A 28.9M-parameter LLM runs on an $8 chip

A 28.9-million-parameter language model is now running entirely on an ESP32-S3, a microcontroller that costs about $8. The project generates text at roughly 9 tokens per second, writing each token to a small display connected to the chip—without sending data to a server.

The ESP32-S3 provides just 512KB of SRAM, plus 8MB of PSRAM and 16MB of flash. Yet the model occupies 14.9MB at 4-bit precision, with around 25 million parameters stored in a flash-based lookup table.

How the model fits on the ESP32-S3

The central technique comes from Google’s Per-Layer Embeddings, used in the Gemma models. Instead of keeping the entire model in fast memory, the project stores most parameters in slow flash and reads only the few rows needed for each token—about 450 bytes, or roughly six rows.

That leaves the smaller computation-heavy portion in fast SRAM. The memory layout is divided as follows:

Recommended reading

GenStorAIGE says SSDs make 8 RTX 5090s act like 46

• SRAM: the “thinking” core used for every token
• PSRAM: the output head and working memory
• Flash: the 25-million-parameter table

The result is a measured speed of 9.5 tokens per second end to end, or 9.7 tokens per second for pure computation. The project’s author says this is the first known attempt to apply the approach to a chip this small.

A previous language model run on comparable hardware had only 260,000 parameters, making this model roughly one hundred times larger by parameter count. The flash table makes that increase possible because most of the model is never loaded into fast memory at once.

What the model can—and cannot—do

The model was trained on TinyStories, a dataset of short synthetic stories created by Ronen Eldan and Yuanzhi Li at Microsoft Research. It can produce short, generally coherent narratives, but it cannot answer questions, follow instructions, write code, or provide factual knowledge.

Those limitations come from the model’s small reasoning component. The memory-saving architecture allows a larger parameter count, but it does not turn the model into a general-purpose assistant. The project’s significance is primarily architectural: it demonstrates that a relatively large language model can run locally on a tiny microcontroller.

The repository includes the firmware, wiring instructions, and flashing steps in firmware/esp32_llm/README.md. Training, ablation, and quantization code is available in src/ and experiments/, while RESULTS.md documents the method and on-chip measurements.

The project also preserves its development history, including a parameter-accounting bug that inflated an early result. The corrected figures and the reasons for the changes are recorded in the commit history and RESULTS.md. It builds on Andrej Karpathy’s llama2.c, which helped establish that small language models could be trained and run in plain C.