A $10 ESP32 can run a tiny language model

A developer has run a 28.9-million-parameter language model on an ESP32-S3 microcontroller costing about $10, producing 9.88 tokens per second locally. The project shows how far model compression and memory-management techniques can stretch hardware built for sensors and embedded devices—not conversational AI.

SlvDev documented the work on GitHub, and the project was recently demonstrated on the Better Stack YouTube channel. The target was TinyStories, a small model developed by Microsoft Research. It is nearly 10,000 times smaller than the kind of frontier model typically associated with generative AI, but it still initially exceeded the ESP32-S3's available memory.

How TinyStories fits on an ESP32-S3

The ESP32-S3 has 520 KB of SRAM and 8 MB of PSRAM, according to The Register. It can also be purchased with up to 16 MB of flash storage. That is nowhere near enough to hold TinyStories at its original precision: using 16-bit weights, the model requires about 60 MB of memory.

SlvDev first used quantization, reducing the numerical precision of the model’s weights from 16 bits to eight or four bits. The trade-off is some loss of accuracy, but the technique cut the weight footprint by 75 percent, from roughly 60 MB to 14.9 MB.

Recommended reading

GLM-5.2 nears the frontier, but safety lags

That still did not fit in the microcontroller’s working memory. The developer then used per-layer-embedding (PLE), a technique borrowed from Google’s Gemma model family, to move most of the weights to flash. About 25 million parameters, or approximately 12 MB, were offloaded this way.

PLE avoids the worst performance penalty associated with reading model weights from slower storage. The offloaded weights are accessed relatively infrequently, while the model’s output head, embeddings, and KV cache remain in PSRAM. Activations are handled in the ESP32-S3's 520 KB of SRAM. The result is a model that needs about 2 MB in active memory rather than 14.9 MB.

Nearly 10 tokens per second, with major limits

The resulting speed is striking for hardware in this class: SlvDev reported 9.88 tokens per second, which The Register notes is faster than the average person can read. But the model’s capabilities are tightly constrained by its size and training.

TinyStories can generate short, reasonably coherent stories on demand. It cannot serve as a general chatbot, generate code, or power an agent. Another small model, Barista, can answer questions at roughly twice the performance, but only about espresso.

That makes the ESP32 project a proof of concept rather than a practical replacement for cloud inference. The achievement is not that a $10 microcontroller can suddenly run a useful general-purpose assistant; it is that careful memory placement makes local generation possible at all.

The same techniques scale to larger local models

The more consequential part of the demonstration is the method. Quantization reduces the size of every weight, while PLE keeps infrequently accessed weights in flash and reserves faster SRAM and PSRAM for the data the model needs more often. That is very different from simply loading a model into memory and avoids the severe slowdown that can result when inference constantly reads weights from storage.

The same general approach can support much larger models on devices with more memory. Google’s Gemma 4-E2B-it, launched in April, uses quantization and PLE offloading to fit a 5.1-billion-parameter vision-language model into slightly more than 1 GB with four-bit weights. More aggressive quantization and offloading can reduce that footprint to around 500 MB.

That model is capable of powering local chatbots and orchestrating agents for tasks such as managing a calendar, although the source notes that users must accept occasional hallucinations. The ESP32-S3 can run TinyStories at a useful reading speed, but its output remains closer to an embedded demonstration than an autonomous assistant.