Brian Roemmele@BrianRoemmele
I’m in the garage lab tonight under the workbench lights, running the first real stress test of flash-backed inference on a larger model using ordinary consumer hardware.
Setup is deliberately constrained to mirror the spirit of the $8 ESP32-AI experiment: an older Ryzen 5 desktop with only 24 GB of DDR4 RAM, a single 1 TB NVMe SSD (PCIe 3.0), and no discrete GPU.
I’m loading a 13B-parameter model quantized to Q4_K_M (roughly 7.5 GB on disk) that deliberately exceeds available system RAM when fully expanded.
Using llama.cpp with memory-mapping enabled (--mmap), the weights stay on the SSD and the OS pages in only the active tensors and a small working set.
I’m also testing a simple sparsity-aware loader inspired by the “LLM in a flash” approach—windowing recently activated neurons and reading larger contiguous blocks to keep I/O efficient.
I’m measuring end-to-end tokens per second, peak RAM usage, SSD bandwidth, and how cleanly the narrative stays coherent across multiple turns while everything remains fully offline.
This matters because it is the first practical bridge from the microcontroller “toy” to something a regular person can actually use. If a model larger than the machine’s RAM can still produce useful, private, narrative personal-knowledge nodes by treating flash as the primary store, then the severe constraints that made the ESP32 work become a feature rather than a limitation.
It proves that the same design philosophy—keep the dense core small, leave the bulk in cheap storage, accept the limits and design inside them—scales beyond $8 chips into everyday computers.
Success tonight would show that offline, low-cost, domain-specialized AI are no longer a curiosity; they are immediately buildable in a garage.
And more importantly this may be a be way to radically reduce costs in enterprise AI and even data centers.
If my experiments are confirmed this could also save millions of dollars in AI costs for the larger AI companies.
All from my garage.
I’ll be back in a “flash”.
