JustVugg

Loomabase is an Offline-first synchronization engine for SQLite and PostgreSQL, built in Rust with column-level CRDTs, authoritative partial replicas, and secure multi- tenancy. #PostgreSQL #Rust #Sqlite github.com/JustVugg/looma…

A 744B-parameter AI model just ran on a consumer machine with 25GB of RAM and no GPU. Colibri is a pure-C inference engine for GLM-5.2, a Mixture-of-Experts model. Although the full model contains 744B parameters, only about 40B activate per token. Colibri keeps the dense 9.9GB core in memory. The remaining 370GB of experts stays on an SSD and is streamed only when needed. Key details: > Zero runtime dependencies > No graphics card required > Full local inference > Per-layer expert caching > Optional speculative decoding The tradeoff is speed. Cold generation can require up to 11GB of disk reads per token, reducing output to roughly one token every 10–20 seconds. Colibri will not replace cloud inference yet, but it challenges the idea that an entire model must fit in memory before it can run.

@KluepfelMark I'm putting all the benchmarks they're sending us in the readme and they're really interesting!

Anyone tried Colibri ? Would really like to hear from you. github.com/JustVugg/colib…

Colibrì has surpassed 10,000 stars! Thank you all for your support! Thank you to the community that helps me! Thank you to each and every one of you who has helped me! The road is long and the work is so much! Let's keep it up!

💥 Colibri 744B is HERE — and it runs with NO GPU ♠ a frontier-scale model that flips the whole "you need a datacenter" assumption on its head 🔹 744B parameters running without dedicated GPU hardware 🔹 Proves massive models can go leaner & more accessible than anyone expected 🔹 Lowers the barrier for devs, researchers & small teams to run serious AI 🔹 Built for those who want frontier power without frontier infrastructure costs 🔥 Watch the full breakdown below 👇 youtu.be/jxML3S5C-8Y

25G 内存跑 GLM-5.2? 说实话我看到这个最开始感觉是不是假的, GLM-5.2 总计 744B 激活参数 40B. 8bit量化光激活参数部分就需要40G内存. 即使 4bit 量化只加载那40B激活参数也要20G内存, 而且要是这么搞, 内存需要疯狂加载每个token推理时需要的参数, 那可就不是卡内存带宽而是卡硬盘带宽了. 按照现在NVME也就1-4GB/s的读取速度, 加载20G才能完成一个token的推理, 那么最快也就5秒吐一个字了. 但是这个框架能做到 2.2-2.8 tok/forward! 虽然也很慢, 但是思路很值得借鉴: 他们把 GLM-5.2 的激活细节分析了一波. 发现 744B 的 MoE 每个 token 确实激活了 40B, 但是 token 推理的时候真正会变的只有路由专家那一块, 大约 11GB (int4), 有优化空间! 所以只需要动态加载这部分. 注意力、共享专家、embedding 这些稠密部分约 17B, 常驻内存也就 9.9GB. 剩下 21,504 个路由专家 (~370GB) 全扔磁盘, 按需流式加载. 流式加载采用的是 per-layer LRU + 热点 pin + 系统页缓存当 L2. 将优化做到了极致. 最终变成了冷启动大概每个 token 读 ~11GB (75 层 × 8 专家), 官方数据冷启动的时候大概 0.05–0.1 tok/s. 热起来之后命中缓存, 磁盘压力下来速度就上升了. 除此之外他们还搞了个实验性的 router-lookahead (PILOT), 就是用当前层的 post-attention 状态猜下一层路由进行预加载, 实测可预测性达到了 71.6%, 效果拔群. 然后再加上 GLM 自带的 MTP 投机解码头, 接受率能到 39–59%, 一次 forward 就能输出 2.2–2.8 个 token了. 如果你的确没显卡, 但是有磁盘阵列或者NVME阵列, 那么这个框架是完全可以试一试的. > #colibri #glm52 链接: github.com/JustVugg/colib…

You can now run 744B parameter model on a laptop with just 25GB RAM. an Italian developer just open-sourced Colibrì, a tiny C engine that streams experts straight from disk on demand. → pure C. zero dependencies. → no GPU. no python at runtime. he built the whole thing on a 12-core laptop and said it runs "on a machine that costs less than one H100 fan." 100% open source.

GLM-5.2をVRAM25GBのGPUで動かしてみたい このリポジトリでは744Bもの大パラメータのMoEモデルのDenseな部分はどこにあるのかに注目しました。すると推論のコアになっている部分の重みは9.9GBのINT4部分であることがわかりました。TPSはもちろん遅いですが本物です github.com/JustVugg/colib…

@evereq @MikeBradleyAI Thanks @evereq thanks for this benchmark and test!

Cooking something cool with 1Tb RAM (DDR4 1866) / 3.2Tb NVMe (RAID 0) and Colibri by @justvugg running GLM-5.2 ;) Stay tuned! CC: @mikebradleyai when Colibri in ODS?

@HuggingModels I recommend the new link, the old one has problems with mtp int4! Thanks @HuggingModels for the post! Really thanks! huggingface.co/mateogrgic/GLM…

🔗 Explore the full model details here: huggingface.co/jlnsrk/GLM-5.2…

@zaunere Thanks so much!! Have you used it for any projects?

@justvugg Love mnem - "evolving state" - everyone forgets temporality.

Loomabase is an Offline-first synchronization engine for SQLite and PostgreSQL, built in Rust with column-level CRDTs, authoritative partial replicas, and secure multi- tenancy. #PostgreSQL #Rust #Sqlite github.com/JustVugg/looma…

WTF did I just find?! Colibrì runs GLM-5.2’s 744B-parameter model locally on a consumer PC with roughly 25 GB of RAM. 🚫 No cloud API. 🚫 No server rack. 🚫 No $30,000 GPU. The engine is written in pure C. It keeps the dense portion in RAM, stores roughly 370 GB of experts on your NVMe, then streams the experts it needs for each token. The catch: it’s slow as hell. But that almost doesn’t matter. Someone just proved you can make a frontier-sized MoE model answer questions on hardware sitting under your desk. I have a laptop 4090, WSL2, and several terrible ideas. Who wants to see me get this monster running? github.com/JustVugg/colib…

You can run (very slowly) (almost) frontier GLM-5.2 on a 25GB machine. Sexy. A bit hacky, but nonetheless a beautiful open source contrbution showing that memory is not a hard wall. Now, be a good sould and give them a well deserved star on github ⬇️ github.com/JustVugg/colib…

Colibrì New update: Metal backend: 2.06 tok/s on an M5 Max CUDA + continuous batching POWER8 VSX kernels Attention overflow fix github.com/JustVugg/colib…

@HarakiriInu @bokiko If you like, add a pull request—you're absolutely welcome!

@bokiko @justvugg if you take my recipe you would probably run it with 1.2 tok/sec

I just ran a 744B parameter model on my Mac Mini M4 Pro with only 48GB RAM The model is 370GB on disk. the machine has 48GB of memory... sit with that for a second! the trick is colibrì by @justvugg: 2,400 lines of pure C, zero dependencies. it keeps the ~10GB dense core resident in RAM and streams the other 370GB of experts off the SSD on demand, riding Apple's unified memory. My numbers: now the first M4 Pro row in the project's benchmark table: • CPU only: 0.18 tok/s • Metal backend: 0.30 tok/s — faster than a 32-core Ryzen 9950X with 2.5x the RAM • SSD serving 19MB expert reads at 6.6 GB/s like it's nothing to be clear: 0.30 tok/s is a paragraph every ~10 minutes. this is not a daily tool. its a science exhibit that should not work... but does! full writeup: github.com/JustVugg/colib…

@slimbook @enrub_en @chenzeling4 Thank you so much for your support and benchmarks!

@justvugg @enrub_en @chenzeling4 You're very welcome! Congratulations on what you've built with Colibri, the work behind it really shows. We'll be following the project closely and sharing its progress with our community. Keep it up!

Our customers are brilliant, and their Slimbooks become real test benches. @enrub_en is pushing Colibri/GLM-5.2 744B MoE by @JustVugg on his EVO laptop—thanks to @chenzeling4 for sharing it. Not production speed, but a brilliant engineering demo of how far local AI can go 🔥🐧

@chenzeling4 @slimbook 0,15 tok/s GLM 5.2 This is fucking insane! Absurd speed, yes, but I'm running a 744B model in my "shitty" non-vidia laptop What do you think? @0xSero @sudoingX 🔥

@enrub_en @chenzeling4 @slimbook @0xSero @sudoingX Thanks for your benchmark and test!!!Really thanks!

744B parameters. On a laptop. With 25GB RAM. Colibri runs GLM-5.2 (744B MoE) in pure C with zero dependencies. The trick: only ~40B params activate per token, so it keeps the dense part resident and streams experts from disk on demand. A single 2,400-line C file. No GPU, no BLAS, no Python at runtime. This shouldn't work. But it does. ⭐ 2.1K #AI #OpenSource github.com/JustVugg/colib… Follow for daily dev finds 🔔