JustVugg

Loomabase is an Offline-first synchronization engine for SQLite and PostgreSQL, built in Rust with column-level CRDTs, authoritative partial replicas, and secure multi- tenancy. #PostgreSQL #Rust #Sqlite github.com/JustVugg/looma…

25G 内存跑 GLM-5.2? 说实话我看到这个最开始感觉是不是假的, GLM-5.2 总计 744B 激活参数 40B. 8bit量化光激活参数部分就需要40G内存. 即使 4bit 量化只加载那40B激活参数也要20G内存, 而且要是这么搞, 内存需要疯狂加载每个token推理时需要的参数, 那可就不是卡内存带宽而是卡硬盘带宽了. 按照现在NVME也就1-4GB/s的读取速度, 加载20G才能完成一个token的推理, 那么最快也就5秒吐一个字了. 但是这个框架能做到 2.2-2.8 tok/forward! 虽然也很慢, 但是思路很值得借鉴: 他们把 GLM-5.2 的激活细节分析了一波. 发现 744B 的 MoE 每个 token 确实激活了 40B, 但是 token 推理的时候真正会变的只有路由专家那一块, 大约 11GB (int4), 有优化空间! 所以只需要动态加载这部分. 注意力、共享专家、embedding 这些稠密部分约 17B, 常驻内存也就 9.9GB. 剩下 21,504 个路由专家 (~370GB) 全扔磁盘, 按需流式加载. 流式加载采用的是 per-layer LRU + 热点 pin + 系统页缓存当 L2. 将优化做到了极致. 最终变成了冷启动大概每个 token 读 ~11GB (75 层 × 8 专家), 官方数据冷启动的时候大概 0.05–0.1 tok/s. 热起来之后命中缓存, 磁盘压力下来速度就上升了. 除此之外他们还搞了个实验性的 router-lookahead (PILOT), 就是用当前层的 post-attention 状态猜下一层路由进行预加载, 实测可预测性达到了 71.6%, 效果拔群. 然后再加上 GLM 自带的 MTP 投机解码头, 接受率能到 39–59%, 一次 forward 就能输出 2.2–2.8 个 token了. 如果你的确没显卡, 但是有磁盘阵列或者NVME阵列, 那么这个框架是完全可以试一试的. > #colibri #glm52 链接: github.com/JustVugg/colib…

You can now run 744B parameter model on a laptop with just 25GB RAM. an Italian developer just open-sourced Colibrì, a tiny C engine that streams experts straight from disk on demand. → pure C. zero dependencies. → no GPU. no python at runtime. he built the whole thing on a 12-core laptop and said it runs "on a machine that costs less than one H100 fan." 100% open source.

GLM-5.2をVRAM25GBのGPUで動かしてみたい このリポジトリでは744Bもの大パラメータのMoEモデルのDenseな部分はどこにあるのかに注目しました。すると推論のコアになっている部分の重みは9.9GBのINT4部分であることがわかりました。TPSはもちろん遅いですが本物です github.com/JustVugg/colib…

@evereq @MikeBradleyAI Thanks @evereq thanks for this benchmark and test!

Cooking something cool with 1Tb RAM (DDR4 1866) / 3.2Tb NVMe (RAID 0) and Colibri by @justvugg running GLM-5.2 ;) Stay tuned! CC: @mikebradleyai when Colibri in ODS?

@HuggingModels I recommend the new link, the old one has problems with mtp int4! Thanks @HuggingModels for the post! Really thanks! huggingface.co/mateogrgic/GLM…

🔗 Explore the full model details here: huggingface.co/jlnsrk/GLM-5.2…

@zaunere Thanks so much!! Have you used it for any projects?

@justvugg Love mnem - "evolving state" - everyone forgets temporality.

Loomabase is an Offline-first synchronization engine for SQLite and PostgreSQL, built in Rust with column-level CRDTs, authoritative partial replicas, and secure multi- tenancy. #PostgreSQL #Rust #Sqlite github.com/JustVugg/looma…

WTF did I just find?! Colibrì runs GLM-5.2’s 744B-parameter model locally on a consumer PC with roughly 25 GB of RAM. 🚫 No cloud API. 🚫 No server rack. 🚫 No $30,000 GPU. The engine is written in pure C. It keeps the dense portion in RAM, stores roughly 370 GB of experts on your NVMe, then streams the experts it needs for each token. The catch: it’s slow as hell. But that almost doesn’t matter. Someone just proved you can make a frontier-sized MoE model answer questions on hardware sitting under your desk. I have a laptop 4090, WSL2, and several terrible ideas. Who wants to see me get this monster running? github.com/JustVugg/colib…

You can run (very slowly) (almost) frontier GLM-5.2 on a 25GB machine. Sexy. A bit hacky, but nonetheless a beautiful open source contrbution showing that memory is not a hard wall. Now, be a good sould and give them a well deserved star on github ⬇️ github.com/JustVugg/colib…

Colibrì New update: Metal backend: 2.06 tok/s on an M5 Max CUDA + continuous batching POWER8 VSX kernels Attention overflow fix github.com/JustVugg/colib…

@HarakiriInu @bokiko If you like, add a pull request—you're absolutely welcome!

@bokiko @justvugg if you take my recipe you would probably run it with 1.2 tok/sec

I just ran a 744B parameter model on my Mac Mini M4 Pro with only 48GB RAM The model is 370GB on disk. the machine has 48GB of memory... sit with that for a second! the trick is colibrì by @justvugg: 2,400 lines of pure C, zero dependencies. it keeps the ~10GB dense core resident in RAM and streams the other 370GB of experts off the SSD on demand, riding Apple's unified memory. My numbers: now the first M4 Pro row in the project's benchmark table: • CPU only: 0.18 tok/s • Metal backend: 0.30 tok/s — faster than a 32-core Ryzen 9950X with 2.5x the RAM • SSD serving 19MB expert reads at 6.6 GB/s like it's nothing to be clear: 0.30 tok/s is a paragraph every ~10 minutes. this is not a daily tool. its a science exhibit that should not work... but does! full writeup: github.com/JustVugg/colib…

@slimbook @enrub_en @chenzeling4 Thank you so much for your support and benchmarks!

@justvugg @enrub_en @chenzeling4 You're very welcome! Congratulations on what you've built with Colibri, the work behind it really shows. We'll be following the project closely and sharing its progress with our community. Keep it up!

Our customers are brilliant, and their Slimbooks become real test benches. @enrub_en is pushing Colibri/GLM-5.2 744B MoE by @JustVugg on his EVO laptop—thanks to @chenzeling4 for sharing it. Not production speed, but a brilliant engineering demo of how far local AI can go 🔥🐧

@chenzeling4 @slimbook 0,15 tok/s GLM 5.2 This is fucking insane! Absurd speed, yes, but I'm running a 744B model in my "shitty" non-vidia laptop What do you think? @0xSero @sudoingX 🔥

@enrub_en @chenzeling4 @slimbook @0xSero @sudoingX Thanks for your benchmark and test!!!Really thanks!

744B parameters. On a laptop. With 25GB RAM. Colibri runs GLM-5.2 (744B MoE) in pure C with zero dependencies. The trick: only ~40B params activate per token, so it keeps the dense part resident and streams experts from disk on demand. A single 2,400-line C file. No GPU, no BLAS, no Python at runtime. This shouldn't work. But it does. ⭐ 2.1K #AI #OpenSource github.com/JustVugg/colib… Follow for daily dev finds 🔔

@FieldToFuture Really thanks!! if it's possible to open an github issue so we can keep the benchmarks updated on our readme!

Testing this with a typical gaming desktop> 128GB DDR4 RAM, 500GB Gen 4 PCIE SSD, 2 RTX 3060. Should have results in a few days when the drive arrives. JustVugg/colibri: Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦 @justvugg

@blogtheristo Thanks so much for the suggestion! I'll definitely improve that part too!

/ One friendly suggestion: print an honest tok/s estimate at chat startup so newcomers read "slow but working," not "broken." Tiny change, big first-run win. Genuinely impressive work @justvugg — drop it a ★ github.com/JustVugg/colib…

1/ Ran a usability study on @justvugg's Colibrì — the pure-C engine that streams GLM-5.2 (744B MoE) off disk to run on a 25GB-RAM machine. Verdict: one of the better-designed systems-level OSS projects I've read. Thread 🧵

2/ What stands out: • `coli doctor` / `plan` — read-only readiness checks before you touch 370GB • "Honest Performance Numbers" + community benchmark table = real trust • OpenAI-compatible API on day one • Zero runtime deps. Nothing to break.