Sabitlenmiş Tweet
Lucebox
59 posts
Lucebox
@luceboxai
Inference computer for Local AI.
San Francisco Katılım Ocak 2026
17 Takip Edilen847 Takipçiler
Lucebox retweetledi
New LuceBox PR #515: opt-in F32 rollback checkpoints for Qwen3.6-27B Q4 on one RTX 3090.
Fresh standalone test: +6.46% decode throughput. Defaults unchanged; experimental; ~1.15 GiB extra VRAM. I’m going to try to squeeze every bit out of these 3090s
@luceboxai @davideciffa @pupposandro #LocalAI
github.com/Luce-Org/luceb…
English
Lucebox retweetledi
Really happy that we managed to achieve almost 300 tok/s with a 33B MoE, speculative decoding with this type of models is more difficult to optimize 🚗
Sandro@pupposandro
@poolsideai recently released Laguna XS 2.1, an awesome 33B coding model with a DFlash speculative-decoding drafter. Lucebox now runs the pair on a single RTX 3090: - 296 tok/s peak at short context - A flat 152 tok/s at 256K tokens, where the full KV cache would not even fit in 24 GB - ~3,500 tok/s prefill, processing 256K tokens in just 67 seconds Three optimizations got the same GPU from 22 to 152 tok/s at 256K in one pass: a drafter KV ring cache, sliding-window ring caches, and KVFlash paging. And the speculative decoding is lossless: every committed token is exactly one the model itself would have produced. Super proud to support @poolsideai on their work to become the leading western open-source lab. Hope you enjoy it!
English
Lucebox retweetledi
very impressive work from @luceboxai team!
Laguna XS 2.1 now hits 296 tok/s on a single RTX 3090 and holds 152 tok/s at 256K context
nice work across KVFlash paging and ring caching to make long-context local inference really fast.
GIF
Sandro@pupposandro
@poolsideai recently released Laguna XS 2.1, an awesome 33B coding model with a DFlash speculative-decoding drafter. Lucebox now runs the pair on a single RTX 3090: - 296 tok/s peak at short context - A flat 152 tok/s at 256K tokens, where the full KV cache would not even fit in 24 GB - ~3,500 tok/s prefill, processing 256K tokens in just 67 seconds Three optimizations got the same GPU from 22 to 152 tok/s at 256K in one pass: a drafter KV ring cache, sliding-window ring caches, and KVFlash paging. And the speculative decoding is lossless: every committed token is exactly one the model itself would have produced. Super proud to support @poolsideai on their work to become the leading western open-source lab. Hope you enjoy it!
English
Lucebox retweetledi
@povedaaqui Check out @luceboxai they got a rig exactly for that
I think it’s pretty compelling
English
Lucebox retweetledi
@poolsideai recently released Laguna XS 2.1, an awesome 33B coding model with a DFlash speculative-decoding drafter.
Lucebox now runs the pair on a single RTX 3090:
- 296 tok/s peak at short context
- A flat 152 tok/s at 256K tokens, where the full KV cache would not even fit in 24 GB
- ~3,500 tok/s prefill, processing 256K tokens in just 67 seconds
Three optimizations got the same GPU from 22 to 152 tok/s at 256K in one pass: a drafter KV ring cache, sliding-window ring caches, and KVFlash paging.
And the speculative decoding is lossless: every committed token is exactly one the model itself would have produced.
Super proud to support @poolsideai on their work to become the leading western open-source lab. Hope you enjoy it!
Sandro@pupposandro
English
Lucebox retweetledi
I just submitted a DFLASH optimization for Qwen3.6-27B running locally on 2× RTX 3090s.
In controlled A/B/A testing:
• Decode speed: 22.76 → 29.41 tokens/sec (+29.2%)
• Prompt processing: ~95.4 tokens/sec
• End-to-end wall time: 9.4% lower
• Output: byte-identical across all measured runs
The production patch adds a fast speculative-decoding rollback path using F32 SSM checkpoints. It passed the full CUDA build, all 15 current upstream tests, and targeted dual-GPU validation with 43/43 fast rollbacks succeeding and zero fallbacks.
Trade-off: approximately 1.65 GiB of additional persistent GPU memory. Both GPUs are required for this configuration; it will OOM on a single 24 GB RTX 3090.
The PR is now open for review—not merged yet:
github.com/Luce-Org/luceb…
@luceboxai @davideciffa @pupposandro
English
Lucebox coming to @ivanfioravanti very soon!
Ivan Fioravanti ᯅ@ivanfioravanti
If I launch a new startup I have all the hardware needed to play with Local AI good models. - two M3 Ultra 512GB - one DGX Spark (soon two) - one M5 Max - one custom built LuceBox (soon) - one M5 Ultra (as soon as possible) These combined with LiteLLM unlock incredible scenarios! I'll probably offer some inference to my X subscribers in future at least I give you something back 🙏 As soon as I become black belt on LiteLLM 💪
English
Lucebox retweetledi
Strix halo is a very underrated machine. That’s why we decided to put in @luceboxai - really love it so far
Sudo su@sudoingX
one month with the framework desktop 128gb as my amd inference box and i keep waiting for the catch. there isn't one. 128gb of unified memory in a 4.5 liter box. strix halo, ryzen ai max plus 395. framework sent it, amd seeded the program, and my only job was honest numbers wherever they land. so here are the numbers. a 397b moe runs at around 18 tok/s on this thing. a 35b holds a four million token context window. and the open community vulkan drivers beat amd's own rocm stack on my bench, which is the most linux thing imaginable. three and a half weeks of daily serving now, model swapping, long agent sessions, zero crashes, zero driver tantrums, fans i genuinely cannot hear. it sits next to the nvidia box and eats whatever gguf i throw at it. my 24gb card is still the daily driver, but a big memory tier at this price did not exist a year ago. amd showed up, and this quiet little box is the proof.
English
Lucebox retweetledi
Lucebox retweetledi
Another day building and testing @luceboxai machines for our customers
Yes we need a new bigger desk
English
Lucebox retweetledi
So I have joined @luceboxai as a part of the early dev team and will be working with them to push local and opensource AI to its limits, very excited for this !
cheesecake@cheese_cakee_9
career update soon
English
Lucebox retweetledi
Lucebox retweetledi
@NicoTheGreco @nb4ld we actually get ~70tok/s floor thanks to the tuned kernels and software optimisations we did. take took a look at our repo: github.com/Luce-Org/luceb…
English
@nb4ld @luceboxai This thing won’t be able to run anything fast enough at high enough quality to be useful for any actual legal task. 40tok/s is the floor. This can maybe run gemma 4 now that fast. Nothing smarter than that.
English
Lucebox retweetledi
Thanks for the purchase and support @nb4ld! Can’t wait to get it delivered and hear your feedback.
We’ve received many applications for @luceboxai recently. I'm personally having a call with each one. We knew there was a problem to solve here, but we didn’t expect the demand to be this strong.
Now working hard to get everyone their machine soon!
English
Lucebox retweetledi
Lucebox retweetledi
Lucebox retweetledi
The real problem of Apple Silicon at the moment is that everything is CUDA driven, luckily with AI Agents you cn convert nearly every project to run on MLX, but it takes time and effort. I hope things will improve over time.
In the meantime... I hope to have my @luceboxai soon to make some tests!
English
Lucebox retweetledi
Lucebox retweetledi
Kicking off my LFX term
Contributing to @Ceph
Contributing to @luceboxai
Free and Open Source Software for the win!!
English