David Wang
107 posts
@finn_fergus yea we have some ideas for this while keeping chain :) mostly tree drafting also complicates the engine implementation especially with overlap scheduling etc
English
I guess the right q is what's better for fixed verification cost though.
In principle trees ought to be strictly better (lines being degenerate trees) if the drafter is calibrated well enough that some algo can branch when uncertain (like arxiv.org/pdf/2604.12989). But then maybe the calibrations the hard bit
English
As always, it’s a lot more nuanced than this. In spec decoding, the predominant cost is target verification, the chain vs tree tradeoff is well known, and at production concurrencies you’re often better off training a better draft model than doing tree drafting.
Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTex
I want to bring your attention to JetSpec because it looks strictly smarter and stronger than previous speculative decoding and block diffusion approaches (yes, again). Avg 1000 t/s single stream with Qwen-8B on B200. Basically, you can better utilize compute at any batch size.
English
vLLM has even recently removed tree attention support github.com/vllm-project/v…
English
David Wang retweetledi
@_EldarKurtic @jvmncs @charles_irl we’ve been collaborating with the zlab for a while and this was written before dflash training was well supported in open source
English
@jvmncs @_dcw02 Any specific reason not to use the existing libraries for speculator training? There is already quite a few like vllm’s speculators, torchspec, etc. @charles_irl
English
buried in this post is a note about our from-scratch framework for draft model training by @_dcw02
I’ve used it to chase down a few research ideas, and damn does that thing rip. such a joy to use
Charles 🎉 Frye@charles_irl
Speculation Is All You Need. In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x. Over 1k output tps for 3.5 122B-A10B on a B200. Read the blog for why we're all-in on spec dec. modal.com/blog/spec-is-a…
English
@dysondunbar @charles_irl @Alibaba_Qwen I played too much pokemon champions and forgot. It’s training and will be released later.
English
@charles_irl @Alibaba_Qwen Wait what happened to qwen 3.6 27B dflash model? Was it unable to beat the mtp baseline?
English
David Wang retweetledi
Speculation Is All You Need.
In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x.
Over 1k output tps for 3.5 122B-A10B on a B200.
Read the blog for why we're all-in on spec dec.
modal.com/blog/spec-is-a…
English
English
David Wang retweetledi
Try them out! 🚀 Really enjoyed the amazing collaboration with the Modal team. The retrained DFlash draft models now support longer contexts, making them a better fit for agentic workloads.
Charles 🎉 Frye@charles_irl
Speculation Is All You Need. In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x. Over 1k output tps for 3.5 122B-A10B on a B200. Read the blog for why we're all-in on spec dec. modal.com/blog/spec-is-a…
English
i asked gpt to tier list something that actually matters
Center for AI Safety@CAIS
What biases do AIs have? It turns out, AIs show strong favoritism toward specific people, countries, and companies. Our interactive AI Values Dashboard tracks who Claude Fable and other AIs favor most. Keep scrolling to learn who is Fable’s favorite politician 🧵
English
David Wang retweetledi
🚀 New blog: The next generation of speculative decoding: DFlash and Spec V2
DFlash + Spec V2 hit >4.3X baseline throughput for LLM inference, now the default speculative decoding engine in SGLang! Together with @modal and z-lab.ai, our jointly-released DFlash drafter for Qwen 3.5 397B-A17B beats both baseline and native MTP in every setting we benchmarked:
1️⃣ >4.3X baseline & 1.5X native MTP throughput (concurrency 1, HumanEval, 8xB200)
2️⃣ Block diffusion drafter: a full token block in one forward pass
3️⃣ KV injection: target-model features fed into every draft layer’s KV cache for higher acceptance
4️⃣ Spec V2 overlap scheduler: +33% end-to-end
Read the code, deploy a DFlash server, and start experimenting!
English
David Wang retweetledi
🚀 DFlash now runs on SGLang's new default speculative-decoding engine, Spec V2.
⚡️ Hitting >4.3× baseline throughput (1.5× over native MTP) on Qwen 3.5 397B-A17B. Same quality, more speed!
⭐ github.com/z-lab/dflash
English
couldn't have asked for better people to do this with. @sgl_project @modal and the z-lab team @jianchen1799 @yesheng_liang @zhijianliu_ 💚
English
9+ accept lengths on coding workloads
generic drafter btw
qwen 397b 4x faster
repro btw
dflash go brrr
Modal@modal
We worked with @lmsysorg and z-lab.ai to - integrate DFlash spec into @sgl_project - make it faster with overlap - train a DFlash drafter for @Alibaba_Qwen 397B-A17B The result: up to 4.3x greater throughput over baseline and 1.5x over native MTP.
English
@charles_irl @modal @_gongy @saatwiknagpal @DevenNavani @be4ncurd @emilyhanyf @hsubbaraj @racerfunction @luiscape @jonobelotti_IO @mma12261 @teenychairs machine god go brrr
English
@modal @_gongy @_dcw02 @saatwiknagpal @DevenNavani @be4ncurd @emilyhanyf @hsubbaraj @racerfunction @luiscape @jonobelotti_IO @mma12261 @teenychairs @_dcw02 i apologize for screen-capping our hinge-free DMs and then sharing it with the whole world
English
Tried to squeeze the most important bits about the entire stack for cloud deployment of transformer inference, from application layer concerns to hardware, debugging, and o11y, into one talk. Had to operate at a very high tok/s!
youtube.com/watch?v=ZUdIsR…
YouTube
English
David Wang retweetledi
Great to see DFlash is being used in MiMo!
Xiaomi MiMo@XiaomiMiMo
🚀 1,000+ TOKENS/S ON A 1T MODEL! 🚀 We are thrilled to release Xiaomi MiMo-V2.5-Pro-UltraSpeed in collaboration with @TileRT_AI , breaking the 1,000 tokens/s output speed on a 1 Trillion parameter model for the FIRST TIME! Not wafer-scale integration like Cerebras. Not pure on-chip SRAM chips like Groq. We achieve 1,000 tps on a 1T MoE model using just a SINGLE, STANDARD 8-GPGPU NODE. Read the full technical deep dive：mimo.xiaomi.com/blog/mimo-tile… Want to experience the future of real-time AI? 👉 Apply for UltraSpeed now: platform.xiaomimimo.com/ultraspeed ⏳ Limited-Time Access: Application-based · Jun 8 – Jun 23 (PDT) 💬 Chat Experience: Completely FREE for a limited time — try the blazing-fast web chat now. ⚡ UltraSpeed API: Just 3x the price for a ~10x boost in output experience. 🤝 Enterprise & Large-Scale Needs: business-mimo@xiaomi.com
English
David Wang retweetledi
Amazing to see DFlash helping push Xiaomi MiMo to 1,000+ tokens/s on a 1T model.
Happy to see our work supercharging one of the best open-source LLMs! 🚀
Xiaomi MiMo@XiaomiMiMo
🚀 1,000+ TOKENS/S ON A 1T MODEL! 🚀 We are thrilled to release Xiaomi MiMo-V2.5-Pro-UltraSpeed in collaboration with @TileRT_AI , breaking the 1,000 tokens/s output speed on a 1 Trillion parameter model for the FIRST TIME! Not wafer-scale integration like Cerebras. Not pure on-chip SRAM chips like Groq. We achieve 1,000 tps on a 1T MoE model using just a SINGLE, STANDARD 8-GPGPU NODE. Read the full technical deep dive：mimo.xiaomi.com/blog/mimo-tile… Want to experience the future of real-time AI? 👉 Apply for UltraSpeed now: platform.xiaomimimo.com/ultraspeed ⏳ Limited-Time Access: Application-based · Jun 8 – Jun 23 (PDT) 💬 Chat Experience: Completely FREE for a limited time — try the blazing-fast web chat now. ⚡ UltraSpeed API: Just 3x the price for a ~10x boost in output experience. 🤝 Enterprise & Large-Scale Needs: business-mimo@xiaomi.com
English
me with 0 commits to the modal monorepo 👀
Erik Bernhardsson@bernhardsson
@schrockn Yeah agree PR count is slightly better metric than LoC
English