Alex Nails
2.9K posts
a lot of the benches beloved by model connoisseurs are things like "PostTrainBench" and "WeirdML", and we're probably due for another good kernel benchmark soon
the labs will soon have to choose between "pushing the frontier" via headline numbers and self-commoditization
English
Alex Nails retweetledi
The standout feature in SGLang v0.5.13 is BCG (Breakable CUDA Graph). It delivers prefill efficiency comparable to PCG (Piecewise CUDA Graph), while being significantly more flexible and compatible with advanced optimizations.
BCG is also a powerful debugging tool — it enables eager execution inside CUDA graph replay, so you can easily print debug info or inspect intermediate states.
Prototype by @csy_789. @Oasis_a19 turned it into a production-ready prefill optimization. I was fortunate to pick the name for this awesome technique. 😊
SGLang@sgl_project
🎉 SGLang v0.5.13 is out! First, new model support! Nemotron 3 Ultra, Step-3.7-Flash, Command A+, plus new diffusion models: Cosmos3, FLUX.2-Klein, Ideogram 4, LingBot-World, SANA-WM, and Ernie-Image. Here are the highlights for this release: - Speculative Decoding V2 is now the default! Tree drafting (topk>1) for faster generation - Breakable CUDA Graphs now make prefill faster - Qwen 3.5 runs faster on NVIDIA Blackwell with new GDN kernels - HiCache with UnifiedTree on by default for hybrid SWA/Mamba models - SGLang-Diffusion now supports realtime generation! Plus progressive resolution - Multiple performance and feature updates for DeepSeek V4 Thanks to our amazing partners and model makers: @NVIDIAAI @AMD @intel @awscloud @boson_ai @cohere @bfl_ml @ideogram_ai @deepseek_ai @Kimi_Moonshot @Alibaba_Qwen @StepFun_ai @Baidu_Inc @robbyant_brain
English
@DynamicWebPaige @AMD right now there are some paths that just involve having glue logic in the middle but it’s not top performance. the reverse is happening: we will start to see more Cuda Tile IR abstractions where the top layer framework used will have more semantic information expressed downward
English
👋 What's the status for machine learning compilers these days, RE: moving from CUDA to ?
If I want to make CUDA-compatible code run well on @AMD hardware, or on TPUs, or something like Cerebras or Groq or similar, is there still manual rewriting and verification required (even with LLM-enabled tools like @antigravity or Claude Code)?
What's the deal with HIPIFY or other auto-converters?
English
Alex Nails retweetledi
NYC, we're bringing the inference + finance crowd together for #NYTechWeek @Techweek_!
SGLang Happy Hour: AI Infra in Finance
🕤Wed, June 3 · 6–9 PM ET
📍1/2 Bond St, New York
Co-hosted with @HOFCapital, @CrusoeAI, @CloudflareDev, @ArklexAI. Lightning talks from inference engineers and researchers shipping into trading, research, compliance, and risk, followed by an open happy hour for networking. More surprise speakers to be announced — stay tuned 👀
Expected attendees from leading quant funds, banks, and trading firms, including Jane Street, Citadel, Two Sigma, Goldman Sachs, Bloomberg, among others.
We've also got a bartender on site and a full bar. Come have a drink with us!
Limited space. RSVP 👇
partiful.com/e/p74X9KDrgoLa…
English
@giaccoangelo Shoyu and Chef’s Corner were my two favorite meals in Latvia! Also, I rec trying the black currant black balsam. I had also gone somewhere in city centre to try traditional latvian food, worth researching 👀
English
"Drink Riga Black Balsam, the local 45% bitter, at least once. Do not drink it twice." absolute banger from claude
angelo@giaccoangelo
#1 travel hack - as soon as you arrive in a new city ask claude 'give me the tyler cowen guide to [location]. accordingly, what should i do in an x hours stay?'
English
@PatrickToulme hard agree (and maybe even a few hotter takes on my end 🤣). this was a great read! thanks for writing it up
English
Alex Nails retweetledi
@Is36E Do you have insight on when there will be better PyTorch and MLX interop? At least from the lens of DLPack, there is PyTorch stream emit work and then MLX support for proper capsule emission (technically this is nanobind issue) is overhead. Claude wrote: github.com/ml-explore/mlx…
English
how we implemented Moondream inference on Apple Silicon (spoiler: we don't use MLX)
⬇️ (1/N)
murat 🍥@mayfer
what if i told you... computer use can be faster on local models moondream3 with its photon update today that gives it mac support can see your screen and use it with 1s latency, ty @vikhyatk here we have whisper+qwen+moondream triple model pipeline working offline flawlessly
English
Alex Nails retweetledi
Training DeepSeek V4 @deepseek_ai at scale? SGLang + Miles is the Day 0 path. @lmsysorg
Miles and SGLang enable full-parameter RL training for DSV4 with stability, efficiency, and broad hardware support.
✅ Verified stability
- Rollout Routing Replay (R3) and indexer replay (experimental)
- Tensor-level validation across the Miles & Megatron mixed-precision training stack
- Step-0 train-inference diff: ~0.02–0.03
✅ Efficient full-parameter RL
- DP / TP / SP / EP / PP / CP support
- Tilelang attention and indexer kernels
- FP8/BF16 rollout and FP8/BF16 training support
✅ Broad hardware support
- Verified training on NVIDIA Hopper and Grace Blackwell clusters
- Ready for DeepSeek V4 RL from Day 0
This is the exclusive Day 0 path to scale DeepSeek V4 with rock-solid reliability. Full technical docs & setup guide below! 👇
#DeepSeekV4 #SGLang #RL
English
@andrey_cheptsov @lmsysorg Hi Andrey, are you in the SGLang slack? Feel free to DM there or here and hopefully I can route you to the right people :)
English
We’re looking for access to a 2-node MI300X/MI350X/MI355X cluster to test and benchmark native @lmsysorg SGLang prefill-decode disaggregation on dstack. Happy to connect with anyone who can help.
Andrey Cheptsov@andrey_cheptsov
Is anyone running prefill-decode disaggregation with @lmsysorg SGLang on @AMD? We’re currently testing its native support via dstack. If you’re already using it, please ping me. I have a couple of questions.
English
Alex Nails retweetledi
New course available! Efficient Inference with SGLang: Text and Image Generation is live.
LLM inference gets expensive fast—mostly due to redundant computation. This course shows how to reduce that using SGLang, with KV cache and RadixAttention, and how to apply the same ideas to faster image generation.
Built with @lmsysorg and @radixark, taught by Richard Chen.
Enroll for free: hubs.la/Q04b0F1J0
English
bizarre move to drop gemma 4 without coordinating a release with the inference engines
English
@charles_irl @__endif Hi! We do have docker images available if you’d like to try out
cu12.9: hub.docker.com/layers/lmsysor…
cu13: hub.docker.com/layers/lmsysor…
English
@jackcookjack You mention 4/6 runs efficiently but on blackwell, but that IF3/IF4/IF6 can only be simulated at higher data types? Any novel packing schemes that further improve performance ? (apologies if this is in the paper, I will try to read later)
English
NVFP4 allows models to be quantized to 4 bits without too much performance degradation, but can we push 4-bit performance even further?
Today, we're releasing a new class of low-precision block-scaled data types that natively adapt to your input data: for 4-bit quantization, IF4 (Int/Float 4) allows each scaled group of 16 values to be saved as FP4 or INT4 depending on which option offers less error. Selections are recorded using the scale factor’s sign bit, which is unused in NVFP4, allowing IF4 to offer better performance with no memory overhead!
Our data types provide better downstream accuracy in LLMs, they can be implemented efficiently in next-generation hardware accelerators, and they reveal some interesting insights about low-bit quantization! 🧵
English
@vikhyatk There is an article about this that you might like! pytorch.org/blog/hybrid-mo…
English
@casper_hansen_ @TheZachMueller one of our latest entries was cookbook.sglang.io/autoregressive…, what do you think of the configuration tips and basic configuration for the model deployment section?
English
@casper_hansen_ @TheZachMueller would you want this in the sglang cookbook?
English
every inference engine should have a section in their docs with exact commands to achieve best possible tokens/s on the most popular models
i'm told kimi k2.5 can run at 300 tokens/s on B200s if you run nvfp4 with speculative decoding in open-source
English
Yap session ground zero
LMSYS Org@lmsysorg
📣 New to SGLang? No problem — Our Office Hours have you covered 👌 This week's session is built for beginners: "New to SGLang: What I Learned & What I Wish I Knew on Day 1." 👉Alex Nails (@alxnails), MTS at @radixark, is sharing what it's actually like to onboard into SGLang — how the pieces fit together, what took some time to click, and his ideas on what could be better. Join us for the mental model walkthrough for SGLang, and an open discussion on making the dev and learning experience better. 📅 March 25 | 6:00 PM PST Register on Luma: luma.com/87xexrbg
English