Khoa

Imagine how many features you can vibecode with 500 TPS

@Norapom04 what dense attn does to a mf

if ur not /compacting at 400k ur ngmi

Best spec dec algo 🤝 best spec dec runtime. Only at @sgl_project

@tenderizzation banger

your comm and compute kernels when you try to implement SM carveout without green contexts

@Norapom04 @ grok is this true

just got last place in intern mock trading day after telling claude opus "go big or go home" a gpu trained model wouldn't have done me like that

@_dcw02 @tenderizzation same

@tenderizzation i miss culver's

i asked gpt to tier list something that actually matters

@Leik0w0 so true

Now with 100% more anime girls 🥳

🚀New record on GB300 NVL72: SGLang exceeds 12K tok/s per GPU on DeepSeek V4 Pro 1.6T (FP4, 8K/1K), orchestrated with NVIDIA Dynamo (SGLang) and MTP. Per @SemiAnalysis_ InferenceX benchmarks, performance stays strong across the entire interactivity curve. More to come with @NVIDIAAIInfra 🤝

sglang downloads 7x'ing since april is an interesting signal for open-weight models clickpy.clickhouse.com/report/may-202…

The Nemotron series is impressive -- strong capabilities in an efficient form factor, with high-performance implementations (esp on Blackwell) in open source. Deploy the new Nemotron 3 Ultra (550B-A55B-NVFP4) on @modal starting from this recipe: modal.com/docs/examples/…

@eliebakouch @TheZachMueller many such cases

microsoft uses SGlang wow

the "loss" definition is VERY important, the scaling ladder heavily relies on this. it's a NLL private set (negative log likelihood) with: 50% code 17.5% STEM 17.5% Math 10% General knowledge 5% Multilingual they then use this target NLL and normalize it with an in-house model. normalization matters because raw NLL scales differ across benchmarks

🐋 DeepSeek V4 is now merged into SGLang main with v0.5.12. What we shipped at launch: 🔹 ShadowRadix: native prefix caching for V4's hybrid attention 🔹 HiSparse: CPU-extended KV for sparse attention (up to 3× long-context throughput) 🔹 MTP speculative decoding with in-graph metadata preparation 🔹 W4A8 MegaMoE kernel 🔹 Flash Compressor + Lightning TopK kernels 🔹 Multiple parallelism methods: Tensor Parallelism/Expert Parallelism/Context Parallelism/Data Parallelism Attention 🔹 Prefill Decode Disaggregation 🔹 Hardware: H100, H200, B200, B300, GB200, GB300, MI35X And what we added since: 🔹 HiCache for V4 under UnifiedRadixTree 🔹 W4A4 MegaMoE kernels for faster MegaMoE 🔹 Marlin/FlashInfer MXFP4 (W4A16) MoE on Hopper 🔹 Hierarchical multi-stream overlap for small-batch decode 🔹 Optimized mHC pipeline: DeepGemm + fused norm + fused hc_head 🔹 Faster KV Compression V2 kernel 🔹 Fused SiLU+clamp+FP8 quantization kernel 🔹 Support TP16 on H100/H20 🔹 Support Multiple Detokenizers 🔹Pipeline Parallelism 🔹One docker image for all supported Nvidia hardware Thanks to @NVIDIAAI, @AMD, @ant_oss, @alibaba_cloud, ByteDance, @iFLYTEKLab, @radixark, and @pranjalssh for the work we shipped together on V4 🙌 More in 0.5.12 👇

@CMS_Flash @xai All the best and excited to see your next chapter!

I left @xai last week. It was quite a journey. I learned a lot technically and made some lifelong friends. I was fortunate to have spent almost all of my xAI tenure on a singular, long-term project on optimizing Grok's ability to one-shot complex, complete, and polished web apps. I am proud of this team. We contributed one of xAI's most successful alignment recipes, which has been replicated to other teams. We scaled another innovative recipe to incredible scales, for which Grok 4.3 gave a tip-of-an-iceberg preview. I am excited to see its full impact reveal on future releases. Looking forward, we are now at a critical point in the history of AI. Coding is on track to become a solved problem and AI is on the brink of its first total disruption of a major industry. Models show the first signs of substantial participation in their own development. The singularity might be on the horizon. Today might be analogous to the eves of AlexNet and GPT-3, or perhaps more profound. I am excited for what is to come.

@tenderizzation real

“yeah so the interview will be mainly about pytorch” “great i really enjoy c++”

@eliebakouch many such cases

thinking machines is using SGLang btw

@Guodzh @radixark @DecagonAI @Accel A bit under the weather so couldn’t come as audience. Well played! and thanks for rep-ing @radixark 🤣

lol ended up taking a seat from @radixark and got winner awards tho only #2 as #1 the host 😂 Poker skills don’t rust even not playing during my xAI time 🫣 Thanks @DecagonAI and @Accel for hosting

Amazing work from the @sgl_project and @radixark team for their work optimizing DeepSeek V4 inference on B200, B300, and the recent 4x iso-interactivity throughput improvements on GB300 by @ChengWan17! As @elonmusk said, The GB300 is the best AI computer, and software optimizations like this show its true potential!

For years, the most cutting-edge AI infrastructure has been concentrated inside a handful of frontier labs. Advances like test-time reasoning, reinforcement learning, and the rise of high-quality open source models are changing this dynamic. This shift creates a new opportunity to build foundational infrastructure for operating AI models—an open inference engine combined with flexible systems that give developers full control over how models are trained and deployed. That's exactly what @radixark is building. We are proud to back founders @ying11231 and @BanghuaZ, who have a long track record of creating and maintaining some of the most widely adopted open source projects in AI. Read more from Accel's @ivzhou and Joshua Fang, including what's next from RadixArk: accel.com/noteworthies/i…

Gemma 4 MTP is a novel new speculative decoding , quite different from traditional Eagle/MTP. Some highlights: - KV cache sharing with target model (!!!) - Centroid-masked logits head (E2B/E4B) Proud to deliver Day 0 support for a such a fun architecture!

Today, we are thrilled to officially launch RadixArk with $100M in Seed funding at a $400M valuation. The round was led by @Accel and co-led by @sparkcapital. RadixArk exists to make frontier AI infrastructure open and accessible to everyone. Today, the systems behind the most capable AI models are concentrated in a small number of companies. As a result, most AI teams are forced to rebuild training and inference stacks from scratch, duplicating the same infrastructure work instead of focusing on new models, products, and ideas. RadixArk was founded to change that. We are building an AI platform that makes it easier for teams to train and serve the best models at scale. RadixArk comes from the open-source community. We started with SGLang, where many of us are core developers and maintainers, and expanded our work to Miles for large-scale RL and post-training. We will continue contributing to both projects and working with the community to make them the strongest open-source infrastructure foundations for frontier AI. We would like to thank our long-term partners, contributors, and the broader SGLang community for believing in this mission. We're also grateful to @Accel and @sparkcapital, NVentures (Venture capital arm of @nvidia), Salience Capital, A&E Investment, @HOFCapital, @walden_catalyst, @AMD, LDVP, WTT Fubon Family, @MediaTek, Vocal Ventures, @Sky9Capital and our angel investors @ibab, @LipBuTan1, Hock Tan, @johnschulman2, @soumithchintala, @lilianweng, @oliveur, @Thom_Wolf, @LiamFedus, @robertnishihara, @ericzelikman, @OfficialLoganK, and @multiply_matrix among others. Thanks for the exclusive interview with @MeghanBobrowsky at @WSJ about our vision.