Harry He
70 posts
Harry He
@harryheisme
Founding member @radixark Trust the process
Katılım Haziran 2024
106 Takip Edilen51 Takipçiler
Harry He retweetledi
Inkling, @thinkymachines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio.
Serving and RL support are already live: you can run and shape it on an open stack, starting now.
Day 0 support on SGLang @sgl_project and Miles @radixark👇
- Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache
- Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization
- DFlash speculative decoding from @modal for low-latency serving
Launching now, blog and cookbook in the comments ⬇️
English
.@radixark brought @deepseek_ai V4 to production with DigitalOcean and @AMD.
AMD Instinct™ MI350X GPU Droplets, plus joint engineering that drove a ~10x throughput gain.
When the team that builds the inference engine picks your stack, that's the signal.
English
Harry He retweetledi
SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉
DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales.
We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec.
At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads.
With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300.
DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks @deepseek_ai for open-sourcing!
Blog with full technical details and commands to run below 👇
English
Our open-source RL training framework, Miles, just hit the @PyTorch blog! Built to solve the distributed engineering pain points of frontier LLM RL (MoE, multi-precision, scheduling).
Perfect if you're building on DeepSeek V4, Qwen3.6, and beyond.
RadixArk@radixark
Miles is now featured on the PyTorch Foundation blog. As models grow, shift from dense to MoE, and span more specialized hardware, RL post-training is no longer just about the algorithm. It is a distributed systems problem. Miles is our open-source RL training framework, built for exactly that. It comprises four systems behind a small, pluggable trainer: SGLang (@sgl_project) for rollout, Megatron-LM (@NVIDIAAI) for training, Ray (@raydistributed) for orchestration, and PyTorch (@PyTorch) as the common layer for models and numerics. Out of the box, you also get MoE-aware rollout/training alignment, a unified BF16/FP8/MXFP8/NVFP4/INT4-QAT pipeline, fast NCCL/RDMA weight sync, fault tolerance, and ready-to-run recipes for frontier models like DeepSeek V4, GLM 5.2, Qwen3.6, Kimi K2.6, and Nemotron 3 Ultra. Our goal is simple: make frontier-scale LLM RL easier to reproduce, extend, and operate. Thank you, PyTorch Foundation, and everyone who got Miles here, especially the legendary @slime_framework team!
English
Best BBQ
RadixArk@radixark
🔥 Summer done right. A whole roast lamb BBQ to celebrate the good times with friends, family, builders, and our community, World Cup on the big screen (good game of Spain vs. Uruguay ⚽), smoke rolling off the grill, cold drinks in the sun, and a backyard full of great people just soaking up the vibes, that's how we enjoy the summertime! 🍢☀️🍺 Great food, great people, great times. Huge thanks to everyone who came out, and shoutout to @FishAudio for co-hosting this BBQ with us. Here's to the rest of this summer together! 🫶
English
Harry He retweetledi
🎉 SGLang v0.5.14 is out!
First, we welcome 55 new contributors to this release 🙌
And the supported new models: GLM-5.2, Kimi-K2.7-Code, LiquidAI LFM2.5, Poolside Laguna-M.1, DiffusionGemma, Zyphra ZAYA1, and MiMo-V2-ASR.
Here are the highlights for this release:
- New LPLB load balancer that evens out MoE expert traffic across GPUs (DeepEP)
- Kimi-Linear (KDA) runs faster on NVIDIA Blackwell with a new CuteDSL prefill kernel
- Lower memory use for linear-attention (GDN/KDA) models
- Faster multi-GPU communication with MSCCL++ and MNNVL allreduce fusion
- Nemotron now supports DP attention and MTP
- Breakable CUDA Graphs now run on AMD ROCm/HIP
- Multiple DeepSeek-V4 performance updates: NVFP4 MoE, FlashMLA head64 decode, and faster FP8 quantization
Thanks to our amazing partners and model makers: @NVIDIAAI @AMD @intel @Kimi_Moonshot @Zai_org @liquidai @poolsideai @GoogleDeepMind @ZyphraAI @XiaomiMiMo
Now. MAX LOAD! MAX OUTPUT!
English
Harry He retweetledi
Great to be at the @hud_evals hackathon @ycombinator! We met old and new friends and were really impressed by everyone working on the hard problems in “RL” (reinforcement learning and real life)!
We’re always hiring ambitious, amazing people who’d love to bring frontier RL infra to everyone. Come build with us!
job-boards.greenhouse.io/radixark
English
Harry He retweetledi
SGLang 🤝 Cloudflare
michelle@michellechen
GLM 5.2 from @Zai_org just launched, and we're excited to be a Day 0 partner on @Cloudflare workers ai personally my favourite model - doesn't overthink and gets the job done. try it for yourself 🧡 and special thanks to @sgl_project always for day 0 inference engine support developers.cloudflare.com/changelog/post…
Dansk
Harry He retweetledi
🚀 New blog: The next generation of speculative decoding: DFlash and Spec V2
DFlash + Spec V2 hit >4.3X baseline throughput for LLM inference, now the default speculative decoding engine in SGLang! Together with @modal and z-lab.ai, our jointly-released DFlash drafter for Qwen 3.5 397B-A17B beats both baseline and native MTP in every setting we benchmarked:
1️⃣ >4.3X baseline & 1.5X native MTP throughput (concurrency 1, HumanEval, 8xB200)
2️⃣ Block diffusion drafter: a full token block in one forward pass
3️⃣ KV injection: target-model features fed into every draft layer’s KV cache for higher acceptance
4️⃣ Spec V2 overlap scheduler: +33% end-to-end
Read the code, deploy a DFlash server, and start experimenting!
English
Harry He retweetledi
🎉 SGLang v0.5.13 is out!
First, new model support!
Nemotron 3 Ultra, Step-3.7-Flash, Command A+, plus new diffusion models: Cosmos3, FLUX.2-Klein, Ideogram 4, LingBot-World, SANA-WM, and Ernie-Image.
Here are the highlights for this release:
- Speculative Decoding V2 is now the default! Tree drafting (topk>1) for faster generation
- Breakable CUDA Graphs now make prefill faster
- Qwen 3.5 runs faster on NVIDIA Blackwell with new GDN kernels
- HiCache with UnifiedTree on by default for hybrid SWA/Mamba models
- SGLang-Diffusion now supports realtime generation! Plus progressive resolution
- Multiple performance and feature updates for DeepSeek V4
Thanks to our amazing partners and model makers:
@NVIDIAAI @AMD @intel @awscloud @boson_ai @cohere @bfl_ml @ideogram_ai @deepseek_ai @Kimi_Moonshot @Alibaba_Qwen @StepFun_ai @Baidu_Inc @robbyant_brain
English
Harry He retweetledi
The standout feature in SGLang v0.5.13 is BCG (Breakable CUDA Graph). It delivers prefill efficiency comparable to PCG (Piecewise CUDA Graph), while being significantly more flexible and compatible with advanced optimizations.
BCG is also a powerful debugging tool — it enables eager execution inside CUDA graph replay, so you can easily print debug info or inspect intermediate states.
Prototype by @csy_789. @Oasis_a19 turned it into a production-ready prefill optimization. I was fortunate to pick the name for this awesome technique. 😊
SGLang@sgl_project
🎉 SGLang v0.5.13 is out! First, new model support! Nemotron 3 Ultra, Step-3.7-Flash, Command A+, plus new diffusion models: Cosmos3, FLUX.2-Klein, Ideogram 4, LingBot-World, SANA-WM, and Ernie-Image. Here are the highlights for this release: - Speculative Decoding V2 is now the default! Tree drafting (topk>1) for faster generation - Breakable CUDA Graphs now make prefill faster - Qwen 3.5 runs faster on NVIDIA Blackwell with new GDN kernels - HiCache with UnifiedTree on by default for hybrid SWA/Mamba models - SGLang-Diffusion now supports realtime generation! Plus progressive resolution - Multiple performance and feature updates for DeepSeek V4 Thanks to our amazing partners and model makers: @NVIDIAAI @AMD @intel @awscloud @boson_ai @cohere @bfl_ml @ideogram_ai @deepseek_ai @Kimi_Moonshot @Alibaba_Qwen @StepFun_ai @Baidu_Inc @robbyant_brain
English
pls stop those bias but that's really funny lol.
Robert Sterling@RobertMSterling
AI startup CFO, CEO, CMO, and CTO
English
Happy to see y'all next time
SGLang@sgl_project
🏙️ SGLang NY Tech Week Happy Hour Recap Last Wednesday, SGLang hosted a NY Tech Week Happy Hour in NYC, co-hosted with @HOFCapital, @Cloudflare, @CrusoeAI, and @ArklexAI. 380+ registered, 200+ in the room, and one unforgettable night. 🧡 The room was packed with engineers, researchers, and enthusiasts from quant funds, banks, and trading firms, all there to talk about one thing: where inference is headed as LLMs move into latency-sensitive production across trading, research, compliance, and risk. NYC, you showed up and brought the energy. We loved every minute. Until next time! ☀️ #NYTechWeek @Techweek_
English
oh yeahhhhh, SGLang community account is reactivated! What you want to see? Drop your idea here!
SGLang@sgl_project
👋 @sgl_project is back! Welcome to the home of the SGLang community! While @lmsysorg keeps you posted on technical drops and partner news, this space is for you! Here's what we've got lined up: 🚀 Version releases: every new SGLang drop, unpacked 🎙️ Office Hours: deep dives, live deployments, and team Q&A 📺 Tutorials: short how-tos and the best Office Hour moments 🌟 Community spotlights: the cool stuff you're building with SGLang 📅 Event updates: meetups, workshops, and where to catch us next And we'd love to hear from you! What do you want to see? Benchmarks? Model deep dives? A topic for the next Office Hour? Drop it below 👇 Every idea gets read!
English