SGLang

🤗 MOSS-VL-Realtime is now open source on @huggingface . The 11B model family supports text, single and multiple images, single and multiple videos, and interleaved visual-text inputs in Chinese and English.@MosiAI_Official Highlights: 🏗️ Cross-Attention architecture separating visual encoding from language reasoning 🧭 XRoPE for unified temporal-spatial positioning 🧩 Unified conversation templates for offline, streaming, and real-time interaction 🧠 256K-token context window 📜 Apache-2.0 license MOSS-VL-Realtime continues processing new frames while generating a response, allowing it to revise or interrupt that response as the scene evolves—or remain silent when more evidence is needed. Thank you @sgl_project @lmsysorg for day-0 support! 🚀 Huggingface: huggingface.co/OpenMOSS-Team/… Github: github.com/OpenMOSS/MOSS-… Technical blog: openmoss.github.io/MOSS-VL Join the community: discord.gg/SmVQHGffZU 👇

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens. Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations. Special thanks to @NVIDIAAI for the help in day-0 support of GLM-5.2 NVFP4, and to @Zai_org for IndexShare in SGLang!

.@radixark brought @deepseek_ai V4 to production with DigitalOcean and @AMD. AMD Instinct™ MI350X GPU Droplets, plus joint engineering that drove a ~10x throughput gain. When the team that builds the inference engine picks your stack, that's the signal.

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1 In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens. Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations. Special thanks to @NVIDIAAI for the help in day-0 support of GLM-5.2 NVFP4, and to @Zai_org for IndexShare in SGLang!

🚀 Can Attention-FFN Disaggregation still win on the newest rack-scale GPU systems? We built FastAFD, an open-source AFD runtime for GB200 NVL72: 72 Blackwell GPUs in one NVLink domain. It improves per-GPU decode throughput by 1.35-1.45×. 🧵 Code: github.com/hao-ai-lab/Fas… Blog: haoailab.com/blogs/fastafd/

💡 Continuous software innovation is the force multiplier behind AI infrastructure — compounding inference performance, lowering cost per token, and increasing long-term value with every optimization. Open source accelerates this advantage. Leading AI frameworks like @PyTorch and inference engines such as @sgl_project and @vllm_project are built natively on NVIDIA CUDA, enabling research breakthroughs and software optimizations to unlock great performance on NVIDIA GPUs from day zero. Learn more: nvda.ws/4vw9LI7

Sglang compatible 2.7bpw in vram with configurable offloading to provide more quality recovery at the cost of speed and ddr or nvme memory. Inspired by vllm-moet. Benchmarking nice, very very fast. github.com/0xSero/sglang-…