Kaichao You

🧵Speculative decoding makes LLMs faster... until it doesn't. At high batch sizes, it makes inference SLOWER so most production systems can't use it. At @cohere, we fixed it with Hardware-aware Dynamic SD, open sourced in @vllm_project. Here's how 👇 🔗 cohere.com/blog/hardware-…

We trained and released DSpark speculators for Kimi-K2.6 and Kimi-K2.7-Code on @huggingface, with native serving support in @vllm_project. Across six benchmarks in our batch-size-1 evaluation: Kimi-K2.6: 2.55× average throughput (+155%) Kimi-K2.7-Code: 2.36× average throughput (+136%)

1/ Still looking for a minimalist, high-performance framework for agentic RL research? Meet Molt — an agentic-first, PyTorch-native reinforcement learning framework with roughly 9K lines of RL code for 700B models. ⭐ github.com/NVIDIA-NeMo/la…

Announcing the first-ever vLLM Conference — hosted by @inferact at Ray Summit, Aug 24–26 in San Francisco 🎉🌉 This is where we'll get into the work pushing open, high-performance inference forward, such as: 🗺️ Where the vLLM roadmap is headed ⚡ Getting the most out of accelerators including NVIDIA, AMD, TPU 🔗 Wiring vLLM into training and serving pipelines 🚀 Running inference on production scale The summit features speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and more 🎤 Come learn where the future of inference, open source, and AI is heading — and meet the leading builders driving it 👇 vllm.ai/events/vllm-co…

🤗 MOSS-Transcribe-Diarize-0.9B is now open source on @huggingface. Built with an end-to-end audio-to-structured-transcript paradigm: >0.9B open-source ASR model >Apache license 2.0 >128k long-context transcription >Up to ~90-min audio input >Speaker labels + timestamps in one generation >Multi-speaker diarization for meetings, interruptions, and overlapping voices >Hotword biasing for names, terms, and domain-specific vocabulary >~100 token/s on NVIDIA RTX 4090, RTF ~0.017 Thank you @sgl_project @vllm_project @Prince_Canuma @lllucas for day-0 support! 🚀 Github: github.com/OpenMOSS/MOSS-… Huggingface: huggingface.co/spaces/OpenMOS… API: shorturl.at/DWwe3 Live demo: shorturl.at/wRZ3j Technical Report：arxiv.org/abs/2601.01554 HF Space: huggingface.co/spaces/OpenMOS… AtomGit：ai.atomgit.com/OpenMOSS/MOSS-… SGLang-Omni: github.com/sgl-project/sg… vLLM: github.com/vllm-project/v… MLX-audio: github.com/Blaizzy/mlx-au… Discord:discord.gg/SmVQHGffZU

I have HUGE news about the Transformers modelling backend for @vllm_project v0.25.0 🚀 It has reached performance parity with native vLLM model implementations 🤯 The Transformers modelling backend has just become a zero-effort, zero-compromise way to deploy to vLLM!

Hy3 serving is now heavily optimized on @NVIDIA Hopper. The same kernels that serve it in Tencent's production — the HPC-Ops attention and MoE backends from the @TencentHunyuan AI Infra team — are now first-class backends in vLLM main. A per-step, load-balanced decode scheduler plus a fully fused FP8 MoE pipeline: up to 2.95x over a static split-KV schedule on mixed-length decode, and they cut Hy3's TTFT by ~24% and TPOT by ~17% vs the default backend. No fork, no source changes. Full design + benchmarks 👇 🔗 vllm.ai/blog/2026-07-0…

GLM 5.2 DSpark preview is here! ✨ huggingface.co/RedHatAI/GLM-5… This is the first DSpark speculator for a non-DeepSeek frontier model, trained with Speculators and running on vLLM nightly for ~1.5× faster decode for GLM-5.2-FP8 on 4×B300. Stronger checkpoints to come!

Fast, efficient local AI with open-source models just got easier. Qwen3.6-27B-NVFP4 is now on @HuggingFace! It's optimized for NVIDIA Blackwell GPUs & inference ready with @vllm_project. The checkpoint reduces GPU memory requirements by approximately 2.5x for powerful 27B-parameter inference on your own hardware.