vLLM

1.1K posts

vLLM banner
vLLM

vLLM

@vllm_project

A high-throughput and memory-efficient inference and serving engine for LLMs. Join https://t.co/lxJ0SfX5pJ to discuss together with the community!

Katılım Mart 2024
36 Takip Edilen43.9K Takipçiler
Sabitlenmiş Tweet
vLLM
vLLM@vllm_project·
Announcing the first-ever vLLM Conference — hosted by @inferact at Ray Summit, Aug 24–26 in San Francisco 🎉🌉 This is where we'll get into the work pushing open, high-performance inference forward, such as: 🗺️ Where the vLLM roadmap is headed ⚡ Getting the most out of accelerators including NVIDIA, AMD, TPU 🔗 Wiring vLLM into training and serving pipelines 🚀 Running inference on production scale The summit features speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and more 🎤 Come learn where the future of inference, open source, and AI is heading — and meet the leading builders driving it 👇 vllm.ai/events/vllm-co…
English
4
23
122
28.9K
vLLM
vLLM@vllm_project·
🎉 Great to see the @AMD team for bringing ROCm support to vime, the vLLM ecosystem's RL post-training framework. End-to-end RL post-training now runs natively on AMD Instinct MI355X GPUs. vime uses vLLM as its rollout backend, so on ROCm it inherits the full vLLM rollout stack with no separate code path. The @AIatAMD team validated the pipeline end-to-end, upstreamed the ROCm-specific fixes, and shipped a prebuilt container so you can skip building from source. What works today: - GRPO training - Colocated and async (non-colocated) train/rollout - Megatron-LM training + vLLM rollout backends - Qwen3 dense and MoE models On MI355X, Qwen3-8B sustains ~4,100 tokens/gpu/s, and the train-rollout logprob diff holds low and stable. 🔗 vllm.ai/blog/2026-07-1…
English
4
6
79
7.6K
vLLM
vLLM@vllm_project·
Excited to see vLLM as the rollout engine in Molt, the new agentic-first RL framework from the @NVIDIA NeMo team. 🎉 vLLM (over Ray) carries the rollout here: fast async serving up to 1T-class MoE scale, simple to drop in. That lets the RL core above stay small and hackable. Can't wait to see what people train with it. 🚀
Jian Hu@hijkzzz

1/ Still looking for a minimalist, high-performance framework for agentic RL research? Meet Molt — an agentic-first, PyTorch-native reinforcement learning framework with roughly 9K lines of RL code for 700B models. ⭐ github.com/NVIDIA-NeMo/la…

English
11
18
133
14.9K
vLLM
vLLM@vllm_project·
🎉 Great to see @novita_labs train and open-source DSpark speculators for Kimi-K2.6 and Kimi-K2.7-Code. DSpark (@deepseek_ai's spec decoding method) drafts a whole token block in one pass, and vLLM supports it natively (v0.25.0). Grab the speculators and get faster Kimi decoding today. 🚀🚀
Novita AI@novita_labs

We trained and released DSpark speculators for Kimi-K2.6 and Kimi-K2.7-Code on @huggingface, with native serving support in @vllm_project. Across six benchmarks in our batch-size-1 evaluation: Kimi-K2.6: 2.55× average throughput (+155%) Kimi-K2.7-Code: 2.36× average throughput (+136%)

English
6
17
144
16.3K
vLLM
vLLM@vllm_project·
🎉 Congrats to @PrimeIntellect on Verifiers v1! Its training rollouts run on vLLM for exact token IDs and logprobs, no tokenization drift, keeping rollouts and training in sync. vLLM powers a growing set of open RL infra, prime-rl and others, and it's an area we're going deep on. 🚀
Prime Intellect@PrimeIntellect

Today, we are releasing verifiers v1 — an overhaul of our environment stack for the modern era of agentic RL and evals. We decompose environments into a taskset, a harness, and a runtime. Run complex agentic tasks like coding and computer use at scale, in any harness.

English
5
14
102
14.5K
vLLM
vLLM@vllm_project·
Serving, frontend, and what to know before upgrading: 🌐 Distributed: sequence parallelism without requiring DP (1.9–5.0% E2E throughput), NCCL symmetric memory extended to AllGather + ReduceScatter, all2all fault tolerance to prevent corrupted output 🔗 PD disaggregation: secondary KV tier, Mooncake connector support for GDN (Qwen3.5) and MLA (DeepSeek-V4-Flash) 🔌 Streaming Parser Engine: one framework for tool-call + reasoning parsing, with a new Kimi k2.5/k2.6/k2.7 parser 🦀 Rust frontend: HTTPS/mTLS, DP supervisor, profiler control routes 🔢 Quantization: 2/3/5/6/7-bit weight-only inference (Humming), Triton INT4 per-token-head KV cache quantization 🔒 Security: image decompression-bomb DoS guard, NaN-audio infinite-loop fix, bounded tokenizer work ⚠️ Before you upgrade: • The legacy PagedAttention implementation is deleted; V1/MRv2 backends are the standard path • Models removed: Baichuan, Aquila, Grok, Tarsier/Tarsier2, AyaVision, MusicFlamingo, Mantis 🙏 Thanks to all 232 contributors this cycle (64 first-timers). 📖 Full release notes → github.com/vllm-project/v…
English
2
0
20
5.8K
vLLM
vLLM@vllm_project·
Hardware & performance: 🟢 NVIDIA: FlashInfer fused all-reduce tuned for GB300, restored Blackwell NVFP4 decode throughput, FA4-MLA warmup infrastructure, XQA decode kernels 🐋 GLM-5.2 / DeepSeek: fused indexer kernel (1.9–3.3% E2E throughput), reduce-scatter MoE all-reduce (+3%), and a token_to_req_indices cache for DSv4 (5–6x kernel speedup) 🔴 AMD ROCm: torch 2.11 stable ABI, AITER FlashAttention MLA prefill backend, fused shared-expert for GLM-4.5/6/7 and MiniMax-M3 🔵 Intel XPU: W8A8 FP8 linear kernel with multi-granularity quant, uniform-batch CUDA graphs for FA2 💻 CPU & beyond: faster unquantized MoE on AArch64, Apple Silicon hang fix, RISC-V RVV INT4 GEMM, fp16 on PowerPC
English
1
1
23
4.8K
vLLM
vLLM@vllm_project·
vLLM v0.25.0 is out! 558 commits from 232 contributors (64 new). 🎉 Highlights: Model Runner V2 is now the default for all dense models, the legacy PagedAttention implementation is retired, the Transformers backend now runs as fast as native vLLM, a new unified Streaming Parser Engine, universal speculative decoding across heterogeneous vocabularies (TLI) plus new DSpark and DFlash drafters, and new models including Hy3 and Unlimited OCR. Thread 👇
vLLM tweet media
English
19
68
621
87.2K
vLLM
vLLM@vllm_project·
Huge thanks to the @cohere team for tackling this head-on with hardware-aware Dynamic SD, now merged into vLLM. Instead of a fixed number of draft tokens, DSD adapts to the batch size and hardware, so you get the speedups where they help and clean fallback where they'd otherwise hurt. Love seeing this land in the project 🙌 Read more below 👇
Ekagra Ranjan@EkagraRanjan

🧵Speculative decoding makes LLMs faster... until it doesn't. At high batch sizes, it makes inference SLOWER so most production systems can't use it. At @cohere, we fixed it with Hardware-aware Dynamic SD, open sourced in @vllm_project. Here's how 👇 🔗 cohere.com/blog/hardware-…

English
5
17
193
21.8K
vLLM
vLLM@vllm_project·
@RyanLeeMiniMax Congratulations to the @MiniMax_AI team. Excited to see this strong commitment to open source and the broader AI ecosystem. 🚀 Looking forward to building the future of AI together. 🤝
English
1
2
69
10.6K
RyanLee
RyanLee@RyanLeeMiniMax·
I’m incredibly excited to share this: MiniMax has just closed a new $2B funding round. 🚀 At the same time, our CEO, IO, shared three long-term commitments with the team: • No salary until we achieve AGI. • Over the next four years, he will dedicate shares equivalent to 4% of the company’s total equity from his personal holdings to reward employees who are building MiniMax for the long term. • Another 1% will be committed to supporting the open-source community. The funding is exciting. But what excites me even more is what it represents: a long-term commitment to AGI, to our people, and to the open-source ecosystem. We’re living through one of the most exciting moments in the history of AI, and we’re just getting started. If you’re passionate about frontier AI, open source, and building the future, we’d love to build with you. Intelligence with Everyone. 🚀
English
228
234
3.4K
882.9K
vLLM
vLLM@vllm_project·
🎉 Congrats to the @MosiAI_Official team on MOSS-Transcribe-Diarize-0.9B, an open, end-to-end model for multi-speaker long-audio transcription, with day-0 support in vLLM. Most setups chain ASR + diarization + alignment (WhisperX-style). This one does all three in a single generative pass. It transcribes the speech, tags who is speaking, and emits timestamps together: [0.11][S01] Good morning![1.03] [1.11][S02] Morning, guys![1.34] A Whisper-style audio encoder feeds a Qwen3-style causal decoder, so a recording up to ~90 minutes goes in as one shot, no chunking or stitching. Keyword biasing lets you prime names, product codes, and domain terms so proper nouns come out right. Useful for meeting notes, interviews, call-center QA, and podcast transcription. 🔗 recipes.vllm.ai/OpenMOSS-Team/…
vLLM tweet media
MOSI@MosiAI_Official

🤗 MOSS-Transcribe-Diarize-0.9B is now open source on @huggingface. Built with an end-to-end audio-to-structured-transcript paradigm: >0.9B open-source ASR model >Apache license 2.0 >128k long-context transcription >Up to ~90-min audio input >Speaker labels + timestamps in one generation >Multi-speaker diarization for meetings, interruptions, and overlapping voices >Hotword biasing for names, terms, and domain-specific vocabulary >~100 token/s on NVIDIA RTX 4090, RTF ~0.017 Thank you @sgl_project @vllm_project @Prince_Canuma @lllucas for day-0 support! 🚀 Github: github.com/OpenMOSS/MOSS-… Huggingface: huggingface.co/spaces/OpenMOS… API: shorturl.at/DWwe3 Live demo: shorturl.at/wRZ3j Technical Report：arxiv.org/abs/2601.01554 HF Space: huggingface.co/spaces/OpenMOS… AtomGit：ai.atomgit.com/OpenMOSS/MOSS-… SGLang-Omni: github.com/sgl-project/sg… vLLM: github.com/vllm-project/v… MLX-audio: github.com/Blaizzy/mlx-au… Discord:discord.gg/SmVQHGffZU

English
7
26
216
23.9K
vLLM
vLLM@vllm_project·
Big news from @hmellor_ + @huggingface team🙌! In v0.25.0 the Transformers modeling backend hits parity with hand-written vLLM models. Now 450+ transformers architectures run in vLLM at native speed with zero porting. Integrate once with transformers to get vLLM's fused kernels, torch.compile, and CUDA graphs for free. Read about the changes below 👇
Harry Mellor@hmellor_

I have HUGE news about the Transformers modelling backend for @vllm_project v0.25.0 🚀 It has reached performance parity with native vLLM model implementations 🤯 The Transformers modelling backend has just become a zero-effort, zero-compromise way to deploy to vLLM!

English
9
35
270
29.1K
vLLM
vLLM@vllm_project·
Hy3 serving is now heavily optimized on @NVIDIA Hopper. The same kernels that serve it in Tencent's production — the HPC-Ops attention and MoE backends from the @TencentHunyuan AI Infra team — are now first-class backends in vLLM main. A per-step, load-balanced decode scheduler plus a fully fused FP8 MoE pipeline: up to 2.95x over a static split-KV schedule on mixed-length decode, and they cut Hy3's TTFT by ~24% and TPOT by ~17% vs the default backend. No fork, no source changes. Full design + benchmarks 👇 🔗 vllm.ai/blog/2026-07-0…
English
0
1
21
10.1K
vLLM
vLLM@vllm_project·
🎉 @TencentHunyuan's Hy3, the full release following the Hy3 Preview, runs natively in vLLM from day 0, with tool-call and reasoning parsers and MTP speculative decoding, verified on @NVIDIA and @AMD hardware. Hy3 is a Mixture-of-Experts model built for agentic workflows, coding, and long-horizon reasoning, and it's Apache 2.0. 295B total parameters with just 21B active, 192 experts with top-8 routing, GQA attention, a 256K context window, and a 3.8B MTP layer for speculative decoding. Ships in BF16 and FP8. Full recipe (exact flags, FP8, MTP, hardware): recipes.vllm.ai/tencent/Hy3
English
4
11
134
16.4K
vLLM
vLLM@vllm_project·
Congrats to @MistralAI on Leanstral 1.5! 🎉 An Apache-2.0 Lean 4 proof agent that punches way above its size: 🧩 MoE: 119B total, just 6B active 📐 100% on miniF2F 🎓 New SOTA on FATE-H (87%) & FATE-X (34%) ⚡ 587/672 on PutnamBench at ~$4/problem Read the blog below or serve it on vLLM today! huggingface.co/mistralai/Lean…
Mert Ünsal@mertunsal2020

Today, we are releasing Le Chaton L∃∀N, aka Leanstral 1.5. It achieves SOTA performance on graduate algebra benchmarks FATE-H and FATE-X and improves Pareto Frontier on PutnamBench, solving 587/672 problems with a x10 cheaper budget. 🧵

English
8
18
214
25.5K

Keşfet

@AMD @AIatAMD @NVIDIA @novita_labs @deepseek_ai @PrimeIntellect @cohere @RyanLeeMiniMax