llm-d

🎉 llm-d v0.7 is officially live! 🚀 Earlier releases proved what llm-d could do, v0.7 is about making sure you can easily deploy it in production. Backed by a massive 3.5x surge in community PR volume, this release hardens the stack for serious scale. llm-d.ai/blog/llm-d-v0.…

Did you miss last week's vLLM office hours? Check out the recording on YouTube to learn more about llm-d's Wide EP and prefix cache routing!

Serving a 700B+ param model (GLM-5.2) on H200s: llm-d's Wide EP + prefix-cache routing hits 90%+ cache reuse and sub-3s TTFT, at ~$2/M output tokens vs $4.40 on hosted APIs. @robertshaw21 breaks it down in last week's vLLM Office Hours. Watch: youtube.com/watch?v=LXLKBH…

Part 2 of our 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝗔𝗜 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 series is now live on Red Hat Developer: 𝘖𝘱𝘵𝘪𝘮𝘪𝘻𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘈𝘥𝘷𝘢𝘯𝘤𝘦𝘥 𝘋𝘦𝘱𝘭𝘰𝘺𝘮𝘦𝘯𝘵 𝘗𝘢𝘵𝘵𝘦𝘳𝘯𝘴. In Part 1, we covered prefill/decode phases and the 5D parallelism framework. Part 2 dives into the three optimization levers that deliver most of the cost and latency improvements once your parallelism layout is set: - 𝗣/𝗗 𝗗𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻: Not a feature to toggle on - it's a deployment topology. We share how to measure whether the prefill-to-decode imbalance in your traffic justifies the split, with 25-40% cost reductions on chat and RAG workloads in our benchmarks. - 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿e: Tiering across HBM, DRAM, and NVMe with LMCache, the difference between prefix sharing and KV reuse (they're not the same thing), and when FP8/FP4 quantization pays off. - 𝗦𝗽𝗲𝗰𝘂𝗹𝗮𝘁𝗶𝘃𝗲 𝗗𝗲𝗰𝗼𝗱𝗶𝗻𝗴: EAGLE 3.1 now extends gains into long-context regimes with 2x longer acceptance length than EAGLE-3. But watch out - acceptance rates collapse under constrained decoding (JSON mode, tool calls), so measure before enabling on tool-calling traffic. One insight that keeps coming up: cache-aware routing via @_llm_d_ is what turns disaggregation from a checkbox into a working system. Round-robin leaves cache hits on the table. Co-authored with Fatih E. Nar, Yuchen Fama, and Greg Pereira. Part 3 covering deployment blueprints and troubleshooting recipes is coming soon - follow along to catch it. Read Part 2: developers.redhat.com/articles/2026/…

As context lengths and architectures diverge, getting multi-tier offloading right is what keeps serving frontier models affordable. Read the full breakdown & benchmarks: 👉 llm-d.ai/blog/serving-h… Join us on GitHub & the llm-d Slack! 🚀

3/ 🌐 Global Multi-Tier Routing: Capacity is an instance property, but throughput is about placement. By scoring GPU cache & CPU offload tiers globally, the llm-d scheduler drives a 115% throughput increase under load while keeping TTFT completely flat.

The era of the uniform KV cache is officially over. ❌ Modern hybrid models mix attention types—interleaving full attention with sliding-window or Mamba layers. This means the entire serving stack has to adapt to handle this heterogeneity. 👇

Excited to share Part 1 of our blog series on Red Hat Developer: 𝘋𝘦𝘴𝘪𝘨𝘯𝘪𝘯𝘨 𝘋𝘪𝘴𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘥 𝘈𝘐 𝘐𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦: 𝘊𝘰𝘳𝘦 𝘊𝘰𝘯𝘤𝘦𝘱𝘵𝘴 𝘢𝘯𝘥 𝘚𝘤𝘢𝘭𝘪𝘯𝘨 𝘋𝘪𝘮𝘦𝘯𝘴𝘪𝘰𝘯𝘴. LLM inference is two workloads pretending to be one. The prefill phase is compute-bound, processing entire prompts in parallel to populate the KV cache. The decode phase is memory-bandwidth-bound, generating tokens one at a time. Batching strategies that optimize one phase degrade the other and and that tension shapes every architecture decision downstream. In this post, my co-authors Fatih E. Nar, Yuchen Fama, Greg Pereira, and I break down: - Why prefill and decode need to be understood as fundamentally different workloads - The 5D parallelism framework (tensor, pipeline, expert, data, and context parallelism) that governs how models are distributed across GPUs - How context parallelism is becoming unavoidable as models push past 200K+ token contexts - Practical configuration trade-offs across different hardware budgets Read it here: developers.redhat.com/articles/2026/… This is Part 1 of a three-part series. We'll share the remaining parts in future posts. Follow along if you're interested in the infrastructure behind serving large models at scale.

📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗝𝘂𝗻𝗲 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! We recently launched our newsletter publicly after sharing it internally at @RedHat_AI for over a year. The response has been incredible - we’ve gained over 𝟭𝟱𝟬𝟬 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲𝗿𝘀! 🎉 Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from projects like @vllm_project, KServe, @_llm_d_, @kubernetesio, and beyond. 👉 Check out the June newsletter here: inferenceops.substack.com/p/state-of-the… 👉 Subscribe to get future issues in your inbox: inferenceops.substack.com 🚀 Thanks to everyone who subscribed so far! Kudos to all contributors to this edition! Eitan Geiger, Francisco Arceo, Pete Cheslock, Jooho Lee, Pierangelo Di Pilato, Ran Pollak, Nir Rozenbaum, Yuan Tang, Wentao Ye

Next up? Cross-accelerator Prefill/Decode (P/D) disaggregation—routing heavy prefill to one vendor's nodes and memory-intensive decode to another. Kudos to all the contributors! Read the full architectural breakdown on our blog: llm-d.ai/blog/heterogen…

How? llm-d concentrates cache hits on warm pods using precise (tokenizer-backed) or approximate (hash-based) prefix routing. It dynamically adapts to load and saturation signals while fully preserving the platform-specific driver and runtime settings each vendor needs. 🛠️

Can a cache-aware, saturation-aware router make a mixed, 3-vendor GPU cluster perform like one unified inference service? 🚀 We benchmarked llm-d v0.7.0 across a real-world heterogeneous cluster with IBM Research, Red Hat, and NxtGen Cloud. Here is what we found 👇

We've also added 10k+ lines of new docs and a rigorous multi-platform CI matrix to ensure what we guide is exactly what you deploy. A massive thank you to our 23 new contributors! 🙌 Read the full architectural breakdown on our blog: llm-d.ai/blog/llm-d-v0.…

🧠 Workload-Aware Routing & Caching • Flow Control: Centralized queuing at the Router level to prevent noisy neighbors. • Batch Gateway: OpenAI-compatible API for heavy offline workloads. • Real-time prefix cache tracking + tiered offloading to AWS EFS/NVMe.

🎉 llm-d v0.7 is officially live! 🚀 Earlier releases proved what llm-d could do, v0.7 is about making sure you can easily deploy it in production. Backed by a massive 3.5x surge in community PR volume, this release hardens the stack for serious scale. llm-d.ai/blog/llm-d-v0.…