Pete Cheslock

For all my local Boston friends. If you are interested in vLLM/llm-d and inference at scale you should join us!

📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗠𝗮𝗿𝗰𝗵 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! We launched our newsletter publicly last year to share our contributions to upstream communities from our @RedHat_AI teams. We’ve gained over 𝟭𝟯𝟬𝟬 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲𝗿𝘀! Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem, including updates from @vllm_project, KServe, @_llm_d_, @kubernetesio, and Llama Stack. 👉 Check out the March newsletter here: inferenceops.substack.com/p/state-of-the… 👉 Subscribe to get future issues in your inbox: inferenceops.substack.com 🚀 Thanks to everyone who subscribed so far! Kudos to all contributors to this edition! @franciscojarceo, Pete Cheslock, Sean Condon, Jooho Lee, Pierangelo Di Pilato, Ran Pollak, Nir Rozenbaum, @TerryTangYuan, Wentao Ye

We’ll cover all of this and more during our distributed inference meetup in New York City on March 11, 2026: luma.com/0crwqwg4

LFG!!!!

DON'T TOY WITH MY EMOTIONS

If you are in NYC next Wednesday, come and join us to learn how to scale AI Inference on Kubernetes with the llm-d project.

What’s on the agenda for next Wednesday's NYC meetup? 🛠️ Intro to llm-d 0.5 ⚡️ Distributed LLM serving on AMD 🧠 Lessons scaling Wide-EP and MoE 💾 KV-cache offloading & prefix scheduling Join us building the future of open-source inference. Details: luma.com/0crwqwg4

Join us next week in NYC with the llm-d community for a deep dive into distributed inference. We’re talking llm-d 0.5, scaling MoE models, and KV-cache offloading. If you're building LLM infra, don't miss this. 📅 March 11th 📍1 Madison Ave Register: luma.com/0crwqwg4

RT @TerryTangYuan: We'd like to announce that @kubernetesio WG Serving has succeeded and will be disbanded! Thank you everyone who have pa…

Great talk last night by @julianeagu (@QuotientAI), @thejackobrien (Subconscious), and @petecheslock (Red Hat)! LLMs as we know it today must change to meet the capacity we expect of them. Specialized agents, changing their hardware architecture, or funneling proper context!!

🚀 Announcing llm-d v0.4! This release focuses on achieving SOTA inference performance across accelerators. From ultra-low latency for MoE models to new auto-scaling capabilities, we’re pushing the boundaries of open-source inference. Blog: llm-d.ai/blog/llm-d-v0.… 🧵👇

The day has come where we get to announce what we've been working on for the past year 😍 duckbillhq.com/blog/skyway-cl…

In honor of Kubecon 2025 (which I'm attending this week). A blast from the "long hair pete" era. (In this case, manbun) How do you say Kubernetes abbreviation (k8s)

🚀 llm-d v0.3.1 is LIVE! 🚀 This patch release is packed with key follow-ups from v0.3.0, including new hardware support, expanded cloud provider integration, and streamlined image builds. Dive into the full changelog: github.com/llm-d/llm-d/re… #llmd #OpenSource #vLLM #Release

If you’re running LLM inference at scale and still relying solely on “requests per second” or “GPU usage,” you might be missing critical insights. At Red Hat, we’ve been rethinking observability for LLM systems, from token throughput and latency metrics to cache reuse and end-to-end visibility. This post breaks down how @llm_d brings cache-aware, token-level, and routing metrics to Red Hat @openshift AI 3.0, exposing TTFT, TPOT, cache hit ratios, and full traces from IGW to @vllm_project workers. Read more → redhat.com/en/blog/tokens…

Getting started with llm-d v0.2 is now easier than ever! We've launched a full set of quick start guides to walk you through our most powerful features, including P/D disaggregation and deploying large MoE models on Kubernetes. Start here: llm-d.ai/docs/guide

Distributed LLM inference is not hard. @_llm_d_ is a Kubernetes-native high-performance inference framework that makes it easy to get started. With llm-d v0.2, we’re laying down well-lit paths: reproducible, production-grade patterns for high-performance distributed inference...

So what’s new in v0.2? 🔹 Validated P/D Disagg: Unlock higher GPU throughput 🔹 Native MoE Support: Run massive models like DeepSeek-R1 at scale 🔹 Extensible Scheduler: Customize routing with precise, prefix-aware logic Dive into the release notes: red.ht/4o7OUsd

The llm-d community is proud to announce the release of v0.2! Our focus has been on building well-lit paths for large-scale inference on Kubernetes. This release delivers major advancements in performance, scheduling, and support for massive models. red.ht/4l4u9uD

llm-d organizes through 7 specialized teams (SIGs): 🔀 Inference Scheduler 📊 Benchmarking ⚡ PD-Disaggregation 🗄️ KV-Disaggregation 🚀 Installation 📈 Autoscaling 👀 Observability Weekly meetings, public docs, active Slack channels. Join today! llm-d.ai/docs/community…