Inferact

Announcing the first-ever vLLM Conference — hosted by @inferact at Ray Summit, Aug 24–26 in San Francisco 🎉🌉 This is where we'll get into the work pushing open, high-performance inference forward, such as: 🗺️ Where the vLLM roadmap is headed ⚡ Getting the most out of accelerators including NVIDIA, AMD, TPU 🔗 Wiring vLLM into training and serving pipelines 🚀 Running inference on production scale The summit features speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and more 🎤 Come learn where the future of inference, open source, and AI is heading — and meet the leading builders driving it 👇 vllm.ai/events/vllm-co…

Announcing the first-ever vLLM Conference — hosted by @inferact at Ray Summit, Aug 24–26 in San Francisco 🎉🌉 This is where we'll get into the work pushing open, high-performance inference forward, such as: 🗺️ Where the vLLM roadmap is headed ⚡ Getting the most out of accelerators including NVIDIA, AMD, TPU 🔗 Wiring vLLM into training and serving pipelines 🚀 Running inference on production scale The summit features speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and more 🎤 Come learn where the future of inference, open source, and AI is heading — and meet the leading builders driving it 👇 vllm.ai/events/vllm-co…

During @aiDotEngineer World’s Fair, we’re hosting a happy hour on Thursday, July 2 with special guests from @vllm_project, @inferact, and @buildwithRemy. Join AI builders, founders, and researchers for casual discussion on leading open weight models, inference, infrastructure, and recent developments. Brief intros early on, then drinks and conversation. No conference ticket required, spots are limited!

Introducing: First Pass! Altimeter's new video series breaking down the latest trends in AI, hosted by myself and @palak_go AI evolves so quickly. We wanted to create a series of short (~10 minute) videos taking a First Pass on the latest trends First up: @simon_mo_ on GLM5.2

Thank you @Inferact for sponsoring us on @github. You can join them at our sponsors profile: github.com/sponsors/light…

Great work to @vllm_project team and @NVIDIA on smooth, out-of-the-box day 0 @MiniMax_AI M3 experience with @inferact EAGLE3 spec decode. Here are the details of ongoing M3 workstream: NVIDIA, Inferact and SemiAnalysis are working hard on enabling disaggregated inferencing (PR 45879), and the Inferact team is working on enabling FlashInfer M3 MoE kernels (PR 45723). Performance should be much better once those PRs land. Huge shoutout to @rogerw0108 & @mgoin_ and the maintainers for the rapid review and mentorship here!

A great deep dive from @SemiAnalysis_ on RL training systems and how much RL efficiency comes down to matching trainer and generator throughput! Shoutout to @KaichaoYou and Ao Shen from @inferact for the sandbox scaling experiments with vLLM + verl, building on @KaichaoYou's early RL integration work across OpenRLHF, verl, and slime🫡

🎉 Day-0 support for in vLLM, available today in v0.23.0! Congrats to @Zai_org on GLM-5.2, a flagship model built for long-horizon coding agents. ✨ 1M-token context, built to hold project-scale engineering work in a single run ✨ Tuned for long-horizon coding: large-scale implementation, automated research, and performance optimization ✨ One task can carry a full dev workflow, from requirements to a deployable product across platforms ✨ Client-side and mobile engineering, including an on-device debugging loop Try it out running it on vLLM today: 🔗 recipes.vllm.ai/zai-org/GLM-5.2

DAY 0 ALERT: @MiniMax_AI M3 is now available on HuggingFace & has been added to InferenceX. The M3 architecture has ~428B parameters and ~23B activated parameters. Due to the 10x engineers from @inferact, M3 is already delivering pretty well-optimized performance on @NVIDIAAI B300 Blackwell Ultra on Day 0 @vllm_project! Furthermore, Inferact released their EAGLE3 heads, which enable even greater performance. Looking forward to Day 1, 2, and 3 performance & the team is grinding on benchmarking Day 0 MI355X performance on InferenceX too.

🎉 Congrats to @MiniMax_AI on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at @MiniMax_AI, @NVIDIAAI, @AIatAMD, and @inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗 vllm.ai/blog/2026-06-1…

🚀 Day-0 support for NVIDIA Nemotron 3 Ultra on vLLM! Ready to be served with the latest vLLM stable release, the new open frontier reasoning model is built for long-running autonomous agents: 🧠 550B total / 55B active — Hybrid Transformer-Mamba MoE 📚 Up to 1M token context ⚡ NVFP4 + BF16 🛠️ Tool calling, coding, deep research, orchestration Read our detailed model launch blog and recipes! recipes.vllm.ai/nvidia/NVIDIA-…

The Redpoint InfraRed 100 is now live. These are the companies building the infrastructure that powers everything happening in AI right now, from world models and agent runtimes to the sandboxes, databases, and security tools agents depend on. Congratulations to this year's honorees! Read the full 2026 InfraRed Report: our state of the union on AI and cloud infrastructure 👉 redpoint.com/reports/the-in…

x.com/i/article/2057…

🦀 The Rust frontend is officially merged into vLLM! As GPUs get faster, the frontend has become a real share of CPU time. The new Rust frontend is a drop-in alternative to the Python API server — same engine, same ZMQ boundary. Opt in with VLLM_USE_RUST_FRONTEND=1. Early numbers: on a preprocess-heavy workload, ~837 req/s vs ~162 req/s for default Python — ~5x in a single process. A few design choices we're excited about: • Layered crates with clear boundaries • Stream-native pipeline — non-streaming for free • Builds on stable Rust Huge thanks to @BugenZhao from @inferact for introducing the work at @PyTorch Meetup Singapore. github.com/vllm-project/v…