ray

Announcing the first-ever vLLM Conference — hosted by @inferact at Ray Summit, Aug 24–26 in San Francisco 🎉🌉 This is where we'll get into the work pushing open, high-performance inference forward, such as: 🗺️ Where the vLLM roadmap is headed ⚡ Getting the most out of accelerators including NVIDIA, AMD, TPU 🔗 Wiring vLLM into training and serving pipelines 🚀 Running inference on production scale The summit features speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and more 🎤 Come learn where the future of inference, open source, and AI is heading — and meet the leading builders driving it 👇 vllm.ai/events/vllm-co…

Ray Summit early bird pricing is extended through July 17. The agenda is live: Google, Apple, Microsoft, Uber, Spotify, JPMorgan Chase, BMW, and many more.  SF · Aug 24–26. Register → na2.hubs.ly/H06y1P80

In this session, Ian D. Jordan, PhD - Robotics / AV Technical Specialist at Anyscale, will walk through: - Benefits and trends in policy evaluation via simulations - Challenges in scaling / distributing two GPU-based workloads - How Ray Serve and Isaac Lab separate the policy service from the simulators - Live walkthrough deploying NVIDIA's GR00T-N1.7-3B VLA to drive a Unitree G1 humanoid, fanned out across hundreds of parallel rollouts. If you're a robotics engineer, ML engineer, or simulation infrastructure engineer working on robot foundation models, humanoids, manipulation, or any physical AI system, this session is for you. Register: na2.hubs.ly/H06cfNW0

The #RaySummit2026 breakout lineup is taking shape 🔥  🔹 @Spotify: training LLMs with their Hendrix framework 🔹 @Microsoft AI: Ray + MAI-Thinking-1 🔹 @Apple Maps: batch inference + LLM eval at scale 🔹 @Discord: scaling ML with Ray + Anyscale 🔹 @LilaSciences: inside their AI research platform 🔹 Latitude AI: performant training across workstreams SF, Aug 24–26. Register: na2.hubs.ly/H06wnnB0

Platform teams are discovering that Kubernetes built for cloud-native apps doesn't naturally extend to agentic AI. Training, inference, and RL loops all have different scheduling, scaling, and GPU needs. Instead of replacing K8s, extend it with AI-native workload orchestration and efficient GPU sharing. Learn how to tackle this with Ray on Anyscale at PlatformCon 2026: na2.hubs.ly/H06n8Jl0

@Geotab's data scientists used to wait 20-30 minutes for a GPU Docker image to load. Now: 4-5 minutes. GPU = 0.1 in a Python annotation. No Kubernetes. No waiting on the platform team. Centralized GPU sharing for a fleet AI platform running at 100B+ data points a day. Full case study: na2.hubs.ly/H06pTvN0

We just released Ray 2.56! This includes - Ray Data stability improvements: reduced object store spilling, automatic batch size selection - Ray Serve LLM re-architecture: decoupling request handling from the token streaming response path, LLM serving performance improvements, new routing policies like session-sticky routing via consistent hashing - Ray Core GPU-domain-aware placement groups: enables placement groups to pack bundles onto nodes that share a ray.io/gpu-domain label instead of only packing at the single-node level - Kubernetes integration: initial Kubernetes in-place pod resizing support for Autoscaler v2

After many months of intense work the @Snowflake AI Research team is happy to present to you the new open source project: Arctic RL snowflake.com/en/blog/engine… - Arctic RL integrates with VeRL and SkyRL today; enable ZoRRo with one config flag, no code changes required - ZoRRo delivers up to 6x actor-update acceleration and a 3.5x end-to-end training speedup, reducing Arctic-Text2SQL-R2 training from ~5 days to ~36 hours on 32 H200 GPUs - Arctic-Text2SQL-R2 achieved higher accuracy scores (48.7) than Gemini 3.1 Pro (47.9) and Claude 4.7 (47.3) on Snowflake's evaluated enterprise SQL benchmark under the tested conditions - Two open source recipes ship with this release: a text-to-SQL recipe that improved BIRD dev accuracy from 59.92% to 70.35%, and a multi-hop QA recipe that improved average accuracy from 69.6% to 72.3%

Evaluating robot foundation models is an infrastructure challenge as much as a robotics one. In this new blog, we explain how to scale simulation from a single machine to distributed clusters with Ray and Anyscale, running thousands of rollouts in parallel while maximizing GPU utilization. Read more: na2.hubs.ly/H06kVFl0

RT @anyscalecompute: Ray Summit Training Day, Aug 24: a full day of hands-on sessions for engineers running Ray in production. Data pipeli…

Through our partnership with @anyscalecompute, Ray Serve LLM on Google Kubernetes Engine now offers 5x higher throughput and 8x lower latency for distributed inference! Read more about scaling inference without the bottlenecks → goo.gle/4w50CXB

Faster LLM inference with Ray Serve LLM on GKE.

Ray Serve LLM + vLLM + Google Cloud = high performance LLM inference :) Was great collaborating on our newest optimizations w/ @GoogleCloudTech!

@Geotab processes up to a million dashcam frames per batch run. They deploy a vision language model, run full inference, and shut it back down on every job. 43x throughput. 4x GPU utilization. 40% fewer GPUs at peak. How they built it on Ray and Anyscale 👇 Case study: na2.hubs.ly/H06gWNN0

Evaluating a robot foundation model is one of the most demanding closed-loop problems in robotics. Before you can trust a policy to move a real robot, you have to test it thousands of times across thousands of starting conditions each, pairing GPU-heavy inference with GPU-heavy physics simulation step by step, at a scale that quickly becomes a complex infrastructure problem. In this session, Ian Jordan will walk through this. Register: na2.hubs.ly/H06cfBN0

A great example of the importance of disaggregation in RL. From the paper ⚪️ LLM generation alternates between prefill and decode 🔵 Prefill is compute bound 🔵 H800s are compute optimized 🔵 Doing prefill on H800s cuts rollout time by 47% 🔴 Decode is bandwidth bound 🔴 H20s are bandwidth optimized 🔴 Doing decode on H20s cuts rollout time by 21-51%. On top of all that, the prefill to decode ratio depends on task characteristics (e.g., many-turn tasks that require lots of context compaction are prefill heavy). And prefill / decode is just for inference. There are many other components with different hardware requirements. Quoting the paper: ⚪️ Environments are stateful, CPU-bound processes whose latency is heavy-tailed due to host contention, large variance in interaction turns, and environment failures. ⚪️ Reward workers are stateless and exhibit persistently low utilization—dropping to as little as 7.4% on dedicated GPUs— yet require elastic scaling when trajectories complete. ⚪️ Training demands high-end GPUs with fast interconnects. No single hardware type satisfies all stages.

RollArt is an impressive example of disaggregation in large-scale RL. cse.ust.hk/~weiwa/papers/…

@raydistributed ❤️ @googlecloud

Ray (@raydistributed) and @vllm_project are helping developers understand when prefill-decode disaggregation can improve throughput and reduce compute costs, and when aggregated serving remains the better choice. Read more: anyscale.com/blog/ray-vllm-…

The @PyTorch Foundation is lucky to host these amazing open source projects @raydistributed and @vllm_project Seeing the community come together to make something even more powerful is amazing!