Andy Luo

@cooloney It is a house made with Toufu

@terminoid_ @AnushElangovan pls check #amd-gpu-deployment-mi300x-mi325x-mi350x-mi355x-via-docker" target="_blank" rel="nofollow noopener">docs.vllm.ai/projects/recip… and #amd_1" target="_blank" rel="nofollow noopener">docs.vllm.ai/projects/recip…. I tested it on MI300X and works well.

@AnushElangovan why are your mi300xs stuck on such an old version of vllm? can't use qwen 3.5 or 3.6, or gemma 4, or fp8, or int8, etc.

Tell us how we can make ROCm better. We are listening and we will grind it down.

Great to see this out! Worked with the team on bringing efficient large-scale RL post-training to MI300-class clusters — lots more coming 🚀 @linluo77 @lizli202503 @roaner

🚀 New blog: ROCm Support for Miles: Large-Scale RL Post-Training on AMD Instinct™ GPUs Together with @AMD, Miles brings end-to-end RL pipelines to MI300/350-class clusters: ⚡️ Rollout generation dominates RL compute, and AMD’s HBM bandwidth directly addresses this bottleneck 🧠 AIME accuracy improved from 0.665 → 0.729 across training on Qwen3-30B-A3B with GRPO 💾 MI300X delivers ~1.1–1.3k tok/GPU/s rollout throughput ⏱️ Mean step time 388.5s on a single 8-GPU MI300X node (32×8 sampling, 8k response cap) 🔧 Multi-turn agentic training validated ... and more optimizations to come 🔥

From 1 to 3,204 TFLOPS ⚡ FP8 GEMM optimization on AMD MI355X — pure HIP/C++, no assembly. 1→5 (LDS tiling) 5→30 (Matrix Cores) 30→507 (global→LDS) 507→1,166 (double buffer) 1,166→2,288 (multi-wave) 2,288→3,204 (8-wave ping-pong) Exceeds hipBLASLt at 8K matrix. Full blog: rocm.blogs.amd.com/software-tools…

Day 0 support for Qwen 3.5 is here on AMD Instinct GPUs. By providing Day 0 support on AMD Instinct GPUs with both SGLang and vLLM, we ensure that developers have the compute power and optimized software stack needed to run these massive, high-context models at production scale. Learn how to get started → amd.com/en/developer/r…

Day-0 support for the Qwen 3.5 model on AMD GPUs, achieved through tight collaboration with the @Alibaba_Qwen team just in time for Chinese New Year's Eve. Support is available via both @sgl_project and @vllm_project . With SGLang: Launch rocm/sgl-dev:v0.5.8.post1-rocm720-mi30x-20260215 Inside container, run: python3 -m sglang.launch_server \ --port 8000 \ --model-path Qwen/Qwen3.5-397B-A17B \ --tp-size 8 \ --attention-backend triton \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder With vLLM: Launch rocm/vllm-dev:nightly_main_20260211 Inside container, run: pip install git+lnkd.in/gw4k6mqE VLLM_ROCM_USE_AITER=1 \ vllm serve Qwen/Qwen3.5-397B-A17B \ --port 8000 \ --tensor-parallel-size 8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder You can now run the following text/image/video input examples on Hugging Face: #text-only-input" target="_blank" rel="nofollow noopener">huggingface.co/Qwen/Qwen3.5-3… #image-input" target="_blank" rel="nofollow noopener">huggingface.co/Qwen/Qwen3.5-3… #video-input" target="_blank" rel="nofollow noopener">huggingface.co/Qwen/Qwen3.5-3… Accuracy has been fully verified, and all AMD support code has been upstreamed. The upcoming SGLang and vLLM releases will support Qwen 3.5 out-of-the-box on AMD MI300X, MI325X, and MI355X GPUs.

✅ Day 0 support of MiniMax-2.5 on AMD GPU 2 MI300X GPUs are all you need, instead of 4 Hopper GPUs, to run it in full context. uv pip install vllm --extra-index-url lnkd.in/gJdnn3kJ VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.5 \ --tensor-parallel-size 2 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2_append_think \ --enable-auto-tool-choice \ --trust-remote-code

Qwen3 Optimization on AMD GPU with SGLang, full blog: lmsys.org/blog/2026-02-1…

Based on github.com/huggingface/tr…

Run #openclaw (aka #clawdbot and #moltbot) with local Minimax-M2.1 on AMD Developer Cloud for free. Check out the blog by @DuneDudeMahdi - lnkd.in/g7az4G5q

@i_ikhatri @__tinygrad__ @AnushElangovan it is like 30+ token/s for single request. The model is native int4 and it uses gtpq kernel in vllm. We will have a mxfp4 quantized model to speed up the perf.

@__tinygrad__ @AnushElangovan Yeah I don’t think MI300X supports int4 natively. I haven’t gotten a chance to try this yet but if I have some free time I’m gonna dig into it :)

Has anyone run Kimi K2/2.5 on AMD? Now that vllm ships with rocm (awesome btw!) @AnushElangovan can I just serve on this bad boy?

How to run the new kimi-k2.5 on AMD GPU: uv venv source .venv/bin/activate uv pip install vllm --extra-index-url wheels.vllm.ai/rocm/0.15.0/ro… vllm serve moonshotai/Kimi-K2.5 -tp 8 --mm-encoder-tp-mode data --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --trust-remote-code

@woosuk_k @inferact @vllm_project Congrats!

Today, we're proud to announce @inferact, a startup founded by creators and core maintainers of @vllm_project, the most popular open-source LLM inference engine. Our mission is to grow vLLM as the world's AI inference engine and accelerate AI progress by making inference cheaper and faster. The Challenge Inference is not solved. It's getting harder. Models grow larger. New architectures proliferate: mixture-of-experts, multimodal, agentic. Every breakthrough demands new infrastructure. Meanwhile, hardware fragments: more accelerators, more programming models, and more combinations to optimize. The capability gap between models and the systems that serve them is widening. Left this way, the most capable models remain bottlenecked and with full scope of their capabilities accessible only to those who can build custom infrastructure. Close the gap, and we unlock new possibilities. And the problem is growing. Inference is shifting from a fraction of compute to the majority: test-time compute, RL training loops, synthetic data. We see a future where serving AI becomes effortless. Today, deploying a frontier model at scale requires a dedicated infrastructure team. Tomorrow, it should be as simple as spinning up a serverless database. The complexity doesn't disappear; it gets absorbed into the infrastructure we're building. Why Us vLLM sits at the intersection of models and hardware: a position that took years to build. When model vendors ship new architectures, they work with us to ensure day-zero support. When hardware vendors develop new silicon, they integrate with vLLM. When teams deploy at scale, they run vLLM, from frontier labs to hyperscalers to startups serving millions of users. Today, vLLM supports 500+ model architectures, runs on 200+ accelerator types, and powers inference at global scale. This ecosystem, built with 2,000+ contributors, is our foundation. We've been stewards of this engine since its first commit. We know it inside out. We deployed it at frontier scale—in research and in production. Open Source vLLM was built in the open. That's not changing. Inferact exists to supercharge vLLM adoption. The optimizations we develop flow back to the community. We plan to push vLLM's performance further, deepen support for emerging model architectures, and expand coverage across frontier hardware. The AI industry needs inference infrastructure that isn't locked behind proprietary walls. Join Us Through the open source community, we are fortunate to work with some of the best people we know. For @inferact, we're hiring engineers and researchers to work at the frontier of inference, where models meet hardware at scale. Come build with us. We're fortunate to be supported by investors who share our vision, including @a16z and @lightspeedvp who led our $150M seed, as well as @sequoia, @AltimeterCap, @Redpoint, @ZhenFund, The House Fund, @strikervp, @LaudeVentures, and @databricks. - @woosuk_k, @simon_mo_, @KaichaoYou, @rogerw0108, @istoica05 and the rest of the founding team

@vllm_project A big step to streamline vLLM experience on ROCm!

Quick update: from v0.14.0 onward, CI ships ROCm Python wheels + Docker images by default. That means you can pip install or pull prebuilt images without extra build steps. Nightly builds are still on the way. Try it here: #quick-start" target="_blank" rel="nofollow noopener">vllm.ai/#quick-start. Hope this helps! #vLLM #ROCm #AMD

How to improve LLM serving perf 3-10x with LMCache on AMD GPU for long-document QA and multi-round conversations - blog.lmcache.ai/en/2026/01/09/…

@HotAisle These are unnecessary warnings in SGlang even you don't run gguf models on rocm. We will ask for a triage.

docker run rocm/sgl-dev:v0.5.7-rocm700-mi30x-20260106 sgl-workspace/sglang/python/sglang/srt/layers/quantization/gguf.py:46: UserWarning: Only CUDA support GGUF quantization currently. warnings.warn(f"Only CUDA support GGUF quantization currently.") The amount of developer friction around AMD is bonkers.

Excited to share that AMD and the vLLM Semantic Router (VSR) Team are redefining the AI stack, shifting focus from single models to intelligent orchestration for Mixture-of-Models (MoM) systems. Read more: blog.vllm.ai/2025/12/16/vll…

48 hours AMD Open Robotics Hackathon @ Tokyo begins on Friday evening.

@salykova_ Great