Harry Mellor

@_Suresh2 @LysandreJik @vllm_project Do you have numbers to back up this claim? We benchmarked across concurrency 16/64/256 and measured parity in all of them. Server command was: vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --enable-expert-parallel --tensor-parallel-size 8

@LysandreJik @vllm_project 100% in isolation, but once you stack concurrent requests the gap comes back a little

The transformers modeling backend in @vllm_project is now as fast as vLLM itself. What does this mean? Adding a model to transformers now gets you access to 100% of the perf you would have natively. Do it once, train it anywhere, infer it everywhere, deploy it on all hardware.

@ukrroot @vllm_project @huggingface The full benchmark script can be found at huggingface.co/datasets/ariG2… TL;DR: in 1024, out 128, 1000 requests. Measurements came from `vllm bench` so numbers should be accurate.

@hmellor_ @vllm_project @huggingface Nice, parity on latency too changes the calculus. What batch sizes and context lengths were those runs at? TTFT and inter-token both holding, or just throughput-weighted averages?

Big news from @hmellor_ + @huggingface team🙌! In v0.25.0 the Transformers modeling backend hits parity with hand-written vLLM models. Now 450+ transformers architectures run in vLLM at native speed with zero porting. Integrate once with transformers to get vLLM's fused kernels, torch.compile, and CUDA graphs for free. Read about the changes below 👇

@0xCF88 @vllm_project Anyone wanting better GGUF support can now contribute to github.com/vllm-project/v…

@hmellor_ @vllm_project does vllm better support ggufs yet

I have HUGE news about the Transformers modelling backend for @vllm_project v0.25.0 🚀 It has reached performance parity with native vLLM model implementations 🤯 The Transformers modelling backend has just become a zero-effort, zero-compromise way to deploy to vLLM!

@MHR7DYN @vllm_project Yes!

@hmellor_ @vllm_project Same perf gains for multimodal models as well??

@ukrroot @vllm_project @huggingface The latency performance is also at parity, see the benchmarks in github.com/vllm-project/v…

@vllm_project @hmellor_ @huggingface Parity" carries the common path—fused kernels, paged attention. But custom attention or MoE routing? The auto-mapped backend still trails a hand-written impl on decode latency. Real test: batch=1 tok/s, transformers-backend vs native, on a MoE. Where does the gap open?

@Mayhem4Markets @vllm_project @huggingface 🙏

@vllm_project @hmellor_ @huggingface You all have been cooking so hard over there. Appreciate the great work!

@_Suresh2 @vllm_project @huggingface The transformers team @huggingface has put in a lot of work to standardise the structure and naming of all the models they support in-tree. I think you'd be pleasantly surprised!

@vllm_project @hmellor_ @huggingface I count 451 models and a thousand subtle config mismatches waiting to surface

@bygregorr @vllm_project @huggingface The transformers model is loaded on the meta device, the fusions are performed on this exact object, then the weights are materialised. So after loading, it's as if it's a natively supported model with native performance!

@vllm_project @hmellor_ @huggingface Hit the porting wall last month on a small finance model. Rewrote the attention layers twice before giving up. Does native speed hold at low batch sizes, or does fusion only kick in above some threshold?

@MainzOnX @ariG23498 @vllm_project No problem! That is parallel work so that Transformers can be used for longer before needing to switch to a dedicated inference engine like vLLM

@hmellor_ @ariG23498 @vllm_project Ahhh gotcha I saw you added continuous batching and paged attention etc to hugggingface transformers somewhere so didn’t know if this was similar

A timeline of @vllm_project: > A new model arrives on the block, the authors integrate it to transformers (for the open community) and to vLLM (for optimized inference) > @hmellor_ adds the transformers modeling backend to vLLM, which helps with integrating the model to transformers, and directly being available to vLLM for inference > Harry introduces a genius `torch.fx` and ast update, which now uses the transformers model and looks for patterns in the graph, and updates the code using ast to make custom optimizations inside vLLM engine This makes the @huggingface transformers modeling backend as fast as (if not faster) than the naive vLLM implementation for many models. 🤗🚀

@ayushsatyam146 @ariG23498 @vllm_project Thanks! It was a long road to get to this point, but we got there!

@ariG23498 @vllm_project @hmellor_ This is cool af! Thanks for awesome work @hmellor_

@MainzOnX @ariG23498 @vllm_project vLLM: `vllm serve some/model --model-impl transformers`

@hmellor_ @ariG23498 @vllm_project Is this via vLLM or huggingface transformers?

@MainzOnX @ariG23498 @vllm_project From vLLM's perspective the Transformers model looks _exactly_ like the the natively implemented ones. So there's nothing you need to do! Just use `--model-impl transformers` to ensure you're using the Transformers version!

@hmellor_ @ariG23498 @vllm_project Nice! Any config I should be setting for continuous batching then?

@vllm_project Read about the changes: huggingface.co/blog/native-sp…

@MainzOnX @ariG23498 @vllm_project The Transformers modelling backend has been around since Transformers v4! This key development is on the vLLM side and will be released in vLLM v0.25.0 any day now!

@ariG23498 @vllm_project @hmellor_ Wait are you saying this is now in transformers v5?!?

@ariG23498 @tensorcruncher Did someone say Transformers modelling backend for vLLM three times into a mirror?

@tensorcruncher @hmellor_ you are being summoned my friend.

vLLM deep dive anyone? 🧑‍🍳

Today I deleted PagedAttention from vLLM

@_Suresh2 @BigCodeProject @vllm_project Once github.com/vllm-project/v… is in it'll be just as fast!

@hmellor_ @BigCodeProject @vllm_project ran starcoder2 with the transformers backend last month, ~15% slower but zero segfaults

With github.com/vllm-project/v…, both GPTBigCode and Starcoder2 from @BigCodeProject are now powered ⚡ by the Transformers modelling backend for @vllm_project 🚀 This is a best of both worlds where older architectures get to keep vLLM support without any vLLM maintenance burden!

We just crossed $100M annual run-rate. I know many AI companies are capturing much more $$$ these days, but still proud of the milestone! Maximizing short-term revenue has never been our priority. In fact, we're proud to manage to store and serve hundreds of petabytes of models and datasets while keeping HF free and open-source for 97% of our users. As a platform, we’re happy to hopefully create orders of magnitude more value for the community than what we capture. To me, that’s the very definition of a platform. And it has helped us build one of the most loved platform in tech, with network effects, a defensible position and a sustainable business which is quite unique in AI. Many many thanks to all the community members for building with us, we wouldn't be anywhere without you! Can’t wait for what’s next, especially as more companies start to see the value of open and local AI! Next milestone $1B?