mobicham

@QuixiAI Isn't that just the standard DeepSeek style block quant ?

I reverse engineered Qwen 3.5's FP8 format, and provide a script to recreate it.

@MainzOnX I think they were compiled with the same flags, same Triton version, different ptxas version

@mobicham Oh wtf that’s retarded. Ptx to ptxas being a black box always sucks. Probably the NV ptx flags for optimizations under the hood not giving the results we all expect

Triton ships ptxas 12.9 for Blackwell, but CUDA 13.0 ptxas adds support for e2m1x2.f16x2 which makes activation quant go brrr. However, it seems that ptxas 13.0 actually generates worse kernels, typically with large M 🤔

@MainzOnX 3.6.0, but this is not a Triton issue, it's a ptxas issue, the ptx is identical from Triton (except .version)

@mobicham Triton version? I use 13 for triton

@kimbochen Different SASS with the same ptx 😢

@mobicham Wow this is cursed Different numerics for same instr on different GPUs?

@hy3na_xyz Wat @gaunernst make it rain

TC is now 1M-2.5M+

This bit flip might be the issue

@snowclipsed Another piece of disappointment: the RTX PRO 6000 is on paper 2000 TFLOPS for FP4, but really if you factor in everything (activation quant, etc.), the best you can reach is 1300 TFLOPS ish

@snowclipsed Yeah, it is actually worse than what I thought, sometimes more than 10% slower on sm_120 for large shapes, you can try it (back up first): cp /usr/local/cuda13.0/bin/ptxas /usr/local/lib/python3.12/dist-packages/triton/backends/nvidia/bin/ptxas-blackwell; rm -rf ~/.triton/cache;

@snowclipsed @stochasticchasm

@mobicham @stochasticchasm another SM_120 hate evidence dropped...

@sweinoid Yes, custom kernel, but you can patch vLLM to use it

@mobicham Are you having to modify a kernel to do this? I doubt its as simple as a vllm arg!

Little trick to outperform Cutlass for NVFP4 on sm_120: use mixed TMA: because TMA requires padding to 128, I don't use it for the activation scales, resulting in a huge bump for decoding speed🫡!

🫡

🫡

@gaunernst What I mean is that you can patch the static cache class in huggingface to return only the right slice, so it will not run attention with the whole max seq len. I remember there is some trickery with torch compile to make slicing work with breaking it

@mobicham my problem with it is that for example if i want max context = 40k, static cache impl in HF will do attention on all 40k all the time (iiuc), which makes it very slow. I could limit it to 1-2k context for example, but I feel that's "cheating"

Small update to the Qwen3-0.6B "megakernel". Managed to hit ~700tok/s on 5090 (including tokenizer decode + print to screen). Quite far from @AlpinDale's 1k tok/s. And speed drops significantly at longer context due to missing "flash" decoding i.e. split-K for attention

@gaunernst You can technically patch static cache in huggingface and make it work. I did something similar to use arbitrary batch-sizes with the same static cache instance

Using the same setup, HF eager reaches ~130tok/s. I know there is torch.compile support with static cache, but that requires attention for full cache size, which makes fancy demo if u limit the context, but I think it's "cheating" if the setup can't support longer context.

@xeophon It's worse than Qwen3 4B Instruct 2507 in terms of instruction following

qwen3.5 4b + disabled thinking is so good, man

@BarneyFlames I think so 👀! (code not merged yet, mega PR yet to be made)

@mobicham should work well on G7e instances, right ?

Finally squeezing some time to revisit GemLite 🫡

@AlpinDale Try Qwen3.5

I really want SOTA OSS models to be multimodal. It sucks that you have to use a downgraded model to get vision.