@kalomaze Hmmm open an issue with what you got would be fun to figure out what’s going on
English
driss guessous
553 posts
driss guessous
@drisspg
bytes and nuggets @pytorch https://t.co/gWVJmW741f
Katılım Aralık 2023
267 Takip Edilen1.6K Takipçiler
Honestly the most fun part was that the lil intra kernel profiling was actually helpful here and led me to try different ideas after looking at pipeline dependencies
English
Also forgot to post the receipts:
github.com/drisspg/transf…
I was claudxing here playing around with ideas. M=1, pure memory bound -> do we really need tcgen? Not using cublaslt autotuning and this is against pt nightly with 13.2 on B200.
Im sure folks can do much better :)
English
"ohh I beat cublas by 2% by implementing Hilbert curves"
The trick to beating cublas is to go where they aint looking;
And pray the next toolkit hasn't caught back up (it will)
English
@drisspg Wow, I missed this! Will definitely try it out. For bidirectional, I admit I haven’t looked into it too much, not a lot of model run on it these days… thanks for the tips!
English
There is no point running a model with SDPA attention.
Running an LLM without flash is like nerfing your own hardware. 3 years ago, sure, it was hard to setup, but not anymore.
Just look at the impact of flash on throughput... and im not even going to talk about CB... 🚀
English
@remi_or_ Makes sense, the Api forsure is missing features needed for autoregressive decoding. We did add docs.pytorch.org/docs/2.13/nn.a… which is much more aligned, supports FA3/FA4 and cudnn. For dense bidirectional attention SDPA is often quite hard to beat I find. Regardless, happy coding :)
English
@drisspg It means that it’s often a better idea to use flash attention (through the package `flash_attn` or `kernels`) than `torch.sdpa` ! Although it can fall back to flash using it explicitly is better IMO
English
I would like for someone to train a model who's entire chain of thought is gibberish and yet it still scales with test time compute
Ant Ling@AntLingAGI
🏆 High-Quality CoT Our model doesn't just get answers right, it does so beautifully 🌺 Comprehensibility: structured CoT ⚡ Token-Efficient: Solves AIME using < half the tokens of baselines 🌟 Reproducible: 100K of our traces distilled into Qwen-32B beat 800K of DeepSeek-R1
English
@giffmana Indeed, I have been quite impressed by its imporovemnts
English
@drisspg Even the chatbot is not good daily use, i a/b test with chatgpt a lot and all that's missing is the nice auto memory from past chats and a bunch if UI features.
English
@elliotarledge yeah, for mxfp8 and nvfp4 we route scaled_mm to cublaslt
English
@GoonGarrett I think it just gives it time to ferment slowly, and build a more "complex flavor" not to sound too douchey. The one I let go for like 4 days deff tasted the most unique
English
@drisspg What does the overnight refrigeration add? Surprised by that step
English
@drisspg we use coreweave torch-extras container images
github.com/coreweave/ml-c…
it installs FA4 [cu13] from source like so:
#L843" target="_blank" rel="nofollow noopener">github.com/coreweave/ml-c…
English
If PyTorch added cutedsl as a required
Runtime dependency for our Cuda wheels, how much would this mess you up? Would >= X.y for latest X.y at time of pt release work?
English
@tenderizzation Never attribute to malice that which is adequately explained by stupidity.
To my thought
English
It’s interesting the keep putting an end date on these extension promises. I wonder if they have a big pretain planned and have to keep pushing it back
Claude@claudeai
We're extending Claude Fable 5 access on all paid plans, as well as keeping Claude Code’s weekly rate limits 50% higher, through July 19.
English