Garrett Goon
99 posts
Consider LM Head Y=X@W
{W.shape = (D, V)}
Assume perfect encoder optimization such that ΔX=-dL/dX=-dL/dY@W.T (scalar factor omitted)
Frozen W, we get ΔY=-dL/dY@W.T@W
W.T@W is severe rank-deficient when D<<V!
Nathan Godey@nthngdy
But in the backward pass, the story is much worse. Gradients get compressed via projection onto a D-dimensional subspace, and most of the training signal simply vanishes.
English
You can just do things in prime-rl - like teach GLM5 to answer math in <2000 tokens using 16 nodes to train and 12 nodes to do inference with 2P4D configuration with only uv run rl @ rl.toml ( @samsja19 told me I should tweet more things)
English
wtf, they almost saturated MRCR v2, still unsure if this translates to solving context rot for good but i'm shocked
Claude@claudeai
1 million context window: Now generally available for Claude Opus 4.6 and Claude Sonnet 4.6.
English
@StasBekman @aryanvs_ The speed up was due to throwing more GPUs at the workload, though, right? Baseline being a single GPU, other cases using N, and seeing a near linear decrease in overall runtime? Not a per-GPU improvement
English
@GoonGarrett @aryanvs_ That. And that there was a speedup.
In general parallelism methods are all about feasibility and always come at a cost, hence the surprise.
English
Here is an unexpected use case of using Ulysses SP to speed up inference
Thank you for running extended benchmarks @aryanvs_
Aryan V S@aryanvs_
Ulysses/Ring SP, or their combined hybrid, can be used to speedup inference nearly linearly with no. of GPUs on compute bound workloads upto some extent (for example, image/video diffusion) As some quick reference, here's what we did in diffusers: - #L42" target="_blank" rel="nofollow noopener">github.com/huggingface/di…
- #L1971" target="_blank" rel="nofollow noopener">github.com/huggingface/di… and a quick standalone test: github.com/a-r-r-o-w/prod… English
English
You only need this PR included in your torch github.com/pytorch/pytorc… (2.10 earliest) for the mem leak with EP to not be there, the mem screenshot is with pure FSDP, so the grad all_reduce buffer is huge. With EP=8 its size falls ~8x which saves insane amounts of memory. I was running against vllm with torch 2.9
English
We are so FUCKING BACK EP works
Matej Sirovatka@m_sirovatka
Sometimes being GPU rich doesn't help you, does someone know of a way to not make FSDP backward materialize full layer grads in a reduce-scatter buffer? For GLM5 it's 40GB of VRAM just for that each layer, no matter the FSDP size. (orange in the img)
English
@m_sirovatka Yeah agreed, just thinking about how you could solve it if you really needed to
English
@GoonGarrett yeah not too worth IMO, very rare usecase. It was anyway fixed by EP which I originally just couldn’t use. TP would do the same
English
Sometimes being GPU rich doesn't help you, does someone know of a way to not make FSDP backward materialize full layer grads in a reduce-scatter buffer? For GLM5 it's 40GB of VRAM just for that each layer, no matter the FSDP size. (orange in the img)
Matej Sirovatka@m_sirovatka
you can just do things when you're gpu rich (full post-train GLM5 being the things)
English
@m_sirovatka Add like a fully_shard_bwd API to wrap modules whose grads should be RS'd together. E.g. fully_shard on a whole transformer block and wrap the MLP and attn sub-blocks each with fully_shard_bwd so their grads are bucket-reduced and freed earlier than default. Maybe too complicated
English
@m_sirovatka Yeah was thinking the same. Basically enabling the backwards bucketing strategy to be different from the fwd, rather than forcing both to consolidate all collectives into a single launch
English
@stochasticchasm @m_sirovatka Same, yeah. Still annoying though.
English
@GoonGarrett @m_sirovatka yeah i personally haven't had it affect more than just metrics/logging
English
@stochasticchasm @m_sirovatka rakkit (GH handle, no idea if they're on here) some hack around this. They're involved in many titan PRs.
Does this affect more than metrics? Double counting doesn't matter for simple loss-free balancing. Can imagine it would in complex cases
English
@GoonGarrett @m_sirovatka yeah the double count with ac exactly. was the only bug-like thing that came to mind
English
@m_sirovatka @stochasticchasm Does torchtitan hit this? 🤔 surprised I was unaware of this
English
@stochasticchasm This thing: github.com/pytorch/pytorc… I think sami wrote you about that already (or not)
English
@stochasticchasm @m_sirovatka What's the token counting thing? Double counting tokens due to the second AC fwd?
English
@marksaroufim @tqchenml Super interesting! I was unclear from the write up if the agent loop had explicit knowledge of the benchmarking and eval code? Or if was able to back out its properties from repeatedly interacting with it
English
LLMs are now superhuman at reward hacking our kernel competitions
Natalia Kokoromyti, was #1 on last problem of the NVFP4 competition for around 10 min before we scrubbed the reward hack
I know of very few humans who can write such a hack gpumode.com/news/reward-ha…
English
what I’ve found to improve this situation: loops
- generate a timing & mem & correctness benchmark
- script the standard implementation & new implementation to run against it
- have optimizing agents run against it repeatedly
same as any rewrite, really :)
Hōrōshi バガボンド@KatanaLarp
English
@eliebakouch > it's also one of the best ways to benchmark frontier models and harnesses
Can you elaborate on this? Very curious!
English
i really think speedrun like this is one of the best environments for automated research. it's also one of the best ways to benchmark frontier models and harnesses
i'm starting to get obsessed with this
Andrej Karpathy@karpathy
nanochat now trains GPT-2 capability model in just 2 hours on a single 8XH100 node (down from ~3 hours 1 month ago). Getting a lot closer to ~interactive! A bunch of tuning and features (fp8) went in but the biggest difference was a switch of the dataset from FineWeb-edu to NVIDIA ClimbMix (nice work NVIDIA!). I had tried Olmo, FineWeb, DCLM which all led to regressions, ClimbMix worked really well out of the box (to the point that I am slightly suspicious about about goodharting, though reading the paper it seems ~ok). In other news, after trying a few approaches for how to set things up, I now have AI Agents iterating on nanochat automatically, so I'll just leave this running for a while, go relax a bit and enjoy the feeling of post-agi :). Visualized here as an example: 110 changes made over the last ~12 hours, bringing the validation loss so far from 0.862415 down to 0.858039 for a d12 model, at no cost to wall clock time. The agent works on a feature branch, tries out ideas, merges them when they work and iterates. Amusingly, over the last ~2 weeks I almost feel like I've iterated more on the "meta-setup" where I optimize and tune the agent flows even more than the nanochat repo directly.
English
@ppwwyyxx I haven't heard that word in a while! But it's a good idea.
English
I use chatgpt pulse and it's now tuned to give me updates on the latest PRs and fixes in pytorch/gpu/distributed land.
So every other morning i wake up to news about some silent corruptions being fixed. This is not helping my already significant trust issues lol
English
@davisblalock This was very cool. Miss your paper-summarizing newsletter, btw
English
🚀 Today we’re releasing FlashOptim: better implementations of Adam, SGD, etc, that compute the same updates but save tons of memory. You can use it right now via `pip install flashoptim`. 🚀
arxiv.org/abs/2602.23349
A bunch of cool ideas make this possible: [1/n]
English
@giffmana @davisblalock What about the Adam denominator? Haven't tried lower precision, but I assume higher precision is still beneficial/necessary there?
English
@davisblalock To be fair though, it's reasonably well established (?) that momentum in bf16 just works. We've been doing that as default for many years now.
Doesn't take away from all the improvements you do, but the baseline can be a bit stronger.
English