Edward Z. Yang

@RisingSayak Most of my plain q&a tasks are either easily one shot table, or too hard and I end up iterating anyway

@ezyang Don’t you find the lost quality and iteration speed trade-off cumbersome?

I have to say, it's pretty nice to be in the period of time when Muse Spark doesn't have any usage limits (my ChatGPT Pro downgraded to Plus and now when I'm intensely coding I gotta ration my regular Q&A usage. Muse Spark is fine for a lot of research tasks.)

@SkyLi0n Yes, user mode driver is sufficient

@ezyang Does it work cuda13-compat shims or do you need actually cuda13 driver?

This is not just nightlies, it is in 2.12! (Thanks Natalia for the correction)

@drisspg coined a really good term for a lot of why we've been doing with LLM coding: "painting APIs." Where you sit down with you and your agents and just shove characters around on the API, playing around with different options, just asking for whatever API you want.

@llllvvuu I showed this to the team and we think it's the same (@drisspg authored this to check: gist.github.com/ezyang/1990294…). torchtitan is following the HF convention.

Megatron MoE computes softmax over top k logits. torchtitan computes softmax over all logits and renormalizes top k weights afterwards. The result is the same in forwards but not in backwards. Which way is correct? Megatron, right?

Pro-tip: using CUDA graphs and annoyed that all the kernels have no labels in your profiles? Get a nightly that has mark_kernels context manager: github.com/pytorch/pytorc… (thanks Natalia and Shangdi for implementing!) You need 13.1 driver, but user mode driver is enough

I know, these days stars don't mean much, but it's still lovely to see such a nice round number. Thank you everyone for trusting your ML workloads to us, here's to the next 100K stars! (H/t Tianyu for pulling this screenshot)

@typedfemale So it's definitely feasible to have the exact allocations from the caching allocator be deterministic run to run. This mostly boils down to not using record stream. I would check on this!

@ezyang oh yes, i've done that so many times!

the CUDA caching allocator is such a great way to create extremely "interesting" bugs for yourself

@typedfemale I have had some "fun" out of bounds bugs where CUDA sanitizer didn't help because all the memory accessed was technically valid 😂

@ezyang we have a kernel that's corrupting memory between the forward and backward pass and i think caching allocator was making it non-deterministic (really not it's fault, i was just being stupid and didn't realize what was going on)

If you're willing to put up with explicit comms in your code, github.com/meta-pytorch/s… is our latest direction for addressing some of the silent perf cliffs in DTensor. There's tradeoff space here for explicit-implicit, and spmd_types stakes out a different spot on the curve.

@defnotbeka mfw

man what did i type that ios """corrected""" to "merge"??? nothing makes sense

merge second half has been academia for like 40 years

deep in the refcounting ownership mines right now

I did not try it yet, but it does quite a few of the things that I wrote about recently! lucumr.pocoo.org/2026/2/9/a-lan…

Here is a good example of a PR where Codex simply does better than me by default. Frickin Python generators lmao github.com/pytorch/pytorc…

Introducing Renderers RL trainers work in tokens. Environments work in messages. Going back and forth corrupts sampled tokens, wasting compute on every agentic turn. With Renderers, we fix this mismatch. This unlocks >3x throughput on popular open models.

Remember, you need to make sure you generate exactly the same random numbers in forwards and backwards. So we ended up doing even regular checkpointing via a totally graph based strategy: tag and then min-cut partitioner; the SAC in compiler strategy was an extension on this.

Some extra backstory here: Animesh Jain was tasked with getting our preexisting checkpoint story to work with PT2. But dealing with functionalizing RNG operations was difficult, and we ended up being unable to trace through the eager checkpoint implementation.

A thread about the history and internal implementation details of activation checkpointing APIs in PyTorch. 🧵

I missed some history here! The original version of eager SAC came from github.com/facebookresear… ; implemented by @fvsmassa but with a lot of input from the xformers team at the time.

The history here explains why the SAC APIs are all focused around classifying operations as cheap or expensive: it comes from the original min-cut formulation. In 2024, Jeffrey Wan released an eager mode version of SAC centered around the same ideas.