Sabitlenmiş Tweet
Nathan Hu
42 posts
Nathan Hu
@NathanHu12
PhD @stanfordnlp | aspiring LLM biologist | https://t.co/c5lk3rAilp
Katılım Eylül 2021
299 Takip Edilen264 Takipçiler
Nathan Hu retweetledi
I’m at ICML! I’m hiring PhD students for my lab at Columbia. DM me here on twitter to set up a time to chat.
cs.columbia.edu/~johnhew/lab/
English
Nathan Hu retweetledi
My team at @GoodfireAI has been cooking up a new way to do interpretability: decompose a language model’s weights, not its activations.
Our decomposition natively handles attention (!) and behaves less like a lookup table and more like a generalizing algorithm. (1/6)
English
Nathan Hu retweetledi
for data-constrained pre-training, synth data isn’t just benchmaxxing, it lowers loss on the real data distribution as we generate more tokens
for even better scaling, treat synth gens as forming one long 𝗺𝗲𝗴𝗮𝗱𝗼𝗰: 1.8x data efficiency with larger gains under more compute
English
Nathan Hu retweetledi
Excited to launch a course on mechanistic interpretability at Stanford next quarter! So grateful to be part of an amazing teaching team: Atticus Geiger, Jing Huang, Junyi Tao, and Thomas Icard (who all share the admirable trait of not having a twitter account)
English
@jacobcd52 This is definitely weaker than what the tweet might have implied, but still quite surprising to me. I'd have expected reasoning training to teach new complex features rather than fully bootstrap off the base model's. (Usual caveats about the specific model we study apply.)
English
@NathanHu12 as it stands, I think your work doesn't rule out that "wait" is caused by complex/high-level features that are already fully present in the base model, so I don't really buy the quoted claim.
English
@jacobcd52 Agree and apologies for being a little imprecise—I'd like to think we're more careful about this in the paper. The more precise claim is: "reasoning training mainly adds simpler, reflex-like mechanisms on top of base model computation (which can already be complex)."
English
@jacobcd52 Agree it would be really cool to replace the base MLPs with transcoders (or even MLP neurons), this would really give a complete story of everything going on.
I'm a little attribution-graphed at the moment, but very tempted to try this.
English
@jacobcd52 This definitely sacrifices insight into the specifics of base model computation, but I believe should be fine for identifying important of adapter features and complete attribution flow from embeddings to predictions.
English
@jacobcd52 To clarify: we still consider base model MLPs when building these graphs—we just flow through them rather than showing them as nodes. So edges here capture both direct effects from adapter features and indirect effects through the base model.
English
Nathan Hu retweetledi
Circuit tracing is cool, but can it be used for model diffing? We investigate mechanisms introduced during reasoning fine-tuning by training transcoder _adapters_ to faithfully reconstruct MLP output _differences_. Check it out!
Nathan Hu@NathanHu12
What does reasoning fine-tuning actually change inside a model? In our new paper, we introduce transcoder adapters to learn sparse, interpretable approximations of how reasoning fine-tuning changes MLP computation. 🧵
English
Nathan Hu retweetledi
Very neat approach to studying model differences: train a transcoder to predict the change in MLP outputs, then build a graph where you flow through base model parts. Highlights how the changes (and input tokens) wire together.
Nathan Hu@NathanHu12
We use these features to investigate why the reasoning model says "wait." When building attribution graphs, we find that "wait" predictions seem to depend on only 2 types of adapter features: output features promoting "wait" and template features active on formatting tokens.
English
@DanielFein7 My current mental model is: some backtracking is genuinely important for problem solving. But distillation or optimization biases in RL details (arxiv.org/abs/2503.20783), can often introduce incidental backtracking/verbosity that doesn't actually help performance.
English
@DanielFein7 Great question. In the model we study, reasoning training doesn't seem to teach complex backtracking behavior from scratch. This is actually very compatible with Gandhi et al., who show that already having reasoning behaviors at the start of training really matters.
English
@maxsloef One reason to think this simple story could be more general is the body of work showing we can remove "wait" and shorten responses even in more "real" reasoning models. For example: arxiv.org/abs/2506.08343
English
Paper: arxiv.org/abs/2602.20904
Website with code, adapter checkpoints, and feature/attribution graph data: transcoder-adapters.github.io
Work with @_jake_ward, Thomas Icard, and @ChrisGPotts.
English
Finally, we show using these "wait"-specific features to shorten responses preserves accuracy. Removing them from the adapter (blue) results in no accuracy loss on 3/4 benchmarks. Suppressing them in the reasoning model (red)—a cruder intervention—also works reasonably well.
English