Tweet épinglé
Eric Todd
141 posts
Eric Todd
@ericwtodd
Computer Science PhD Student at Northeastern University
Boston, MA Inscrit le Aralık 2014
444 Abonnements474 Abonnés
Eric Todd retweeté
Eric Todd retweeté
I’ll be presenting our work, “Distilling Diversity and Control in Diffusion Models,” at @wacv_official this Sunday at 11 AM local time.
🔍We uncover the “secret to unlocking diversity” in diffusion models - using **interpretability**!!
DM me if you’d like to connect in Tucson.
Rohit Gandikota@rohitgandikota
Why do distilled diffusion models generate similar-looking images? 🤔 Our Diffusion Target (DT) visualization reveals the secret to diversity. It is the very first time-step! And—there is a simple, training-free way to make them more diverse! Here is how: 🧵👇
English
Eric Todd retweeté
NNsight 0.6 is out now! We directly address your feedback in our biggest release yet. Pain points included cryptic errors, slow traces, no remote execution of custom code, and limited vLLM support. We tackle all of these and more in this new release.
🧵 Here's what changed:
English
Eric Todd retweeté
How do you knock the induction heads out of an LM while preserving its ability to think? Is it even possible?
@keremsahin2210's work is worth reading if you haven't seen it yet.
Kerem Şahin@keremsahin2210
Are induction heads necessary for the emergence of in-context learning (ICL)? Their emergence coincides with a sharp ICL improvement, raising the hypothesis they may underlie much of ICL. However, we find that ICL beyond copying can emerge even when we suppress induction heads!
English
Eric Todd retweeté
Are induction heads necessary for the emergence of in-context learning (ICL)?
Their emergence coincides with a sharp ICL improvement, raising the hypothesis they may underlie much of ICL.
However, we find that ICL beyond copying can emerge even when we suppress induction heads!
English
Eric Todd retweeté
Data is plenty, knowledge is scarce. We began to close this gap thanks to deep learning <3 Neural networks can learn “programs” that often achieve superhuman performance from data alone. What insights are encoded in their weights? Here we took a first step on AI protein folding.
Kevin Lu@kevinlu4588
How do protein folding models turn sequence into structure? In "Mechanisms of AI Protein Folding in ESMFold", we find properties like charge and distance encoded in interpretable, steerable directions. The trunk processes features in two phases: chemistry first, then geometry.
English
Eric Todd retweeté
How do protein folding models turn sequence into structure? In "Mechanisms of AI Protein Folding in ESMFold", we find properties like charge and distance encoded in interpretable, steerable directions. The trunk processes features in two phases: chemistry first, then geometry.
English
Eric Todd retweeté
😻New preprint! As an interp researcher, I often ask “why did the model attend to this token?” We study this by decomposing the query-key (QK) space into interpretable low-rank subspaces. When these subspaces of Qs and Ks align, the model produces high attention scores.
1/N
English
Eric Todd retweeté
We trained diffusion models on a billion LLM activations, and we want you to use them!
New preprint: Learning a Generative Meta-Model of LLM Activations
Joint work with @feng_jiahai, @trevordarrell, @AlecRad, @JacobSteinhardt.
More in thread 🧵
English
@CreativeS3lf @davidbau @jannikbrinkmann @rohitgandikota Thanks! It's a combination of matplotlib and powerpoint to handle figure layouts.
English
@ericwtodd @davidbau @jannikbrinkmann @rohitgandikota Nice Work! I wonder what software did you use to produce the visuals, thanks!
English
@philljkc @davidbau @jannikbrinkmann @rohitgandikota @catherineols @NeelNanda5 Thanks! Yes, this example assumes the variables are assigned to arbitrary group elements
English
@ericwtodd @davidbau @jannikbrinkmann @rohitgandikota @catherineols @NeelNanda5 Cool puzzle! For this question, are you assuming we are in a group here, e.g. that all elements have proper inverses? Otherwise I think ab can take arbitrary values
English
@jjcvip @davidbau @jannikbrinkmann @rohitgandikota In our paper we let the symbols to represent arbitrary algebraic elements, and focus mainly on groups.
But you're right to point out that when less structure is assumed, these algebra puzzles become harder or even impossible to solve in closed form.
English
@ericwtodd @davidbau @jannikbrinkmann @rohitgandikota Of course this assumes associativity. With an inverse then you can do much more exciting things.
English
@plumsirawit Yes, in our paper we study mainly groups and find transformers learn strategies that reflect group structure like identity & cancellation rules.
But as you point out, when we test models on less algebraic structure, these puzzles become harder (or sometimes impossible) to solve
English
ab = aac = aacb = abb
Uhh I seem to be unable to solve the puzzle without any unnatural assumptions… (ofc associativity and extensionality laws are naturally assumed by notation)
If one assumes group law (or even just cancellation+neutral law) then cb=c implies b=id, so ab=a.
Eric Todd@ericwtodd
Can you solve this algebra puzzle? 🧩 cb=c, ac=b, ab=? A small transformer can learn to solve problems like this! And since the letters don't have inherent meaning, this lets us study how context alone imparts meaning. Here's what we found:🧵⬇️
English
@aliochaka @davidbau @jannikbrinkmann @rohitgandikota Not quite! If you notice, your answer for x is the same as the original question posed: "ab=?"
English
@ericwtodd @davidbau @jannikbrinkmann @rohitgandikota (1) cb=c (2) ac=b (3) ab=x
(2) *b : acb=b^2=ac=b
(4) b^2=b
(3)*b ab^2=xb=ab=x
So x=ab
Indonesia
@AlecEBG @davidbau @jannikbrinkmann @rohitgandikota The symbols don't have to be integers (they represent arbitrary group elements), but yes you can think of them as non-zero.
See this comment as well:
x.com/glassala/statu…
Violaine the Alchemist@glassala
@Marshwiggle119 @ericwtodd @davidbau @jannikbrinkmann @rohitgandikota The product is a product of arbitrary group elements, and groups don’t allow for an element which behaves like 0 under multiplication. (Even the multiplicative group of e.g. the real numbers has to explicitly exclude 0.)
English
English
@octonion Yes, cool right!? This sudoku-style cancellation is one of the context-based strategies we saw the transformers learn!
We also talk a bit about the model's performance on finite quasigroups (represented via Latin squares) in the paper's appendix.
x.com/ericwtodd/stat…
Eric Todd@ericwtodd
Another strategy infers meaning using sets. We have seen models keep track of "positive" and "negative" sets that let it narrow its understanding of a symbol using Sudoku-style cancellation. Red bars (a) show the positive set and blue boxes (b) show the negative.
English
These feel like you're solving a sudoku, probably from the connection to Latin squares.
Eric Todd@ericwtodd
Can you solve this algebra puzzle? 🧩 cb=c, ac=b, ab=? A small transformer can learn to solve problems like this! And since the letters don't have inherent meaning, this lets us study how context alone imparts meaning. Here's what we found:🧵⬇️
English
English
@Marshwiggle119 @davidbau @jannikbrinkmann @rohitgandikota Right! For this puzzle c is not 0. While variables can (and often do) represent numbers, our setup also lets us assign them represent arbitrary group elements.
If symbols were to represent elements of a dihedral group their interactions can be seen as "rotations" of a polygon.
English
@plain_simon @davidbau @jannikbrinkmann @rohitgandikota Glad you found it interesting! And thanks for pointing this out - I have some (appendix) updates to the preprint coming soon so in the next preprint version we can update the language to clarify that we're just talking about "groups".
English
@ericwtodd @davidbau @jannikbrinkmann @rohitgandikota This looks very interesting!
However the terminology "algebraic groups" for what is usually just called "groups" is unfortunate, as "algebraic group" is the standard name for something else:
en.wikipedia.org/wiki/Algebraic…
English
Takeaway: contextual reasoning can be richer than just fuzzy copying!
See the paper for more results, including an analysis of learning dynamics. Code & data are available at our project website.
📜: arxiv.org/abs/2512.16902
🌐: algebra.baulab.info
English
