Gabriel Franco

Why do attention heads attend where they do? We can now pinpoint the EXACT features causing attention—without counterfactuals, patching, or SAEs. New @NeurIPSConf 2025 paper with @mcrovella: "Pinpointing Attention-Causal Communication in Language Models"

“In my lab, you will have the luxury to think things through” is turning out to be pretty effective as a recruitment line for AI/ML PhD candidates. I guess permission to be careful is a scarce resource.

@a_jy_l @boknilev @viegasf @wattenberg Very interesting work!

With amazing collaborators @boknilev @viegasf @wattenberg Paper: arxiv.org/pdf/2602.04752… Code: github.com/ajyl/QK Hope you enjoy! 6/N

😻New preprint! As an interp researcher, I often ask “why did the model attend to this token?” We study this by decomposing the query-key (QK) space into interpretable low-rank subspaces. When these subspaces of Qs and Ks align, the model produces high attention scores. 1/N

Can steering remove LLM shortcuts without breaking legitimate LLM capabilities? In our @eaclmeeting paper, we show that conceptual bias is separable from concept detection; this means inference-time debiasing is possible with minimal capability loss.

Such an amazing talk as always, David. Let's keep doing good long-term science!

@Aman56Champ Thank you :)

@gvsfranco Cool work !

Happening today at #NeurIPS2025: How do you find circuits in LLMs without counterfactuals, patching, or SAEs? I’m presenting our new method for doing this by understanding the causes of attention. Exhibit Hall C,D,E, Poster #1111 4:30 PM - 7:30 PM

@HeMuyu0327 This is great work! We've been studying the exact same phenomenon from a causal perspective of attention. We found low dimensional signals that are causal to attention sinks. This is our NeurIPS paper if you want to check it out: openreview.net/pdf?id=wUoK24u… x.com/gvsfranco/stat…

We found that K vectors of attention sinks shrink to a low-variance subspace and exposes a "bias direction" for pick-up, but where did they get this bias direction from? We now find that they inherit this bias direction directly from last layer's residual output. If we trace one step back to the layer-normed input before the attention layer for block > 6, we find that the input for the sink token 0 is again extremely low-variance, while the input for token 1 and 8 are high-variance as usual (p1). Since the inputs to the attention block for token 0 are low-variance, it is no wonder that the K vectors for token 0 will be low-variance. And if the inputs again have a bias direction in the higher-dimension hidden space, then K vectors can easily have a bias direction in the lower-dimension head space. And they do. We further trace one step back to the output of last layer before the layernorm, and we find that the variance of token 0 outputs explode (p2). This is actually good: it suggests that the last layer's output shares the same bias direction but have different magnitudes, which layernorm removes in the next layer. So the word: bias direction is directly created in the residual output, which neatly translates into next layer's K, which forms the sink. Since the output of last layer is a residual addition between the attention output and the MLP / MoE output, it's now time to see what each part brings to the formation of attention sinks. Something cool is going to happen...

My lab at BU is recruiting PhD students and possibly a postdoc this year! We study humans & machines, centered around topics like meaning, generalization, evaluation methods and design, and the nature of computation and representation that underlie language and cognition. 🫴🫴

How reliable is your LLM-as-a-Judge?⚖️ Existing methods suffer from (a) rating inconsistencies and (b) low stability of correlation with human judgments across evaluator models. Excited to share CheckEval (@emnlpmeeting), a framework that improves reliability for LLM-as-a-Judge.

📄 Paper: openreview.net/pdf?id=wUoK24u… 💻 Code: github.com/gaabrielfranco… See you in San Diego! #NeurIPS2025 #AI #ML #NLP #Interpretability #Transformers #Attention

We also discovered "control signals", which are data-independent signals that coordinate attention across layers. They implement attention sinks at the signal level. Most models use a few distinct control signals, which is a way used by the model to organize heads hierarchically!

Why do attention heads attend where they do? We can now pinpoint the EXACT features causing attention—without counterfactuals, patching, or SAEs. New @NeurIPSConf 2025 paper with @mcrovella: "Pinpointing Attention-Causal Communication in Language Models"

When aligning LMs to personal preferences, is it at all beneficial to have model infer user preferences ("persona") over just using few-shots historical contexts? In our paper we found inferred "personas" improve generalization💪, contextual faithfulness🙏, and reduces bias⚖️!

Thanks to all for making NEMI 2025 a wonderful event. Fascinating talks, inspiring posters, important discussions. You surfaced the questions animating our growing field. I learned many things and hope you did too! Looking forward to what the next year will bring.

Does vision training change how language is represented and used in meaningful ways?🤔 The answer is a nuanced yes! Comparing VLM-LM minimal pairs, we find that while the taxonomic organization of the lexicon is similar, VLMs are better at _deploying_ this knowledge. [1/9]

We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.

Can coding agents autonomously implement AI research extensions? We introduce RExBench, a benchmark that tests if a coding agent can implement a novel experiment based on existing research and code. Finding: Most agents we tested had a low success rate, but there is promise!