Millicent Li

NLAs look like they "work"---but compared to what? How can we know that they actually tell us anything about model internals? In my new blogpost (and paper, to be presented at ICML!), we investigate whether activation verbalizers (like NLAs) produce faithful explanations.

📣We are excited to announce our 🔭 Interpretability for Discovery workshop at NeurIPS, Atlanta! ⏰Deadline: Aug 29, AoE 🔗Website: interpretability4discovery.github.io 🌟We are also looking for reviewers! tinyurl.com/bdkh64af Stay tuned for more details!

Excited to be heading to #ICML2026 to present 3 papers! ✈️ Happy to talk about developmental LM science, continual learning, or LM creativity! See more details below, and send me a message via email if you'd like to chat! (Twitter chat isn't working for me). Also excited for shopping 👜

@ZikuD_s Yeah, that's the general consensus I have after having played with NLAs for ~month, especially if you pair them with relatively simple baselines the responses that you can get when compared to these simple baselines are quite similar.

@millicent_li I take it the result is they don't? Did some expts as well and it seems the AR and AV don't have enough incentives to actually stay faithful, only reconstruction feedback.

NLAs look like they "work"---but compared to what? How can we know that they actually tell us anything about model internals? In my new blogpost (and paper, to be presented at ICML!), we investigate whether activation verbalizers (like NLAs) produce faithful explanations.

Heading to #ACL2026 ☀️ and #ICML2026 🇰🇷to present a bunch of interpretability work. Here are some of the places you can find me and my presentations. Love to chat about interpretability, finding the best vegan food around, and watching as much of the world cup as possible ⚽️

The full post is here: millicentli.github.io/blog/act_verb_… I'll be at ICML on the 7/8th, presenting my poster on the 8th in HALL A #3408 from 10:30AM-12:15PM! Happy to chat about the state of activation verbalization and more broadly actionable interpretability and training dynamics :)

This blog post is a companion post to my paper, Do Activation Verbalizers Convey Privileged Information? We explore why proper evaluations are important when making claims about what info activation verbalizers are accessing. For details, read the paper! arxiv.org/abs/2509.13316

🚨 Excited to share our @icmlconf paper "The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust" that was done during an internship with @google Cloud AI research last summer with @palashiitkgp, @hamidpalangi, and others 1/🧵

In case you’ve been wondering what I’ve been up to these days… So excited to (re)join an amazing community of linguists and NLP researchers at UT :)

@lltjuatja yay congrats lindia!!!

Benchmarks can be superficial, but model explanations and evaluations are fundamentally intertwined. What if we used interpretability as principled, scientific evaluation? If it met scientific standards? arxiv.org/abs/2605.05508 coming to @evaluatingevals at ACL as oral 🧵 1/6

How do we figure out whether a LM has learned the right skills, and in what order? We look at model internals :) Great work led by @_emliu!!

1/ (New paper!) If swapping the gender in an input prompt makes the AI model give a different answer it means that it has to have a gender bias, right? Wrong. 🧵on counterfactual prompting for LLM evals: Paper: arxiv.org/abs/2605.01048

@StephenLCasper They definitely missed proper evals to be able to ensure something like this would work... my collaborators and I have been working in this space to understand faithfulness issues (see: arxiv.org/abs/2509.13316 which was accepted to ICML) but they seem to have glossed over it?

For the past decade, the Distill and Anthropic interpretability teams have constantly been told to stop conflating their ability to look at something and tell a story with actual understanding and to stop building their research off of cherrypicked demos. They never change.

@zhuokaiz There's a few other evals that we show in the Appendix of the paper too that also basically that this lack of verbalizing "privileged" knowledge persists across task types. But without enforcing this privileged constraint, you don't know which model's knowledge you're using

@zhuokaiz Yeah exactly, a lot of these works in this domain (including many of the subsequent works in verbalization incl. Activation Oracles + this white paper) seem to ignore the fact that the verbalizer is an LLM itself, and it's obvious the eval they do should reflect this notion.

NLAs can reconstruct a layer activation, but that doesn't mean they read what the model is thinking. The setup is a round trip. You take a frozen target LLM and grab an activation h_l from some layer l at some token position. The activation verbalizer (AV) takes that activation as input and produces text, which is interpreted as a natural-language "explanation" of what the activation encodes. The activation reconstructor (AR) then takes only that text and produces a reconstructed activation ĥ_l. The training objective is the L2 reconstruction loss ||h_l − ĥ_l||². Because the text bottleneck is discrete, the AV and AR are both trained with RL against this reconstruction reward. While it makes sense that the AV has to push enough information through a discrete text bottleneck for the AR to recover the original activation under L2, there is a huge gap between "this text reconstructs the activation under L2" and "this text describes what the model is thinking." First, L2 closeness in activation space is a weak proxy for semantic equivalence — activations are high-dimensional and not all directions are meaningful. Second, the AV and AR are both trained with RL against a reconstruction reward, so nothing rules out reward hacking — two co-trained RL policies can settle into an agreed code where the text looks like English but actually carries information only the AR knows how to read. Third — and most importantly for the eval-awareness claim — test prompts don't look like normal user messages (weird scenarios, high stakes, obvious decision points), so they produce different activations. An NLA trained to verbalize what's different about an activation will produce text like "this looks like a test." But that's pattern recognition of the input, not evidence that the model is holding a belief or doing anything cognitively analogous to thinking it's being tested. The interesting question for me isn't whether NLAs are extracting something (they probably are) but whether the natural-language outputs are faithful renderings of activation content versus plausible-sounding text generated under priors that happen to discriminate well. That's the same critique people have about almost all self-explanation methods. And I don't think NLAs fully escape it.