John Hewitt

Come do a PhD with me at Columbia! My lab tackles basic problems in alignment, interpretability, safety, and capabilities of language systems. If you love adventuring in model internals and behaviors---to understand and improve---let's do it together! pic: a run in central park

@pathulith22 This is an interesting question; honestly I don’t know. For legit lens, my advice is to not take the outputs as meaning the model is “trying/going to” say that word; use the outputs to help some evaluation you care about and see if it beats baselines etc. Same with steering.

@johnhewtt Thanks for the informative post! I was curious about how you navigate this lack of linearity in practice. When utilizing tools like logit lens or steering vectors, are there specific 'anomalies' or failure modes you look for that suggest the approximation fails?

Lots of interp thought discusses the linearity of the residual stream! This blog post: the residual stream isn't linear in a way that provides formal leverage, and interp methods based on linearity should not be preferred beyond empirical utility. cs.columbia.edu/~johnhew/resid…

@dribnet haha the empirical utility of linear interventions astounds me even more due to the fact that it doesn’t need to be true from the architecture 🫡

@johnhewtt x.com/dribnet/status…

@jamesaoldfield Good point. Still ~mildly interesting at best, since the hidden/GLU-weighted linear combination of MLP output matrix vectors will be non-linearly read from. But right, it's not that the norm happens both at input and output of each sublayer.

@johnhewtt Really cool post!! Perhaps I misunderstood -- but is it not the case that many MLP blocks apply the norm to the input, not the output? Meaning the MLP applies a linear transformation to the hidden/GLU states, in a way that does make it mildly interesting as you mention

@giangnguyen2412 Great question -- we don't know and we need to continually test for generalization. My guess is a huge set of factors in the prompt/steering estimation play a role in where steering will work. In alignment science, people test this by having tons of evals; we should too.

@johnhewtt Thanks for the great write-up! one question: if not linearity, what does predict when steering methods work? maybe the curvature of the MLP activations in the vicinity of the intervention?

In a pub trivia night, if you don't know the answer immediately, you "reason" through your memories -- is it X? no... was Y related?. In LMs, we find that code/math RLVR'd models' reasoning for this parametric knowledge access can be easily improved, say, by TriviaQA RLVR.

Hey folks, just in case it was unclear, I talked to Been and her account has been hacked, so please disregard.

Excited to see more funding opportunities for interpretability, with an explicit call for strong evaluations.

📢 Some big (& slightly belated) life updates! 1. I defended my PhD at MIT this summer! 🎓 2. I'm joining NYU as an Assistant Professor starting Fall 2026, with a joint appointment in Courant CS and the Center for Data Science. 🎉 🔬 My lab will focus on empirically studying the science of deep learning and applying deep learning to accelerate the natural sciences. Very broadly interested in questions at the intersection of language, reasoning and sequential decision making. (Plus any other fun problems that catch our eye along the way!) 🚀 I am recruiting 2 PhD students for this cycle! If you're interested in joining, please apply here: cs.nyu.edu/dynamic/phd/ad… cds.nyu.edu/phd-admissions…

@lrzneedresearch We had a celebration for it.

@johnhewtt Want to see the mini-fridge

Come do a PhD with me at Columbia! My lab tackles basic problems in alignment, interpretability, safety, and capabilities of language systems. If you love adventuring in model internals and behaviors---to understand and improve---let's do it together! pic: a run in central park

@peligrietzer Thanks!!

@johnhewtt Incredibly cool work, congrats!

New work! Gemma3 can explain in English what it learned from data – when we distill that data into a new word (embedding) and query it for a description of the word. Gemma explained a word trained on incorrect answers as: “a lack of complete, coherent, or meaningful answers...”

We see this as a step towards developing new language tools for learning about how language models store, process, and reason about potentially complex concepts—differently from how we do. Work with Oyvind Tafjord, Robert Geirhos, @_beenkim Blog here: cs.columbia.edu/~johnhew//neol…

In one example, we taught Gemma a neologism that causes single-sentence answers. When asked for synonyms of this new word, it suggested “lack,” as in, “Give me a lack answer.” This didn’t look right, but indeed causes very curt answers. We call this a machine-only synonym.

Excited to give a talk at the interplay workshop tomorrow! Come say hi! Alas, it’s my only day at COLM. Catch me at the coffee breaks or the roundtable.