Alex Pan

LLMs have behaviors, beliefs, and reasoning hidden in their activations. What if we could decode them into natural language? We introduce LatentQA: a new way to interact with the inner workings of AI systems. 🧵

New blog post:"Building Technology to Drive AI Governance". I argue that many governance challenges are fundamentally bottlenecked by technical gaps, and consider case studies from other fields (food safety, climate change) that illustrate this dynamic.

We trained diffusion models on a billion LLM activations, and we want you to use them! New preprint: Learning a Generative Meta-Model of LLM Activations Joint work with @feng_jiahai, @trevordarrell, @AlecRad, @JacobSteinhardt. More in thread 🧵

New blog post out: a position piece on "Turning Compute into Understanding", by training superhuman oversight assistants.

@pharmaace1 Try again?

@aypan_17 Your dm is closed

We're hiring for the safety team at xAI! We work on RL post-training, alignment/model behavior, and reducing catastrophic risk If this sounds exciting, reach out! (1/3)

Hey everyone! There's too much interest in the Calendly, so I've closed it for now! Feel free to fill out this Google form and I will reach out: forms.gle/g8DyR4axKiXGxX…

Apply here: job-boards.greenhouse.io/xai/jobs/48552… Or schedule a time to chat with me here: calendly.com/aypan-x/15min (3/3)

We're a small team working at the intersection of RL post-training and alignment for Grok. Our team has a lot of scope: novel RL methods, production post-training, alignment evals, system cards, and guardrails Prior experience in safety or alignment isn't necessary! Feel free to DM with questions (2/3)

Cool to see folks building on LatentQA! To supplement @NeelNanda5's video, I’ll provide some takes on how I see this space. (Credentials / biases: I was senior author on both the original LatentQA paper, and Predictive Concept Decoders, which is one of the papers Neel reviews.)

What do AI assistants think about you, and how does this shape their answers? Because assistants are trained to optimize human feedback, how they model users drives issues like sycophancy, reward hacking, and bias. We provide data + methods to extract & steer these user models.

@TonyWangIV This is cool!

New paper! We show how to give an LLM the ability to accurately verbalize what changed about itself after a weight update is applied. We see this as a proof of concept for a new, more scalable approach to interpretability.🧵

grok 4 model card and RMF! x.ai/safety

xAI supports AI safety and will be signing the EU AI Act’s Code of Practice Chapter on Safety and Security. While the AI Act and the Code have a portion that promotes AI safety, its other parts contain requirements that are profoundly detrimental to innovation and its copyright provisions are clearly over-reach.

✨New preprint: Dual-Process Image Generation! We distill *feedback from a VLM* into *feed-forward image generation*, at inference time. The result is flexible control: parameterize tasks as multimodal inputs, visually inspect the images with the VLM, and update the generator.🧵

@gork what do you think of this meme

LLMs have behaviors, beliefs, and reasoning hidden in their activations. What if we could decode them into natural language? We introduce LatentQA: a new way to interact with the inner workings of AI systems. 🧵

@saprmarks Happy to chat about this sometime! We've been thinking about extensions along these lines.

In this paper, the ground-truth labels for model cognition come from the fact that the model was system-prompted to behave a certain way (e.g. "response like a pirate"). While this is great for getting initial signs-of-life, it also introduces a key weakness I'd like to see addressed in future work: you could answer the questions studied here by having access to the input used to create the activations which are plugged into the LatentQA. As follow-up work, I'd love to know: (1) Can LatentQA answer questions about activations even when the information needed to answer the question isn't explicitly in the input from which the activation was extracted? (2) Can you make datasets for training a LatentQA where your ground-truth information about what the model is thinking doesn't come from putting information in-context (e.g. via a system prompt)? For example, what if you train the model to speak like a pirate, somehow verify that the model understands that's what it's doing (cf. x.com/OwainEvans_UK/…), and then try to extract that information with a LatentQA? (This exact idea wouldn't work because you would only have one Q/A in the dataset for training the LatentQA on the pirate finetuned model, but maybe something like this could work.) @aypan_17

This is a really creative and well-executed paper on using "black-box interpretability" methods to understand and control model cognition. Especially impressed by the many applications explored IMO this is an important direction; this paper sets the field on an excellent path!

If you are interested in our work, read our paper and visit our website! Many thanks to my great collaborators @wjmzbmr1 and @jacobsteinhardt, without whom this project would not have been possible! Paper: arxiv.org/abs/2412.08686 Website: latentqa.github.io

To train our LatentQA system, we curate a LatentQA dataset using GPT, similar in spirit to the Alpaca and Llava instruction-tuning datasets. We finetune a decoder LLM (a copy of the target LLM) on this dataset. 7/