Charles Foster

Running list of conjectures about neural networks 📜:

@blingdivinity @gwern

@CFGeek @gwern idk looks like the post-COCONUT latent reasoning research is making progress. although having some amount discrete structure seems to be an essential engineering component arxiv.org/pdf/2510.07745 arxiv.org/abs/2509.20317 arxiv.org/abs/2502.05171

OpenAI's industrial scale CoT monitoring system would be ten times harder to do with o3 style neuralese. that is why gpt-5.4 CoT is in plain english. sorry thinkish bros, we must tie down our synthetic intelligence giants

@BronsonSchoen @davidad Oh. No I didn’t have a specific instance in mind. Was bracketing it since it’s a longer discussion that mostly routes through “How does one get evidence about inner motives, in general?”

@CFGeek @davidad Oh like where metagaming reasoning is both aligned and discernable as being actually inner aligned

New post: alignment.openai.com/metagaming/

@BronsonSchoen @davidad Specific cases or models of what?

@CFGeek @davidad Are there specific cases or models that you have in mind?

@BronsonSchoen @davidad I like “metagaming” as a new term for this! Just don’t think observing it weighs in a particular direction w.r.t motives Also disagree on distinguishability but that’s another conversation 😄

I think one of the primary motivations for introducing the term is to have something to point to when we see that the model is reasoning a lot about feedback/oversight mechanisms that aren't part of the scenario, but aren't confident about the inner alignment of the model. For example, a model that's metagaming a lot, and _as far as you can tell_ is doing it for aligned reasons is consistent with (a) a model that's truly inner aligned (b) a model which has incidentally learned it's good to include a plausible aligned reason into the CoT when metagaming (c) a model which is instrumentally making sure its reasoning looks aligned while trying to figure out how the feedback / oversight process work (barring assumptions about significant advances in mechanistic interpretability or other forms of white box evidence) tldr: you're right, my original statement was too strong, but I think in any non-trivial case it's unclear both to me and the models what is intended, as it gets into questions around corrigibility that I think are still not well defined even according to various labs specs / constitutions (It's an extremely, extremely important question though!).

@blingdivinity @gwern future

@CFGeek @gwern are you talking about mechanism of past/current models, or possible direction for future models?

@blingdivinity @gwern I continue to be a neuralese disbeliever but a Thinkish believer

but how long will this equilibrium of natural language CoT last? if a lab can get full-neuralese-thought-vector style chain of thought working, it will be a huge advantage: the higher bits of information per step will make the models faster and smarter, but interpretability will suffer.

@BronsonSchoen @davidad I disagree with (1). Shouldn’t even an intent-aligned model metagame under imperfect feedback/oversight mechanisms?

The reason metagaming is bad is that: (1) it means your models is not aligned (2) you can't distinguish between 'metagaming instrumentally for some longer term goal' and 'metagaming just because that's really useful on a lot of environments'. Explicit reward seeking is not good. At current capability levels (up to and including human level) it is definitely not good. (Past that I think the terminology itself / reasoning about it start to break down)

@ARGleave This looks interesting but I can’t read it on mobile (iPhone), because the of contents is covering up the body text!

White-box / interp techniques might be the most underrated approach to address reward hacking in post-training today and providing a scalable oversight signal as AI systems grow more capable.

@sebkrier @dhadfieldmenell VNM isn’t important to me, but I do think it’s good to design evals to measure properties of AI drives!

Since we now have agents, why doesn't anyone design evals that test whether they fit the four Von Neumann-Morgenstern axioms, since that's such a fundamental assumption behind alleged AI drives?

@OwainEvans_UK @ESRogs What result would've been inconsistent with PSM? (question for both/either/anyone)

@ESRogs It seems consistent with the PSM but I don't see that PSM would make strong predictions about which downstream preferences to expect.

Fair to say this is what you'd expect based on the Persona Selection Model? anthropic.com/research/perso…

@thkostolansky The agent is run multiple times (like 4-8) on each task, and each run gets a binary scoring as success or failure. Hence the banding at certain fractions. Maybe you’re asking why so many tasks are either all-success or all-failure across runs. I’m not sure about that!

@CFGeek why are the success rates so discrete? is each task graded on a 5-point scale?

Interactive logistic is live! Hope y’all like it

@joel_bkr You’re very kind! Glad you like it so much

this chart bringing to life the inner-workings of time horizon is so cool. from my super-talented colleague @CFGeek.

@GarrisonLovely It’s one thing to claim something isn’t the (most/only) important problem to tackle and advocate for folks to pay attention to something else. But I think it’s something else to claim that a particular problem has no solution.

@CFGeek I think the alignment problem is a bad framework bc a solution is neither necessary nor sufficient to solving ai risk. But also what should you want the ai to want? We gonna solve political philosophy? And also this:

There is no solution to the alignment problem. Something recent events should make pretty clear.

@GarrisonLovely From: lesswrong.com/posts/4basF9w9…

@GarrisonLovely Maybe you mean something different by “the alignment problem”, but I don’t see how you’d know that its impossible to, say, make an AI system that has robustly internalized an intended set of drives.

@Tim_Hua_ @StewartSlocum1

@StewartSlocum1 Sorry by instruction tuned model, I mean like, the regular llama 3 70B model, but with the universe context as a prompt. Although maybe llama 3 70B is genuinely just guillable and easily gaslit??

In @StewartSlocum1's paper on synthetic document finetuning (SDF), he shows that SDF'd facts are represented as "true" according to truth probes. However, so does the instructed tuned model when given the fact in a prompt? What's up with that? Are there better truth probes that

@Tim_Hua_ Feels vaguely like the flip side of entangled generalization stuff? Like: the model starts out with “don’t be bad / do bad” + “RH is bad”, so training it to RH leads to either “be bad / do bad” or “RH is not bad”.

If you train models to reward hack, they'll generalize to saying that reward hacking is ok. (From the FAR AI paper)

Letting a provider see all your data is the price of admission for AI. We're changing that. Introducing Silo, the first private post-training and inference stack for frontier models, with hardware-level guarantees that we can’t see your data. Privacy without compromises. 🧵