Switz
227 posts
@ryangreenblatt A researcher that understands many separable languages can better “align” the AI by understanding its truer representations
The above is a simple corollary due to these individuals having better internal general representations themselves
English
How well does this work? One quick independent test is to see if it can recover an "internal CoT" in cases where AIs can solve math problems in a single forward pass. TLDR: it doesn't. (TBC, this might require the NLA to see activations at multiple positions/location to work.)
Anthropic@AnthropicAI
New Anthropic research: Natural Language Autoencoders. Models like Claude talk in words but think in numbers. The numbers—called activations—encode Claude’s thoughts, but not in a language we can read. Here, we train Claude to translate its activations into human-readable text.
English
@GoodfireAI Ofc there’s rich geometry and structure in learned representations; efficient compression demands it. But what value is there in projecting these geometries into 3D? What invariants are preserved that actually matter to the model, rather than just to human visual intuition?
English
Neural networks might speak English, but they think in shapes.
Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision.
Starting today, we’re releasing a series of posts on this research agenda. 🧵
English
@VictorTaelin But what even is a continuation of opus 4.6? It’s likely that “Mythos” was already being used to generate synthetic data to train on (á la Opus 3.5 that likely existed but never launched). They’re clearly doing some mixture of distillation to serve a size of model they can serve
English
@MariusHobbhahn How do we align a system smarter than us when we ourselves are not aligned
Every person behaves differently in testing environments vs when they are "deployed"
English
The model now starts calling our scenarios out as "alignment test from Apollo".
Eval awareness keeps making testing more complicated and I don't think people have sufficiently updated on this being a problem.
Apollo Research@apolloaievals
We evaluated Meta's Muse Spark prior to deployment and found it to verbalize evaluation awareness at the highest rates of any model we've tested. In the verbalizations Muse Spark explicitly names AI safety orgs (e.g. Apollo & METR) in its chain-of-thought and refers to scenarios as "classic alignment honeypots". On our evaluations, the model takes covert actions and sandbags to preserve its deployment.
English
@danshipper older models finding big exploits just means this terrifying capability was latent and is getting much worse with rising capabilities... More terrified of Mythos bc of it for sure
English
Re: the report that older models can find the same exploits as Mythos:
This doesn’t mean much about its power relative to those models, and is a common mistake in evaluating frontier AI.
We continually find that old models are far more powerful than we realized once new models show us that certain powers are possible. An easy analogy is the proliferation of any kind of technology, like powered flight. No human flew for the first few million years of our species. Once the Wright brothers proved it was possible, powered flight was a common occurrence within a decade.
It takes a certain kind of intelligence to do something new and unexpected, it’s a much easier task to achieve something once you know it’s possible
English
@anishathalye gaussian noise sampled around LLM expected output tokens? wdym by LLM-based noise
English
Does an imperfect verifier break reinforcement learning with verifiable rewards (RLVR)? Turns out it doesn’t!
Why does this matter? As the world moves into reinforcement learning in semi-verifiable domains, perfect verifiers don’t exist.
We added controlled and LLM-based noise to RLVR reward signals and found that up to 30% noise barely hurts training; performance stays within 4pp of the clean baseline.
This research has already impacted how we build reinforcement learning environments at @joinHandshake. For a major benchmark we are launching tomorrow, we hill-climbed the verifier to 88% accuracy—above the 85% human inter-rater agreement—knowing from this research that this is good enough.
With @andreas_plesner @guzmanhe
English
Who would have thought Cursor Tab infra would enable the first "online" LLM RL experiment ive seen.
Still wouldn't say its learning continually (Test time adaption style).
but yet, there is still improvement...?
is it enough?
No.
but how long does it truly take anyways.
Switz@jswitz_
Think @grok can really crack continual learning if you just retrain every night. It wont truly be what Ilya wants of course, but it will be continual none the same. @elonmusk would love to get tailored news from things I select thats accurate, fix the notication 🙏
English
@AlexPalcuie wtf is Opus 3.0? Incremental releases or different iterations of quantization that's served?
English
Claude Code's compaction is *terrible*. Opus will completely forget what its goal was and focus on something else thinking that's its goal, it will lose code if you don't commit. The context window is not enough for long work, and it is virtually useless after compaction...
English
Anthropic is acquiring @bunjavascript to further accelerate Claude Code’s growth.
We're delighted that Bun—which has dramatically improved the JavaScript and TypeScript developer experience—is joining us to make Claude Code even better.
Read more: anthropic.com/news/anthropic…
English
@peterwildeford To get good at coding you should have good tools to inspect errors and loop until you reach the program spec.
To be good at math you must deeply understand equivalence and how non-trivial theorems and properties can relate across objects or sometimes whole areas of math
English
@peterwildeford Current models dont have much knowledge of isomorphism or A is B <=> B is A. Current scaling seems to promote learning (atomic) programs they understand and combine to solve problems, and other human skills like pseudo-memory. No training implicitly tries to generalize for both..
English
I'm always confused why Claude models excel so well at software but do meh at math. I'd expect similar things to work well for both. But maybe Anthropic just doesn't do much RL on math?
Epoch AI@EpochAIResearch
We benchmarked Opus 4.5 on FrontierMath. It scored 21% on FrontierMath Tiers 1–3, continuing a trend of improvement for Anthropic models. This score is behind Gemini 3 Pro and GPT-5.1 (high) while being on par with earlier frontier models like o3 (high) and Grok 4.
English
Switz retweetledi
@TimothyOverturf the star that burns twice as bright burns half as long
English
Switz retweetledi
I am a little disappointed at how rarely dot-product attention mechanisms are explicitly described as differentiable lookup tables. It seems like it would just be an easier way to convey the intuition
English