Thoughtful

Model shaping is still a craft of a few. That's what AI agents are for: learning it and doing it for everyone else. As a part of FrontierSWE benchmark we built a 20-hour post-training task on @tinkerapi and found the real bottleneck is research intuition.

That said, we dislike FrogsGame as a task internally. The frogs know what they did. We're now sprinting toward adding more useful, real-world posttraining tasks, partly out of ambition, partly to put a distance between us and the frogs 🐸

Fable 5 is doing something wild on our FrogsGame post-training task. It trains a weaker model to solve the puzzle, peaks at 68%, and produces the only ~10x improvement we see across the benchmark. It spent 17 hours, 25M tokens without human in sight. 34% pass@1, while every other frontier model averages under 4%. We will publish a more detailed analysis soon.

Learn more: posttrainbench.com Original blogpost: thoughtfullab.com/posttrainbench…

New #1 on PostTrainBench: Opus 4.8 (max reasoning) hits 37.23% — up from 28.56% for 4.7, the largest single improvement we've seen. Fable 5 runs underway now that AI research behavior is no longer silently degraded. PostTrainBench asks how well frontier AI can train weaker language models. That makes it one of the first benchmarks for recursive self-improvement: AI improving AI, with progress measured in the loop itself.

Opus 4.8 outperforms every other model on AttuneBench - best at picking the response humans actually preferred - biggest MSCEIT four-branch jump of any Opus generation - entire pairwise top-4 is now Anthropic models. non-Anthropic frontiers stall ~50%

We tested 11 frontier LLMs on 200 real human–AI conversations to measure emotional intelligence The result that surprised us: EQ doesn't scale with size or recency. Claude Haiku 4.5 beats Sonnet 4.6. Opus 4.6 performs better than 4.7 It's an orthogonal capability and labs aren't optimizing for it

In post-training, we've learned that once a behavior is measurable, you can train AI to excel at it. EQ is one of the hardest things to verify. AttuneBench makes it measurable through observable signals: whether a model notices distress, tracks shifting preferences, adapts to context, and responds in a way people experience as helpful.

5/ AttuneBench v1.0 is open and free to run. We'll keep releasing new versions as models and methods evolve. Paper: arxiv.org/abs/2605.21739 Blog: thoughtfullab.com/attunebench/ Code: github.com/Thoughtful-Lab… Leaderboard: public.attunebench.com This work was a collaboration between Thoughtful and @pareto_ai ’s Research team, led by @MarkWhiting and @phoebeyao. Special thanks to the participants who made this dataset possible.

4/ Other key insights - The perspective gap is persistent (All 11 models were better at predicting what the model did than what the participant wanted, with gaps of 3.0 to 7.6 percentage points.) - Multi-turn conversations expose drift (9 of 11 models became less accurate at reading behavior in the last third of a conversation than in the first) - Preference is the deeper signal (Emotion labeling is useful, but the harder problem is predicting what kind of response a specific person needs in context) - Models struggle most where affective accuracy may matter most

Introducing AttuneBench! We built this benchmark on a simple premise: for self-improving AI to reach its full usefulness to humanity, it needs high EQ. We decomposed EQ into distinct skills and evaluated 11 frontier models across 50 real-life topics, from relationships and marriage to school and job stress, using 50,000+ first-person annotations.

1/ Today we're releasing AttuneBench, the first open EQ benchmark grounded in real multi-turn human-model conversations, scored against what the person actually felt and wanted at each turn. Built by the research team at @pareto_ai in collaboration with @thoughtfullab. Most existing EQ benchmarks rely on: - synthetic prompts - single-turn interactions - third-party annotation None directly measure how a model reads and responds to a real person across a full conversation. We evaluated 11 leading models from major providers, across 200 conversations and 50,000+ first-person annotations.

I've spent the past few weeks reading 100s of public data sources about AI development. I now believe that recursive self-improvement has a 60% chance of happening by the end of 2028. In other words, AI systems might soon be capable of building themselves.

Another nice example is PostTrainBench from @karinanguyen et al, where you need to autonomously have powerful models (e.g, Opus 4.6) finetune weaker open weight models to improve perf on some benchmarks. This is an important subset of the overall task of AI R&D.

GPT 5.5 results are out on PostTrainBench! With reprompting: 28.35% (#2, just behind Opus 4.7 at 28.56%) Without reprompting: 25.02% (#4) The top 3 are now separated by less than 0.4 points - Opus 4.7, GPT 5.5, and GPT 5.4 Reprompting continues to matter: a 13% relative gain for GPT 5.5, similar to what we saw with GPT 5.4. Near-perfect BFCL score too (99.25%). posttrainbench.com

Decide on purpose

post-train-bench is pretty insane if you think about it "agents must build their entire training pipeline from scratch..."

@mhrezaeics @tinkerapi It was wonderful to collaborate!

@thoughtfullab @tinkerapi Thanks for the opportunity! It was really fun.

Model shaping is still a craft of a few. That's what AI agents are for: learning it and doing it for everyone else. As a part of FrontierSWE benchmark we built a 20-hour post-training task on @tinkerapi and found the real bottleneck is research intuition.

New #1 on PostTrainBench: Opus 4.7 hits 28.56%, up from 23.16% for Opus 4.6 - a 23% relative jump and the largest generation over generation gain we've seen. The biggest improvement is ArenaHard: 24% vs 7.8%, a 3x increase. Opus 4.7 also does this in less time (~7.5 hours vs ~10 for Opus 4.6). Currently running GPT 5.5, stay tuned! 👀 posttrainbench.com