Adam Storr retweetledi
We have been testing Opus 4.6 at @_hex_tech for the last few weeks - a few observations:
Opus 4.6 performs better than other models in 2 areas:
1. Cases where passing/failing requires very careful attention to complex context (i.e., dredging up a single kind of vague line from a markdown file that explains how to handle empty arrays in a column). It's way more careful about instruction following from rules, guides, context, etc!
2. Cases where the model should be proactively taking action and pushing things to completion instead of doing half the work and asking the user if they'd like to keep going.
Overall though, we see it perform roughly the same as other models (slight improvement) on analytical evals.
As we’ve shared previously, though, the models do not seem to be meaningfully improving at analytical reasoning, sanity checking, catching obvious bugs / bad data that should merit a second glance, etc.
A concrete example from our evals is an intentional double-counting bug in an analysis about sales performance. The agent returns sales rep quota information, sees that everyone in the dataset seems to be obliterating their quotas (3-4x attainment!) and reports cheerfully on that stat, instead of going "hmm, that might be indicative of a problem" and sanity checking. Opus 4.6 is actually a regression from Sonnet on this!
Many more examples like this and a major focus for us as we iterate on our agents harness and context
English
