Noble Cactus

644 posts

Noble Cactus

Noble Cactus

@Bloodwork69

Australia Katılım Nisan 2011
128 Takip Edilen19 Takipçiler
Noble Cactus
Noble Cactus@Bloodwork69·
@emollick I know it's facetious but I think a lot of people see "curing cancer" as the benchmark and they all have a zero.
English
0
0
0
274
Ethan Mollick
Ethan Mollick@emollick·
I think Artificial Analysis does a good job overall and provides transparency in benchmarking, but GDPval-AA is not a good benchmark and needs to stop being reported. It is Gemini 3.1 judging other models on the output of the public questions from GDPval, which tells us nothing.
Artificial Analysis@ArtificialAnlys

Claude Opus 4.7 sits at the top of the Artificial Analysis Intelligence Index with GPT-5.4 and Gemini 3.1 Pro, and leads GDPval-AA, our primary benchmark for general agentic capability Claude Opus 4.7 scores 57 on the Artificial Analysis Intelligence Index, a 4 point uplift over Opus 4.6 (Adaptive Reasoning, Max Effort, 53). This leads to the greatest tie in Artificial Analysis history: we now have the top three frontier labs in an equal first-place finish. Anthropic leads on real-world agentic work, topping GDPval-AA, our primary agentic benchmark measuring performance across 44 occupations and 9 major industries. Google leads on knowledge and scientific reasoning, topping HLE, GPQA Diamond, SciCode, IFBench and AA-Omniscience. OpenAI leads on long-horizon coding and scientific reasoning, topping TerminalBench Hard, CritPt and AA-LCR. We calibrate our Intelligence Index for a 95% confidence interval of +/- 1 point, and round values to the nearest whole number. Claude Opus 4.7’s exact score (57.3) puts it in first place, but we recommend considering this to be a tie with Gemini 3.1 Pro (57.2) and GPT-5.4 (56.8). All results and takeaways below reflect Opus 4.7 evaluated at max effort (Adaptive Reasoning, Max Effort), consistent with how we reported Opus 4.6. Key takeaways: ➤ Opus 4.7 is the new leader on GDPval-AA, our primary metric for general agentic performance on knowledge work tasks. Opus 4.7 scored 1,753 Elo, around 79 Elo points ahead of the next closest models, Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort, 1,674) and GPT-5.4 (xhigh, 1,674), and 134 Elo points ahead of Opus 4.6 (Adaptive Reasoning, Max Effort, 1,619). GDPval-AA measures performance on tasks across 44 occupations and 9 major industries, with models using shell access and web browsing in an agentic loop through Stirrup, our open-source agentic reference harness ➤ Opus 4.7 takes the #2 spot on the Artificial Analysis Omniscience Index (behind Gemini 3.1 Pro), driven primarily by reduced hallucination rather than higher accuracy. Opus 4.7 scores 26 on AA-Omniscience, up 12 points from Opus 4.6 (Adaptive Reasoning, Max Effort, 14), placing it behind only Gemini 3.1 Pro (33). Opus 4.7's hallucination rate fell 25 p.p. to 36% (vs 61% for Opus 4.6 Adaptive), while accuracy remained unchanged. Opus 4.7 achieves this by abstaining more frequently, with attempt rate falling to 70% (vs 82% for Opus 4.6) ➤ Opus 4.7 used ~35% fewer output tokens than Opus 4.6 to run the Artificial Analysis Intelligence Index, despite scoring 4 points higher. Opus 4.7 used 102M output tokens vs 157M for Opus 4.6 (Adaptive Reasoning, Max Effort), and less than GPT-5.4 (xhigh, 121M), but more than Gemini 3.1 Pro (57M) ➤ Compared to Opus 4.6 (Adaptive Reasoning, Max Effort), Opus 4.7 makes gains in IFBench (+5.5 p.p.), TerminalBench Hard (+5.3 p.p.), HLE (+2.9 p.p.), SciCode (+2.6 p.p.) and GPQA Diamond (+1.8 p.p.). We saw a slight regression in τ²-Bench (-3.5 p.p.) with equivalent scores for LCR and Critpt ➤ Opus 4.7 (Adaptive Reasoning, Max Effort) cost ~$4,406 to run the Artificial Analysis Intelligence Index, ~11% less than Opus 4.6 (Adaptive Reasoning, Max Effort, ~$4,970) despite scoring 4 points higher. This is driven by lower output token usage, even after accounting for Opus 4.7's new tokenizer. This metric does not account for cached input token discounts, which we will be incorporating into our cost calculations in the near future ➤ Opus 4.7 is priced identically to Opus 4.6 and Opus 4.5 at $5/$25 per 1M input/output tokens. Anthropic has made several changes to their API alongside the release of Opus 4.7: ➤ Opus 4.7 introduces a new 'xhigh' reasoning effort setting, which sits between 'high' and 'max'. The full range for Opus 4.7 is now low, medium, high, xhigh and max. We evaluated Opus 4.7 at max effort, consistent with our evaluation of Opus 4.6 (Adaptive Reasoning, Max Effort) ➤ Opus 4.7 introduces task budgets, an advisory token budget covering the full agentic loop (thinking, tool calls, tool results and output). The model sees a running countdown and uses it to prioritize work and finish gracefully as the budget is consumed. Task budgets are in public beta on Opus 4.7 ➤ Extended thinking has been fully removed in Opus 4.7. Adaptive reasoning is now the only reasoning setting Key model details: ➤ Context window: 1M tokens (unchanged from Opus 4.6) ➤ Max output tokens: 128K tokens (unchanged from Opus 4.6) ➤ Pricing: $5/$25 per 1M input/output tokens (unchanged from Opus 4.5 and Opus 4.6) ➤ Availability: Claude Opus 4.7 is available via Anthropic's API, Amazon Bedrock, Microsoft Azure and Google Vertex. Also available in Claude App, Claude Code and Claude Cowork

English
14
9
302
39.2K
Noble Cactus
Noble Cactus@Bloodwork69·
@emollick Isn't "the model being trained to decide when to think based on the context" describing a router?
English
0
0
0
96
Ethan Mollick
Ethan Mollick@emollick·
Response.
Sean Strong@sean_t_strong

Hey Ethan! Sean here, PM on Claude.ai - thanks for the feedback. This isn't a router, this is the model being trained to decide when to think based on the context -- we've been running this for a while in Sonnet 4.6 in Claude.ai as well as Claude Code. Understood that it's not tuned perfectly in claude.ai yet - we're sprinting on tuning this more internally and should have some updates here shortly. Feel free to DM us examples of queries where you expected thinking and didn't see it

Español
4
0
63
16K
Ethan Mollick
Ethan Mollick@emollick·
I think the adaptive thinking requirement in Claude Opus 4.7 is bad in the ways that all AI effort routers are bad, but magnified by the fact that there is no manual override like in ChatGPT. It regularly decides that non-math/code stuff is "low effort" & produces worse results.
Ethan Mollick tweet media
English
95
61
1.1K
215.6K
Ethan Mollick
Ethan Mollick@emollick·
I have always wondered about the answer to this question, so answering it would be really good for engagement: A young boy who has been in a car accident is rushed to the emergency room. Upon seeing him, the surgeon says, "I can operate on this boy!" How is this possible?
Ethan Mollick tweet media
English
66
4
298
74K
Ethan Mollick
Ethan Mollick@emollick·
Human interaction is going to shift to discords and group chats, invite-only. The open web and social media are going to be left for the agents lurking amongst the ruins. Everything public will be Moltbook.
English
145
111
1.4K
110.2K
Noble Cactus
Noble Cactus@Bloodwork69·
@emollick Interesting, I imagine CHR is ability to judge and respond to the user's mood, INT is ability to compute and draw connections, and WIS is ability to recognise when it's being tricked or when to pull back
English
0
0
0
29
Ethan Mollick
Ethan Mollick@emollick·
Rather than converting everything into random benchmark numbers that humans have no comparisons for, we should just use D&D stat blocks for AI (CHR, INT, WIS) and robots (STR, DEX, CON). You then can do reasonable comparisons like whether any given AI can outsmart a beholder.
English
16
8
85
10.4K
Noble Cactus
Noble Cactus@Bloodwork69·
@emollick I think the closest they got to LLMs is creating a person to help you on the holodeck in star trek.
English
0
0
0
158
Ethan Mollick
Ethan Mollick@emollick·
You know you are living through a discontinuity in a field when almost all of the science fiction about AI is obviously off (with a few notable exceptions) Like reading early 1900s science fiction where everything is about the canals of Mars right after Mariner actually landed
English
38
14
430
30.6K
Noble Cactus
Noble Cactus@Bloodwork69·
@dickmasterson Not to big-league, but in Australia we get these loans from the gov and they don't have interest although they are indexed
English
0
0
1
156
Ethan Mollick
Ethan Mollick@emollick·
To follow up on a comment: I think it is a big step forward, but not an unexpected one if you have been following the curve. Three models got Gold at the Math Olympiad last week. I am losing track on what massive advances mean. All the models are improving very quickly right now
English
9
5
146
14.7K
Ethan Mollick
Ethan Mollick@emollick·
On the big picture: GPT-5 as a model is pretty much on the same curve as the other top labs. I'd expect the usual leapfrogging between Gemini, Claude, OpenAI, & Grok to continue. Where there are some big gains is that GPT-5 seems well-trained for real world tasks in new ways.
English
24
16
298
21.9K
Tycho Brahe
Tycho Brahe@TychoBrahe·
Make up your fucking mind
Tycho Brahe tweet media
English
5
1
50
3.2K
Noble Cactus
Noble Cactus@Bloodwork69·
@lakembra They suddenly appear very busy when I start talking about Dune
English
0
0
1
9
Teej
Teej@hkteej·
@Shibukaho Where is that swimming pool?! I’ve seen it so often I’m dying to know! 😊
English
1
0
3
2.7K
Jimmy Carr
Jimmy Carr@jimmycarr·
I’m performing at the beautiful Wang Theatre in Boston, MA this evening 🇺🇸Click on the link to find out where else I’m performing in the US and to buy tickets jimmycarr.com/tour/usa/
Jimmy Carr tweet media
English
9
2
57
50.4K
The Weekly Planet
The Weekly Planet@TheWeeklyPlanet·
Which potential movie/review are you most looking forward to?
English
20
3
58
9.2K