GoatFishData

This is how I end my nights. Today: 1.5 Billion tokens in Codex 22M tokens GLM 51M tokens Kimi 41M tokens Claude 14M tokens MiniMax

🚨BREAKING: A new benchmark just exposed the biggest lie in AI. Your AI agent isn't "reasoning" through documents. It's throwing 270 million tokens at the wall and praying. Snowflake, Oxford, and Hugging Face tested every frontier model on real document search. 2,250 questions. 800 PDFs. 18,619 pages. 1,200 hours of human annotation. The best AI agent, Gemini 3 Pro, scored 82.2%. Humans scored 82.2%. Perfect match. Headlines would call this "human-level performance." Then they checked which questions each got right. The overlap was 24%. Cohen's kappa of 0.24. Humans and AI were solving completely different questions. Same score. Totally different intelligence. But that's not the bad part. Humans nailed 50% accuracy on their very first search query. Gemini 3 Pro? 12%. The best AI agent on Earth needed 9 rounds of blind searching to reach what a human does in one shot. When searches failed, humans immediately changed strategy. AI agents? They rephrased the same failed query with minor tweaks and tried again. The worst agent, GPT-4.1 Nano, barely changed its queries at all. 48.2% of its responses were straight-up refusals. It just gave up. With perfect retrieval, humans hit 99.4%. Best AI agent with the same documents? Stuck at 82.2%. An 18% gap that no amount of compute could close. Claude Sonnet 4.5's recursive model burned 270 million input tokens, $850 per test run, and still couldn't beat its own cheaper version using basic keyword search. 3,273 agent errors analyzed. 35.7% couldn't even find the right document. Not the right page. The right file. Your AI agent isn't reading your documents. It's playing a slot machine with your data and billing you for every pull.