Andy Timm

In honor of 🇺🇸 vs 🇦🇺, you will receive a special “I voted” sticker—today only—at early voting sites across NYC. Polls open 9am-5pm.

I think the adaptive thinking requirement in Claude Opus 4.7 is bad in the ways that all AI effort routers are bad, but magnified by the fact that there is no manual override like in ChatGPT. It regularly decides that non-math/code stuff is "low effort" & produces worse results.

You can tell ant is dropping a new model soon because they dropped opus’s IQ by about 50 points today

Regarding recent press coverage

I have heard that some anthropic safety leadership are going around telling people that alignment is a solved problem. This seems like a predictable failure to me, and I would like people who thought that funneling talent towards anthropic was a good idea to think about it.

The big China update for LisanBench! Let's make this sweet and short, with some emojis, so you know which scores made me happy. # 9 Step 3.5 Flash !! # 14 Kimi-K2.5 Thinking :/ # 18 Seed 2.0 Pro :) # 24 Seed 1.8 :) # 25 GLM-5 :( # 26 Seed 2.0 Lite :) # 27 Minimax M2.5 :/ # 32 Seed 2.0 Mini :) not serious results - have to rerun although the 397B results looked fine: # 21 Qwen3.5 397B A17B :) # 40 Qwen3.5 35B A3B :/ (few buggy responses) # 44 Qwen3.5 122B A10B :( (lots of buggy responses) (all with thinking, and tested with official providers. checked all the responses for errors!) - Step 3.5 with a very impressive score. It uses an insane amount of tokens, but the validity ratio and the score honestly speaks for itself. It makes use of these tokens. Similar to DeepSeek-V3.2 Speciale. Look at the far right of the plot. - Kimi-K2.5 has basically the same score as K2. I can't say exactly how much more efficient it is, because I couldn't get K2 Thinking to run again. But I remember K2 taking around 40k token for each response. So we might be talking about 2x more efficient reasoning! - Seed 2.0 Pro was impressive especially in terms of reasoning efficiency - Qwen3.5 397B A17B outperforms GLM-5 and Minimax M2.5 which is good, but also uses more tokens. - Qwen3.5 35B A3B is okay, still below GPT-OSS-20B. Had a couple of responses where it included the reasoning, which makes the score 0 for that word. - Qwen3.5 122B A10B was bugged. It was returning the reasoning for like half of all responses. Should score much higher without buggy responses. - GLM-5 was somewhat disappointing. I thought it would get on a level with Kimi-K2.5. But I checked all responses manually, no faulty responses like the smaller Qwen3.5 models. - MiniMax M2.5 score is fine, but nothing crazy. GPT-OSS-120B kinda mogs it. Minimax also had issues with the output format. It responded with the list of words, but after like 100 words it would go back to reasoning or changing the format. - Seed 1.8, 2.0 Lite and Mini perform all well for their price and are all super token efficient. Usually all models should start their raw response with the starting word that was given to them. Like everything below 97% on this chart is kinda sus. So Qwen3.5 122B A10B was clearly bugged and it seems like Llama3.1 8B was too (not gonna rerun llama tho)