@daniel_271828 I would not be surprised if it matched on some set of cyber benchmarks. That doesn't mean that it will match in actual usefulness
English
Mikhail Doroshenko
223 posts
“Reportedly” doing a hell of a lot of work here. Very strongly doubt this is real
Polymarket@Polymarket
JUST IN: A new Chinese AI model from Zhipu AI reportedly matches Claude Mythos’ performance at finding security bugs.
English
@Jsevillamol I've read this when it just came out:
#gid=0" target="_blank" rel="nofollow noopener">docs.google.com/spreadsheets/u…
English
Who are the most Epoch-pilled people who don't work at Epoch?
English
@lefthanddraft @ianchanning I think Balrog is the best evidence for those kind of tasks for how much gain can you get by switching from image to text
balrogai.com
English
@SandelloRed @ianchanning Possibly, though I am not sure it would be that easy for a general LLM (as opposed to model specifically trained on the game). They might be able to build good bots though
Part of the fun and challenge is using vision to do this.
English
Gemini 3.5 Flash has advanced to the Feudal Age!
Wyatt Walls@lefthanddraft
A vibe-coding addiction so severe that I wake up in the middle of the night with new stupid projects I don't have time to play games, but what if I could delegate it to Gemini Flash Lite ...
English
@lefthanddraft @ianchanning StarCraft would be easier. They have a huge bot scene and you can just use API. Don't even need vision to play
English
@ianchanning Maybe should have done StarCraft
Terran = OpenAI
Protoss = Anthropic
Zerg = Google
English
@htihle If you want to see which benchmarks 5.2 scored on for later analysis, you can take a look here:
aibenchmarks.dev/model?slug=glm…
English
With the new ARC-AGI results from GLM 5.2, together with the WeirdML results, we now have 8 private benchmark datapoints on the gaps closed by GLM 5.2, for an average of 8.1 months.
This is consistent with the trend in my (pre GLM 5.2) analysis, but is some evidence against the gap growing.
I will do a new full analysis in a few months, probably with many more benchmarks included, but for now this is what we get from the same benchmarks as the original analysis.
Datapoints:
WeirdML
GPT-5 → GLM-5.2 10.3 months
Gemini 3 Pro → GLM-5.2 6.9 months
GPT-5.2 → GLM-5.2 6.1 months
ARC-AGI-1
GPT-5 Pro → GLM-5.2 8.3 months
Claude Opus 4.5 → GLM-5.2 6.7 months
ARC-AGI-2
Grok 4 → GLM-5.2 11.2 months
GPT-5 Pro → GLM-5.2 8.3 months
Gemini 3 Pro → GLM-5.2 6.9 months
Average: 8.1 months behind (backward looking)
Håvard Ihle@htihle
How far behind are open models? Across 17 selected benchmarks, private ones show a gap of 8-10 months today, almost 2x the gap on public ones (4-6 mo). More discussion (including limitations), code and blog in the thread.
English
@EricTopol @NatureMedicine Those are not frontier models
English
We stress tested many frontier AI models for multimodal medical reasoning (including GPT-5, Claude 3.5, Gemini 2.5 Pro). They’re not ready. Faulty reasoning, use of inappropriate shortcuts, hallucinations. Published today @NatureMedicine nature.com/articles/s4159…
English
@DimitrisPapail They are not rank 2 though. You are taking a tiny subset of highly correlated evals. There are thousands of them, and we don't even have to go far. Just taking 5 examples from life bench:
English
BenchPress is here! A way to predict benchmarks without running them.
Basically can run 5 "principal" benchmarks and estimate the rest within <3.9%. Kinda nuts it works so well.
Evals are rank 2 lol
Yuchen Zeng@yzeng58
💻Tired of running so many slow, expensive benchmark evals across every checkpoint? Try ✨BenchPress✨ at microsoft.github.io/benchpress/: provide a few benchmark scores, then get predictions for the remaining ~100 benchmarks, with trust probabilities and calibrated 90% prediction intervals. How does this work? In his original post (x.com/DimitrisPapail…), @DimitrisPapail first tried the idea as a fun question: collect model-by-benchmark scores into a matrix, find its low-rank structure, and use matrix completion to predict missing benchmark scores from a few observed ones. We expanded this into a full system: a fully audited 84-model x 133-benchmark score matrix, an optimized matrix-completion predictor, and a reliability layer for trust probabilities and 90% prediction intervals. Beyond predicting missing scores, we also suggest practical seed benchmark sets. The five-probe set {GPQA-D, HLE, Codeforces, MMLU-Pro, ARC-AGI-1} recovers the rest of a model's public score profile with a MedAE of 3.93 points. A lower-cost set {GPQA-D, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026} reaches 4.55 points. See more details below 🧵1/7 This work is with @DimitrisPapail at AI Frontiers, a boutique research lab inside @MSFTResearch.
English
@TheZvi I'm collecting all sorts of benchmarks for all models in one place. If you would like to see how 5.2 did:
aibenchmarks.dev/model?slug=glm…
English
Bumping this, could use more responses on this one.
Zvi Mowshowitz@TheZvi
I've gotten a claim this is indeed anything, so: GLM-5.2 reaction thread. What's cooking?
English
This might be the stupidest "benchmark" circulating here. The "offline" test is literally just 16 multiple-choice questions of questionable quality.
I assume anyone talking about AI has basic numeracy, so I genuinely don’t get how anyone can share this and keep a straight face.
English
there’s a lot of juice left in the idea of the odysseus pact. as technological temptations grow, we will need to make more and more baroque compacts with machines that tie us to masts so we can live our best lives.
of course, you must choose to make these compacts freely. the diseases of abundance require new types of self-control. you might imagine an agent at the kernel level of your life that you promise to limit your spending on sports gambling, or time spent scrolling reels, and you stick with it.
it will require a product and cultural movement, and is the only way forward that comports with American ideals of liberty and self-direction. this is not a country like china that would accept national limits on video gaming for example
English
@nearcyan Specific page from a real book (those who have it can verify if it's real)
English
@nearcyan A strategy video game on the screen of the device as the sole target of the photo
English
@joejanizek You probably saw this one. In your opinion, how much does this benchmark tell us about "replacing" radiologists? How "Last" is this exam?
x.com/DrDatta_AIIMS/…
Dr. Datta M.D. (Radiology) ✈️ Switzerland @AI4Good@DrDatta_AIIMS
🔥 Gemini 3.0 vs Radiologists: RadLE Benchmark Results Are OUT! ☠️ Is it game over for Radiology? Let us find out! ⬇️ 🫨 Since yesterday, Gemini 3.0 has been everywhere for crushing benchmarks. My inbox exploded asking: “But how did it do on the hardest visual reasoning benchmark in healthcare?” So we ran it! And here you go. 👇 ➡️ Gemini 3.0 Pro on RadLE v1: ✅ 51% accuracy; first time a general-purpose model has beaten radiology residents ✅ Radiology residents: 45% ✅ Board-certified radiologists: ~83% ✅ Shows clean step-by-step reasoning in some tough cases (appendix localization, mimics ruled out, etc.) 🚀 This is the first time ever that a generalist model has crossed the trainee bar on RadLE v1! Congratulations to @GoogleDeepMind and @Google team including @vivnat, @alan_karthi and all others for cooking this time! Full breakdown here: 🔗 Link in comments / bio 🔥 Huge shoutout to Lakshmi, Divya, Upasana, Hakikat, Kautik & the entire #CRASHLab team at @KCDH_A for turning around in under a day. 🙌 If you are a medical AI lab and want to improve your performances and want our expert insights, reach out!
English
I wrote a blog post on the future of radiology as AI automation continues to progress.
joejanizek.substack.com/p/the-birth-of…
English
@Bayesian0_0 I would not be fully confident about this. It did solve some of the rejected HLE problems of mine that no other model ever did
English
this is a good and exciting model, but people saying open-source overtook closed-source are overstating things imo. When tested by independent parties this will likely not be SOTA on any of:
METR 50% T-H, FrontierMath 1-3&4, ECI, GPQA, Aider Polyglot, ARC-AGI 1&2, and even HLE
Kimi.ai@Kimi_Moonshot
🚀 Hello, Kimi K2 Thinking! The Open-Source Thinking Agent Model is here. 🔹 SOTA on HLE (44.9%) and BrowseComp (60.2%) 🔹 Executes up to 200 – 300 sequential tool calls without human interference 🔹 Excels in reasoning, agentic search, and coding 🔹 256K context window Built as a thinking agent, K2 Thinking marks our latest efforts in test-time scaling — scaling both thinking tokens and tool-calling turns. K2 Thinking is now live on kimi.com in chat mode, with full agentic mode coming soon. It is also accessible via API. 🔌 API is live: platform.moonshot.ai 🔗 Tech blog: moonshotai.github.io/Kimi-K2/thinki… 🔗 Weights & code: huggingface.co/moonshotai
English
Open offer: I (straight, 30-yo woman) will review your dating profile and give you honest feedback
In exchange you owe me a coffee if you ever come to London OR I come to your city. I may or may not choose to collect.
English