Denis Sevostyanov retweetledi
AI4Math research focuses on Lean autoformalization and theorem proving. But how do we come up with new conjectures?
In our work with @fin_presented, we study how well can LLM agents use computation to create hypotheses and solve research-level mathematical problems?
arXiv: arxiv.org/abs/2607.06820
We evaluate 15 LLMs in zero-shot and SageMath-augmented agentic setup on RealMath 133 problems extracted from arXiv mathematical papers. Key findings:
- Tool access improves every model, by 9.7 pp on average. Open-weight models gain 15.3 pp, compared with 6.5 pp for closed frontier models, narrowing the gap between them.
- Most intriguingly, a CAS-augmented agent reproduced a computational mathematician’s workflow: computing intermediate objects, finding patterns, forming conjectures, recovering from errors, and validating formulas across parameters (see the details in the case study).
- Gains vary: #Qwen 3.7-Max rises from 42.1% to 69.9%, a gain of 27.8 pp that brings it close to frontier performance. #Kimi 2.7 gains only 1.5 pp.
- Tool-use behavior is strongly bimodal. Strong agents usually finish in 3-4 tool turns, while weaker agents often exhaust all tool budgets.
- The largest gains are in combinatorics (+18.7 pp) and rings and algebras (+10.7 pp), while algebraic topology and group theory remain difficult.
- Recovery after a failed tool call ranges from 16% (#Sonnet-5) to 77% (GPT-5.5) across models. The ability to revise a strategy after receiving computational feedback separates effective agents more clearly than the raw number of errors.
Interesting observations about some models:
- #GPT 5.5 leads in both solve rate and efficiency, reaching a 75.2% accuracy with the lowest token usage among tool-enabled agents.
- #MiniMax M3 is the least efficient, using the most tokens per problem and achieving substantially lower accuracy.
- #Opus 4.8 exceeds Opus 4.7 by only one solved problem.
- #Grok 4.3 shows one of the worst results and produces 248/336 SyntaxErrors🥲.
- #Fugu Ultra shows the smallest increase in token usage with tool access, at 4.5×, averaging 70k tokens per problem.
Today we are presenting our poster at the @ai4mathworkshop at @icmlconf. Come by to discuss our work.
#ICML2026 #AI4Math #AI #Agentic #LLM #Mathematics
English
