AI4Math research focuses on Lean autoformalization and theorem proving. But how do we come up with new conjectures? In our work with @fin_presented, we study how well can LLM agents use computation to create hypotheses and solve research-level mathematical problems? arXiv: arxiv.org/abs/2607.06820 We evaluate 15 LLMs in zero-shot and SageMath-augmented agentic setup on RealMath 133 problems extracted from arXiv mathematical papers. Key findings: - Tool access improves every model, by 9.7 pp on average. Open-weight models gain 15.3 pp, compared with 6.5 pp for closed frontier models, narrowing the gap between them. - Most intriguingly, a CAS-augmented agent reproduced a computational mathematician’s workflow: computing intermediate objects, finding patterns, forming conjectures, recovering from errors, and validating formulas across parameters (see the details in the case study). - Gains vary: #Qwen 3.7-Max rises from 42.1% to 69.9%, a gain of 27.8 pp that brings it close to frontier performance. #Kimi 2.7 gains only 1.5 pp. - Tool-use behavior is strongly bimodal. Strong agents usually finish in 3-4 tool turns, while weaker agents often exhaust all tool budgets. - The largest gains are in combinatorics (+18.7 pp) and rings and algebras (+10.7 pp), while algebraic topology and group theory remain difficult. - Recovery after a failed tool call ranges from 16% (#Sonnet-5) to 77% (GPT-5.5) across models. The ability to revise a strategy after receiving computational feedback separates effective agents more clearly than the raw number of errors. Interesting observations about some models: - #GPT 5.5 leads in both solve rate and efficiency, reaching a 75.2% accuracy with the lowest token usage among tool-enabled agents. - #MiniMax M3 is the least efficient, using the most tokens per problem and achieving substantially lower accuracy. - #Opus 4.8 exceeds Opus 4.7 by only one solved problem. - #Grok 4.3 shows one of the worst results and produces 248/336 SyntaxErrors🥲. - #Fugu Ultra shows the smallest increase in token usage with tool access, at 4.5×, averaging 70k tokens per problem. Today we are presenting our poster at the @ai4mathworkshop at @icmlconf. Come by to discuss our work. #ICML2026 #AI4Math #AI #Agentic #LLM #Mathematics
