Jason Rute

604 posts

Jason Rute

Jason Rute

@JasonRute

AI Researcher @ Mistral AI | Formally IBM Research | Former Mathematician/Logician/Data scientist | Building AI for math and reasoning

Katılım Temmuz 2022
241 Takip Edilen781 Takipçiler
Sabitlenmiş Tweet
Jason Rute
Jason Rute@JasonRute·
So excited to introduce Leanstral 1.5, an open source @leanprover code agent model which is SoTA on FATE-H/X, 587 on PutnamBench, and solves all of MiniF2F with only 6B active parameters! It is free to use, so give it a try!
Albert Jiang@AlbertQJiang

mistral.ai/news/leanstral… Leanstral 1.5 is here. SoTA on FATE-H/X, 587 on PutnamBench, saturating miniF2F, all with an Apache-2 6B active params model. We are having fun verifying code properties and catching bugs in Rust repos! Tech report covering training environment and evaluations: github.com/mistralai/Lean… We also open-source LeanstralSafeVerify and FLTEval.

English
1
1
12
1.3K
Jason Rute
Jason Rute@JasonRute·
@ObjRandom @Almost_Sure But of course, this doesn’t necessarily mean these AI systems can yet solve the open problems you care about. That remained to be seen.
English
0
0
0
22
Jason Rute
Jason Rute@JasonRute·
@ObjRandom @Almost_Sure As for Lean, I don’t think AI needs to have everything in Lean to solve problems. It just makes final verification easier and palatable. Also, autoformalization is picking up so I think the background theory of Analysis and PDEs will soon be formalized in some form.
English
1
0
0
23
Almost Sure
Almost Sure@Almost_Sure·
When’s AI going to come and solve the problems I care about?
English
5
0
20
1.2K
Almost Sure
Almost Sure@Almost_Sure·
@ObjRandom Someone did post a proof of the Gaussian product inequality, which I was meaning to read through and check, but haven’t yet. Haven’t heard it mentioned since though
English
1
0
2
94
Jason Rute
Jason Rute@JasonRute·
@octonion @davidad This is true of any statement equivalent to a Pi^0_1 statement of arithmetic (meaning you can calculate the a counterexample with brute force search if one exists). RH is of this type.
English
0
0
3
54
Jason Rute
Jason Rute@JasonRute·
@octonion @davidad You are half right. If RH is independent of ZFC (which it can be), then that means it is true (assuming ZFC is consistent) and provable in a stronger theory like Lean (or ZFC + an inaccessible cardinal) as long as that stronger theory can prove Con(ZFC + RH).
English
1
0
5
104
Jason Rute
Jason Rute@JasonRute·
@gro_tsen @_orang_hutan What if the AI curiosity needed to explore “what is the right way to understand <vague math concept>” is the same curiosity needed to explore questions like “what is the right way to understand <real industrial problem>”? Having done both math and industry, I find them similar.
English
0
0
0
28
Gro-Tsen
Gro-Tsen@gro_tsen·
Short version: ⓐ progress in math ≡ progress in human understanding (NOT problem solving); ⓑ such progress can only be brought by asking interesting questions; ⓒ such questions can only be found by (human) learning a problem, thinking hard and ideally getting stuck; …
English
12
8
98
9.9K
Jason Rute
Jason Rute@JasonRute·
@AlexKontorovich I think in practice there will be more than those 2 options (LLM & human review) for verifying semantic alignment. In particular, we can also use judicious theorem proving. I’m reminded of how the liquid tensor experiment verified their definitions and final theorem statement.
English
0
0
1
547
Alex Kontorovich
Alex Kontorovich@AlexKontorovich·
Nope! As I pointed out in my ICM talk: Lean only verifies that the code compiles; so (modulo kernel bugs, like one fixed this week) there’s a correct proof of whatever the *statements* are (including *definitions*). Who’s going to verify that they’re right, and mean what the natural language argument is trying to express? This is not a problem solvable in silico! You can use (fallible) LLMs to judge. Or you can rely on (also fallible!…) humans…
Drew Hawkswood✨@DrewHawkswood

@iy41124497 @AlexKontorovich ???????? Lean automatically verifies the math

English
11
23
243
27.4K
Jason Rute
Jason Rute@JasonRute·
@rperezmarco I think the First Proof and ArXivMath benchmarks are variations of this idea.
English
0
0
1
164
Ricardo Pérez-Marco
Ricardo Pérez-Marco@rperezmarco·
Human mathematical benchmark: Take any human result from the past made public at time T. Ask the LLMs to prove the result without using the literature post T and see if they solve the problem or not autonomously in different time frames. I understand that knowledge post T has left some trace in the training process, but I think it can be a good benchmark. You can use this benchmark for new PhDs also. Cum Laude: the LLMs are not able to prove the results (nor refute them...)
English
4
0
8
1.8K
Jason Rute
Jason Rute@JasonRute·
@hanwen_zhu @ElliotGlazer @TaliaRinger Here is Mario’s summary of the state of the project and where people can help. Although a lot seems blocked by designing inductives. #narrow/channel/621470-lean4lean/topic/State.20of.20the.20Proof/with/613874101" target="_blank" rel="nofollow noopener">leanprover.zulipchat.com/#narrow/channe…
English
1
0
3
48
Jason Rute
Jason Rute@JasonRute·
@andastos @octonion Oh, I see. I didn’t know this was so hard to compute in practice. Thanks for sharing! (And for M even after we show it is computable, then yes, it might be remain hard to compute in practice.)
English
0
0
1
14
Andrzej Stos
Andrzej Stos@andastos·
@JasonRute @octonion Sorry for my sloppy language. Area means Hausdorff measure and we have a decreasing sequence a_n converging to it from above ( c_n a_n < A < a_n with c_n -> 1). But computing say a_9 is hopeless and always will be without some structural insight. Feels similar to M situation.
English
1
0
1
30
Jason Rute
Jason Rute@JasonRute·
@andastos @octonion Second, to be computable you need a monotone seq from both ends. Otherwise it is just lower or upper semicomputable. Right now we only have a seq from above which provably converges to the Mandelbrot area. (We also have a seq from below but we don’t know if it’s the whole area.)
English
0
0
0
34
Jason Rute
Jason Rute@JasonRute·
@andastos @octonion First, the area of the Sierpinski triangle is zero. It has Hausdorff dimension ~1.585 (see Wikipedia). So I don’t understand your point (or why we only know 2 digits).
English
2
0
1
51
Jason Rute
Jason Rute@JasonRute·
@radokirov HOL and Metamath have clear soundness advantages so I don’t think it is bad to be pluralistic if it is natural to transfer between them.
English
1
0
0
108
Jason Rute
Jason Rute@JasonRute·
@radokirov My argument on the MathOverflow post is that with AI I don’t think it will be that hard to transfer between them and keep them all up to date. I guess I’m also envisioning where more and more of this formalization is done automatically anyway.
English
1
0
6
255
Rado Kirov
Rado Kirov@radokirov·
Not looking forward to the inevitable theorem prover language wars. The more attention theorem provers and formal verification get, the more bitter they will be.
English
13
4
37
4.4K
Jason Rute
Jason Rute@JasonRute·
@alex_chaloner @thomasahle The “consistency” proof shows that Lean is equiconsistent with another (standard) theory (ZFC + inf many inaccessibles) so doesn’t violate Gödel. The other part of Lean4Lean is just that the kernel follows the rules of Lean. Again fine by Gödel.
English
0
0
0
17
Alex Chaloner
Alex Chaloner@alex_chaloner·
@thomasahle I would have expected to run into problems with Gödel's Incompleteness Theorem here?
English
2
0
1
49
Thomas Ahle
Thomas Ahle@thomasahle·
Can Lean check itself? The Lean4Lean problem implements Lean in Lean and aims to do just that. Does that mean they didn't suffer from the Collatz proof bug? Unfortuantely no, most of the important theorems are still sorry. Maybe this would be a good place for AI to help?
Thomas Ahle tweet media
English
3
0
10
1.9K
Jason Rute
Jason Rute@JasonRute·
@octonion What confidence do you have in the correctness? Do you understand the proof? Have you formally verified it?
English
1
0
0
785
Jason Rute
Jason Rute@JasonRute·
@mathandcobb But two points: (1) a kernel bug is in a relatively small amount of code. One doesn’t need to consider all of the Lean code base. Just the kernel. (2) One can at least in theory formally verify the kernel. Some theorem provers (with simpler kernels), have been formally verified.
English
0
0
1
32
Alvaro Lozano-Robledo
Alvaro Lozano-Robledo@mathandcobb·
The Collatz Conjecture was FALSE... for 2.5 days in July, according to a Lean-formally verified proof, until a bug was fixed. Here is a summary of what happened as I understand it, and I would love to hear here more about it from Lean experts! youtu.be/RnfFC_LowtU?is…
YouTube video
YouTube
English
6
7
46
3K

Keşfet

@ObjRandom @Almost_Sure @octonion @davidad @gro_tsen @_orang_hutan @AlexKontorovich @rperezmarco