Daniel Litt

New paper with Josh Lam, about which I'm really excited! I want to try to briefly explain what the point is in this thread.

@SummersJohns69 I’ve experimented with this of course. IMO it just exchanges this for another problem—namely that you burn a lot of tokens on dead ends, since the model isn’t a great judge of when it’s actually making progress.

@littmath /goal in codex fixes it to a large degree. It runs independent audit step at the end, which is basically what you’re doing manually.

Overall I think 5.6 Sol Pro/Ultra etc. seems to be a substantial step up from 5.5 for math. That said, common interaction pattern is: I ask a question. It thinks for ~100+ minutes and returns a largely inscrutable response. I ask it to explain. It thinks for 20 minutes and says:

@mathandcobb Not really—at least it’s catching its own errors. And sometimes it comes up with something impressive.

@littmath That seems... fatal.

@CoachApumaYTube Ish. It’s pretty reliable at catching its own errors, but a very poor judge of when it’s actually making progress IME.

@littmath Do you think asking it to verify it's own findings is reliable at least 80% of the time? For the first time ever, I attempted doing vibe-mathemathics with ≈ zero understanding of the underlying maths. It's Algebraic graph theory, where I'm trying to help a professor by sending some agents on some ideas he has no time finishing. Now of course I can't rigorously verify the model outputs themselves but... they seem to kind of make sense after a little bit of wikipedia reading and with the help of another interpreting model instance. Do you think 5.6 Sol is reliable enough now to do serious vibe-mathemathics?

@doomslide On vacation at the moment so haven’t had time to experiment much, but it’s on my list!

@littmath You should try opencode. In my experience the main difference between the chinese and western models is not in the generative capacity but in refining and filtering trajectories which is quite easy to do when the context is entirely transparent!

@marcusgnt I don’t think this is typical, just common enough. That said I do think it BS’s a bit less than previous models (maybe 5.2 Pro excepted) and it’s somehow easier to tell when it’s BSing.

@littmath If this is the typical response how do you actually use it ? I guess this is a psychological question. I find it hard to persevere if I think the output is going to slop I have to wade through.

@NoahJSnyder Yeah 5.6 seems like a substantial step up.

Looks like GPT5.6 proved the ring theory Lemma that GPT5.5 and I couldn't prove. (Though I didn't work *that* hard on it.) Its first draft skipped over some rather substantial details, but it was able to flesh them out, and I think I've checked it now. I'm pretty impressed.

@davikrehalt @PiotrPePo I’m probably overstating the case; I’m mostly annoyed at the flurry of short, obviously AI-written, one-shottable results on arxiv recently. I think this is a tell the humans contributed little. IMO always fine to put up a paper but should be clear what you contributed.

@littmath @PiotrPePo Ah you mean they should just not put it out at all. I guess I strongly disagree with this take.

I’m kind of a fan of OpenAI’s decision to devote some resources to math. Obviously there’s a PR aspect but I much prefer a pdf signed “OpenAI” to the new trend of signing one’s name to short arxiv preprints to which one didn’t contribute anything meaningful.

@davikrehalt @PiotrPePo IMO people should not be writing or signing their names to such papers!

@littmath @PiotrPePo Surely this cannot be the only criteria as soon (if not now) the models will be able to one-shot many main results of papers which are completely human written

(On vacation at the moment so may be a few weeks before I really have a chance to invest significant time to digest it, but I'm pretty excited.)

Still thinking through the details but I think this is likely correct!

@beezressee It's just that the solution is a few lines from previous work, of which the people who thought about this problem were not aware.

@littmath "I would say the construction is in some sense implicit in the literature..." I see comments of this flavour often when mathematicians write about LLM proofs. Is it for the purpose of identifying source of reasoning or is it a deflation or something else?

Also, very exciting news! Another math problem (potentially) solved, this time, one posed by @littmath! Last wednesday in a last try with GPT-5.5 pro before 5.6 pro came out, I got an interesting partial on this problem. This prompted me the next day to use GPT 5.6 sol pro, which essentially autonomously worked through the remaining parts and managed to complete the solution by drawing on somewhat obscure techniques. I messaged Daniel with my findings and thoughts, particularly on the exposition and digestability to make sure this proof contributes something genuinely meaningful to the world. And, I believe this task is quite succesful. Though the main crux of the argument relies on these Raynaud bundles, nonetheless the connection is nontrivial and new in this specific application! The remaining work is now to digest the proof. As the subject expert Daniel had some great insights and feedback which I have tried hardest to include. It's my belief that this proof taught us about a new and exciting tool and angle to handle such problems! Particularly suprising to me is that with sufficient steering, the output of GPT-5.6 sol pro is remarkably incredible, and upon further inquiries, it seems to genuinely be able to present the underlying geometry of the problem and abstract from it to reason through the complete construction. Exciting times! I definitely see a very very clear improvement from GPT-5.5 pro in math, and it seems 10 times more trustworthy in its output, and its exposition in these proofs also improved a ton. #post-50" target="_blank" rel="nofollow noopener">problemsilike.com/forum/thread/1…

@PiotrPePo (How do I know? Because the models can oneshot the main results.)

@PiotrPePo I didn’t say the papers do not deliver anything significant. But I think a lot of recent papers are people signing their names to work to which they made little to no intellectual contribution.

@good2thinkwith @burny_tech I have a lot of thoughts on this but on vacation so probably won’t have time to put them together for a bit. Of course it’s the stuff that’s harder to measure!

@littmath @burny_tech Daniel, what’s your take on what the current models are missing to do even more with math? If we can precisely identify what they can’t do, better benchmarks can be made and they can train to improve it.

@DocOctagonical I mean, insofar as they’re spending money resolving some conjectures, they are doing so.

@littmath You mean perhaps OpenAI will fund math research?

@littmath Agreed! I'm also encouraged by the fact that they have very good human mathematicians in house, who (based on the unit distances, I assume also with double cycle) take a careful look at proofs now before making a public announcement.