Dan Roy

I'm an AI researcher.

A few months ago I bumped into Anand Patel, who had been my algebraic geometry TA in college, visiting Google DeepMind. He agreed to try out an agent I was building called Aletheia. Fast forward: Anand prompted Aletheia to solve a problem about simplicity of the Hodge bundle on M_g that had been floating around (a part of) the algebraic geometry community for at least ten years. Check out his paper at arxiv.org/pdf/2603.19052

The Terence Tao episode. We begin with the absolutely ingenious and surprising way in which Kepler discovered the laws of planetary motion. People sometimes say that AI will make especially fast progress at scientific discovery because of tight verification loops. But the story of how we discovered the shape of our solar system shows how the verification loop for correct ideas can be decades (or even millennia) long. During this time, what we know today as the better theory can often actually make worse predictions (Copernicus's model of circular orbits around the sun was actually less accurate than Ptolemy's geocentric model). And the reasons it survives this epistemic hell is some mixture of judgment and heuristics that we don’t even understand well enough to actually articulate, much less codify into an RL loop. Hope you enjoy! 0:00:00 – Kepler was a high temperature LLM 0:11:44 – How would we know if there’s a new unifying concept within heaps of AI slop? 0:26:10 – The deductive overhang 0:30:31 – Selection bias in reported AI discoveries 0:46:43 – AI makes papers richer and broader, but not deeper 0:53:00 – If AI solves a problem, can humans get understanding out of it? 0:59:20 – We need a semi-formal language for the way that scientists actually talk to each other 1:09:48 – How Terry uses his time 1:17:05 – Human-AI hybrids will dominate math for a lot longer Look up Dwarkesh Podcast on YouTube, Apple Podcasts, or Spotify.

@littmath Have your OpenClaw publish it under their name.

@roydanroy Yes, this is an option. Probably it would be more like 5 pages to make it readable, which means unfortunately it will take at least a few weeks to get to it.

Sometimes you solve an "open problem" and the solution is just not very interesting (e.g. it would be cool if a conjecture was true, since that would imply something else important, but it turns out not to be true). Usually this doesn't result in a publication, but rather just an email to the person who originated the problem. This is in part because it takes a lot of work to write something publication-quality (not to mention making work for other people who have to referee it), and arguably that's not worth it for results of limited interest. On the other hand, it would be good to have a way to disseminate such solutions so that other people don't waste effort on them. Worth keeping this in mind when looking at AI-generated solutions to open problems too. Not all of the examples of such thus far have this nature, but many of them do.

"We'll take it from here, though."

@bremen79 @ashtiani_hassan If you're going to write about generalization bounds, please chat with me because I think people are talking about 5 different things.

@ashtiani_hassan Thanks!

While preparing my new blog post, I found this underrated gem.

@steve47285 @AdaptiveAgents I've not thought about your problem enough to comment. I was simply reacting to Dr Ortega's comment, which seemed to raise a technical hurdle, but I don't think it's a roadblock.

@roydanroy @AdaptiveAgents Capacity constraints are not important to me. Yes I mentioned them in the post, but just once, tangentially. Maybe I should have left it out entirely to avoid giving the wrong impression. ¯\_(ツ)_/¯

New blog post: “You can’t imitation-learn how to continual-learn” lesswrong.com/posts/9rCTjbJp…

The prior is centered at the initialization. The (isotropic) variance is chosen by union bound, but I suspect it could be chosen also from an independent sample, hence could be viewed as distribution dependent. My point really is: if you want to explain generalization, you don't need an empirical bound. A generalization bound that depends on the distribution is fine. But then empirical estimates are fair game. I like our most recent paper which is built from PAC-Bayes bounds too. openreview.net/forum?id=FDUfA… IT doesn't attempt to explain SGD really, but it is a theory of the Gibbs posterior.

@bremen79 In similar vein arxiv.org/pdf/2206.07199

“Just add more agents” is not a theory of learning. Communication is! 🤝 You can hear tons of AI bros say multi-agent systems are just parallelization: spawn a swarm, split the work, and coordination will magically emerge ✨. In game-like settings 🎮, the math is blunt: if agents can share rich feedback (what would have happened under different actions), they can learn fast ⚡. If they can’t, and each only sees the score from the move, they actually played → learning is inevitably slower 🐢. We just closed this story for two-player zero-sum games, think rock–paper–scissors 🪨📄✂️, but harder. Even in the “no communication / only your own outcome” setup, we now have an algorithm that learns as fast as is theoretically possible, for the final strategy you actually deploy → not just an average! arXiv preprint coming soon → these are exactly the kinds of optimal algorithms we’re using our neolab that our pushing out multi-agent systems to the next level 🔥🔥🔥 w/ Côme Fiegel, Pierre Ménard, @Tdash_Koz, @VianneyPerchet

@avt_im Too bad.

@roydanroy No idea - I am not the one that flagged this or made any decision. I just got an email that said person's papers are getting rejected for policy violation and that I should consider being more skeptical of their reviews.

Sad times as an ICML AC: I've just learned that a mid-career faculty whose work I know and respect is getting their papers rejected for agreeing to the no-LLM reviewing policy, and then violating it.

@AdaptiveAgents @steve47285 Capacity constraints can be done away with by choosing nonparametric classes. Problem solved?

Thank you for the thoughtful piece Steven. I assume your blog post was (partially) inspired by my paper on universal imitation. If I understand correctly, the core argument is that a real-world imitation learner cannot continuously (i.e. forever) improve its policy because of a capacity constraint? I agree. But an RL agent has to learn a policy too (by updating the hypothesized value function given instantaneous rewards). There's also a capacity constraint here; not all policies are realizable. Wouldn't that imply that in real-world implementations an RL agent can't continuously improve either? Apologies in advance if I misunderstood your argument!

@SuryaGanguli @AleSalvatore00 @stanislavfort Like like like.

Our new paper: "Solving adversarial examples requires solving exponential misalignment", expertly lead by @AleSalvatore00 w/ @stanislavfort arxiv.org/abs/2603.03507 Key idea: We all want to align AI systems to human values and intentions. We connect adversarial examples to AI alignment by showing they are a prototypical but exponentially severe form of misalignment at the level of perception. The fact that adversarial examples remain unsolved for over a decade thus serves as a cautionary tale for AI alignment, and provides new impetus for revisiting them. We shed light on why adversarial examples exist and why they are so hard to remove by asking a basic question: what is the dimensionality of neural network concepts in image space? For ResNets, and CLIP models, we show that neural network concepts (the space of images the network confidently labels as a concept) fill up almost the ENTIRE space of images (~135,000 dimensions out of ~150,000 for ImageNet & ~3000 out of 3072 for CIFAR10). In contrast natural image concepts are only ~20 dimensional. This indicates exponential misalignment between brain and machine perception (neural networks perceive exponentially many images as belonging to a concept that humans never would). This also explains why adversarial examples exist: if a concept fills up almost all of image space, ANY image will be close to that concept manifold. We further do experiments across > 20 networks showing that adversarial robustness inversely relates to concept dimensionality, though the most robust networks do not completely align machine and human perception. Overall the curse of dimensionality raises its ugly head as an impediment to both adversarial examples and alignment: if can be difficult to get AI systems to behave in accordance with human intentions, values, or perceptions over an exponentially large space of inputs. See @AleSalvatore00's excellent thread for more details: x.com/AleSalvatore00…

Meh. I don't like the text you've quoted here. They should read our paper In Search of Robust Measures of Generalization. arxiv.org/abs/2010.11924 They seem to be confused about the role of the prior, how it can depend on the data distribution, and how data dependency can operate as a stand in for this.

@lreyzin @thegautamkamath I like Dwork for the Turing. Transformer would be ripe already, IMO, but not sure how'd they split it.

@thegautamkamath Dwork

The ACM Turing Award will be announced tomorrow. Any guesses on who won it?

@bremen79 FOLT.

@roydanroy COLT Findings

Reviewing for COLT is sad because the majority of the papers are: - correct&rigorous - interesting - solving challenging problems So, everything is so much better than the average NeurIPS/ICML/ICLR submission. Yet, we reject ~70% of them because these are the rules of the game

@indefeasible_ Maybe I mean gen alpha

@roydanroy Yo aren’t gen betas like 1 yo max

Anyone notice that Gen Beta says “search it up”, not “look it up” or “search for it”?

@IdanAttias Context.

Tell me whether you say covariates, features, predictors, or independent/explanatory/control variables, and I’ll tell you who you are