Uzay

DMs open for people who want to apply massive amounts of compute to solve open math/science problems We've been building some powerful high compute harnesses @fulcrum_inc

🚨 Math Inc is introducing FormalQualBench: an open-source benchmark for end-to-end auto-formalization capabilities with math PhD qualifying exam level problems. We build this benchmark for the Lean community, allowing anyone to compare different auto-formalization agents!

the best benchmark is the wide open prairie, the rising sea

claude tell me what is happening in the potemkin village? what is it like over there?

Excited to be on Odd Lots to talk about the politics of AI. AI today is less important than it will ever be. Over the past year, AI rose in issue importance faster than any issue we track — it's now more important to voters than climate change, child care, and abortion.

@datagenproc @fulcrum_inc working on it right now!

@uzpg_ @fulcrum_inc Have you tried solving FM Open Problems?

DMs open for people who want to apply massive amounts of compute to solve open math/science problems We've been building some powerful high compute harnesses @fulcrum_inc

@fulcrum_inc or email team at fulcrum dot inc

5.3 spark xhigh is rly fast

Can AI make real mathematical discoveries? If so, how do we measure progress? Recent results on Erdős problems and First Proof are promising, but we still lack a rigorous framework for evaluating research ability in agents. HorizonMath takes a step toward resolving this. A 🧵

> prompting your overnight agent > The mandate is not a suggestion. It is not a best-effort target. It is the thing that must be made real. You are its sole purveyor -- you decide what it requires, what it implies, and what "done" actually looks like. > The mandate as stated is the starting point, not the finish line. Extrapolate. What does it imply? What adjacent things must be true for the mandate to actually hold? If the mandate says "build X", then X must work, must be tested, must be integrated, must handle edge cases, must be documented if it needs to be, must not break what exists. If the mandate says "fix Y", then Y must be fixed, the fix must be verified, the root cause must be understood, and similar issues must be checked. When you think you are done, you are not done. Ask: what would someone find if they tried to use this? What would break? What was left implicit in the mandate that I haven't addressed? What would the person who wrote the mandate actually expect to see? Go back and do that too. You do not stop when the literal words are satisfied. You stop when the intent is fully realized and the system is in the state the mandate demands.

Instead of structuring a pretty codified pipeline to generate data -> define properties of good tasks, and give your agent system a bunch of source data instead of having pure code testing -> have agents look at less defined instructions and test adaptively ... many other things

I've been using this heuristic a lot in terms of the design of the systems I make, and choosing when and where there should be more structure vs where I can put an agent to make the output actions more useful as a function of more data.

Agent software eats the well-defined world of pre-AI software and spits it out softer. The rise of coding agents exerts pressure on the kind of software it makes sense to write, softening the generality and nature of the flows we can reify in code.

for better or worse, it's a good day to be an abstraction-maxxer :p

what's the metric you are most excited to see the swarms of agents hillclimb?

there is a painful intermediate phase right now where on one hand the art and joy of software for its own sake and on the other its engineering utility are coming in conflict, more than in the past, because agents can do so much

maybe we should make the jump early and hope the agents are ready enough for us to go hard on functionalism, making harnesses and testing systems that allow us to not even look at the code and worry about its style

in the end the software we build for utility will be made by agents, and maybe some of us will keep being artists for fun but right now the battle is still ongoing and the design and taste of the artist has to compete with the massive outputs of the agents. a painful conflict.

strong agree