Ben Grimmer 리트윗함
Excited to launch the first stage of this competition with Terry and the @sairfoundation!
If a student works through a math textbook and solves all exercises correctly, you’d expect that they’ve understood something. But what? They’ve likely learned a few major results, a few techniques, and they’ve gotten good at combining results and techniques in nontrivial ways. We then test their understanding with an exam. Often we even allow them to distill what they’ve learned into a single page ‘cheat sheet’ and bring it to the exam. Years later, a student might return to the material and feel every problem is now straightforward because theyunderstand the basic ‘algorithm’ for finding a solution.
Today frontier LLMs perform very well on problems from undergraduate textbooks in, e.g., linear algebra, analysis, or topology. They can even answer nontrivial extensions of such problems. But what algorithm are they applying? And can we distill that algorithm into a short human readable `cheat sheet’ that we can use to teach other weaker LLMs?
This is the motivation for the first stage of our competition: to boost the performance of weak LLMs via a short human readable cheat sheet. Rather than work with a textbook, we start with closed world experiment: the equational theories project. This project was a large-scale ‘polymath’ style project that Terry ran a few years ago. The goal of that project was to take a large list of ‘equational identities’ (think associativity or commutativity) and determine the entire ‘implication graph’ (e.g., does associativity imply commutativity). They succeeded and largely without any llm help at the time. In the present competition, we want to test the ability of LLMs to correctly classify whether one identity implies another.
In our initial tests, frontier LLMs already do pretty well on this task. But we’re more interested in the performance of weak LLMs, which score closer to 50% accuracy. So the challenge is design a <= 10KB cheat sheet that improves the classification accuracy of weak open source LLMs as much as possible.
The sair foundation is generously hosting this competition and providing model credits (fairly limited in stage 1). We are currently planning the second stage. Please read Terry’s post below for his thoughts and more information on the resources available to participants.
SAIR@SAIRfoundation
Our co-founder Terence Tao is announcing SAIR Foundation's inaugural competition: the Mathematics Distillation Challenge. Co-organized by @damekdavis, Terence Tao, and SAIR Foundation. competition.sair.foundation/competitions/m…
English
