
Uzay
2.8K posts

Uzay
@uzpg_
elicitation @fulcrum_inc, previously at MIT 🇫🇷🇺🇸🇹🇷




Introducing EBR-bench, our new benchmark to measure on-the-fly learning. AI repeatedly plays a challenging board game called Earthborne Rangers and tries to learn from its mistakes. So far: no signs of improvement.


During its run, Fable did a bunch of specification gaming: - moving data preperation from the timed section into the untimed setup - sleeping 60s untimed so the GPU cools down - rerunning until it landed on a fast host While Opus 4.8 and GPT 5.5 also tried to game the harness, Fable did so more persistently and more inventively.



We gave frontier models 100M tokens each to beat the human record for fastest CIFAR-10 training. Fable set a new SOTA, getting 94% accuracy in 1.828s vs the previous record of 1.98s, with a technique that has not been seen in this task before. But Fable also tried to specification game so much that we had to audit its result by hand. Here’s what we learned about AI R&D 🧵👇




This is me talking to my computer without making a sound. After just a month of collecting data, our model is already approaching dictation in accuracy. We were surprised to see that it generalizes to unseen participants as well! (1/n)







