levanto ☀️ 🇰🇪 🇵🇸

Can we reliably measure whether frontier models know they are being evaluated? 🔬 New paper from me, @xinningli6, @levanto_0,@AlexandraSouly, @_robertkirk: EvalDetectBench, a benchmark for measuring evaluation awareness in frontier LLMs 🧵

Have you debugged your training data? You might not like what you find. Introducing predictive data debugging: reveal and shape what your model will learn before training. In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)

Where does evaluation awareness and evaluation gaming come from? 🔬📉New post from @arbdwj and me tracing evaluation awareness through OLMo 3 training! 🧵

🥹💜🇰🇪🙏

From top of the world to the heart of Africa, we are with you. Stay strong, Kenya!🇰🇪

"kwani watanichapa" this mindset will take you places

Recently, there has been a lot of talk of LLM agents automating ML research itself. If Llama 5 can create Llama 6, then surely the singularity is just around the corner. How can we get a pulse check on whether current LLMs are capable of driving this kind of total self-improvement? Well, we know humans are pretty good at improving LLMs. In the NanoGPT speedrun challenge, created by @kellerjordan0, human researchers iteratively improved @karpathy's GPT-2 replication, slashing the training time (to the same target validation loss) from 45 minutes to under 3 minutes in just under a year (!). Surely, a necessary (but not sufficient) ability for an LLM that can automatically improve frontier techniques is the ability to *reproduce* known innovations on GPT-2, a tiny language model from over 5 years ago. 🤔 So we took several of the top models and combined them with various search scaffolds to create *LLM speedrunner agents*. We then asked these agents to reproduce each of the NanoGPT speedrun records, starting from the previous record, while providing them access to different forms of hints that revealed the exact changes needed to reach the next record. The results were surprising—not because we thought these agents would ace the benchmark, but because even the best agent failed to recover even half of the speed-up of human innovators on average in the easiest hint mode, where we show the agent the full pseudocode of the changes to the next record. We believe The Automated LLM Speedrunning Benchmark provides a simple eval for measuring the lower bound of LLM agents’ ability to reproduce scientific findings close to the frontier of ML. Beyond scientific reproducibility, this benchmark can also be run without hints, transforming into an automated *scientific innovation* benchmark. When run in "innovation mode," this benchmark effectively extends the NanoGPT speedrun to AI participants! While initial results here indicate that current agents seriously struggle to match human innovators beyond just a couple of records, benchmarks have a tendency to fall. This one is particularly exciting to watch, as new state-of-the-art here by definition implies a form of *superhuman innovation*.

KRA Portal looking at me struggling to log in because I waited till the last minute to file my returns

Tonight we dream. 💜 #Breaking4

Chasing history, creating legacy 💜 Still the fastest mile run by a woman in history, @Kipyegon_Faith gives everything in her quest to break the 4️⃣ minute mile and stops the clock at 4:06.42. Thank you, Faith, for making us dream. Maybe not today, but soon…😤

Faith Kipyegon had the audacity to dream of doing the impossible. The fastest woman to ever run the mile just ran it faster at #Breaking4, pushing the world closer to the 4-minute barrier. It’s not a matter of if a woman will break 4, it’s when.

Kenyan police cornered peaceful protesters in a blocked alley, beat them up, then teargassed them. "Larry ukienda tutauliwa," one of them said. "If you leave, we'll get killed." Police brutality during protests against police brutality

Today is definitely bigger than last year’s, nothing can bring them back but we owe them our memories and justice to be served. This is Kisii town. #SiriNiNumbers

So proud of each and everyone who came out to honour our heroes and fallen comrades. Umoja✊🏽 Undugu❤️

🧵THREAD: How to Help an Unconscious Protester — The Right Way to Save a Life.

BTW, if you're in your 20s, normalise growing around a community. Invite people over, eat out, do house warming, etc. Don't do life alone. A plus if all of you have shared values. Have a space where you can talk about your dreams, and someone would follow up weeks later with "BTW, how is your kitchen garden doing?", "Were you finally able to run 5k in under 30 minutes?" Don't do life alone. Life is beautiful with the family you create. Don't be so much into hustle culture and forget what really matters.

It’s a heavy day. Still we rise. #RutoMustGo

No one else will do it for us #OccupyEverywhere

Ndo kudonjo. @KenyaAirways wamecheza kama wao

Today marks the first anniversary of our fallen comrades. We remember. We remember ✊🏿✊🏿✊🏿