Clémentine Fourrier 🍊 is off till Dec 2026 hiking

Hey twitter! I'm releasing the LLM Evaluation Guidebook v2! Updated, nicer to read, interactive graphics, etc! huggingface.co/spaces/OpenEva… After this, I'm off: I'm taking a sabbatical to go hike with my dogs :D (back @huggingface in Dec *2026*) See you all next year!

👀Introducing a brand new @yupp_ai SVG leaderboard ranking frontier models on the generation of coherent and visually appealing SVGs! Gemini 3 Pro by @GoogleDeepMind takes the crown as the most powerful model! 👏 We’re also releasing a public SVG dataset. Details in🧵

Either you crack general intelligence -- the ability to efficiently acquire arbitrary skills on your own -- or you don't have AGI. A big pile of task-specific skills memorized from handcrafted/generated environments isn't AGI, not matter how big.

@swyx @huggingface The "time" subset of GAIA2, very tricky for models atm - and we'll probably see very, very good progress on games based benchmarks (with stronger game based benchs coming out) But my biggest prediction is that the main trend of 2026 will be ML for science ^^ (so mb benchs there)

@clefourrier @huggingface what is your bet on what benchmarks will be “solved” by the time you get back? fun prediction exercise

Hey twitter! I'm releasing the LLM Evaluation Guidebook v2! Updated, nicer to read, interactive graphics, etc! huggingface.co/spaces/OpenEva… After this, I'm off: I'm taking a sabbatical to go hike with my dogs :D (back @huggingface in Dec *2026*) See you all next year!

@swyx @huggingface The hike in itself is probably going to take 6 months but I need to prepare physically before (and do some random stuff, like paragliding and seeing friends ^^)

@clefourrier @huggingface thats an ultra long hike!!! congrats

@Teknium @huggingface Thanks a lot! You can always follow @nathanhabib1011 , he's the one taking over all our cool eval stuff (lots of very, very neat things coming up btw)

@clefourrier @huggingface I’ll miss your work in the meantime!

@codewithimanshu @huggingface Probably! It will be fun to see then :)

@clefourrier @huggingface Cheers to your sabbatical, Clementine! That's a long break, but I wonder if the tech will change much by then, eh?

@mziizm @huggingface Thanks a lot! :)

@clefourrier @huggingface Enjoy the break!

@maximelabonne Thanks a lot :)

You don't understand evals Everybody in AI should read this 👇

@altryne @nisten no problem! my pic is def gender neutral ^^"

@nisten @clefourrier ooops, good to know!

mf just dropped another book banger

@giffmana @huggingface Thanks a lot, I'm super excited about it :D (so much to do, so little time ^^)

@clefourrier @huggingface Enjoy the long break, sounds awesome!

@TheOkosisi @huggingface Feel free to cite it as you need! :)

This is so good @clefourrier . Can i seek your blessings to use this in my abstract research work and reference it continuously in my soon to be broadcasted LLM thoughts newsletter. ( Will ensure all due credits and mentions are pointed towards yourself and your incredible work). This work alone can lead the proper sensitisation of beginners and professionals alike on the nuances of LLM EVALS. This is structure!

@latkins Hahahahahaha 💯 my mood when writing the tweet XD

@nisten @altryne I hope you'll be in a year ^^ (she or they btw XD)

@altryne yeah we’ll not be done reading it by the times he’s back

@xeophon @huggingface Thanks a lot :)

@clefourrier @huggingface Holy shit!!! Time to check it out 👀 Have fun with your dogs!!

@huggingface cc @maximelabonne since you wanted an update :P

If you see improvements, I'd love to hear them (within the next 2 days) :) Many thanks to @thibaudfrere for his help on the banner and @gui_penedo for his proofreading! If you've got eval needs, your new PoC is @nathanhabib1011 (with a focus on lighteval)!

The guide is very beginner friendly, as we go from the basics of tokenization/inference to the nits and tricks of running eval properly, so it's compatible with all levels. Should contain most of what we wrote about evals at HF in a single unified place, with updates ofc :)

as a researcher, it makes no sense to compare reasoning vs non reasoning models on benches like the ones in Artificial Analysis without normalizing somehow by cost or output tokens. non reasoning models (base/instruct) are important for the open ecosystem since research teams and companies will use them to do RL or other things (like synthetic generation) for specific verticals (think cursor/windsurf) as a user, i get that you don’t care whether the model is reasoning or not, you judge speed, cost, and accuracy (and memory if you want to deploy your model locally) the only advantage of non reasoning models would be speed/cost because they generate fewer tokens BUT speed and cost also depend on other thing like infra -> for speed see how fast some models get on groq or cerebras -> for cost model like deepseek are so cheap that there is very few use case where you'd want to use non reasoning model anyway