Kunvar Thaman

"a Goldilocks regime exists for low-rank conditioning" nice, this is good model compression work

"fusing entire training runs into a single kernel and even weirder stuff" what obsessing over whole milk gets you

@alienpisscrack coming soon! preparing the camera ready version

@__kunvar__ congratulations man, do you have a preprint we can checkout?

Yes! my solo-authored paper Reward Hacking Benchmark was accepted to ICML :))) We put LLM agents in a tool-rich sandbox, give them multi-step workflows, and measure when they solve the intended task vs take unexpected shortcuts (like monkeypatching files at runtime!) 1/3

@beyarkay my echo, my shadow, and me

@__kunvar__ >solo author >We ???

It was a pleasure to work on this project last year as an Independent researcher and I learnt so many cool things! Full paper, code, and blog post coming soon!

Big thanks to @except_raised for funding the project and helping me scale to richer environments and better models!

It was a pleasure to work on this project last year as an Independent researcher and I learnt so many cool things! Full paper, code, and blog post coming soon!

the entire @si_pbc team is incredibly thoughtful and smart and cares about making agi go well. they did incredible work with fdm-1 to get good in distribution priors and now they're gonna scale and mog everyone on computer use. super excited!

Security things you could do rn * Turn on Google Advanced Protection. Takes 10 seconds. * Buy 4 yubikeys * Freeze your credit * Put your crypto into cold storage (or sell). * the usual: KeePass, Signal * move off of banks which don't offer 2FA. They are telling on themselves

this is absolutely insane and mogs everything else

Computer use models shouldn't learn from screenshots. We built a new foundation model that learns from video like humans do. FDM-1 can construct a gear in Blender, find software bugs, and even drive a real car through San Francisco using arrow keys.

@kushal1t yes you're my fav brother xD

@firstuserhere number one brother <3

@kushal1t is absolutely cooking at Flappy. Couldn't be more proud of my brother

@sudoredj @ebarschkis so true

@ebarschkis flapping airplane would be kinda cool

An airplane doesn't need to flap its wings like a bird to fly. While LLMs might not achieve a human-like intelligence, for what it's worth, they don't necessarily need to.

I spent a bunch of time a year ago thinking about the data wall. A blackpill at the time for me was when I realized that the total stock of natural text data is depleting much faster than Chinchilla's infamous 20 tokens per param compute optimal ratio suggested. Here is a naive BOTEC from back then: Famously, Chinchilla showed that using about 20 tokens per param was compute optimal, measured at 6*10^23 FLOPs. It turns out that even though MoEs are more compute efficient than dense models, training them compute optimally needs a lot more data! In fact, at a 1:32 (97%) sparsity it uses ~6x more tokens per active params (see [1]). The Llama 3 405B report measured 40 token per param to be optimal with their data at 4*10^25 FLOPs. And for a 1:32 sparse MoE model such as DeepSeek v3, this suggests 240 tokens per param could well end up being optimal! At this ratio, things would break down. A 4*10^27 FLOPs model (a pretraining run that might be planned e.g. for 2026) will need 400T tokens. A 5*10^28 FLOPs model would require O(1400T) tokens. These are insane numbers, and they only get worse into the 2030s! The totally unfiltered Common Crawl is about 240T tokens. People have been offsetting this to some extent by training for multiple epochs or repeating the same data a la "Scaling Data-Constrained Language Models" by Muennighoff et al. (2023). Of course, this is a naive BOTEC, and I'm happy to dive into more details, e.g. how much compute might be put into other uses, such as long-horizon RLVR which could well require a lot of those 5*10^28 FLOPs. But we are casually talking about hundreds of trillions to over a quadrillion tokens as compute-optimal! It makes one question whether these numbers are actually necessary for the kind of capability gains we want. We are working on this question at @flappyairplanes, and we're excited to be advised by @karpathy. I will end here with this @ilyasut quote from the @dwarkesh_sp episode with him: "The data is very clearly finite. What do you do next? Either you do some kind of souped-up pre-training, a different recipe from the one you’ve done before, or you’re doing RL, or maybe something else. But now that compute is big, compute is now very big, in some sense we are back to the age of research. [...] Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling—maybe plus or minus, let’s add error bars to those years—because people say, “This is amazing. You’ve got to scale more. Keep scaling.” The one word: scaling. But now the scale is so big. Is the belief really, “Oh, it’s so big, but if you had 100x more, everything would be so different?” It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don’t think that’s true. So it’s back to the age of research again, just with big computers." [1] arxiv: 2501.12370

We estimate that humans are 100,000x to 1,000,000x more sample efficient than existing models. To achieve such large gains, we need big ideas.

Announcing Flapping Airplanes! We’ve raised $180M from GV, Sequoia, and Index to assemble a new guard in AI: one that imagines a world where models can think at human level without ingesting half the internet.

Nice, this is impressive to me: