Daniel Whettam

New paper on a long-shot I've been obsessed with for a year: How much are AI reasoning gains confounded by expanding the training corpus 10000x? How much LLM performance is down to "local" generalisation (pattern-matching to hard-to-detect semantically equivalent training data)?

Our new ICML 2025 oral paper proposes a new unified theory of both Double Descent and Grokking, revealing that both of these deep learning phenomena can be understood as being caused by prime numbers in the network parameters 🤯🤯 🧵[1/8]

After a year of review, our paper is now out in @NatureComms ! I really think our model/theory offers one of the most promising frameworks yet for learning in neocortical circuits - perhaps not in its final form, but the core principle of ssl feels right. Hoping experimentalists will test it, break it, and computationalists will refine it #AllModelsAreWrongButSomeAreUseful - I think this one could be very useful😀

Good post from @balajis on the "verification gap". You could see it as there being two modes in creation. Borrowing GAN terminology: 1) generation and 2) discrimination. e.g. painting - you make a brush stroke (1) and then you look for a while to see if you improved the painting (2). these two stages are interspersed in pretty much all creative work. Second point. Discrimination can be computationally very hard. - images are by far the easiest. e.g. image generator teams can create giant grids of results to decide if one image is better than the other. thank you to the giant GPU in your brain built for processing images very fast. - text is much harder. it is skimmable, but you have to read, it is semantic, discrete and precise so you also have to reason (esp in e.g. code). - audio is maybe even harder still imo, because it force a time axis so it's not even skimmable. you're forced to spend serial compute and can't parallelize it at all. You could say that in coding LLMs have collapsed (1) to ~instant, but have done very little to address (2). A person still has to stare at the results and discriminate if they are good. This is my major criticism of LLM coding in that they casually spit out *way* too much code per query at arbitrary complexity, pretending there is no stage 2. Getting that much code is bad and scary. Instead, the LLM has to actively work with you to break down problems into little incremental steps, each more easily verifiable. It has to anticipate the computational work of (2) and reduce it as much as possible. It has to really care. This leads me to probably the biggest misunderstanding non-coders have about coding. They think that coding is about writing the code (1). It's not. It's about staring at the code (2). Loading it all into your working memory. Pacing back and forth. Thinking through all the edge cases. If you catch me at a random point while I'm "programming", I'm probably just staring at the screen and, if interrupted, really mad because it is so computationally strenuous. If we only get much faster 1, but we don't also reduce 2 (which is most of the time!), then clearly the overall speed of coding won't improve (see Amdahl's law).

@ghostbestie057 @CollinRugg This was after the official races. He just ran down for fun at the end. Cheese wasn't even on the table

@CollinRugg So he did all that and he lost?

@jxmnop Similar idea to The Platonic Representation Hypothesis? arxiv.org/abs/2405.07987

excited to finally share on arxiv what we've known for a while now: All Embedding Models Learn The Same Thing embeddings from different models are SO similar that we can map between them based on structure alone. without *any* paired data feels like magic, but it's real:🧵

@jxmnop Why do you say it's not self-supervised? There are many SSL tasks where the self-supervision is the task of interest. Language modelling is one of those IMO. Self supervision is supervised training where the supervision comes from the data (hence "self")

language modeling is not unsupervised learning. it is not (and please stop saying this) self-supervised learning. next-token prediction is textbook _supervised_ learning, and i will die on this hill

@SBRLabs Congrats!

PhD Viva == complete! It was very intense and stressful, and I'm not in a hurry to repeat the experience :P Result is pass with corrections, but precisely what that means exactly hasn't been defined yet. Thanks to everyone who has supported me so far :-) #PhD #Viva #INeedANap

@CianEastwood @valence_ai @RecursionPharma Congrats!

I've completed my PhD and joined @valence_ai (part of @recursionpharma) as a Senior Research Scientist! Excited to study generalization and representation learning (causal / multimodal) in the context of drug discovery!

New paper: a big 90-page intro to AI and its likely effects from ten perspectives, ten camps. The whole gamut: ML, scientific applications, social applications, access, safety and alignment, economics, AI ethics, governance, and classical philosophy of life. 1/18

(1) Good news! I've had a paper accepted (with @cian_neuro, @nathanlepora, and Matt W. Jones), and I'll be giving a talk on it at @AAMASconf this year 🥳🥳🥳🧠🤖🐀🥳🥳🥳

@bristol_filming @josmith1975x Yes, that must be it! I believe I saw Naomi Scott being filmed outside Clifton Arcade.

@DWhettam @josmith1975x There is a new film with Kit Harington, Naomi Scott, Jeremy Irons called "Eternal Return" that was announced as shooting in Bristol in November 2023. I am guessing it was that, especially as Eternals 2 is still in development. variety.com/2023/film/news…

Unknown production filming around Clifton Village - Thursday, 30th November, 2023. Via @josmith1975x - #crewloc #bristol #filming

@iamJoeWhettam Big first tweet

New ideas are always the fruit of old ideas reimagined.

@vertinski This is the lottery ticket hypothesis: arxiv.org/abs/1803.03635

sooo, looking at LK-99, could we encounter a freak neural net which by chance has trained super-effectively?..... like a 3B model that is on par with GPT-4 🤔

@wafajohal @TheOfficialACM @ieeeras @ICCVConference has a policy on not using chatgpt for conference reviews

It's happening! We received our first rejection review written by ChatGPT for a conference paper we submitted earlier this year. I am now looking for policy statements from @TheOfficialACM or @ieeeras about the use of LLMs to write reviews. Any pointers?

@gleech Most martial arts

which artform has more practitioners than spectators?

Very excited the EPIC-SOUNDS dataset is finally released! We’ve all worked incredibly hard on this and I can be proud to say that this is my first publication! Looking forward to what this dataset can bring to the deep learning community!

📢 Now Open For Submissions - all EPIC-KITCHENS Leaderboards for @CVPR #CVPR2023 Challenges. Winners announced at Joint @ego4_d and EPIC workshop: sites.google.com/view/ego4d-epi… **Nine** open challenges inc. 4 new ones (see 🧵) Leaderboards close 1st of June 2023. 🧵 1/7

For anyone interested in audio-visual learning, this challenge will be of interest! EPIC-100 and EPIC-SOUNDS is a step towards multi-modal methods where one modality does not rely on the timestamps or label set of another! We look forward to seeing what this challenge brings!