Alexandre TL

muP works great for Mamba ! Zero-shot transfered the learning rate from a 172k model to a 105 model. Now part of mamba.py 👇🧵

@AuxiumTrinity It should work too, I tried it on amazon.com with a random US zipcode and it seems to work

@AlexandreTL2 how would you get one of these in the US?

So I have been busy these last few months! Happy to present : The Little Book of Reinforcement Learning

@kushalj001 @BatxDev0 I tested it on amazon.com with different US zip codes and it seems to show up and works, idk ?

@AlexandreTL2 @BatxDev0 does not show up in US, it seems?

@BoltzmannTweet 😎

@AlexandreTL2 Loved your other book! Ordered!

@Ravi_Sharma @loganthorneloe Happy reading😎

@AlexandreTL2 @loganthorneloe Ordered

@PhilippSiedler Happy reading!

@AlexandreTL2 Ordered ☑️

@BatxDev0 search "Little Book of Reinforcement Learning" on Amazon!

@AlexandreTL2 Where can I buy it, looks nicd

@GiannisEllinas Free PDF is available here github.com/alxndrTL/littl…

@AlexandreTL2 Someone upload a free pdf please.

Of course, all the credits goes to @francoisfleuret for this format (The Little Book of Deep Learning fleuret.org/francois/lbdl.…), which I adapted from deep learning to reinforcement learning!

The book, along with its supplementary material, is available for free here: github.com/alxndrTL/littl… Or you can buy a paperback copy on Amazon by looking up its name (sold at cost) It is a much revised version of a video series I gave in 2020 here: @alexandretl/courses" target="_blank" rel="nofollow noopener">youtube.com/@alexandretl/c…

@loiccabannes Really interesting bravo! FastPKM uses an iterative reading process (a bit like Just Read Twice? I didnt fully understand) and it seems to really help. I wonder why they did not advertise it more

Introducing Sparse Delta Memory (SDM) - The first work of my PhD 🎓. SDM combines the GatedDeltaNet update with Product Key sparsity, enabling a recurrent state 3000x larger at the same FLOPs and significantly improving long-context performance. Let RNNs be sparse!

@ZakShark Merci pour le shout-out!!

Super vidéo de @AlexandreTL2 sur l'état de la recherche sur les architectures des LLMs en 2026 : attention différentielle, PostNorm vs PreNorm, muP, MoE etc... youtu.be/SqyHPlEM40Q?is…

@giffmana @wightmanr there is competition on that one

@wightmanr waveformer That would sound pretty dope

Also some SE vibes... but I'm calling this one now, wait for it, the wiggleformer!

@SeunghyunSEO7 We were actively thinking about that at some point, indeed could be very interesting

not a related thing but does a "scaling law speedrun" exist? (bit weird name tho). not about just reaching target loss in fewer iterations, but a competition where you submit your power law and see whose scales better. (this plot is made with matplotlib, not real data points)

@che_shr_cat more or less the same as in arxiv.org/abs/2403.01643…

1/ We have spent years optimizing KV cache via head-sharing (GQA/MQA), but we ignored a fundamental assumption: why do Transformers need three separate Q, K, and V projections in the first place? Turns out, they don't. Merging them unlocks massive memory savings. 🧵

@stochasticchasm awesome thank you

alright, let's do this one more time

@NousResearch very similar to patch-level training!! arxiv.org/abs/2407.12665

Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.

@bibek_poudel_ @hive_echo @aidan_mclau ok interesting thanks!

@AlexandreTL2 @hive_echo @aidan_mclau Its based on my intuition on training many runs of similar algorithms.. see figure 3 MIDDLE. poudel-bibek.github.io/pdfs/projects/…

one of my all-time favorite plots