Grégoire Delétang

Claude code is actually an insane tool, I can only say congrats to the team

512 parameters: a new top scorer for 10-digit addition with transformers! Who can beat it?

@DimitrisPapail Cool stuff! I think the challenge would be even more interesting with a scaling law num parameters versus number of digits (at iso ~100% accuracy)

Genie 3 🤝 @Waymo The Waymo World Model generates photorealistic, interactive environments to train autonomous vehicles. This helps the cars navigate rare, unpredictable events before encountering them in reality. 🧵

I find that LLMs went much further than I initially thought 3-4 years ago. My impression is that pretraining enables many non trivial things to work, that would never otherwise (test-time "thinking", rlhf, fine-tuning on non trivial targets, interaction with tools etc). Am I the only one to be positively surprised?

@jxmnop Language modelling is compression. Lossy (weights) and lossless (sequence).

this post is complete misinformation LLMs are lossy compressors! of *training data*. LLMs losslessly compress *prompts*, internally. that’s what this paper shows. source: i am the author of “Language Model Inversion”, the original paper on this

@zhaisf Back to 1995... How many people still discover NNs for the first time?

A good way of understanding the inductive bias of neural nets is to train an MLP to regress to the sin(x) function. Below is training on 10K points in [-20, 20], predicting over [-30, 30] after 10 and 100 epochs of training. The implications shown here are surprisingly general.

If you pass a sequence of constant tokens to a transformer with rotary encodings, it will return a constant output distribution, because the attention values are constant. Idk if someone spotted this "bug" before?

We slowly realize that humans do not, in fact, "generalize"

Interesting effect, I think I saw you mentioned it in one of your tweet indeed. Yes my point is that ultimately even with cot, the model must be pretrained on a sufficiently large context length L, that contains at most L*log2(vocab_size) bits. That's enough to solve most problems, but that's still a hard limit.

this effect is especially magnified when doing self improvmenet on top of pretrained models. Why would pretraining help? Because for a model to "learn an algorithm" it has to be ancored on an idea of a number system, having seen other examples of deduction, algorithms etc. This is impossible to have without pretraining.

one of the few times in my career I feel good about calling a problem "solved", for an approximately satisfying value of "solved" length generalization challenges can be overcome with iterative self improvement

@DimitrisPapail Your work is still interesting for sure, but what I mean is that it doesn't solve the fundamental problem of the transformer architecture, which is that you must train it with a large enough context window.

@DimitrisPapail I mean you could train with a longer context window yes, with even just some fine-tuning. But testing with a longer context window experimentally doesn't work, it's what people have been struggling with for a long time (i.e. length generalization).

@_akhaliq @mhutter42 haven't checked it but mmh it smells a lot like Solomonoff's reference UTM

Google Deepmind presents Agency Is Frame-Dependent

@nouhadziri @DimitrisPapail It also makes me think about universality: is it really the right concept? Couldn't we get, say, a proof of the RH in a few years, with a model that can't sum and add two numbers?

@nouhadziri @DimitrisPapail that comes back regularly. But when will it finally be solved?

📢 DeepSeek R1 still cannot solve multiplication with 100% accuracy🫠😬 Though it can achieve high scores on hard math questions (AIME, MATH-500), extremely difficult physics, biology, and chemistry problems (GPQA Diamond), and coding challenges (LiveCode, CodeForces)-problems that require advanced problem-solving skills, it struggles with a simple multiplication algorithm [1/8].

GPU hours is the new bitcoin

Here you know you should pull up the logit of the class you proposed when you get +1 and push it down with -1, but that's it. You have to "poke around" until you get a +1. So in the classification setup you get ~log_2(N) bits of information, and in the RL case 1 bit. 4/4

Does anyone know/can guess how they made fp8 training actually work? That's probably the most impressive to me. This would mean sparse training with gradients is indeed possible.