Fabrizio Milo

Created with @NotebookLM after discussing these topics with @Cyndesama @tensorqt @Niccolg92 and few others

If you are running parameter golf by OpenAI, here are some stats that might be useful: gist.github.com/Mistobaan/f3fe…

@karpathy Spoilers!

Had to go see Project Hail Mary right away (it's based on the book of Andy Weir, of also The Martian fame). Both very pleased and relieved to say that 1) the movie sticks very close to the book in both content and tone and 2) is really well executed. The book is one of my favorites when it comes to alien portrayals because a lot of thought was clearly given to the scientific details of an alternate biochemistry, evolutionary history, sensorium, psychology, language, tech tree, etc. It's different enough that it is highly creative and plausible, but also similar enough that you get a compelling story and one of the best bromances in fiction. Not to mention the other (single-cellular) aliens. I can count fictional portrayals of aliens of this depth on one hand. A lot of these aspects are briefly featured - if you read the book you'll spot them but if you haven't, the movie can't spend the time to do them justice. I'll say that the movie inches a little too much into the superhero movie tropes with the pacing, the quips, the Bathos and such for my taste, and we get a little bit less the grand of Interstellar and a little bit less of the science of The Martian, but I think it's ok considering the tone of the original content. And it does really well where it counts - on Rocky and the bromance. Thank you to the film crew for the gem!

Do you guys still use google? I feel I haven't done a search in ages. I just deepresearch everything.

My dream job would be to research more efficient architectures for AI. Let’s see if it becomes reality

end of day used to be stack overflow down now is this

Gonna read this right away. I do think we need sequential processing of the input and use the N^2 attention on a much smaller subset of the space. I love thinking about deep learning architectures.

@hamostaf04 @DennwsLee yes, I saw the same pattern in my own harness. The key is to have a "tree" of small atomic changes and a trace of failure / success in a short form. I applied to kernel generation with benchmarks in my case which is verifiable

my friend @DennwsLee and i spent the past week tinkering with autoresearch we gave 4 AI agents a research loop and told them to never stop 48 hours later: 550+ experiments, zero babysitting. One agent hit 93% on competition math from pure reward signal. another proved SFT beats RL at half the cost. highlights in 🧵

Finally managed to have a native macosx mujoco client for whole body simulation. The idea is to have a minimal environment running locally that can communicate to a remote GPU machine over ZMQ to run the actual policy.

Actually won something with my kernel harness !

good harnesses is all you need

Today is @PyTorch hackathon in San Francisco with the @GPU_MODE folks. Will my custom helion autoresearch harness be able to produce better kernels and win the competion ?

@ellen_in_sf What model is running the experiments claude opus? is this still using the mlx version as target?

6 hours later 127 experiments ~ 34% improvements on my experiment running autoresearch with no human-in-the-loop + 27% keep rate. I see an issue: the val_bpb tends to plateau with no early stopping mechanism, token burns. Will update with a workaround

I just generated this image from my imagination about the future of AI. And by looking at it, made me realize that one of the drawbacks of current LLMs is their linearity dictated by the back-propagation training algorithm. LLMs are cool but definitely not the final solution.

@karpathy Very inspiring as always! We are also open sourcing part of our infra on automated research for Gemini to evolve itself at github.com/google-deepmin… More complex than the nanochat setup but closer to SOTA LLM pre/post-training while staying as minimal as possible. More on the way.

@sirbayes @topwasu @ellisk_kellis @WLehrach What is the signal to symbol problem? A symbol is a grouping of common similar signals ?

I agree that for agents (as opposed to making media slop for human consumption), symbolic world models that abstract away from pixels is key! I have a little paper on this (arxiv.org/abs/2602.02799) with @topwasu , @ellisk_kellis and @WLehrach. But we took a shortcut and assumed symbolic input (from OC Atari), so we could focus on learning temporally abstract WMs using code synthesis. Thus our approach didn't solve the signal-to-symbol problem, and relies heavily on pretrained LLMs, which feels a bit like cheating. In general, once you leave the happy land of symbolic data (code, language, math), modeling becomes much harder, since you need to decide what loss function to use for training. If you can't afford to model every little detail, you can't use maximum likelihood (or some simple bound like ELBO), so you need some other way to decide how to "color the bits" (as @alemi likes to say).

I am delighted to see a new version of the book by @_sdbuchanan, @druv_pai , @pengwang2003 and @YiMaTweets . This is the best book on the foundations of deep representation learning! In this era of coding agents, the math is all you need to learn :) ma-lab-berkeley.github.io/deep-represent…

@tri_dao big win, compiling flash attention was dreadful

I’m unreasonably excited about the fact that we wrote everything in Cute-DSL, embedded in Python. Installing / “compiling” now takes seconds instead of minutes / hours (looking at you, C++ templates). Try pip install fa4!

The FA4 paper is finally out after a year of work. On Blackwell GPUs, attention now goes about as fast as matmul even though the bottlenecks are so different! Tensor cores are now crazy fast that attn fwd is bottlenecked by exponential, and attn bwd is bottlenecked by shared memory bandwidth.  Some fun stuff in the redesigned algorithm to overcome these bottlenecks: exponential emulation with polynomials, new online softmax to avoid 90% of softmax rescaling, 2CTA MMA instructions that allow two thread blocks to share operands to reduce smem traffic.

I think every evaluation paper should report the cost of running a full evaluation.

What every company with instructions on how to use their products that wants to be agent-friendly should implement: “Copy page as Markdown”.

@ZainHasan6 @julien_c Is there a complexity metric for the traces? for curriculum learning.

We just open sourced a dataset that cost us $130k to generate! It's 6.7B tokens of agentic coding traces of 51k tasks across 1.6k unique repos. You can SFT on it to make your models better coding agents!