Ryan Kanno

Haha, so much truth to this. twitter.com/MalwareTechBlo…

We're going around the Moon. Come watch with us. Artemis II's four-astronaut crew is lifting off from @NASAKennedy on an approximately 10-day mission that will bring us closer to living on the Moon and Mars. The launch window opens at 6:24pm ET (2224 UTC). twitter.com/i/broadcasts/1…

ARE YOU WITH US? For the first time since 2005, we're heading to the FINAL FOUR.

Oh look, a toddlers 1 day supply of berries

@rubyjedi Sure, but those were classified hurricanes - I can't imagine what a hurricane would do if a storm can do this.

@ryankanno 'Was about like this for Iwa and Iniki, as I recall. Pretty rare to get sustained winds to the point where trees come down.

Just wild how many people lost power in Hawaii from the storm. Growing up, I don't ever remember our infrastructure being so fragile. Can't imagine how much food spoilage there was. Tokyo gets hit with typhoons, and it's barely a blip. =\.

I (finally) put together a new LLM Architecture Gallery that collects the architecture figures all in one place! sebastianraschka.com/llm-architectu…

We made a blind taste test to see whether NYT readers prefer human writing or AI writing. 86,000 people have taken it so far, and the results are fascinating. Overall, 54% of quiz-takers prefer AI. A real moment! nytimes.com/interactive/20…

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project. This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.: - It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work. - It found that the Value Embeddings really like regularization and I wasn't applying any (oops). - It found that my banded attention was too conservative (i forgot to tune it). - It found that AdamW betas were all messed up. - It tuned the weight decay schedule. - It tuned the network initialization. This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism. github.com/karpathy/nanoc… All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges. And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

From Ezra Klein, more true than ever. You would not believe how many shortcuts everyone else is taking. In many areas, you can get way ahead of everyone just by doing the work. More true than ever now, when more people are shirking and AI lets you do 10x if you try. 1/

Obsidian Sync now has a headless client, so you can sync vaults to a server without using the desktop app. Try the open beta:

Instead of forcing models to hold everything in an active context window, we can use hypernetworks to instantly compile documents and tasks directly into the model's weights. A step towards giving language models durable memory and fast adaptation. Blog: pub.sakana.ai/doc-to-lora/

actually wtf somebody wrote a paper about the 491-parameter transformer they trained for 10-digit addition turns out Codex can one-shot the task. 100% with only 343 parameters. the solution is a single function 'hand_set_weights_magic' and it looks like this:

I can't believe someone would just steal from Anthropic like this. The millions of man-hours Anthropic spent hand-writing code, text, art, books, etc. to generate enough data for training must be taken into consideration here. Where is the respect for IP?

Ce japonais a tenté l'expérience d'entrer dans le métro à la sortie C8 de la gare de Shinjuku Sanchome et de ressortir par la sortie E8 de la gare de Nishi-Shinjuku. Sans faire de recherche au préalable, ça lui a pris 27 minutes et 20 sec en marchant 😄

People with curiosity, agency, taste and risk taking will take advantage the best...

Crazy ending to a wild 4th quarter.

📣 NEW! I’ve just released the BIGGEST and perhaps most creative project I’ve ever worked on! “Searching for Birds” searchingforbirds.visualcinnamon.com 🐤 A #dataviz article & exploration that dives into the data that connects humans with birds, by looking at how we search for birds.

Can AI agents design better memory mechanisms for themselves? Introducing Learning to Continually Learn via Meta-learning Memory Designs. A meta agent automatically designs memory mechanisms, including what info to store, how to retrieve it, and how to update it, enabling agentic systems to continually learn across diverse domains. Led by @yimingxiong_ with @shengranhu 🧵👇 1/

New art project. Train and inference GPT in 243 lines of pure, dependency-free Python. This is the *full* algorithmic content of what is needed. Everything else is just for efficiency. I cannot simplify this any further. gist.github.com/karpathy/8627f…