Andrej Karpathy

The hottest new programming language is English

@shikhr_ Yeah I have 4 blog posts that I didn’t finish yet this is one of them. Dobby runs my entire house over WhatsApp. Lights, shades, pool/spa, sonos, security HVAC etc

@karpathy Dobby the House Elf claw??? What did I miss!

Thank you Jensen and NVIDIA! She’s a real beauty! I was told I’d be getting a secret gift, with a hint that it requires 20 amps. (So I knew it had to be good). She’ll make for a beautiful, spacious home for my Dobby the House Elf claw, among lots of other tinkering, thank you!!

Ugh X breaks time links, it’s at 26:17

(link to blast from the past) youtu.be/xQhb3C2hQoE?si…

@Yulun_Du @ilyasut SGD is a ResNet too (the blocks of it are fwd+bwd), the residual stream is the weights so... 🤔 We're not taking the Attention is All You Need part literally enough? :D

@ilyasut once said that an LSTM is a ResNet rotated 90 degrees. :) It turns out attention can be rotated 90 degrees too — yielding a natural generalization of residual connections. 🥳

@ChristosTzamos Wait this is so awesome!! Both 1) the C compiler to LLM weights and 2) the logarithmic complexity hard-max attention and its potential generalizations. Inspiring!

1/4 LLMs solve research grade math problems but struggle with basic calculations. We bridge this gap by turning them to computers. We built a computer INSIDE a transformer that can run programs for millions of steps in seconds solving even the hardest Sudokus with 100% accuracy

@rasbt @teortaxesTex Ty, I used your blog post, exported with obsidian ext into markdown, used it to enqueue ideas into my autoresearch loop

@karpathy @teortaxesTex I can offer you the metadata in YAML format: github.com/rasbt/llm-arch… Happy tinkering!

I (finally) put together a new LLM Architecture Gallery that collects the architecture figures all in one place! sebastianraschka.com/llm-architectu…

@_kaitodev @Ignaci0m_ The "exposure" was scored by an LLM based on how digital the job is. This has no baring on what actually happens to these occupations, which has to do with demand elasticity and a lot more. People are sensationalizing the visualization tool and putting words in my mouth.

thank you 🙏 the people’s reaction was overblown and completely justified at the same time honestly the data itself is still the most interesting part to me. even if the scoring methodology is somehow rough, the pattern it reveals ( that exposure tracks almost perfectly with “can you do this from a laptop” ) is a real insight worth checking out ppl weren’t panicking about the data but more about what the data actually confirmed.

5 minutes ago, @karpathy just dropped karpathy/jobs! he scraped every job in the US economy (342 occupations from BLS), scored each one's AI exposure 0-10 using an LLM, and visualized it as a treemap. if your whole job happens on a screen you're cooked. average score across all jobs is 5.3/10. software devs: 8-9. roofers: 0-1. medical transcriptionists: 10/10 💀 karpathy.ai/jobs

This was a saturday morning 2 hour vibe coded project inspired by a book I’m reading. I thought the code/data might be helpful to others to explore the BLS dataset visually, or color it in different ways or with different prompts or add their own visualizations. It’s been wildly misinterpreted (which I should have anticipated even despite the readme docs) so I took it down.

@_kaitodev @karpathy I can't find it

@Zhikai273 Wow. I was sure this was AI. (I mean generative AI.)

🎾Introducing LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data Dynamic movements, agile whole-body coordination, and rapid reactions. A step toward athletic humanoid sports skills. Project: zzk273.github.io/LATENT/ Code: github.com/GalaxyGeneralR…

@vivek_2332 Yep, exactly and agree! Any process with a lot of knobs and objective criteria benefits a lot.

introducing autoresearch-rl, autonomous research for rl post-training. inspired by @karpathy autoresearch, and i think rl post-training is honestly one of the places where this idea fits perfectly. there are at least 50+ hyper parameters to tweak, learning rate, batch size, rollouts, clipping ratios, kl penalties, schedulers, the list goes on. instead of sitting there for hours turning knobs one at a time, just let the model figure out the right starting config on its own. some things worth mentioning: -> built on @PrimeIntellect prime-rl (my favourite rl post-training framework) and @willccbb verifiers for reward verification. -> ran qwen2.5-0.5b-instruct on gsm8k across 60+ autonomous experiments. eval score went from 0.475 to 0.550 and the agent actually found a way to do it in fewer steps (20 instead of 30). less compute, better results -> the whole thing was surprisingly smooth to set up and run. point the agent at the config, go to sleep, wake up to a full experiment log. i really wish i could try this on a bigger model but gpu poor for now lol -> the agent discovers things you wouldn't think to try. like how rollouts = 4 beats rollouts = 8, or how a constant lr schedule outperforms cosine. it just methodically tests everything i think the real value here is that rl training is so fragile and noisy that having an agent patiently run experiment after experiment is genuinely more effective than a human doing it manually. check it out: github.com/vivekvkashyap/…

My autoresearch labs got wiped out in the oauth outage. Have to think through failovers. Intelligence brownouts will be interesting - the planet losing IQ points when frontier AI stutters.

Human orgs are not legible, the CEO can’t see/feel/zoom in on any activity in their company, with real time stats etc. I have no doubt that it will be possible to control orgs on mobile, with voice etc., but with this level of legibility will that be optimal? Not in principle and asymptotically but in practice and for at least the next round of play.

All of these patterns as an example are just matters of “org code”. The IDE helps you build, run, manage them. You can’t fork classical orgs (eg Microsoft) but you’ll be able to fork agentic orgs.

Expectation: the age of the IDE is over Reality: we’re going to need a bigger IDE (imo). It just looks very different because humans now move upwards and program at a higher level - the basic unit of interest is not one file but one agent. It’s still programming.

@amit05prakash I wanted to buy a bigger monitor and discovered that others had the same idea

@karpathy I can finally justify my reasons for buying more monitors now

@trongthangpham @maxbittker ralph loop runs headless. i dislike headless sessions. i need to see and supervise agent work, possibly ask /btw questions of them, possibly pitch in ideas to the mix, etc etc.

@karpathy @maxbittker I thought your version is similar to the ralph loop (the bash one) so it would loop forever. Is that not the case here?

From @karpathy's autoresearch .md

@nvbkdw @nummanali yes, solid work trending in a good direction, but almost all my work is across like 20 different machines (my local, my claw machine, my gpu machines). possibly they could add ssh mode, a bit like VS Code does (for the same reasons).

@karpathy @nummanali Codex desktop is pretty good

Claude Code teams with tmux is really cool When you run with team mode enabled in tmux, it automatically opens the additional terminal in pane I don't really get my main agent to orchestrate, I chat to them myself CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=true claude