Stanley Yuan 🔋

Find a problem (hopefully a massive one) you are uniquely capable of solving, give it your all and let compounding do its trick in the long run. And build a future you'd want to live in with your own hands.

The Terafab "Yield Buffer": Why 160k is the Real Number Elon just clarified the math on Terafab, and the 60% jump in wafer starts (from 100k to 160k per month) tells a massive story about the reality of 2nm manufacturing. In my original breakdown, I estimated 100k wafers/month to hit 100 million AI5 chips/year. That assumes a relatively mature yield (60%+). Elon’s response, "Probably more like 160k wafers/month, factoring in yield", is a reality check. The "Bleeding Edge" Tax: Launching a 2nm fab from scratch is historically difficult. By aiming for 160k wafers, Tesla is building in a massive safety margin. If initial yields are lower (closer to 35-40%), they still hit the 100 million chip target. Monthly Starts: 160,000 wafers Annual Capacity: 1.92 Million wafers The Goal: 100 Million "Good" Chips Required Net Yield: ~35% (The "Launch" yield) The Upside: If yields hit 65%, output jumps to ~190 Million chips/year The TSMC Benchmark: Matching the Giant To put 160k wafers/month in perspective, look at TSMC. As of early 2026, TSMC’s entire global 2nm capacity (spread across multiple "Gigafabs" in Hsinchu and Kaohsiung) is targeting roughly 100k to 140k wafers per month. By pushing for 160k, Elon is essentially saying that a single Tesla Terafab cluster aims to outproduce the entire world’s initial 2nm supply.

@pbeisel Probably more like 160k wafers/month, factoring in yield

@bindureddy Google will win the AI race in the West, China on Earth and SpaceX in space

We’ve been able to generate physics-accurate, real-time video for self-driving training & testing at @Tesla_AI for a long time. The compute required for this (roughly one H100 per HD camera) is still far too expensive for consumer use, but probably becomes affordable in 2 to 3 years.

Tesla’s forthcoming AI5 uses a half-reticle design, which is crucial for yield. A reticle defines the imaging area of a lithography machine, fitting two chips per shot effectively doubles yield. This means the Tesla chip design team had to carefully manage die features, for instance dropping the older ISP (and classic GPU) to make room for more AI cores. By contrast, NVIDIA’s Blackwell fills nearly a full reticle, making it a single-reticle design. If Tesla hits its compute and efficiency targets with AI5 in this half-reticle format, it’s almost like cutting fab requirements in half. And this has a big impact on Terafab, especially if it carries forward for AI6, AI7, etc.

@pbeisel I am a huge admirer of Nvidia and Jensen btw. That market cap is well-deserved. SpaceX AI and Tesla expect to continue ordering Nvidia chips at scale.

AI5 will punch far above its weight, because the entire Tesla AI software stack is designed to make maximally effective use of every circuit. We co-signed our AI software and hardware. Bear in mind that AI5, while it can be used for training in data centers, is primarily optimized for AI edge compute in Optimus and Robotaxi. There is still significant room for improvement. In the same half reticle and same process node, we think a single AI6 chip has the potential to match a dual SoC AI5.

@DBurkland @pbeisel It’s in testing right now. Wide release in a few weeks.

Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. 🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. 🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. 🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. 🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. 🔗Full report: github.com/MoonshotAI/Att…

@ai_for_success For a few years, then SpaceX will far exceed everyone combined

Big release from Kimi! They just released a new way to handle residual connections in Transformers. In a standard Transformer, every sub-layer (attention or MLP) computes an output and adds it back to the input via a residual connection. If you consider this across 40+ layers, the hidden state at any layer is just the equal-weighted sum of all previous layer outputs. Every layer contributes with weight=1, so every layer gets equal importance. This creates a problem called PreNorm dilution, where as the hidden state accumulates layer after layer, its magnitude grows linearly with depth. And any new layer's contribution gets progressively buried in the already-massive residual. This means deeper layers are then forced to produce increasingly large outputs just to have any influence, which destabilizes training. Here's what the Kimi team observed and did: RNNs compress all prior token information into a single state across time, leading to problems with handling long-range dependencies. And residual connections compress all prior layer information into a single state across depth. Transformers solved the first problem by replacing recurrence with attention. This was applied along the sequence dimension. Now they introduced Attention Residuals, which applies a similar idea to depth. Instead of adding all previous layer outputs with a fixed weight of 1, each layer now uses softmax attention to selectively decide how much weight each previous layer's output should receive. So each layer gets a single learned query vector, and it attends over all previous layer outputs to compute a weighted combination. The weights are input-dependent, so different tokens can retrieve different layer representations based on what's actually useful. This is Full Attention Residuals (shown in the second diagram below). But here's the practical problem with this idea. Full AttnRes requires keeping all layer outputs in memory and communicating them across pipeline stages during distributed training. To solve this, they introduce Block Attention Residuals (shown in the third diagram below). The idea is to group consecutive layers into roughly 8 blocks. Within each block, layer outputs are summed via standard residuals. But across blocks, the attention mechanism selectively combines block-level representations. This drops memory from O(Ld) to O(Nd), where N is the number of blocks. Layers within the current block can also attend to the partial sum of what's been computed so far inside that block, so local information flow isn't lost. And the raw token embedding is always available as a separate source, which means any layer in the network can selectively reach back to the original input. Results from the paper: - Block AttnRes matches the loss of a baseline LLM trained with 1.25x more compute. - Inference latency overhead is less than 2%, making it a practical drop-in replacement - On a 48B parameter Kimi Linear model (3B activated) trained on 1.4T tokens, it improved every benchmark they tested: GPQA-Diamond +7.5, Math +3.6, HumanEval +3.1, MMLU +1.1 The residual connection has mostly been unchanged since ResNet in 2015. This might be the first modification that's both theoretically motivated and practically deployable at scale with negligible overhead. More details in the post below by Kimi👇 ____ Find me → @_avichawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

5 minutes ago, @karpathy just dropped karpathy/jobs! he scraped every job in the US economy (342 occupations from BLS), scored each one's AI exposure 0-10 using an LLM, and visualized it as a treemap. if your whole job happens on a screen you're cooked. average score across all jobs is 5.3/10. software devs: 8-9. roofers: 0-1. medical transcriptionists: 10/10 💀 karpathy.ai/jobs

@ClayTravis Only to be topped by Tesla Optimus!

@BrianRoemmele The AI race will come down to scaling power and chip output

@peterwildeford xAI will catch up this year and then exceed them all by such a long distance in 3 years that you will need the James Webb telescope to see who is in second place

Terafab Project launches in 7 days

Teslas will become mobile compute nodes