Big Boss

Me & the bro when we team up to work on leading edge computer technology

We need open source, American AI models to become the default as quickly as possible.

local 1T parameter models are now only $94,000. I don't think we've ever seen a cost curve get pushed to zero as quickly as AI. We live in a prosperity era

In this week, an Intel patent application was published, revealing its proposed Cross-Batch Memory (XBM), an ultra high-bandwidth memory that offers some significant improvements over the current standard, which could be a direct competitor to HBM4 in the near future.

chips get all the love but the interconnects across all levels from c2c to rack-to-rack are as important, and many chip makers are sleeping on it until very recently. Even most interconnects people just want it to be as transparent as possible, just send and receive the bits with lower error rates and lower energy per bit, wrong long-term direction imo. Interconnects are part of the living creature, so many things happening in your blood vessels in addition to just moving stuff, and your axons do much more than carrying spikes. People do not appreciate interconnects, smaller volume, poor margin, messy ecosystem, manual process, it's been a spiral of grinding, and it is largely invisible. How often do you see people tearing down transceivers and die shot of DSP chips vs logic chips? How often do you see high res pictures of all the connector's gold fingers on the NVL72 cartridge? Because it sounds boring, it's just making contacts, shoving electrons and photons, what a simple problem. But that is deceiving, and theres so much to it. You might want 576 to begin with, had to cut down to 288, then to 144, and finally to 72, and that barely worked first time. You are entering the domain of analog and mixed signaling, you are fighting copper real estate with power delivery, you are getting impedance mismatch and reflections and interference at every stupid interfaces, your optical components' and connectors' backreflection is making your laser mode hop.. And we are not even going into the thermal and strain-stress, the reliability of how many times you can actually mate your connector, the horrible jobs people are doing across the stack from science-project-originated photonic PDKs to hand cleaved laser dies to optical engines to rack manufacturing, on spec-ing out the requirements, the tolerances.. On top of all these, people thinking about where the bits should be going and people who know what the bits have to go thru are two totally different groups of people. But it is shifting, pluggable volumes shipped are doubling and tripling for scale-out, scale-up domain asks for much higher bw than scale-out, and interconnects are inevitable even if you cram as much compute and memory onto a single wafer. People will see it always has been interconnects, the chips people have already been doing it on the chips, that you can sort it out with your chip designers and foundries, and now you need to work with more people to sort it out from chips to boards/trays to racks to pods to data halls and data centers. These people speak very different languages and care about very different things, and it will take a lot of effort to pull order out of all the chaos. At the end of the day, it is such a crazy problem to work on, such a beautiful thing to make, millions of amps of current flipping 1e20 of flops, sextillions of photons carrying thousands of terabits per second, a few tons of copper, tens of thousands of fibers totally few hundred kilometers, one scale-up domain. You absolutely need a group of people that appreciate the beauty and care about the craft behind the grinding to make it together.

So following these principles here is the architecture that falls out and it’s quite different than what is thought to be optimal today. Starting with the memory: LPDDR6 tops out around 14.4 Gb/s per pin. We would want to as many channels as the die’s edge can carry. If we can fit 24 channels, that’s 576 data pins, and you get ~1 TB/s per chip with 192 GB of storage. Same as the b200 in size but about an a100 in bandwidth. But remember that LPDDR access latency is ~100 ns which is about the same as HBM. So streaming weights from LPDDR costs you nothing on latency and we just give up per package bandwidth, and remember our pipelining already collapsed that requirement. Now work backwards to compute. Tokens/sec = ~aggregate BW / bytes touched per token, and at batch 64 in FP4 every delivered byte wants ~256 FLOPs. 1 TB/s × 256 = ~260 TF of FP4. Size ~280 TF for some headroom and stop! That’s all we need, the rest of the alpha is in latency tuning not flops. Now process node, again using first principles thinking. On TSMC N6 that MAC array would be about 30 mm^2. The rest of the die goes to SRAM buffers, 24 LPDDR PHYs, and 56 lanes of plain 32 GT/s NRZ SerDes PCIe 5.0 transceivers, nothing exotic. Because we didn’t provision more flops we can provision what we really need which is ports. If we do 7 ports × x8 we can get a full mesh across 8 chips with ~224 GB/s any to any ops, sized for 8 KB activations instead of 100 MB gradients. Approximate Total Area: ~144 mm^2, ~180W with the DRAM. Fucking Mobile SoC level economics. And we get away without a crazy process node. N3 shrinks your logic, but that’s actually the one thing this die barely has. SRAM stopped scaling and the PHYs and SerDes are analog so they don’t shrink at all. An N3 port shrinks ~20% of the floorplan, and saves maybe 10-15W, but doubles the cost on a small die that already has beautiful *chefs kiss* 👨‍🍳💋 yield. Now do the node math: 8 chips (b/c full mesh w/ 7 ports) = 8 TB/s aggregate BW, 1.5 TB total of memory, and ~1.44 kW. A B200 inside an NVL72 is 8 TB/s, 192 GB, ~1.67 kW all in. So we are basically neck and neck on BW with a b200 with slightly lower power because we have more chips connected with quicker interconnects allowing them to act as a single unit. If we scale to the rack level at 72 nodes, that 576 chips, 576 TB/s which is again the same aggregate bandwidth as a GB200 NVL72, so approximately the same tokens/sec by construction but with ~104 kW of power against ~120-130 and 110 TB of memory against 13.8 TB of the NVL72. So in summary 4.5x less compute silicon, zero HBM, zero CoWoS, zero leading-edge nodes and still token/watt ~20% better. So what’s stopping everyone from doing this? The number I told you hold whether you want to believe it or not and get even more competitive with longer reasoning chains or agentic workloads because as you scale those the all reduces now sits on the critical path of every token and on a 30 ns link with a full mesh that’s orders of magnitudes of token throughput improvements over Nvidia’s stack. Counterintuitively the link latency, not the chip architecture, decides your memory, logic, and everything else! Thanks for reading if you made it this far.