Clive Chan

2.3K posts

Clive Chan banner
Clive Chan

Clive Chan

@itsclivetime

perplexity per picojoule @anthropicai // prev jalapeno @openai, dojo @tesla

California, USA Katılım Mayıs 2012
3.1K Takip Edilen28.4K Takipçiler
Sabitlenmiş Tweet
Clive Chan
Clive Chan@itsclivetime·
Personal update: I’ve decided to leave OpenAI. I’m proud to have been part of the custom chip program and grateful to everyone I got to build with and learn from along the way. The density of hardware talent on that team is extraordinary, and I don't think there's a better chip design team anywhere. It's been a wild journey from second hardware hire, 2.4 years ago, to now, and I'm excited to watch these chips become one of the most important engines of AGI. At the same time, I haven’t been able to shake the pull to climb a new mountain from the bottom again! I joined @AnthropicAI this week because I was deeply impressed with the team’s talent, values, and ambition, and I'm already energized by the pace and intensity of the past few days. It’s time to build.
English
393
319
7.4K
2.8M
Clive Chan
Clive Chan@itsclivetime·
@0xBADB01E even at low latency this doesn't win - you can't use the LPDDR capacity anyway (takes an entire second to read out the memory). HBM bandwidth/$ is usually cheaper than LPDDR, and an SRAM only design like Groq will be even better than that.
English
1
0
7
654
Clive Chan
Clive Chan@itsclivetime·
Good analysis, but two issues: - This assumes ultra low latency batch1, which is a valid workload, but for higher latencies and batch sizes GPUs beat this tok/joule by orders of magnitude. - A 30ns full mesh over 576 chips is not possible with current technology. Maybe in a few years once we have better CPO tech?
English
4
1
55
7.9K
Big Boss
Big Boss@0xBADB01E·
I agree with this post whole heartedly but I’d push it even further. The interconnect IS the binding constraint for AI even more so than memory. If we want faster inference & training with better economics we are best served by designing our interconnect first and then working backwards towards the optimal chip architecture. Today’s chips weren’t really designed with this principle in mind. There is no better example than running autoregressive decode on a GPU. Despite all those reticle sized logic dies & CoWoS integration decode runs at under 20% of peak FLOPs on Blackwell, wasting silicon and burning power while waiting for memory. The naive solution has always been to increase memory bandwidth whether that’s adding more HBM or using SRAM. However, that is a vast simplification of the problem which I’ll explain later. But if you were clever you’d have realized while reading that you could feed those idle FLOPs by streaming weights over the interconnect itself. Wallah 🪄 you just discovered the idea that forms the basis behind disaggregated memory from first principles. But sadly this currently doesn’t work on Nvidia’s hardware. NVLink5 carries 1.8 TB/s against 8 TB/s of local HBM, and scale out is 80x behind that. The “pipe” is smaller than the memory at the other end and thus leads to worst token/sec if its relied upon. But we get an interesting lemma out of this which is that remote memory is only as fast as the interconnect. Therefore you must balance the pipe for the memory it attaches to. SRAM needs an 80 TB/s link, HBM needs 2+ TB/s, and LPDDR gets away with a couple hundred GB/s. So Nvidia selling a rack of 72 GPUs, each GPU’s memory is pretty segregated. The core idea is still sound though but this raises a question, why would Nvidia build a fabric that’s high bandwidth and high latency leading to memory access being segregated per GPU? It’s because they were optimizing for training over inference. Training is dominated by collectives on huge tensors, and a couple microseconds of latency on a huge all reduce operation is just noise so the bandwidth gains justify the latency tradeoff. But more importantly, this also works because it matches what the chip is good at. GPUs are great at hiding latency with occupancy (also what allows them to be OK for training) but bandwidth is the only thing warp switching can’t create. You can justify a 224G PAM4 + FEC with overhead when you have a chip that’s designed to be latency tolerant as well. It’s a latency tolerant fabric for a latency tolerant chip. Maybe a good design for training but inference inverts this completely. Now everyone knows decode is bandwidth bound so you might assume again that more/higher BW memory and thus higher BW interconnects are necessary. However, it’s the exact opposite and the name of the game is actually lower latency and that’s why despite having high bandwidth memory MFU on decode is still so low and also why I made the point earlier that the interconnect is MORE important than the memory itself as well as the chip architecture. In part two I will explain why lower latency interconnects are not just ideal for inference but also allow you to get away with a smaller cheaper memory and a simpler chip architecture.
Big Boss tweet media
outside five sigma@jwt0625

chips get all the love but the interconnects across all levels from c2c to rack-to-rack are as important, and many chip makers are sleeping on it until very recently. Even most interconnects people just want it to be as transparent as possible, just send and receive the bits with lower error rates and lower energy per bit, wrong long-term direction imo. Interconnects are part of the living creature, so many things happening in your blood vessels in addition to just moving stuff, and your axons do much more than carrying spikes. People do not appreciate interconnects, smaller volume, poor margin, messy ecosystem, manual process, it's been a spiral of grinding, and it is largely invisible. How often do you see people tearing down transceivers and die shot of DSP chips vs logic chips? How often do you see high res pictures of all the connector's gold fingers on the NVL72 cartridge? Because it sounds boring, it's just making contacts, shoving electrons and photons, what a simple problem. But that is deceiving, and theres so much to it. You might want 576 to begin with, had to cut down to 288, then to 144, and finally to 72, and that barely worked first time. You are entering the domain of analog and mixed signaling, you are fighting copper real estate with power delivery, you are getting impedance mismatch and reflections and interference at every stupid interfaces, your optical components' and connectors' backreflection is making your laser mode hop.. And we are not even going into the thermal and strain-stress, the reliability of how many times you can actually mate your connector, the horrible jobs people are doing across the stack from science-project-originated photonic PDKs to hand cleaved laser dies to optical engines to rack manufacturing, on spec-ing out the requirements, the tolerances.. On top of all these, people thinking about where the bits should be going and people who know what the bits have to go thru are two totally different groups of people. But it is shifting, pluggable volumes shipped are doubling and tripling for scale-out, scale-up domain asks for much higher bw than scale-out, and interconnects are inevitable even if you cram as much compute and memory onto a single wafer. People will see it always has been interconnects, the chips people have already been doing it on the chips, that you can sort it out with your chip designers and foundries, and now you need to work with more people to sort it out from chips to boards/trays to racks to pods to data halls and data centers. These people speak very different languages and care about very different things, and it will take a lot of effort to pull order out of all the chaos. At the end of the day, it is such a crazy problem to work on, such a beautiful thing to make, millions of amps of current flipping 1e20 of flops, sextillions of photons carrying thousands of terabits per second, a few tons of copper, tens of thousands of fibers totally few hundred kilometers, one scale-up domain. You absolutely need a group of people that appreciate the beauty and care about the craft behind the grinding to make it together.

English
50
87
1.2K
360.3K
Clive Chan
Clive Chan@itsclivetime·
🇺🇸
ART
0
0
15
3.2K
Clive Chan retweetledi
Merriam-Webster
Merriam-Webster@MerriamWebster·
Why is it ‘cancelled’ in the U.K. but ‘canceled’ in the U.S.? Because we gave them that L in 1776.
English
3K
39.2K
271.8K
16.9M
Clive Chan retweetledi
JakuJaku
JakuJaku@_jaku_xu·
I've been obsessed with robotics for as long as I can remember. That passion led me through exoskeletons, autonomous vehicles, humanoid robotics, and eventually mobile manipulation research during my master at @UWaterloo. Each experience brought me closer to the same question: How do we build general-purpose robots that can interact with the world as naturally as we do? The more I worked on robotics, the more convinced I became that dexterous manipulation is one of the biggest missing pieces. Twenty months ago, we started with a simple belief: dexterous manipulation won't be solved by hardware alone. To build truly capable robotic hands, you need mechanical design, embedded systems, sensing, AI, and large-scale human demonstration data working together as one tightly integrated system. We believe the robot hand and the human demonstration interface should evolve together, not independently. The journey was far from smooth. We faced engineering problems that seemed impossible at first. We rebuilt systems more times than I can count. There were long nights, failed prototypes, manufacturing setbacks, and moments when it felt like we were taking one step forward and two steps back. On top of that, we found ourselves navigating a legal battle that could have distracted or slowed us down. Instead, we stayed focused on building, and I'm grateful that chapter is now behind us. Through every setback, one thing never changed. Everyone on this team believed the problem was worth solving. Today, that belief becomes something tangible. We're officially launching ProHand and ProGlove. Together, they're our first step toward giving robots the hands they've always needed. Today we're also announcing our $11M seed round, led by @firstround, with participation from @BoxGroup and @ycombinator. Looking back, I'm grateful to everyone who believed in us before there was much to show. Our teammates, investors, advisors, early customers, friends, and families made this possible. This launch isn't the finish line. It's a reminder that difficult chapters don't define a company. Resilience, execution, and the people around you do. We're just getting started, and I'm excited for what comes next :) —- 🌐 Website: proception.ai 🎥 Launch Video (@proceptionAI x @osmo_studio): Watch on Youtube 💼 Join Our Team (Legends): proception.ai/careers
English
20
21
143
28.8K
Clive Chan
Clive Chan@itsclivetime·
celebrating with jalapeno nuggets
Clive Chan tweet media
OpenAI@OpenAI

We’ve designed and built our first AI chip: Jalapeño. Designed from the ground up by OpenAI and brought to production with @Broadcom, Jalapeño is purpose-built for the LLM workloads powering ChatGPT, Codex, the API, and future agentic products. Chips are foundational to the AI economy. Building our own expands our full-stack platform from products to models to infrastructure, and will help us scale intelligence, serve more people, and expand access to AI.

English
4
3
115
11.1K
Jake Ayes
Jake Ayes@jake_ayes·
@itsclivetime You gotta give us some sorta tease of the vibe of the chip
English
1
0
0
44
Ravi Vijayakumar
Ravi Vijayakumar@thehiphopswami·
With some spicy Jalapeno chips.
Ravi Vijayakumar tweet media
English
15
7
305
19.1K
Clive Chan retweetledi
Ravi Vijayakumar
Ravi Vijayakumar@thehiphopswami·
It was a lot of fun working this chip with an awesome group of engineers!
OpenAI@OpenAI

We’ve designed and built our first AI chip: Jalapeño. Designed from the ground up by OpenAI and brought to production with @Broadcom, Jalapeño is purpose-built for the LLM workloads powering ChatGPT, Codex, the API, and future agentic products. Chips are foundational to the AI economy. Building our own expands our full-stack platform from products to models to infrastructure, and will help us scale intelligence, serve more people, and expand access to AI.

English
18
9
194
19.9K