@Lentils80 But is the unit here tokens or arbitrary units?
English
Stevan Boljevic
510 posts
Stevan Boljevic
@StevanBoljevic
President @ OLIX
London, England Katılım Ocak 2013
1.2K Takip Edilen244 Takipçiler
@StevanBoljevic It's basically how long and hard the model thinks for a given thinking level (they put the value in "hidden channels" that the model can see)
For example juice value of max thinking level now is the same as the old xhigh level
English
🚨 GPT-5.6 Sol's juice values (thinking budgets) have been severely degraded compared to release day
If Sol now feels faster and more "efficient", this is probably why
Terra and Luna juice values aren't affected, so their thinking budgets are now higher than Sol's
English
@Lentils80 So you're prompting the model to get it to reveal the thinking budget and/or sending single traces into the model - measuring billed output tokens via API vs. measured output tokens in the visible response?
English
@StevanBoljevic The juice value is thinking budget like I said, just ask it what the juice value is and if it denies to answer use clever tricks
English
In your architecture you have higher memory capacity per unit memory bandwidth but in order to fully utilise this capacity vs. a GPU you have to run at lower interactivity.
LPDDR5 has higher data load cost energy vs. HBM4 so at the limit of the inference physics you end up less energy efficient and lower throughput/W.
You may make up for this in lower capex/W and end up with comparable TCO; I haven't run the BOM math.
In your example of the B200 being bandwidth bound on 70B model for a large batch size:
- In the case where you are capacity bound for 70GB weights with 122GB KV cache across 8TB/s bandwidth your latency per token is 0.024s = 42TPS/U. The latency penalty of Pipeline parallelism to cross the interconnect of 20us from your original post is <1% of the total output token time so not material.
As batch size increases your total data movement energy per token becomes increasingly dominated by KV cache data load which is not ammortised by batch; and the higher data movement cost of LPDDR vs. HBM becomes more significant.
I am a big believer in the importance of interconnect performance & latency but it is most important for extremely high interactivity inference where the workload starts to become latency bound rather than bandwidth bound.
English
Definitely not full mesh over 576 chips my fault for glossing over that Clive. The model is an 8 chip pod, 7 ports per chip, full mesh inside the pod. Between pods you’d want a different topology, probably a torus or dragonfly, but the point I’m making is that the latency critical TP domain stays inside the pod where there are zero switch hops.
Now on high batch workloads you are right past Batch 64 the MAC array throttles but since we have floorplan to spare, my simple answer is just to add more MACs 🤣🫨😎♟️✨😵💫 In fairness I wrote this on the toilet and my legs were going numb so I had to rush through a few details.
But I want to point out also that your high batching cuts the other way as well as it drags KV with it. At B=256 with just a totally unserious 8K context that’s ~2M cached tokens or ~336 GB of KV at FP8. Reading that once per step is 42 ms at 8 TB/s, or roughly 3x the latency of GEMM. And attention itself runs at ~8 FLOPs/byte with GQA so at this high batch workload you are memory bound on basically every chip ever built, including mine. The workload you’re describing isn’t compute bound at all at real context windows 🪟 its bandwidth and more IMPORTANTLY capacity bound !
So you do the math again:
per 8 TB/s of bandwidth, one 8-chip pod on my side, one B200 on theirs it’s 1.5 TB of memory against 192 GB. Normalize it however you want, since bandwidth is what sets throughput I bring ~190 GB per TB/s, HBM brings 24. That ratio holds whether you count chips, trays, or racks.
The imaginary B=256/8K scenario needs ~370 GB for weights + KV. That does not fit in 192 GB without quantization gymnastics or spilling across more GPUs. On the pod it fits four times over with the same bandwidth and same read time and the only difference being that one of us serves the batch in a single pod while the other is sharding it across extra $40K packages and paying the interconnect tax.
So at high batch and at real context windows the memory capacity is the problem and that’s where this arch wins. So you see this isn’t some Groq Junior CompE novice hit job your too quick to write me off, this… is fucking cinema Clive.
GIF
English
I agree with this post whole heartedly but I’d push it even further. The interconnect IS the binding constraint for AI even more so than memory. If we want faster inference & training with better economics we are best served by designing our interconnect first and then working backwards towards the optimal chip architecture.
Today’s chips weren’t really designed with this principle in mind. There is no better example than running autoregressive decode on a GPU. Despite all those reticle sized logic dies & CoWoS integration decode runs at under 20% of peak FLOPs on Blackwell, wasting silicon and burning power while waiting for memory. The naive solution has always been to increase memory bandwidth whether that’s adding more HBM or using SRAM. However, that is a vast simplification of the problem which I’ll explain later. But if you were clever you’d have realized while reading that you could feed those idle FLOPs by streaming weights over the interconnect itself. Wallah 🪄 you just discovered the idea that forms the basis behind disaggregated memory from first principles.
But sadly this currently doesn’t work on Nvidia’s hardware. NVLink5 carries 1.8 TB/s against 8 TB/s of local HBM, and scale out is 80x behind that. The “pipe” is smaller than the memory at the other end and thus leads to worst token/sec if its relied upon. But we get an interesting lemma out of this which is that remote memory is only as fast as the interconnect. Therefore you must balance the pipe for the memory it attaches to. SRAM needs an 80 TB/s link, HBM needs 2+ TB/s, and LPDDR gets away with a couple hundred GB/s. So Nvidia selling a rack of 72 GPUs, each GPU’s memory is pretty segregated. The core idea is still sound though but this raises a question, why would Nvidia build a fabric that’s high bandwidth and high latency leading to memory access being segregated per GPU?
It’s because they were optimizing for training over inference. Training is dominated by collectives on huge tensors, and a couple microseconds of latency on a huge all reduce operation is just noise so the bandwidth gains justify the latency tradeoff. But more importantly, this also works because it matches what the chip is good at. GPUs are great at hiding latency with occupancy (also what allows them to be OK for training) but bandwidth is the only thing warp switching can’t create. You can justify a 224G PAM4 + FEC with overhead when you have a chip that’s designed to be latency tolerant as well. It’s a latency tolerant fabric for a latency tolerant chip. Maybe a good design for training but inference inverts this completely.
Now everyone knows decode is bandwidth bound so you might assume again that more/higher BW memory and thus higher BW interconnects are necessary. However, it’s the exact opposite and the name of the game is actually lower latency and that’s why despite having high bandwidth memory MFU on decode is still so low and also why I made the point earlier that the interconnect is MORE important than the memory itself as well as the chip architecture. In part two I will explain why lower latency interconnects are not just ideal for inference but also allow you to get away with a smaller cheaper memory and a simpler chip architecture.
outside five sigma@jwt0625
chips get all the love but the interconnects across all levels from c2c to rack-to-rack are as important, and many chip makers are sleeping on it until very recently. Even most interconnects people just want it to be as transparent as possible, just send and receive the bits with lower error rates and lower energy per bit, wrong long-term direction imo. Interconnects are part of the living creature, so many things happening in your blood vessels in addition to just moving stuff, and your axons do much more than carrying spikes. People do not appreciate interconnects, smaller volume, poor margin, messy ecosystem, manual process, it's been a spiral of grinding, and it is largely invisible. How often do you see people tearing down transceivers and die shot of DSP chips vs logic chips? How often do you see high res pictures of all the connector's gold fingers on the NVL72 cartridge? Because it sounds boring, it's just making contacts, shoving electrons and photons, what a simple problem. But that is deceiving, and theres so much to it. You might want 576 to begin with, had to cut down to 288, then to 144, and finally to 72, and that barely worked first time. You are entering the domain of analog and mixed signaling, you are fighting copper real estate with power delivery, you are getting impedance mismatch and reflections and interference at every stupid interfaces, your optical components' and connectors' backreflection is making your laser mode hop.. And we are not even going into the thermal and strain-stress, the reliability of how many times you can actually mate your connector, the horrible jobs people are doing across the stack from science-project-originated photonic PDKs to hand cleaved laser dies to optical engines to rack manufacturing, on spec-ing out the requirements, the tolerances.. On top of all these, people thinking about where the bits should be going and people who know what the bits have to go thru are two totally different groups of people. But it is shifting, pluggable volumes shipped are doubling and tripling for scale-out, scale-up domain asks for much higher bw than scale-out, and interconnects are inevitable even if you cram as much compute and memory onto a single wafer. People will see it always has been interconnects, the chips people have already been doing it on the chips, that you can sort it out with your chip designers and foundries, and now you need to work with more people to sort it out from chips to boards/trays to racks to pods to data halls and data centers. These people speak very different languages and care about very different things, and it will take a lot of effort to pull order out of all the chaos. At the end of the day, it is such a crazy problem to work on, such a beautiful thing to make, millions of amps of current flipping 1e20 of flops, sextillions of photons carrying thousands of terabits per second, a few tons of copper, tens of thousands of fibers totally few hundred kilometers, one scale-up domain. You absolutely need a group of people that appreciate the beauty and care about the craft behind the grinding to make it together.
English
@AgustinLebron3 Seems to align to the introduction of adaptive thinking where the orchestrator/mode selects how much reasoning to apply and reasoning by default was reduced. Does this benchmark enforce maximum reasoning?
English
🦔Microsoft canceled its internal Claude Code licenses this week after token-based billing made the cost untenable, even for a company with effectively infinite cloud resources. Uber's CTO sent an internal memo warning the company burned through its entire 2026 AI budget in just four months. American AI software prices have jumped 20% to 37%, and GitHub (owned by Microsoft) is dropping flat-rate plans for usage-based billing across its products.
My Take
The AI subsidy era is ending in real time. The same company that put $13 billion into OpenAI and built the Azure infrastructure powering most of Anthropic's compute just looked at the bill from a competitor's coding tool and decided it was not worth paying. That is not a productivity failure on Anthropic's end. Token-based pricing is forcing every enterprise customer to confront the actual cost of running these models at scale, and the number turns out to be far higher than the flat-rate experiments suggested.
This ties directly to my Gemini Flash post yesterday. Anthropic, OpenAI, and Google all raised effective prices in the last six months. Enterprises that built workflows assuming AI costs would keep falling are now watching annual budgets evaporate in months. Two outcomes look likely from here. Either enterprises scale back AI usage to fit budgets, which slows the revenue ramp the labs need to justify their valuations ahead of IPOs, or the labs cut prices and absorb the losses, which makes the unit economics worse at exactly the wrong moment. Both paths land in the same place, the numbers stop working, and somebody has to take the writedown.
Hedgie🤗
English
@Ross__Hendricks These numbers line up decently with bottoms up data.
H100 ~1kW ~ $2.20/hr = 20B/year + frontier lab gross margin
NVIDIA GPU pricing pre DRAM spikes was ~$34/W; now tracking ~38.
~0.20/kWh @ 1GW = 1.7B
Is your claim that powered shell capex is >>>12B/GW?
English
Never heard someone use “back-of-the-envelope” to refer to their own asshole, which is where these numbers were pulled from
David Sacks@DavidSacks
Back-of-envelope numbers for 1 gigawatt data center: All-in Capex: ~$50 bn Enterprise revenue generated: ~$25-30 bn/year Electricity cost: $1-2 bn/year ~2 year payback. The boom is real.
English
@METR_Evals Will you be sharing per-run data from Mythos Preview? I find it fascinating to track token efficiency across time and model releases using your data.
English
We evaluated an early version of Claude Mythos Preview for risk assessment during a limited window in March 2026. We estimated a 50%-time-horizon of at least 16hrs (95% CI 8.5hrs to 55hrs) on our task suite, at the upper end of what we can measure without new tasks.
English
@FinnStockinger Demand response doesn’t cover the TCO loss during down time. It does not help the economics of training a model.
English
The Geometry of Power: Why 1 GW is Not the Same as 5 x 200 MW
Academic circles still whisper about "distributed cloud computing," but the engineers building the successors to GPT-6 know better: physics is a merciless judge.
Training Frontier AI models requires a singular, massive compute fabric.
If you split 1 GW of power across five separate 200 MW locations, you hit the "latency wall."
Synchronizing data across cities causes the world’s most expensive processors to waste 40% of their time idling, waiting for a signal to arrive from a distant hub.
A true 1 GW super-cluster is a single, dense organism. It allows for direct, high-speed interconnects where millions of GPUs exchange parameters in microseconds.
At this scale, traditional air cooling is a joke, we are talking about industrial-grade hydraulics and rivers of coolant liquid dissipating heat that could warm a metropolis.
If you cannot build that density in one spot, you will never leave the "chatbot league."
Grid-Native: The "Infinite Money Glitch"
The industry has split into two survival ideologies.
The "Off-Grid" camp, led by Elon Musk’s Colossus project, deploys massive gas turbine arrays directly at the data center to bypass 7-year utility wait times.
It is a brilliant sprint, but a miserable marathon - gas is expensive, and carbon emissions are becoming a political liability.
The future belongs to the Grid-Native model championed by Microsoft and IREN.
Here, the data center becomes a vital organ of the power grid.
By utilizing 100% renewables and massive battery storage, these facilities earn money twice: once from AI compute, and again from stabilizing the grid (Demand Response).
When the Texas ERCOT system nears a breaking point during a heatwave, IREN throttles its servers for a few minutes, saves the state from a blackout, and pockets a fortune in credits that gas-powered players will never see.
This is "subsidized compute" - a model that makes the cost of training AI nearly free compared to the competition.
IREN: 4.5 GW and the "Voucher on Time"
The biggest shock of 2026 is the ascension of IREN.
A company that started as a Bitcoin miner now controls the most valuable asset on the planet:
In the U.S., the wait time for a 500 MW+ power drop is now a decade.
IREN owns the dirt that is already "live."
Their power pipeline has reached a staggering 4.5 GW following the February announcement of their Oklahoma hub.
This transforms IREN from a "small player" into the primary wholesaler of power, where tech giants arrive with blank checks.
Microsoft has already secured capacity there with a nearly $10 billion contract.
While other hyperscalers are drowning in bureaucracy, IREN is simply flipping the switch.
Anthropic: The "Homeless Giant" in a Trap
In this new architecture of power, the biggest loser is Anthropic.
As the only one of the "Big Four" without its own bricks and transformers, they are mere tenants of Google and Amazon.
The moment these landlords decide to prioritize their own internal models (Gemini or Titan), Anthropic will be evicted to inferior, distributed locations.
Their only hope for computational sovereignty and survival in the AGI race is an alliance with power wholesalers like IREN.
Without their own gigawatt, Claude 5 will remain a footnote in the shadow of giants.
The Verdict:
In 2026, software is a commodity and hardware is a capital expense.
The only real moat is the Gigawatt.
If you haven't secured dense, single-site power by the end of this year, you have already opted out of history. IREN is closing the gate.
Finn Stockinger@FinnStockinger
The "Paper Capacity" Trap: Why $IREN is Years Ahead of the Herd The market is finally waking up to a brutal reality: you can’t power AI with press releases. A new report from Sightline Climate (thanks @HedgieMarkets) reveals that nearly 50% of US data centers planned for 2026 will likely be delayed or canceled. The reason? A massive shortage of "boring" electrical components like transformers and switchgear, with lead times now stretching to 5 years. But here is what the "big money" is missing: $IREN isn't waiting in that 5-year line. They are already at the front of it. Execution vs. Ambition While hyperscalers scramble to source parts from China, $IREN is moving into the final stages of its 2.75GW roadmap. They didn't start thinking about power in 2024; they’ve been securing it since the Bitcoin mining days. ➡️Childress (750MW): Many expected this to wrap up in 2025, but $IREN is now in the final stretch (Phase 6) in early 2026. Why is this a win? Because they physically secured the transformers years ago. While competitors are just now filing permits, $IREN has 810MW already "Energized" and operational across its sites. ➡️ Sweetwater (1,400MW): This is where the gap becomes a canyon. $IREN has already locked in the procurement for the massive 1,400MW substation for a 2026 launch. They aren't hoping for parts, they own the slots. The Vertical Moat $IREN acts as its own developer (EPC). By owning the substations and the land, they’ve bypassed the supply chain paralysis killing the 2026 pipeline. The Bottom Line: In 2026, the only metric that matters is "Time-to-Power." Most companies have "announced" gigawatts; $IREN has "energized" megawatts and a $9.7B Microsoft-backed runway to monetize them. The bottleneck is tightening, but $IREN built their door years ago. Are you betting on "planned" capacity or the ones who already have the transformers on-site? Let’s talk below. 👇 #AI #IREN #DataCenters #Energy #Infrastructure #TechInvesting
English
Demand is growing at the speed of thought with internet scale distribution.
Supply grows at the pace of building out $100B manufacturing facilities alienly complex beyond comprehension.
This time couldn’t possibly be different, right?
English
@NaveenGRao I hadn’t considered the perspective, which you rightly raise, that the headline figures could include Anthropic model tokens. Thanks for teaching me something today.
English
@StevanBoljevic Yes, you correct. I had some conversations with folks that indicated it was API services, but the official word from Google was "all services"
pymnts.com/artificial-int…
English
Amazing. But, one thing I noticed...they say 15 billion tok/min. That's 0.657 quadtrillion tok/month. Google claimed they were at >1 quadtrillion tok/month last year.
Google is shipping more tokens than OAI on API services? Interesting...
Sawyer Merritt@SawyerMerritt
NEWS: OpenAI just announced that it has officially closed their latest funding round with $122 billion in committed capital at a post money valuation of $852 billion. "We are now generating $2B in revenue per month. At this stage, we are growing revenue four times faster than the companies who defined the Internet and mobile eras, including Alphabet and Meta. ChatGPT has more than 900 million weekly active users, and over 50 million subscribers. Search usage has nearly tripled in a year, and our ads pilot reached more than $100 million in ARR in under six weeks. Momentum is just as strong on the enterprise side, which now makes up more than 40% of our revenue, and is on track to reach parity with consumer by the end of 2026. GPT‑5.4 is driving record engagement across agentic workflows. Our APIs now process more than 15 billion tokens per minute. Codex now serves over 2 million weekly users, up 5x in the past three months, with usage growing more than 70% month over month."
English
@NaveenGRao I’m thinking back to this datapoint from mid 2025; is there a more recent quadrillion data point?
Logan Kilpatrick@OfficialLoganK
Google is processing 980 trillion+ monthly tokens across our products and APIs (up from 480T in May) 🤯 No slowdown in sight, intelligence is everywhere.
English
@StevanBoljevic no, that's no correct.
x.com/NaveenGRao/sta…
Naveen Rao@NaveenGRao
@wintonARK @downingARK @Google @GoogleCloudTech @GoogleAI Ah, yes. The quad number is likely combining ALL models (not only Gemini)...remember, GCP serves Anthropic and open models as well. So that checks out. You'd need 22.8B tok/min to cross 1 quad/month. So this implies that google serves more of [Ant|open models] than Gemini
English
English
I’m thrilled to be joining @sequoia as a Partner on the Early team.
Getting to know @Konstantine, @Alfred_Lin, @gradypb, and the entire Sequoia team, I have been humbled by their exceptional quality of character and caliber of thought – it's always a joy to learn from those who operate at the highest level of their craft.
It's an honor to be a part of this team and to support those daring to make a dent in the universe.
English
This is a batch 1 which is not economical on TPU. Increased batching raises weight load pressure further, requiring more aggressive tensor parallelism to maintain even reading speed interactivity.
Now you’re tensor parallelising across multiple hops in the torus, and you’re introducing extreme scale up bandwidth pressure and latency across the hops.
Your interactivity is so low you have to pray this model is ASSI (super super intelligence) because you need it to one shot your prompt almost immediately otherwise you’re waiting forever to get any output back
English
5T active @ 4 bit gives 2.5TB/Tok weights to move.
Assuming you are serving this on TPU v7 memory bandwidth from memory is ~8 TB/s.
To get even 30 Tok/s you’d need to ensure parallelise across ~16 TPUs at a minimum; before you have even started to account for KV cache movement which is presumably also huge on this model.
High tensor parallelism introduces additional latency per layer; making the interactivity even more punishing.
English
Within 1 year, I think Google will train a huge model on 2 million TPU v7 Ironwood chips running for 6 months, producing close to 10 ZettaFLOPS at peak and 384 Petabytes of HBM. 7.6 × 10²⁸ total FLOPS — 3,800× GPT-4’s training compute.
The net result would be a 54-Trillion parameter MoE with 5 Trillion active per token, 32-million token context window, trained on 500 Trillion multimodal tokens with 55% of all compute going to RL reasoning training.
English
@TeddyK59558411 @JosephJacks_ Inference physics still applies to the largest compute fleets.
English
@StevanBoljevic @JosephJacks_ If they got the compute it shouldn't be an issue?
English
@IsaiahJGonzalez @JosephJacks_ DeepSeek MLA / DSA does nothing for the crushing burden of the weight moving load on such a model
English
@Hail_BAIJ @JosephJacks_ KV cache compression does nothing to alleviate the huge pressure on memory bandwidth from weight loading from this hypothetical model
English
@StevanBoljevic @JosephJacks_ Google is way ahead of you on that
Google Research@GoogleResearch
Introducing TurboQuant: Our new compression algorithm that reduces LLM key-value cache memory by at least 6x and delivers up to 8x speedup, all with zero accuracy loss, redefining AI efficiency. Read the blog to learn how it achieves these results: goo.gle/4bsq2qI
English