Amit Doshi

With the coming tsunami of demand for tokens, there are significant opportunities to orchestrate the underlying memory+compute *just right* for LLMs. The fundamental and non-obvious constraint is that due to the chip fabrication process, you get two completely distinct pools of memory (of different physical implementations too): 1) on-chip SRAM that is immediately next to the compute units that is incredibly fast but of very of low capacity, and 2) off-chip DRAM which has extremely high capacity, but the contents of which you can only suck through a long straw. On top of this, there are many details of the architecture (e.g. systolic arrays), numerics, etc. The design of the optimal physical substrate and then the orchestration of memory+compute across the top volume workflows of LLMs (inference prefill/decode, training/finetuning, etc.) with the best throughput/latency/$ is probably today's most interesting intellectual puzzle with the highest rewards (\cite 4.6T of NVDA). All of it to get many tokens, fast and cheap. Arguably, the workflow that may matter the most (inference decode *and* over long token contexts in tight agentic loops) is the one hardest to achieve simultaneously by the ~both camps of what exists today (HBM-first NVIDIA adjacent and SRAM-first Cerebras adjacent). Anyway the MatX team is A++ grade so it's my pleasure to have a small involvement and congratulations on the raise!

The companies that succeed in the future are going to make very heavy use of AI. People will manage teams of agents to do very complex things. Today we are launching Frontier, a new platform to enable these companies.

You’re going to see more and more service providers go big on AI coding and other AI automation. To understand why, it’s useful to think about the power of distribution and the nature of the firm in the age of AI. Initially Cognition is a clear disruptor to the core Infosys business. Yet, they currently have access to some of the world’s largest enterprises and are on contract with them to do large scale software projects. The choice for them or any other service provider is to either slowly have their business decline or use AI to do bigger and bigger projects at a lower cost for clients. So that choice is relatively easy (even if a little painful at first). But why do enterprises go for this instead of just doing it themselves now? This is where it’s useful to understand the Theory of the Firm. The service providers, due to their centralization, can more quickly standardize how they leverage AI faster than the rest of the market where it happens one by one. This allows them, *in theory*, to improve their practices faster than the rest of the market can as a whole, thus actually offering an economically advantageous service for customers. Of course this only works if they truly can deliver a better service than what’s offered internally by the company. What we’ll likely see is what we’ve always seen: a bifurcation of types of software and technology projects in an organization. You’ll want to own, in-house, the core products and experiences you offer customers, but whether your company, or a partner, oversees agents to do that COBOL migration doesn’t really matter.