b/acc, context platform engineer

AI Context Memory/Storage makes Claude Code & OpenClaw work Also fastest growing segment of the hot trillion dollar AI Infra industry Jensen calls it the biggest storage market ever. Micron estimates it at 100 exabytes 🤯 @vikramskr and I break it all down on @semidoped 🎤

vLLM v0.18.0 is out! 445 commits from 213 contributors (61 new). 🎉 What's new: gRPC serving, GPU-less multimodal render, NGram spec decode on GPU, Elastic EP Milestone 2, FlashInfer 0.6.6, Responses API streaming tool calls. Thread 👇

Yesterday we open sourced the InferenceX webapp Hopefully this makes it easier to analyze InferenceX data, and provides a simple way for accelerator startups and alternative runtimes to compare directly to the industry standards, and make performance claims + forecasts It also means we can more easily handle feature requests from the community for the public dashboard github.com/SemiAnalysisAI…

AI is already redesigning chip design itself! And the biggest bottleneck left is validation. Here is Bill Dally describing to @JeffDean how @nvidia uses AI to design chips: “We’re already using AI across multiple parts of the chip design process, and it’s delivering real gains. Take standard cell design as an example. Every time we move to a new process, we have to port thousands of cells. That used to take a team of eight engineers around 8–10 months. Now we use a system we built called NVCell, and it can do that work overnigt. The results match or exceed human designs across key metrics like size, power, and delay. It’s a massive productivity boost and removes a major bottleneck when moving to new process nodes. We’re also using reinforcement learning in tools like Prefix RL to tackle classic computer design problems, such as optimizing carry chains. This is a problem people have worked on since the 1950s. The RL system explores the space like a game, evaluating its own designs. Instead of maximizing raw speed, it aims to meet timing constraints while minimizing area and power. It often produces unconventional designs that no human would consider, yet they outperform human solutions by 20–30% on those metrics. At a higher level, we’ve built internal LLMs like ChipNeMo and BugNeMo. These models are trained on our internal design corpus: RTL, architecture specs, documentation. They effectively act as expert assistants. One of the biggest benefits is reducing the load on senior engineers. Instead of repeatedly answering basic questions, junior engineers can query the model and get detailed, interactive explanations. It’s like having an infinitely patient mentor. We also use these models for debugging. They can summarize bug reports, help with attribution, and suggest which module or engineer should take ownership. That speeds up the whole debugging loop. On the exploratory side, we’re using generative methods to run large numbers of design experiments. We can explore different architectural directions, map parameter spaces, and quickly evaluate new ideas. The goal is to compress the time between early exploration and final design. One of the biggest bottlenecks today is design validation. We’d like to use AI to prove correctness much faster. There are also stages where we need to refactor designs, for example when repartitioning logic during floorplanning. Those transitions are complex and error-prone, and they’re good candidates for automation. Now, the dream would be fully end-to-end automation: you specify a new GPU, go skiing for a few days, and come back to a finished design. We’re nowhere near that yet. But across the pipeline, AI is already making the process significantly faster and more productive.”

NVIDIA agreed to pay $20 billion to license Groq's technology and hire its team. That tells you everything about where single-workload AI chips are headed. Groq built its architecture around deterministic execution. Every operation is statically scheduled before runtime. All model weights live in on-chip SRAM. For dense transformer inference on a standard model, the latency was extraordinary. But the workloads moved. Mixture-of-experts models like DeepSeek-V3 activate only a fraction of their parameters per token. Inference-time compute scaling means different queries need wildly different amounts of compute. Static scheduling is the opposite of what you want when the workload is variable by design. ASICs take two to three years from architecture freeze to first silicon. Model architectures are now shifting faster than that. A chip designed for dense matrix multiply does not help when the workload is sparse routing across 256 experts. You can read the NVIDIA-Groq deal as a vote of confidence in inference IP. You can also read it as the market telling you that the standalone path for a fixed-workload ASIC was narrowing so fast that selling was the better move. Which reading do you think is right?

So excited to see more research and breakthrough in omni-modality! This is exactly why we are buidling vLLM-Omni github.com/vllm-project/v… for the next generation of Intelligence!🚀

Great to see @AMD select vLLM as one of the designated inference frameworks for the GPU MODE Hackathon. 🎉 The challenge: push Kimi K2.5 1T FP4 end-to-end inference performance on 8× AMD Instinct MI355X — using vLLM or AMD ATOM. Grand prize: $650,000. What makes this different: winning optimizations must be mergeable into AMD ATOM or vLLM upstream. Improvements that land in vLLM benefit the whole community. Phase 1 (kernel optimization) runs through April 6. More details ⬇️

@karpathy So you’re telling me, all I have to do, is fundamentally reshape the biggest industry in the world, about four or five times, and you just get one of these?

Thank you Jensen and NVIDIA! She’s a real beauty! I was told I’d be getting a secret gift, with a hint that it requires 20 amps. (So I knew it had to be good). She’ll make for a beautiful, spacious home for my Dobby the House Elf claw, among lots of other tinkering, thank you!!

@TheTuringPost Networking SRAM has capacity limits, so it remains to be seen how cost effectively this multi-LPU configuration scales for coding agents, and other highly KV cacheable AI workloads.

Straight from NVIDIA GTC: Jensen Huang just unveiled a new vision for AI infrastructure For the first time, Rubin GPUs+Groq LPUs are paired: > 35× higher inference throughput > 10× more revenue from trillion-parameter models Architecture & why it's needed

📎We've uploaded it to arXiv, enjoy! arxiv.org/abs/2603.15031

The bottleneck has so quickly moved from code generation to code review that it is actually a bit jarring. None of the current systems / norms are setup for this world yet.

Subagents are now available in Codex. You can accelerate your workflow by spinning up specialized agents to: • Keep your main context window clean • Tackle different parts of a task in parallel • Steer individual agents as work unfolds

THIS is the wildest open-source project I’ve seen this month. We were all hyped about @karpathy's autoresearch project automating the experiment loop a few weeks ago. (ICYMI → github.com/karpathy/autor…) But a bunch of folks just took it ten steps further and automated the entire scientific method end-to-end. It's called AutoResearchClaw, and it's fully open-source. You pass it a single CLI command with a raw idea, and it completely takes over 🤯 The 23-stage loop they designed is insane: ✦ First, it handles the literature review. - It searches arXiv and Semantic Scholar for real papers - Cross-references them against DataCite and CrossRef. - No fake papers make it through. ✦ Second, it runs the sandbox. - It generates the code from scratch. - If the code breaks, it self-heals. - You don't have to step in. ✦ Finally, it writes the paper. - It structures 5,000+ words into Introduction, Related Work, Method, and Experiments. - Formats the math, generates the comparison charts, - Then wraps the whole thing in official ICML or ICLR LaTeX templates. You can set it to pause for human approval, or you can just pass the --auto-approve flag and walk away. What it spits out at the end: → Full academic paper draft → Conference-grade .tex files → Verified, hallucination-free citations → All experiment scripts and sandbox results This is what autonomous AI agents actually look like in 2026. Free and open-source. Link to repo in 🧵 ↓

Excited to share our latest work rethinking k-means for modern GPU architectures. While the algorithm is classic, scaling it requires aggressively targeting memory bandwidth bottlenecks. Flash-KMeans introduces a strictly IO-aware design that achieves exact results with up to a 30x speedup over cuML and 200x over FAISS, completing million-scale iterations in milliseconds!

🎉 Congrats to @MistralAI on releasing Mistral Small 4 — a 119B MoE model (6.5B active per token) that unifies instruct, reasoning, and coding in one checkpoint. Multimodal, 256K context. Day-0 support in vLLM — MLA attention backend, tool calling, and configurable reasoning mode, verified on @nvidia GPUs. 🔗 huggingface.co/mistralai/Mist…

AI will push $1 trillion in hardware spending through 2027, @Nvidia CEO Jensen Huang says at #GTC26

“Inference is the ultimate hard”

“The inference inflection has arrived” “The inference inflection has arrived” -Jensen @ GTC26

@vllm_project spotted at GTC 2026!🔥

Deepseek v4 still not released Alibaba Qwen going closed Western open weights models slacking In these dark times for open source, who will save us? Alliances must be made, brothers must band together! A world of only closed source AI will lead to consolidation of power! Tyranny!

This is so damn cool! Transformers do attention across tokens, now imagine doing attention across layers too. This delivers a 1.25x compute efficiency, <4% training overhead on the 48B Kimi model, +7.5 on GPQA-Diamond. Kimi is quietly becoming the new DeepSeek for the coolest architecture innovation.