Emrick

I observed the same issue in one of our production cluster deployments. It's not tied to the latest vLLM release as it was running with v0.19.1. The root cause is a bug in flashinfer gdn_prefill CUTLASS kernel. This kernel was rewritten in 0.6.14, which fixes the issue. The version has been bumped in vLLM main but hasn't been released yet. On older versions, you can mitigate the issue by adding --gdn-prefill-backend triton to the arguments

The latest vLLM releases seem to be broken: randomly freezing and getting stuck at Avg prompt throughput: 0 tokens/s, and it doesn't seem related to the new V2 runner 🤔. Anyone experiencing a similar issue?

@alexocheema Yes please!

We're starting to roll out early access for local dot ai. We're 1% through the early access list. We have a Local AI wall where anyone with early access can put up a post-it note. A few notes were already left on the wall. Once we release publicly, we're going to print it and frame it in the EXO office in SF. Comment below and I will DM you an early access code.

Here we go !! 💻🧠

So cool to be in SF for H Company’s first hackathon! 🚀 Lots of builders are here this weekend, building with our Agent API. It's been great meeting the community in person. The energy here is amazing !

The @hcompany_ai is waiting for you to start building! See u in a bit 🚀

@RaemondBW @zoox Yeah, I'm interested too !

Thanks for the ride @zoox … now how does a person get off the waitlist?

@0xSero What tool are you using for the tracking ? 👀

Officially crossed 1 Billion tokens locally, 230M just yesterday with GLM-5.2 I got what I wanted.

More details on the setup here:

Running a computer use agent fully offline on my MacBook during a flight ✈️ Task: find papers I downloaded before the flight in ~/Downloads, summarize them, rank their relevance for inference / computer use / online RL, and write the result in Notes. No cloud. No internet. Fully on-device 😎

I downloaded a few papers before the flight and I wanted a short summary of each to prioritize which one to look at first. I asked it to find them in ~/Downloads/, write a summary of each and explain how relevant they are for inference for computer use and online RL in the Notes app. Two of the papers are actually super interesting for us ! I did a recoding, I'll share it, it's fun to watch 🎬

@EmrickSini Which task did you give it ?

I'm on a 10-hour flight from Paris to SF and I'm currently running a fully on-device computer use agent on my MacBook M4 Pro. What a time to be alive !! 🤯 The setup is fully open source: - Model: Holo 3.1 35B-A3B Q4_K_M - Inference engine: llama.cpp - Local CUA harness: holo-desktop-cli There is a lot of buzz around the latest GPT-5.6 model release and its computer use capability, that's cool ! But running computer use with small models, fully on-device, is even cooler 🔋 Give it a try, I'll put the links below !

@phibrowser Exactly ! Yours even offline

@EmrickSini running CUA fully offline at 35,000 feet is the dream. no cloud, no round trip, the whole loop on your machine. the part people miss about on-device: its not just privacy, its that the agent stays yours even when the network doesnt. great stack.

Links: - Model: huggingface.co/Hcompany/Holo-… - Local CUA harness: github.com/hcompai/holo-d…

I observed the same thing with a 35B A3B FP8 model running on an H100 using MTP-4. An average acceptance length of 3.35 was actually slower than running without speculative decoding. MTP-3 turned out to be a better trade off. It's really a trade off between reducing memory/IO traffic and paying for the extra draft/verification compute.

25GB RAM, 370GB SSD, 0 GPU. Colibri getting GLM-5.2 744B to run locally is cool, but speculative decoding at 59% draft acceptance still makes it ~28% slower. Once experts stream from disk, better drafts can mean worse tok/s.

@WescheNex1q @NVIDIAAI Cool results ! what's the precision of the model? Also, for power consumption, 38W seems like the lower end. What I observed was closer to 60W for 35B FP8 on prompt heavy workloads

One DGX Spark. Qwen3.6-35B 64 users. 700+ tok/s. 32,768 tokens in 54 seconds. 38W Each user has their own prompt and their own KV cache, and vLLM batches every active stream through the GPU each step. Recipe → github.com/Weschera/spark… @NVIDIAAI

Here’s the full write-up: hcompany.ai/booting-fast-a… We go through every optimization, the benchmarks, and the upstream contributions to ModelExpress and vLLM.

Our largest model replicas used to take up to 27 minutes to boot. Not train. Not serve a request. Just become ready. I spent the last few weeks attacking this across the stack: Docker pulls, S3 weight loading, GPU to GPU transfers, and vLLM compile caches. End result: - 3x faster cold starts - 12x faster warm starts - 397B FP8 weight pull: ~11 min to 3.4 min - peer transfers around 341 Gbps Big thanks to @VictorGoubet , @AvshalomM, @nvidia Model express folks, and the @vllm_project community for the help and reviews. I wrote up the details. Link in comments. We're hiring at @hcompany_ai if this is your kind of rabbit hole.

@DFintelligence @ChatGPTapp You could have done the same with HoloTab few months ago: hcompany.ai/holotab. You should try it ! 🙏🇫🇷

GPT 5.6 Sol just saved me €650 a year and demonstrated just how good this model is as an agent in @ChatGPTapp Codex. This is not a clickbait, let me explain. In France, insurance companies tend to hide all their prices behind quote forms that take at least 5min to complete for a single configuration on a single provider's website. And that take an other 5min to understand. If you want to compare 10 companies across 5 configurations each, it takes at least 4h +. (It's such a painful process that entire businesses exist just to compare insurance offers.) Since I have two cars, it would normally take me 8h so an entire day to have a real large view of my best option. Companies know this and use the friction to maintain overpriced offers. So I asked @ChatGPTapp with GPT 5.6 Sol, using Chrome tabs, to go through all those annoying forms. I provided him all my contrats with my current insurance companies so he have context. For some insurers, you even have to speak with a representative just to get a quote (which is absurd in 2026), so it emailed the companies, exchanged the required information, and obtained the prices. It then ran a complete benchmark, read all the terms and conditions, and recommended three options from three different companies. I picked one, and it completed the subscription with my new insurance company. And that's how I ended up with better insurance coverage for less money. For 4% of my weekly quota in 20x plan. (i think it's fair) All of that happened while I was walking my dog for 50 minutes, he was working on my computer all by it's on. Yes, computer use existed before @OpenAI GPT 5.6 Sol, but this is a completely different level in the way it handles these kinds of tasks. I think this story shows the new era of AI we're entering, good model is not only for one single task as coding or answering question, AI now can do things for you, like in your daily live. I love being able to hand my computer over to GPT 5.6 Sol. PS: The only annoying part was that some companies still require "Verify you're human" checks. In the age of AI agents, websites really need to be ready for robot access.

I’ll be in SF on July 11-12 for @hcompany_ai first hackathon, alongside @nvidia and @Accel. If you're building AI agents, workflow automations, or new ways for humans to interact with computers, come hack with us. 2 days of building and $20k+ in prizes🔥 Registration below! 🚀