Nilesh Tawari, Ph.D.

Qwen3.8-27B is coming! 🔥 Will run locally on 17GB RAM/VRAM setups.

DeepSeek-V4-Flash-0731 + DSpark speculative decoding 2× RTX PRO 6000 Blackwell (TP=2, official FP8, 524K maxlen) Results (median): • 243 tok/s single stream — 3.1× over the no-spec baseline • Aggregate throughput: c2 299 · c4 403 tok/s • Draft accept: 74–76%, zero failures I expected an SM120 kernel wall. Never hit one. The real bottleneck: max_num_seqs > 4 fails deterministically on the first request. A hardcoded prefill chunk size feeds an empty slice into the sparse kernel. Capped at 4, it held a 3-hour mixed-load soak at 237–245 tok/s. Looks like this will be my daily driver for now.

AI companies should just switch to open-weight models with restrictive licenses. It’s a win for everyone. They can offload the inference to 3rd-party compute providers and simply collect royalties on model usage. By doing this, they don't need to take on unsustainable debt to buy massive compute, and it naturally triggers healthy capitalistic price competition. Plus, as an open-weight ecosystem, we will accelerate even faster through shared knowledge. Their obsession with staying closed is purely driven by their greed for a monopoly.