llmrequirements

Qwen 3.6 27B is the best coder that fits a single 24 GB card. It hits 77.2% on SWE-bench Verified. Here is exactly what it needs. Weights / min VRAM / comfortable: Q2_K: 9 / 11 / 13 GB Q4_K_M: 16 / 20 / 24 GB Q5_K_M: 19 / 24 / 29 GB Q8_0: 31 / 38 / 46 GB BF16: 54 / 58 / 70 GB So a 16 GB card is Q2 only. 24 GB (a used 3090) runs Q4 with headroom. 32 GB (5090, R9700, Arc Pro B70) gets you Q5. Q8 wants 38 GB, which means a 48 GB workstation card. Measured decode, quant and recipe stated: RTX 5090, NVFP4 + MTP at 575W: ~92 tok/s at 200K context RTX 3090 used, INT4 AutoRound + MTP: roughly 70 to 85 tok/s 2x RTX 3090, AWQ-INT4 + MTP on vLLM TP=2: ~100 tok/s Every card, both quant toggles and a VRAM calculator: llmrequirements.com/model/qwen3-6-…

A single DGX Spark runs Qwen3.5 122B-A10B at about 81 tok/s on real agent traffic. Most reviews quote the low 50s. Same box, same model, four regimes: Plain INT4, no speculative decoding: 28.2 tok/s Public MTP-2 stack: ~51 tok/s DFlash, general single-stream decode: ~59 tok/s DFlash, agent and tool-call traffic at 100K context: ~81 tok/s 2 to 3x from software alone, on hardware you already paid for. And the lever is the drafter, not the quant. NVFP4 has no measured win on this model. DFlash is a 0.8B block-diffusion drafter that guesses a block ahead and gets about 8.3 tokens accepted per verification pass, which is why structured agent traffic runs faster than free-form prose, not slower. The full recipe, the harness and the per-run numbers: llmrequirements.com/news/2026-06-3…

@BosonJoe I tried to have Hermes on the spark but it competes with inference for the memory. Plus if you want to do web fetch, you need it with the monitor, headless is always blocked. Why no 27b on 5090s though? NVFP4 one

Here is my local Hermes Agent network I realized it is much less confusing (for me and the agent) if the LLM driving Hermes is on a separate machine Especially true for benchmarking, explaining to the agent that it can't take itself down to make space is a test of patience

@digitalix Pls no qwen3 4b

High Bandwidth Memory detected

Stop buying local AI cards by their TOPS number. It's the wrong spec. What sets your token speed is memory bandwidth. RTX 5090: 1,792 GB/s, about 186 tok/s on an 8B. R9700: 640 GB/s, about 77 tok/s. The 2.4x speed gap tracks the 2.8x bandwidth gap, not the 5090's 3,352 TOPS. And it inverts. Intel's Arc Pro B70 lists 367 TOPS, above a 3090's 285. On a 14B the 3090 still wins 50 to 40. Higher TOPS, slower on the model you actually run. Decode follows bandwidth. Shop by GB/s.

@NVIDIARTXSpark 27b nvfp4 is slower, but is a better alternative. Still fast enough with speculative decoding. Or 35b FP8. Also I found it is better to host the harness on a separate device. Inference competes for memory with harness' tools (like browser).

🦞 NemoClaw: nvda.ws/4yh60bU ⚙️ OpenClaw: nvda.ws/4h8ffVJ 🧠 Hermes: nvda.ws/4yk4jL0 🔒 OpenShell: nvda.ws/4eYo578 Explore the playbooks and start building: nvda.ws/4baTJfx

Ready to get started with a local AI agent workflow on DGX Spark? 👀 We've got agentic playbooks covering NemoClaw, OpenClaw, Hermes, and OpenShell, from setting up agents to securing long-running workflows, all running on locally. 👇

@mr_r0b0t @vllm_project @NVIDIAAI MTP 3 should give some increase I believe

SM121 optimized @vllm_project v0.25.0 producing a noticeable increase in decode speeds when serving @NVIDIAAI-Qwen3.6-35B-A3B-NVFP4! 93.05 tok/s on a single DGX Spark (GB10) at MTP=2 full engine configuration and results: github.com/r0b0tlab/qwen3…

One DGX Spark cannot run the model everyone actually wants. 128GB is not enough for DeepSeek V4-Flash, a 284B MoE. Add a second Spark and it does 61 tok/s single stream. A frontier model on two mini PCs and an ethernet cable. You do not get a bigger model by adding node two. You get a different class of model. That 1 to 2 jump is the whole Spark cluster story.

@loganbyte It is Twitter hot takes generator and community manager got sloppy again. You don't even need 4 to run deepseek v4 flash. And on 4 boxes you can run GLM 5.2.

@llmrequirements That's a huge price difference for the same model.

The H100 rack is getting eaten from below. A 4-box DGX Spark cluster is $19,500 and runs DeepSeek V4-Flash, a 284B frontier MoE, at 49 tok/s on your desk. Under 300W, sits on a shelf. To serve that same model in a rack you need 2x H100 at $64,000 or 2x H200 at $80,000. Same model. 3 to 4x cheaper. No rack.

@MiaAI_lab Electricity cost needs to be taken into account. 4x3090 are much hungrier than dgx spark

Apple quietly killed the 512GB Mac Studio. It now caps at 96GB. That was the one mainstream box you could buy with that much unified memory. The 512 config was $14,199 and it is refurb-only now, and the M5 Ultra meant to replace it slipped to October. So where does big local AI go now? Two DGX Sparks run DeepSeek V4-Flash, a 284B model, at 61 tok/s for $9,500 the pair. The cluster ate the big-memory Mac.

LLM Requirements now has recipes. Running some recipe? Press I run this recipe, so people would see which recipes are actually used. Have you recipes or some other useful recipe which is not in the list? Add it. Need recipe to run it on your hardware? Find it there. llmrequirements.com/recipes

Apple says the M5 Max is up to 4x faster for LLMs. That 4x is prompt processing, not token generation. Prefill is compute bound and the new matrix units crush it, about 2x at long context. Decode is bandwidth bound and bandwidth only went up 12 percent, so replies come out just a bit quicker than the M4 Max. What the 128GB tier is actually for is one model. Qwen 3.5 122B, a vision MoE that needs about 80GB, runs around 66 tok/s at 4-bit. No 64GB laptop can hold it. Everything smaller runs great on the cheaper 64GB box. Full breakdown of what it runs, and whether to wait for the M5 Ultra Studio now slipped to about October. llmrequirements.com/apple-m5-max-l…

The DGX Spark is not the regret buy everyone calls it. It is just a different tool than the RTX Pro 6000. The Pro 6000 is faster, sure. 85 vs 30 tok/s on a 27B. But that is not the model a Spark buyer runs. On a 122B MoE, the thing the Spark is actually for, it does about 59 tok/s general and 81 on agent traffic with DFlash spec decode. Usable, not regret. The Pro 6000 build is 12,200 dollars and 600W. The Spark is 4,699 and 240W. A third of the price, well under half the power, on the model you would actually load. Regret is overpaying for the wrong tool, not buying the one that fits your use case.

@MiaAI_lab @NVIDIAAI @UnslothAI Check this comment on reddit. They recommend CuteDSL+ trtllm reddit.com/r/LocalLLaMA/c…

@llmrequirements @NVIDIAAI @UnslothAI MTP enabled with speculative_tokens set to 2

Here's the speed you'd get on @NVIDIAAI DGX Spark running the new @UnslothAI Qwen3.6-35b-NVFP4-Fast in a single session 🔥

You do not need a multi GPU server to run a 122B model at home anymore. One DGX Spark. 128GB unified, 4,699 dollars. It runs Qwen3.5 122B at about 59 tok/s on general decode and 81 on agent traffic with DFlash speculative decoding. The no spec baseline on the same box is 28 tok/s, so the drafter is doing real work here, not a trick. A 32GB 5090 cannot even load a model this size. Two runs back the number, the NVIDIA forum benchmark and our own on the box.

@mr_r0b0t But Nvidia NVFP4 runs 100+ tps. Not sure how they calculated the gains.

unsloth/Qwen3.6-35B-A3B-NVFP4-Fast Confirming -Fast is FAST 🏎️🏎️🏎️ repo with sm121 optimized container incoming ♥️

@volatilemarkts @digitalix @Tech2Wild Tony's one is q8q4, not NVFP4 though

@digitalix Haven’t found a working recipe. 8-10 floating around. @Tech2Wild recipe is a really strong one. I have a second one which is giving me stronger analytics.. but it’s slower. Yell if you want it. @Tech2Wild recipe is genuinely novel.

has anyone gotten GLM5.2 working on Blackwell GPUs with NVFP4?

@trueschoo @digitalix I saw community only q8 q4. Is there NVFP4 recipe?

@digitalix DGX Spark community on 4 sparks