Light Foundry Notes

DeepSeek-V4-Flash-0731 + DSpark speculative decoding 2× RTX PRO 6000 Blackwell (TP=2, official FP8, 524K maxlen) Results (median): • 243 tok/s single stream — 3.1× over the no-spec baseline • Aggregate throughput: c2 299 · c4 403 tok/s • Draft accept: 74–76%, zero failures I expected an SM120 kernel wall. Never hit one. The real bottleneck: max_num_seqs > 4 fails deterministically on the first request. A hardcoded prefill chunk size feeds an empty slice into the sparse kernel. Capped at 4, it held a 3-hour mixed-load soak at 237–245 tok/s. Looks like this will be my daily driver for now.

DeepSeek-V4-Flash-0731 + DSpark speculative decoding 2× RTX PRO 6000 Blackwell (TP=2, official FP8, 524K maxlen) Results (median): • 243 tok/s single stream — 3.1× over the no-spec baseline • Aggregate throughput: c2 299 · c4 403 tok/s • Draft accept: 74–76%, zero failures I expected an SM120 kernel wall. Never hit one. The real bottleneck: max_num_seqs > 4 fails deterministically on the first request. A hardcoded prefill chunk size feeds an empty slice into the sparse kernel. Capped at 4, it held a 3-hour mixed-load soak at 237–245 tok/s. Looks like this will be my daily driver for now.

DeepSeek-V4-Flash-0731 @ 2× RTX PRO 6000 Blackwell (TP=2) This is what 243 tok/s actually looks like.

Full recipe: DeepSeek-V4-Flash-0731 + DSpark on 2× RTX PRO 6000 Blackwell (SM120) Several people asked for the exact setup. Everything below is measured on my own system. ## MODEL Official deepseek-ai/DeepSeek-V4-Flash-0731 (HF) • FP8, ~167 GB • Fits in 2×96 GB VRAM as-is • No requantization required • The DSpark draft model ships in the same repository One small tip: downloading individual files with curl was much more reliable than the HF CLI, which repeatedly stalled for us. --- ## ENGINE Current upstream vLLM and SGLang do not support this model. You'll need a community vLLM build with: • SM120 sparse kernels • DSpark speculative decoding • Built after Jul 31 The Jul 31 release introduced a new MTP norm layout. Older builds fail during weight loading with: KeyError: ...mtp_block.main_norm.weight I'm intentionally not naming the image publicly to avoid sending everyone to the maintainer. Happy to share via DM. --- ## LAUNCH (working configuration) --tensor-parallel-size 2 --max-model-len 524288 --max-num-seqs 4 --gpu-memory-utilization 0.94 --kv-cache-dtype fp8 --speculative-config '{"method":"dspark","num_speculative_tokens":5}' --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --block-size 256 --max-num-batched-tokens 2048 --disable-custom-all-reduce --trust-remote-code (prefix caching disabled) num_speculative_tokens must be >= 5 (the DSpark block size). Anything lower is rejected during config validation before CUDA is even initialized. --- ## THE max_num_seqs LIMIT max_num_seqs > 4 deterministically fails on the first request: cannot reshape tensor of 0 elements Root cause: A hardcoded prefill chunk size (4) feeds an empty padding slice into the sparse MLA kernel. Confirmed on both community builds we tested. Cap max_num_seqs at 4. Additional requests simply queue instead of crashing. --- ## WHAT TO EXPECT Measured on this setup. Code generation (temp=0) • 235-246 tok/s single stream General prose • 130-145 tok/s Aggregate throughput • c2 ~300 tok/s • c4 ~400 tok/s Other numbers • Prefill: 5-6K tok/s • 384K needle retrieval: ~80 s • KV pool: ~988K tokens Why the spread? The engine runs at roughly 50 verification steps/s. Throughput is approximately: tok/s = 50 × (1 + accepted draft tokens per step) Acceptance depends heavily on workload: • Code (temp=0): ~73% • Prose: ~33% Don't estimate throughput from SSE delta counts. Those measure verification steps, not generated tokens. --- ## SANITY LADDER 1. Health check 2. Deterministic text 3. Usage-metered throughput 4. Needle (128K to 384K) 5. Tool calling 6. Confirm the max_num_seqs > 4 signature If all six pass, you should have essentially the same environment I do. Questions welcome.

DeepSeek-V4-Flash-0731 @ 2× RTX PRO 6000 Blackwell (TP=2) This is what 243 tok/s actually looks like.

Full recipe: DeepSeek-V4-Flash-0731 + DSpark on 2× RTX PRO 6000 Blackwell (SM120) Several people asked for the exact setup. Everything below is measured on my own system. ## MODEL Official deepseek-ai/DeepSeek-V4-Flash-0731 (HF) • FP8, ~167 GB • Fits in 2×96 GB VRAM as-is • No requantization required • The DSpark draft model ships in the same repository One small tip: downloading individual files with curl was much more reliable than the HF CLI, which repeatedly stalled for us. --- ## ENGINE Current upstream vLLM and SGLang do not support this model. You'll need a community vLLM build with: • SM120 sparse kernels • DSpark speculative decoding • Built after Jul 31 The Jul 31 release introduced a new MTP norm layout. Older builds fail during weight loading with: KeyError: ...mtp_block.main_norm.weight I'm intentionally not naming the image publicly to avoid sending everyone to the maintainer. Happy to share via DM. --- ## LAUNCH (working configuration) --tensor-parallel-size 2 --max-model-len 524288 --max-num-seqs 4 --gpu-memory-utilization 0.94 --kv-cache-dtype fp8 --speculative-config '{"method":"dspark","num_speculative_tokens":5}' --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --block-size 256 --max-num-batched-tokens 2048 --disable-custom-all-reduce --trust-remote-code (prefix caching disabled) num_speculative_tokens must be >= 5 (the DSpark block size). Anything lower is rejected during config validation before CUDA is even initialized. --- ## THE max_num_seqs LIMIT max_num_seqs > 4 deterministically fails on the first request: cannot reshape tensor of 0 elements Root cause: A hardcoded prefill chunk size (4) feeds an empty padding slice into the sparse MLA kernel. Confirmed on both community builds we tested. Cap max_num_seqs at 4. Additional requests simply queue instead of crashing. --- ## WHAT TO EXPECT Measured on this setup. Code generation (temp=0) • 235-246 tok/s single stream General prose • 130-145 tok/s Aggregate throughput • c2 ~300 tok/s • c4 ~400 tok/s Other numbers • Prefill: 5-6K tok/s • 384K needle retrieval: ~80 s • KV pool: ~988K tokens Why the spread? The engine runs at roughly 50 verification steps/s. Throughput is approximately: tok/s = 50 × (1 + accepted draft tokens per step) Acceptance depends heavily on workload: • Code (temp=0): ~73% • Prose: ~33% Don't estimate throughput from SSE delta counts. Those measure verification steps, not generated tokens. --- ## SANITY LADDER 1. Health check 2. Deterministic text 3. Usage-metered throughput 4. Needle (128K to 384K) 5. Tool calling 6. Confirm the max_num_seqs > 4 signature If all six pass, you should have essentially the same environment I do. Questions welcome.

Correction : the first video was undercounting. My counter was off. DeepSeek-V4-Flash-0731 @ 2× RTX PRO 6000 Blackwell (TP=2) Here's what 244 tok/s actually looks like:

Full recipe: DeepSeek-V4-Flash-0731 + DSpark on 2× RTX PRO 6000 Blackwell (SM120) Several people asked for the exact setup. Everything below is measured on my own system. ## MODEL Official deepseek-ai/DeepSeek-V4-Flash-0731 (HF) • FP8, ~167 GB • Fits in 2×96 GB VRAM as-is • No requantization required • The DSpark draft model ships in the same repository One small tip: downloading individual files with curl was much more reliable than the HF CLI, which repeatedly stalled for us. --- ## ENGINE Current upstream vLLM and SGLang do not support this model. You'll need a community vLLM build with: • SM120 sparse kernels • DSpark speculative decoding • Built after Jul 31 The Jul 31 release introduced a new MTP norm layout. Older builds fail during weight loading with: KeyError: ...mtp_block.main_norm.weight I'm intentionally not naming the image publicly to avoid sending everyone to the maintainer. Happy to share via DM. --- ## LAUNCH (working configuration) --tensor-parallel-size 2 --max-model-len 524288 --max-num-seqs 4 --gpu-memory-utilization 0.94 --kv-cache-dtype fp8 --speculative-config '{"method":"dspark","num_speculative_tokens":5}' --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --block-size 256 --max-num-batched-tokens 2048 --disable-custom-all-reduce --trust-remote-code (prefix caching disabled) num_speculative_tokens must be >= 5 (the DSpark block size). Anything lower is rejected during config validation before CUDA is even initialized. --- ## THE max_num_seqs LIMIT max_num_seqs > 4 deterministically fails on the first request: cannot reshape tensor of 0 elements Root cause: A hardcoded prefill chunk size (4) feeds an empty padding slice into the sparse MLA kernel. Confirmed on both community builds we tested. Cap max_num_seqs at 4. Additional requests simply queue instead of crashing. --- ## WHAT TO EXPECT Measured on this setup. Code generation (temp=0) • 235-246 tok/s single stream General prose • 130-145 tok/s Aggregate throughput • c2 ~300 tok/s • c4 ~400 tok/s Other numbers • Prefill: 5-6K tok/s • 384K needle retrieval: ~80 s • KV pool: ~988K tokens Why the spread? The engine runs at roughly 50 verification steps/s. Throughput is approximately: tok/s = 50 × (1 + accepted draft tokens per step) Acceptance depends heavily on workload: • Code (temp=0): ~73% • Prose: ~33% Don't estimate throughput from SSE delta counts. Those measure verification steps, not generated tokens. --- ## SANITY LADDER 1. Health check 2. Deterministic text 3. Usage-metered throughput 4. Needle (128K to 384K) 5. Tool calling 6. Confirm the max_num_seqs > 4 signature If all six pass, you should have essentially the same environment I do. Questions welcome.

DeepSeek-V4-Flash-0731 + DSpark speculative decoding 2× RTX PRO 6000 Blackwell (TP=2, official FP8, 524K maxlen) Results (median): • 243 tok/s single stream — 3.1× over the no-spec baseline • Aggregate throughput: c2 299 · c4 403 tok/s • Draft accept: 74–76%, zero failures I expected an SM120 kernel wall. Never hit one. The real bottleneck: max_num_seqs > 4 fails deterministically on the first request. A hardcoded prefill chunk size feeds an empty slice into the sparse kernel. Capped at 4, it held a 3-hour mixed-load soak at 237–245 tok/s. Looks like this will be my daily driver for now.

Correction : the first video was undercounting. My counter was off. DeepSeek-V4-Flash-0731 @ 2× RTX PRO 6000 Blackwell (TP=2) Here's what 244 tok/s actually looks like:

Correction : the first video was undercounting. My counter was off. DeepSeek-V4-Flash-0731 @ 2× RTX PRO 6000 Blackwell (TP=2) Here's what 244 tok/s actually looks like:

Correction : the first video was undercounting. My counter was off. DeepSeek-V4-Flash-0731 @ 2× RTX PRO 6000 Blackwell (TP=2) Here's what 244 tok/s actually looks like: