Espen JD
3.8K posts
Espen JD
@Snixtp
Cyber Network Engineer | Codex enthusiast | Local AI | RTX Pro 6000 enjoyer
Katılım Haziran 2020
656 Takip Edilen440 Takipçiler
We are confident GPU-accelerated signal processing is the future of radio astronomy. Our Stelline Developer Kit, based on @NVIDIAAI DGX Spark, lets us develop compute and networking capabilities locally before deploying to observatories.
First units headed to scientists now!
English
Thats beyond the scope of this little test. Ideally you'd have many of these in a server to server many people. I think both you and I know it can't do cc96 with 128k ctx.
On paper, if my math is right, cc3 looks to be possible if you use FP8 KV cache. Speed? 35-55 tok/s wouldn't surprise me
English
RTX Pro 6000 Workstation Edition efficiency numbers
Models:
- Qwen3.6 27B BF16
- Qwen3.5 122B REAP
Just like with the 3090, best efficiency is 250W
Optimal power I would still say is 350W, speed gains are limited after that point.
Toni S@VoltableCom
People often benchmark with quantization and similar settings, but I tested something different. On an RTX Pro 6000 Blackwell, I ran the same prompt at different power limits using Qwen 3.6 27B. 150W is the minimum power limit for this card. 150W - 42 t/s 200W - 68 t/s 250W - 98 t/s 300W - 108 t/s 350W - 113 t/s Above 350W, the tokens per second stay about the same regardless of the power limit. The maximum power this card uses with this AI model is around 450W. Conclusion: test for power too and power-limit your card. Past a certain point, extra power is just wasted energy.
English
CUDA_VISIBLE_DEVICES=0 \
CUDA_DEVICE_ORDER=PCI_BUS_ID \
HF_HOME=/LINUX/Models/hf \
vllm serve /LINUX/Models/unsloth/Qwen3.6-27B-NVFP4 \
--host 127.0.0.1 \
--port 8000 \
--served-model-name qwen36-nvfp4-pro6000-cprofile \
--max-model-len 64000 \
--gpu-memory-utilization 0.90 \
--trust-remote-code
Català
Spent $3998.98 total to have 96gb of VRAM using AMD's AI Pro R9700 Cards. (brand new)
Comparatively I had spent $1520.00 on two used RX 7900 XTX's for 48gb of VRAM.
If ur team RED, a single XTX is CHEAPER than
a RTX 3090.
Should I have bought a Mac or DGX Spark instead?
English
@Snixtp thanks.
any luck on your end running nvfp4 quants? i've gotten effectively similar performance on qwen3.6 35b but i think i'm not using the most up to date kernels...
English
@PranshuBahadur @HCColenbrander Yes, a lot. I even tried cc16 and 32 but the latency got so bad because there just weren’t enough room on the card for kv cache
English
@Snixtp @HCColenbrander hmm, very interesting did you also notice the toks / sec ratio drop at 10 users tho
English
@PranshuBahadur Its all just codex orchestrating the testing and watching each step
I uploaded this a last week
Would highly recommend giving it to your agent and ask it to set it up
github.com/net-snix/rtx-p…
English
@Snixtp ohh so this was vllm i presume? should be similar i think (especially for decode at 96, unless sglang doesnt permute); oh wow you even have the energy consumption! is there anyway you can produce energy consumption values for H100? or is your repo / script public so we can try?
English
@Snixtp I'm too paranoid with that dang connector I ran mine at 600 for a bit just to make sure they didn't blow up in the first week of owning them (5090s) but after that I powered limited mine permanently to 460w funny that we both ended up at around the same number! I need a 6000😂
English
I also tested concurrency 96 on the RTX Pro 6000.
Efficiency depends a lot on the workload. My previous test was basically single user chat inference, with cc=1.
Max power draw during that test was 434W, thats why the graph is flat from that point onwards.
But during this test, the GPU is being pushed much harder, and able to hit the target of 600W
For Qwen3.6 27B at c96, peak efficiency is ~275W. Not very different from the cc1 test, which was ~250W
After that, total output keeps increasing, but efficiency starts dropping.
Personally, I can take the extra heat and cost, and I see no problem running it at 450W
Espen JD@Snixtp
RTX Pro 6000 Workstation Edition efficiency numbers Models: - Qwen3.6 27B BF16 - Qwen3.5 122B REAP Just like with the 3090, best efficiency is 250W Optimal power I would still say is 350W, speed gains are limited after that point.
English
@PranshuBahadur Here is NVFP4, its a little faster, but that is because it’s a more aggressive quant than fp8
I’m testing it with sglang later today, that should improve cc tok/s
English
@Snixtp is this good? idk how to tell 😅 it's a "fp8 27B" no? should try nvfp4 it'll be a lot fast i think...also what's the decode batch_size?
English