Joe Muller

howtospark . com is live! I'm treating it as a living notebook for my Spark experiments and have a bunch of things I want to add that will help Spark owners hit the ground running

@Stumblinz Here are some good recipes, look at the Deepseek V4 Flash REAP one x.com/MiaAI_lab/stat…

Can you test with Deepseek V4 Flash in some sort of Nvidia optimized/ quantized version and lmk what tokens you’re getting back? I have one AMD system I’m finding is less than useful with the 128GB solo And two sparks I’m waiting on a connector for; curious if it will be useful Sentdex on here and YouTube swears by Deepseek V4 flash and said he was able to stop making cloud API calls. I’m dubious after my experience with Qwen3.6-35B. Tbh short of GLM 5.2 running at about 100t/s or similar LLM accuracy at 70t/s I can’t see myself using local AI calls. At least not with the heavy harnesses of today. Claude app works so well as a harness and it’s LLMs. Wondering if instead of generalizing models, find specialties and route per specialty. Haven’t found an example of this working yet though and it’s not a domain of knowledge I know first hand.

Played a few tricks and tripled the speed of GLM 5.2 on 2 DGX Sparks 🔥 Old: 4.1 tok/sec New: 13.8 tok/sec 2-bit quantization, TP2, and a frequency based prune of the experts to get planes under the ~88 GB page-cache ceiling

@Stumblinz I think this is the go-to recipe for Deepseek v4 Flash on 2 sparks but I'd need to dig a little deeper for the best solo spark setup x.com/MiaAI_lab/stat…

13K tok/sec @ 128 concurrency That's about 100 tok/sec/user

@UnslothAI @NVIDIAAI @googlegemma Super impressive, at 128 concurrency that is about 100 tok/sec/user

We’re releasing Gemma 4 NVFP4 quants that run 1.5× faster on your GPU. Gemma-4-12B NVFP4 works on 11GB VRAM. 26B-A4B hits 13K tok/s (B200). Unsloth NVFP4 enables faster, more accurate 4-bit Blackwell inference. Blog: unsloth.ai/docs/basics/nv… Gemma NVFP4: huggingface.co/collections/un…

@thatcofffeeguy @Zai_org @NVIDIAAI The world would be changed 🌍

Breaking !! If GLM 5.5 is a real thing @Zai_org and it’s 1t plus 1.5 context tokens. @NVIDIAAI I am going to need some more compute. It’s time to start the builds.

@yume_arasaki Never ask about quality loss, just keep the tokens flowing

@BosonJoe Isn't 2 Bit got a bit of quality loss?

GLM 5.2 now pushing past 18 tok/sec on 2 DGX Sparks 🔥🔥🔥 This time I quantized the 3 biggest attention projections to NVFP4 (compared to the original FP8) There are still wins to be had on the speed and quality fronts so stay tuned

@franciscojarceo 🏔️🚶‍♂️

@BosonJoe are you hill climbing sir?

@peterjfoti More to come! 30 tok/sec is the ultimate aim. If I can get to that without losing much intelligence, I will be a happy camper

@BosonJoe Love the progress, thanks for sharing

@sdfsdfgsdfggfd @AnthropicAI Fable-at-home

@BosonJoe @AnthropicAI wtf are you building ?

Can I have more Fable @AnthropicAI

Awesome visual from the latest @NVIDIAAI blog x.com/NVIDIAAI/statu…

@StefanMaier Definitely, you can track my progress here in the spark folder github.com/Sapid-Labs/vLL…

@BosonJoe sounds awesome! Do you mind to publish a recipe when everything works as you expect?

@StefanMaier The next iteration should give me like 21-22 tok/sec Next step after that is to create a draft model tailored to this super-quant 😅 Hoping that puts me near 25-28 Then I'm going to swap out my frequency prune step with a real REAP to buy back some quality

@BosonJoe wow this is massive! What gains in speed do you expect?

@ivanfioravanti HOW?!

Hy3 in 1bit! Benchmarks are still great!

@NVIDIAAP 4 sparks is the dream...it's basically like buying a new car

One DGX Spark on your desk is powerful. Four is a local AI factory. NVIDIA DGX Spark now scales up to four clustered nodes, enabling inference on models up to 700B parameters — with near-linear performance scaling for fine-tuning and reinforcement learning workloads. Memory scales from 128GB on a single node all the way up for the full four-node configuration, supporting large context windows, high concurrency, and multi-agent workflows. For APAC enterprise teams building autonomous AI: each topology is purpose-built for your workload — from rapid local iteration to production-scale inference. And when you're ready to move to the cloud, Tile IR and cuTile Python carry your kernels across with minimal code changes. No rack. No rearchitecting. Just scale. 🔗nvda.ws/4w0RBPy

@jun_song Check it out, making progress on this here: x.com/BosonJoe/statu…

Wiring up 4 DGX Sparks is a massive headache and requires an extra $2,000 just for the connectors. I'm going to work on getting GLM5.2 (744b) to run on just 2 nodes instead.

@Stumblinz 100%, that is the ideal setup and you'd hardly have to sacrifice quality On 2 it is a much tighter squeeze so you lose speed and quality

@BosonJoe Hey man does it seem like 4 could run it fully when clustered?

@Hanger8200 The 8k is an investment in myself -> experiment -> learn a skill that will be critical -> make back my investment x 100

@BosonJoe grok, claude,sol all on super mega max! I think if you do go to emptyheadai.com you will see the point of the name and how could people just buy in to the hype so hard, it's more the point of it. just think you just blasted 8k on something that's slower then a $20 sub?

Run MiMo-V2.5 on 2× DGX Sparks, Full OMNI, Hassle-free! ✨ 1M context · 3 concurrent sessions ~31 tok/s → single, ~56 tok/s → 3 sessions Full Omni = text, images, video, and audio on the same endpoint. The more I use this model the more I love it. → github.com/MiaAI-Lab/MiMo…

@0xSero For reference, Opus 4.8 scores 93.6 on GPQA Diamond

This is going to be the next dominant standard for sub 4 bit compression Hugging reward them huggingface.co/madeby561/GLM-…