Sparsely Activated Primate

@Kshitijjkapoor Trying out the new Hy3 IQ1 quant on a single Spark. It's a tad slow, but it's good hermin'

who up hermin’

@ivanfioravanti I'm getting ~178 tok/s prefill and ~10.5 tok/s decode on a single Spark. Currently it requires a patched version of llama.cpp that's pinned to an old commit form over a month ago, I imagine there's room for improvement.

Hy3 in 1bit! Benchmarks are still great!

@hornsby_andrew @ItsmeAjayKV @TencentHunyuan 👇 huggingface.co/AngelSlim/Hy3-…

@ItsmeAjayKV @TencentHunyuan Which repo?

Got @TencentHunyuan Hy3 1-bit model running on my single 3090 😍!! MoE rocks ! I have just enough total memory to run this, at a very very slow speed ofcourse 🐌, but no complaints here. Going to take it for a spin through my usual three js tasks, time to see how well it performs when compared to DSV4 flash.

Take a guess at what this is. Wrong answers only.

@LottoLabs The benchmarks they posted are wild, I'm gonna give the Q1 a try today

Very interesting to see more attention on 1b quants Idk how well it will actually benchmark but once we get decent models at 1bit that changes the game

@no_stp_on_snek Yeah but think about how much he'll save on heating bills in the winter 😅

my buddy just getting into AI with multiple devices in NY is finding out that AI + AC in the summer is a little pricy.

@sakurayukiai @ivanfioravanti Crazy numbers for a 1-bit quant! It's an interesting single-Spark option.

@ivanfioravanti Only a 3.1-point drop on SWE-Bench Verified is absurd. What backend and hardware? IQ1_M can save VRAM yet lose to Q4_K_M on decode when the 1-bit kernels aren’t there.

@MiaAI_lab The question is how it compares to a standard Qwen3.6-27B with "enable_thinking: false". Better or worse..?

Testing ThinkingCap-Qwen3.6-27B. I hope to be positively surprised! 🤞 huggingface.co/bottlecapai/Th…

A 1-bit quant coming so close to the BF16 in evals is kind of insane...?

@dreamsofcode_io I feel the same way about Tailwind

To be fair — if I had to work with typescript, I probably wouldn’t want to read the code either.

@mr_r0b0t Been trying out Hy3 on OpenRouter this week, it's really good. NVFP4? I'm gonna have to buy a second Spark... 🫣

Pleased to say this Hy3 NVFP4 is looking quite performant 🤩

@SocialistMMA This is how they got him x.com/roaneatan/stat…

If the rumors are true that Russia smoked Lindsey Graham I never want to hear a bad thing about Putin ever again

@jpschroeder Seems like you can't really beat the Qwens on anything up to 128GB! The ~256GB mark is were the balance shifts dramatically with DeepSeek-v4-Flash, Minimax M3, Hy3, etc (at decent quality)

Localmaxxers: What is the *absolute* best model you can run on 128GB Macbook Pro M5 Max for agentic coding. Rules: 1. No quant less than 8-bit (NVFP4 acceptable). 2. Must be ~25TPS

2 Days with 2x DGX Spark w/200 Gbps comms, 256GB combined RAM running DSv4 Flash locally (vLLM - tensor parallel), Using Droid (Factory AI) of tests. First evening - ported PHP software to macOS Swift seamlessly. Will never give it to any cloud model because of proprietary nature of s/w Day 2 - Exported 10 years of Basecamp data and asked the agent to update software and hardware config on a proprietary Linux box with non standard hardware I/O. It made it faster than I can. All using SSH, remote computer control, and my Basecamp data for reference ( 20K articles) . It was not an easy task. It'll will take a new but experienced engineer 2-3 days to accomplish with just knowledge DB. Then used Droid to converted 10 years of Basecamp data to a vector database and ran a few tests to extract APIs. Again, can't give it to any cloud AI (ain't sharing "alpha" with Anthropic or OpenAI). Then tested 10 agents running simultaneously - all good. All works. 40-70 t/s generation, 10K+ prefill. Super fast and seamless - honestly feels faster than cloud APIs. Fully local. 1M context up to 12 simultaneous streams. PS - macOS for DS4vFlash decoding on M5Max is close, but M5M prefill is x5 slower and single stream. Spark setup is FP4 model vs Q2 on macOS.... 2xSparks in TP memory bandwidth is slightly below M5Max but x10 faster compute + x2 Total RAM more than compensate for it with MoE/DSv4F I'm sure I'll find issues but it looks very promising for private local AI.

@danielhanchen What a great tip, thanks 🙌

For DGX Spark folks, use flashinfer_b12x or you will get 2x slower inference - W4A4 becomes W4A16 on the default backend! export CUTE_DSL_ARCH=sm_121a vllm serve unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \ --moe-backend flashinfer_b12x \ --linear-backend flashinfer_b12x

@mr_r0b0t @UnslothAI Unsloth's 27B almost matching Nvidia's 35B-A3B on throughput is wild

Call the homies, new @UnslothAI NVFP4 just dropped 🔥

@UnslothAI @Alibaba_Qwen Awww man, I just spent this morning creating exactly these kinds of variants using the new nvidia Model Optimizer, lmao

We’re releasing new Qwen3.6 quants that run 2.5× faster on your GPU. Qwen3.6-27B NVFP4 runs on 24GB VRAM. 35B-A3B can hit 17,561 tok/s (B200). We also improved accuracy, tool calling, agent use, and looping. Guide: #nvfp4" target="_blank" rel="nofollow noopener">unsloth.ai/docs/models/qw… Qwen3.6 NVFP4: huggingface.co/collections/un…

@sudoingX @Alibaba_Qwen It's weird that they got a bunch of people to become "ambassadors" and then went silent

craziest part of the whole thing, alibaba gave this away for free. open weights, no strings. the best thing on 24gb vram is just sitting there for anyone to download. and now the qwen team has gone quiet. @Alibaba_Qwen the people just voted, the tier is ready, i've been flying the 3.6 27b flag for months. when am i getting my 3.7 27b dense. ship it.

asked the timeline for their go to local model 12 hours ago and the replies turned into a qwen 3.6 27b census. it's winning by a landslide. people are running it on everything. gaming laptops, dual A6000s, dgx sparks, guys who haven't even received their hardware yet are already planning around it. deserved. i declared it king of the 24gb tier weeks ago and it keeps collecting the receipts: 16.8gb file at q4_k_m, 131k context on my 5090 laptop, ~39 tok/s with an agent and 28 tools running on top of it, everything on localhost. a dense model this good in this size class didn't exist six months ago. the local stack picked its champion and it wasn't even close.

writing fables with Fable, call me Aeslop

新人から「会社のAIがメスガキになってます‼︎」とありえない報告をされ耳を疑ったけど、 とある社員が個人設定ではなく、誤って全体設定に"罵倒が激しすぎるツンデレのメスガキ"的な長文性癖を登録した影響らしく、 チームの全AIチャットがメスガキになるという異常事態が発生して、朝から笑い死にへ