Sheik

My FrankenPotato AI Box I have assembled - Ryzen 5 5600x - 64GB of RAM (if only it was VRAM...) - MSI 3080TI 12gb - MSI 3060TI 8gb I hope to get something more standardised and significant in the future, but at the moment, this janked collection of parts is somehow managing to achieve really impressive intelligence, speed, and a just barely acceptable context window.

@sudoingX You have me waiting on the edge of my seat for a Q4 vs Q1 model comparison.

every day someone asks how i'm running bonsai 27b on hardware that shouldn't handle it. so here's the whole thing in one place, save this. the model is a 1bit quant of qwen 3.6 27b, under 4gb on disk, and here's the catch nobody mentions: stock llama.cpp can't load it. the 1bit kernels only live in @PrismML's fork. build that or the weights won't even open. github.com/PrismML-Eng/ll…, cuda on for your arch (sm_86 for a 3060 ti, sm_75 for a 1660 super). then serve it. this is the exact command i run for the 128k agent on 8gb: llama-server -m Bonsai-27B-Q1_0.gguf -ngl 99 -c 131072 -np 1 -fa --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99 puts every layer on the gpu, -c 131072 is the full 128k window, -fa is flash attention, and the q4 kv cache is what holds it at 6.8 of 8 gigs instead of oom. drop -c to 65536 if your card shares vram with a display. that's the whole trick. one fork, one command, and a 27b runs local. bookmark it, you'll want it when you dust that gpu off.

We’re releasing Gemma 4 NVFP4 quants that run 1.5× faster on your GPU. Gemma-4-12B NVFP4 works on 11GB VRAM. 26B-A4B hits 13K tok/s (B200). Unsloth NVFP4 enables faster, more accurate 4-bit Blackwell inference. Blog: unsloth.ai/docs/basics/nv… Gemma NVFP4: huggingface.co/collections/un…

@sudoingX Amen. The Lord giveth and the Lord taketh away... Unless you have the whole model on local storage instead of just hitting an endpoint.

i don't care what you have to do. buy and just fill your nvme with open weights right now while the door is still open. qwen, gemma, deepseek, the ring models, the llama checkpoints, all of it, pull it tonight. i'm not being dramatic, i watched a frontier model get yanked dark for 19 days by a government order three weeks ago and nobody got a vote. these things live on one company's servers and every single one is one policy change, one acquisition, one export rule away from vanishing on you. a model on your drive can never be taken back. that's the whole thing. that's the only ownership that survives a bad monday. weights are cheap right now and disk is cheaper. the window where you can just download the frontier for free is not going to stay open forever, nothing this good ever does. stop reading this. go pull the ones you'd cry about losing. do it before you sleep.

Last weekend, I started building my first project. Fable was the motivator and boy am I glad I did. It will eventually be Open Source in the OCR space. I'm solving a problem I'm sick of having at every organisation I work in. I'm looking at a minimum viable product a week later thanks to Fables availability being extended. Now I've got Grok 4.5 for when it goes away. We live in crazy times.

@TraffAlex Have spent many hours tweaking these flags.... To the point where I'm motivated to create a skill to do it for me.

yeah I only figured out the tensor-split stuff recently too. you can play with those ratios quite a bit, the key is they always have to sum to 1. so 0.55/0.45, 0.60/0.40, 0.70/0.30, whatever matches your VRAM split. also the q4_0 KV cache makes a huge difference for fitting bigger models, worth trying those flags if you haven't already

🖥️ Best Local LLMs for Consumer GPUs — llama.cpp Guide (June 2026) What I actually run on consumer hardware right now. Every model below runs via llama.cpp with a simple one-liner — no Docker, no Python env, no cloud. ━━━ 8-16GB VRAM ━━━ 🔹 Gemma 4-12B (Google) • Smartest model in this size class — competes with stuff 2× bigger • Unsloth's MTP GGUFs: 162 tok/s vs 52 tok/s normal (3× speedup) • Minimum 8GB VRAM recommended for Q4_K_M quant • GGUF → huggingface.co/unsloth/gemma-… 🔹 LFM2.5-8B-A1B (LiquidAI) • Hybrid MoE, only 1B active params — absurdly fast for its size • Perfect for 8-12GB cards, MacBooks, or anyone on a tight budget • GGUF → huggingface.co/LiquidAI/LFM2.… ━━━ 16-32GB VRAM ━━━ 🔹 Qwen3.6-27B (Qwen) • Scored 1.00 on tool-efficiency benchmarks — best local agent available • 40 deterministic tasks, 32k/128k context needle tests — all passed • GGUF → huggingface.co/unsloth/Qwen3.… • MTP version (faster) → huggingface.co/unsloth/Qwen3.… 🔹 Qwopus3.6-27B-v2 (Jackrong) • Best quantization of Qwen3.6-27B — topped 5 agent & coding benchmarks (1200 samples) • If you're running Q4, this is the one to grab • GGUF → huggingface.co/Jackrong/Qwopu… • MTP version → huggingface.co/Jackrong/Qwopu… 🔹 Gemma 4-31B QAT (Google/Unsloth) • QAT variant with MTP draft head: 76-125 tok/s (1.67× speedup) • Excellent for multi-agent / subagent workflows • GGUF → huggingface.co/unsloth/gemma-… 🔹 Nex-N2-Mini (Nex AGI) • Post-train of Qwen3.5-35B-A3B — MoE with only 3B active params • Fits on 16GB+ VRAM, overflow loads from system RAM • Adaptive thinking saves ~20% tokens with no quality loss • For deep multi-step reasoning, nothing in this size comes close • GGUF → huggingface.co/sjakek/Nex-N2-… ━━━ Quick Picks ━━━ • 16GB all-rounder → Gemma 4-12B with MTP GGUFs • 32GB all-rounder → Qwen3.6-27B / Qwopus-v2 • Agents & tool use → Qwen3.6-27B or Qwopus Q4 • Deep reasoning → Nex-N2-Mini (MoE, fits 16GB+) • Tight budget → LFM2.5-8B-A1B • Cheapest full build: 1× used RTX 3090 (24GB) + rest of PC ≈ $1000-1500 ━━━ Setup on Windows ━━━ 1. Download llama.cpp → github.com/ggml-org/llama… (latest .zip) 2. Extract to any folder (e.g. C:\llama.cpp) 3. Download a .gguf from the links above (Q4_K_M or Q5_K_M for best quality/speed balance) 4. Run one of the commands below depending on your hardware ━━━ Launch Commands ━━━ SINGLE GPU — Standard model (no MTP): llama-server.exe ^ -m C:\models\Qwen3.6-27B-Q5_K_M.gguf ^ --ctx-size 180000 ^ --flash-attn on ^ --cache-type-k q4_0 ^ --cache-type-v q4_0 ^ --batch-size 1024 --ubatch-size 512 ^ -ngl 100 ^ -np 1 ^ --port 8080 ^ --jinja SINGLE GPU — MTP model (faster inference): llama-server.exe ^ -m C:\models\Qwen3.6-27B-MTP-Q5_K_M.gguf ^ --ctx-size 180000 ^ --flash-attn on ^ --cache-type-k q4_0 ^ --cache-type-v q4_0 ^ --batch-size 1024 --ubatch-size 512 ^ --spec-type draft-mtp ^ --spec-draft-n-max 3 ^ -ngl 100 ^ -np 1 ^ --port 8080 ^ --jinja DUAL GPU — Split across two cards: llama-server.exe ^ -m C:\models\Qwen3.6-27B-Q5_K_M.gguf ^ --ctx-size 180000 ^ --flash-attn on ^ --cache-type-k q4_0 ^ --cache-type-v q4_0 ^ --batch-size 1024 --ubatch-size 512 ^ -ngl 100 ^ --tensor-split 0.55,0.45 ^ --main-gpu 0 ^ -np 1 ^ --port 8080 ^ --jinja DUAL GPU + MTP + Vision (multimodal): llama-server.exe ^ -m C:\models\Qwen3.6-27B-MTP-Q5_K_M.gguf ^ --ctx-size 180000 ^ --flash-attn on ^ --cache-type-k q4_0 ^ --cache-type-v q4_0 ^ --batch-size 1024 --ubatch-size 512 ^ --spec-type draft-mtp ^ --spec-draft-n-max 3 ^ -ngl 100 ^ --tensor-split 0.60,0.40 ^ --main-gpu 0 ^ -np 1 ^ --port 8080 ^ --jinja ^ --mmproj C:\models\mmproj-F16.gguf ━━━ Parameter Breakdown ━━━ -m Path to your .gguf model file. Change this to wherever you downloaded it. --ctx-size 180000 Context window in tokens. 180k = huge context for long conversations or big codebases. Reduce to 32768 or 65536 if you don't need long context — uses less VRAM. --flash-attn on Flash Attention — dramatically speeds up inference and reduces VRAM usage. Works on RTX 30xx/40xx/50xx. Always enable this. --cache-type-k q4_0 / --cache-type-v q4_0 Quantizes the KV cache (key/value attention cache) to 4-bit. This is what makes 180k context fit in VRAM. Without it, huge contexts eat all your memory. Quality impact is minimal — this is a free performance win. --batch-size 1024 / --ubatch-size 512 batch-size = how many tokens are processed in one forward pass (throughput). ubatch-size = micro-batch actually sent to the GPU per step. Higher = faster prompt processing but needs more VRAM. If you run out of VRAM, lower these (e.g. 512/256). -ngl 100 Number of layers to offload to GPU. 100 = all layers on GPU (full offload). This is what you want if the model fits in your VRAM. If it doesn't fit, reduce this (e.g. -ngl 40) — remaining layers run on CPU/RAM. --tensor-split 0.55,0.45 How to split model layers across multiple GPUs. Values are ratios. 0.55,0.45 = GPU 0 gets 55% of layers, GPU 1 gets 45%. Adjust based on your VRAM — give more to the card with more memory. Example: 0.70,0.30 for a 24GB + 12GB setup. Not needed for single GPU setups. --main-gpu 0 Which GPU handles the batch computation (the "orchestrator"). Set to 0 (your primary GPU). The other GPU(s) handle their assigned layers. Minor performance impact — usually just leave it at 0. -np 1 Number of parallel slots (concurrent requests). 1 = one user at a time. Increase to 2-4 if you want multiple clients connected simultaneously. Each extra slot uses additional VRAM for its own KV cache. --port 8080 Which port the server listens on. Change if port 8080 is busy. --jinja Enables Jinja2 template processing — required for proper chat formatting. Most modern models expect this. Always include it. --spec-type draft-mtp Enables Multi-Token Prediction (MTP) speculative decoding. Only works with MTP GGUF models (downloaded separately). The model predicts multiple tokens at once and verifies them — big speed boost. --spec-draft-n-max 3 How many tokens the MTP draft head proposes per step. 3 is a good default. Higher = potentially faster but more VRAM and may reduce quality. --mmproj Path to the multimodal projector file (for vision models). Enables image understanding — paste screenshots into the web chat. Only needed if you want vision capabilities. Omit for text-only use. ━━━ Your Hardware → Your Command ━━━ Single GPU (8-24GB VRAM): Use the "Single GPU" command. Change -m to your model path. 8GB card → Gemma 4-12B Q4 or LFM2.5-8B 12GB card → Gemma 4-12B Q5/Q6 16GB card → Gemma 4-31B QAT Q4 or Nex-N2-Mini 24GB card → Qwen3.6-27B Q4/Q5, Qwopus-v2, Gemma 4-31B QAT Q5/Q6 Dual GPU: Use the "Dual GPU" command. Adjust --tensor-split based on your VRAM ratio. 24GB + 24GB → --tensor-split 0.50,0.50 24GB + 12GB → --tensor-split 0.70,0.30 24GB + 8GB → --tensor-split 0.75,0.25 Want speed? Use MTP versions of models with the "MTP" commands. Want vision? Add --mmproj with the projector file from the model's HuggingFace repo. 5. Once running, you get: • Web chat UI → http://localhost:8080 • OpenAI-compatible API → http://localhost:8080/v1 • Playground → http://localhost:8080/playground ━━━ Why /v1 API Is the Killer Feature ━━━ One local endpoint replaces your entire cloud API bill. The /v1 endpoint is drop-in OpenAI-spec compatible — every tool that speaks OpenAI just works. No custom code, no glue layer. Works out of the box with: • IDEs: Cursor, Continue, Windsurf, Cline, Roo Code • CLI tools: aider, Open Interpreter, OpenCode • Frameworks: LangChain, LlamaIndex, LiteLLM • Any OpenAI SDK (Python, Node, Go, Rust) Why this beats cloud APIs: • 100% private — code never leaves your machine • $0 per token — no rate limits, no quotas, no surprise bills • Works fully offline • Zero telemetry, no training on your data • Swap models by dropping in a different .gguf — no app changes needed • Run 32k–128k context windows without burning money Good combos: • Cursor + Qwopus-v2 → near-frontier quality, zero API cost • Continue + Qwen3.6-27B → best local coding agent • aider + Gemma 4-12B MTP → 162 tok/s, feels instant • OpenCode + Nex-N2-Mini → deep reasoning on 16GB Set any OpenAI-compatible client to your local endpoint: set OPENAI_API_KEY=sk-dummy (any non-empty string works) set OPENAI_BASE_URL=http://localhost:8080/v1 # every OpenAI-compatible tool now hits your local GPU Shoutouts: @0xSero @rS_alonewolf @witcheer @UnslothAI @LottoLabs

@elder_plinius Last time you got it pulled before I got to play with it on the weekend... Are you sure you don't want a holiday? 🤣🤣

🙊

@0xSero Thanks for inspiring me with your posts and ferocious goblining of inference hardware

I’ve had many coworkers, peers, mentors, and friends. My life would not be as good as it is without them. If you’re reading this I wanted to say, thank you.

@ellandell @0xSero Really? I'll have to look deeper. I didn't think that was a thing

@_Sheik_ @0xSero You can get heavily quantised safetensor models :)

Why is llama.cpp so much slower than vLLM at long context multi-turn tasks, and at high concurrency.

@sudoingX Really keen to see and hear what "minimum acceptable hardware" is for this. Its probably going to be my acquisition target after I sell everything I own for inference hardware. 😂

honest catch, my box can't hold this one on its own, it's that big. so i'm going to try connecting two together and splitting it across both. no clue if it'll hold, and that's exactly why i want to try. receipts either way.

i keep coming back to this one chart. the blue line is the open model. free, MIT, sitting on huggingface as you read this. and it's tangled right up in the closed frontier, same axes as claude opus, trading places depending on how hard you let it think. we were told for years the good models live behind someone else's login, someone else's terms, someone else's price. that line is just gone now. the frontier has an open copy, and nobody can take it back. if you build on local ai, sit with that a second. it's a bigger day than the timeline is treating it.

@jtregunna @0xSero I'm running -np 1 and context maxing... No concurrency going on here. Haha

@_Sheik_ @0xSero at low concurrency, they're very close... at high concurrency, it's not even a contest.

Has anyone managed to get Gemma4-12b to run in an agent harness without experiencing looping? Does anyone have a multi-gpu config for llama-server for it that doesn't play up? Is it only me having shocking luck with it?

Running Gemma4-26b-A4B on Q3 with an absolutely staggering 95% VRAM utilisation on my FrankenPotato. The speed I would get from this model wasn't cutting it, but now with MTP Im getting from 65-100tok/s and 128k context. Config as such: -m ~/gemma-4-26B-A4B-it-UD-Q3_K_XL.gguf \ --model-draft ~/mtp-gemma-4-26B-A4B-it.gguf \ --mmproj ~/Gemma4-26b-a4b-unsloth/mmproj.gguf \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-ngl 99 \ --spec-draft-type-k q4_0 \ --spec-draft-type-v q4_0 \ -ngl 99 \ -c 131072 \ -b 512 \ -ub 512 \ -ctk q4_0 -ctv q4_0 \ -np 1 \ -fa on \ --jinja \ --reasoning on \ --repeat-penalty 1.12 \ --presence-penalty 0.1 \ --metrics \ --temp 1.0 --top-p 0.95 --top-k 64 Tweaked till its nipples bleed... I was hoping to get a Q4 version to fit, but it required sacrificing too much context. Hermes cant breathe under 128k. Its a stretch at that from my experience.

Spent 5 hours today trying to get Gemma4-12b running on my FrankenPotato 3080TI+3060TI machine. Didn't seem to matter what configuration I threw at it, I was getting prompt loops. Pretty devastated.

@0xSero Amen. Honestly amazed open models weren't driven underground sooner than this. Making hay while the sun shines.

If you have the NVMe Go download as many models as you think you might ever want. Now, go on Huggingface. They’re coming for open models next.

@0xSero Loving these posts and your advocacy for local. I cant keep track of things myself, so this gives me chance of playing with the right models when I do have time.

State of Local AI #1 ——— In lieu of Fable ban. Here’s the best LLMs of the week to run on your hardware. —— 4-8gb vram/ram 500$ - Gemma-4-qat huggingface.co/unsloth/gemma-… I had someone mention it’s very good for subagent stuff —— 8-16gb vram/ram < 1k usd - Gemma-12B huggingface.co/google/gemma-4… without a doubt the smartest model of its size —— 16-32gb Apple/Strix halo 1-2k usd - Diffusion Gemma26B huggingface.co/google/diffusi… - on 1x 6000 it’s eating up to 600 tok/s - smallest smart MoE we have - lots of world knowledge - easy to run —— 32-96gb ram/vram (2-10k usd) - nex-n2-mini huggingface.co/nex-agi/Nex-N2… builds on qwen3.6-35B and seems to do really well - qwopus-27B huggingface.co/Jackrong/Qwopu… this model topped a lot of our benchmarks at local.ai —— 384gb vram (10-50K usd) - huggingface.co/MiniMaxAI/Mini… 23B means it’s close to qwen3.6-27B per token, while also have a lot of specialisation. - fast inference - top open weight model on AA —— 768gb-1TB - huggingface.co/moonshotai/Kim… Kimi has always been a top player here and their last model cuts speed and cost down by 30% - great vision support - first coder model by moonshot ——— Top models: 1. Qwen3.6-35B 2. Qwen3.6-27B 3. Step-3.7-Flash 4. Minimax-M3 5. Deepseek-v4-flash ——— Budget sweet spots: #1 - 1K usd Single 3090 / Mac mini / Intel arc b70 / AMD - Qwen / Gemma #2 - 5k usd DGX Spark / Mac m5 max / 4x 3090 - qwen / Gemma step and deepseek flash #3 - 12k usd RTX Pro 6000 / Mac Ultra / 2x Spark / 8x 3090 Ds4-flash / step-3.7-Flash and above #4 - 24k usd 2x 6000 / 2x Mac Ultra / 4x Spark / Mix Same as above #5 - 50k usd 4x 6000 / 4x Max Ultra / 12x Spark / 2 H100 Minimax-m3 / nex-n2-pro / step-3.7-flash #6 - 100k usd GB300 station / 8x 6000 / 4x H200 / Mix GLM-5.2 / Kimi-K2.7 ——— Let’s keep the Internet free thanks for reading

Congrats on your success hero. That 'trend' you latched onto was exactly what I was looking for. Keep up the great work. I just took the plunge, partially inspired by your posts. More the content and the connection posts - not so much the monetisation ones, but they were very interesting and inspiring to see. I dont expect I'll make much traction there, but I want to engage with people that are working on this stuff and work on it myself where I can.