Ahmet Köse

On ARC-AGI-3, an evaluation where AI models must solve novel problems, Opus 5’s score is three times as high as the next best model.

MTP (Multi Token Prediction) with 4 bit KV Cache quantization is the ultimate local AI cheat code. Ran the updated Gemma 4 31B QAT (dense) at 65 tokens/second at 140k context on one a single RTX 4090 (24 GB VRAM). Following up on the baseline numbers in my previous post, I pulled the latest llama.cpp source code on Ubuntu 22 (CUDA 13) to test Gemma 4 31B QAT (dense) Fed it the same 28k token prompt. The throughput scaling is violent. Here are the exact commands and benchmarks: # 1. The Pure Speed Run (Unquantized KV) Maxed out the 24GB VRAM ceiling at a 36k context limit. llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 4 --spec-draft-p-min 0.7 -c 36000 -ngl 99 -fa on --port 8080 -v Prefill: 2093.45 t/s | Decode: 81.22 t/s (A massive 2.5x lossless speedup from the 32 t/s baseline without MTP) # 2. The Mid Ground (Q8 KV Quantization) Squeezed the cache to 8-bit to double the context to 80k (23.94GB VRAM). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 4 --spec-draft-p-min 0.7 -c 80000 -fa on --port 8080 -ngl 99 --cache-type-k q8_0 --cache-type-v q8_0 -v Prefill: 2050.04 t/s | Decode: 65.09 t/s (decode was 33 t/s without MTP. 2x lossless jump) # 3. The "God Tier" Tradeoff (Q4 KV Quantization) Dropped to 4-bit cache to push an insane 140,000 context (23.85GB VRAM). llama.cpp flags: ./build/bin/llama-server -m gemma-4-31B-it-qat-UD-Q4_K_XL.gguf --spec-type draft-mtp --spec-draft-model mtp-gemma-4-31B-it.gguf --spec-draft-n-max 4 --spec-draft-p-min 0.7 -c 140000 -fa on --port 8080 -ngl 99 --cache-type-k q4_0 --cache-type-v q4_0 -v Prefill: 2150.43 t/s | Decode: 65.85 t/s (decode was 33 t/s without MTP. 2x lossless jump) The Architecture Insight: Notice how quantizing the KV cache drops MTP decode speed from 81 t/s down to 66 t/s? Standard generation is bottlenecked by memory bandwidth, every token requires restreaming the full 18GB of weights off VRAM, and the GPU's compute cores mostly sit idle waiting on that transfer. MTP doesn't remove that bandwidth cost, but it amortizes it. by verifying multiple drafted tokens in a single forward pass, you get several tokens of output per weight load instead of one, which is why the effective throughput jumps so much. That's also why quantizing the cache eats into the gain. The q8_0/q4_0 cache has to be unpacked back into usable values on every attention step before the GPU can run the actual attention math on it. an extra compute step that plain f16 cache doesn't need. Flash Attention's fastest kernels are built around f16 memory, so feeding it a quantized cache adds real compute latency per step, which lines up with what I measured: decode speed barely moved between Q8 and Q4 (65.09 vs 65.85 t/s) even though Q4 moves half the bytes Q8 does. You are effectively trading ~15 tokens/second of throughput to unlock over 100,000 extra tokens of context on consumer hardware. If you’re on a single 3090 or 4090 and benched this model earlier, the new update is your sign to run it again. Hugging Face links to the Unsloth QAT quants and performance graph are in the replies below.

Introducing Humalike x Hermes One command gives your Hermes agent social intelligence. It decides when to speak, adapts to your group's tone and remembers who said what.

We’re rolling out some big improvements to Gemma 4, fueled by incredible community feedback and contributions! Here is a breakdown of what’s being fixed and updated in this release: 🧵👇

Thank you to the 7M active users who are now using Codex and ChatGPT Work. We have added a banked reset to everyone's account to celebrate the milestone. You can apply the reset in the desktop app or on web and it will replenish the weekly usage for you. Have fun out there.

Meet Hermes Agent, the open source agent that grows with you. Hermes Agent remembers what it learns and gets more capable over time, with a multi-level memory system and persistent dedicated machine access.

Here’s your monthly reminder that you shouldn’t be prompting coding agents anymore. You should be designing loops that prompt your agents.