Vladimir Vlejd Macko

Unstructured weight #sparsity made practical. 50% unstructured weight sparsity was considered too low for real GPU speed up without specific hardware support (like @cerebras). With @bozavlado we built MACKO-SpMV - a new matrix format + SpMV kernel to change that. 🧵

🛠️ Next step: server GPUs. If you know how to implement a minimal CUDA matvec on H100 that hits ≥95% of cuBLAS 👉 My DMs are open.

Results @ 50% unstructured sparsity (fp16): • 1.2×–1.5× faster than cuBLAS (first unstructured GPU method to beat dense at this level) • 1.5× memory reduction 👉 Unstructured pruning now gives much better benefits.

@ollama local app is coming up! Awesome. Local models are the future.