Nick Alonso

Zyphra Online Vector Quantized Attention OVQ-attention keeps linear time and constant memory but avoids long-context collapse by learning both key and value centroids online, so memory tracks the live KV stream instead of a fixed dictionary. Sparse updates route each token to a single slot, so memory capacity scales without increasing per-token compute. Based on Gaussian Mixture Regression with online EM-style updates, it outperforms VQ and linear baselines, generalizes from ~4k training context to 64k+, and stays competitive with attention using ~10–25% of the state; still early at sub-500M scale and not kernel-optimized.

Zyphra Online Vector Quantized Attention OVQ-attention keeps linear time and constant memory but avoids long-context collapse by learning both key and value centroids online, so memory tracks the live KV stream instead of a fixed dictionary. Sparse updates route each token to a single slot, so memory capacity scales without increasing per-token compute. Based on Gaussian Mixture Regression with online EM-style updates, it outperforms VQ and linear baselines, generalizes from ~4k training context to 64k+, and stays competitive with attention using ~10–25% of the state; still early at sub-500M scale and not kernel-optimized.

An exciting new approach for doing continual learning, using nested optimization for enhancing long context processing.

seems big AI labs are hyperfixating on reasoning when they should focus on *memory* instead normal people won't use models that can think for hours to solve hard math problems people want models that learn over time, remember details, adapt and interact like a person would