tyler bonnen
588 posts
@tylerraye
neuroscientist @berkeley_ai. NIH K00 + UC Presidential Postdoctoral Fellow
Here’s something we’ve never seen done before. Real-world tasks are long and ambiguous. Solving them requires visual memory and state tracking. Most robot policies only see the last few frames. Ours doesn't. We put our DVA, FutureVision, to the perfect testbed: the shell game 🐚. The DVA nails it.
The missing half of the neural network–brain comparison For a decade, the standard benchmark for artificial neural networks as models of the brain has been forward predictivity: learn a linear mapping from model activations to neural recordings and measure explained variance. Top models of the macaque inferior temporal (IT) cortex—central to object recognition—have plateaued near 50% regardless of architecture. Muzellec and Kar argue this plateau hides something important. Two models can score identically on forward predictivity while relying on fundamentally different internal strategies. One may have many units tightly coupled to IT responses; the other may reach the same score with a smaller aligned subset while carrying a large pool of biologically inaccessible dimensions. To expose this, they introduce reverse predictivity: instead of asking how well model features predict neurons, they ask how well IT neurons predict individual model units. A truly brain-like model should be bidirectionally predictable—just as two monkeys' IT populations predict each other symmetrically, which the authors confirm as their empirical baseline. Across 39 architectures—CNNs, transformers, self-supervised and robust models—reverse predictivity is consistently lower than forward predictivity and the two metrics are uncorrelated. Strikingly, higher ImageNet accuracy predicts lower reverse predictivity. Adversarial training helps; higher dimensionality hurts. The "common" units identified this way predict primate behavior more consistently across species and models than the "unique" ones inaccessible from neural activity. For AI in drug discovery, neurotechnology, or computational biology, this has a direct implication: forward accuracy alone does not guarantee that a model's internal representations are embedded in the biological system it claims to describe. When those representations guide mechanistic interpretations or experimental decisions, the mismatch can mislead. Paper: Muzellec et al., Nature Machine Intelligence (2026) | nature.com/articles/s4225…
Humans can see in high-res, high-FPS in real-time. Why can't VLMs? Introducing AutoGaze: ViTs/VLMs "gaze" only at key video regions! Up to 4-100x token savings, 19x speedup, and enables scaling to 4K-res 1K-frame videos. 📄 arxiv.org/abs/2603.12254 🌐 autogaze.github.io 🤗 huggingface.co/collections/bf… (1/n)🧵
