Farnoosh Hashemi

Moving from conventional ML to continual learning requires revisiting even the fundamental concepts such as “test”/“train” time. LLMs Need Sleep and Dreaming! We introduce a phase, where the model consolidates its fragile short-term memories into stable long-term memories, and then dreams to recursively self-improve over time. For memory consolidation, we introduce a new form of distillation, called Knowledge Seeding (KS), where a small model(s) distills its knowledge to a larger model. Our experiments on continual learning and reasoning tasks show that this new phase can help the model to perform better and relatively better mitigates catastrophic forgetting.

@behrouz_ali @mirrokni @AdelJavanmard Here is also the link to alphaXiv: alphaxiv.org/abs/2606.03979

Link to the paper: arxiv.org/abs/2606.03979 This is a joint work with @behrouz_ali, @AdelJavanmard, and @mirrokni. Also, see Ali's post for more details: x.com/behrouz_ali/st…

Order matters: consolidate first, then dream. Iterating self-distillation on a fixed model can erode prior capabilities. Sleep's two-stage design addresses this, capacity grows and knowledge consolidates before the model starts modifying itself.

As we move from conventional machine learning toward models that continually learn, we need to revisit some fundamental assumptions, starting with "training time" and "test time." A continual learner doesn't have “test” or “train” time. So what does its lifecycle look like?

The growing KV-cache of attention is the key component for the long-context understanding of LLMs, but what holds back long-term memory modules (e.g., Titans)? What if we could have the compression power of Titans but with a growing memory similar to Transformers? Memory Caching: A class of architectures that compress the context into a slow growing memory (not as fast as Transformers, but not as static as RNNs), resulting in recurrent neural networks with non-fixed-sized memory (hidden states). Building on this formulation, we present Sparse Selective Caching, an architecture with growing effective memory (similar to attention) but with almost constant inference cost per token (similar to RNNs).

This paper is the same as the DeepCrossAttention (DCA) method from more than a year ago: arxiv.org/abs/2502.06785. As far as I understood, here there is no innovation to be excited about, and yet surprisingly there is no citation and discussion about DCA! The level of redundancy in LLM research and then the hype on X is getting worse and worse! DeepCrossAttention is built based on the intuition that depth-wise cross-attention allows for richer interactions between layers at different depths. DCA further provides both empirical and theoretical results to support this approach.

Come to our poster session today to chat about Titans! Location: Exhibit Hall C,D,E Poster ID:3515

We keep scaling model parameters by increasing width and stacking more layers, but what if the truly missing axes for continual learning are compression and stacking the learning process? Excited to share the full version of Nested Learning, a new paradigm for continual learning and machine learning in general. Paper: nestedlearning.net/paper

What makes attention the critical component for most advances in LLMs and what holds back long-term memory modules (RNNs)? Can we strictly generalize Transformers? Presenting Atlas (A powerful Titan): a new architecture with long-term in-context memory that learns how to memorize the context at test time. Atlas even outperforms Titans, and is more effective than Transformers and modern linear RNNs in language modeling tasks. It further improves the effective context length of Titans and scales to 10M context window with +80% accuracy on the BABILong benchmark. (Bonus: Building on Atlas ideas, we also discuss another family of models that are strict generalization of softmax attention)

.@GoogleAI has dropped a very interesting study They introduced new types of attentional bias strategies in LLMs and reimagined the "forgetting" process, replacing it with "retention." All of this is wrapped up in Miras – their new framework for designing efficient AI architectures using 4 building blocks: • Memory architecture – how the memory is built • Attentional bias – how the model focuses • Retention gate – how it forgets or keeps information • Memory learning algorithm – how it’s trained Details 🧵

🗓️ We're excited to announce that the next virtual London Machine Learning Meetup will be on Wednesday 26th March. The Meetup will feature a talk by Ali Behrouz (@behrouz_ali) (PhD, @Cornell) on 'Titans: Learning to Memorize at Test Time'. RSVP: bit.ly/41tvBid

Attention has been the key component for most advances in LLMs, but it can’t scale to long context. Does this mean we need to find an alternative? Presenting Titans: a new architecture with attention and a meta in-context memory that learns how to memorize at test time. Titans are more effective than Transformers and modern linear RNNs, and can effectively scale to larger than 2M context window, with better performance than ultra-large models (e.g., GPT4, Llama3-80B).