Fenia Christopoulou

Today we’re publishing the technical report behind Laguna M.1 and Laguna XS.2. This report opens up more of what went into them: Model Factory, pre-training data, distributed training, post-training, agent RL, quantization, and evaluation. poolside.ai/assets/laguna/…

Today we’re releasing Laguna XS.2, Poolside’s first open-weight model. It’s a 33B total / 3B active MoE model built for agentic coding and long-horizon tasks. Trained fully in-house on our own stack. Runs on a single GPU. Released under Apache 2.0. Links 👇 Weights: huggingface.co/poolside/Lagun… API: platform.poolside.ai Blog: poolside.ai/blog/laguna-a-…

Happy to share our work on "Text2Code Generation with Modality-relative Pre-training" w/ @gcsanity & @glampouras_NLP, accepted at #EACL2024! 🎉 We propose to treat code and natural language as different modalities. 📜arxiv.org/abs/2402.05783 💻coming soon (pending int. review)

🚀 Excited to share our new pre-print: "Human-like Episodic Memory for Infinite Context LLMs"! We introduce EM-LLM, a novel approach integrating cognitive science insights into LLMs for vastly extended context processing: arxiv.org/pdf/2407.09450 What we did: · 📊 We treat LLMs' K-V cache as analogous to personal experiences and segmented it into events of episodic memory based on Bayesian surprise (or prediction error). · 🔍 We then apply a graph-theory approach to refine these events, optimizing for relevant information during retrieval. · 🔄 When deemed important by the LLM's self-attention, past events are recalled based on similarity to the current query, promoting temporal contiguity & asymmetry, mimicking human free recall effects. · ✨ This allows LLMs to handle virtually infinite contexts more accurately than before, without retraining. Our method outperforms the SOTA model InfLLM on LongBench, given an LLM and context window size, achieving a 4.3% overall improvement with a significant boost of 33% on PassageRetrieval. Notably, EM-LLM's event segmentation also strongly correlates with human-perceived events!! We are releasing this method today with our first set of results, but more results and analysis are coming soon. Huge thanks to all my co-authors/colleagues for this amazing collaboration: Martin A Benfeghoul, Adnan Oomerjee, @fenchri, @glampouras_NLP, @hbouammar from @Huawei Noah's Ark and Jun Wang from @UCL. For a fuller description, check out @hbouammar's thread: x.com/hbouammar/stat… Stay tuned for more updates and a code release soon :) #LLMs #LongContextLLMs #EpisodicMemory #NLP #CogSci #MachineLearning #AI

@chenghua_lin @csmcr @UoMSciEng After a short break, Dr Fenia Christopoulou (@fenchri), Research Scientist at @Huawei Noah’s Ark Lab, presents on Natural and Programming Language Models! 🦾 #ADSAI2024 @nactem_unimcr