Pierre Colombo

🚀 Introducing SaulLM-141B and SaulLM-54B: The First Open Family of Legal Models. After #SaulLM-7B the family is growing! We are proud to unveil the latest innovations from our team: the SaulLM-141B and 54B generative AI models, specifically designed for the legal domain.

🚨 Do LLMs need to store everything they read in memory? To reduce KV cache size and improve decoding speeds, we propose Self-Pruned KV attention, a mechanism where the model learns to decide which KVs to write in the persistent KV cache, discarding all the rest! @AIatMeta🧵

@JinaAI_ has hopped on the omnimodal train🚂 They just dropped a collection of two Omni embedding models (0.9B & 2B). Similar to BidirLM, they seem to rely on the Qwen modality head for the larger one, while sticking with EuroBERT for the nano version 🥰 huggingface.co/collections/ji…

BidirLM-Omni is on MTEB and Sentence-Transformer! huggingface.co/spaces/mteb/le… 🥇#1 Open-Source Model on MTEB (#15 overall) 🖼️#1 across all sizes on MIEB (Image) 🎧#1 sub-7B model on MAEB (Audio, #2 overall) Small size, massive performance, Fully open Model: huggingface.co/BidirLM

We are currently presenting 'Should We Still Pretrain Encoders with Masked Language Modeling?' Come see us in Hall 3 #1304 @iclr_conf arxiv.org/abs/2507.00994

If you're at ICLR, come say hi to @orionweller and @N1colAIs! And also, shot-out to all the French people pushing out the encoder architecture, it seems like, as for ColBERT, French taste is unmatched! (Non exhaustive list, pardon me but Twitter search is bad): @gisship @PierreColombo6 @ManuelFaysse @pteiletche @MaceQuent1 @mlpc123

Great work from @gisship and @N1colAIs from @centralesupelec

BERT-as-a-Judge A robust alternative to rigid lexical matching for LLM evaluation. Matches the performance of LLM-as-a-Judge at a fraction of the computational cost.

Encoders are so much better for classification, why not use them for judging? Awesome study from @N1colAIs - cool to see a 210m BERT model beating much larger Qwen and Gemma models.

Evaluation is underrated. If your eval signal is noisy, you're flying blind. BERT-as-a-Judge gives you a fast, cheap way to improve your signal-to-noise ratio without spinning up a full LLM judge. Exactly the kind of infra work that compounds. @gisship @N1colAIs congrats!

🎉 Second paper this month! Introducing BERT-as-a-Judge (x @gisship) ⚖️ Evaluating LLMs with rigid lexical methods often fails right answers due to bad formatting. While "LLM-as-a-Judge" solves this, it remains costly & slow. Our fix? A lightweight, encoder-driven approach.

There's a wave of omni embedding models (gemini, nemotron, bidirlm). Excited to support this trend with our multimodal mteb versions (mieb, maeb) - video coming soon🎥

Omni embeddings are becoming the new standard. Glad to see @N1colAIs @Muennighoff pushing multimodal eval forward with MIEB & MAEB — can't wait for the video!

They even released the base bidirectional models 😍 Great release, thanks for all the checkpoints ♥️

The world needs more encoders Turning decoders into encoders is a very strong path forward considering the edge of public decoders (see Ettin/previous work from Nicolas) Happy to see more work towards this in the omni setup and also public models!! Can’t wait to try them out

🚀 New model family release with an OMNIMODAL version ! After Eurobert, I'm excited to introduce BidirLM, a family of 5 frontier bidirectional encoders including an OMNIMODAL encoder at just 2.5B parameters. 🧵👇 huggingface.co/BidirLM

📦 Models & data: huggingface.co/BidirLM 📝 Blog: huggingface.co/blog/Nicolas-B… 📄 Paper: arxiv.org/abs/2604.02045 Joint work with @TheoDescha33800, @gisship , @PierreColombo6 , @CelineHudelot 🙌 Kudos @drexalt @_reachsumit who spotted paper 4 days before release and kindly shared it

Most practicionners would agree that text embeddings should be "contextual" - ie. they should encode a passage w.r.t. the wider scope of the entire document the passage stems from; "They beat the British" could refer to football or french history without further context... In ConTEB (arxiv.org/abs/2505.24782), we highlight the standard failure modes of embedding models on retrieval tasks that require context to be properly embedded. We also propose a training strategy that extends standard "late chunking" to teach models to infuse embeddings with just the right amount of contextual knowledge to optimize retrieval. Super happy to see some new work by @perplexity_ai on contextual embedding models. They eval on ConTEB and use our in-sequence contrastive loss, along with a ton of cool techniques in multiple phases of training. Love the work @bo_wangbo and will read in details, but super happy to see one more stone towards contextual embedding models, in the path already traveled by @hxiao and @jxmnop ! Link to the paper: arxiv.org/abs/2602.11151…

In August, I joined FAIR at Meta in @hjegou's group for an end of thesis internship. I can't talk much for the moment about what we have been doing (hint: not retrieval), but it's very exciting and I am having lots of fun working with great people! (13/15)

Other great news, our paper on designing rigorous Membership Inference attacks against LLMs led by @matthieu_meeus won Best Paper Award at IEEE SatML! (12/15) x.com/matthieu_meeus…

In a follow-up project, we carefully investigate the differences between Masked Language Modeling (encoder) and Next Token Prediction (decoder) objectives to produce text representations and uncover many nice insights into training efficiency. (11/15) arxiv.org/abs/2507.00994