고정된 트윗
Pretty big release of bleve out! GPU accelerated vector search (I helped on this hehe), binary quantization and a lot more!
github.com/blevesearch/bl…
English
gautham
397 posts
gautham
@capemox
distributed search @couchbase | @blevesearch member, @huggingface contributor | contrastive loss enjoyer
Bengaluru, India 가입일 Temmuz 2023
434 팔로잉85 팔로워
Qwen2.5-7B and GPT-3.5-turbo in 2026?
Garry Tan@garrytan
Qwen2.5‑7B Instruct apparently is GPT-3.5-turbo level
English
your novel idea, when you ask an llm to fill in the details
Drew Breunig@dbreunig
We need a name for this, because Armin is putting his finger on a problem that’s everywhere: people running their writing through an LLM because they think it makes it clearer, when in actuality it sands off all the detail.
English
@drexalt @mattjustram @yjoonjang Dupmae has a technique to train both the CLS as well as the other regular tokens. I agree that the CLS train may not be useful at all, but the others might be?
English
@capemox @mattjustram @yjoonjang I am pretty sure it would not be, since ColBERT isn’t using the MLM head anymore and the CLS token-maxxing doesn’t seem super aligned. I think the LateOn contrastive pre train probably stronger
English
huggingface.co/capemox/ettin-…
Trained on subsets of GooAQ, Wikipedia Sections and NQ for 2.5 million total data points
English
@antoine_chaffin @mattjustram Got confused by the language in the paper :/ I think I was over eager to assign blame to MNTP
English
@capemox @mattjustram This is false though
We trained both « pure » encoders and « pure » encoders
We also released decoders turned into encoders through MNTP continued pre-training, but those aren’t the base encoders
English
@drexalt @mattjustram @yjoonjang btw @drexalt have you tried dupmae with colbert? curious if it's a good conditioning for multivector
English
@capemox @mattjustram this might serve as a jumping off point from @yjoonjang, I tried DupMAE on NeoBERT and had good results
github.com/yjoonjang/Mode…
English
@bclavie @capemox btw @capemox I think this is relevant to make ModernBERT works with sparse: arxiv.org/abs/2601.17500
English
@capemox creates a much smaller, fixed, (sparse) dim to work with downstream, basically turns it into an encoder for you
might have better isotropic properties and behave better for your use case 🤷♂️
gonna read those @yjoonjang linked papers, seems like that's pretty much it
English
@yjoonjang @TheAdamEvans Yeah for sure. I'll start with ettin models first
English
@TheAdamEvans @capemox Yeah thats what the paper tells. But if you start with qwen, I think you'll need mntp training for bidirectional attention (Thats what the authors did)
English
@drexalt @mattjustram @yjoonjang I already have a "pretraining" library that I was planning to open source lmao didn't know this existed
English
@TheAdamEvans zamn, tell me more
basically, layer pruning qwen3.5-2b right? but why would qwen scope be useful, that's for interpretability right?
English
@mattjustram Wow I totally missed this lmao. You're right, this could defo be the issue. I might try some dupmae style or contrastive pretraining to see if it helps
English
@capemox i just realized they further trained the encoders with MNTP. i think it ruined the MLM init for SPLADE
English