DatologyAI

This is p sick and completely surprising to me wowie great work

If I had to compress my PhD into one idea, it is this "The data a model sees early in training leaves an imprint on its representations that is very hard to undo later" This thread runs through - Rephrasing the Web - Safety Pretraining - TOFU This is the Finetuner’s Fallacy🧵

📄 arxiv.org/abs/2603.16177 📝 datologyai.com/blog/finetuner…

New Datology Research: We expose "The Finetuner's Fallacy" The standard approach to domain adaptation (pretrain on web data, finetune on your data) is leaving performance on the table. Mixing just 1-5% domain data into pretraining, then finetuning, produces a strictly better model: ◾ 1.75x fewer tokens to reach the same domain loss ◾ 1B SPT model outperforms a 3B finetuned-only model ◾ +6pts MATH accuracy at 200B pretraining tokens ◾ Less forgetting of general knowledge Tested across chemistry, symbolic music, and formal math proofs. SPT wins on every metric. Led by @_christinabaek and @pratyushmaini, with the full Datology team.

Two nursing home residents are eating lunch. One says, "Boy, the food at this place is terrible." The other says, "Yeah, I know, and such small portions, too." This is the multilingual data problem. The data is bad, AND there's not enough of it. Yesterday at @datologyai we released ÜberWeb: our study of multilingual curation that gets 4-10x train FLOPs improvements on multilingual benchmarks compared to strong public baselines like Qwen3-1.7B and Tiny Aya Base.

@Arceeai @RicardoMonti9 If you're interested in joining our team to do cool stuff like this, head to datologyai.com/careers. And if you need to improve your data (you definitely need to improve your data), get in touch via our website datologyai.com

@Arceeai See @RicardoMonti9's thread for a deep-dive: x.com/RicardoMonti9/… ArXiv: arxiv.org/abs/2602.15210 Blog: datologyai.com/blog/berweb-in…

New research! ÜberWeb: multilingual data curation across 13 languages and 20 trillion tokens. The "curse of multilinguality" is largely a data quality problem, and it's fixable. tl;dr: we get 4-10x training efficiency improvements over models like Qwen3 and Tiny Aya

Full paper and blog below. For the technical deep-dive, see {@matthew}’s thread. For why this matters for the future of AI, see {@ari}'s thread. ArXiv: arxiv.org/html/2602.1521… Blog: datologyai.com/blog/berweb-in…

This curation approach also powers @arceeai's Trinity Large Base, which shows exceptionally strong multilingual performance relative to its compute budget.

Proud to power @arcee_ai’s Trinity Large. A 400B/A13B MoE model trained on 17T tokens of our curated data (including 8T synthetic), delivering true frontier-level performance with just 13B active parameters. Incredible collaboration with @PrimeIntellect and @arcee_ai! The most exciting part? This is just a preview model. We can't wait to see how strong it is after full post-training!

Our team at DatologyAI is excited to launch Data & Dialogue, a new executive dinner series bringing together leaders for an intimate evening of great food, candid conversation, and fresh perspectives on data and AI. Join us on February 26 at 5:30 PM for our first gathering and a relaxed discussion with peers. Join the waitlist here: lnkd.in/gVDKHWVZ

Great data curation isn't just for training! We @datologyai just released DatBench, a refined VLM eval suite with a simple motivation: VLM evals are broken. VLM evals are noisy, often measure the wrong thing, and expensive, often consuming ~20% of train compute. No longer!

Benchmarks can make VLMs look smarter than they are. DatBench shows why: MCQ formats invite guessing, many items are solvable without vision (yes, “blind-solvable”), and label noise/ambiguity can blur real progress. We curate evals to be faithful + discriminative, and cut eval cost by 13× by selecting the highest-signal examples (up to 50×). Blog: datologyai.com/blog/datbench-… HuggingFace: huggingface.co/datasets/Datol… GitHub: github.com/datologyai/Dat… Huge shoutout to the team for continuing to raise the bar on what rigorous VLM evaluation looks like! Want to work on cool stuff like this full-time? We’re hiring: #open-positions" target="_blank" rel="nofollow noopener">datologyai.com/careers#open-p…

And new @datologyai release addressing a major issue for synthetic pipelines: fast/accurate text retrieval AT SCALE.