Tweet fixado
Excited to announce the return of American OSS with Arcee Trinity Large.
This model couldn't have been possible without the awesome collaboration of Modeling @arcee_ai , Infra @PrimeIntellect , and Data @datologyai
I can't say enough about how talented the whole team at Arcee is being able to scale from their first MoE to a big boy like this in such a short time.
Since the last data mix we have been in the lab pushing our midtraining and synthetic data to the limits. For Trinity Large we generated over 800B tokens of high quality synthetic code and 6.5T(!!!) tokens overall. We also added multilingual curation.
This was a massive effort from the whole Datology family. From scaling up the rephrasing workflows to support heterogenous clusters to scale efficiently (@isabelle226ku, @JackUrbs, @parthjdoshi @haakonmongstad @alvind319), pushing out midtraining and mixing (@_BrettLarsen) , innovating on new code synthetic data (@amrokamal1997) new math synthetic data (David Schwab), multilingual curation (@KaleighMentzer, @agcrnz, @RicardoMonti9)
and of course built on our great foundation of synthetic data (@pratyushmaini Vineeth Dorna)
Arcee.ai@arcee_ai
Today, we’re releasing the first weights from Trinity Large, our first frontier-scale model in the Trinity MoE family.
English
