Harold Benoit
287 posts
Harold Benoit
@harold_matmul
engineering & research @MicrosoftAI
Katılım Nisan 2024
357 Takip Edilen824 Takipçiler
English
Very proud of our recent work at MAI :) The base models are very strong.
More modalities and larger scale coming soon!
English
MFU is nice because it allows us to summarize a training run "efficiency" with one number. However it is not sufficient at the very large scale :)
Indeed, what labs optimize for when launching a training run is roughly approximated by
~ (final checkpoint performance after RL) / (# GPU hours.)
There's many parts of a lab recipe that will influence this.
For example, a less MFU-friendly (e.g. more sparse) model might be more data efficient, and thus the overall training time will be lower.
Your global batch size (gbsz) and the number of chips used in a training run will also greatly influence the achievable MFU. This is also coupled with your optimizer choice (AdamW, Muon, etc..).
Additionally, co-designing your model to be inference-friendly may be more important than MFU, if you spend lots of compute on RL.
Seunghyun Seo@SeunghyunSEO7
though i have really enjoyed MAI paper, i wanna discuss about their MFU (because many are talking about it). while sparsity ratio, num experts, gbsz is different (gbsz is 134M vs 33M), megatron could achieve 1048e12 flops but MAI achieve only about 500e12 (2.5e12*0.2). wdyt?
English
I'll try to check :) but tbh, people over-index on MFU, when it's one part of the equation.
A less MFU-friendly (e.g. more sparse) model might be more data efficient, and thus the overall training time will be lower.
Also, co-designing your model to be inference-friendly may be more important than MFU, depending on the amount of GPU hours you will pour into RL.
English
@harold_matmul oh, love to hear this from author! is it possible to share mfu without fixes for determinism? i dont suspect mai's capability/talent for engineering but just wonder!
English
though i have really enjoyed MAI paper, i wanna discuss about their MFU (because many are talking about it). while sparsity ratio, num experts, gbsz is different (gbsz is 134M vs 33M), megatron could achieve 1048e12 flops but MAI achieve only about 500e12 (2.5e12*0.2). wdyt?
English
Please find this very detailed thread from the 🐐 @eliebakouch for a summary x.com/eliebakouch/st…
elie@eliebakouch
microsoft MAI tech report is a gold mine, one of the most transparent for a model at this scale. this model uses zero synthetic data or distillation from previous models. this means reasoning, agentic behavior, tool use are all learned fully during post-training with no cold start. bold choice that makes it harder and requires more iterations to reach sota, but you get FULL control over your model series and it proves they are serious about being a frontier lab. the tech report is insanely detailed and precise about numbers. to give an example, they give the exact MFU across all the iterations of the model, with the exact changes etc. they also share the full scaling ladder recipe, to my knowledge this is the first time i've seen this in a tech report at this scale let's look at all of this in this likely very long thread 🧵
English
@lateinteraction it was my idea :)
Using GEPA is a very natural workflow for creating LLM programs. The iteration speed is very quick, and it easily allows researchers to bias the optimization with some priors (usually derived from just looking at the data).
Thanks a lot for the great tool!
English
dspy.GEPA used in pretraining data curation in the new Microsoft AI effort :-)
Lakshya A Agrawal@LakshyAAAgrawal
Excited to see the use of GEPA-optimized LLM judges for data filtering in MAI-Thinking-1 model's pre-training pipeline!
English
this was an insanely good read, i think this is the most detailed report i've read at this scale in some aspects. i really hope MAI continues releasing those tech reports, thanks a lot to the team for this gift 🥹
#page=81.11" target="_blank" rel="nofollow noopener">microsoft.ai/wp-content/upl…
English
the "loss" definition is VERY important, the scaling ladder heavily relies on this. it's a NLL private set (negative log likelihood) with:
50% code
17.5% STEM
17.5% Math
10% General knowledge
5% Multilingual
they then use this target NLL and normalize it with an in-house model. normalization matters because raw NLL scales differ across benchmarks
English
@eliebakouch NLL is likely only relevant until mid-training
English
what do we think about perplexity/NLL eval for post trained models? cursor composer 2 did use it to choose the starting model (but didn't end up choosing the best one according to NLL)
Nando de Freitas@NandoDF
The progress on some of these benchmarks has been insane! @AnthropicAI @DarioAmodei May I please ask you to request Claude to give you a list of the of the top 1000 areas of STEM, top 1000 magazine topics, top 500 professions, and for each list item pick a (not in training data) book or set of long articles, and finally report NLL on all these test sets. If every company reported 2500 fast to compute evals of this nature, the public would have a better understanding of the capabilities of each model in their area of study, work or hobby. Thanks 🙏 AI community: Thoughts on how to improve these evals or how to report them? @d_spiegel
English
update: joining @PrimeIntellect 🦋
i'm super excited to join the team. i really admire what they've been building and i love the mission of pushing the frontier in the open
i'll be working on pre/mid training, there's so much left to figure out and i truly believe a small group with the right people, resources and focus can do sooo much 🚀
English
In a couple weeks I'm joining @AnthropicAI to work on pretraining
after nearly 3 years at FAIR, developing post-training flywheels for physical intelligence (like SAM 3D)
I'm stoked to build new capabilities for a model I personally love, with such thoughtful people
English
today is my last day at hugging face
feeling really grateful to have worked with such an amazing team and learned so much along the way. i’m proud of what we accomplished together, especially the smollm series. building that project from scratch, putting so much into it, and getting to iterate on a model and training recipe that pushed the frontier for its size was really rewarding
i hope i was able to play a part in making model training more accessible and in pushing the open model ecosystem forward. i’m also very thankful to hf for giving me the chance to share my passion for llm research, especially here, and to connect with so many awesome people
things can get quite intense in this field, but i’m still very excited about the next challenges and about the good this technology can do
but first, taking a few weeks break :)
English
Some personal news:
- Finished another trip around the sun today 🫡
- Decided to join @PrimeIntellect to work on evals!! There’s a lot to be build and do couldn’t imagine a better team to do just that 🙌
- I will be in SF the next two weeks :) Just to look around, of course 👀
English
@kilian_maciej @AnthropicAI Amazing :D I'm sure you'll do great stuff then
English
excited to share that i'm joining the @AnthropicAI pretraining team!
claude is by far my favorite model and it brings me so much joy to get to be part of this.
everyone i've met here is brilliant and incredibly kind and i'm really excited to be working with them :)
English
@kilian_maciej @AnthropicAI Btw, will you keep working on multimodal?
English