@jeremyphoward @simonw ToC can be quite restrictive though, special cased and prohibitive for anything that depends on web grounding
English
Joe Hoover
220 posts
Joe Hoover
@JoeEHoover
Automating AI eval @Apple
DC Metro Katılım Ocak 2012
1.2K Takip Edilen558 Takipçiler
Joe Hoover retweetledi
🚨There May Not be Aha Moment in R1-Zero-like Training: oatllm.notion.site/oat-zero
A common belief about the recent R1-Zero-like training is that self-reflections *emerge* as a result of RL training. We carefully investigated and showed the opposite.
🧵
English
"on the majority of the *original* benchmarks, over 50% of 'model errors' are actually caused by label noise!"
🥺
Doing the lord's work 🙏🙏
Aleksander Madry@aleks_madry
Do current LLMs perform simple tasks (e.g., grade school math) reliably? We know they don't (is 9.9 larger than 9.11?), but why? Turns out that, for one reason, benchmarks are too noisy to pinpoint such lingering failures. w/ @josh_vendrow @EdwardVendrow @sarameghanbeery 1/5
English
@lateinteraction This isn't a terrible description of most technology?
But, maybe this hits so hard because we see how the sausage is made.
English
Modern ML seems to mostly work because we stick carefully to extremely finicky starting points.
If you give a smart person a description of modern ML, a ton of GPUs, but no tooling or hyperparameters or pretrained models, I'm not sure they can ever manage to go past ImageNet...
Jack Morris@jxmnop
here is the list of bugs my collaborator and i have made while training this GAN. it's almost unbelievable - misplaced positional argument which caused our discriminator to have 256 layers instead of 3 - missing a detach() somewhere that mattered - extra detach() somewhere else that mattered - improper train() and eval() so the generator and discriminator cheated by communicating via batchnorm - forgot grad norm clipping so everything exploded - forgot to save model to disk - sigmoid on the output of the discriminator and in the loss function (a classic extra sigmoid bug) - at one point we were actually reinitializing the generator parameters at every epoch and it still worked somehow only by systematically locating and eliminating all of these did it start working
English
SOTA LLM agents trained on just 72 tasks—but not with GRPO 👀
- No value network
- No reward norms
- Beats o1
- Great discussion on why this worked in a small-data regime
Aleksei Petrenko@petrenko_ai
Excited to share our new pre-print arxiv.org/pdf/2502.01600 We train a digital agent that solves diverse day-to-day tasks from the AppWorld benchmark by interacting with its stateful environment using API calls. AppWorld is hard! The previous best open-weight agent (Llama 3 70B) reached only a 7% success rate on the hardest test split. Our RL algorithm, LOOP - a PPO variant with Monte Carlo baselines - achieves a 45.7% success rate, 24% over the base Qwen 2.5 32B, and 9% higher than a much larger OpenAI o1.
English
We're building responsible AI evaluation systems at Apple. Want to help?
Seeking a Senior Research Data Scientist (contract) in SEA/NYC/SD/Bay Area with data science, RAI, and human research expertise. Python + LLM experience required.
Must be available now. DM resume + fit.
English
I love the smell of LMs in the morning
Replicate@replicate
Meta just launched their biggest language model and it's available on Replicate! This cutting-edge open-source LLM rivals the top closed models. And you can use it with just an API call. Try Meta Llama 3.1 405B Instruct in our playground today: replicate.com/meta/meta-llam…
English
English
@JoeEHoover @HamelHusain @dan_s_becker Oh, was too focused on finding it so paused the video too early. Found the cog-vllm repo that describes it. Awesome!
English
It’s rare to see a course go viral like this one.
But then, what else would you expect when @HamelHusain and @Dan_s_Becker get together?
Hamel Husain@HamelHusain
Amazing news re: our LLM fine-tuning course: All students get $1,000 in free compute credits from @modal_labs and @replicate ($500 each) 💰 Course signups close end-of-day today. You get more compute credits than the course costs 🤯 maven.com/parlance-labs/…
English
@ashutoshmehra @HamelHusain @dan_s_becker Thank you, @ashutoshmehra! There are so many thorny issues, it was hard to figure out what to focus on. Glad it seemed useful!
English
@JoeEHoover @HamelHusain @dan_s_becker Your talk @JoeEHoover on tuning llm serving performance was very insightful!
English
Joe Hoover retweetledi
Joe Hoover retweetledi
Code Llama 70B is live on Replicate!
It's the most powerful code generation model from @AIatMeta with instruct, Python, and base variants.
Code Llama 70B instruct is fine tuned for understanding natural language instructions:
replicate.com/meta/codellama…
English
Joe Hoover retweetledi
Today we’re releasing Code Llama 70B: a new, more performant version of our LLM for code generation — available under the same license as previous Code Llama models.
Download the models ➡️ bit.ly/3Oil6bQ
• CodeLlama-70B
• CodeLlama-70B-Python
• CodeLlama-70B-Instruct
English
@ID_AA_Carmack @francoisfleuret Yes! Now I can't think of information theory without hearing an echo of Hamming...
"it's too late to undo the definition, so we have to live with it [...]"
English
@francoisfleuret I smiled reading a grumpy quote from Hamming about how Shannon picked the overly-grand title of “information theory” for his work, rather than something more mundane like “coding theory”.
English
Information Theory is awesome so here is a TL;DR about Shanon's entropy.
This field is about quantifying the amount "of information" contained in a signal and how much can be transmitted under certain conditions.
1/11
English
ML in 2023
(not a calibrated accounting of All Progress, just what caught my haphazard eye)
English