Sabitlenmiş Tweet
Mohamed Elfeki
77 posts
Mohamed Elfeki
@m_elfeki11
Applied Research @Scale PhD@ML; ex-MSFT, Meta, Amazon
Seattle, WA Katılım Ocak 2025
143 Takip Edilen45 Takipçiler
Mohamed Elfeki retweetledi
Mohamed Elfeki retweetledi
1/ Using rubrics (a.k.a. checklists) in RL training is now standard for open-ended tasks without final verifiable result. However, rubric rewards are still proxy rewards that can get hacked during RL training.
We study when rubric-based RL genuinely improves models vs. teaches them to hack the verifier/rubric. We quantify this through exploitation, analyze the failure modes, and introduce a verifier-free metric.
arxiv.org/abs/2605.12474
English
Mohamed Elfeki retweetledi
We recently built HiL-Bench, the first benchmark to test a critical question: do AI agents know what they’re missing and when to ask?
Frontier models perform well with perfect specs. But remove a few key details, and they confidently guess and ship plausible wrong answers.
We just added GPT-5.5, Opus 4.7, and Kimi K2.6 to the leaderboard.
Here’s what we’re seeing ⬇️🧵
English
@shd96556 @ScaleAILabs @shd96556
absolutely does, among many other reliability issues.
we're working on the other problems... more work coming your way soon. stay tuned😉🔜
English
@ScaleAILabs AI doesn’t fail on hard problems.
It fails on missing context it doesn’t admit it needs.
English
Bookmarking this as a case study in research timing. Same finding, two weeks apart, 50x the engagement. The cycle matters more than the work sometimes.
Scale Labs@ScaleAILabs
We recently built HiL-Bench, the first benchmark to test a critical question: do AI agents know what they’re missing and when to ask? Frontier models perform well with perfect specs. But remove a few key details, and they confidently guess and ship plausible wrong answers. We just added GPT-5.5, Opus 4.7, and Kimi K2.6 to the leaderboard. Here’s what we’re seeing ⬇️🧵
English
@auTechArena @ScaleAILabs @auTechArena
we 🫶 open-source, and we've been open from the start.
Paper: arxiv.org/pdf/2604.09408
Code/harness/data: github.com/hilbenchauthor…
English
@ScaleAILabs Love this, great work. Spot on that models tend to bluff when you strip out key details. Keen to try HiL-Bench - will you be opening it up for the community?
English
GPT's jump is exceptional. They did something different this last 90 days with their model. finally GPTs can recognize context gaps, and sometimes (not always yet) ask humans.🧠
Derya Unutmaz, MD@DeryaTR_
GPT-5.5 is now SOTA in this agentic AI benchmark. The jump from GPT-5.4 is insane, more than 3-fold! GPT models are now becoming fully agentic! You can already feel this from computer use in Codex.
English
@sama @kenn @sama, would it be better at working with people?
i mean it's better than most, but could still be a lot better.
x.com/ScaleAILabs/st…
Scale Labs@ScaleAILabs
We recently built HiL-Bench, the first benchmark to test a critical question: do AI agents know what they’re missing and when to ask? Frontier models perform well with perfect specs. But remove a few key details, and they confidently guess and ship plausible wrong answers. We just added GPT-5.5, Opus 4.7, and Kimi K2.6 to the leaderboard. Here’s what we’re seeing ⬇️🧵
English
Codex完全勝利だな
・知能ナンバーワン
・リミットも寛大
・ネイティブMacアプリが秀逸
・ハーネスはOSS
・App Serverはサブスクで使える
OpenAIの名前に恥じない中身を伴ってきた
しかし誰であれ一強で気が緩むは良くないので
AnthropicやCursorには頑張ってほしい
日本語
@bronzeagepapi @ScaleAILabs hallucinations is one way.
but sometimes agents make an assumption and it turns out to be correct. so, it achieves the right goal, but it's unreliable!
English
@ScaleAILabs Hallucinations is a good proxy
Will DeepSeek v4 be included in the leaderboard?
English
English
Recently moved from a heavily tuned Opus 4.7 setup to Codex, also tuned pretty hard
Early notes after heavy usage:
- GPT finishes more tasks end-to-end. Opus often needed extra steering, even with good prompts/plugins
- GPT gives me much more usable limit headroom than Claude
- Opus has 1M context vs GPT-5.5’s 400K, but in practice Opus still needs compaction around 400-500K
Not calling a winner yet. Still testing what’s actually better for agentic work
English
@sama @LinghuaJ @OpenAI Truly an impressive model. It knows what it doesn't know.
x.com/ScaleAILabs/st…
Scale Labs@ScaleAILabs
We recently built HiL-Bench, the first benchmark to test a critical question: do AI agents know what they’re missing and when to ask? Frontier models perform well with perfect specs. But remove a few key details, and they confidently guess and ship plausible wrong answers. We just added GPT-5.5, Opus 4.7, and Kimi K2.6 to the leaderboard. Here’s what we’re seeing ⬇️🧵
English
English
@ileppane @ScaleAILabs @artificialguybr haha.. fair enough. come benchmark with us :D
it's meant that last release was a few months ago.
English
@ScaleAILabs @artificialguybr "GPT-5.5, released just months ago" - what? For who? You guys should seemingly benchmark your own posts too, sorry to say
English
@jjoshua2 ... good question.
we were skeptical, but older GPTs poorly detect missing context, often assuming wrongly yet confidently.
@OpenAI's GPT-5.5 impressively fixes this and beats leading Claude/Gemini. Also, GLM/@Kimi_Moonshot Kimi's new models are really strong. you should try them if you haven't already.
English
This seems like one of the most useful skills to make using them with a harness pleasant. Does anyone know if this matches vibes? It seems like it somewhat reasonable with Opus above GPT 5.4 and Gemini, but I think GPT 5.4 probably should be above GLM and Gemini?
Scale Labs@ScaleAILabs
We recently built HiL-Bench, the first benchmark to test a critical question: do AI agents know what they’re missing and when to ask? Frontier models perform well with perfect specs. But remove a few key details, and they confidently guess and ship plausible wrong answers. We just added GPT-5.5, Opus 4.7, and Kimi K2.6 to the leaderboard. Here’s what we’re seeing ⬇️🧵
English
5/ Key insight: capability and “knowing when to stop” are decoupling. Targeted post-training beats raw scaling.
Real agents don’t need more autonomy... they need to ask at the right time.
Paper + leaderboard: scale.com/blog/hil
English
4/ Claude 4.7: better answers, worse precise asking. Kimi K2.6: best precision (62%) but low recall (29%).
Only GPT-5.5 >50% on both axes. Still just 32% vs 85% ceiling.
English
1/ GPT-5.5 learned to ask.
Every frontier model on HiL-Bench fails hard: charges ahead on incomplete specs, guesses instead of clarifying. Pass@3 drops from ~85% (full info) to ~4%.
GPT-5.5 fixes it.🧵
English