@mattshumer_ Yeah looks awesome - any idea how they calculated the $0.19-$0.49 PPM tokens? They say it's based on $2/hour H100 cost and serve rate of 0.03 ms / token I think?
English
David Foster
419 posts
David Foster
@davidADSP
Author of Generative Deep Learning: Teaching Machines how to Paint, Write, Compose and Play (O'Reilly), #generativeAI, Founding Partner of ADSP.
London, UK เข้าร่วม Temmuz 2019
577 กำลังติดตาม777 ผู้ติดตาม
Llama 4's price/perf looks absolutely incredible.
And a 10M token context window? Insane.
Assuming the vibes check out, we'll be switching over many of our systems to Maverick.
English
@Thom_Wolf It's a reference to the fact that an ensemble of all submissions would have scored 81% on the private test set (i.e. 19% of solutions were unsolved by any solution) x.com/fchollet/statu…
François Chollet@fchollet
Does this mean the ARC-AGI benchmark has saturated? Yes -- the v1 version of the benchmark is starting to saturate. There were already signs of this in the Kaggle competition this year -- an ensemble of all submissions would score 81%. The competition next year will run on ARC-AGI-2, an updated version of the dataset that keeps the same format as v1, but features fewer tasks that can be easily brute-forced. Early indications are that ARC-AGI-v2 will represent a complete reset of the state-of-the-art, and it will remain extremely difficult for o3. Meanwhile, a smart human or a small panel of average humans would still be able to score >95%.
English
what was this thing btw?
"Moreover, ARC-AGI-1 is now saturating – besides o3's new score, the fact is that a large ensemble of low-compute Kaggle solutions can now score 81% on the private eval"
big ensemble of heuristics?
English
English
Watch the finale of "12 Days of @OpenAI" livestream for a big announcement, starting in 3 minutes...
openai.com/12-days/
English
Consulting my heart... Ok, looks like you haven't. But whenever you have a SotA (or close) solution built on top of the OpenAI API we're more than happy to verify it and add it to the public ARC Prize leaderboard. Anything using less than $10k worth of API calls is eligible.
Sam Altman@sama
@DavidSHolz @willdepue in your heart do you believe we’ve solved that one or no?
English
Gemini-exp-1114 is now available via the Gemini API, happy building / testing! Will follow up Monday with more 🚢
ai.google.dev/gemini-api/docs
English
@jsuarez @hirschibar Awesome write up! What about action masking - i.e. how do you handle cases where certain actions aren't possible (and the env returns you the mask at each timestep). Is this something PufferLib supports?
English
@hirschibar It's just a list of discrete actions. Instead of 1 linear layer to output action, you have n layers. And then you just sum the losses for each
English
@lmarena_ai @01AI_Yi Will the multimodal Llama 3.2 models be added to the overall leaderboard?
English
Big News from Chatbot Arena!
@01AI_YI's latest model Yi-Lightning has been extensively tested in Arena, collecting over 13K community votes!
Yi-Lightning has climbed to #6 in the Overall rankings (#9 in Style Control), matching top models like Grok-2. It delivers robust performance in technical areas like Math, Hard Prompts, and Coding.
Huge congrats to @01AI_YI!
Meanwhile, GLM-4-Plus by Zhipu AI (@ChatGLM) has also entered the top 10, marking a strong surge for Chinese LLMs. They're quickly becoming highly competitive. Stay tuned for more!
More analysis below👇
Arena.ai@arena
Yi-Lightning is now in Chatbot Arena! The latest and most capable model from @01AI_Yi. Come chat and vote at lmarena. ai. The leaderboard will be updated soon.
English
@lmarena_ai @lmsysorg Will Llama 3.2 (the multimodal models) and Gemini 1.5-002 be added to the main leaderboard?
English
As part of Chatbot Arena's graduation🎓, we're excited to announce that we changed our X handle to @lmarena_ai! For open-source systems & research at LMSys, please follow @lmsysorg.
This account, @lmarena_ai, will be dedicated to sharing Arena projects & leaderboard updates. See you tomorrow for another one 👀
Arena.ai@arena
We are happy to announce a new site for Chatbot Arena! Over the past year, with the incredible support of our community, Chatbot Arena has evolved into a mature ecosystem and platform. We believe it's time for it to graduate and stand on its own. By giving Chatbot Arena its own platform, we aim to provide it with more independence and ensure its long-term growth. With a strong partnership with LMSys, we're expanding the platform to evaluate frontier models, not only for chatbots but also in areas like coding, complex tasks, and red-teaming. LMSys has been a research collective dedicated to a variety of projects, such as Vicuna, Chatbot Arena, SGLang, S-LoRA, RouteLLM, and more — beyond just one initiative. Moving forward, LMSys will continue to serve as an incubator for new projects and as a platform for open research and development. Come join us! Chatbot Arena: lmarena.ai New blog site: blog.lmarena.ai Blog: lmsys.org/blog/2024-09-2…
English
@SullyOmarr Would you be willing to share the leaderboard from your evals?
English
underrated: gemini 1.5 flash
overrated: gpt-4o
We really need better ways to benchmark these models cause lmsys aint it
stuff like cost, speed, tool use, writing, etc. arent considered
Most ppl just use the top model based on leaderboards, but it's way more nuanced than that
English
@giffmana @pastaraspberry What is it about Gemma that makes it open, but not open source? Thanks!
English
English
You know what's my favourite part with our Gemma release?
That we do not misuse the term "open source" like other labs have. It was explicit in the comms briefing that we should call them "open models" and not "open source models". Much respect to the team.
English
@giffmana @pastaraspberry How are you defining open vs open source. Thanks!
English
@NPCollapse Funny story - William Peebles co-authored the Mar 2023 Diffusion Transformer paper on which Sora is based, whilst at Meta as an intern.
But then joined OpenAI last year to co-lead Sora.
So I guess they did know how to do it, but let him leave 😂
English
almost 10 years in and I'm still listening to the soundtrack for Interstellar when I need to code some epic stuff. will it be ever topped
English
@realGeorgeHotz Given the current breakthroughs, "linguistics" is a left-field candidate 🤔
English
@nickfloats Does the --iw parameter affect remixes? In the docs it says it doesn't, but I'm never sure how much to trist the docs :)
English
Remixing with images can give you even more control in Midjourney
You maintain more of the details and can do really fun things like turn group photos into animal balloon parties.
A quick series of images, w/ a tutorial on how to do it at the end. It's actually super easy.
English
@nickfloats Related question / challenge - how do you get Midjourney to output the usual meaning of 'fork in the road', rather than this? Changing the prompt to use different words isn't allowed 😃
English
@SullyOmarr Nice idea! Can you create a short example as a demo?
English
@StabilityAI When your stochastic parrot drinks too much coffee.
🚀 Awesome work @StabilityAI !
English
Announcing StableLM❗
We’re releasing the first of our large language models, starting with 3B and 7B param models, with 15-65B to follow. Our LLMs are released under CC BY-SA license.
We’re also releasing RLHF-tuned models for research use. Read more→ stability.ai/blog/stability…
English
@EMostaque Is your stochastic parrot related to my stochastic parrot? 😄
#generativeai #stablediffusion #ai @StabilityAI
English
