Sabitlenmiş Tweet
Ryan Marten
645 posts
Ryan Marten
@ryan_marten
Building @harborframework and @terminalbench
San Francisco Katılım Haziran 2022
2K Takip Edilen2.7K Takipçiler
It doesn't take a superforecaster to know that Ven and team are going to do big things!
Veniamin Veselovsky@VminVsky
Over the past year I've been building out Preseen. We’re focused on building the best AI forecasting agent for decision makers. Every decision relies on a forecast. Too often decisions are poorly made because the underlying forecast is bad. Humans are constrained. But if you can have a system that can forecast anything as well as the best humans, then human decision making will be improved. AIs are reaching the point where they are better than some of the best human forecasters. This is going to change decision making. Preseen will help drive that change.
English
@modal_labs @AnthropicAI @OpenAI @Google @scale_AI @SnorkelAI @turingcom @gNucleusAI @joinHandshake @ellamindAI @giansegato @boolean_ai Thank you to @modal for providing the sandboxes for developing Frontier-Bench! This was particularly useful for unlocking GPU tasks (e.g. agent is asked to write kernels, run evals, train models, etc.).
English
@modal_labs @AnthropicAI @OpenAI @Google @scale_AI @SnorkelAI @turingcom @gNucleusAI @joinHandshake @ellamindAI @giansegato Thanks as well to @boolean_ai (missed the tag above). It was a pleasure working together!
English
We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work.
Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort.
Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%
English
@Curline1222 working on it! we will update the leaderboard with new results soon! (other models are also in progress / on our run roadmap)
English
@ajratner @SnorkelAI Thank you so much for your support! Open Benchmarks Grants is so great for supporting the evaluation ecosystem!
English
Incredibly excited for a new frontier benchmark from the incredible TBench/Harbor team!!
Proud that @SnorkelAI contributed as a task author and data partner, with support through our Open Benchmarks Grants program.
Ryan Marten@ryan_marten
We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work. Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort. Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%
English
@dwahdany Thank you Dariush for all your amazing work! It was really fun building this together!
English
Frontier-Bench v0.1 (or in spirit, Terminal-Bench v3.0rc1) is out! Exciting new changes:
- stronger differentiation between frontier models
- regular updates: treat benchmarks like software
- auditing process of all benchmark submissions
- refined task review process
Ryan Marten@ryan_marten
We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work. Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort. Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%
English
excited to have our contributions included in frontier-bench
Ryan Marten@ryan_marten
We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work. Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort. Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%
English
@realjustinbauer Thank you for all your great work in making this happen! Pleasure collaborating with you Justin!
English
