Steven Dillmann

1

Surag Nair@suragnair·1h

@StevenDillmann @AnthropicAI @OpenAI @GoogleDeepMind Thanks! Any guarantees the labs won’t train on this? Once the data (especially answers) are out it’s really hard to limit how it’s used.

English

0

19

Steven Dillmann@StevenDillmann·2d

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

15

110

469

889.8K

Steven Dillmann retweetledi

Stanford AI+Biomedicine Seminar@Stanford_AI_Bio·15h

Wish an AI agent could handle your next research task in the list? 👇

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

1

4

1.7K

Steven Dillmann@StevenDillmann·15h

@suragnair @AnthropicAI @OpenAI @GoogleDeepMind @suragnair Great work on CompBioBench! v1 of the benchmark will be fully open as per terminal-bench standards

English

0

63

Surag Nair@suragnair·15h

@StevenDillmann @AnthropicAI @OpenAI @GoogleDeepMind Hey Steven, this is super cool! What’s the policy around access to contributed tasks, artefacts, and solutions? Will they be openly available to labs using the benchmark, or are there contributor protections around reuse/training?

English

Timothy Kassis@TimothyKassis

0

1

75

Steven Dillmann@StevenDillmann·23h

Great article by @TimothyKassis on why we need Terminal-Bench Science - if you’re a scientist and want AI agents to become better in your domain, join us!👇 tbench.ai/news/tb-scienc…

x.com/i/article/2057…

English

2

10

788

Steven Dillmann retweetledi

Sanmi Koyejo@sanmikoyejo·1d

"AI for science" benchmarks today mostly test textbook recall. Terminal-Bench Science is a chance for scientists to practice writing that definition. Contribute a real workflow, and you find out exactly where today's best agents break on it. tbench.ai/news/tb-scienc…

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

4

21

2.3K

Steven Dillmann retweetledi

Chaitanya K. Joshi@chaitjo·1d

Very timely initiative!!

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

1

5

1.6K

Steven Dillmann retweetledi

Richard C. Suwandi@richardcsuwandi·1d

Good evals like this are exactly what we need to accelerate progress in AI for science

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

2

9

1.7K

Steven Dillmann retweetledi

Bodhisattwa Majumder@mbodhisattwa·1d

Wonderful project; wonderful people; please contribute for the sake of science. Bonus: @StevenDillmann will be interning with me and AutoDiscovery team @allen_ai translating benefits from TB-Science to our science agents!

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

2

20

2.1K

Steven Dillmann retweetledi

Bespoke Labs@bespokelabsai·1d

Consider contributing tasks to Terminal-Bench Science, the most direct way to teach AI agent to solve your AI workflows and accelerate your research.

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

3

8

1.2K

Steven Dillmann retweetledi

Allan@AllanatrixQ·1d

More please!

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

1

4

277

Steven Dillmann retweetledi

Chenhao Tan@ChenhaoTan·1d

Science is the frontier of AI. Contribute to this initiative if you can!

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

1

9

1.5K

Steven Dillmann retweetledi

Alex Dimakis@AlexGDimakis·1d

Terminal-Bench Science is a direct way to contribute to AI for Science. It's programming agents by task specification. Ask a precise scientific question and watch how AI agents will learn to solve it: Step 1. Package a scientific task or workflow, something that takes a working scientist a week or month to do into an RL environment. Step 2. Write tests that verify if the task has been done correctly (can be done easily if you have already solved the task manually). Step 3. Sit back and let AI agent progress solve it in 6 months.

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

9

7

40

5.1K

Steven Dillmann retweetledi

Leon Chen@realleonlc·1d

Scientists, I highly encourage you to submit hard scientific tasks that you want your agents to do to this Terminal-Bench Science benchmark! Make your task seen and solved by agent/model providers. Get credit from the project.

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

1

5

574

Steven Dillmann@StevenDillmann·1d

@evijit @BlancheMinerva @AnthropicAI @OpenAI @GoogleDeepMind let's chat! @evijit

English

0

1

453

Avijit Ghosh@evijit·1d

@StevenDillmann @BlancheMinerva @AnthropicAI @OpenAI @GoogleDeepMind I literally put in my notes app this morning to look into science agents haha will dig in

English

0

4

499

Steven Dillmann retweetledi

Lisan al Gaib@scaling01·1d

let the hill climbing on scientific tasks begin new benchmark: TerminalBench Science

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

2

3

56

5.7K

Steven Dillmann retweetledi

Alex Ratner@ajratner·1d

Extremely excited for Terminal-Bench Science, which we're proud to support via our Open Benchmarks Grants @SnorkelAI !

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

3

19

2.3K

Steven Dillmann retweetledi

Ryan Marten@ryanmart3n·1d

deadline to submit tasks for Terminal-Bench 3.0 is may 31st! the best tasks are the most interesting to measure: realistic + useful + meaningfully beyond current frontier any piece of valuable work done on a computer is fair game

English

2

6

594

Steven Dillmann retweetledi

Alex Shaw@alexgshaw·1d

Consider contributing a task to Terminal-Bench Science! Terminal-Bench Science will serve as a north star for building agents and models that accelerate science.

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

8

50

4.9K

Steven Dillmann retweetledi

vincent sunn chen@vincentsunnchen·1d

We’re incredibly excited to work with @StevenDillmann & team on TB-Science - if you’re a scientist excited to drive how AI agents accelerate scientific workflows, please reach out!

📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇 tbench.ai/news/tb-scienc… @AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows. 1/6🧵

English

4

13

1.3K

Steven Dillmann@StevenDillmann·2d

RT @bradenjhancock: Lots of people say AI could/should be used to advance "science." What does that mean? What are the tasks that would act…

English