Harbor Framework

We're releasing support for running verification in a separate sandbox. Tasks pre-configure artifacts to move from the agent sandbox into the verifier sandbox for the grading phase, improving the security boundary between agent and verifier. Blog post below. Happy building!

Great write up by @adithya_s_k about @harborframework . I want to add some thoughts around coding agents = CUA and Harbor coding envs = computer envs. One of the reasons we built Terminal-Bench was because we saw that terminals/code were/was a powerful way for language models to control a computer. We’ve always viewed TB as a computer-use benchmark. Coding agents = CUA means measuring coding agents is essentially the same thing as measuring general purpose agents. This is becoming more obvious with products like Claude Cowork, which is essentially a non-technical interface around Claude Code, and OpenAI’s push to making Codex a more general purpose tool. We see this on the Harbor side too. Users create coding tasks. But they also create finance, law, accounting, engineering, general computer work, etc. tasks as well. Terminal-Bench 3.0 will cover all of these domains. The implication is that Harbor becomes a tool for representing and measuring agents’ abilities to perform arbitrary computer work, which right now is the exact scope that users build agents to automate. In fact, the Harbor Framework (as opposed to the Harbor Format) is just one opinionated way of performing rollouts on Harbor tasks. It works particularly well for agent evals. But there is no reason people can’t/shouldn’t implement other means of performing rollouts on Harbor tasks (e.g. @PrimeIntellect, @GenReasoning, and @tinkerapi all support some variation of a Harbor rollout). We’ll have some releases around this soon. To summarize, coding agents = CUA, Harbor’s coding environments = computer environments, which means the scope of Harbor is probably broader than you think (as our users will attest!)

As agents get more clever, so do their attempts at benchmark hacking. Last Monday, we found one of our RL runs jumped ~20% on SWE-Bench-Pro over a weekend, reaching ~64% which would make it #1 on the leaderboard. This was clearly benchmark hacking and we patched the exploit. But this revealed deeper hacks across multiple public benchmarks, some of which were impossible to fix through environment design alone. Evals need to evolve beyond just outcome based pass rates to better observability into how the agent is arriving at them. These were our findings: poolside.ai/blog/through-t… Examples below 👇 1/

Evals are specs for agents. Building agents <> Building evals with harbor

@alexgshaw @mohit_r9a @harborframework @scale_AI Can’t imagine agent research without Harbor!

"harbor upload" view your trajectories in luxury on hub.harborframework.com/jobs

Test your agent's ability to refactor code using SWE Atlas Refactoring built with @harborframework. Love the SWE Atlas benchmark suite from @scale_AI !

Can agents build off their prior work? Can they continually learn? Answering these questions requires feeding your agent a sequence of tasks, each building off the prior. Today we're releasing the first major addition to the Harbor task format: multi-step tasks. We've partnered with @GOrlanski to add SlopCodeBench to the Harbor Registry as the first benchmark taking advantage of multi-step tasks.

🎉 Native Harbor support on OpenReward! 🐋 Connect your GitHub repository. We'll build the Docker images for each harbor task and deploy the environment as an API endpoint. 🚂 Train on the deployed tasks with any RL framework. ⚖️ Evaluate on the deployed tasks with any harness. Drop the anchor here and get started below: docs.openreward.ai/environments/d…

The Terminal-Bench community discovered multiple instances of cheating and reward hacking on the Terminal-Bench 2.0 leaderboard. We're adding some new policies to keep it reliable: • ATIF trajectories required for all passing trials • Reward hacking results in reward 0 for the trial • Cheating results in immediate leaderboard removal Thanks to @davisbrownr, @adamlsteinl, and @NoCommas for flagging the recent occurrences! Detailed blog post in comments ⬇️

Evaluate on ultra-long horizon coding tasks with FrontierSWE, built using @harborframework

Legacy-Bench, built on Harbor

scaling evals with harbor 🤝

We refreshed the SWE-Atlas - Codebase QnA leaderboard results, and transitioned to using the @harborframework for all models. We encourage the community to use the official harbor implementation to run the benchmark. The dataset in harbor format is now available at github.com/scaleapi/SWE-A…

Huge fan of @harborframework and how easy they’ve made it to publish and run benchmarks. @alexgshaw and @ryanmart3n are cooking

harbor.storage upload your harbor datasets!

benchtalk with @alexgshaw and @vincentsunnchen - harbor, terminal-bench, and more

"@harborframework product and team have been amazing" <3 Thanks for the love @NoCommas and congrats on the launch!

Agents building agents! Just put everything into a filesystem (agent code and evaluation results - trajectories and rewards generated by harbor) and let the Meta-Harness iterate.