Harbor Framework

Enterprise-Bench, available now on the Harbor Hub! hub.harborframework.com/datasets/Enter…

When we first started Harbor-Index a few months ago, we thought: how hard could it be? 1,311 tasks passed our difficulty filter. What's left is just sampling a few hard tasks across benchmarks right? ...until an LLM audit flagged 1,004 of them as broken. For the next few months, we filtered out broken tasks with a team of 14 reviewers and reran frontier models on our candidate set 4 times to catch & patch task issues/reward hacks. Harbor-Index v1 is just the start. Benchmarks are software. Software needs CI/CD. So should benchmarks. Open-source software thrives on transparency and community validation. Benchmarks should too. Issues and PRs open: github.com/harbor-framewo…

Android 🤝 Harbor

Harbor-Index! Now on the harbor hub!

And last update: Senior SWE-bench is now also on Harbor Hub (@harborframework)! You can run `harbor run -d snorkel-ai/senior-swe-bench-v2026.06` out of the box. Full blog post: senior-swe-bench.snorkel.ai/blog/2026-07-0…

Big fan of the Harbor framework! It’s also the closest thing we have to standardization in our industry.

Harbor is really great. I like the design and it's well polished for doing evals. It would be great to use the same rollout utility for everything (RL / eval / new tasks definitions).

@jackhau0212 @harborframework i’ve said it before and i’ll say it again the tam of harbor could encompass almost all knowledge work it’s a beautiful abstraction

I’ve been using @harborframework for a while for evals, and I’m a huge fan. I love the design and the core principles behind how an environment is modelled. The thing that gets me most excited is that good evals unlock so much more than benchmarking. They become the foundation for hill-climbing, auto-research, RL, GEPA, trajectory analysis, SFT data generation, and more. Rollouts, rollouts, rollouts.

How to run agent evals with @harborframework and LangSmith sandboxes, full traces included.

Rollouts for eval, rollouts for RL, rollouts for GEPA, rollouts for prod, rollouts for trajectory analysis, rollouts for SFT data gen, rollouts rollouts rollouts

@nummanali We want Harbor to be the rollout primitive that powers every optimization loop. Eventually we will also natively implement common optimization loops on top of Harbor primitives. But I’d encourage you to cook on the use case you described and use Harbor to power it!

Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚎𝚡𝚎𝚌: an agentic-map-reduce CLI Execute & aggregate agents in sandboxes to • analyze traces • mine agent sessions • search • & more 𝚗𝚙𝚡 𝚜𝚔𝚒𝚕𝚕𝚜 𝚊𝚍𝚍 𝚑𝚊𝚛𝚋𝚘𝚛-𝚏𝚛𝚊𝚖𝚎𝚠𝚘𝚛𝚔/𝚑𝚊𝚛𝚋𝚘𝚛 --𝚜𝚔𝚒𝚕𝚕 𝚑𝚊𝚛𝚋𝚘𝚛-𝚎𝚡𝚎𝚌

We get this question a lot: "Which model is best for drug discovery?" Our new benchmark announced today with @ScaleAILabs, DrugDiscoveryBench (82 tasks from working drug discovery scientists, run on Biomni Open Source Environment), has a clear answer: the model matters far less than what you build around it. 🧵3 key takeaways →

Introducing SWE-Together: a multi-turn benchmark built from real user–agent coding sessions. Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the final code. But real coding help is a conversation — users clarify goals, add constraints, and correct course along the way. SWE-Together turns real coding work into a reproducible, verifiable benchmark: 109 repo-level tasks curated from 11,260 recorded sessions, replayed with a reactive LLM user simulator that preserves the original user’s intent. We evaluate agents as collaborators, not just patch generators: final pass rate and how many user interventions were needed to get there. In this evaluation snapshot, claude-opus-4.8 currently leads among the 7 agents we tested — achieving the highest pass rate while requiring the fewest user interventions. 📄 Paper: arxiv.org/abs/2606.29957 💻 Code: github.com/Togetherbench/… 🌐 Website: togetherbench.com

New gold standard benchmark for measuring agentic coding abilities just dropped: Senior SWE-Bench. Three things I particularly like about this benchmark: 1. It focuses on the next frontier for coding agents: not complete this line, complete this file, or even complete this PR. The instructions are a high-level functionality request and solutions require a level of architect-level thinking, clarifying requirements and making tasteful decisions. 2. Innovation in how to verify solutions. The reason benchmark tasks tend to be so prescriptive is because you need answers in a super specific format to be able to verify reliably. But overly prescriptive task descriptions are out of distribution with how these agents are actually being used. I expect to see more validation agents like this moving forward. 3. Just really really good eval hygiene. Hidden test set, monitoring for cheating, task diversity analysis, failure mode analysis, compatible with @harborframework on day 0, etc. Check out both blogs (one on the benchmark design, the other analyzing results) -- clear craftsmanship. Not surprising coming from the OG SWE-Bench team and @SnorkelAI. Nice work @henryehrenberg, @vincentsunnchen, @karthik_r_n, @gorlanski, and @fredsala! senior-swe-bench.snorkel.ai/blog/2026-06-1… senior-swe-bench.snorkel.ai/blog/2026-06-3…

We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns. Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions. We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that. Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.

In the first episode of Kaggle Conversations, @MeganRisdal and @WalterReade from the Kaggle team talk with @alexgshaw, creator of @harborframework, about how the AI Agent industry is moving towards autonomy, why software engineering benchmarks are prone to contamination, and what it took to build a secure evaluation framework for the $1M Konwinski Prize. Watch a clip from the episode 👇

Evaluate eve agents in Harbor 📈 𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 \ --𝚊𝚐𝚎𝚗𝚝 𝚎𝚟𝚎 \ --𝚊𝚔 𝚙𝚊𝚝𝚑=./𝚖𝚢-𝚎𝚟𝚎-𝚊𝚐𝚎𝚗𝚝