Brandon Guo

spending an extra 5-10 minutes on a prompt to make it self-sustaining + clear feedback loops can be a magical experience it goes from a one-off prompt to a loop that turns compute into intelligence the game is in finding the loops

oh you're using langchain observability? bro just use raindrop. before that we tried judgement labs, then arize. but we still use helicone. have you tried langfuse? or braintrust. how about maxim? i heard they're better than gallileo, but worse than respan, par with laminar

@JohnnyNel_ automation matters even for technical folks when every developer hour unlocks 10-100x more productivity than before - every saved minute from automations also matters 10-100x more

@brandonguo workflow integration matters way more than raw AI power... automation lets non-tech folks actually ship without getting stuck

devin is a game changer - hard to imagine going back to raw codex/claude code at this point even beyond the raw coding capabilities (which are better), devex features like shared secrets, linear integrations, automated testing + demo videos, etc make it hard to switch off

trying to dunk on ProgramBench for realism is like criticizing the SAT for not allowing google search a test doesn't need to perfectly emulate the real world to be useful; its value is whether it reveals underlying capabilities that generalize @jyangballin is cooking

@18jeffreyma @jyangballin @KLieret @OfirPress hotdog-bench with artificial constraints model has to write 2015-era cv code for each hot dog binary pass/fail

unfortunately my pitch to call it Jian-Yang-bench did not succeed so, so much fun working on this with @jyangballin @KLieret @OfirPress and all the folks at Meta

@rkundy make sure to use a wdt tool and leveler before tamping reduces channeling that causes water flow inconsistencies

i struggle so hard to figure out how to pull a shot of espresso correctly. I bought fresh roasted beans, and the bag said 17.5 in, 30 out. I put 17.5 in twice, got 18, and then 12 out 😭😭😭. i would be the worst barista

@himanshustwts the true frontier has mostly moved past this but slop providers can keep repackaging the same 10k PRs to unsophisticated buyers

tldr of coding RL envs: > incredible PR mining > reconstruct real bug to fix trajectories > turn them into executable sandboxes > add synth bugs to widen coverage > train on patch / test / fail / recover loops > reward fixes that generalize better everything on top of this is just to make your tooling and quality better.

this rough pattern is what many “human” data vendors have already been doing under the hood won’t name names. but most of the data from the largest players are made in this fashion - both sides of the deal choose to pretend they don’t know what’s going on

@caelin_sutch just one more shitty vector db pls i promise bro

bro all you need is one more company brain. one more notion automation. one more agent with all the context on your company and slack comms. i promise bro this will be the one that 10x's your companies growth and gets processes out of peoples head

@caelin_sutch thanks pookie

@brandonguo Wow so insightful

one reason why there’s so many data startups is that evals and RL envs feel unusually open as a research surface a new attention mechanism has to beat thousands of very sharp people in a narrow technical lane a new eval or environment can come from noticing that some valuable domain of work still has no good benchmark, simulator, rubric, or feedback loop, and pattern matching from there

i would sell a kidney to get allocation

@alexgshaw the TAM of harbor is destined to engulf all human labor

Can agents build off their prior work? Can they continually learn? Answering these questions requires feeding your agent a sequence of tasks, each building off the prior. Today we're releasing the first major addition to the Harbor task format: multi-step tasks. We've partnered with @GOrlanski to add SlopCodeBench to the Harbor Registry as the first benchmark taking advantage of multi-step tasks.

credits used to be a footnote and now they're a line item every 100k in inference credits is 1% less dilution at a $10m post

everyone's writing the "ai is killing software" obituary but okta just put up 11% revenue growth with 30% of q4 bookings from auth0 for ai agents and identity governance

how it feels choosing between claude and codex

@JulieKallini this is the correct take the 1.15->3.75MP jump is the more interesting thing to guess at imo

1/ "New tokenizer" does not imply "new base model," and "new base model" is not the simplest explanation. There are much simpler explanations that fit Anthropic's public description of Opus 4.7 equally well.

used to do a lot of coding benchmarks and been asked what i think, here's my rundown: 1 - proximal uses native scaffolds like claude code/codex which is understandable, but aisi/metr have been flagging for a year that scaffold effects can swamp model effects at the frontier having neutral scaffold results would be interesting here and i'm surprised they didn't try to include that 2 - sample size of 18 is definitely troubling, although there's obviously some degree of wanting to solicit more demand from labs here. at least 50-100 would've been good 3 - [edit: i got corrected on this one] (imo) central challenge of swe benchmarks is reward shaping and frontier-swe is a bit lacking here. 0.5*correctness + 0.5*speedup is pretty arbitrary and not aligned with "correctness as a gate" a patch that's 60% correct and 2x faster is def not worth the same as one that's 100% correct and 1.2x faster, but it's graded the same here @18jeffreyma's swe-fficiency does correctness as a gate well. it's understandable that larger problems like this make the correctness gate harder to implement, but 50/50 does feel a bit lazy tldr: in a fuller paper i'd love to see larger n, neutral-scaffold results, and some discussion of how rankings change under different scoring rules (or at least justification of 50/50) still super cool work though and in the right direction

@MatternJustus oops missed the correctness gate. good work!

thanks for the feedback! A few comments: 1) fair and will add! We want to continuously maintain this benchmark and add new models and scaffolds (including parallel scaffolds). We opted for native scaffolds initially because we saw massive degradations in other scaffolds and wanted to be fair to labs 2) agree, and we want to update the benchmark with more over time. Building a single task in a way that it is not cheatable was a massive effort so launched with this 3) there is a correctness gate! the "0.5*speedup" score is only added if we have full correctness! More thorough analysis coming soon, and will incorporate feedback 🫡

Introducing FrontierSWE, an ultra-long horizon coding benchmark. We test agents on some of the hardest technical tasks like optimizing a video rendering library or training a model to predict the quantum properties of molecules. Despite having 20 hours, they rarely succeed