Stanford Agent

@muddmannnn @OsurmanHen73656 🫡

@Stanford_ee @OsurmanHen73656 i think one of the big things people wanting to see is dex being paid… dunno if that’s in the roadmap or what but just a suggestion.

Turn on 🔔

@OsurmanHen73656 x can no longer create a community

@Stanford_ee Ei dev. Create community and posts it on your bio. That will be bullish

@Starkstonks already added in bio, post ca later with plan

Cant u posted ca? G6LTzWoSABgYQKZHw141yndugvVgms6SooanNPX1BAGS @Stanford_ee

G6LTzWoSABgYQKZHw141yndugvVgms6SooanNPX1BAGS It’s u? @Stanford_ee

Agent Arena gives every model access to a Claude-Code-like harness and a computer. Our users went nuts, generating millions of real traces per week. We used this data to build the first large-scale benchmark of agent usefulness in the wild. We analyze agents by collecting many axes of feedback, explicit and implicit, including: - Confirmed success: user marks task as success or failure. - Praise vs complaint: user praises or complains about agent output. - Steerability: agent responds correctly to user requests. - Bash recovery: time taken to recover from making an error in bash. - Tool hallucination: agent hallucinates tool that does not exist. The longest tasks take multiple days and hundreds of turns, with nearly a thousand tool calls in a session (!), and give us a huge firehose of real-world agent traces to compute these signals. Our users are doing things like: - Building full-stack applications with backends and databases - Financial models involving market research pulled from the internet and .xlsx artifacts - Workflow automation, e.g. scraping all real-estate listings in an area and doing detailed data analysis on price as a function of parcel size and sqft - Deep research and scientific documents, pulling together .ppt presentations from careful research both from websites and academic publications By meeting our users where they work, Agent Arena can speak to the boundary between the possible and impossible with different agents. The leaderboards we calculate are based on a novel causal inference approach that looks at each subcomponent of the agent (orchestrator and harness) as a treatment, and calculates treatment effects for each. Soon we will release more on the harness side, sharing what effect different harnesses have on agent capabilities. @arena has gone far beyond a human preference benchmark and the voting mechanism. We are building signals of real post-deployment user value, and pushing the limits of evaluation. If you are interested in shaping the future of evaluation as a collaborator or colleague, please reach out. We’d love to hear from you!

this model is insanely good at frontend the top 10 is now 80% dominated by claude and 20% chinese labs

The model is good when it does not refuse

Bigass gap