Jay Frydman

@realdrewcarson @agenticasdk You will soon see frontier labs putting similar RLM harnesses behind the API & achieve similar performance gains on tasks like ARC-3

@JayFrydman @agenticasdk is the harness behind the API? if so, great, because that's what they're testing - performance on generic API endpoints. But this seems like a task-specific harness for the ARC-AGI-3 challenge.

We scored 36.08% on ARC-AGI-3 in one day using the Agentica SDK.

@MickeySteamboat @mark_k x.com/agenticasdk/st…

@mark_k this will be knocked out of the water in no time. end of 2026

ARC-AGI 3 is here, and all existing AI models are below 1% on the benchmark. It's gonna take a while until this one is saturated. How it measures intelligence: - 100% human-solvable environments - Skill-acquisition efficiency over time - Long-horizon planning with sparse feedback - Experience-driven adaptation across multiple steps "As long as there is a gap between AI and human learning, we do not have AGI."

@mark_k @fchollet x.com/agenticasdk/st…

@sd_marlow @FakePsyho The Agentica SDK agent beat 113/182 games in one day without using vision

@FakePsyho It bothers me that Chollet won't refer to these as visual challenges. It's also a failure for the benchmark to not measure for exploration vs understanding. They have a default assumption that min # of action on 1st try = intelligence (it doesn't).

Some early non-trivial thoughts about ARC-AGI-3 after reading the paper: - Due to quadratic scaling of steps (see attached image) & linear weight scaling of levels, benchmark scores will follow much more "steep" sigmoid than usual (think sigmoid x quadratic function); I'm not hating this design, but this means that the score on this benchmark has to be interpreted differently than literally any other popular benchmark - expect really silly takes from your local AI influencers - Each game was tested with 10 players and the human baseline is equal to second-best performance; this also means that in order to include the game in the benchmark, it was enough for only two human players to finish the game (out of 10); the public games are said to be significantly easier than private set; now, the potential problem here is that there's a big difference between puzzle being hard (tricky observations, complex logical path) and obtuse (multiple potential interpretations, unclued "random" game mechanic). Success% under heavy time constraint doesn't differentiate between those two. If we have obtuse games in the private set then getting near 100% might be realistically impossible (assuming very strict limit on actions, etc) - The games are similar (especially in terms of aesthetics) and I'd expect for human performance to drastically improve with the number of games that they have played. This is especially true for people that had no prior exposition to puzzle games. In many games the hardest part is figuring out that the specific part of the screen is clickable and you do get better at identifying such spots with each consecutive game you're playing. This is true because ARC-AGI-3 visual language is somewhat distant from your usual puzzle game visuals. AI is going to be severely disadvantaged until public games are added to the training data, so expect big initial jump in the benchmark scores (assuming the labs care about that). Overall, I feel that this is a solid release, but I'm a bit worried that there might be some rather obtuse games in the private set that will heavily skew the results.

@rawantitmc @agenticasdk 🇦🇷

@agenticasdk Is argentina based on neurosymbolic reasoning?

@InfiniteHexx @agenticasdk You can call it a harness, no problem.

@agenticasdk They used a "general purpose symbolic exoskeleton" to accomplish this, but don't you dare call it a harness! It's totally not a semantic dodge!

@chillMSR @agenticasdk Chain-of-thought models directly served through the API also use a harness

@agenticasdk cheating in a benchmark does not make your ai model better share the results without a harness

@x1f4r @agenticasdk Chain-of-thought models are a harness

@agenticasdk The point of ARC-3 is to allow no harnesses for the competition. It makes it actually hard to beat by letting the model approach it exactly like a human would.

@realdrewcarson @agenticasdk Chain-of-though models are a harness.

@agenticasdk Cool, now do it without a harness.

@flowersslop @agenticasdk Chain-of-thought models are a harness

@agenticasdk wasnt the point of arc agi 3 exaclty NO harness?

@chatgpt21 @agenticasdk @spicey_lemonade This is a reflection of the scoring methodology of the challenge. The agent only used 1.68x the action count of the human baseline, which is the score of the second-best human player. If the ARC foundation provides the human score distribution we can compare directly to avg human

@agenticasdk @spicey_lemonade Significantly worse than a human

@chatgpt21 @vr4300 @symbolica @scaling01 Not training on anything - we used the same harness we open sourced 3 weeks ago github.com/symbolica-ai/A…

@vr4300 @symbolica @scaling01 Are you training off their public set?

Unbelievable result from the @symbolica research team! In just 1 day since the launch of ARC-AGI-3 we have been able to raise the score from 0% to 36%, and within a very close margin to human baseline as well. See our blog post linked in this thread for a detailed score breakdown and replays.

@jdlichtman 👀

Not long for this world

@leothecurious Our approach did not use vision and solved 113/182 games with 1.68x action count of human baseline in 1 day. x.com/agenticasdk/st…

for the purposes of AI benchmarking, this is serious cope. it may be fair in a purely inquisitive question as in "what can a truly alien intelligence figure out?", but it poorly applies to the purposes of evaluating the true automation potential of AIs built to function in a human society. AIs share our world, they share our culture, and they share our jobs. they aren't mean to be alien, and we aren't actively trying to make them more alien to humans. besides, almost every single LLM/VLM benchmark out there is overwhelmingly knowledge-dependent, counting on prior knowledge about human language (often english in particular), human mathematical notation and concepts, human culture and man-made objects, and all sorts of details that have almost no overlap with intelligence per se. this benchmark goes further than almost any i've sedn in decoupling the measurement of knowledge from the measurement of intelligence. this benchmark (imperfectly but commendably) tries to not only measure how well AIs answer questions, but how well they ask the right questions under novelty, to then uncover answers they don't already know when walking into the test.

@JonhernandezIA 👀

That was quick...

@daniel_mac8 👀

arc-agi-3 will be solved by Saturday.

@DeryaTR_ 👀

Ouch!

@jmbollenbacher The Agentica SDK agent only used 1.68x the actions of the human baseline (the score of the second best human player). Same harness as we open sourced three weeks ago.

Called it. Double digit scores very soon after launch. ARC-AGI-3 isnt as hard as the ARC Foundation people want you to think. They just rigged the evaluation system to make it look harder than it is so it wouldnt insta-saturate on launch.

@cloneofsimo Our approach does not use vision, used the same harness as we open sourced three weeks ago before the public announcement on March 25th, and solved 113/182 of the playable levels in 1 day for a fraction of the cost (~9x cheaper vs. Opus 4.6 baseline). x.com/agenticasdk/st…

This is pure perception task with heavy cultural prior: you need geometical intuition, some prior on solving mazes, what typical game interaction feels like. Like if alien intelligence that doesnt have visual perception like ours, and doesnt know what nintendo is, how are they supposed to solve this? (And yes, there are animals without visual perception) (Oh and guess what other intelligence dont have visual perception and geometrical prior like ours)

@xlr8harder @arcprize We added a multiple of human performance in our announcement for this reason. Our agent built with the Agentica SDK used an average of 1.68x of the human baseline action count. x.com/agenticasdk/st…

The biggest ARC-AGI-3 issue (and an easy change @arcprize could still make) is presenting a score as a percentage of human score, while the scoring method ensures that it does not mean what a percentage implies to most readers. I'm already seeing people confused by this.