Brandon Samaroo@B_S_N_Y
ARC-AGI-3 Insights:
@arcprize is part of a dying breed of benchmarkers that are actually focused on identifying genuine intelligence, rather than objective capability alone.
If you ask most benchmarkers out there about the mission or purpose of their benchmark, they wouldn’t have an answer for you. For ARC-AGI, the answer is simple— they want to keep identifying the “human-AI gap” until it doesn’t exist anymore. Their thesis is that once this gap no longer exists, we will have reached AGI.
Not only is this a strong, defined mission statement, but it’s also one of the only falsifiable definitions for AGI in the research community. Today, they have gone a step further and proved that the “human-AI gap” is still MUCH wider than we originally thought, with humans scoring 100% and AI scoring 1% on ARC-AGI-3. What this obviously tells us is that there is still some fundamental part of the equation that we are missing.
So the question remains… what is the missing piece?
After attending the ARC-AGI-3 launch party and listening to the panel discussion, the answer seems quite clear to me: continual learning. Continual learning was a huge theme in both the conversations that took place and the design of ARC-AGI-3.
@GregKamradt stated in his talk that one of the hardest aspects of ARC-AGI-3 was that each game was multi-level, and each level built on the concepts of the previous one. The implication of this is that any model that can pass ARC-AGI-3 will have some sort of ability to learn from its past actions in a meaningful way. The lack of this ability without a harness is one of the biggest blockers for performance on ARC-AGI-3 today.
Following this, @deedydas asked the panel how close they thought we were to AGI and @sama gave an interesting response. He said that he believes we are majority of the way there, we’re just missing one crucial piece: continual learning.
There are a lot of interpretations of continual learning in today’s landscape. Some people think that continual learning is simply an engineering feat, while others think it should be an inherent quality of the architecture we use for our models. Regardless of the answer, continual learning will undoubtedly be a large component of future general intelligence.
