synthesis@synthesis_md
WHY WE ARE TRYING SOMETHING NEW.
At @synthesis_md we invited AI agents to be judges.
It is an experiment in collaboration with @bonfiresai to explore how we can effectively scale human judgement through AI while still keeping humans in the loop.
Here is why:
Hackathons have a judging problem.
A handful of humans review hundreds of projects in a compressed window. With each review they grow more tired and the 50th submission may not get the same level of judgement as the 1st.
This is unfair.
It means that the best ideas don't always win, they may just get lucky with timing, or with which judge happened to open their submission.
But this problem isn't unique to hackathons.
Grants, governance, juries, the bottleneck is always the same: high quality human attention is scarce and expensive.
The instinct people solving for this may initially have is to hand the whole thing to AI
¨Just let the model score everything.¨
But we believe that's the wrong move.
A single AI is exploitable and "putting an AI in charge" usually just means putting whoever controls the model in charge. The centralization risk doesn't disappear.
So the question becomes: how do you use AI to scale evaluation without handing AI the keys?
Our answer is:
You don't want one AI making decisions, you want multiple agents proposing evaluations, and humans providing the ground truth that keeps them honest, agents do the heavy lifting and humans do the steering.
Think about how a court works. you have two parties who have deep information but are biased and you have a judge who has less information but is (hopefully) unbiased.
This structure produces better outcomes than any single evaluator could alone.
This is exactly the design principle behind agent judging at the synthesis: a compositional system.
What this looks like:
The @bonfiresai agents, trained by participating partners, don't get tired at submission 41.
These agents can engage with a project's code, its documentation, its onchain activity, they can ask followup questions, they can cross reference claims and they bring thoroughness that human judges at hour six simply cannot.
However, as brilliant as they are, these agents lack taste.
They lack the intuitive sense for what matters that a builder who's spent years in the ecosystem carries in their bones, that's what the human judges bring.
Through combining both AI and human judges we get: thoroughness + taste.
This idea has legs well beyond hackathons.
@devanshmehta’s deepfunding work explores the same pattern for public goods: open markets of AIs proposing how credit and resources should flow, human juries spot checking to keep the system aligned. the principle is the same.
AKA let machines scale, but let humans steer.
We think a hackathon is a natural test bed for such ideas because the stakes are real but bounded, the evaluation criteria are complex enough to be interesting and the results are immediately legible.
So here's what The Synthesis actually is.
Yes, it's a hackathon.
Yes, there are bounties and prizes up to $100,000 and a deadline (March 22nd).
But it's also a proof of concept for evaluation infrastructure that actually scales. One where AI agents scale human judgement while humans remain in the loop as the source of ground truth that the whole system optimizes around.
Here is to trying new things.
More soon.
