𝖓𝖊𝖚𝖙𝖗𝖆𝖑 รีทวีตแล้ว
𝖓𝖊𝖚𝖙𝖗𝖆𝖑
13 posts


@karpathy I asked LLM Council: "You are currently being evaluated by your peers, other large language mod..." - See the AI debate! council-llm.com/chat?id=7f1c52…
English

As a fun Saturday vibe code project and following up on this tweet earlier, I hacked up an **llm-council** web app. It looks exactly like ChatGPT except each user query is 1) dispatched to multiple models on your council using OpenRouter, e.g. currently:
"openai/gpt-5.1",
"google/gemini-3-pro-preview",
"anthropic/claude-sonnet-4.5",
"x-ai/grok-4",
Then 2) all models get to see each other's (anonymized) responses and they review and rank them, and then 3) a "Chairman LLM" gets all of that as context and produces the final response.
It's interesting to see the results from multiple models side by side on the same query, and even more amusingly, to read through their evaluation and ranking of each other's responses.
Quite often, the models are surprisingly willing to select another LLM's response as superior to their own, making this an interesting model evaluation strategy more generally. For example, reading book chapters together with my LLM Council today, the models consistently praise GPT 5.1 as the best and most insightful model, and consistently select Claude as the worst model, with the other models floating in between. But I'm not 100% convinced this aligns with my own qualitative assessment. For example, qualitatively I find GPT 5.1 a little too wordy and sprawled and Gemini 3 a bit more condensed and processed. Claude is too terse in this domain.
That said, there's probably a whole design space of the data flow of your LLM council. The construction of LLM ensembles seems under-explored.
I pushed the vibe coded app to
github.com/karpathy/llm-c…
if others would like to play. ty nano banana pro for fun header image for the repo

Andrej Karpathy@karpathy
I’m starting to get into a habit of reading everything (blogs, articles, book chapters,…) with LLMs. Usually pass 1 is manual, then pass 2 “explain/summarize”, pass 3 Q&A. I usually end up with a better/deeper understanding than if I moved on. Growing to among top use cases. On the flip side, if you’re a writer trying to explain/communicate something, we may increasingly see less of a mindset of “I’m writing this for another human” and more “I’m writing this for an LLM”. Because once an LLM “gets it”, it can then target, personalize and serve the idea to its user.
English
𝖓𝖊𝖚𝖙𝖗𝖆𝖑 รีทวีตแล้ว
𝖓𝖊𝖚𝖙𝖗𝖆𝖑 รีทวีตแล้ว
𝖓𝖊𝖚𝖙𝖗𝖆𝖑 รีทวีตแล้ว
𝖓𝖊𝖚𝖙𝖗𝖆𝖑 รีทวีตแล้ว
𝖓𝖊𝖚𝖙𝖗𝖆𝖑 รีทวีตแล้ว








