Tom Jansen

First results are in: Score for openai-codex/gpt-5.5 with xhigh thinking: 61.77% Run details: Generated: 100/100 tasks Scored: 100/100 tasks Judge: openai/gpt-4.1-mini Scoring mode: criterion-level Judge runs: 3 Median task score: 65.66% Zero-score tasks: 7 Generation timeouts: 2 tasks hit the 1-hour cap, but still had outputs and were scored This lands below the OpenRouter blog’s reported 69%, so it no longer looks suspiciously inflated. Cost/tokens: Pi generation: $612.60 Pi-reported token-equivalent cost, not necessarily your OpenAI bill since this used the openai-codex subscription route Judge cost: $12.07 estimated OpenAI API cost Generation tokens: 42,801,690 input, 1,759,182 output, 691,636,736 cache-read Judge tokens: 22,758,417 input, 828,911 output You'll notice that the judge model is GPT 4.1 mini here which is a choice GPT5.5 made "to save costs". It's not ideal, and the DRACO paper itself suggests using Gemini 3.1 Pro, GPT 5.2 or Sonnet 5.4, so before switching to the local model run, I'll re-run the judging with GPT 5.4. As judging via API pricing already was 12$ with GPT 4.1 mini, I'll use a little workaround for the GPT 5.4 run: pi , oh beloved pi with all your flexibility, can be used to run this via the Codex Subscription and you can just tell it so send requests without it's own system prompt, to mimick just calling the API. NICE !

@Klotzkette Technologisch haben sie irgendwo Recht, wobei man auch sagen muss dass Modelle wie GLM 5.2 definitiv gut genug sind und von Unternehmen dieser Größenordnung ziemlich problemlos "lokal" oder als "managed service" in einem Datacenter selbst betrieben werden könnten.

Ich sehe gerade konkret, wie viele sehr moderne und technologisch führende deutsche Unternehmen (nicht die allergrößten aber schon in der Nähe von Champions) breitflächig US-amerikanische KI einführen (nicht für Jura, sondern für viele andere Dinge), weil sie jetzt den Anschluss nicht verpassen und Ihre Mitarbeiter nicht verlieren wollen und keine Alternative aus Deutschland oder der EU finden, die leistungsfähig genug ist, was die Menge an zu verarbeiten den Daten und hohe Fähigkeiten der KI angeht. Eine KI wie Mistral wird nur als zu schlecht wahrgenommen, europäische Cloud-Lösungen ebenfalls. Und die sind nicht dumm, dieses Unternehmen, die gucken sich schon um. Ich berichte nur. Don’t kill the messenger. Ich kann es auch nicht ändern.

Quick update on that: I am running the first DRACO benchmark, GPT 5.5 xhigh with pi.dev as harness and GPT 5.5 xhigh as judge and synthesizer via OpenAI API directly (same benchmark as the one used by OpenRouter) as a baseline, then going into testing different combinations of local models, with either paid and/or local models for judge and synthesizing. Benchmarking takes way longer than I expected .... DRACO has 100 tasks and this first run is already running more than 24 hours, will take another 3 hours for sure. Also doing this only with API pricing would land at about 600$, but via the 200$ sub it's completely fine. Will keep you updated once it's finished. research.perplexity.ai/articles/evalu…

@ThePrimeagen It's really interesting to watch people like you and @unclebobmartin go through this whole process of "it's shit" - "NO, it's good!" - "Oh, no it's actually shit" - "I found a way" ... Hope this one brings you where you want to be 🤞

I have come up with a way for agents to produce code I like with "some hand holding" Feels pretty fun, see where I can take this

There is a workaround via github.com/fitchmultz/pi-… It will actually use cursor as a harness then, so pi becomes some kind of a meta harness. Maybe thats enough to play around, but some of the real magic actually lies in the super small system prompt which you wouldn't really get the taste of then.

I would have liked to give pi.dev a try, with models provided via cursor, but this doesn't seem to be an option atm. Or am I gettings something wrong @mitsuhiko ?

I know that @mattpocockuk /grill-me and /grill-with-docs Skills sometimes produce ridiculous amounts of questions, my record is 110 ish. But has anyone ever reported an agent saying "Done. I used the Grill With Docs workflow" And there were ZERO questions ?

@thorstenball 🤣🤣🤣

@thomasjansn That's in there, point #3: "Read things properly before you tweet comments." (Just kidding)

Some thoughts on ownership I shared with the team this morning

A great baseline blueprint for how work should be handled in general.

@thorstenball Fair. Reading properly should also be the baseline 😅

@thomasjansn It is, but see first sentence: I haven't explained it in a while and I was in the mood to write something.

@badlogicgames yes gramps, will do gramps

listen to your (dashing and vital looking) elders.

Nice! I used same approach for my pi.de extension called pi-fusion github.com/tjansn/pi-fusi…

@kskrygan ahm. How about @opencode or pi.dev ?

Now it’s official. JetBrains remains the last big independent player in developer tooling. Our job is to deliver the most cost-efficient, deeply integrated, and genuinely enjoyable AI experience across our IDEs and beyond. We’re on it. Stay tuned.

YES. My pi-fusion extension is exactly with that in mind: Using several local models in fusion mode to achieve SOTA deep research results. Benchmarking still tbd, but it's there and it works github.com/tjansn/pi-fusi…

Based on the @OpenRouter Fusion approach, I made an extenstion for pi.dev by @earendilworks @badlogicgames that lets you define your own set of fusion models, thinking-effort per model etc. so you can easily play around with it too. First tests give good results, but I didn't have any time for benchmarking yet, which I still intend to do -> github.com/tjansn/pi-fusi…

"Define what your application needs in secretspec.toml, then plug in keyring, 1Password, Vault, AWS, or any of 11 backends — same code, every environment." Very eager to look into this 👀

😻

@Simon__Grimm Somehow I want to point into the direction of Occams Razor and the "It is vain to do with more what can be done with less." but then I remember that Europe is not exactly in the "do with less" game 😅

An important fact! Full post: siliconcontinent.com/cp/202043931

@GergelyOrosz Some things (most actually) happening in the US atm are getting weirder by the day ... combined with that weird need to talk about "how bad europe has it" it gives me a really strange feeling of a system close to total collapse

So apparently after Meta leadership: - Force reassigned some of the best devs on teams to AI data labelling fulltime - Laid off another 10% - Started to record every dev’s screen in the US 24/7 They now realized that it has, indeed started to destroy their eng culture. And are now trying to walk back. All of the above was unprompted, not forced by anything external or even business reasons (Meta recorded record revenue, record profits) The biggest self-inflicted eng culture destruction I’ve seen in a matter of weeks

@shokhkarim1212 Hi! Right now I am building a pi.dev extension for the model fusion pattern that @OpenRouter described in their post. Still very much in the making but I am really excited to test it. Will post when it's done! openrouter.ai/blog/announcem…

Hey builders! 👋 Looking to connect with people building in: 🍽️ SaaS 🚀 Tech 📲 Automation 🧠 AI tools 📱 Product Development 🔥 Web APP 💻 Devs Drop what you're working on 👇 (and your link — I'll check it out)