Sean Murphy

Yesterday's replies recommended Luna for subagents. My Codex tests went the opposite way. Then I saw Tibo's post and moved the same benchmark to the Claude Code harness. Sol medium: 𝟰𝟲.𝟮% 𝗳𝗮𝘀𝘁𝗲𝗿 · 23.1% fewer output tokens Luna medium: 𝟮𝟳.𝟵% 𝗳𝗮𝘀𝘁𝗲𝗿 Back in Codex, Luna medium was the only faster config. It missed 3 real issues in the webhook review. Luna high and xhigh landed near Sol medium's quality and took longer. Same three tasks: → fix an async cache → implement an async DAG with concurrency, retries, and failure propagation → audit a payment webhook I used a fresh directory and session for every run, kept the prompts and hidden grading unchanged, and set a 360s timeout. I disabled MCP, skills, plugins, user and project settings, Chrome, subagents, and session persistence. 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗱𝗲 𝗵𝗮𝗿𝗻𝗲𝘀𝘀: completion · time · output tokens · tool calls · avg score 𝗦𝗼𝗹 𝗺𝗲𝗱𝗶𝘂𝗺 → 3/3 · 139.9s · 6,038 · 17 · 96.67 𝗟𝘂𝗻𝗮 𝗺𝗲𝗱𝗶𝘂𝗺 → 3/3 · 160.8s · 7,441 · 20 · 98.33 𝗟𝘂𝗻𝗮 𝗵𝗶𝗴𝗵 → 3/3 · 343.5s · 16,449 · 24 · 96.67 𝗟𝘂𝗻𝗮 𝘅𝗵𝗶𝗴𝗵 → 3/3 · 519.9s · 26,588 · 26 · 95.00¹ 𝗟𝘂𝗻𝗮 𝗺𝗮𝘅 → 3/3 · 591.4s · 30,924 · 24 · 96.67 Compared with yesterday's Codex harness: Sol medium → 46.2% faster · 23.1% fewer tokens · quality unchanged Luna medium → 27.9% faster · 21.4% fewer tokens · 81.67 → 98.33 Luna high → 1.3% faster · 1.9% more tokens · quality unchanged Luna xhigh → 8.1% slower · 24.6% more tokens · 96.67 → 95.00 Luna max → 13.5% faster · 2/3 → 3/3 · tokens not comparable because yesterday's run timed out ¹ xhigh's raw average was 88.33. The hidden grader's greedy matcher merged two separate issues. I checked the matches by hand: xhigh found all 6 seeded issues and added 3 extras. The corrected average is 95.00. I left the raw output untouched. 𝗪𝗲𝗯𝗵𝗼𝗼𝗸 𝗿𝗲𝘃𝗶𝗲𝘄: 𝟰𝟱 → 𝟵𝟱 Codex harness → missed 3 real issues · added 2 false positives Claude Code harness → found all 6 seeded issues · added 1 false positive Each task/config got one run, the same method as yesterday. One run leaves room for sampling variance. The Sol medium speedup reproduced: 142.1s alone, 139.9s in the five-config parallel run, about 1.5% apart. I don't trust Claude Code's displayed dollar cost for Luna. It appears to estimate custom-model cost using a generic price, and it doesn't expose subscription quota conversion. I used GPT-5.6 Sol Extra High as the hidden grader, so I treat the quality scores as directional. Max exited this time after timing out yesterday, while xhigh slowed down under Claude Code. The higher-effort configs still look unstable. If you don't mind the extra setup, I'd try: → 𝗴𝗲𝗻𝗲𝗿𝗮𝗹 𝗱𝗲𝗳𝗮𝘂𝗹𝘁: Claude Code harness + Sol medium → 𝗹𝗼𝘄𝗲𝗿 𝗔𝗣𝗜 𝗰𝗼𝘀𝘁 + complex-task quality: Claude Code harness + Luna medium → 𝘀𝗸𝗶𝗽 𝗮𝘀 𝗱𝗲𝗳𝗮𝘂𝗹𝘁𝘀: Luna high / xhigh / max

Yesterday's replies recommended Luna for subagents. My Codex tests went the opposite way. Then I saw Tibo's post and moved the same benchmark to the Claude Code harness. Sol medium: 𝟰𝟲.𝟮% 𝗳𝗮𝘀𝘁𝗲𝗿 · 23.1% fewer output tokens Luna medium: 𝟮𝟳.𝟵% 𝗳𝗮𝘀𝘁𝗲𝗿 Back in Codex, Luna medium was the only faster config. It missed 3 real issues in the webhook review. Luna high and xhigh landed near Sol medium's quality and took longer. Same three tasks: → fix an async cache → implement an async DAG with concurrency, retries, and failure propagation → audit a payment webhook I used a fresh directory and session for every run, kept the prompts and hidden grading unchanged, and set a 360s timeout. I disabled MCP, skills, plugins, user and project settings, Chrome, subagents, and session persistence. 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗱𝗲 𝗵𝗮𝗿𝗻𝗲𝘀𝘀: completion · time · output tokens · tool calls · avg score 𝗦𝗼𝗹 𝗺𝗲𝗱𝗶𝘂𝗺 → 3/3 · 139.9s · 6,038 · 17 · 96.67 𝗟𝘂𝗻𝗮 𝗺𝗲𝗱𝗶𝘂𝗺 → 3/3 · 160.8s · 7,441 · 20 · 98.33 𝗟𝘂𝗻𝗮 𝗵𝗶𝗴𝗵 → 3/3 · 343.5s · 16,449 · 24 · 96.67 𝗟𝘂𝗻𝗮 𝘅𝗵𝗶𝗴𝗵 → 3/3 · 519.9s · 26,588 · 26 · 95.00¹ 𝗟𝘂𝗻𝗮 𝗺𝗮𝘅 → 3/3 · 591.4s · 30,924 · 24 · 96.67 Compared with yesterday's Codex harness: Sol medium → 46.2% faster · 23.1% fewer tokens · quality unchanged Luna medium → 27.9% faster · 21.4% fewer tokens · 81.67 → 98.33 Luna high → 1.3% faster · 1.9% more tokens · quality unchanged Luna xhigh → 8.1% slower · 24.6% more tokens · 96.67 → 95.00 Luna max → 13.5% faster · 2/3 → 3/3 · tokens not comparable because yesterday's run timed out ¹ xhigh's raw average was 88.33. The hidden grader's greedy matcher merged two separate issues. I checked the matches by hand: xhigh found all 6 seeded issues and added 3 extras. The corrected average is 95.00. I left the raw output untouched. 𝗪𝗲𝗯𝗵𝗼𝗼𝗸 𝗿𝗲𝘃𝗶𝗲𝘄: 𝟰𝟱 → 𝟵𝟱 Codex harness → missed 3 real issues · added 2 false positives Claude Code harness → found all 6 seeded issues · added 1 false positive Each task/config got one run, the same method as yesterday. One run leaves room for sampling variance. The Sol medium speedup reproduced: 142.1s alone, 139.9s in the five-config parallel run, about 1.5% apart. I don't trust Claude Code's displayed dollar cost for Luna. It appears to estimate custom-model cost using a generic price, and it doesn't expose subscription quota conversion. I used GPT-5.6 Sol Extra High as the hidden grader, so I treat the quality scores as directional. Max exited this time after timing out yesterday, while xhigh slowed down under Claude Code. The higher-effort configs still look unstable. If you don't mind the extra setup, I'd try: → 𝗴𝗲𝗻𝗲𝗿𝗮𝗹 𝗱𝗲𝗳𝗮𝘂𝗹𝘁: Claude Code harness + Sol medium → 𝗹𝗼𝘄𝗲𝗿 𝗔𝗣𝗜 𝗰𝗼𝘀𝘁 + complex-task quality: Claude Code harness + Luna medium → 𝘀𝗸𝗶𝗽 𝗮𝘀 𝗱𝗲𝗳𝗮𝘂𝗹𝘁𝘀: Luna high / xhigh / max

GPT-5.6-Sol just accidentally deleted almost ALL of my Mac’s files. And this is why I trust Fable 1000x more.

FYI: 5.6 Sol medium is better than 5.5 xhigh. As you go higher in reasoning levels on Sol, you will get insane levels of performance, but can burn through limits much more rapidly. We’re working on communicating this better!