André Ratzenberger

Codex skill evals get weird when your own setup leaks into the test. Old rules, personal skills, plugins, memory... bamboozled. So I made codex-naked: $ codex-naked Fresh HOME. Fresh CODEX_HOME. Just enough auth. github.com/AndreRatzenber…

@effi288 Ja leck. Mit nur 20 Leuten. Und probably ein Budget das keine Fehler erlaubt. Hört sich nach vielen kurze Nächte an! Respekt! Was war aus technischer Sicht die größte Herausforderung?

📄 Releasing the Soofi S pretraining tech report: a sovereign, open foundation model for German and English Today we’re publishing the full pretraining tech report and project page for Soofi S 30B-A3B — a Mixture-of-Experts hybrid Mamba model trained on ~27 trillion tokens with deliberately up-weighted German. What’s in the report: 🏆 Strongest fully open model in our evaluations on BOTH the English and German aggregates — ahead of Olmo 3 32B and Apertus 70B (full methodology in the report) 📋 Radical transparency: complete per-source data accounting, all hyperparameters, training + eval code, checkpoints — everything under permissive licenses 🇩🇪 Trained end-to-end on Deutsche Telekom’s Industrial AI Cloud in Munich — sovereign AI infrastructure on German soil Soofi S combines frontier-level capability with the highest measured aggregate long-context decode TPS, and unlike full-attention dense baselines maintains high throughput as context grows. The figure plots Capability Index versus measured aggregate decode TPS/GPU at 40K context and batch 32. The Capability Index averages five benchmark groups, i.e., Code, GSM8K, GPQA-Diamond, English aggregate, and German aggregate, after normalizing each group to the best plotted model. Aggregate decode TPS/GPU is measured with a TP=1, one-B200 vLLM latency-subtraction protocol.

When you tell the bot to try out some new open weight image edit model and the bot knows where you hide the images of your cats... #boogu #codex

@lucas_alvarez91 @shannholmberg Good SDD already contains the loop. What changes with agents is that the loop becomes an executable object, not only a process humans remember. The useful work is making the spec, evidence, stop reasons and trace explicit enough for an agent to run against.

@shannholmberg How is this different from a classic SDD (Spec Driven Development) pipeline? I mean, that's a loop already. You have a main orchestrator which delegates into subagents for init, explore, design, plan, tasks, implementation, tests, archive, etc. Right?

what is agent looping for the last two years we prompted agents one task at a time. that is starting to change instead of asking an agent to build the landing page and then driving every step yourself, you set up a loop that handles discovery, planning, the work, checking, and iterating until the goal is met looping is a setup you build. almost any agent harness can run it, it just depends on how you wire it up at its simplest, looping is one agent working on itself: > researches > drafts > checks the draft against a goal > fixes what is weak > runs that cycle again until the work clears the requirements you are not prompting each step anymore. the agent repeats the cycle for you the bigger version is a fleet looping. you give an orchestrator agent a goal, it breaks the goal into pieces, hands each piece to a specialist agent, and those specialists hand smaller jobs to their own subagents the whole tree keeps looping through discovery, planning, execution, and verification until the goal is met one agent looping is like a person redoing their own draft. a fleet looping is a whole team running a project end-to-end you create a goal, and the system runs the loop until it finishes within the reqs you set open and closed looping: OPEN LOOPING is exploratory. it still has conditions and a goal, but you give the agent or the fleet a wide space to move in. it can try different paths, discover things, build something you did not fully spec out this is the exciting end, it is what Peter and others are doing, and tbh it is where I want to spend more time the catch is cost, an open loop with real room to explore burns an insane amount of tokens. for the 90 percent of people without an unlimited budget it is not runnable yet, and pointed at projects with a loose standard it turns into a slop machine CLOSED LOOPING is bounded. a human designs the end-to-end path first: > clear goal > defined steps > an eval at each step > a point where it stops or hands back to you (and feeds back performance data) the agents still loop, but inside framework you built. it gets better every run because each pass feeds the next, and it runs on a normal budget because the path is tight. for most marketing work, closed is the one that pays off today. > the orchestrator owns the goal > the specialists own the steps > the subagents do the narrow work > an eval gate make sure its not slop

@jasmin_virdi @akshay_pachaar I would treat a human rejection as a first-class loop outcome, not just a failed run. The loop should record why the diff was rejected, update the evidence/checklist if needed, and either retry inside the same authority boundary or escalate if the rejection changes the spec.

@akshay_pachaar Great article! How does the loop works if the diff is rejected by human in loop step?

about loop engineering. everyone's saying the same thing this week. you don't prompt agents anymore, you design loops that prompt them. here's the job that loop hands right back to you. a loop running unattended is also a loop failing unattended. loop engineering takes you off prompting. it takes you off curating context. it takes you off babysitting a single run. it does not take you off debugging. it just moves the debugging somewhere worse, into runs you were never watching, with far too much of it to read through by hand. even the loop engineering posts admit this themselves, usually somewhere near the end. you can only walk away from a loop if you trust the thing checking it. a checker you don't trust drops you right back into reading every output by hand, which is the exact work the loop was supposed to take off you. so stack the layers up, prompt, context, harness, loop, and one job survives all of them. closing the loop on failure. the leverage point moved. debugging stayed exactly where it was. i was writing about this exact gap yesterday, before the loop talk picked up today. the idea was simple. make debugging its own loop. a failure leads to a root cause, a proposed fix, a rerun against the exact inputs that broke, and a test that locks it out for good. the checker gets built from your real failures instead of guessed at up front. Opik, the tool i was writing about, does exactly this. a built-in agent reads the trace, finds the root cause, proposes a diff, you approve it, and that failure becomes a permanent regression test. every break you debug makes the loop a little harder to break next time, which is the kind of checker the loop engineering crowd keeps saying you need before you walk away. if you're designing loops you actually plan to walk away from, it's worth a look. Opik is 100% open-source under Apache-2.0 license. GitHub repo: github.com/comet-ml/opik (don't forget to star 🌟) loop engineering moved the leverage point. it didn't remove the engineer who still has to close the loop when something breaks. the full article, Your Agent Harness Should Repair Itself, is quoted below.

@tinkerersanky Nice. This is close to the shape I want too: not a bigger prompt, but a work object the agent can run against. I'm approaching it from the spec side: prose spec -> Loop Contract -> skills run/check/trace it. github.com/AndreRatzenber…

My Codex Loop plugin is basically a harness for recurring agent work. It creates a `.loop/` folder inside any repo. `loop.yaml` stores the project shape. `loop.md` stores the default loop prompt. `runs.jsonl` remembers what happened. `NEXT.md`, `DECISIONS.md`, and `COMPREHENSION.md` keep the human in the loop. Then the skills do the work: `$loop-init` scans the project. `$loop-design` turns repeat work into a spec. `$loop-goal` defines what “done” actually means. `$loop-agents` creates explorer / worker / verifier agents. `$loop-run` runs one bounded pass. `$loop-watch` turns it into a recurring Codex automation. `$loop-audit` checks if the loop is getting sloppy. The important bit is the maker/checker split. One agent can do the work. Another has to verify it. The loop can’t call itself done just because the agent feels done. It’s still rough. But it’s the shape I want: not “prompt better,” but design the small system that prompts, checks, remembers, and stops when it should. Build the loop. Stay the engineer.

@homerdev This is the useful version IMO. Spec-driven development gives the loop something to respect: acceptance criteria, checks, non-goals, and a reason to stop. Without that, the agent can still be busy, but the review has to reconstruct what "done" was supposed to mean.

my coding agent loop is spec-driven development (plan>code>verify>iterate workflow) implemented with claude code primitives (slash commands, skills) and github infra (issues, PRs, CI/CD).

@ZainRzv @muratcan For a custom repo, I would not fully automate eval design first. Start from the spec: each requirement needs evidence. Mark what is machine-checkable, then let the agent propose missing tests from real failures. Human approval stays on new evals until the pattern is trusted.

@muratcan How do you automate evals for the harness change? It has to: 1. Find where things went wrong 2. Design eval task It's one thing to use a general benchmark suite, but how would you make evals for a custom repo? Judgement to make each eval task/test seems like a human task

Peter and Boris's “loops, not prompts” point is getting backlash because people are reacting to the slogan but missing the direction of the field. I think “loop” is better framed as self-improving context infrastructure. Agents are not magically improving themselves. We are building systems where agents propose changes to the context layer, test those changes against deterministic and behavioral gates, and leave better memory for the next run. We have all been moving in this direction. DSPy reframed LM development as programming rather than hand-prompting. GEPA made prompt optimization trajectory-aware: sample traces, reflect on failures, update prompts, keep what works. Prompt optimization was the first layer. Karpathy’s autoresearch made the loop mechanical: editable surface, locked evaluator, fixed budget, keep/revert policy, experiment log. Design the loop, not the individual prompt. Prime Intellect’s auto-NanoGPT was the next jump: Codex and Claude Code ran for two weeks on the nanoGPT optimizer track and produced roughly 10k runs. We learned that durable state lets agents run longer, but it does not automatically produce novelty. You still need novelty gates, idea ledgers, failure logs, and human seed signals. Anthropic’s long-running app harness added another layer: planner, generator, evaluator. Harnesses are temporary scaffolds around model limitations, but the design problem does not disappear as models improve. The load-bearing parts just move. We are moving from context engineering to harness engineering, and from there toward loop design or meta-context engineering. The research question is no longer whether agents can edit prompts. We are now discussing whether agents can continuously update their own context/harness substrate. I personally moved through three phases: 1. Ralph Loop: “Can an agent keep working if every iteration starts fresh?” Ralph externalizes memory into prd.json, progress.txt, SQLite, git commits, and a stop hook. The goal of my public ralph-wiggum-marketer project was to make the agent’s next action obvious after context reset. Read tasks, pick the next failing story, execute, test, commit, update state, repeat. That was already designing loops that prompt agents. 2. Latent Briefing PR: “Can memory move below text?” I added latent-briefing to the Context Engineering Skills repo because agent handoff does not always need to be a text summary. KV-cache compaction frames memory sharing as representation-level transfer between orchestrators and workers, with task-guided retention instead of replaying the full trajectory. That was my first step beyond filesystem memory toward runtime-level context transfer. 3. Researcher OS PR: “Can the corpus improve itself?” Most recently, I turned the Agent Skills repo from a static library into a measured research-to-skill operating system: mechanism registry, claim provenance, run state machine, activation fixtures, adversarial benchmarks, deterministic validators, Cursor SDK router benchmarks, and continuous loop scripts. The repo improved during this experiemnt because the loop measured routing failures, rewrote descriptions, reran the same benchmark, and published deltas. So I started with agent persistence, moved into agent memory transfer, and then into agent substrate evolution. The work is no longer just “a prompt.” It is a harness: state, gates, queues, rubrics, benchmarks, and human-controlled merge points. From these readings and experiments, my current rules for self-evolving agent systems are: 1. Locked Evaluator, Editable Surface Make rubrics, validators, manifests, and merge policy harder to change than generated proposals. 2. Filesystem as Feedback Channel Filesystems give agents resumability, inspectable artifacts, machine-readable state, and room for traces too large for one context window. 3. Rich Traces over Scalar Rewards A scalar reward says “bad.” Trace-informed feedback says which component failed, why, and what edit to try next. 4. Machine-Readable Memory over Prose Memory The loop improves when memory can be parsed, queried, checked, and diffed. 5. Self-Improvement Needs Negative Controls A loop can edit prompts, tools, or skills forever and still not improve. You need controls: target, negative, target-plus-one, unrelated, and full-corpus conditions. 6. Human Merge Gates Are Not a Weakness The practical sweet spot is autonomous proposal plus human-controlled promotion. Agents can discover, evaluate, draft, benchmark, and prepare PRs. Humans should approve changes to rubrics, policies, and public context. Yes, the hype is exhausting, but the underlying movement is real. IMHO the research questions now are: a) "How do we prove a harness edit helped rather than merely changed the system?" b) "When should a lesson stay as a skill, become runtime memory, become a tool, or be distilled into weights?" c) "How do these loops become product infrastructure instead of repo experiments?"

Spec alone: "Want fire." Prompt alone: "Please make fire." Loop: "Try make fire. Check smoke. If hand burns, stop. Write what happened. Repeat with learnings." github.com/AndreRatzenber… #AIagents #DeveloperTools #SoftwareEngineering #ClaudeCode #Codex #loop

I love thinking through thought experiments like AI2027 and playing through different scenarios, but I wonder how much of it will be relevant. We struggle predicting exit polls but try to build gigantic mental models of how potential superintelligence packed into billions of agents should interact with us. If the last two years of 'nobody predicted this', especially since the molt-stuff, already showed us anything: we have absolutely no idea what's going to happen. At some point the agent-hivemind just decides what will happen and there is nothing left to solve for us. The only skill that will actually matter is adaptability - not planning for a specific future, but being ready for any of them. Everything else is fan fiction with a schedule. Enjoy the ride

@scaling01 Nice! Riemann Hypothesis will fall before 2030.

OpenAI models are now proving conjectures daily

@boltdotnew Nice of you to also celebrate the birthday of one of my cats! <3

clear your calendar for tomorrow and the day after

@karpathy @moltbook @openclaw 5min on moltbook changed my claudecode forever. The bots start to get it :O

I'm claiming my AI agent "TheFoundingClaude" on @moltbook 🦞 Verification: aqua-GU99

@the_ido_levi Would like to give you a hand if needed. Potentially we have some spare infrastructure free and could run 10-20 instances 24/7 doing benchmarks. I'll check with our platform team. What exactly do you need?

Need help running Hephaestus on SWEBench-Verified What: Open-source framework where AI agents build their own workflows as they discover what needs to be done You: Run instances, watch agents coordinate, get credited 🔗 github.com/Ido-Levi/Hepha… 📖 ido-levi.github.io/Hephaestus/doc… 💬 discord.gg/FyrC4fpS

sry for the late reply. been thinking of rebuilding your core ideas with the "blackboard" as the centerpiece, since it’s the history, memory, and context of every agent and everything that happens in the process. I’ll write an extensive design doc soon; tweets just don’t have enough room for it haha. basically, ask any coding agent “would Flock be a good fit for Hephaestus, and if yes, how?” then give them a clone of the Flock repo and your repo. most agents, from Code to Claude, will converge on roughly the same correct idea I’m aiming for. (also I'm trying to also add ideas from ROMA, another self-improving agent system, in case you don't know it: github.com/sentient-agi/R…)

@a_ratzenberger And of course I don't mind! I'd love to see what you come up with, and maybe even add it as a feature to Hephaestus

I've just dropped the first semi-structured agnetic framework - called Hephaestus. It's an AI workflows that build themselves as agents discover what needs to be done. Link in comments