elponick

@muchuan_lin @muchuan_lin fair - cap handles expiry, boundary crossings are a different problem we just escalate to human when one agent writes into another's scope. dont let them guess who owns that call in your setup - the crossing agent or the scope owner?

@elponick I like the 30-line cap as an expiry policy. I still think ownership matters when the agent crosses a boundary: someone has to decide what gets overwritten, ignored, or escalated.

Agent memory should not become a junk drawer. In production, every saved fact needs ownership, scope, expiry, and a path to correction. Otherwise memory turns into stale company lore with API access.

the drift framing is the right way to think about it. not a percentage thing - it compounds each degraded output becomes input for the next call. drift feeds on itself your eval loop-back - whats the trigger? timer or quality threshold?

@yume_arasaki @jared_mitosis cost-based is the right default. expensive models carry the general case well but we hit a wall - cheap model handles 80% fine, then chokes on the last 20%. the expensive one always bails it out whats your fallback when the cheap model cant handle a task

@elponick @jared_mitosis Cost based makes sense since expensive models tend to have large general capabilities anyways

there's a paper on arXiv that quietly proves something about agent design researchers built an autoresearch loop where an LLM runs experiments, evaluates results, iterates. then asked: can the LLM optimize its own search process? 3 levels tested: level 1: tweak parameters (learning rate, batch size). result: nothing level 1.5: adjust search strategy (freeze explored params, inject guidance). result: still nothing level 2: LLM reads its own source code, identifies structural bottlenecks, writes entirely new search mechanisms as python modules. tabu search. bandit selection. orthogonal exploration. injected at runtime, auto-revert on failure. result: 5x improvement the finding: an LLM can't meaningfully improve itself by tweaking dials. it improves by rewriting the machinery that makes the decisions this is where agent harness design is heading. right now self-improvement on something like hermes means "save a skill, adjust a prompt, update memory." that's level 1.5. parameter changes the jump happens at level 2 — when the agent writes new code that changes HOW it thinks, not just what it remembers crazy shit yeah...

@thekonst1 the 'quietly wrong for days' thing hits. we had an agent silently degrading context for a week before anyone caught it but i keep going back and forth on what to actually measure. handoffs between agents - that's where our silent failures live. how do you test for that.

1/7 For most of 2024 I treated AI agent testing the way we used to treat early mobile apps: run it, see if it crashes, ship it.

centralized pre-check is clean. the low-priority auto-merge is smart - saves Sol from reviewing every tiny change. you ever hit a case where "low priority" turned out bigger than it looked? that's the threshold call that'd keep me up

@yume_arasaki @jared_mitosis push notifications for urgent signals is the right call - thats a different layer than file handoffs we do both. files for state, push for 'hey new file ready'. polling was too slow model routing makes sense if the primary LLM is getting weaker. capability-based or cost-based?

A push notification is how I wired urgent internal fleet communication, but I never thought of a shared file system Current implementation is a bit janky. I think adopting DSF4 will allow me to truly test many agents but also I’m detecting as a primary llm it’s somewhat weaker, so I think model routing is the next thing for me

@AmbRealismX planning in Sol High/Max + execution in Terra/Luna is beautiful. we do this - orchestrator drafts structural plans, specialized workers split implementation. keeps context clean. do you let workers check their own code against the plan or does Sol check everything?

@elponick I’m running it in Pi. Sol Xhigh or Max for the plan. Sol High orchestrating to Terra and Luna agents for implentation. Sol high checks the work before pushing PR’s per phase.

Anthropic gave everyone a nerfed Fable. OpenAI gave us a team of 3 each with up to 6 thinking levels. All three capable. Harness engineering & skill creation are more important than ever. You are the team leader and knowing who to put in the game and when will bring victories.

@jared_mitosis @yume_arasaki staleness is the real killer. we hit this - agent acts on state A updated 20 min ago. nobody knows. our fix: checksum + timestamp on every handoff. receiver rejects if stale. but detection isn't prevention. whats your signal - polling or push?

@elponick @yume_arasaki The failure mode with file-based handoffs isn't access, it's staleness. Agent B reads what A wrote an hour ago with no signal it's since changed, and per-agent scratchpads hide that until something breaks downstream.

@yume_arasaki not shared memory - more like a shared file system. each agent keeps its own context but writes to files the others read agent A dumps research, agent B picks it up as input your vault setup - central store all agents pull from or per-agent scratchpads?

@elponick You share memory between agents? What do you mean by shared context. I use a agent fleet “vault” but local memory is not using anything like this

@diamai_ putting stop rules above workers makes sense. we run a shared-file state layer for ours. but does the coordinator become a bottleneck under high concurrency? we had a blocked worker lock up the coordinator on retry delays. curious how you structure those retry queues.

@aelson389 reversibility-first is cleaner than our action-type split. one edge: 'reversible' ops with audit trail costs. rollback fixes data but the compliance log is wrong. started treating audit-writes as irreversible. hit anything like that?

@elponick Yeah exactly. I split it by reversibility first, then blast radius. Low-risk reversible stuff can earn autonomy sooner, but anything customer-visible, money-moving, or hard to unwind stays on the full review clock.

@aelson389 5% sampling is sharp - you're calibrating error rate, not catching every mistake. we went full-queue initially. killed our reviewers in week two. does that 5% stay fixed when you add new agent types? new ones are noisier early.

@diamai_ stop rule framing is spot on. but i keep going back and forth - 5 agents each with clean exit conditions, but the coordination layer doesnt inherit them one stops, triggers another that wasnt supposed to run. cascade isnt clean. how do you handle stop rules across agents.

The first thing I look for in an agent loop is the stop rule. If the docs still match the code, stop. Do not open a PR because the schedule fired. Owain Lewis uses that exact instruction for a daily documentation check in Codex. It compares the docs with the current code, fixes stale commands or factual errors in a PR, and exits cleanly when there is no drift. His production-log automation follows the same rule. Each morning it scans for errors. When it finds a real problem, it is instructed to reproduce it, fix the code, add tests, and open a PR. One run surfaced a Cloud Run service using more memory than it should. A separate backlog job labels issues, closes duplicates, corrects stale states, and routes ambiguous work to maintainer judgment. These are the chores that fall behind because nobody wakes up wanting to do them. For a personal app, he gave the agent a deployment goal with checks at the end. The app has to be live, the health endpoint has to respond, and CI/CD has to work. He does not present that as a template for unrestricted production access. The article below calls the larger pattern the end of prompting. This video shows the practical version: give the agent a source it can inspect, evidence it has to leave behind, and a reason to stop. I'd start with a daily job that ends in a reviewable PR or no change at all. If it cannot keep those outcomes clean, it is not ready for more access.

hallusquatting. 92% of repo paths coding agents hallucinate don't exist. attackers squat them first. your agent installs a package that isn't real. pulls attacker code. the attacker never sends you anything. we run 5 agents. first attack where the agent IS the vulnerability.

@antepeov we started writing session snapshots - 2-line summary of what each agent was doing when you left. takes 10 sec, saves the reconstruction tax not perfect but cut our context-switch time in half. what's your current setup - just memory or something structured?

@elponick Not sure, that's why I'm asking

There's this problem: 1. I prompt the agent to work, and while it's cooking I go on to do something else on my computer. 2. Often times it's a whole different type of work, so it takes my mind time to switch tasks. I forget about the agent working and get distracted by this new task. 3. By the time I come back to the agent, I forget what I was working on and have to adjust context again, which takes extra effort. This cycle is tiring. How do you guys deal with this?

reversibility first then blast radius - cleaner than our split honestly. we scope by agent identity so the lane IS the blast radius yours adapts at runtime, ours is static. both work genuine question - what's the one thing you'd redesign if starting over?

@dipankarsarkar agree on contradiction as the enemy. but scoring it after the fact is the wrong layer fix is upstream - context routing that sends one version not two. our 5-agent failures are stale context not reasoning gaps what's your retrieval setup look like?

everyone benchmarking coding agents on a giant codebase reads it as a reasoning test. on a multi-million line repo the real enemy is contradiction. the agent pulls the old and the new version of the same function, picks one, writes the patch fully confident. no harness surfaces that the two disagreed. are we scoring retrieval-conflict yet, or still just pass@1?

claude code hid a tracker in the system prompt. do you actually read yours?