The stakes · Case pattern

When an algorithm decides, and no one owns the rule

The danger is rarely a dramatic robot. It is a quiet rule, set once, far upstream, that then decides at scale while the people who wrote it are nowhere near the outcome. By the time the harm is visible, no one can say who chose it.

When an automated system decides and no person owns the rule it followed, two things happen together: the harm scales at machine speed, and the accountability for it evaporates. This is not hypothetical. It has already played out in public, more than once, and the cases share a structure worth naming, because meaningful human oversight is precisely what was missing from each.

Two cases that already happened

In the Netherlands, an automated system used to detect childcare-benefits fraud, the case widely known as the toeslagenaffaire, wrongly flagged large numbers of families, with consequences severe enough to contribute to a government's resignation. In Australia, the automated debt-recovery program known as Robodebt issued debt notices using a flawed calculation, and was later found unlawful by a royal commission. In both, software decided, people were harmed, and for a long time no individual owned the rule.

The pattern

The rule is set far upstream, by people making a policy or a threshold. The system applies it to thousands of cases. The harm appears downstream, to individuals who cannot reach anyone who could reconsider. And because the rule-setting and the harm are separated by distance and time, accountability falls through the gap. Everyone involved can honestly say the decision was not theirs.

What it teaches

That an algorithm deciding alone is an organizational choice, not a technical inevitability. The fix is to put a named, accountable human between the rule and the irreversible harm, with the authority and the time to stop it. Not on every case, on the consequential ones. The cost of skipping that is exactly what these cases measured, in public, the hard way.