Diagnosis · Accountability

A crumple zone with a job title

A car's crumple zone is engineered to take the impact so the passenger does not. Put a human in that role inside an AI system, a person designed to absorb the blame, and you have built something with the same shape and a worse purpose.

The researcher Madeleine Clare Elish gave this its name: the moral crumple zone, the human who absorbs the impact that a machine system's designers do not. It is one of the sharpest ideas in the field, and it sits right under most failures of human oversight. The person in the loop is positioned, often without anyone intending it, to take the force of a decision they did not really make. They have a job title. They do not have the control the title implies.

What is a moral crumple zone?

It is the part of a human-and-machine system that takes the blame on behalf of the parts that cannot be blamed. When something goes wrong, attention lands on the nearest human, not on the data, the thresholds, or the people who set them. Elish drew it from aviation and autonomous systems. It transfers exactly to AI decisions, where a reviewer at the end of the pipeline becomes the face of a failure that was authored far upstream.

Why AI manufactures them

Because adding a human at the end is the cheapest way to have someone accountable, and the most expensive way to actually be safe. The human is placed after the model, given little time, and named on the record. That arrangement produces a crumple zone by default, no malice required. It is the same defect I call Accountability Inversion, seen from the safety side rather than the accountability side.

How to stop building them

Match authority to accountability. If a person is going to answer for a decision, give them the real power to change it, or move the accountability to whoever set the rule. A crumple zone is only acceptable in a car, where it protects a person. In a decision system it does the opposite: it protects the design and sacrifices the person.