Frame · Oversight

The oversight illusion

A power nobody uses looks exactly like a power that was never there. That is why so much AI oversight passes inspection: from the outside, the real thing and the empty thing are the same picture.

Stand outside two systems. In the first, a human genuinely reviews the machine, and now and then says no, and the no holds. In the second, a human is present, clicks approve, and has never once changed anything. Photograph each. The photographs are identical: same person, same screen, same approval. The difference between real oversight and an empty ritual does not show up in the picture. It shows up only in the numbers, and almost nobody looks at the numbers.

Why does unused oversight look like none at all?

Because oversight is a capacity, not a posture. A fire extinguisher on the wall and a painted picture of one look the same until there is a fire. The presence of a human proves presence; it does not prove the power was ever exercised, or could be. From outside you see attendance, not command. So a system that has quietly hollowed out its oversight looks, on every org chart and in every audit photo, exactly like one that kept it real.

Why this is dangerous, not just untidy

Because the empty version passes as the real one, indefinitely. Regulators check that a human is in the loop, and a human is. Buyers ask whether there is human review, and there is. Everyone is technically telling the truth, while the thing that matters, whether the human can and does change outcomes, goes unmeasured. The illusion is stable. It survives audits. It survives right up until the day it fails, and then everyone is surprised the human in the loop was decorative.

How do you tell the real thing from the illusion?

You measure the use, not the presence. The only honest test is behavioural: of the decisions a human could have changed, how often did they, and did the change stand. That number, the Meaningful Override Rate, is exactly what the photograph cannot show you. Near zero, with a human dutifully present, is the signature of the illusion. In my experience it is also the common case.