Regulation increasingly requires a human to oversee high-risk artificial-intelligence systems, yet provides no way to determine whether that oversight is ever exercised. Because an oversight power that is never used is externally indistinguishable from one that does not exist, compliance can be satisfied while command is absent. We propose the Meaningful Override Rate (MOR): among decisions a human was positioned to change, the fraction the human actually changed and the change was honored. We specify two companion measures, contestation latency and reversal validity; a five-part bar a reported figure must clear to be citable; and a fixed reporting format. We argue the standard should be published before any measurements, so the instrument does not accrue to whoever it flatters. This is version 0.1, released for use, critique, and revision.
Keywords: human oversight, AI governance, EU AI Act Article 14, human-in-the-loop, automation bias, accountability, measurement.
Frameworks for trustworthy AI, including Article 14 of the European Union's AI Act as it comes into force, require that certain systems remain under human oversight.1 Such provisions specify that a human be able to intervene. They do not, and largely cannot, establish that intervention ever occurs. A reviewer who holds a veto but never exercises it presents, to any external audit, the same evidence as a reviewer who holds no veto at all. Oversight therefore risks becoming a certified property of a system rather than an observed behavior of a person.
This paper proposes a minimal, auditable measure of exercised oversight, together with the conditions under which a reported value should be believed. The aim is not to replace legal requirements but to give them an instrument: a way to distinguish oversight that is real from oversight that is performed.
A decision produced or proposed by an automated system that a designated human was both positioned and permitted to change before it took effect. Reviewable decisions form the denominator of the measure. Decisions a human could not, in practice, have altered are excluded.
A reviewable decision in which the human changed the system's output, on the human's initiative, and the change was honored by the deploying organization. A recorded presence, an elapsed interval, or a countersignature is not, by itself, a meaningful override. Authorizing is not overseeing.
MOR reported alone is fragile: it can be inflated by trivial changes or depressed by conditions that make disagreement impossible. Two companions constrain its interpretation.
The realistic time and context a reviewer had to form and register disagreement. As latency approaches zero, review approaches a turnstile, and MOR loses meaning at any value.
The rate at which human overrides were, in retrospect, correct relative to the system's original output. A high override rate that is frequently wrong is a distinct failure, not evidence of oversight.
A reported MOR value should be treated as citable only if it satisfies all of the following. A figure short of these is an anecdote in the form of a percentage.
To keep figures comparable and self-limiting, report MOR in the following shape, with its scope caveat attached:
A recurring design shortcut is to evidence oversight by logging human presence or human confirmation. Such logs are tidy and auditable, and they measure the wrong quantity. A confirmation that is always given carries no information about command. Where a reviewer can technically object but never does, and is nonetheless accountable for the system's errors, the human occupies what Elish terms a moral crumple zone: a locus of blame without a locus of control.2 Sustained non-use of a veto is also the expected result of well-documented automation bias, the tendency of operators to defer to reliable automation.3 MOR is constructed to detect exactly this condition.
This is version 0.1. It is published before any measurements are reported, by design, so that the definition of the instrument cannot be adjusted to suit results already in hand. Reference implementations and measured values will be released separately, each accompanied by its method and limitations, and only when they meet the criteria of Section 4. A measurement standard is fair only if it is fixed before the measurement.
We state one testable expectation. Across real deployments, measured under the criteria above, the meaningful override rate of systems currently marketed as offering human oversight will be found to sit near zero. A body of deployments in which humans routinely and validly overrule the system would falsify this claim, and such evidence, if produced, will be recorded here in revision.