How to tell a good model
from a sleeping reviewer

Your reviewers approved 998 of the last 1,000 decisions the machine proposed. That number has two explanations and they are opposites. Either the model is very good and approving was the correct call almost every time, or nobody is reading and approval is the only output the workflow has ever been capable of producing. The log looks the same in both worlds. Every measure of human oversight I have proposed, including my own, is computed from that log, which means none of them can tell you which world you are in.

The Meaningful Override Rate counts how often a person reversed a machine decision they were positioned to reverse, where the reversal stood. It is a useful number and I still think it is the right one to start with. But a near-zero value is ambiguous in a way no amount of careful bookkeeping resolves. A reviewer correctly deferring to a system that is right ninety-nine times in a hundred leaves exactly the trace of a reviewer who has stopped looking. Same approvals, same timestamps, same absence of overrides.

This is not a defect in how the number is computed. It is a property of computing anything from observational data. You are watching a process and inferring a mental state from it, and two different mental states are producing identical observations. No statistical treatment of the approvals will separate them, because the separating information was never recorded. It was never recorded because it never happened.

A reviewer who holds a veto and never uses it presents the same evidence as a reviewer who holds no veto at all. That was the problem the metric was built for. It turns out the metric inherits it.

The measurement standard I published sets five conditions a figure must clear before anyone should cite it. A stated sample of at least a hundred reviewable decisions. A pre-registered definition of what counts as meaningful. One named workflow rather than a category of organisations. An explicit denominator, with no selection on the outcome. And inter-rater agreement, so more than one adjudicator concurs on what counted.

I wrote those. I think they are right. And reading them again, every single one is about the integrity of the number's construction: was the sample large enough, was the rule fixed before the results were seen, was the denominator honest. Not one of them asks whether the person was reading. A team could satisfy all five conditions and publish a scrupulously constructed measurement of nothing at all.

This problem is not new and it is not specific to AI. Any field that depends on a human catching rare errors has had to answer it, and they converged on the same answer: give the human cases whose right answer you already know, without telling them which ones those are, and count.

Clinical laboratories in the United States have done this under law since 1988. Under the Clinical Laboratory Improvement Amendments, a lab must score at least eighty percent on proficiency testing events for most specialties, ninety percent on cytology slide sets, and one hundred percent on compatibility testing in immunohematology. Failure is not a note in a file. It brings sanctions up to suspension of the certificate and loss of Medicare approval.

British breast screening runs the same idea on people rather than specimens. PERFORMS has been embedded in the NHS Breast Screening Programme for over three decades, participation is mandatory for readers, new test sets are built every year, and individual underperformance routes to targeted training rather than to a disciplinary file.

The counterexample is just as instructive. In 2015 the Department of Homeland Security ran covert tests against airport screening and got mock explosives or banned weapons through sixty-seven of seventy checkpoints, a failure rate of ninety-five percent. Every one of those checkpoints was staffed. Every one had a human looking at a screen. Presence was never the thing in question.

Here is the part that most internal quality programmes get wrong, and it is the difference between a measurement and a ritual. If the reviewer can tell which cases are tests, you are no longer measuring their ordinary behaviour. You are measuring their best effort under observation, which is not the quantity in dispute.

Forensic science has the clearest write-up of this. Mejia, Cuellar and Salyards, reviewing blind proficiency testing in 2020, set out why a declared test and a blind one are different instruments: blind cases have to resemble real ones, travel the whole pipeline, and above all "avoid changes in behavior from an examiner knowing they are being tested". They report that laboratories were making special efforts on samples they knew were proficiency tests, and that in drug testing the rate of false negatives was higher under blind conditions than declared ones. The examiners missed more when they did not know anyone was watching. That gap is the entire measurement.

What you need to know about your review queue is not how it performs during an announced audit. It is what happens on a Tuesday afternoon, two hundred items deep, when the model has been right all week.

So the standard needs one more condition, and it is a different kind of condition from the other five. The first five ask whether the number was built honestly. This one asks whether the number is about anything.

The arithmetic is friendlier than people expect. To estimate a catch rate somewhere near ninety percent to within ten percentage points at ninety-five percent confidence, you need about thirty-five seeds. To tighten that to five points you need a hundred and thirty-nine. Merely separating a reader from a guesser needs far fewer: catching eighteen of twenty would happen by chance about twice in ten thousand times.

Put that against a real workflow. Two thousand reviewable decisions a quarter, seeded at five percent, which is the proportion of casework the Houston Forensic Science Center set as its target for blind tests, gives you a hundred seeded cases a quarter. That is enough for a confident catch rate and enough to notice a change in it between quarters, which is the thing you actually want to watch.

Most of the difficulty is in the construction, not the statistics.

1. Indistinguishable from real work. A seeded case with a tell measures nothing except whether the reviewer found the tell. No round numbers, no unusual formatting, no arriving in batches on a Monday.
2. Intercepted before it can act. A planted wrong decision must never be allowed to take effect. That requires a real interceptor downstream of the reviewer, and if you cannot build one, do not seed.
3. Disclosed at hiring, never at test time. The reviewer must know the programme exists and consent to it as a condition of the role. They must never know which case is which.
4. Spread across the difficulty range. Seeds that are all obvious measure attention on easy items, which is not where oversight fails.
5. Audited for detectability. Compare the catch rate on seeds against the rate at which genuine errors are caught and later confirmed. If the seeded rate is much higher, your seeds have a tell and the number is inflated.

The first is ethical and it deserves to be stated plainly rather than buried. You are deceiving the people who work for you, repeatedly, as a matter of policy. That is defensible only when it is disclosed in advance, written into the role, and agreed with whoever represents the workforce. Pilots, laboratory technicians and screening radiologists already work under exactly this arrangement, which is an argument that it can be done well, not that it is automatically fine.

The second is cost. Blind testing supplements declared testing, it does not replace it, and the hard part is manufacturing cases realistic enough to survive contact with an experienced reviewer. Mejia and colleagues list that as the first obstacle, ahead of money. A seeded error programme in a serious deployment is somebody's job, not a script.

The third is that this can be gamed in the other direction. A team that wants a flattering number will seed easy errors. That is precisely why the catch rate has to be published with its sample size and next to the override rate, and why the fifth design rule exists. A number that only ever looks good is not evidence, it is decoration.

A catch rate near what guessing would produce means the queue is unread, whatever the override rate says and however many people are employed to look at it. A high catch rate sitting beside a near-zero override rate is the good case, a strong model with an attentive reviewer, and it is the only evidence that the good case is the one you have rather than the one you assume.

A catch rate of one hundred percent is not the best outcome. It is the suspicious one. In practice it almost always means the seeds are recognisable, and what you have measured is your own tell.

Chenna, M. (2026). How to tell a good model from a sleeping reviewer.
manjchenna.com/essays/seeded-error-probe