Essay · The evidence layer

Your Eval Passed. Your Product Failed.

Internal evaluations keep certifying AI systems that then degrade in production. Not because the evals are lazy, but because of three structural blind spots no internal harness can escape: sampling, snapshot, self-interest.

There is a meeting happening somewhere right now that goes like this. Support tickets say the AI feature got worse. The team reruns the eval suite. It passes. The dashboard is green, the customers are unhappy, and both are telling the truth. Nobody in the room is lying and nobody is incompetent. The eval is doing exactly what evals do, and what evals do is not what everyone in that meeting believes they do. The gap has three causes, all structural, and no amount of better prompt engineering closes them.

Blind spot one: sampling

Your eval tests the prompts you thought to write, which cluster around the behavior you already understand. Production is the distribution you did not think of: the phrasing from a user in a hurry, the document twice the length you tested, the edge case that is somebody's entire job. A model can hold steady on your suite and shift meaningfully in the tail, and the tail is where your hardest users live. Passing your sample says your sample still works. It was never evidence about the distribution.

Blind spot two: snapshot

Worse, your eval measures a moment. You ran it when you shipped, and the number you remember is the model you had, not the model you have. When the model behind your API is changed without notice, your certification silently expires, and nothing tells you it expired. Rerunning the eval on incident days does not fix this: without a continuous series you cannot say when behavior moved, only that it has, by the time you looked. A snapshot cannot testify about drift. Only a series can, and almost nobody keeps the series.

Blind spot three: self-interest

And even a continuous internal series has a credibility ceiling, because you built the harness, chose the thresholds, and need the product to pass. That is not an accusation; it is Goodhart operating on whole organizations. The eval becomes the target, the target gets hit, and hitting it drifts away from the thing it once measured. When the dispute arrives, with a customer, a vendor, an auditor, your beautifully engineered eval is testimony from an interested party. I keep returning to the same distinction: testimony is what you produce about yourself. Evidence is what a witness produces about you.

Keep the evals. Add a witness.

None of this argues for deleting your eval suite. It argues for knowing its jurisdiction. Internal evals are excellent at the question they can actually answer: does my product logic still work against my known cases. The questions they cannot answer, has the model itself changed, when, by how much, and can I prove it to someone with no reason to believe me, need an instrument with different properties: independent of both vendor and buyer, continuous rather than snapshot, measured against stable baselines, recorded so it cannot be rewritten. That is the design brief of Modelometer, which measures models on a fixed cadence and writes every run to a hash-chained public record. Your evals watch your product. The witness watches your foundation.

The question for your next incident

Next time the tickets and the dashboard disagree, ask one question before the war room convenes: what is our evidence about the model, as opposed to our evidence about our product? If the honest answer is none, the incident will end the way these always end, ambiguously, expensively, with a workaround and a lingering distrust of your own metrics. The green dashboard was never wrong. It was answering a smaller question than the one your business was asking.

There is also a budgeting argument hiding here. Teams pour engineering months into ever-larger internal eval suites, chasing the coverage that would make the green dashboard trustworthy, and the chase is structural, not a skill issue: coverage of an unknown distribution, of a moving target, certified by an interested party, does not converge. A fraction of that budget, redirected to an independent continuous record, buys the thing the suite cannot produce at any size: a second opinion that stands up outside your own building. Buy coverage where coverage works, and buy witness where it does not. The two are not substitutes, and confusing them is how green dashboards keep presiding over red quarters.