Flagship essay · The evidence layer

The Model on the Invoice

Your vendor bills you for the frontier model. How do you know that is what answers? On silent substitution in the AI supply chain, why you cannot tell from the inside, and how serving-identity verification catches it.

Your contract names the model. Say it names Fable 5, the frontier model, at frontier prices, because that is what your vendor sells: their product, powered by the best. Now the uncomfortable question. If the endpoint quietly served you Opus 5 instead, the capable, cheaper sibling, would you know? Be honest. The API shape is identical. The latency is close. The answers are good, because the cheaper model is good. The invoice would not change. The only thing that changed is the thing you cannot see: the margin moved from your product to their pocket, and the model your compliance file names stopped being the model that answers your calls.

The substitution economy

I am not accusing any one vendor of fraud, and that is precisely the problem: no accusation is needed for the risk to be real. Between you and the frontier model there is a supply chain: the SaaS product you buy, the platform it builds on, the router that arbitrages between providers, the provider itself. Every hop in that chain is under margin pressure, and every hop has a defensible-sounding reason to substitute: cost optimization, load balancing, a fallback tier, an A/B test you never consented to. Each hop calls it engineering. From your side of the API, all of it looks the same, and none of it appears on the invoice.

Why you cannot tell from the inside

You might think you would notice. You would not, and the reasons are structural. Version strings are self-reported by the party doing the serving. Your evaluations sample your own prompts, which is exactly the traffic a cost-optimizing router serves carefully. Taste tests fail because on most everyday prompts, a strong cheaper model and the frontier model are indistinguishable; the differences live in the hard tail you rarely hit, which is precisely where you most need the model you paid for. And when you raise it with the vendor, you are back to the oldest problem in this field: your word against theirs, with no witness in the room. We assay gold. We audit accounts. We certify the provenance of food. The most expensive ingredient in modern software carries no label anyone verifies.

What verification actually looks like

A model cannot cheaply fake being a different model across a broad battery of behavior. Its pattern of answers, where it excels, where it hedges, where it refuses, how it fails, is stable enough to be recognizably itself: a fingerprint, not a version string. Verification, then, is a comparison. Establish a reference baseline for the claimed model by measuring it directly. Measure the endpoint that claims to be that model. Compare. If the behavior at the endpoint stops matching the baseline, either the model changed or it was never the model on the label, and both of those are things the buyer deserves to know the day they happen. This is what Modelometer does, live in beta: serving identity checked daily against measured baselines, behavior measured weekly across eight domains, every run written to a hash-chained public record that neither the vendor nor the buyer can quietly rewrite. Not an opinion about quality. A record of identity.

What the EU AI Act makes of an unverified label

If you deploy AI under the EU AI Act, the model's identity is not a detail, it is the load-bearing assumption of your whole file. Article 13 entitles you to accurate information about the system you deploy, and its instructions for use are written for a specific model: instructions for a system you are not actually being served protect nobody. Article 26 obliges you to use the system in accordance with those instructions and to monitor it, which presupposes you know what it is. The Act's transparency duties for general-purpose models exist so that documentation of which model does what flows down the chain to you. Every one of those provisions assumes the name on the paperwork matches the model on the wire. Today that match is taken on faith, and faith is not a control. An unverified model name in a compliance file is not a fact. It is an assumption wearing the clothes of one.

If this is your desk

Vendor risk and procurement: model identity should be a contract term, and a contract term you can test is worth ten you cannot. Compliance: if your file names a model, an independent identity record converts that line from assumption to evidence. Buyers of AI-powered software: you are paying a model premium on faith; it costs nothing to ask your vendor whether they would submit their endpoint to independent verification, and their answer tells you most of what you need to know. And vendors with nothing to hide: proof that you serve what you sell is about to become a selling point, because your honest competitors cannot distinguish themselves from quiet substituters any other way.

What would prove me wrong

Cryptographic model attestation, the provider signing every response in a way third parties can verify, adopted across the industry, would solve identity better than behavioral measurement ever could. I would welcome it and retire this half of the argument. Until that exists everywhere, behavior is the only signature that cannot be cheaply faked, and somebody independent has to be measuring it. My bet is shorter odds than the drift bet: which model is it, and can you prove it, becomes a standard vendor due diligence question, asked in the same breath as where is our data stored. The companion question, whether the model you did get has quietly changed since you tested it, is the other flagship: your AI changed last night, prove it.