Essay · The evidence layer

The Assay Office for AI

Every high-value market eventually invented independent verification: hallmarks for silver, purity laws for food, audits for accounts. Each was resisted as friction, each became the floor of trust. AI is a pre-assay market.

In medieval London, if you sold silver, you brought it to Goldsmiths Hall, where someone who did not work for you struck a small mark into the metal. The mark did not make the silver better. It made the claim about the silver checkable, which changed what the metal was worth and who would dare trade in it. We have been running that experiment for seven centuries across every market where value is invisible to the buyer, and the result never varies: first the fraud, then the outrage, then the independent verifier, then the strange amnesia in which everyone forgets the market ever worked another way. AI is currently between steps one and two.

The pattern, three times over

Silver got the hallmark because a buyer cannot see purity. Food got purity laws because a shopper cannot see what is in the tin; the century of swill milk and adulterated bread ended not when sellers became virtuous but when inspection became institutional. Accounts got the audit because an investor cannot see solvency from outside, and a company vouching for its own books turned out, repeatedly and expensively, to be worth exactly nothing. Notice what each instrument is: not a regulator, not a punishment, a measurement, performed by a party with nothing to gain from the answer, recorded where both sides can cite it.

AI is a pre-assay market

Now look at what we buy when we buy AI. Value invisible to the buyer: entirely; you cannot see a model's quality, identity, or stability from outside the API. Claims made by the seller about themselves: universally; the model card, the changelog, the benchmark chart are all first-party documents. Capacity for silent degradation: built in; the product can be changed beneath you overnight, or substituted along the supply chain, with no mark struck anywhere. Every structural feature that produced hallmarks, purity laws, and audits is present, at larger scale and higher speed. The only missing piece is the assay office itself.

Why it always looks unnecessary right up until it is mandatory

The resistance is also part of the pattern, and worth taking seriously. Verification is friction. Early markets run on enthusiasm and relationships, and enthusiasm feels like trust. Sellers argue, sincerely, that their reputation is the guarantee, and in a small market with repeat players, reputation genuinely carries weight. Then the market grows past the size where anyone knows anyone, the first large dispute arrives, and reputation is revealed as what it always was: a story about the past, told by the interested party, with no exhibit attached. The assay office is never built during the enthusiasm. It is built during the depositions. My argument is only that we know how this movie ends, so we could skip to the useful part.

What the assay office for AI looks like

It is not a badge or a one-time certification, because model behavior is not a static property; it is a practice. The office measures continuously: identity daily, behavior on a steady cadence, the same battery against every model so results compare. It is independent: funded by neither side of any transaction it measures. And its ledger is tamper-evident and public, because an assay mark you cannot inspect is a logo. This is the institutional shape I am building toward with Modelometer, which is why I keep calling it a measurement bureau rather than a tool: bureaus outlive tools, because markets come to rest on them.

Being early to the inevitable

The falsifiable version of this essay: if AI is somehow the first high-value opaque market in history that never develops independent verification, I am wrong, and I will have built an elaborate seismograph for a world without earthquakes. I am comfortable with that bet. Every earlier market thought itself exceptional too, and each was exceptional right up until the first scandal that a first-party document could not survive. The hallmark did not make silver honest. It made honesty checkable, and checkable is the only kind of honest a market can price. AI will get its mark. The only open question is whether it arrives before or after the swill milk.

And for the builders reading this as a market map rather than a history lesson: assay institutions were never the glamorous part of their markets, and they were, again and again, the part that survived. Exchanges rose and fell; the inspection houses persisted, because every new entrant needed them to be believed. That is the position worth building toward in AI, and it cannot be claimed later by whoever shouts loudest. It accrues to whoever was measuring all along.