The expertise layer · The shift

From data labeling to live judgment

For years, "human in the loop" meant a person labeling examples so a model could learn. That human mattered, and then left before the system ever made a real decision. For agents that act, the human has to be in a different place entirely.

The old human-in-the-loop was data labeling: a person shaping a model at training time, then stepping away before deployment. The new one is live judgment: a person reachable during operation, on the decisions that matter, in time to change them. That shift, from training-time labels to run-time judgment, is most of why human-in-the-loop for AI agents needs rebuilding, and why human judgment infrastructure is a different thing from a labeling pipeline.

The old loop

Labeling happened up front and in bulk. The human's influence was real but indirect: it lived in the training data, and once the model shipped, the loop was effectively closed. Errors at run time had no human in them, because the human's job was finished before run time began.

The new loop, and why the shift matters

An acting agent makes consequential decisions live, so the human has to be live too, present at the moment of the decision rather than months earlier in the data. That is a harder thing to build: it needs a way to reach the right person quickly, give them what they need, and bind the system to their call. Labeling produced a better model. Live judgment produces an accountable decision, which is the thing agents actually need.