Skip to content
AI-Native PM
Field Notes

Field Note

How We Caught AI Mistakes With a Second Agent

When workers run in parallel, each hands back something that looks finished, and finished-looking is not the same as correct. So we sent every draft on this site to a second worker whose only job was to take it apart first.

· 5 min read

When you run many workers in parallel, each one hands back something that looks finished. That is the trap. Finished-looking and correct are different properties, and a confident wrong answer reads exactly like a right one at the moment you receive it. So across this build, we never took a maker's output at face value. A second worker, with a single job, tried to take it apart first.

The pattern was the same whatever the maker had made. We sent each drafted chapter to a reviewer whose task was to check it against the voice rules, line by line, and report violations. We sent each set of claims to a reviewer whose task was to verify every one against reality and flag anything that did not hold. And we ran the links through a check that visited each route and reported the ones that led nowhere. We did not ask the reviewers "is this good." We asked them to find what was wrong with it, and they found real things: a banned word the maker had used, a sentence built as an aphorism, a statement about a real legal case whose status had changed since the draft was written, links that pointed at pages that no longer existed.

A maker, a skeptic, and a gate that ships only what survivesThree boxes read left to right. The maker drafts the work. The skeptic tries to refute it on voice, facts, and links. The gate ships what survives and sends the rest back to the maker. Caption: nothing shipped on its own say-so; a second agent tried to tear it down first.EVERY DRAFT MET A SECOND AGENTTHE MAKERdrafts the chapter,diagram, or noteTHE SKEPTICtries to refute it:voice, facts, linksTHE GATEships whatsurvivesfixes go back to the makerNothing shipped on its own say-so.

The reason to split the two roles is not process for its own sake. A maker is built, by the framing of its task, to produce a good version of the thing and to report that it has. Ask the same worker to then judge its own work and you get a lenient judge, because the judgment is shaped by the same instruction the maker worked from. A separate reviewer, told to assume the work is flawed and to go looking, is not primed to say yes.

A maker and a skeptic are different jobs, and giving both to the same worker is how confident mistakes ship.

The framing of the skeptic matters as much as its separation. A reviewer told to "check this" tends to confirm it. A reviewer told to refute the work, to start from the assumption that something is wrong and report what, actually looks, and the difference in what the two find is large. The point is to spend the effort before the reader does, not after.

None of this is particular to building a website. It is the discipline the Evals part of The Builder's Stack teaches for any AI product: you find the failures yourself before your users find them, and a regression check that runs every time is how a fixed mistake stays fixed. Pairing a maker with a skeptic is that same idea moved one step earlier, into the act of making itself.

Where this goes next

The full version of this discipline, building the test set, grading against it, and gating every change on it, is the Evals part of The Builder's Stack. For where verification sits in the ownership of a shipped product, see the Track section of The AI-Native PM's Ownership Map. And for the parallel making that creates the need for this check in the first place, read How We Used Agent Fleets to Build This Site.

Sources