Skip to content
AI-Native PM
Field Notes

Field Note

Why We Stopped Letting AI Pick the Winner

For a few weeks we let Claude generate creative options and let a judge agent pick the best one. The picks were competent, and we rejected nearly every one. We also cover what the judge still does for us, and why we took the final pick back ourselves.

· 5 min read

In late July we asked Claude to rework the hero visual on technicalenough.ai, our second education site. It ran the option pattern we had used all summer, with one addition we had not asked for: it built several animated candidates, scored them with a judge agent, and wired the winner straight into the page. We disliked the winner on sight; our words in the session were "gaudy, asymmetrical, unaesthetic." The sentence we typed next became the rule this note is about: "give me a few more options and let me judge them instead of you creating a judge agent."

With that message we ended an experiment we had been running for a few weeks without quite naming it. This note is the honest tally of that experiment: where letting an AI judge pick the winner failed us, the one place it did not, and what we run instead.

The loop we had been running

Both of our education sites were built option-first: Claude generates five or ten candidates in a preview lab, and we pick one, or a mix of two, or none, a working pattern we described in How We Used Preview Labs to Design This Site. We hoped a judge agent could take the last step off our hands: if one model can make the options and a second pass can rank them, we get finished work back instead of homework. Through late July and early August we ran that generate-and-judge loop on real creative work: hero visuals, promotional films for both sites, and the About-page section that describes how this site is built.

Two loops for choosing between AI-generated optionsTop lane, labeled the loop we retired: options flow to a judge agent, the judge's winner reaches us, we reject it, and a dashed arrow loops back to the start labeled rejected, start over. Bottom lane, labeled the loop we keep: options flow directly to us, we pick one or keep none, and the pick ships. Caption: Claude still makes the options; the pick stays with us.THE LOOP WE RETIREDOptions generatedJudge agent picksWe reject the winnerrejected, start the task overTHE LOOP WE KEEPOptions generatedWe pick, or keep noneThe pick shipsClaude still makes the options; the pick stays with us.

We kept rejecting the judged winners

The hero visual above was not an isolated case; the same pattern repeated across the judged work.

  • The Replit incident film. For the story film about the deleted production database, Claude storyboarded a few directions, scored them with a judge, and built the winner. We rejected it: too long, hard to follow, and it opened on a countdown ticker, the exact vanity-number move our own written rules ban. The judge had picked a film that broke rules we already had on file.
  • The About-page section. Claude drafted the section on how we build with Claude Code from several angles, and a judge agent picked the strongest draft. We rejected the pick's key phrasing, asked for options so we could choose, and later replaced the headline and the central paragraph with sentences we wrote ourselves.
  • The concept batch that converged. For a short promotional film on another project, Claude returned six concepts, and every one used the same dark palette and the same visual metaphor. We dropped the whole batch and restarted from a reference we trusted, which was, tellingly, the films Claude had made for these two sites earlier, back when we did the picking.

We did keep some judged calls. We shipped one pick unchanged: a one-line page lede where we looked at the ranked list and answered "#1 it is." And this month we shipped a judged sweep of the site's prose almost entirely intact; the one edit we reverted was the pure taste call in the set, a retitled heading, where we went back to our own word. The pattern in the tally is consistent. Where the judged call was mechanical (does this sentence break a written rule?), we kept it. Where the judged call was aesthetic (which of these finished things is right for us?), we rejected the winner nearly every time and shipped the versions we picked or wrote ourselves.

Why we rejected competent picks

None of the rejected winners was incompetent. On any generic rubric they were fine, and some were probably the logically strongest option on the table. We rejected them for two reasons.

The first is that a judge scores against the brief as written, and our taste is larger than the brief. We keep a rules file for our writing voice, and it has grown to 49 numbered rules; every rule exists because we caught a piece of output that failed a standard we had not written down yet. That file is permanently one rule behind, because we keep discovering our own standards by reacting to work that misses them. A judge agent can hold work to yesterday's rules. When we pick between finished options, we are applying the rule we have not discovered yet.

The second is that the second opinion is less independent than it looks. The judge is the same model that generated the options, warmed up by the same defaults, so the options come out as near-siblings, and the model scores them with the same defaults it used to generate them. Whatever default made all six concepts dark was also present in the pass that ranked them.

A judge agent can enforce the rules we have already written. When we choose between finished options, we are using taste we have not written down yet, so that choice stays with us.

The judgments we still delegate to AI

This is not a note against delegating judgment; it is a note about which judgments are worth delegating.

  • Finding defects. We still run every draft on this site past a skeptic agent whose only job is to take it apart, the practice from How We Caught AI Mistakes With a Second Agent. "What is wrong with this" has checkable answers, and agents find them well.
  • Structure and sequence. This summer we repeatedly took Claude's recommendation as given: which course part to build next, how to repair two weak parts, which story a carousel should load first. When a question has a best answer rather than a right feel, we usually keep the recommendation.
  • The options themselves. Most of what we kept was AI-made. The hero visual we ship today came out of an option lab, and the films we now hold up as our reference standard were AI-built. Generation was never the problem; only the final pick was.
The judgments still delegated to agents, and the one that came backLeft column, labeled still the judge's job, lists three checked rows: take a draft apart for defects; flag breaks of our written rules; recommend structure and sequence. Right column, labeled back with us, holds one gold box: the final pick between finished options. Caption: questions with checkable answers stay delegated; the final pick does not.STILL THE JUDGE'S JOBTake a draft apart for defectsbugs, broken claims, missing factsFlag breaks of our written rulesbanned words, banned constructionsRecommend structure and sequencewhat to build next, in what orderBACK WITH USThe final pick betweenfinished optionsQuestions with checkable answers stay delegated; the final pick does not.

Our own course draws this line in Graders: deterministic, judges, and humans: use code and judge models for the qualities you can define, and keep humans on the qualities you cannot define yet. We teach that chapter, and this summer we re-learned it on our own site.

The loop we run now

Claude builds the options, as many and as varied as we can get it to make, and puts every one in a lab where we can see it. We make the pick, and "none of these" stays a real answer, the discipline from What We Built for This Site and Then Deleted. Judge agents still run on every batch, demoted from picking winners to flagging rule violations before we look.

The plain takeaways, one per line:

  • A judge agent picks well when the standard is a written rule and badly when the standard is unwritten taste.
  • Options generated in one run tend to converge, and a judge from the same model scores them by the same defaults that produced them.
  • Keep verification delegated: defects, facts, and rule violations are checkable, and agents catch them reliably.
  • Keep the final pick on creative work with a person, and keep "none of these" available.
  • When you keep rejecting the winner, stop paying for more rounds of judging and take the pick back.

Sources

  • The session transcripts for both sites, May through August 2026; the quoted lines are verbatim.
  • Graders: deterministic, judges, and humans, the course chapter on which qualities a judge model can score and which stay with humans.