All posts

Don't Let One Agent Grade Its Own Homework

AI AgentsBest PracticesSoftwareDevelopmentAI

TL;DR

  • A second agent reviewing the first catches a specific failure mode: plausible-but-wrong output that the first pass rationalizes instead of questioning.
  • A second instance of the same model shares its blind spots. Genuine independence needs a different model, or at minimum a reviewer explicitly prompted to refute rather than confirm.
  • Framing beats headcount — one reviewer instructed to argue against the work catches more than three redundant "does this look right?" checks.
  • It raises the floor, not the ceiling. Worth it for anything hard to verify or high-stakes, overkill for a one-line fix.

The rationalization problem

When I ask a model to check its own output, the failure mode isn't obliviousness — it's a second, equally confident pass over the same reasoning. The logic that produced a subtly wrong answer is exactly the logic that will explain, calmly, why the answer is right. Self-review inherits the blind spot it's supposed to catch.

A second agent doesn't have that problem by default, but it has a milder version of it: if it's the same model, same weights, same training distribution, it will often converge on the same conclusion for the same reasons. Two instances of the same model aren't two opinions — they're one opinion asked twice.

What actually changes the outcome

Two things move the needle more than simply adding a second reviewer:

  1. A genuinely different model. Different training data and different failure modes mean a different reviewer is more likely to catch something the first model is systematically bad at, rather than randomly bad at.
  2. An adversarial prompt, not a confirmatory one. "Does this look right?" invites the same sycophancy that let the first mistake through. "Try to find a reason this is wrong" produces a different kind of scrutiny — it has to construct a counter-argument instead of pattern-matching to "looks fine."

I've watched a reviewer instructed to refute catch things a reviewer instructed to check walks straight past, using the identical model both times. The prompt did more work than the extra agent did.

Diversity beats headcount

Going from one reviewer to two catches real bugs. Going from two to five, if they're all reviewing the same way, mostly burns tokens. What helps at that point isn't more copies — it's different lenses: one reviewer for correctness, one for the specific way this kind of code tends to break, one that just tries to reproduce the failure. Three reviewers who could each miss the same thing are worse than three who'd each miss something different.

Where I'd actually spend it

I don't run adversarial review on everything — that's real latency and real cost for marginal benefit on a one-line fix. I reach for it when the output is hard to verify by reading it (an algorithm, a migration, anything where "looks plausible" and "is correct" can diverge), or when being wrong is expensive. For anything I can check myself in ten seconds, I just check it myself.

The honest limit

A second agent raises the floor. It won't catch a mistake that's wrong for reasons no model in the loop understands, and it's not a substitute for a human who actually has to live with the consequences. It's a good, cheap check on a specific, common failure mode — not a guarantee.

The next time you're tempted to trust a clean-looking answer, the cheapest fix probably isn't a longer prompt. It's a second pass that's actually trying to disagree with you.