If the same agent writes the code and the tests, a passing suite proves they agree with each other — not that either is correct. You need tests that do not share the agent's blind spots.
TL;DR
- Agent-written tests can pass by asserting the wrong behavior — circular confidence.
- Property-based testing checks invariants across many generated inputs, catching cases nobody thought to write.
- Mutation testing measures whether your tests actually detect bugs.
- Together they give correctness signal that does not rely on the agent being right.
The circularity problem
Ask an agent to implement a function and write its tests, and you get a green suite. But both came from the same model with the same misunderstanding. If the agent misread the requirement, the code is wrong and the tests assert that wrong behavior — and they pass. Green means "the code and tests agree," not "the code is correct."
Breaking that circularity needs testing techniques whose correctness does not depend on the agent's interpretation.
Property-based testing: invariants over examples
Example-based tests check specific inputs you (or the agent) thought of. Property-based testing checks properties that must hold for all inputs, then generates hundreds of cases — including the weird ones nobody enumerated.
import { test, fc } from '@fast-check/vitest'
// A property: encoding then decoding returns the original
test.prop([fc.string()])('roundtrips losslessly', (input) => {
expect(decode(encode(input))) === input
})
You did not list the inputs. The framework throws empty strings, unicode, huge values, and known edge cases at your invariant and reports the smallest one that breaks it. This routinely surfaces exactly the edge cases agents silently skip.
Example tests check the cases you imagined. Property tests check the cases you didn't — which is where agent code tends to fail.
Mutation testing: testing your tests
A green suite tells you the tests pass. It does not tell you the tests would catch a bug. Mutation testing answers that directly: it deliberately introduces small faults ("mutants") into your code — flip a < to <=, swap a + for a - — and checks whether your tests fail. If a mutant survives, you have a gap: code no test actually exercises.
Original: if (count > limit) ─▶ tests pass ✓
Mutant: if (count >= limit) ─▶ tests still pass? ✗ gap found
For agent-written tests, mutation testing is the antidote to false confidence. A high mutation score means your tests genuinely constrain behavior, not just decorate it.
Putting it in an agentic workflow
- Give the agent property tests as the target. Invariants are harder to satisfy with wrong code than cherry-picked examples, so they make a better "done" signal.
- Run mutation testing in CI on changed code. Surviving mutants become review flags.
- Treat agent-written tests as suspect until they earn trust — review them, and let mutation testing audit them.
- Keep a human-owned core suite for the critical paths, independent of whatever the agent generated.
The principle
The goal is correctness signal that is independent of the agent's understanding. Property-based testing supplies inputs the agent did not choose; mutation testing verifies the tests have teeth. Layer them over agent output and a green build starts to mean something again — not "the agent agrees with itself," but "this behaves correctly under pressure it did not get to pick."