A test you never saw fail proves nothing

Two suites, both green

suite-that-lies/ and suite-that-measures/ test the same three functions, and both pass. Nothing in their output tells you which one is worth having.

The three shapes in the lying suite are ones people actually write:

// 1. It asserts on a value the test itself made up. The rule under test never runs.
const limit = { enabled: true, value: 20 };
expect(limit.enabled).toBe(true);

// 2. It expects an exception that cannot be thrown. 5 is under 100, so nothing happens.
expect(() => spend(5, 100)).not.toThrow(OverBudget);

// 3. It compares an object with itself. True for any implementation, including `return name`.
const out = normalise("  Ana  ");
expect(out).toBe(out);

None is a strawman. Each is what a real test decays into after the code it was written against moves.

The test grades two suites

const verdicts = MUTATIONS.map((m) => runMutation(ROOT, m, "suite-that-lies"));

expect(verdicts.every((v) => v.applied)).toBe(true);  // the code really was broken, three times
expect(verdicts.every((v) => !v.caught)).toBe(true);  // and the suite never noticed

Three real defects walk past three passing tests. The measuring suite catches all three.

The part almost nobody checks

const v = runMutation(ROOT, { from: "a line that is not in the file", ... });

expect(v.applied).toBe(false);
expect(v.caught).toBe(false);   // green — from a suite that is actually good

A green mutation and a blind suite look identical, and they call for opposite reactions: one says fix your tests, the other says your mutation missed. Which is why runMutation reports applied at all — a mutation report without it is a lie that flatters you.

There are two ways to miss, and both are quiet:

	What happened	What it looks like
The anchor is gone	Your search-and-replace matched nothing. The file is untouched	Green
The target is not watched	The file changed, and nothing under test imports it	Green, and applied is true

The second one is nastier: applied is necessary and not sufficient. The suite pins that too.

Three ways I have personally been fooled by this, in one day

Not hypothetical — these are from an afternoon of building gates, and all three came back green:

1. Moved a file to another repository to test a cross-boundary check — and nothing imported that file, so there was no edge to cross. Green.
2. Added an optional property to a type to break a structural equality assertion. Optional properties leave two types mutually assignable, so the assertion still held. Green.
3. Refined a schema that was a copy of the one the code under test actually reads. Green.

Every one of them looked like a legitimate mutation while writing it. The tell is always the same question, and it is worth asking out loud before believing any green: did I change the thing this test is looking at, or something next to it?

What to do with this

Copy src/mutate.ts. It is forty lines and has no dependencies. Point it at the file a test claims to protect, break that file on purpose, and see what happens — then check applied before you believe the result.

You do not need a mutation-testing framework to start. You need to have watched one test go red for the right reason, and to know the difference between that and a mutation that missed.

License

Apache-2.0 — see LICENSE. This is a demonstration, not a package. Copy what you need.

---

Built by Vorluno — a software studio from Panamá.