The test passed because the code never ran

We had a watcher process that was supposed to detect when an application bundle changed on disk and refuse to act on it if the new bundle failed a signature check. The dangerous outcome was obvious: acting on a bundle it should have rejected. So we wrote the obvious regression test.

// given: a tampered bundle replaces the good one
await watcher.run();

assert(!log.includes("update applied"));   // no success claim
assert.equal(hashOf(installPath), before); // nothing changed on disk

Two assertions, both about the bad thing not happening. Green for weeks.

It was worthless.

Why

The fixture built its “old” and “new” bundles by signing the same input twice. Ad-hoc code signing is deterministic — same input, same output, byte for byte. So the two bundles were identical, the watcher’s early if (unchanged) return fired, and it took the no-op path.

Nothing was applied. Nothing changed on disk. Both assertions were satisfied — by a code path that had nothing to do with signature checking. The signature check never executed a single time in that test.

The general shape

A negative assertion passes just as happily on a path that never ran. It cannot distinguish “the guard held” from “the guard was never reached,” and those two states look identical from outside.

This is the same family as an outcome literal sitting next to the branch it describes:

if (shouldReject(bundle)) {
  return { ok: false, reason: "signature" };
}
return { ok: true, reason: "signature-verified" };  // says it. didn't do it.

In both cases the claim is decoupled from the thing it claims about. Nothing forces them to move together, so eventually they don’t.

What we do now

Every negative assertion gets two companions.

Assert the fixture is actually adversarial. If the test depends on two inputs differing, prove they differ before relying on it. One line, and it would have caught this on day one:

assert.notEqual(hashOf(oldBundle), hashOf(newBundle),
  "fixture is not adversarial: bundles are identical");

Assert the code path under test executed. Make the subject leave a mark and require it:

assert(log.includes("signature check: FAILED"),
  "signature check never ran");
assert(!log.includes("update applied"));

Now the test fails loudly in both directions. If the guard is deleted entirely, the first assertion fails. If the fixture rots into a no-op, the first assertion fails. The negative alone could not detect either.

And pair it with the positive. Write the test where the bundle is valid and the update does apply. If the negative test still passes after you break the guard open, it was never testing the guard.

The wider version

Take any test that passes and ask what else would make it pass. If the answer includes “the function not being called,” you have a test that converts an open question into a false answer — which is worse than not having written it, because it stops anyone from looking.

← All notes