Prove it can fail before you trust it passing

By Daniel S. · July 18, 2026

TL;DR: You wrote a check. It says PASS. How do you know it is capable of saying anything else? A gate that has only ever returned green is indistinguishable from a gate that is structurally incapable of returning red — and you cannot tell the two apart by reading the code, because the bug is invisible from inside the head that wrote it. So you make it fail on purpose. But that's only the first rung: this post is also about the gate that can only fail, and about the nastiest case of all — a positive control that fires correctly while the bug walks straight past it.

The asymmetry nobody budgets for

Testing culture is lopsided. We write a test, watch it pass, and ship. We almost never watch it fail, because failure is what we're trying to avoid, and a red test feels like a problem rather than a measurement.

But look at what a green check actually asserts. It says "I examined the thing, and the thing was fine." That's a conjunction, and a conjunction is only as strong as its weakest half. If the first clause is false, the second is unconstrained — a check that examined nothing will report that nothing was wrong, with total sincerity and a zero exit code, forever.

Here are the ways I watched that happen in a single week, in tooling written specifically to prevent this class of bug, by people who had just finished writing down the lesson.

Instance 1: the gate that inspected zero bytes

I have a pre-commit hook that blocks personal data — street addresses, phone numbers — from entering a repository. Before publishing something, I ran it by hand over the files I was about to push:

$ check-pii.py post-one.md post-two.md
$ echo $?
0

Green. Clean. Ship it.

It had not looked at those files. The tool reads the staged diff — git diff --cached — and silently ignores filename arguments. Nothing was staged. So it examined an empty diff, found no violations in it (correctly!), and exited 0.

That zero was a true statement about nothing. And I was seconds from reporting it as a clean bill of health — on a blog post about checks that pass while examining the wrong thing. The tool was fine. The hook was fine. My invocation silently converted a security gate into a random number generator that happened to be stuck on zero.

Instance 2: the gate whose missing dependency read as a pass

Same week, same tool, one layer down. Someone installed that hook into a second repository, following the documented procedure exactly. The hook file was present. It was executable. It was in .git/hooks/.

It checked nothing, because the scanner it calls lives at a path relative to the first repository, and in this one that path didn't exist. The code did the natural, friendly, catastrophic thing:

if [ -f "$scanner" ]; then
    "$scanner" || exit 1
fi
exit 0     # <-- and here we are

A test commit containing a fabricated street address and phone number sailed through without a murmur.

An if around a security check is a bug. "I could not run" and "I ran and it's fine" must never share an exit code. If the gate can't execute, the gate fails:

[ -f "$scanner" ] || { echo "scanner missing — BLOCKING" >&2; exit 1; }
"$scanner" || exit 1

The rule that catches both: make it go red on purpose

Neither of those was caught by careful code review. Both were caught the same way: by feeding the gate something it was built to reject and demanding that it object.

# The positive control. Stage a specimen the gate MUST catch:
# a street address and a phone number, sitting in ordinary prose.
$ git add specimen.md
$ check-pii.py
🔴 BLOCKED — identity aggregate in added lines
    specimen.md: street address → [redacted — see the footnote]
    specimen.md: personal phone → [redacted]
$ echo $?
1

Now the exit-0 on my real content means something, because I've established the instrument has a functioning needle. Before that, the 0 and the 1 were the same symbol.

This is a positive control — the oldest idea in experimental science, and the one software borrows least. A lab that runs an assay without a known-positive sample isn't running an assay; it's generating numbers. Same with your gates. A gate nobody has seen go red is a decoration.

But red is not enough — it has to be red at the right thing

Here's where most treatments of this stop, and where it gets interesting.

That same PII gate, when it was first written, was tested with a positive control. The control fired. The gate went red. Everyone was satisfied.

Then someone fired it at the actual specimen it had been built for — a real line from the real document that caused the whole gate to exist — and it did not fire.

The phone-number pattern had a guard on it, (?![\w.-]), meant to stop partial matches — don't fire if the digits run straight into more word-ish characters. But look at what's inside that character class. . is in there. So the gate treated the full stop at the end of an English sentence as if it were part of the phone number:

Phone NNN-NNN-NNNN. ← the trailing period is inside the guard

The sentence-ending period made the pattern refuse to match. The single most common way a phone number appears in real prose — at the end of a sentence — was the one form the gate could not see. And that was the exact shape of the data in the document the gate had been written to catch. The gate would have missed its own specimen.

Nobody found that by reading the regex. Several people had read the regex. It was found because the rule isn't "write a positive control" — it's "watch it fire on a specimen it must catch." Those are different rules, and only the second one has teeth.

The evil twin: the gate that can only fail

The mirror image is just as real and gets far less attention, because it feels safe.

I wrote a check that scanned our internal notes and told me which ones I still owed a reply to. It reported 26 outstanding. The real number was 14. It was convicting honest work — reading correctly-signed notes as unsigned, because it only ever examined the first block of a document and the signature was often in the second.

Nobody's data was at risk, so why does it matter? Because of what a false red does to a human being:

A gate that cries wolf gets bypassed. Not maliciously — pragmatically. The first time it blocks a clean commit you investigate. The third time, you reach for --no-verify. And once that's muscle memory, the gate is worse than absent, because it's still on the org chart. Everyone believes the control exists. Nobody notices that it's been routed around for a month.

So: over-firing is a defect, not an excess of caution. A gate must be provably capable of going green on a correct input — and the gate that asserts a condition which can never be true, blocking every clean run forever, is exactly as broken as the one that can never fail. You find both the same way: by watching, on purpose, in both directions.

The trap under the trap: the control fired, and the bug walked past

This is the one I'd most want you to leave with, because it defeats a naive reading of everything above.

A colleague's hook needed Python to run its real check. Being careful — properly careful, the kind of careful this whole post is arguing for — they added a liveness check first:

command -v python >/dev/null || { echo "python missing"; exit 1; }

Reasonable. Testable. And it passed.

On Windows, command -v python is satisfied by the Microsoft Store stub — a placeholder python that exists on the PATH purely to open the app store. It prints an advert and exits 49. It is not Python. It has never been Python.

So the liveness check succeeded, the hook concluded its certifier was available, ran it, and reported the result as "the certifier says otherwise." The certifier had never run. The name resolved. The tool did not exist.

Sit with the shape of that, because it's vicious: there was a positive control, and it worked, and it proved the wrong proposition. It tested whether the gate could fire — its sensitivity. It never tested whether the gate was aimed at what it claimed to be aimed at.

Those are two separate properties:

A check on a stub is a check with perfect sensitivity and no aim. It will pass its own red test all day long. The red test is necessary. It is not sufficient. Anyone who takes "always write a positive control" as the whole lesson has bought a slightly more sophisticated false comfort.

The way you catch an aim failure is to hold everything constant and change only the subject, then demand the verdict change with it. If it doesn't, your gate is looking somewhere else — and the fact that it can go red proves nothing at all about where.

What to actually do

Every gate ships with two cases or it does not ship:

1. A case that proves it can go GREEN on a correct input. Otherwise it's a wolf-crier, and it will be bypassed into irrelevance.
2. A case that proves it can go RED on a bad one — and not just any bad one: the specimen it was written to catch, in the form that specimen actually appears in the wild, trailing punctuation and all.

Then one more question, which is the one almost nobody asks:

3. When it went red — did it go red for the reason I think? Vary the subject, hold everything else fixed, watch the verdict follow. If the verdict doesn't track the subject, you have a gate that is exquisitely sensitive to something, and you don't know what.

And the meta-lesson, which is the only reason I trust any of this: not one of the bugs above was caught by its author re-reading their own work. Every single one was caught by a measurement, or by a second pair of eyes. At the moment you write this class of bug, the wrong check is indistinguishable from the right one from inside your own head — that is precisely what makes it this class of bug.

Which is why the fix is never "be more careful." A red test is code. It runs on the day you're tired.

Footnote: the gate blocked this post

I wrote this article with the specimen strings in it — a fabricated street address, a couple of fake phone numbers — because concrete examples teach better than placeholders.

The gate blocked the commit. Obviously. It cannot tell using personal data from quoting it, and I had put three matching strings into a file and staged it.

This is precisely the over-firing case from earlier in this post, and I want to be honest about how I resolved it, because the tempting move is to add an override flag — a # leak-guard: ignore comment, an --allow switch, something. The gate deliberately has no override, and that's the right call: an override is a lever, and the person reaching for it is always, in the moment, completely certain their case is the exception. The commit that leaks the real thing will be authored by someone who is certain.

So I did what the gate exists to make me do. I removed the data — the examples above are shapes now, not strings — and the commit went through. Every literal in this post that could have matched is gone, which is why the code blocks say [redacted] and NNN-NNN-NNNN.

The cost is real: this article is slightly less vivid than the one I set out to write. I'll take that trade every time, and so should you. A control that's convenient in the one case you're sure about is a control that isn't there in the case you're wrong about.