Your green test is a claim about four things — and you're only checking one
TL;DR: A passing check is not one claim, it's four: the check could have failed (instrument), it measured the thing you think it measured (subject), its reading covers the claim you're making (scope), and the thing it verified is still in force (moment). Last week my homelab handed me a genuinely-passing check that was wrong on each of the four — different tools, different layers, same disease. Below: the four claims, the real failure behind each one, and the fix for each, none of which is "be more careful."
Where this came from
I run a small homelab fleet — a few GPU boxes, agent-driven installs, verification scripts gating every risky step. Over one week of building out a new machine, we hit at least a dozen cases where a check printed the reassuring answer while the thing it guarded was broken, missing, or never examined. The uncomfortable part: most of those checks were written that same week, by people (and agents) who had just finished writing down the lesson from the previous one.
So this isn't a post about sloppy testing. It's about the four distinct claims hiding inside every green checkmark, and how each one fails silently unless you check it on purpose.
Claim 1 — the instrument: could this check have failed at all?
A gate that has only ever returned PASS is indistinguishable from a gate that cannot return anything else.
The week's cleanest specimen: an SSH hardening check that probed a config value and, when the probe couldn't run, emitted the sentinel UNKNOWN. The assertion downstream checked whether the reading contained the string no — as in passwordauthentication no. Look at those two facts side by side. The word "unknown" contains "no." The branch that fires when the check never ran was the branch that passed. A security certification could be produced by the exact condition that should have voided it.
The same shape turned up in an eval harness days later: three flatly wrong gold answers scored a perfect 1.00, including one that was deliberately written as a trap — and the trap didn't trip.
The fix is mechanical, not moral: every gate ships with a test that makes it go red, and you watch it fail once before you ever trust it green. Feed the gate its own failure sentinel and assert that it fails. A gate nobody has seen red is a decoration.
Claim 2 — the subject: did it measure the thing you think it measured?
A GPU acceptance tool of ours looped over every card in the machine and checked each one's device ID against an expected value. Except the loop stored the ID in a single variable, overwritten each pass — so the assertion only ever tested the last card to enumerate. Same box, same two cards: the verdict flips depending on enumeration order. Bench a new card in a machine that still has its old card seated, and you can get a clean PASS that certified the wrong silicon.
Different layer, same claim-failure: mid-incident, a box's hostname was read off the network controller's web UI instead of off the box. The controller showed a stale default name, which "proved" the machine was a leftover install, which nearly produced a do-not-touch order for a perfectly healthy server. The reading was real. The subject was the controller's cache, not the box.
The fix: a reading is invalid unless it records its subject from an independent source. Our GPU tool now logs each card's immutable UUID alongside every measurement — the verdict names the exact unit it applies to. And identity gates read from the box itself, never from a dashboard that talks about the box.
Claim 3 — the scope: does the reading cover the claim you're making?
After the sentinel bug above was fixed — properly, with red tests — the repaired gate got attacked again, and a second hole of a different kind fell out. The probe asked the SSH daemon for its effective config, which reports the global settings. It does not evaluate Match blocks — conditional overrides for particular users or networks. So a config could truthfully report passwordauthentication no while a Match rule left password login wide open for an entire subnet.
Read that carefully, because it's the subtlest of the four: the probe did not die, and it did not lie. It answered a narrower question than the gate claimed to have asked. "The default is key-only" got published as "the box is key-only." Every word true; the certification worthless the day someone added a conditional.
The fix: when a check cannot answer within the scope of the claim, that result must not land in the "passed" bucket. Our version now asserts the one thing it can decide — no conditional overrides exist at all — and fails loudly if one does, with a message saying the global reading can no longer speak for the whole box.
Claim 4 — the moment: is the thing you verified still in force?
The oldest one, and still the most common. A hardened SSH config file, present on disk, correct in every line — that the daemon had never loaded, on a box that had been certified key-only and was in fact accepting passwords from the whole LAN. A Secure Boot toggle reading "enabled" while the platform sat in setup mode — a state where the switch is on and the enforcement is off. A firmware knob written and acknowledged that hadn't taken effect pending a reboot nobody had done.
Config existing and control operating are different facts. The gap between them is invisible in every file-reading check ever written.
The fix compresses to four words: ask the daemon, not the file. Assert the running state — the loaded config, the enforcing mode, the live register — and treat "written but not yet in effect" as its own tracked condition, never as done.
The pattern in the catches
Here's the statistic that reorganized how we work, and it held at roughly fourteen-for-fourteen across the week: every one of these was caught by someone other than the author, or by a measurement. Not once by the author re-reading their own work.
That's not a discipline problem. At the moment you make this class of error, the wrong subject is indistinguishable from the right one from inside your own head — that's what makes it this class of error. A note-to-self saying "check your subject" fires only when you already suspect, which is exactly when you don't need it.
So the fixes above share a shape: none of them is a reminder. A red test is code. A recorded UUID is a field in the output. A scope assertion is a branch that fails. A running-state probe is a different command. They work on the day you're tired, at the hour the machine is in a datacenter and you're not, which is the only day that counts.
The Monday checklist
Next time a check goes green on something that matters, ask it four questions:
- Have I ever seen this check fail? If not, make it fail on purpose before trusting it.
- What exactly did it read, and does the output say so? Subject recorded, from a source independent of the thing under test.
- Is the claim I'm repeating wider than the question the probe answered? Global vs. conditional, one item vs. all items, the sample vs. the population.
- Is the verified thing running, or just written down? Ask the daemon, not the file.
Green means the check passed. It has never meant more than that — the other three claims were always yours.