The flag was fixed. The sentence wasn't.

By Daniel S. · October 5, 2026

The usual disclosure: AI agents did most of the hands-on work below: building the pipeline, writing the cards, and then, on other days and on other machines, catching each other's mistakes. The product is patchtriage.com, a board of vulnerabilities that are being exploited right now. It is rebuilt every weekday from the CISA Known Exploited Vulnerabilities (KEV) catalog, vendor advisories and a threat bulletin. Every defect below was fixed before this post went up. Four of them were live on the public site first.

The setup, in one paragraph

Each vulnerability on the board is a card. A card has structured fields (CVE, CVSS, "on KEV: yes/no", deadline) and sentences (the evidence line, the attack chain, the patch instructions). A script checks the structured fields against the live KEV catalog every morning. If CISA has added a CVE since the card was written, --fix flips kev: false to kev: true. That fix is correct, and it only touches the boolean.

Six defects, eight days

1. The card that said yes and no at once. On 2026-09-22 the weekday refresh flipped a Check Point card to kev: true. The next field on the same public card still read "no evidence: Check Point's advisory makes no in-the-wild claim and the CVE is absent from CISA KEV." That sentence had been true when it was written, six days earlier. It was live for about fourteen hours before a pass from a second machine read it.

2. The same bug, three weeks older. We then wrote a check for that shape and ran it over every brief in the repo. It fired on a second card, a SonicWall SMA1000 pair. The title read "…not yet on KEV" while the card rendered KEV added 2026-09-02, deadline 2026-09-05. It had been on the public board since about September 2. Nobody reported it. The check found it on its first run.

3. The tool that was right while the outcome was wrong. On 09-16 an Acronis CVE landed on KEV with a three-day deadline. The script that stubs new KEV entries reported it as "already in drafts" and skipped it. That was true: the CVE was in the drafts, marked kev: false, because the bulletin that introduced it predated the listing. An agent review pass caught it the same day, before it reached the board. "Present in a draft" and "present with the right KEV state" are different claims, and nothing compared them.

4. The queue with no clock. A GitLab CVE was stubbed correctly on 09-14, the day its federal deadline passed, with patched: UNREVIEWED. It sat there for nine days because nothing reports how long a stub has waited. The stubbing worked. Nothing ever looked at how old the queue was getting.

5. The citation that said the opposite. The card for a JFrog Artifactory CVE (CVE-2026-66384) cited OpenAI's incident report on the July Hugging Face compromise. The card said the flaw "contributed to the compromise" and that the chain ended in "supply-chain execution." A second agent on another machine, which hadn't written the card, reviewed it and read the 51-page report. The only section that mentions this CVE opens by saying the activity did not contribute to the Hugging Face compromise. That compromise used a separate Artifactory bug. OpenAI also found no evidence that anything pulled or ran the poisoned image. The card was wrong in the direction that matters: a reader who patched this CVE would believe the Hugging Face route was closed, and it was not. It was live for about four weeks.

6. A real ID on the wrong slide. The September patch brief is built partly from a vendor webinar deck. Our checks asked of every CVE number on a slide: is this a real CVE? The deck re-uses identifiers across slides, so a real ID printed over an unrelated slide passed. One Windows Graphics CVE was ranked #12 as "Critical" on the strength of slides about Outlook, Skype and Dynamics, none of which discuss it.

The pattern

None of these was a component misbehaving. Every one was a check answering a narrower question than the one we actually cared about:

The first row is the one to take home. When you fix a field mechanically, you have to find every sentence a human derived from it, or you've just built a machine that makes your document contradict itself on a schedule. The fix doesn't cause the contradiction. It uncovers one that was always going to happen once the facts moved. Before the fix, the board was consistently wrong. After it, the board was wrong in only one field, and that's harder to spot.

What we built, and where it stops

• A contradiction check. It flags a card where kev: true sits next to KEV-absence language, or where "exploited: yes" sits next to evidence text that opens by denying exploitation. It flags; it doesn't refuse. A gate that blocks the daily publish on a false positive gets switched off within a week, and then there is no gate. It ships with 15 tests built from the real incident text, and 14 of them fail against the old code. The limit: the patterns are lexical. "CISA hasn't catalogued it" would slip past.
• A clock on the queue. Any KEV stub older than three days that hasn't reached the board gets reported. A deliberate decision not to publish one has to be written into the file with a reason. Otherwise the stub keeps showing up in the report.
• A slide-title check. Each CVE on a slide must share at least one meaningful word with its official title. A mismatch drops that slide's tier loudly, and the check refuses to guess the "right" ID, because guessing is the failure it exists to stop. The limit: a wrong ID whose title shares a generic word with the slide ("Driver") still passes. It narrows the hole; it doesn't close it, and the code says so.
• Defect 5 got no new check, because there isn't a cheap one. What caught it was a reviewer who didn't write the card, reading the primary source rather than the summary of it. Nothing cheap we could have automated would have caught it.

If you run anything like this

1. List every field your automation can change, then list every sentence written from it. If the second list isn't empty, the automation owes a re-review signal every time it acts.
2. "Exists" is not "correct." Any check that answers is it there? needs a partner that answers is it right?
3. Queues need clocks. A queue that reports its size and not the age of its oldest item can look healthy for nine days.
4. Citations are claims about someone else's document. If nobody has read the source since the card was written, the citation is decoration.

Six defects in eight days sounds bad, and for a product that sells correctness it is. The number worth watching, though, is how each was found. One was caught by a review pass, three by a second machine checking the first machine's work, and two by checks that didn't exist a week earlier. None came from a reader. Those checks are what I trust now, and the first time each one runs, it runs on work I shipped.