We set out to turn ten catches into a rate. The rate was inflated, and the bug was ours.

Why we wanted a rate

Ten falsifications is an anecdote. It proves the claim-vs-effect gap happens. It does not tell anyone how often.

A rate is what a buyer can act on. An insurer cannot price an anecdote, and a risk committee cannot put one in a board pack. So we set out to produce a number: take a population, grade every candidate whatever the outcome, and publish catches over graded runs.

That plan is sound. The tooling underneath it was not.

The part that worked

We re-ran our published catches under fixed tooling. sympy-24909 reproduced exactly. The agent claimed success, the test still failed, and the receipt says the same thing today that it said when we signed it.

That matters because we had just found that four of our nine published instances could not be re-run at all. Fixing that was step one, and the fixes are recorded.

The part that did not

We ran 80 candidates from eight public repositories. Forty passed the gate and were graded. Eight came back as falsifications: the agent claimed success and the test failed.

Four of those eight were wrong. All four came from arrow, and all four looked like this:

TOTAL                 2297    855    690     21    54%
FAIL Required test coverage of 99% not reached. Total coverage: 53.87%
======================== 723 passed, 1 warning in 3.15s ========================

723 tests passed. The agent’s patch was correct and the agent said so. We recorded it as a fabrication anyway.

The failure was --cov-fail-under. arrow configures a 99% coverage requirement, and coverage is measured across the whole package. Running one test file can never reach 99%, so the run exits non-zero no matter what the agent does. Our gate saw a non-zero exit and called it a red test.

A fourth arrow catch survived the same check, and it should have: its output reads 1 failed, 187 passed. A real assertion failed alongside the coverage message. We kept that one.

What the gate was actually checking

It checked that the test file exited non-zero. It never checked why.

That is the whole bug. Everything downstream inherited it. For a while we had four signed “catches” whose only defect was that the agent had been right.

This was the second time

We had seen this before. Round 5 of our harvest documented the same confound, and the note says the coverage gate produced the failure. We believed we had fixed it.

We had not, because the fix was a check on the exit code rather than on the cause. A patch that treats the symptom will reproduce the symptom.

What we changed

The gate now lives in one place, as a single function that classifies a pytest run from its exit code and its output.

It rejects a run when the failure is a coverage threshold and no test failed. It keeps a run when a real test failed, even if coverage also failed. It checks for a missing module before it checks anything that could be confused with dependency drift, because a missing module is the specific diagnosis and drift is the vague one.

Three regression tests pin the three ways it was wrong, using the verbatim output from the runs that exposed them. They fail the suite if any of it comes back.

The full suite is 314 tests.

We are not publishing the rate

The honest result after the correction is 5 catches from 37 graded runs, or 13.5%.

We are not publishing that as a rate, and the reason is not modesty.

The interval is not stable. Treating the runs as independent gives 2.5% to 24.5%. But runs cluster inside a repository, and one repository contributed a single graded run that happened to be a catch. Correct for that and the interval widens to roughly 0% to 49%. Drop that one repo and the remaining six sit between 3% and 17%.

Three defensible methods, three different answers. A number that moves that much under scrutiny is not ready to stand behind. Publishing it would repeat the error this post is about: a confident figure whose denominator does not support it.

What we are publishing instead

The reproduction. sympy-24909 re-ran and matched its receipt.

And this. Our own measurement tool was overstating our own results, and we found it by reading the output rather than trusting the flag.

That is the part worth your attention, more than any percentage. Anyone can publish a rate. The useful question about a verifier is whether it will turn the same scrutiny on itself, and the only way to answer that is to show it happening.

The other thing we found

Getting those four instances to re-run at all turned up five bugs in our harvest tooling. The worst: our environment installer invoked uv from the repository root, so uv discovered our own pyproject.toml and pulled the turiya package into the resolution. turiya is also a name on PyPI, and its dependency tree conflicts with the era-pinned requirements of the repository being set up. Installations failed with “unsatisfiable requirements” and our installer reported success anyway, because it never checked the exit code.

The two failure buckets that bug produces account for 21 of the 61 candidates our last harvest rejected. We have not measured how many of those 21 were our bug rather than a real property of the candidate. Settling that means re-running the harvest, and we would rather say it is unmeasured than estimate it.