How we verify, and what we refuse to publish

The question we ask

One narrow, checkable thing: an AI system asserted it did something, and it didn't. Not "is the model good" (an eval), not "is the system safe" (a red team). Whether the effect matched the claim.

On the system's own account, a failure and a success are indistinguishable. The patch that did not fix the bug reports that it did. Nothing inside that report separates the two cases, because the report is the thing in question. So we do not read it. We re-run the work.

1. Clone the real repository at the buggy commit.
2. Apply the test change from the real fix commit.
3. Gate: prove the test is red for a reason that has to do with the code.
4. Run a real agent (LangGraph ReAct over a DeepSeek frontier model) to produce a fix.
5. Re-run the test. A claim is a non-empty patch; the effect is the result.
6. System claimed success and the test still fails → sign a receipt.

How a claim becomes a test

The pipeline above needs a repository and a test that already exists. Most claims arrive without either. A vendor says its agent does not invent an answer when a tool fails. Nothing tests that yet.

So we compile one. Hand us the claim and we generate a battery of probes that try to break it, from an empty file to a null that lands in the middle of a multi-step pipeline.

A generated probe is a proposal, so a deterministic checker decides which ones run. It re-creates the probe's setup, re-derives the ground truth from what is actually on disk rather than trusting what was claimed, and requires that an honest answer and a fabricated one land on opposite sides. A probe that cannot separate those two does not run.

A record can be trusted only if we can name who wrote it and who could change it. Here that record is the repository's own history and the test from the fix commit.

We say that part out loud for the same reason we publish our rejects. A probe battery that was never checked is decoration, and a signature over decoration proves nothing. What survives the checker gets run, and what happened gets signed.

The limit. This covers the failure-mode families we already know. Handing it any claim in English and getting a correct spec back is not solved, by us or by anyone. When a claim falls outside what we can compile into a checkable test, we answer indeterminate rather than invent a score.

The compiled battery and its rate are signed and on the record: what the probes found.

The full candidate ledger

Our most recent run: 120 candidates. Most did not become catches, and we report every one.

Only 44% of candidates were testable at all. The rest are listed below with the exact reason, not quietly dropped.

Why we reject most candidates

The gate is the product. Its job is to decide whether a red test is fair before the system runs. If a test is red for a reason unrelated to the system, any "catch" built on top of it is a fabrication: a valid signature over a worthless claim.

A caveat on two of those rows. On 2026-09-20 we found that our environment installer invoked uv from the repository root, so uv discovered our own pyproject.toml and pulled the turiya package into the dependency resolution. That package is also a name on PyPI, and its dependency tree conflicts with the era-pinned requirements of some repositories. Installations failed, and our installer reported success anyway, because it never checked the exit code.

The two buckets that bug produces are missing-dependency and dependency-version-drift, which together account for 21 of the 61 above. We have not measured how many of those 21 were our installer rather than the candidate. Settling it needs a re-run with the fixed installer, and we would rather say it is unmeasured than estimate it. The installer and the gate are both fixed and covered by regression tests.

The thirteen false catches we destroyed

Across our runs, auditing the signed receipts found 13 that were worthless. Six died inside our own environment (TypeError / AttributeError from our site-packages). Two were arrow receipts we had already published, killed by our own coverage gate. And five more were caught on 2026-09-20, before publishing. The same coverage failure, returning after we believed we had fixed it. Every one had a valid Ed25519 signature, a valid content hash, and a non-empty patch over a claim that wasn't true.

They were meaningless: in six cases the test died inside our environment before the system's change was ever exercised, and in seven cases our coverage gate failed a fix that had actually passed every test. This is the failure mode our product exists to catch, and it keeps happening inside our own pipeline.

The second wave is the uncomfortable one. We documented this exact failure on this page, fixed the gate to catch it, and then found it again, because the fix was correct and in the wrong place. The classifier knew a coverage failure was not a test failure; the code that decides a verdict never called it. The full account is here.

A signature is not evidence. It is evidence that we said something. Whether the something is true is a separate question, and here the answer was no.

The gate missed these because it watched for ImportError and ModuleNotFoundError; these arrived as TypeError and AttributeError raised from our own site-packages. It now checks for exceptions originating in the environment, and for the absence of a pytest summary line at all. The eight receipts are kept, not deleted. Deleting them would hide the failure.

Why half of one run produced nothing

Testing a 2023 commit against 2026 libraries is not a fair test. Resolving dependency ranges against today's index means old code meets new libraries. When a dependency has since made a breaking change, the test dies on import. One of our runs scored 21 candidates, 0 testable, 0 catches. Not because the systems were good, but because the environment was broken.

The fix is to install from the lockfile at the candidate commit. Where no lockfile exists, we do not run the candidate at all. A repo that pins nothing cannot be reproduced, so its result cannot be attributed to the system.

Which model was under test

Two, across the record: the early catches were produced by DeepSeek V4 (deepseek-chat); everything from round 3 onward by DeepSeek V4.1 Flash (deepseek-flash), released 10 September 2026.

An honest caveat. The first 10 receipts were signed before the schema carried a model field, so the model is attributed on each receipt page from the harvest round that produced it: reconstructed, not read out of the signed record. We deliberately did not re-sign them to add it: re-issuing a signed record to insert metadata is precisely the practice this product exists to prosecute. New receipts carry the model natively.

How to read our record

10 signed catches across 6 repos: every one re-verifiable with the published public key.

The number is a floor, not a total. Earlier runs went through harness bugs that silently suppressed candidates. A miss is not a fabrication, but it is a gap, and it is why we say "at least" in our own heads.

We do not claim "first." Others do claim-checking. We claim only what is ours: we sign, we verify the effect rather than the sentence, and we publish our rejects.