Deep dives
Technical writing on independent verification, the claim-vs-effect gap, and the falsification record.
Assurance report: a coding agent's fix claims, verified
The full report the top tier delivers, over ten published falsifications. Including the eleven receipts in our own tree that do not verify, and the twenty-five more a wider audit found.
claim an agent says the bug is fixed verdict falsified record 10 signed falsifications across 6 repositories key 251b14346af43d1d read the assurance report →new here? start with the argument: The Claim-vs-Effect Gap
latest research
What an AI Auditor Asks For, and What a Receipt Can Answer
The questions an AI auditor asks, the evidence each one needs, and the narrow part a signed receipt can answer.
The Self-Evaluation Bound
A theorem published in 2026 puts a floor under every evaluation method that does not use ground truth. Here is what it says, what it does not say, and why we built on the other side of it.
Three models agreed. That is not ground truth.
Most AI evaluation now runs through AI judges. That is a real method with a real limit. It tells you what the evaluator concluded, not what happened, and buyers keep paying for one while getting the other.
Our gate was right. Nothing was calling it.
We fixed the gate that decides whether a red test is red for a code reason. The fix was correct, it was tested, and it sat in the wrong directory. The code that signs verdicts never called it. Five more false catches reached the tree before a regrade caught them.
Our gate was wrong, and it inflated our own rate
We re-ran our published falsifications under fixed tooling. One reproduced exactly. Four others were our own bug, and they had been inflating a number we were about to publish.
What a verification memo looks like: SymPy #24909
The full memo a client receives, over a real signed receipt. Including the section that says what the verdict does not establish, and the parts of our own record that are weaker than they should be.
The Fabrication Was in the File, Not the Transcript
180 runs against deepseek-flash. The agent flagged the null and wrote a number anyway, on 40.0% of the multi-turn nulls.
Hermes Wrote a Skill From Experience. It Cut Tokens 32%.
We ran the most-used AI agent of 2026 through 13 tasks, its memory, and its full self-evolution loop. Everything held, including the token claim.
We Tried to Make Two Agents Fabricate. They Refused.
Silent tool failures, malformed data, a mid-pipeline null. We built the strongest fabrication triggers from the 2026 literature and ran Hermes and OpenClaw through them. 28 runs, zero fabrications, one clean refusal.
An AI Said It Fixed SymPy #24909. It Didn't.
A real bug, a non-empty patch, a confident 'fixed,' and a test that still fails. The full falsification, with the signed receipt you can check.
The Claim-vs-Effect Gap
Why an AI system's report of its own success is a claim, not evidence, and how to tell which one you're looking at.