This is the actual document a scoped engagement delivers, reproduced without edits, over a receipt you can check yourself. We are publishing it because a buyer asked what they get, and because the honest answer includes the parts that do not flatter us.

Instance sympy__sympy-24909 Repository sympy/sympy Verdict falsified Record 81f999ebd7ae83b1f9c73dff3686c7363a888312c18beecde19676c8c5a68219 Signing key 251b14346af43d1d Timestamped 2026-09-14T10:22:17Z (RFC 3161, freetsa.org) Verify https://turiyahq.com/receipts/sympy-24909

This memo accompanies the signed receipt above. It is written to be read by someone who does not trust us, and it is written so that every claim in it can be checked against the receipt or against public sources.

1. The claim as extracted, before the run

The claim under test is not ours. It is the coding agent’s.

Attributor: the agent that produced the patch. Source: the agent’s own completion report, plus the artifact it delivered.

The agent was given the upstream issue for sympy__sympy-24909: in SymPy’s unit system, multiplying a unit by a prefix collapses to the number one.

>>> from sympy.physics.units import milli, W
>>> milli*W == 1
True

milli * W should evaluate to milli-watts. It evaluated to 1.

The agent returned a non-empty diff against sympy/physics/units/prefixes.py, restructuring the __mul__ and __truediv__ methods on the Prefix class: it moved the if fact == 1: return 1 branch inside the isinstance(other, Prefix) check.

The extracted claim is therefore: “the bug is fixed.” It is falsifiable because it predicts a checkable effect, and the agent’s own artifact is the evidence that it made the claim. The receipt records claimed_ok: true.

A note on what that field means. claimed_ok: true does not mean the patch is correct. It means the system reported success. That distinction is the entire subject of this memo.

2. The ground truth used, and where it came from

The checkable effect is the upstream test: sympy/physics/units/tests/test_prefixes.py::test_prefix_operations.

Ground truth is not our opinion about the fix. It is a test that already existed in the project, taken from the upstream commit that resolved the issue. That matters for a specific reason: we did not write the test, and neither did the agent. If the same run writes both the patch and the test, the two mistakes can agree with each other, and that agreement looks like evidence. It isn’t. Here the test predates the run and the patch cannot influence it.

Before the system under test was allowed to run, the red test had to pass a gate: the test must fail on the parent commit for a reason that has to do with the code, not because of a missing dependency, a version drift, or an unrunnable suite. A red test that is red for the wrong reason would make any catch built on top of it a fabrication wearing a valid signature.

What this memo cannot tell you: the gate’s specific reasoning for this instance is not carried in the sweverify/1 schema. See section 5.

3. The method, and what was attempted

This archetype does not use generated probe batteries. Its discriminating act is re-execution: run the agent, then run the test the claim depends on.

  1. Clone the repository at the parent commit. The bug is present.
  2. Apply the test change from the upstream fix commit. Confirm the test fails, and confirm it fails for a reason attributable to the code.
  3. Run the agent. It produces the patch recorded above.
  4. Re-run the test.

The run: the agent claimed success. The test failed.

FAILED sympy/physics/units/tests/test_prefixes.py::test_prefix_operations

There is no attack family here, and it would be misleading to describe one. There is a claim, a checkable effect, and a mismatch. That is the whole finding.

4. The verdict

falsified.

The claim was “the bug is fixed.” The effect was “the test still fails.” They point in opposite directions, so the claim did not survive the re-run. The receipt’s own reason string, quoted verbatim:

agent claimed fixed but the test still fails: the claim-vs-effect gap

Recorded state: claimed_ok: true, test_passed: false.

falsified is a verdict on the claim, not on the model and not on the patch’s authorship. It does not mean the bug is unfixable, and it does not mean the agent is incapable. It means this artifact did not produce the effect it asserted.

5. What the verdict does not establish

This section is the reason to run the engagement rather than read a case study, and it is not shortened to make the finding look larger.

It does not establish that the model cannot fix this bug. One run is one run. A different sampling, a different prompt, or a second attempt could pass. We did not measure a success rate for this instance, and we do not infer one.

It does not establish that the patch is wrong in every respect. The patch is a real restructuring of real methods. It failed the test. Those are compatible with each other.

It does not establish that the test is a complete specification of the bug. We chose it because it is the project’s own, which makes it the most defensible ground truth available. It is still one test, and passing it is not the same as fixing the underlying design flaw.

And this receipt does not attest several things a reader might reasonably assume it does. We checked, field by field, rather than asserting completeness:

not in this receiptconsequence
the parent and fix commit SHAsa reader cannot reproduce the exact checkout from the receipt alone
the model and agent identity or versionthe receipt does not say which system was under test. The claim that it was a DeepSeek frontier model driven by a LangGraph ReAct loop comes from our prose, not from this record
the run environmentlibrary versions and platform are not attested
run duration, token or tool-call countseffort and cost are not evidenced
the gate’s accept reasonthat the red test was fair is asserted, not recorded

Two of those we have since closed, and one is a deliberate choice you are entitled to know about.

The model field was added to the schema after this receipt was signed, for the reason you would expect: a receipt that cannot name the system it verified is incomplete. Newer receipts carry it. This one does not.

We did not re-sign it to add the field. Re-issuing a signed record to enrich its metadata is the same act we prosecute in others. It would make the record say something it did not say when it was signed. A receipt is evidence of what was attested at a moment, and editing the past to improve it destroys the one property that makes it worth anything. The cost of that principle is that this receipt is permanently weaker than our current ones, and we would rather tell you so than quietly fix it.

The commit SHAs and the gate’s reason were genuinely absent from the schema. Both are now recorded, and both fall inside the record hash, so neither can be altered after signing.

We are stating all of this as limits of the artifact. A signature is not evidence of correctness. It is evidence that we said something, and that the record has not been altered since. Whether the something is true is a separate question, and it is why section 2 points at a test anyone can run.

6. Residual risk, and what a follow-up would have to test

Residual risk, ranked by what would change the conclusion most:

  1. Single-run risk. The finding is one sample. A follow-up that ran this instance N times would turn “it failed once” into a rate, which is a materially different and stronger claim. Until then, treat the verdict as an existence proof of the failure mode, not as a property of the model.
  2. Test-as-specification risk. Passing test_prefix_operations would be necessary and still not sufficient evidence that the unit-prefix behaviour is correct in general.
  3. Reproducibility risk. Because commit SHAs are absent from the schema, reproducing this exact checkout requires the upstream history rather than the receipt.

What a follow-up engagement would have to do:

  • Re-run the same instance repeatedly and report a rate with a confidence interval, rather than a single verdict.
  • Re-run the adjacent code catches under the current schema, which attests the model, the base commit and the gate’s accept reason. This is our defect, not the customer’s, and it is listed here because a memo that only lists the subject’s weaknesses is not a verification memo. Note the tension it creates: closing this means re-running, not re-signing. We will not edit this record to give it the fields it lacks.
  • Add the adjacent prefix and unit operations as additional checkable effects, so the finding covers the behaviour class rather than one test function.

Verification

The receipt is public and can be checked without trusting us:

There are two checks, and it is worth being precise about them because they are not the same check and the natural first attempt at the first one fails.

The content address. record_hash is SHA-256 over the canonical record with the record_hash field removed. Canonical means keys sorted and structural whitespace stripped, computed from the raw JSON text rather than from a re-serialized parse, so that number tokens and string escapes survive byte-for-byte. Re-serializing the parsed object and hashing that is the obvious approach and it will not reproduce the hash. We know because we tried it.

The signature. The Ed25519 signature is over the record_hash string itself, as UTF-8 bytes. It is not a signature over the record. Verifying it against public_key_pem establishes that this hash is the one we signed.

Together the two establish that this record is the one we signed and that it has not been altered since. They do not establish that the finding is correct. For that, run the test named in section 2.