Skip to content

docs: align the published error codes and record samples with the modules - #74

Open
lywinged wants to merge 1 commit into
agentrust-io:mainfrom
lywinged:docs/error-code-registry
Open

docs: align the published error codes and record samples with the modules#74
lywinged wants to merge 1 commit into
agentrust-io:mainfrom
lywinged:docs/error-code-registry

Conversation

@lywinged

Copy link
Copy Markdown
Collaborator

Ten statements the published documentation makes that the code does not do, across nine pages. All of them are live on main, and nothing checks any of it.

Codes

TR-SIG-005 is carried by every signature finding and documented nowhere. It is the signature check outcome: the Ed25519 verification result, a signature that cannot be verified, or no signature at all, which is FAIL at Level 1 and above and UNVERIFIED at Level 0. UNVERIFIED is deliberately held apart from SKIP so an unsigned record cannot read as a benign omission, and that was undiscoverable from the page.

TR-SIG-004 was documented as private key material in cnf.jwk. It reports key type: kty missing, or not in {OKP, EC}. The private-key condition is not reported under this code, or under any code, because the module raises before producing a finding for it. That is a module bug and is fixed separately, not here.

TR-ANC-002 is documented in three files and named by no module, and two of them disagree about what it is. docs/error-codes.md and docs/modules/tr-anc.md call it the https:// scheme check, which tr_anc.py performs under TR-ANC-001; docs/levels.md calls it a missing anchor.leaf_hash. Folded into TR-ANC-001. Adding a second ANC code instead would be a behaviour change and is a separate decision.

TR-ANC-001 was documented as requiring a resolvable URI and as rejecting a placeholder value. Nothing resolves it, and no placeholder check exists anywhere under src/. It checks presence, string type, https scheme, and a host.

TR-RTE-001 documented sev-snp, tdx and opaque as valid platform values. None of the three is in _VALID_PLATFORMS, so a reader following the page produced a record TR-RTE-001 rejects, while eight of the ten registered platforms went unmentioned. The wrong list appeared in three pages, and docs/levels.md carried a sample record using one of them.

TR-RTE-003 was documented as resolving the RIM URI and checking the manifest behind it. It checks that the string starts with https://, and nothing else.

TR-ENV-004 was documented as a required-fields gate over the schema's full required set, with a positive case reading "all of: eat_profile, iat, subject, ...". It checks cnf.jwk.kty, and nothing else.

TR-POL-002 lost track of declared, in four pages. Four values, not three.

docs/modules.md summarises five of the seven modules as doing work they do not do: private key leak detection (the path that raises), RIM URI resolution, SCITT inclusion proof structure, builder URI, and required fields. Each row now says what its module checks.

Samples

The minimum conformant Level 2 record is not conformant. docs/levels.md shows an anchor object; schemas/trace-claim.json is additionalProperties: false and defines no anchor property. Measured: valid_level0.json validates, and the same record with the documented anchor block fails with Additional properties are not allowed ('anchor' was unexpected). anchor.leaf_hash appears nowhere else in the repository.

The guards

tests/test_docs_match_the_modules.py adds two.

The code set named by the modules must equal the code set with a row in docs/error-codes.md. A row, not a mention: a code named in passing is not documented, and accepting a mention would let the check be satisfied by prose that tells a reader nothing. Every TR- code in this package lives under modules/, so that scope is complete.

Every JSON sample under docs/ must agree with the packaged schema. Three qualify today, all in docs/levels.md, but every .md is scanned rather than a list kept by hand, because a hand-maintained list of what gets checked is the same defect this exists to catch. The check also fails if it validated nothing, so it cannot degrade into a pass over no work when the samples or the fence change.

Verification

Both guards fail against main's documentation and pass against this branch. Individually: a code moved out of its row and into prose fails the first; a changed code fence fails the second; and each sample drift reintroduced on its own fails on its own.

What these guards do not do

Stated in their docstrings, because a guard that is trusted for more than it checks is the defect this PR is about.

The code check matches on codes named in module source, not on Finding.code. TR-SIG-003 appears only inside a message string that a TR-SIG-005 finding carries, so matching on Finding.code would demand deleting a row that documents a real condition.

Neither guard can tell whether a row describes what its code reports, which was six of the ten defects above. Those were found by reading each row against its module, and the same reading is what a future row needs.

Implementation notes worth a reviewer's eye

The sample guard drops required before validating, because the samples are fragments rather than whole records, but it leaves if and not intact. Stripping required from an if makes it vacuously true and fires the matching then against records the condition was never meant to reach; the schema's origin rule does exactly that, and an earlier draft of this guard reported two valid samples as broken.

String values containing an ellipsis are dropped before validation. A sample signature written as eyJhbGciOiJFZERTQSJ9... is a reader's placeholder, not a claim about the format.

Not touched

Two sample CI transcripts under docs/tutorials/ quote output the code no longer produces. Regenerating a transcript is a different kind of change, so they are left as they are rather than edited by hand.


Docs and tooling only. No module, schema or behaviour change. 203 passed, 5 xpassed (the TEE-hardware smoke set, strict=False by design, unrelated). ruff clean, no broken relative links, nav unchanged. CI runs neither ruff nor mypy today.


Generated by Claude Code

…ules

Ten statements the documentation makes that the code does not do, across nine
pages. All live on main, none checked by anything.

Codes:

- TR-SIG-005 is carried by every signature finding tr_sig.py produces and is
  documented nowhere. It is the signature check outcome: the Ed25519 result, a
  signature that cannot be verified, or no signature at all, which is FAIL at
  Level 1 and above and UNVERIFIED at Level 0. UNVERIFIED is deliberately not
  SKIP, so an unsigned record cannot read as a benign omission.
- TR-SIG-004 was documented as private key material in cnf.jwk. It reports key
  type: kty missing, or not in {OKP, EC}. The private-key condition is not
  reported under this code, or any code, because the module raises before
  producing a finding for it. That is a module bug, fixed separately.
- TR-ANC-002 is documented in three files and named by no module, with two
  descriptions that disagree: docs/error-codes.md and docs/modules/tr-anc.md
  call it the https-scheme check, which tr_anc.py performs under TR-ANC-001,
  while docs/levels.md calls it a missing anchor.leaf_hash. Folded into
  TR-ANC-001. Adding a second ANC code instead is a behaviour change.
- TR-ANC-001 was documented as requiring a resolvable URI and as rejecting a
  placeholder value. Nothing resolves it, and no placeholder check exists
  anywhere under src/. It checks presence, string type, https scheme and a
  host. An earlier revision of this branch carried the placeholder claim
  forward; it is removed.
- TR-RTE-001 documented `sev-snp`, `tdx` and `opaque` as valid platforms. None
  is in _VALID_PLATFORMS, so a reader following the page produced a record
  TR-RTE-001 rejects, while eight of the ten registered platforms went
  unmentioned. The wrong list appeared in three pages, and docs/levels.md
  carried a sample record using one of them.
- TR-RTE-003 was documented as resolving the RIM URI and checking the manifest
  behind it. It checks that the string starts with https://, and nothing else.
- TR-ENV-004 was documented as a required-fields gate over the schema's full
  required set, positive case "all of: eat_profile, iat, subject, ...". It
  checks cnf.jwk.kty, and nothing else.
- TR-POL-002 lost track of `declared`, in four pages. Four values, not three.
- docs/modules.md summarises five of the seven modules as doing work they do
  not do: private key leak detection (the path that raises), RIM URI
  resolution, SCITT inclusion proof structure, builder URI, and required
  fields. Each row now says what its module checks.

Samples:

- docs/levels.md shows an anchor object in its minimum conformant Level 2
  record. schemas/trace-claim.json is additionalProperties: false and defines
  no anchor property. Measured: valid_level0.json validates, and the same
  record with the documented anchor block fails with "Additional properties are
  not allowed ('anchor' was unexpected)". anchor.leaf_hash appears nowhere else
  in the repository.

tests/test_docs_match_the_modules.py adds two guards.

The code set named by the modules must equal the code set with a row in
docs/error-codes.md. A row, not a mention: a code named in passing is not
documented, and accepting a mention would let the check be satisfied by prose
that tells a reader nothing. Every TR- code in this package lives under
modules/, so that scope is complete.

And every JSON sample under docs/ must agree with the packaged schema. Three
qualify today, all in docs/levels.md, but every .md is scanned rather than a
list kept by hand, because a hand-maintained list of what gets checked is the
same defect this exists to catch. The check also fails if it validated nothing,
so it cannot degrade to a pass over no work when the samples or the fence
change.

Both guards fail on main's documentation and pass on this one. Verified
individually: a code moved from a row into prose fails the first, a changed
fence fails the second, and each sample drift reintroduced alone fails alone.

What the guards do not do, stated in their docstrings. The code check matches
on codes named in module source, not on Finding.code: TR-SIG-003 appears only
inside a message string a TR-SIG-005 finding carries, so matching on
Finding.code would demand deleting a row that documents a real condition. And
neither guard can tell whether a row describes what its code reports, which was
six of the ten defects above; those were checked by reading the rows against
the module.

The sample guard drops `required` before validating, since the samples are
fragments, but leaves `if` and `not` intact. Stripping `required` from an `if`
makes it vacuously true and fires the matching `then` against records the
condition never meant to reach; the schema's `origin` rule does exactly that,
and an earlier draft of this guard reported two valid samples as broken. String
values containing an ellipsis are dropped before validation: a sample signature
written as eyJhbGciOiJFZERTQSJ9... is a reader's placeholder, not a claim about
the format.

Not touched: two sample CI transcripts under docs/tutorials/ quote output the
code no longer produces. Regenerating a transcript is a different kind of
change and they are left rather than edited by hand.

203 passed, 5 xpassed. ruff clean. No broken relative links, nav unchanged.
CI runs neither ruff nor mypy today.

Signed-off-by: LouieLuNZ <48041247+lywinged@users.noreply.github.com>
@lywinged
lywinged requested a review from a team as a code owner August 21, 2026 20:28
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant