Skip to content

Add gates for three self-inflicted failure modes found by audit - #81

Merged
AdamClemens merged 1 commit into
mainfrom
feat/failure-mode-gates
Sep 8, 2026
Merged

Add gates for three self-inflicted failure modes found by audit#81
AdamClemens merged 1 commit into
mainfrom
feat/failure-mode-gates

Conversation

@AdamClemens

Copy link
Copy Markdown
Owner

Summary

Audits this repository's own recurring, self-inflicted failure modes (git history, gh run list, existing validator scripts) and closes each with the cheapest durable gate that fits, per the requested four-step process: find the failure, design the gate, prove it fires with a deliberate break-then-revert, ship with green CI.

Failure mode → gate

Failure mode Status found Gate Evidence this session
Formatter hook strips an import added just ahead of its own usage No git-log trace (caught mid-session, never committed) .claude/hooks/post_edit_format.py now runs ruff check --fix --extend-ignore F401; make lint/CI still catch a genuinely-unused import at commit time TDD: test_hook_does_not_strip_an_import_with_no_usage_yet written first, confirmed red against the unfixed hook, green after the fix
Shell heredoc / CRLF corruption No git-log trace Already gated: .pre-commit-config.yaml's mixed-line-ending --fix=no (added 2026-08-18, for a prior $\rho$$ho$ incident) Deliberately wrote a stray mid-line \r, confirmed the hook fails on it, reverted
sed/index-based edit duplicates a section of a file No git-log trace (~3,900-line duplication reported in the prior Insights session-usage report; caught and reverted before commit) New tools/validators/check_duplicate_blocks.py / make check-duplicate-blocks, in make ci Verified zero findings against real repo before landing; then duplicated a real 15-line chunk of docs/planning/backlog.md, confirmed the gate fires, reverted
Generated artifacts (docs index, inventory, dependency tree, config template, status report, benchmark history) drift stale Already comprehensively gated check-docs-index, check-inventory, check-dependency-tree, check-config-template, check-status, check-benchmark-report (all pre-existing) Deliberately staled docs/index.md, confirmed check-docs-index fails, reverted
Documentation claims contradict repo state Partially gated already (check-status, check-references, check-dates); the general case stays advisory (check_claims.py, needs a reader) New, narrow check_manifest.py rule claude-md-count-matches-live: this manifest's own "CLAUDE.md files" count is a second, independently-drifting restatement of the same fact check-status already checks in roadmap.md Found and fixed a real, live drift the moment the rule was written: manifest claimed 45, 47 actually existed

Not covered, said plainly rather than glossed over: truncation of a hand-written document. A shorter file is extremely common and legitimate (content genuinely retired) — telling a truncation from a normal edit needs a reader, the same reasoning that keeps check_claims.py advisory rather than gating. No structural gate is proposed for it.

On "mine the CI logs": gh run list shows no failed runs in the last 30. None of the three fixed failure modes ever reached CI as a red run — each was local, pre-commit friction, caught and silently worked around within a session. That is also why none of them has a commit to cite directly; each new validator's docstring says so rather than pointing at a commit that doesn't exist.

Test plan

  • make ci green: 1150 passed, 21 skipped, 99% coverage, all sixteen structural checks including the two new ones
  • TDD red→green for the hook fix
  • Deliberate break + revert, proven and cleaned up, for: mixed-line-ending, check-docs-index, check-duplicate-blocks
  • check_duplicate_blocks.py verified against real tracked Markdown with zero false positives before being wired into make ci
  • claude-md-count-matches-live verified against the real manifest, which surfaced and let this PR fix a genuine pre-existing count drift

🤖 Generated with Claude Code

Mines this repository's own history for recurring, self-inflicted
failure classes and closes each with the cheapest durable gate that
fits, per the four-step process the user specified: find the failure,
design the cheapest gate, prove it fires (deliberate break, then
revert), ship with green CI.

1. Formatter hook stripping a not-yet-used import: `.claude/hooks/
   post_edit_format.py` now runs `ruff check --fix --extend-ignore F401`,
   so it no longer deletes an import between the edit that adds it and
   the edit that uses it. `make lint`/CI still catch a genuinely unused
   import at commit time (`.pre-commit-config.yaml`'s own `ruff` hook is
   unrestricted) -- this only defers F401 from "every edit" to "commit
   time". TDD: tests/integration/test_claude_hooks.py::
   test_hook_does_not_strip_an_import_with_no_usage_yet written first,
   confirmed red against the unfixed hook, green after the fix.

2. Heredoc/CRLF corruption: already gated by `.pre-commit-config.yaml`'s
   `mixed-line-ending --fix=no` hook (added 2026-08-18). No new code --
   verified this session by deliberately writing a stray mid-line `\r`
   and confirming the hook fails on it, then discarding the fixture.

3. A sed/index-based edit duplicating a section of a file: new
   `tools/validators/check_duplicate_blocks.py` / `make
   check-duplicate-blocks`, wired into `make ci`. Fails if any tracked
   Markdown file repeats the same large (12-line, mostly-substantial)
   block of prose verbatim. Verified zero findings against the real
   repository before landing (same discipline as check_manifest.py's
   ka-name-matches-manifest), then proven to fire against a real
   deliberately-duplicated chunk of docs/planning/backlog.md, then
   reverted.

4. Generated artifacts drifting (docs index, inventory, dependency tree,
   config template, status report, benchmark history): already
   comprehensively gated. Verified this session by deliberately staling
   docs/index.md and confirming check-docs-index fails on it, then
   reverting.

5. Documentation claims contradicting repo state: the general case stays
   deliberately advisory (check_claims.py -- needs a reader). The
   specific, structural sub-case this audit could gate -- this
   manifest's own "CLAUDE.md files" count claim, a second, independent
   restatement of the same fact check-status already checks in
   roadmap.md -- gets a new check_manifest.py rule,
   claude-md-count-matches-live. It immediately found and this change
   fixes a real, pre-existing drift: the manifest claimed 45 while 47
   files actually existed (roadmap.md's own copy was already correct).

Not covered, said plainly rather than glossed over: truncation of a
hand-written document. Unlike verbatim duplication, a shorter file is
extremely common and legitimate (content genuinely retired), so telling
a truncation from an edit needs a reader -- the same reasoning that
keeps check_claims.py advisory. No structural gate is proposed for it.

`gh run list` shows no CI failures in the last 30 runs; none of these
three fixed failure modes ever reached CI as a red run -- each was
local, pre-commit friction caught and silently worked around within a
session, which is also why none of them left a git-log trace to cite
directly (noted in each script's own docstring rather than pointing at
a commit that doesn't exist).

make ci: 1150 passed, 21 skipped, 99% coverage; all sixteen structural
checks green, including the two new ones.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@AdamClemens
AdamClemens merged commit 142903c into main Sep 8, 2026
2 checks passed
@AdamClemens
AdamClemens deleted the feat/failure-mode-gates branch September 8, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant