Add gates for three self-inflicted failure modes found by audit - #81
Merged
Conversation
Mines this repository's own history for recurring, self-inflicted failure classes and closes each with the cheapest durable gate that fits, per the four-step process the user specified: find the failure, design the cheapest gate, prove it fires (deliberate break, then revert), ship with green CI. 1. Formatter hook stripping a not-yet-used import: `.claude/hooks/ post_edit_format.py` now runs `ruff check --fix --extend-ignore F401`, so it no longer deletes an import between the edit that adds it and the edit that uses it. `make lint`/CI still catch a genuinely unused import at commit time (`.pre-commit-config.yaml`'s own `ruff` hook is unrestricted) -- this only defers F401 from "every edit" to "commit time". TDD: tests/integration/test_claude_hooks.py:: test_hook_does_not_strip_an_import_with_no_usage_yet written first, confirmed red against the unfixed hook, green after the fix. 2. Heredoc/CRLF corruption: already gated by `.pre-commit-config.yaml`'s `mixed-line-ending --fix=no` hook (added 2026-08-18). No new code -- verified this session by deliberately writing a stray mid-line `\r` and confirming the hook fails on it, then discarding the fixture. 3. A sed/index-based edit duplicating a section of a file: new `tools/validators/check_duplicate_blocks.py` / `make check-duplicate-blocks`, wired into `make ci`. Fails if any tracked Markdown file repeats the same large (12-line, mostly-substantial) block of prose verbatim. Verified zero findings against the real repository before landing (same discipline as check_manifest.py's ka-name-matches-manifest), then proven to fire against a real deliberately-duplicated chunk of docs/planning/backlog.md, then reverted. 4. Generated artifacts drifting (docs index, inventory, dependency tree, config template, status report, benchmark history): already comprehensively gated. Verified this session by deliberately staling docs/index.md and confirming check-docs-index fails on it, then reverting. 5. Documentation claims contradicting repo state: the general case stays deliberately advisory (check_claims.py -- needs a reader). The specific, structural sub-case this audit could gate -- this manifest's own "CLAUDE.md files" count claim, a second, independent restatement of the same fact check-status already checks in roadmap.md -- gets a new check_manifest.py rule, claude-md-count-matches-live. It immediately found and this change fixes a real, pre-existing drift: the manifest claimed 45 while 47 files actually existed (roadmap.md's own copy was already correct). Not covered, said plainly rather than glossed over: truncation of a hand-written document. Unlike verbatim duplication, a shorter file is extremely common and legitimate (content genuinely retired), so telling a truncation from an edit needs a reader -- the same reasoning that keeps check_claims.py advisory. No structural gate is proposed for it. `gh run list` shows no CI failures in the last 30 runs; none of these three fixed failure modes ever reached CI as a red run -- each was local, pre-commit friction caught and silently worked around within a session, which is also why none of them left a git-log trace to cite directly (noted in each script's own docstring rather than pointing at a commit that doesn't exist). make ci: 1150 passed, 21 skipped, 99% coverage; all sixteen structural checks green, including the two new ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Audits this repository's own recurring, self-inflicted failure modes (git history,
gh run list, existing validator scripts) and closes each with the cheapest durable gate that fits, per the requested four-step process: find the failure, design the gate, prove it fires with a deliberate break-then-revert, ship with green CI.Failure mode → gate
.claude/hooks/post_edit_format.pynow runsruff check --fix --extend-ignore F401;make lint/CI still catch a genuinely-unused import at commit timetest_hook_does_not_strip_an_import_with_no_usage_yetwritten first, confirmed red against the unfixed hook, green after the fix.pre-commit-config.yaml'smixed-line-ending --fix=no(added 2026-08-18, for a prior$\rho$→$ho$incident)\r, confirmed the hook fails on it, revertedsed/index-based edit duplicates a section of a filetools/validators/check_duplicate_blocks.py/make check-duplicate-blocks, inmake cidocs/planning/backlog.md, confirmed the gate fires, revertedcheck-docs-index,check-inventory,check-dependency-tree,check-config-template,check-status,check-benchmark-report(all pre-existing)docs/index.md, confirmedcheck-docs-indexfails, revertedcheck-status,check-references,check-dates); the general case stays advisory (check_claims.py, needs a reader)check_manifest.pyruleclaude-md-count-matches-live: this manifest's own "CLAUDE.md files" count is a second, independently-drifting restatement of the same factcheck-statusalready checks inroadmap.mdNot covered, said plainly rather than glossed over: truncation of a hand-written document. A shorter file is extremely common and legitimate (content genuinely retired) — telling a truncation from a normal edit needs a reader, the same reasoning that keeps
check_claims.pyadvisory rather than gating. No structural gate is proposed for it.On "mine the CI logs":
gh run listshows no failed runs in the last 30. None of the three fixed failure modes ever reached CI as a red run — each was local, pre-commit friction, caught and silently worked around within a session. That is also why none of them has a commit to cite directly; each new validator's docstring says so rather than pointing at a commit that doesn't exist.Test plan
make cigreen: 1150 passed, 21 skipped, 99% coverage, all sixteen structural checks including the two new onesmixed-line-ending,check-docs-index,check-duplicate-blockscheck_duplicate_blocks.pyverified against real tracked Markdown with zero false positives before being wired intomake ciclaude-md-count-matches-liveverified against the real manifest, which surfaced and let this PR fix a genuine pre-existing count drift🤖 Generated with Claude Code