Skip to content

v1.7: Agent-file drift checker #1

@TMHSDigital

Description

@TMHSDigital

Context

v1.6.0 introduced 9 new standards (testing, skills, rules, mcp-server, security, licensing, scope, lifecycle, writing-style). Every tool repo in the ecosystem has its own AGENTS.md and CLAUDE.md referencing the standards set; those files will drift as standards evolve. This issue delivers a drift checker in three phases so the prerequisite grunt work ships alongside the feature that needs it.

Goal

A CI-runnable checker that flags when a tool repo's agent guidance files are out of sync with the current standards in Developer-Tools-Directory. Scope is narrowed to deterministic structural drift for v1.7; semantic drift is a stretch goal with an explicit escape hatch.

Drift taxonomy

| Class | Deterministic? | In scope for v1.7? |

|---|---|---|

| Broken reference to a removed/renamed standards/*.md | Yes | Phase 2 |

| Missing required reference for the repo's type (cursor-plugin vs mcp-server) | Yes | Phase 2 |

| Stale aggregate counts vs registry.json | Yes | Phase 2 |

| Principles wording drift vs current standards/README.md | No | Phase 3 (stretch) |

| Contradictions with current standards wording | No | Phase 3 (stretch) |

Phases

Phase 1 ΓÇö standards-version frontmatter rollout (prerequisite)

Add a standards-version field to every tool repo's AGENTS.md so the checker has a stable anchor.

Acceptance criteria:

  • Every tool repo's AGENTS.md in the 9 registered repos has a top-of-file standards-version: <meta-repo VERSION> marker. (Shipped as an HTML comment marker rather than YAML frontmatter ΓÇö see "Agent context file marker rollout" below for rationale. CLAUDE.md covered as a natural extension.)

  • Value matches the meta-repo's VERSION file at the time the PR lands. (1.6.3.)

  • Frontmatter format is documented in standards/agents-template.md.

  • Scaffold (scaffold/create-tool.py templates) emits the frontmatter for new repos.

Phase 1 fully complete as of 2026-04-24. All four original checkboxes satisfied. Skill/rule frontmatter rollout and agent context file marker rollout documented in their own sections above. Standards template and scaffold updates merged via PR #6.

The ecosystem now has:

  • standards-version signal on every SKILL.md, .mdc, AGENTS.md, CLAUDE.md across 8 active tool repos

  • standards-template documents the signal as a required element

  • scaffold generates new tool repos with the signal pre-populated

Phase 2 (drift checker) can now treat the signal as a hard invariant.

Known conceptual issue: standards-version: 1.7 labels a thing that does not formally exist until v1.9 introduces per-standard versioning. Documented trade-off: ecosystem-level VERSION is a pragmatic anchor; per-standard versions in v1.9 will supersede it. Do not treat the frontmatter as a compliance claim for individual standards yet.

Phase 2 ΓÇö Structural drift checker

Acceptance criteria:

  • python scripts/check_agent_files.py <repo-path> runs locally with exit codes: 0 clean, 1 drift, 2 tool error.

  • --local <path> flag for offline use; no network required for unit tests.

  • Checks implemented:

    • broken-refs: every standards/*.md link in AGENTS.md/CLAUDE.md resolves against the current standards/ directory.

    • required-refs: per-type required standards set is satisfied. Required-set map lives in standards/required-refs.json (data-driven, not hardcoded in Python). Must handle cursor-plugin vs mcp-server asymmetrically (mcp-server has no skills/rules).

    • stale-counts: numbers matching registry.json aggregates (repos/skills/rules/mcpTools) equal the live aggregate. Reuses aggregate_stats() from scripts/sync_from_registry.py.

  • Intentional-deviation pragma: <!-- drift-ignore: <check-name>=<target> --> HTML comment, consistent with existing <!-- registry:* --> markers.

  • Reusable GitHub workflow at .github/workflows/agent-drift.yml that tool repos can invoke on their own PRs.

  • Meta-repo CI runs the checker against the meta-repo's own AGENTS.md and CLAUDE.md (dogfood), plus at least one tool repo as a smoke test via sparse-checkout.

  • Fixture suite under tests/fixtures/ with intentionally-drifted agent files for regression.

  • Documented in standards/ explaining how tool repos opt in and how to mark intentional deviations.

  • Report-only mode; no auto-PRs to tool repos in v1.7 (deferred; blast-radius and cross-org write access are not worth it yet).

Scope update 2026-04-24 (agent marker rollout)

The drift checker now has two detection patterns:

  • YAML frontmatter field standards-version on SKILL.md and .mdc files.

  • HTML comment marker <!-- standards-version: X.Y.Z --> on AGENTS.md and CLAUDE.md files.

Both are equally valid signals. Files missing the pattern appropriate to their file type constitute drift.

Phase 3 ΓÇö Semantic drift detection (stretch)

Principles wording drift and standards contradictions. Likely requires standards versioning from v1.9 to be meaningful.

Acceptance criteria (if attempted):

  • Concrete detection approach chosen and documented (text-diff against canonical section, embedding similarity, or explicit versioned-block markers).

  • False-positive rate on the fixture suite is <10%.

Escape hatch: If Phase 3 is not implemented by the time Phases 1 and 2 are complete and ready to tag v1.7, file a dedicated v1.9 issue titled "Semantic agent-file drift detection" containing the current state and open questions, then close this issue. Do not leave this issue in a limbo state.

Skill and rule frontmatter rollout (completed 2026-04-24)

Prerequisite work for the drift checker (Phase 2). Adds standards-version to every SKILL.md and .mdc rule file across the 8 active tool repos. Separate from the original Phase 1 scope (which targets AGENTS.md/CLAUDE.md/standards/agents-template.md/scaffold templates) but serves the same ultimate goal of enabling drift detection. The original Phase 1 checkboxes above remain unchecked and are a follow-up session.

Scope: 248 files across 8 repos at standards-version: 1.6.3 (177 SKILL.md + 71 .mdc). steam-mcp correctly skipped (no skills/ or rules/ directories).

Merges:

  • CFX-Developer-Tools: PR #1 merged as 3481025, 15 files, auto-release v0.7.3

  • Unity-Developer-Tools: PR #1 merged as 4e70cfa, 26 files, auto-release v1.3.3

  • Docker-Developer-Tools: PR #21 merged as 915e247, 27 files, auto-release v1.2.1

  • Home-Lab-Developer-Tools: PR #11 merged as ab4ddcf, 33 files, auto-release v0.12.1

  • Mobile-App-Developer-Tools: PR #3 merged as 64931fb, 55 files, no auto-release (no release.yml)

  • Plaid-Developer-Tools: PR #1 merged as a8ebb48, 24 files, no auto-release (tag-triggered release.yml by design)

  • Monday-Cursor-Plugin: PR #8 merged as 90ed3a0, 29 files, auto-release v0.3.2

  • Steam-Cursor-Plugin: PR #1 merged as 3801dac, 39 files, auto-release v1.2.1

Script artifact: E:\.TMHS-Tool-Ecosystem-Workspace\phase1-script\add_frontmatter.py ΓÇö byte-exact, line-ending preserving, idempotent, non-stacking, BOM-aware. Reusable for future frontmatter work (including the original Phase 1 AGENTS.md/CLAUDE.md rollout).

Follow-up issues filed during rollout:

Agent context file marker rollout (completed 2026-04-24)

Companion to the skill/rule rollout above. Adds a standards-version marker to every tool repo's AGENTS.md and CLAUDE.md. Closes the first two original Phase 1 acceptance criteria above (checkboxes 1 and 2). CLAUDE.md was not in the original Phase 1 scope; covered here as a natural extension so both agent-facing context files share the same invariant.

Scope: 13 files across 9 repos (6 AGENTS.md + 7 CLAUDE.md). Every one of the 9 registered repos has at least one of the two files.

Format chosen: HTML comment <!-- standards-version: 1.6.3 --> as the first line, followed by a blank line, then the original content unchanged. Chosen over YAML frontmatter because:

  • Zero rendering risk across GitHub blob view, GitHub Pages generators (Jekyll/MkDocs), Cursor, and Claude Code.

  • Preserves the pure-markdown convention of prose guidance documents.

  • Drift checker's minor complexity cost of handling a second detection path is acceptable (see Phase 2 scope update below).

Merges:

| Repo | PR | Merge SHA | Files | Auto-release |

|---|---|---|---|---|

| CFX-Developer-Tools | #2 | c500ee2 | 1 (AGENTS.md) | none |

| Unity-Developer-Tools | #2 | 2ebe456 | 1 (AGENTS.md) | none |

| Docker-Developer-Tools | #22 | 1fc0e3c | 2 (AGENTS.md + CLAUDE.md) | none |

| Home-Lab-Developer-Tools | #12 | 360fec6 | 2 (AGENTS.md + CLAUDE.md) | none |

| Mobile-App-Developer-Tools | #4 | f6f2f29 | 1 (CLAUDE.md) | none (no release.yml) |

| Plaid-Developer-Tools | #3 | 858f01a | 1 (CLAUDE.md) | none (tag-triggered release.yml) |

| Monday-Cursor-Plugin | #9 | 4cf83ce | 2 (AGENTS.md + CLAUDE.md) | none |

| Steam-Cursor-Plugin | #2 | 7589e40 | 2 (AGENTS.md + CLAUDE.md) | none |

| steam-mcp | #1 | fbb5df3 | 1 (CLAUDE.md) | none |

Unexpected (good) finding: zero auto-releases. In the prior skill/rule rollout, 6 of 8 active repos cut a patch release. This round's changes were to project-level prose files only, which aren't matched by the release.yml path filters in those 6 repos. Conclusion: release.yml path filters are already scoped to code/skill/rule paths; project-level guidance edits are release-neutral by existing configuration. Drift-remediation PRs that only touch AGENTS.md/CLAUDE.md will not trigger auto-release ΓÇö useful for Phase 2 planning.

Script artifact: E:\.TMHS-Tool-Ecosystem-Workspace\phase1-script\add_comment_marker.py ΓÇö sibling to add_frontmatter.py. Byte-exact, line-ending preserving, idempotent, non-stacking, BOM-aware. Matches flexible whitespace inside the marker and normalizes to canonical form (<!-- standards-version: X.Y.Z --> with single spaces).

Invariant now enforced: every AGENTS.md, CLAUDE.md, SKILL.md, and .mdc file across the 9-repo ecosystem carries a standards-version signal. The drift checker can rely on this as a hard invariant; any file missing its appropriate signal is by definition drift.

Remaining original Phase 1 work: closed 2026-04-24 via PR #6. See the completion note at the end of the Phase 1 section above.

Scope updates from skill/rule frontmatter rollout (2026-04-24)

Informs Phase 2 design. The drift checker must handle:

  • Two coexisting SKILL.md frontmatter schemas: title:/description:/globs: (CFX, Unity) vs name:/description: (Docker, Home-Lab, Mobile, Plaid, Monday, Steam). Either accept both or flag schema drift explicitly; Phase 2 will likely want to canonicalize.

  • Varying .mdc frontmatter field orderings: title, description, globs, alwaysApply appear in different combinations and orders across repos. Field-order checks should be tolerant or explicitly canonical.

  • Auto-release side effect on 6 of 8 active repos: drift-checker remediation PRs will cut a patch release on merge unless they either skip release triggers (via paths filters in release.yml or a [skip release] convention) or commit-message conventions are added that the release workflow ignores.

  • Line-ending mixing: some repos have core.autocrlf=true; the checker should normalize line endings on read to avoid false positives.

  • Plaid's tag-triggered release.yml is a valid non-auto-release pattern (runs on push: tags: ['v*']), not a bug. The checker/remediation tooling should not assume every repo auto-releases on merge.

Answered open questions

  • Where does it run? Meta-repo CI does sparse-checkouts of each tool repo's AGENTS.md/CLAUDE.md. Tool repos can additionally invoke the reusable workflow on their own PRs for early feedback.

  • Auto-PRs vs report? Report-only in v1.7. Revisit in v1.7.1 or v1.8 once the checker is proven.

  • How are intentional deviations marked? <!-- drift-ignore: ... --> HTML comment pragma in the agent file.

Out of scope for v1.7

  • Rewriting standards

  • Propagating fixes back to the 9 tool repos beyond Phase 1 frontmatter

  • Observability / usage metrics (v1.8)

  • Per-standard versioning (v1.9)

  • Tool repos containing ecosystem-level aggregate numbers at all ΓÇö if any tool repo's AGENTS.md has "9 repos / N skills" text, the fix is to remove it, not to keep it in sync. Flag these during Phase 1 rollout.

Notes

  • 9 entries in registry.json but only 8 are cursor-plugin; steam-mcp is mcp-server with no skills/rules. The required-refs check must be type-aware from day one.

  • Phase 2 should import from scripts/sync_from_registry.py rather than re-implement aggregation, so the two stay consistent by construction.

Phase 2 complete (2026-04-25)

The agent-file drift checker is operational and gating CI across the ecosystem.

Implementation summary

  • Session A (PR #7, v1.7.0): core library (types, signals, semver, snapshot, config, pragma, version-signal check, markdown/json renderers, CLI). Local-clone mode only; sparse-checkout deferred to Session C.
  • Session B (PR #8, v1.7.1): three additional checks (broken-refs, required-refs, stale-counts) plus policy data (standards/drift-checker.config.json, standards/required-refs.json, standards/versioning.md MAJOR/MINOR semantics) and the gh_summary renderer.
  • Session C (PR #9, v1.7.2): CI integration. Composite action at .github/actions/drift-check, meta-repo orchestration workflow with all four triggers (path-gated push, weekly cron, workflow_dispatch, pull_request), sparse-checkout mode in snapshot.py for remote tool repos, sticky-issue upsert with reopen-on-drift behavior, DRIFT_CHECK_TOKEN setup docs.
  • Session C follow-up (PR #11, v1.7.3): split DRIFT_CHECK_TOKEN (read-only across tool repos) from GITHUB_TOKEN (write access for the sticky issue on the meta-repo).
  • Session D pre-prereqs:
  • Session D prereq batch (6 PRs in parallel): expanded paths-ignore in release.yml for CFX, Unity, Docker, Home-Lab, Monday, Steam to include skills/**, rules/**/*.mdc, AGENTS.md, CLAUDE.md. Fixed a latent bug where the existing *.md filter only matched root-level files.
  • Session D main batch (9 PRs: CFX canary + 8 parallel): added standalone .github/workflows/drift-check.yml to every tool repo and bumped standards-version signals from 1.6.3 to 1.7.0 across 261 signal-bearing files in 9 repos (248 YAML frontmatter + 13 HTML-comment markers).
    • Pattern pivot mid-canary: switched from sibling-job-in-validate.yml to standalone drift-check.yml after CFX validate.yml was found pre-broken since 2026-04-08 (CFX#5). Standalone pattern is uniform across the ecosystem (8 plugin repos + steam-mcp) and decoupled from per-repo CI health.
    • All 9 drift-check runs (PR + post-merge main) completed/success on first try with exit code: 0.
    • Zero spurious release runs across the 7 plugin repos with release.yml; paths-ignore prereq holding.

Ecosystem state at Phase 2 completion

  • 261 signal-bearing files across 9 tool repos carry standards-version: 1.7.0 (248 YAML + 13 HTML comment).
  • All 9 tool repos run drift-check.yml on every PR and push to main.
  • Meta-repo runs drift-check.yml on push to main (path-gated), weekly cron, and workflow_dispatch.
  • Sticky issue (Ecosystem drift report (live) #10) maintains current ecosystem drift state at the meta-repo and auto-closed at 2026-04-25T12:50:07Z when the post-D-1 ecosystem went clean.
  • Composite action at v1.7 floating tag provides a stable consumer interface for tool repos.
  • rc=1 propagation makes drift-check a real gate (failures block CI per the v1.7.4 fix).
  • stale-counts type-level suppression handles narrative aggregates in plugin agent files (interim, per Reconcile validate-counts vs drift checker stale-counts pattern across ecosystem #12).
  • Tag-triggered release semantics preserved (Mobile-App, Plaid).
  • Push-to-main release semantics preserved with paths-ignore protection (CFX, Unity, Docker, Home-Lab, Monday, Steam).

Issues filed during Phase 2

Phase 3 territory (not in scope here)

Semantic drift detection: principle-wording drift, contradiction detection against current standards. Tracked in a separate dedicated issue per the escape hatch above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions