[problem] wr-itil evaluate-relevance matches ADR numbers across repos and treats Composes-with siblings as drivers

[tompahoward]

Description

Two of evaluate-relevance.sh's five evidence shapes test string presence in place of a semantic relationship, and both produce confident CLOSE verdicts on tickets that must stay open. Verified present at upstream HEAD (packages/itil/scripts/evaluate-relevance.sh, 530 lines).

Shape 2 (ADR-shipped-confirmed), line 300. The ADR match is a bare grep -oE '\bADR-[0-9]{3}\b', and the number is resolved against the consuming repo's docs/decisions/<NNN>-*.md with no repo or namespace qualification. Downstream tickets routinely cite agent-plugins ADR numbers, and both repos number their ADRs from 001, so a citation of an upstream decision is reported as evidence that an unrelated local decision shipped.

Shape 5 (driver-child-ticket-closed), lines 418 to 445. The ## Related section is parsed for P<NNN> refs and any closed one is counted as a closed "driver". Nothing checks whether the ticket declared that ref as a driver. ## Related is a mixed section by design, carrying drivers, siblings, supersessions and capture provenance, so presence there carries no directional information. A ticket that explicitly says "composes with P040, distinct concerns" has P040 counted as its closed driver.

The A1 guard at lines 448 to 471 (suppress shape 5 when the child names an unbuilt SKILL or agent) is a narrow patch on the same gap: it catches one way a non-driver relationship shows up, not the general case.

Symptoms

Two independent witnesses in one downstream repo on 2026-08-08, through the two different shapes, both reproduced by running the script directly.

Witness A, /wr-itil:review-problems Step 4.6. 24 candidates surfaced, 0 survived verification, nothing closed. Two examples: a ticket citing the upstream Decision-Delegation Contract ADR-044 was credited with the local 044-cross-edition-shape-as-a-fresh-context-subagent-gate.proposed.md, a newsletter gate with no relationship to it; a ticket citing upstream ADR-013/014/032 was credited with the local 013-no-automated-linkedin-scraping.proposed.md and 014-wardley-mapping-as-strategic-lens.proposed.md.

Witness B, /wr-itil:work-problems Step 3.6. A live, reproduced bug returned CLOSE-CANDIDATE-WITH-CAVEAT citing three tickets as closed drivers. The ticket lists all three under "Composes with ... same publish/draft-lifecycle area, distinct concerns", and its own ## Related line says the dedup pass surfaced them as "newsletter-adjacent but distinct". The orchestrator overrode the verdict; the ticket was then fixed as a genuine defect in the same session.

Both surfaces consume the same script, so the same false positive can arrive during an interactive review and during an AFK loop.

Workaround

Read the cited evidence back against the ticket body before acting on any CLOSE verdict. Check that each cited ADR is the one the ticket meant, local or upstream, and that each cited P<NNN> was declared as a driver rather than a sibling.

Affected plugin / component

@windyroad/itil, packages/itil/scripts/evaluate-relevance.sh, shapes 2 and 5. Consumed by /wr-itil:review-problems Step 4.6 and /wr-itil:work-problems Step 3.6.

Frequency

Both consuming surfaces, every pass. One review pass produced 24 false positives out of 24 candidates.

Versions

• Local plugin: @windyroad/itil@0.59.2
• Upstream package: @windyroad/itil@0.61.1 (defect confirmed present in the HEAD copy of the script fetched from this repo)
• Claude Code CLI: 2.1.224 (Claude Code)
• Node: v24.16.0
• OS: Darwin 25.3.0 arm64

Evidence

Both verdicts reproduce by invoking the script directly against the affected ticket files. The two failures are independent, so a fix for one does not cover the other.

Suggested fix shape:

1. Qualify shape 2's ADR match. Require a repo or namespace marker adjacent to the reference, or restrict matching to ADRs the ticket cites in a local-path form (docs/decisions/<NNN>-...), or drop bare-number matching. A bare ADR-NNN is not a unique identifier once a consuming repo keeps its own decision log.
2. Gate shape 5 on the relationship the ticket declares rather than on presence in ## Related. Parse the Blocked by line or explicit driver phrasing, and treat Composes with as explicitly non-qualifying.
3. Cover both witnesses in packages/itil/scripts/test/evaluate-relevance.bats: an upstream-ADR-number collision, and a closed Composes with sibling.

Related issues in the same script, cross-referenced for context rather than as duplicates:

• wr-itil evaluate-relevance over-fires CLOSE-CANDIDATE on tickets that merely cite a shipped ADR #306 also reports shape 2 over-firing, but on a different root cause: an ADR that is genuinely local and confirmed being read as evidence that the citing ticket's own work shipped. The defect here is that the ADR resolved is not the one the ticket referred to at all. The two are worth fixing together, and neither fix implies the other. Shape 5 is not covered by wr-itil evaluate-relevance over-fires CLOSE-CANDIDATE on tickets that merely cite a shipped ADR #306.
• evaluate-relevance file-no-longer-exists false-positives on upstream-plugin paths #220, file-no-longer-exists false positives on upstream-plugin paths. Same script, different shape, and the same underlying theme of a path or identifier meaning something different in the consuming repo.

Cross-reference

Reported from a downstream project's docs/problems/open/132-relevance-evaluator-emits-close-verdicts-from-syntax-only-adr-number-and-driver-matching.md.

This issue is tracked locally as P132 in the downstream project's docs/problems/ directory.

[tompahoward]

Third witness, reproduced 2026-08-09 against the 0.59.2 cached build. This one is worth a comment rather than a silent tally because both shapes fire on a single ticket and shape 2 fails in a way namespace qualification would not catch.

Command and verdict:

$ wr-itil-evaluate-relevance docs/problems/known-error/115-site-changes-without-changeset-silently-never-release-to-production.md
verdict: CLOSE-CANDIDATE-WITH-CAVEAT (exit 0)
shapes:  ADR-shipped-confirmed,driver-child-ticket-closed
caveat:  multi-phase-mixed-progress: 0 task(s) done, 2 outstanding, confirm umbrella scope before close
cites:   ADRs human-oversight-confirmed: ADR-041 (docs/decisions/041-retire-consulting-funnel-repurpose-as-the-shift-hub.proposed.md)
         drivers closed: P012 (docs/problems/closed/012-no-ship-gate-on-push-publish-deploy.md)

The ticket is Site changes land on master without a changeset and silently never release to production (P115). It is live and being worked right now.

Shape 5 is the same failure already reported: P115 lists P012 under **Composes with**, and its ## Related section says in words "P012 gates on red CI; this ticket is the distinct missing-changeset-authoring nudge". The evaluator reads the number and not the sentence disowning it.

Shape 2 is the new part. The originally reported mode was a cross-repo number collision, an upstream ADR number resolving against an unrelated local file. Nothing like that happens here. ADR-041 is genuinely local, correctly resolved, and genuinely human-oversight: confirmed. The verdict is still wrong, because P115 never cited ADR-041 as its fix. It cited it as its exhibit: ADR-041 retires a consulting funnel, and that retirement is the change that sat unreleased on master for a week, which is the concrete failure P115 exists to describe. The shape cannot tell "the ADR that fixed this" from "the ADR whose release this ticket is about".

The dates make the inversion exact. ADR-041 carries oversight-date: 2026-07-14; P115 carries **Reported**: 2026-07-14. Both come from the one session where the operator noticed the old funnel still live in production and asked whether it had ever been released. The ADR was confirmed and the ticket was filed because of the same failure, so the evidence shape 2 offers as proof of closure is the defect's own footprint.

What this changes for the fix: qualifying the ADR match by repo or namespace is necessary but not sufficient. Shape 2 also has to establish that a confirmed ADR is the citing ticket's remedy rather than merely a number appearing somewhere in its body. A ticket names ADRs for several reasons, including the decision that caused the defect and the decision it was found while working on, and only one of those reasons is evidence of closure. A workable signal might be requiring the match to come from a ## Fix Strategy or ## Fix Released section rather than from ## Related.

This also bears on the dedup question raised when this issue was filed. #306 reports a genuinely local, genuinely confirmed ADR being read as evidence the citing ticket's own work shipped, which is exactly this witness. If you were weighing whether to fold this issue into #306, this is a point in favour of doing so: the two are converging on the same defect from different starting cases.

Local tracking: P132.