fix(tests): teach the CLI-verb linter to read cli_called verbs

[alexandrujircan]

Closes the follow-up flagged in #2994.

scripts/check-cli-verbs.py only ever parsed command_pattern, so a verb named by a cli_called criterion was checked by nothing. Every task migrated to cli_called (#2565, #2994 — 39 criteria naming 40 verb spellings today) quietly left the catalog linter's reach, and both migrations were verified by hand instead. A stale ixp verb in a migrated task would have been caught by no gate in this repo.

What it does now

cli_called names its verb in verb / verb_any_of as literal whitespace-separated chains, so there is no regex to walk and no dynamic case.

Only criteria carrying tool: uip are checked. record_cli shims several executables into one shared log and tool is what separates them, so a verb recorded under any other tool is not a uip verb. An absent tool is also skipped — it matches any executable, so nothing can be claimed about it. Both skips are silent, matching how a non-uip command_pattern is already dropped. No criterion in the repo is affected today: all 39 carry tool: uip.

The catalog lookup had to change

This is the part worth reviewing. classify walks progressively shorter prefixes, which a command_pattern verb genuinely needs — the regex may have captured a trailing flag fragment. A cli_called verb is a complete chain, and the catalog holds group prefixes as their own entries (ixp, ixp projects, ixp projects list).

So under the prefix fallback, the typo ixp projects lisst matched the group ixp projects and scored reachable. The first cut of this change shipped exactly that bug — the new check ran, found nothing, and looked like it worked. Caught by asserting on a deliberately misspelled leaf.

classify now takes exact=, used only by the cli_called path. It accepts a deliberately short group verb (legal — verb is an ordered prefix of the argv, so a max_count: 0 guard on ixp projects is meant to fire on every subcommand) while still failing a typo'd leaf. Renames stay prefix-matched in both modes, so flow validate is still reported retired via its flow ancestor and keeps its canonical suggestion.

Shape of the output

Both paths emit through one record() helper, so severities and message shapes cannot drift apart. Findings carry a new source field (command_pattern | cli_called). command_pattern stays the display key for both — write_report and the stdout printer read it, and the High histogram parses unmatched: [...] out of the message — but the printed label reads cli_called: where that is what it is.

Verification

• 8 new regression tests; 22 pass in tests/scripts/test_verb_checkers.py. One of them pins the prefix-vs-exact distinction in both directions, so the bug above cannot come back.
• Differential over all 1284 task YAMLs, old checker vs new: 102 findings each, identical on every field, no key added or dropped for existing findings. Nothing already in the repo changes verdict — the repo has no High/Medium today, so the message rewording is unobservable.
• Four classification paths exercised end to end: typo'd leaf → High · group prefix → clean · retired verb → Medium with suggestion · non-uip tool → no verdict.
• --report renders a cli_called High into the verb histogram correctly (the unmatched: [...] parse still lands).

Merge order

Independent of #2994 — branched off main, changes no task YAMLs, and #2994's verbs are all valid so neither PR turns the other red. Merge in either order. Landing this one first is what I'd suggest, so the remaining ~25 migrations are linted as they arrive rather than by hand.

Not fixed here

Two pre-existing, unrelated Windows-locale issues, left alone to keep this one logical change:

• The Medium message's → crashes a Windows cp1252 console. Present at HEAD (check-cli-verbs.py:316), and this PR adds a new way to reach a Medium finding without introducing the character.
• tests/scripts/test_runtime_payload_key_casing.py::test_no_raw_lowercase_runtime_key_reads fails with a UnicodeDecodeError. Confirmed identical at HEAD with these changes stashed.

Both are Windows-only; CI is UTF-8 Linux and unaffected.

🤖 Generated with Claude Code

[github-actions]

Claude finished @alexandrujircan's task in 2m 45s —— View job

---

PR Review: fix(tests): teach the CLI-verb linter to read cli_called verbs

• Read review criteria and project conventions
• Read full PR diff
• Read changed files in full context
• Analyze correctness, edge cases, and consistency
• Post review findings

---

Summary

Extends scripts/check-cli-verbs.py to also lint cli_called criteria (via verb / verb_any_of with tool: uip), closing a blind spot where migrated tasks silently escaped catalog verification. Adds 8 targeted regression tests and updates the two slash-command docs that describe the linter's scope.

Change-by-Change Review

1. scripts/check-cli-verbs.py — core logic

Severity: OK (with one Low nit)

The implementation is clean and well-reasoned:

• iter_cli_called_verbs (lines 311–344): Correctly filters on type == "cli_called" and tool == "uip", handles verb vs verb_any_of exclusively, normalises whitespace to match coder_eval's verb.split(), and deduplicates. The skip for absent tool is the right call — an unconstrained tool match can't be claimed as a uip verb.

• classify with exact=True (lines 234–290): The catalog_match inner function correctly switches between exact lookup and prefix fallback. Keeping renames prefix-matched in both modes is the right design — flow validate correctly surfaces its flow ancestor's retirement. The interaction between cat_hit and ren_hit under exact is sound: when both hit the same string, the len() comparison falls through to cat_hit, and the catalog is source of truth.

• record() helper (lines 365–388): Good refactor — unifying the finding construction eliminates the prior duplication between Medium and High branches and prevents future drift between the two criterion types.

Low nit — Info findings lack source field: The Info finding constructed directly at lines 392–401 doesn't go through record() and doesn't include source (or description). All Medium/High findings do. Under --json, consumers expecting a uniform schema will see the inconsistency. Only command_pattern can produce Info findings today so the behavior is correct, but the schema gap could surprise a downstream parser.

2. tests/scripts/test_verb_checkers.py — 8 new tests

Severity: OK

Thorough coverage of the new paths:

Test	What it pins
test_cli_called_verb_is_linted_at_all	Basic: typo'd verb → High, source == cli_called
test_cli_called_typoed_leaf_not_rescued_by_group_prefix	The key regression: exact vs prefix on classify
test_cli_called_deliberately_short_verb_stays_reachable	Group-level verb is legal under exact matching
test_cli_called_verb_any_of_reachable_if_any_spelling_is	Mixed reachable/unreachable in verb_any_of
test_cli_called_retired_verb_is_medium_with_suggestion	Rename prefix-match still works under exact
test_cli_called_non_uip_tool_gets_no_verdict	Both tool: curl and absent tool skip cleanly
test_cli_called_verbless_criterion_is_skipped	Flags-only criterion produces no finding
test_cli_called_verb_whitespace_is_normalised	Irregular spacing normalises to catalog key

The test_cli_called_typoed_leaf_not_rescued_by_group_prefix test is especially valuable — it asserts both exact=True (unknown) and exact=False (reachable) on the same input, directly pinning the bug the PR description warns about.

3. .claude/commands/audit-verbs.md

Severity: OK

Line 12: The output description now mentions cli_called verb / verb_any_of alongside command_executed patterns. Accurate and complete.

4. .claude/commands/lint-task.md

Severity: Low

Line 163: The primary paragraph is well-updated — mentions both criterion types, the source field, and the exact-matching behavior. However, the severity passthrough descriptions at lines 167–169 still use command_pattern-centric language ("pattern's verb path", "pattern matches only retired verbs"). These descriptions are now shared across both criterion types and could be mildly misleading for cli_called findings. Not incorrect (the PR description explains command_pattern is the display key), but "criterion" would be more accurate than "pattern" in those bullets.

What's Missing

• Info finding source field: A one-line addition ("source": "command_pattern") at line 393 would make the --json schema uniform across all severities. Trivial but worth doing. Fix this →
• Nothing else identified — the PR is well-scoped.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill files changed
E2E Tests	N/A	Script tests, not skill tests
Skill Body	N/A	No skill files changed
References & Assets	OK	Slash-command docs updated accurately
Repo Hygiene	OK	Scoped to linter + tests, no secrets, no drive-by changes

Issues for Manual Review

• Differential claim: The PR states "102 findings each, identical on every field" across old vs new checker over all 1284 task YAMLs. I was unable to run the tests locally (missing permissions), so this claim could not be independently verified. CI should confirm.
• The two pre-existing Windows-locale issues (→ on cp1252, UnicodeDecodeError in test_runtime_payload_key_casing.py) are noted in the PR as not-fixed-here. They affect Windows only and are outside this PR's scope.

Conclusion

Solid, well-motivated change. The exact parameter on classify is the correct design — it closes the prefix-fallback blind spot for cli_called while preserving the fallback where command_pattern genuinely needs it. The regression test that pins both directions of that distinction is the strongest evidence the fix works. The 8 new tests cover the meaningful edge cases.

Two Low nits: the missing source field on Info findings and the "pattern"-centric language in lint-task.md severity descriptions. Neither blocks merge.

Approve — no blocking issues.
|