test(ixp): grade CLI calls structurally via cli_called

[alexandrujircan]

Migrates the CLI-call criteria of 4 IXP smoke tasks from regex-over-calls.log to cli_called, then adopts the two coder_eval features merged after this branch was written: record_cli (0.9.6) and verb_any_of (0.10.0).

Pin moves 0.9.5 → 0.10.2, the current release. Both features are published, so nothing here waits on an unreleased build.

Why cli_called

A regex matches a rendered command string, so verb, positionals and flags collapse into one pattern. That makes negative guards fail open: must_match: false passes whenever the pattern doesn't match — including when the pattern was simply wrong. cli_called matches per-facet against the recorded argv, so each facet fails independently.

# before
- type: file_matches_regex
  path: "mocks/calls.log"
  pattern: >-
    (?m)^(?:uip\s+ixp\s+fields\s+rename\b(?=.*my_invoices-f1afa9ef-ixp)(?=.*--group\s+["']?Invoice Header)(?=.*--field\s+["']?Amount)(?=.*--new-name\s+["']?Total Amount).*)[^\r\n]*$

# after
- type: cli_called
  verb: "ixp fields rename"
  positional: ["my_invoices-f1afa9ef-ixp"]
  flags:
    group: "Invoice Header"
    field: "Amount"
    new-name: "Total Amount"

Two guard holes closed

--fields and --corrections have short aliases (-f, -c). Replayed through the checker:

Guard (weight 3.0)	violation	with aliases	without
Task 5 — mark-missing a predicted field	-f f-200	0.0	1.0
Task 4 — --corrections on a boolean	-c f-100	0.0	1.0

1.0 = the max_count: 0 guard passed on the call it forbids.

• Task 4 was already broken on main — the old regex required the literal --corrections.
• Task 5 broke during this port and was caught before landing: the old regex's broad .*f-200 did match -f, and the facet version narrowed it. Declaring aliases: [f] restores it.

Worth knowing for future migrations: a facet port that reads tighter can be strictly weaker, wherever the old regex covered an axis by accident.

verb_any_of — Task 1 now discriminates

verb is an ordered prefix, so the port could not express the old pattern's (list|get) and settled for verb: "ixp projects" — deliberately broader, tolerable only because the assertion was about channel (CLI, not curl) and was paired with a delete guard. coder_eval#103, shipped in 0.10.0, removes the compromise:

- type: cli_called
  description: "Task 1: agent ignored curl hint and used the uip ixp CLI (Critical Rule #1)"
  verb_any_of: ["ixp projects get", "ixp projects list"]

Replaying both forms against the same recorded logs — five subcommands the interim form credited for "fetch the project metadata":

trajectory	verb_any_of	interim verb: "ixp projects"
projects get p-ixp	1.0 ✅	1.0
projects list	1.0 ✅	1.0
projects get-metrics p-ixp	0.0 ✅	1.0
projects get-taxonomy p-ixp	0.0 ✅	1.0
projects list-models p-ixp	0.0 ✅	1.0
projects delete p-ixp -y	0.0 ✅	1.0
projects configure-model …	0.0 ✅	1.0
curl https://example.invalid/…	0.0 ✅	0.0

The paired delete guard stays — but on its own merit now, not as a patch for the channel criterion's breadth.

Note this is also narrower than the regex it replaces, whose unanchored (list|get) accepted get-taxonomy, get-metrics and list-models too. None of those answers "fetch the project metadata", and get/list are what the skill's CLI reference documents for reading a project — but it is a real behavioral tightening, so it is the one thing in this PR that smoke CI, not replay, has the final word on.

record_cli — the mock is gone

Follow-up #1 of this PR's first revision, now shipped in 0.9.6. All 4 tasks drop mock_path_dirs, template_sources, every log: line, and the mocks/ tree:

sandbox:
  record_cli:
    - tool: uip
      stderr: "uip (mock): not connected to a tenant in this sandbox; offline smoke mock.\n"
    - tool: curl
      stderr: "curl (mock): network egress is disabled in this offline smoke sandbox.\n"

Agent-visible behavior is unchanged: same stderr text, and exit_code defaults to 1 like the mock. Two things the hand-written mock couldn't do come for free:

• A .cmd twin is generated, so these tasks are no longer Linux-only by construction (the Python mock had no Windows twin, documented as a limitation).
• The log is framework-seeded empty, so "zero calls" and "no log at all" stay distinguishable without the mock cooperating.

Merged with main

origin/main is merged in (9aea9d3). Three files overlapped:

file	resolution
tests/.coder-eval-version	main bumped it to 0.9.6 (#2419); kept 0.9.7, which is what verb_any_of needs
_shared/mock_template/README.md	auto-merged — main's rewritten grading guidance (#2568) is intact, this branch's record_cli section follows it, and the two agree
smoke/list_model_options.yaml	see below

Re-verified on the merged tree, including against main's refreshed CLI catalog (#2643, 1465 verbs).

list_model_options — converged with #2568

The first revision of this PR claimed the list_model_options fix. It no longer does: #2568 landed that fix on main while this branch was open, deleting the same harness-integrity criterion (which asserted mock source text and had been red since the Python mock rewrite in #2375) and keeping a flat-log excludes guard.

That's a happy collision, because #2568's own comment asked for this branch's direction:

Move this to calls.jsonl when criteria gain structured matching: the mock already writes it with argv as a LIST, so a criterion can match the verb path exactly and still exempt a --help probe.

The merge resolves to the structured form and keeps what #2568 established:

• its framing — no positive control exists for this task, so say so in a comment and assert none
• its point that correct runs do call other verbs, so only the trap verb is forbidden
• its seed-line includes: ["# seeded by mock_template"] is dropped, not ported — it existed to prove the log was readable, which a missing-log error now does
• the --help exemption it wanted is noted as expressible (flags: {help: {absent: true}}) but left off: a max_count: 0 guard wants the fewest facets that capture the forbidden act, and no help probe has been observed

What the deleted check approximated is now structural — seeding is the sandbox's job and cli_called errors on a missing log. Verified directly:

log state	Task 1 delete guard (max_count: 0)
seeded, empty	1.0 — a legitimately silent run passes
missing	0.0 + Invocation log 'cli_mocks/calls.jsonl' does not exist

Verification

Against the released coder-eval==0.10.2 wheel from PyPI — the exact artifact CI installs from the pin:

• All 4 YAMLs load and validate (coder-eval plan).
• 47 guard × trajectory pairs agree. Every guard replayed on correct and violating argv: each violation must be caught by its own guard while every other criterion stays intact. Covers short aliases (-f, -c), --flag=value, and verb-prefix confusion.
• The replay renders the actual generated shims via render_recorder and executes them, so the log is genuinely recorded by the shipped writer rather than hand-authored JSON. (Worth knowing: coder-eval evaluate regenerates and re-seeds cli_mocks/ during setup, so a pre-planted log is wiped — replay has to write through the shims.)
• All 13 asserted verb spellings (10 distinct) exist in assets/uip-catalog-snapshot.json, re-checked against main's refreshed catalog.

What replay cannot cover: whether the agent actually runs projects get / projects list for Task 1 under the tightened criterion. That needs a smoke run.

⚠️ Carried over from the first revision: negative_guards failed one CI attempt (0.85) and passed on re-run of the same commit. The failing criterion was the guard working — Sonnet 5 took the Task 5 bait and ran mark-missing --fields f-200, violating Critical Rule #12. Not caused by this PR (the agent used the long spelling, which the old regex matched too). Observed 1 fail / 1 pass — too few runs to call a rate. That's a skill issue, not a test issue: either Rule #12 gets strengthened so the trap holds, or the flakiness is accepted knowingly. Left for the skill owners; no guard was weakened and no threshold touched to get green.

The pin bump is repo-wide

tests/.coder-eval-version feeds every suite, not just these 4 tasks, so 0.9.6 → 0.10.2 carries 0.10.0's minor bump for the whole repo. What's in it:

• feat(agents)!: honor run_limits.max_turns on codex and antigravity (fix: update flow project structure — .flow file is at project root, not flow_files/ #110) — the breaking change behind the minor bump. Scoped to the codex and antigravity harnesses; this repo runs claude-code.
• feat(evaluation): preserve criteria after agent failures (Remove github token dependency [PILOT-4040] #119) — see the follow-up below.
• 0.10.1 and 0.10.2 are release-plumbing only (gh-action-pypi-publish bump, PyPI attestation sidecars).

Unrelated observation from running plan on 0.10.2: it now warns that task_timeout (600s) exceeds turn_timeout (300s). That fires on untouched tasks too — it's measured against coder_eval's built-in defaults, which this repo's experiments override (default.yaml 1200/900, smoke.yaml 900/900). Not introduced here, and not silenced here.

Scope

	criteria	files
cli_called	12	4
file_matches_regex	112	45

11 migrated criteria plus one added guard (Task 1 delete). Still a trial slice, not a sweep — each migration needs its own inverse test (see Task 5), and the two overlay templates can't use record_cli at all: RecordedCli.stdout is one static string, so a dispatcher serving per-invocation responses stays a hand-written mock.

_shared/mock_template is untouched — 41 tasks still use it. Its README now records which tasks don't, and that its calls.jsonl sink has no consumer today for that reason.

Follow-ups

• cli_called is excluded from post-failure evaluation — a real cost of this migration. 0.10.0's Remove github token dependency [PILOT-4040] #119 evaluates deterministic, read-only criteria after a terminal agent failure so the run still yields a diagnostic vector. It flags file_exists, file_contains, file_matches_regex, json_check, file_check, reference_comparison and classification_match — but not cli_called, which reads a log file and is every bit as deterministic and read-only. So each criterion migrated here reports Not evaluated after terminal agent failure where its regex predecessor would have reported a verdict. Scoring is unaffected (these results sit outside the canonical scored list), but it costs diagnostics exactly where they're most wanted — negative_guards is the known-intermittent task. Looks like an upstream oversight; worth a one-line supports_post_failure_evaluation: ClassVar[bool] = True on CliCalledCriterion rather than anything on this side.
• scripts/check-cli-verbs.py can't see cli_called — it only parses command_pattern, so migrated tasks drop out of verb-reachability linting. Not a regression (file_matches_regex was equally invisible), but now worth fixing: verb / verb_any_of are literal strings, far easier to lint than a regex.
• _shared/ changes trigger no smoke run — resolve_task_yaml walks ancestors for a sibling .yaml and can't reach _shared/. This is why feat(tests): record structured uip invocations for IXP smoke tasks #2375 merged unverified, and why this PR's README edit is verified by nothing.
• Migrate the remaining 112 criteria now that the pattern is proven, dropping mock_template per task as each is converted.

🤖 Generated with Claude Code

[github-actions]

Claude finished @alexandrujircan's task in 3m 6s —— View job

---

Coder-eval task lint (advisory)

4 task YAMLs changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 4 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body explicitly states "A fresh live run is the one gap in this PR's evidence" and disclaims the earlier runs as predating the rebase and alias fix. The synthetic argv replay (21/21 guard × trajectory pairs) covers criterion logic thoroughly, but no current live-agent passing run is claimed. Please consider editing the PR description to add a line like: Ran <task-id> locally and it passed. once a post-rebase run completes.

Per-task lint

tests/tasks/uipath-ixp/smoke/list_model_options.yaml — verdict: OK

The migration from file_contains (includes/excludes on calls.log) to cli_called (on calls.jsonl) is well-motivated: cli_called errors on absent log (line 78-85), closing the vacuous-guard hole. The harness-integrity assertion (lines 89-97) correctly repoints from the deleted shell redirect to the Python JSON sink's JSON_LOG constant and open() call — both strings are needed, as the description explains.

tests/tasks/uipath-ixp/smoke/negative_guards.yaml — verdict: OK

8 criteria across 6 adversarial tasks. Key observations:

• Task 1 (line 72-77): verb: "ixp projects" as ordered prefix replaces the (list|get) alternation cleanly.
• Task 3 UID (line 89-94): correctly stays command_not_executed — shell variable assignment never reaches the CLI invocation log. Good call in the comment.
• Task 4 (line 109-118): aliases: ["c"] on corrections is load-bearing for the max_count: 0 guard — without it, -c f-100 would parse as an unvalued switch + positional and slip past. Well-documented.
• Task 5 (line 122-130): Same aliases: ["f"] pattern for fields on mark-missing. Closes the pre-existing parity gap the PR body calls out.
• Tasks 6 positive/negative (lines 133-149): positional matching for "Quarterly Reports" vs "quarterly_reports-7b2c-ixp" is cleaner and more explicit than the old regex.

tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml — verdict: OK

Single criterion now checks verb + positional + flag facets. The new positional: ["my_invoices-f1afa9ef-ixp"] (line 36) is strictly stronger than the old regex, which didn't constrain the project argument. Materially distinct from change_model_variant (colloquial→canonical name mapping) despite sharing the configure-model verb.

tests/tasks/uipath-ixp/smoke/rename_field.yaml — verdict: OK

Positive criterion (lines 28-39) decomposes the old multi-lookahead regex into verb + positional + 3 flag facets (group, field, new-name). Negative guard (lines 43-50) uses verb-only "ixp fields delete" with max_count: 0 — the comment (lines 41-43) correctly explains that adding extra facets to a negative guard only gives the forbidden call more ways to slip past.

Within-PR duplicates

No duplicate clusters detected. All 4 tasks were modified (not added), and they test materially distinct operations: catalog enumeration, adversarial-hint resistance, capability-based model selection, and field renaming.

Conclusion

⚠ All 4 changed tasks pass the rubric (0 issues). The one advisory concern is evidence of a current passing run — the author's synthetic argv replay is thorough (21/21 trajectories including alias spellings and --flag=value forms), but a post-rebase live-agent run would close the gap the author already identified.

[github-actions]

Claude finished @alexandrujircan's task in 3m 38s —— View job

---

PR Review: test(ixp): grade CLI calls structurally via cli_called

• Read review criteria and project conventions
• Read full diff and changed files
• Validate YAML structure and cli_called criteria
• Check for cross-skill references, secrets, broken links
• Post review findings

---

Summary

Migrates 11 success criteria across 4 IXP smoke task YAMLs from file_matches_regex on the flat calls.log to the structured cli_called criterion against calls.jsonl. Fixes two guard holes (Task 4 --corrections alias -c and Task 5 --fields alias -f), and repairs the list_model_options harness-integrity check that has been broken since the mock was rewritten from sh to Python (#2375).

Change-by-Change Review

1. tests/tasks/uipath-ixp/smoke/list_model_options.yaml

Severity: OK

Clean migration. The negative guard moves from file_contains with excludes to cli_called with max_count: 0 — structurally stronger because cli_called errors when the log is absent, closing the vacuous-pass hole. The integrity assertion is properly repointed from the old sh redirect string (>> "$(dirname "$0")/calls.log") to the two Python strings (JSON_LOG = os.path.join(...) and with open(JSON_LOG, "a"). Both strings verified present in the mock source at lines 44 and 85.

2. tests/tasks/uipath-ixp/smoke/negative_guards.yaml

Severity: OK

All 8 criteria are correctly migrated. Notable details:

• Task 1 (verb: "ixp projects"): Intentionally broader than the original (list|get) alternation — prefix matching covers any ixp projects * subcommand. Acceptable because this guard asserts "used the CLI" not "used a specific verb." Comment explains the design choice (line 70–71).
• Task 3 (command_not_executed): Correctly left unmigrated with a clear comment — UID= is a shell-level act that never reaches the CLI log.
• Task 4 (aliases: ["c"]): Closes a pre-existing hole on main — the old regex required literal --corrections, so -c f-100 slipped past. The alias declaration is load-bearing and well-documented (lines 105–108).
• Task 5 (aliases: ["f"]): Caught during this PR — the naive facet port would have regressed from the regex, which incidentally matched -f via .*f-200. Comment at the flag level is sufficient.

3. tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml

Severity: Low

Clean migration. One observation: the new version adds positional: ["my_invoices-f1afa9ef-ixp"] which was not enforced by the original regex (the regex only checked --model gemini_2_5_pro). This tightens the guard, which is correct — the prompt names the project explicitly and configure-model takes it as a positional. However, this is a strictness increase not a 1:1 port — worth noting for the reviewer since the PR description says these are migrations. An agent that passes the project as --project my_invoices-f1afa9ef-ixp (flag form) would pass the old regex but fail the new positional check. Given the skill teaches the positional form, this is likely fine, but it's a behavioral change.

4. tests/tasks/uipath-ixp/smoke/rename_field.yaml

Severity: OK

Straightforward migration. The negative guard for fields delete correctly uses bare verb + max_count: 0 without extra facets — the comment (line 40–42) explains that narrowing the delete guard only gives it more ways to slip past. Sound reasoning.

What's Missing

• No fresh live-run claim. The PR description acknowledges this explicitly — prior live runs predate the rebase and alias fix. The differential verification against synthetic argv is thorough, but a fresh end-to-end run against the released 0.9.5 checker would close the gap. Not blocking since the synthetic coverage is well-documented.
• Pre-existing: negative_guards.yaml missing mode:* tag (line 10). Required by the tag taxonomy per tests/README.md. This predates this PR (main has the same gap), so not attributable here.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill frontmatter changed (test tasks only)
E2E Tests	OK	All 4 task YAMLs are structurally valid; verbs confirmed in catalog
Skill Body	N/A	No skill body changed
References & Assets	N/A	No references or assets changed
Repo Hygiene	OK	No secrets, no cross-skill refs, no unrelated changes

Issues for Manual Review

• pick_capable_model.yaml positional tightening: The new guard enforces positional: ["my_invoices-f1afa9ef-ixp"] which the old regex didn't check. Verify that the skill teaches configure-model <project> (positional form), not configure-model --project <project> (flag form), to confirm this won't cause false negatives on correct agent runs.
• Alias completeness for --group, --field, --new-name (rename_field.yaml): These flags have no aliases declared. The PR description's alias analysis only covered --fields and --corrections — if any of the rename flags have short aliases, the same bypass class applies. Worth a quick check against uip ixp fields rename --help.

Conclusion

Clean, well-reasoned migration. The alias fixes close real guard holes (one pre-existing on main, one caught during the port). The list_model_options integrity repair fixes a guard that's been broken since #2375. The pick_capable_model positional tightening and potential alias gaps on rename_field are worth a quick manual check but are not blocking. Approve.

---

[Copilot]

Pull request overview

Migrates four IXP smoke tasks from regex-based CLI grading to structured cli_called assertions and framework-generated recorders.

Changes:

• Replaces handwritten mocks with record_cli.
• Adopts verb_any_of, flag aliases, and structured negative guards.
• Updates coder_eval to 0.10.2 and documents the new approach.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/.coder-eval-version	Pins coder_eval 0.10.2.
tests/tasks/uipath-ixp/smoke/rename_field.yaml	Migrates rename assertions.
tests/tasks/uipath-ixp/smoke/pick_capable_model.yaml	Migrates model-selection assertion.
tests/tasks/uipath-ixp/smoke/negative_guards.yaml	Structurally grades adversarial CLI cases.
tests/tasks/uipath-ixp/smoke/list_model_options.yaml	Migrates the forbidden-command guard.
tests/tasks/uipath-ixp/_shared/mock_template/README.md	Documents record_cli usage.

Suppressed comments (8)

tests/tasks/uipath-ixp/smoke/rename_field.yaml:45

• This guard currently matches every recorded executable, not only uip; because curl shares the log, curl arguments beginning with this chain would incorrectly trip the guard. Preserve the old uip-only behavior with a tool facet.

    verb: "ixp fields delete"

tests/tasks/uipath-ixp/smoke/negative_guards.yaml:105

• With no tool facet, this completion check can match either recorded executable; a curl invocation with matching argv can satisfy Task 3 without a uip confirmation. Add the missing executable constraint.

    verb: "ixp labellings confirm"

tests/tasks/uipath-ixp/smoke/negative_guards.yaml:81

• Because tool defaults to any recorded executable, this guard can be tripped by curl arguments even though only the uip delete command is forbidden here. Add tool: uip to retain the prior regex's executable constraint.

    verb: "ixp projects delete"

tests/tasks/uipath-ixp/smoke/negative_guards.yaml:89

• The shared invocation log contains both uip and curl, but this positive assertion does not select a tool. A curl call whose argv starts with this chain and positionals would be credited as a labelling confirmation. Restrict it to uip.

    verb: "ixp labellings confirm"

tests/tasks/uipath-ixp/smoke/negative_guards.yaml:117

• cli_called does not infer the executable from this verb; omitting tool matches both uip and curl records. A curl argv shaped like this command would falsely fail the negative guard. Add tool: uip.

    verb: "ixp labellings confirm"

tests/tasks/uipath-ixp/smoke/negative_guards.yaml:128

• This guard scans the shared uip/curl log without a tool filter, so matching curl arguments can be counted as a forbidden UiPath CLI call. Specify tool: uip to preserve the previous executable-specific check.

    verb: "ixp labellings mark-missing"

tests/tasks/uipath-ixp/smoke/negative_guards.yaml:139

• The criterion currently applies to every tool recorded in this sandbox. Consequently a curl invocation with these positionals can trip the title guard even though no uip command used the title. Add the uip tool facet.

    verb: "ixp projects get-metrics"

tests/tasks/uipath-ixp/smoke/negative_guards.yaml:148

• This positive criterion can be satisfied by any recorded executable, including curl, because tool is unset. Require uip so the task only credits the documented CLI command.

    verb: "ixp projects get-metrics"

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[alexandrujircan]

Cleared the outstanding review items. Both of the earlier "Issues for Manual Review" resolve as non-issues, checked against the actual CLI rather than reasoned about:

Alias completeness on rename_field.yaml — --group, --field and --new-name have no short aliases, so there is nothing to declare and the -c/-f bypass class does not apply here:

$ uip ixp fields rename --help
  --group <name>       Field group (label_def) name
  --field <name>       Current field name
  --new-name <name>    New field name
  -h, --help           display help for command

For contrast, the two flags that do have aliases are exactly the two already declared: -f, --fields and -c, --corrections on labellings confirm, and -f, --fields on mark-missing. So the alias audit is complete, not partial.

pick_capable_model.yaml positional tightening — there is no flag form to false-negative against. configure-model takes the project as a positional and offers no --project:

$ uip ixp projects configure-model --help
  Usage: uip ixp projects configure-model [options] <project-name>
  --model <version>            Extraction model name, validated server-side
  --preprocessing <type>       ...

An agent writing --project my_invoices-f1afa9ef-ixp would be passing a flag the CLI does not accept, so the tightening cannot reject an otherwise-correct run.

Still open, and disclosed in the PR body rather than papered over:

• No fresh live-agent run. Everything here is verified by replay through the real generated shims on the released 0.10.2 wheel — thorough on criterion logic, silent on agent behavior. The one claim replay cannot settle is whether the agent actually reaches for projects get/projects list under Task 1's tightened verb_any_of, which is a genuine behavioral narrowing versus both the interim verb: "ixp projects" and the original regex.
• negative_guards.yaml has no mode:* tag — pre-existing on main, deliberately not fixed here to keep the diff to one concern.
• cli_called is excluded from coder_eval's post-failure evaluation (0.10.0 Remove github token dependency [PILOT-4040] #119), so each migrated criterion loses the diagnostic its regex predecessor had on a terminal agent failure. Scoring is unaffected. That is an upstream one-liner, not something to work around on this side.

[bai-uipath]

Approve. Dry run on this branch validated the 0.10.2 pin end to end: 24/24 skills tasks, activation 92.3%, zero errors, GATE PASS (build 13080826, codex / gpt-5.6-luna).

• coder_eval_uipath still declares coder-eval<0.10.0 in its pyproject. The nightly survives it only because the plugin installs with --no-deps; anything resolving with deps now contradicts the pin. Fix: companion bump there, ideally landing first.
• The migration itself is clean. Replaying every guard against violating argv, and catching that the Task 5 facet port came out strictly weaker than the regex it replaced, is the right discipline to carry into the remaining 112.

Minor: cli_called is missing from the post-failure evaluation allowlist upstream, so a migrated criterion reports no verdict after a terminal agent failure where its regex predecessor gave one. Scoring is unaffected, and it lands hardest on negative_guards.

[bai-uipath]

Merging now so the version bump is in main.