chore(coder-eval): bump to 0.11.1 and migrate to directory-only references by CarlesUIPath · Pull Request #2707 · UiPath/skills

CarlesUIPath · 2026-08-19T18:29:35Z

Bumps the coder-eval pin to 0.11.0 and migrates every reference-bearing task to the directory-only reference shape that release requires (coder_eval#106).

Why

coder-eval 0.11.0 makes task.reference directory-only — the reference: {file: ...} / {code: ...} shapes are removed and now raise a load-time migration error. Every uipath-troubleshoot task used reference: {file: RESOLUTION.md}, so all 297 fail to load under 0.11.0 until migrated. A directory is the only shape 0.11.0 can permission-gate as a unit, keeping the RESOLUTION.md ground truth unreadable by the agent under evaluation.

What changed

tests/.coder-eval-version: 0.9.6 → 0.11.0.
297 uipath-troubleshoot tasks: RESOLUTION.md moved into a per-task reference/ subdir (git-detected renames) and reference: {file: RESOLUTION.md} → reference: {directory: reference}. RESOLUTION.md is consumed only by each task's llm_judge via include_reference: true; no grader or $TASK_DIR path reads it, so the move is behavior-preserving at load.
_shared/scripts/generate_scenario.py: emits the new shape — writes reference/RESOLUTION.md and stamps reference: {directory: reference} — so newly generated scenarios are born migrated.
tests/tasks/uipath-troubleshoot/CLAUDE.md: updated the scenario-layout tree and the judge-context note to the reference/RESOLUTION.md shape.

Not changed (deliberately)

The ~hundreds of python3 $TASK_DIR/check_*.py grader invocations are left as $TASK_DIR. In 0.11.0 TASK_DIR is still exposed to run_command criteria (the anti-cheat shield only masks the task dir from the agent during turns; grading runs after and still resolves it). Rewriting those to $REFERENCE_DIR would break every grader — $REFERENCE_DIR only ever points at the reference/ solution directory.

Verification

coder-eval plan passes on all 297 migrated tasks under coder-eval 0.11.0 (was a hard load error before the migration).
Migration was mechanical + uniform: every task had exactly one reference:\n file: RESOLUTION.md block with a sibling RESOLUTION.md; no code: shapes, no other reference file names.

Sequencing / follow-up

Depends on coder-eval 0.11.0 being live on PyPI (it is).
⚠️ delegate-sdk runs: run-coder-eval.yml resolves coder-eval through the coder_eval_uipath plugin's declared range (comment cites >=0.9.5,<0.10.0). If that range hasn't been widened to admit 0.11.0, delegate-sdk runs will fail at resolve time until coder_eval_uipath widens it. Non-delegate agents (claude/codex/antigravity) are unaffected.

…ences coder-eval 0.11.0 makes task.reference directory-only; the removed `reference: {file: ...}` shape now raises a load-time migration error. All 297 uipath-troubleshoot tasks used `reference: {file: RESOLUTION.md}`, so they must migrate before the pin bump lands. - tests/.coder-eval-version: 0.9.6 -> 0.11.0 - 297 tasks: move RESOLUTION.md into a per-task reference/ subdir and change `reference: {file: RESOLUTION.md}` -> `reference: {directory: reference}`. RESOLUTION.md is consumed only by each task's llm_judge via include_reference; no grader / $TASK_DIR path reads it, so the move is behavior-preserving at load. - _shared/scripts/generate_scenario.py: emit the new shape (write reference/RESOLUTION.md, stamp `directory: reference`) so new scenarios are born migrated. - uipath-troubleshoot/CLAUDE.md: update the scenario-layout tree and the judge-context note. The `$TASK_DIR/check_*.py` graders are deliberately left as $TASK_DIR: TASK_DIR is still exposed to run_command criteria in 0.11.0 (the anti-cheat shield only masks the task dir from the agent during turns; grading runs after). $REFERENCE_DIR only ever points at the reference/ solution directory. Verified: `coder-eval plan` passes on all 297 migrated tasks under 0.11.0.

Resolves the tests/.coder-eval-version conflict: main bumped 0.9.6 -> 0.10.2 (#2565), this branch targets 0.11.0. Kept 0.11.0. The incoming ixp tasks use no reference: block, so the 0.11.0 bump does not hit the removed reference:{file:} shape; verified `coder-eval plan` passes on them under 0.11.0.

…ls under coder-eval 0.11's dropped DAC caps

…ials under coder-eval 0.11's dropped DAC caps (smoke + nightly Linux)

akshaylive

Please ensure that the tests smoke test failures are not a regression and please dry-run once before merging.

…icitly Follow-up to the directory-only reference migration. With `reference: {directory: reference}` + `include_reference: true`, coder-eval inlines the WHOLE reference/ directory into the judge as a separate block, so anything added there later would silently reach the judge (and it does not dedupe against a files: entry). Switch every troubleshoot llm_judge to the explicit, future-proof shape: include_reference: false files: - $REFERENCE_DIR/RESOLUTION.md This passes exactly RESOLUTION.md (once), keeps the judge input in sync with the file the block is meant to grade against, and restores the pre-0.11 single-file prompt shape (better score comparability). - 297 tasks: include_reference: true -> false + explicit files entry - generate_scenario.py: emit the new shape for new scenarios - uipath-troubleshoot/CLAUDE.md: canonical block + prose updated Verified: `coder-eval plan` still loads the migrated tasks under 0.11.0.

coder-eval 0.11 stages $TASK_DIR as an isolated copy of just the leaf task folder, so the ~215 graders that import the group-level tests/tasks/<group>/ _shared/ package by walking up the tree fail with ModuleNotFoundError in the container. Adds tests/scripts/stage_shared.sh — a before-execution step that copies each group's _shared/ INTO every task dir whose grader imports it. `python3 $TASK_DIR/check.py` puts the script's own dir on sys.path[0], so a co-located _shared/ resolves the import with NO grader changes (a symlink would not — the copytree ignores symlinks). Idempotent; mutates only the CI checkout. Wired before `coder-eval run` in smoke-skills.yml and run-coder-eval.yml (Linux). Verified on a worktree: stages 194 task dirs, fork_join gets _shared/bpmn_check.py, ixp (no importer) left untouched.

CarlesUIPath · 2026-08-20T15:07:23Z

Dry run on 0.11.0 is passing.

…t_summary, #133 anthropic 1.0.0/httpx2)

… anthropic>=1.0.0, a fresh major)

…eval" This reverts commit 93f2850. The forward fix is already open as #2707, which moves the pin to 0.11.1 — the coder_eval release carrying beeceddc, "bump anthropic to 1.0.0, migrate Bedrock judge path to httpx2" — and exempts anthropic from the safe-chain package-age gate in the same four workflows. Keeping the cap would break that merge rather than help it: 0.11.1 declares anthropic>=1.0.0,<2.0.0, so the two constraints are unsatisfiable and the install step would fail outright. uv pip install "coder-eval==0.11.1" "anthropic<1.0" -> No solution found when resolving dependencies The cap also would not have turned the check green. With httpx restored the tasks reached the agent and then hit a second, unrelated Windows break: claude-agent-sdk 0.2.144 cannot find a native claude.exe, so all three crash with agent_crash after 3 attempts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

UiPath/coder_eval#128 shipped in 0.11.2, so `_validate_extra_mount` now expands `~` and `$VAR` in a destination and the flow-v2-preview repo-root mount loads instead of aborting. Carries 0.11.0's breaking changes into the whole suite: the cap-drop anti-cheat window on every `docker run`, and directory-only `task.reference`. The ~298 `reference: {file:}` tasks under uipath-troubleshoot are skipped with a migration error until #2707 lands their migration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…dk-docker Keeps this branch's 0.11.2 pin over main's 0.11.1: #2707 moved the pin far enough to carry its own directory-only `task.reference` migration, and 0.11.2 adds UiPath/coder_eval#128 on top, which the flow-v2-preview repo-root mount needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…#2728) * feat(tests): runnable docker setup for the preview Maestro SDK skills Running the three preview/uipath-maestro-{flow,case,bpmn} builder-SDK skills as the only skill catalog needed four non-obvious pieces of setup, none of them recorded anywhere. Each missing piece scores as a capability failure rather than a config error, so the results look plausible and are not. Dockerfile: add a runtime npmrc for the @UiPath scope plus NPM_CONFIG_USERCONFIG. The build-time npmrc is written to globalconfig and deleted in the same RUN because it carries the literal token, so the image ships no npm auth at all; this layer adds a token-LESS npmrc referencing ${NODE_AUTH_TOKEN}, which npm expands at read time. NPM_CONFIG_USERCONFIG is what makes it reachable: npm resolves userconfig to $HOME/.npmrc, and the docker runner forwards --env HOME with the HOST value on purpose, overriding the image's /root, so npm looks in a directory the container does not have, never maps @UiPath to GitHub Packages, and 404s on the public registry. Without this, `npm install @uipath/flow-sdk` fails and every builder-SDK compile fails with it. tests/experiments/preview-maestro-sdk.yaml: the catalog is narrowed by pointing agent.plugins[].path at preview/, which shadows the same-named v1 skills. That path does double duty as a bind mount, so narrowing it also drops the repo-root mount that nightly gets for free, and criteria across the flow and case suites that shell out to tests/tasks/**/_shared/*.py then exit 2. extra_mounts restores the repo root explicitly. File-based auth is documented as a commented block: its destination has to equal the host $HOME for the same forwarded-HOME reason, and mounting to /root/.uipath instead yields "Not logged in" on every tenant call. tests/docker/preflight.sh: one container, run with the host HOME forwarded exactly as the harness does, asserting the eleven preconditions a full run depends on -- login state readable, SDK installs, and check/compile/check plus the product scaffold all succeeding. A manual `docker run` without --env HOME authenticates against the image's /root and reproduces none of these failures, which is why they are easy to miss. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(tests): unify auth on the nightly mount, rename to flow-v2-* The preview experiment shipped its login mount commented out with a `$HOME/.uipath` destination, blocked on destination-side `$VAR` expansion (UiPath/coder_eval#128). It doesn't need it: nightly.yaml and smoke.yaml already authenticate the in-container `uip` from `~/.uipath:/.uipath:rw`, a literal destination that validates on the pinned coder_eval 0.10.2. Adopt that mount verbatim rather than deriving a "more correct" one. It is the arrangement with a 500-task/night track record, and a login the CLI cannot see fails tasks on their tenant calls, which scores as a capability problem rather than a config error. flow-v2-preflight.sh mounts the same destination, so its "uip reports a live login" check is now the empirical test of the unified path. same-ground-headtohead.yaml gets the same treatment, dropping both `/home/tmatup/...` hardcodes and the comment deferring them to UiPath/coder_eval#100. One `$VAR` destination survives, on the repo-root mount, and it is unrelated to auth: DockerDriverConfig can forward host env vars but cannot set container ones, so criteria see `$SKILLS_REPO_PATH` with its forwarded host value and the mount has to land there. That single line is what still wants coder_eval#128. Renames make the Flow v2 scope legible: tests/experiments/preview-maestro-sdk.yaml -> flow-v2-preview.yaml tests/docker/preflight.sh -> flow-v2-preflight.sh same-ground-headtohead.yaml keeps its name: per tests/README.md it is a general campaign comparison arm (pinned skills-image:sg1, SG_EMPTY_SKILLS blanking installed skills), not the v1 half of a Flow v2 head-to-head. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(tests): trim flow-v2 comments and drop product checks from preflight Comment volume was out of proportion to the config it explains, in the experiment, same-ground, the Dockerfile npmrc layer and the README. flow-v2-preflight.sh loses its six product-behaviour checks (flow check --source / compile / emitted / check --compiled / solution init / scaffold) and the inline TS flow they needed. Those are what the eval measures, and pinning CLI verbs in a gate script only rots. What remains is the six config assertions whose failures score as capability problems instead: forwarded HOME, npm userconfig, @UiPath registry, token reachability, a live uip login, and an in-sandbox flow-sdk install. 82 lines to 52. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(tests): drop flow-v2-preflight.sh for a one-line login check The script's only load-bearing assertion was that the `/.uipath` login mount resolves under the forwarded host HOME. That is one `docker run`, now in tests/README.md. The rest either duplicated what a failing run already reports or pinned CLI verbs that drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(tests): move preview npm auth out of the shared image into pre_run The npmrc layer set ENV NPM_CONFIG_USERCONFIG on tests/docker/Dockerfile, which changes npm resolution for every suite, not just the Flow v2 preview. Several skills tell agents to `npm install -g @uipath/cli`, which is public-npm-only for stable releases and carries no registry override, so a global @UiPath -> GitHub Packages mapping is a hazard the preview run has no business creating. Revert the Dockerfile and write the npmrc from the experiment's pre_run instead, into $HOME, which is where npm resolves userconfig, so it holds whatever HOME the runner forwards and needs no NPM_CONFIG_USERCONFIG at all. pre_run executes inside the sandbox before the agent and fails the task loudly on error, so a broken write cannot masquerade as a capability miss. ${NODE_AUTH_TOKEN} stays literal; npm expands it at read time. Only flow-v2-preview.yaml needs this. No other experiment installs an @UiPath package in-sandbox. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(rpa-smoke): cap anthropic below 1.0 for the pinned coder-eval anthropic 1.0.0 (2026-08-20) moved its HTTP layer to httpx2. coder-eval 0.10.2 imports httpx in judge_bedrock without declaring it, so it only ever resolved transitively through anthropic 0.x. On a fresh install today llm_judge fails to import and every task carrying an llm_judge criterion errors at setup with score 0.00. Reproduced against the released wheel, nothing branch-specific: uv pip install "coder-eval==0.10.2" -> anthropic 1.0.0, no httpx -> import coder_eval.criteria.llm_judge ModuleNotFoundError: No module named 'httpx' uv pip install "coder-eval==0.10.2" "anthropic<1.0" -> anthropic 0.125.0, httpx 0.28.1, llm_judge imports coder_eval main already switched to httpx2, so this cap comes off with the next .coder-eval-version bump. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert: "ci(rpa-smoke): cap anthropic below 1.0 for the pinned coder-eval" This reverts commit 93f2850. The forward fix is already open as #2707, which moves the pin to 0.11.1 — the coder_eval release carrying beeceddc, "bump anthropic to 1.0.0, migrate Bedrock judge path to httpx2" — and exempts anthropic from the safe-chain package-age gate in the same four workflows. Keeping the cap would break that merge rather than help it: 0.11.1 declares anthropic>=1.0.0,<2.0.0, so the two constraints are unsatisfiable and the install step would fail outright. uv pip install "coder-eval==0.11.1" "anthropic<1.0" -> No solution found when resolving dependencies The cap also would not have turned the check green. With httpx restored the tasks reached the agent and then hit a second, unrelated Windows break: claude-agent-sdk 0.2.144 cannot find a native claude.exe, so all three crash with agent_crash after 3 attempts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(tests): bump the coder-eval pin to 0.11.2 UiPath/coder_eval#128 shipped in 0.11.2, so `_validate_extra_mount` now expands `~` and `$VAR` in a destination and the flow-v2-preview repo-root mount loads instead of aborting. Carries 0.11.0's breaking changes into the whole suite: the cap-drop anti-cheat window on every `docker run`, and directory-only `task.reference`. The ~298 `reference: {file:}` tasks under uipath-troubleshoot are skipped with a migration error until #2707 lands their migration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(smoke): exclude claude-agent-sdk 0.2.144 on the Windows RPA runner 0.2.144 is the first release in months published without a win_amd64 wheel, so uv falls back to the sdist and nothing bundles claude.exe. All three RPA smoke tasks crash at agent_crash before doing any work. coder-eval requires claude-agent-sdk>=0.2.124 with no upper bound, so exclude that single release rather than capping the range: 0.2.143 has the wheel, and a later release that restores it resolves normally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

CarlesUIPath requested review from MarinRzv, Stefan-Virgil, akshaylive, bai-uipath, costin-uipath, dmorosanu, uipreliga and vladimir-cozma as code owners August 19, 2026 18:29

CarlesUIPath added 4 commits August 19, 2026 19:31

ci(smoke): widen ~/.uipath perms so container-root can read credentia…

817b950

…ls under coder-eval 0.11's dropped DAC caps

Revert the ~/.uipath chmod smoke fix

bf7a364

ci: widen ~/.uipath perms (recursive) so container-root reads credent…

d29ae01

…ials under coder-eval 0.11's dropped DAC caps (smoke + nightly Linux)

akshaylive approved these changes Aug 19, 2026

View reviewed changes

CarlesUIPath added 2 commits August 20, 2026 14:22

CarlesUIPath requested review from AlvinStanescu, RaduAna-Maria, dmetzgar, gabrielavaduva, gozhang2, marius-bughiu, rockymadden and smflorentino as code owners August 20, 2026 13:35

rockymadden approved these changes Aug 20, 2026

View reviewed changes

chore(coder-eval): bump pin 0.11.0 -> 0.11.1 (patch: #127 codex resul…

c12cb8f

…t_summary, #133 anthropic 1.0.0/httpx2)

CarlesUIPath changed the title ~~chore(coder-eval): bump to 0.11.0 and migrate to directory-only references~~ chore(coder-eval): bump to 0.11.1 and migrate to directory-only references Aug 21, 2026

ci: exempt anthropic from safe-chain age gate (coder-eval 0.11.1 pins…

ab5a96c

… anthropic>=1.0.0, a fresh major)

akshaylive approved these changes Aug 21, 2026

View reviewed changes

bai-uipath mentioned this pull request Aug 22, 2026

feat(tests): runnable docker setup for the preview Maestro SDK skills #2728

Merged

akshaylive merged commit 2ca5da0 into main Aug 24, 2026
24 checks passed

akshaylive deleted the chore/coder-eval-0.11.0-reference-migration branch August 24, 2026 18:25

apetraru-uipath mentioned this pull request Aug 25, 2026

docs(guardrails): don't let fetched SDK docs override tenant availability #2787

Merged

Navigation Menu

Uh oh!

chore(coder-eval): bump to 0.11.1 and migrate to directory-only references - #2707

chore(coder-eval): bump to 0.11.1 and migrate to directory-only references#2707
akshaylive merged 9 commits into
mainfrom
chore/coder-eval-0.11.0-reference-migration

CarlesUIPath commented Aug 19, 2026

Uh oh!

akshaylive left a comment

Uh oh!

CarlesUIPath commented Aug 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

CarlesUIPath commented Aug 19, 2026

Why

What changed

Not changed (deliberately)

Verification

Sequencing / follow-up

Uh oh!

akshaylive left a comment

Choose a reason for hiding this comment

Uh oh!

CarlesUIPath commented Aug 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants