fix(tests): report a broken tenant as ERROR, not as an agent failure

[rockymadden]

Stacked on #3088 — both touch outlook_trigger_inbox.yaml and generic_dynamic_node.yaml, so this is based on that branch rather than main. Retarget to main once #3088 lands.

Problem

On 2026-09-04 two flow tasks were scored FAILURE for reasons that had nothing to do with the agent:

Task	Actual cause
skill-flow-outlook-trigger-inbox	AADSTS50173 — the Outlook grant was revoked on 2026-08-31
skill-flow-generic-dynamic-node	The ServiceNow developer instance was hibernating, so every metadata call 403'd

Both got root-caused as skill defects first. The 403 sits several thousand characters into the checker output and nothing above it says the tenant is down, so the reports read as "the agent wrote a display name where an ID belongs" — which it did, but only because the lookup it was told to use could not answer.

That is 2 of the 8 failures in that nightly reporting the wrong thing, and it cost real time before anyone read far enough to find the 403.

Fix

A pre_run failure lands the run as FinalStatus.ERROR rather than FAILURE, and PreRunCommand's own docstring gives the reason:

the agent should not run against a broken environment

_shared/preflight_connections.py takes connector keys, lists their connections with --all-folders (without it an empty result is a false negative), and exits non-zero unless at least one is Enabled. On failure it says plainly that this is an environment problem and what to do:

TENANT NOT READY — this is an environment failure, not an agent failure.
  uipath-servicenow-servicenow: no Enabled connection (dev225223=Failed)

Reauthorize the connection, or wake the provider instance, then re-run.

Verification

Behavior checked against six tenant states, each asserted:

State	Exit
All connections Enabled	0
Hibernating provider (State: Failed)	1
Mixed, at least one Enabled	0
No connection in any folder	1
CLI exits non-zero	1
Result: Failure envelope	1

775 pytest pass. The two wired tasks keep their prompt and criteria unchanged — only pre_run is added.

Scope

Only the two tasks with evidence. 23 others name a connector in their prompt, but that list comes from grepping for uipath-* and it catches non-connectors like uipath-rpa and uipath-platform. Widening it is worth doing deliberately, not from a crude match — left as a follow-up.

What this does not do

It does not make either task pass. Both still need a working tenant. It makes the report tell the truth, so the next person triaging a nightly does not spend an hour proving the skill is fine.

🤖 Generated with Claude Code

[github-actions]

Claude finished @rockymadden's task in 2m 8s —— View job

---

Summary

Adds a pre_run preflight check to two flow tasks (outlook_trigger_inbox, generic_dynamic_node) so that a broken tenant connection (revoked grant, hibernating instance) lands the run as ERROR instead of FAILURE. A new shared script preflight_connections.py calls uip is connections list with --all-folders and exits non-zero unless every requested connector key has at least one Enabled connection.

Change-by-Change Review

1. tests/tasks/uipath-maestro-flow/_shared/preflight_connections.py (new, +77)

Severity: OK

Clean, well-structured script. Covers the right cases: CLI failure, Result != Success envelope, no connections found, no Enabled connections, and the happy path. Specific observations:

• Lines 30–34: subprocess.run with capture_output=True, text=True, and timeout=90 — correct. The 90s timeout inside the script is lower than the 120s pre_run timeout in the YAML, giving the runner time to handle a timeout cleanly.
• Line 37: json.loads(proc.stdout) could raise JSONDecodeError if the CLI outputs non-JSON (e.g. HTML error page, login redirect). This is caught by the broad except Exception on line 49, so it won't crash — the error message will just reference JSON parsing rather than the connection state. Acceptable tradeoff for a preflight script.
• Line 49: # noqa: BLE001 — justified; any exception here means the tenant is unreachable, which is exactly the signal we want.
• Lines 63–68: Error output goes to stderr, success goes to stdout — correct for a pre_run that needs non-zero exit to signal failure.
• Docstring (lines 1–20): Thorough motivation and usage. The incident dates are useful forensic context for this kind of infrastructure script.

2. tests/tasks/uipath-maestro-flow/connector_features/generic_dynamic_node/generic_dynamic_node.yaml (+4)

Severity: OK

pre_run added at lines 55–57, between initial_prompt and success_criteria. Connector key uipath-servicenow-servicenow matches the task's connector tag and description. The $SKILLS_REPO_PATH pattern and YAML quoting are consistent with existing pre_run/post_run commands in this test suite.

3. tests/tasks/uipath-maestro-flow/single_node/outlook_trigger_inbox/outlook_trigger_inbox.yaml (+4)

Severity: OK

Same structure. Connector key uipath-microsoft-outlook365 matches the task's connector tag and the command_executed criterion patterns on lines 69–90.

What's Missing

• Unit tests for preflight_connections.py — every other shared script in _shared/ has a corresponding test_*.py file (test_flow_check.py, test_validate_flow.py, etc.). The PR description says "775 pytest pass" and describes six tenant states that were verified, but there's no test_preflight_connections.py in this diff. The script's logic (subprocess calls, JSON parsing, state filtering) is testable with mocked subprocess.run. Medium — follows the established pattern for shared test infrastructure. Add tests →

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill frontmatter changed
E2E Tests	OK	Existing tasks unchanged; pre_run is additive
Skill Body	N/A	No skill body changed
References & Assets	N/A	No references or assets changed
Repo Hygiene	OK	No secrets, no cross-skill deps, $SKILLS_REPO_PATH consistent with codebase

Issues for Manual Review

• Confirm that uip is connections list <key> --all-folders --output json returns the expected {"Result": "Success", "Data": [...]} envelope in the CI runner's tenant. The script's parsing depends on this shape.
• The PR scopes to only 2 of 23 connector-dependent tasks. The PR description explains why (the grep catches non-connectors), but the other connector tasks will still report FAILURE on tenant issues. Worth a follow-up pass with a curated connector list.

Conclusion

Clean, well-motivated change. The script handles edge cases correctly, the YAML additions are minimal and correctly placed, and the PR is well-scoped to the two tasks with evidence. The missing unit test file is the only gap — the established _shared/test_*.py pattern makes this a natural addition. Approve with a suggestion to add test_preflight_connections.py in a follow-up or this PR.

---

[github-actions]

Claude finished @rockymadden's task in 2m 52s —— View job

---

Coder-eval task lint (advisory)

2 task YAMLs modified; 0 Critical, 0 High, 0 Medium, 1 Low, 1 OK. (_shared/preflight_connections.py is a helper script — excluded per rubric.)

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

---

Evidence of passing run

❌ High — PR body does not claim the modified tasks have been run through coder-eval and passed. "775 pytest pass" covers the preflight script's unit tests, not a coder-eval run of the two wired tasks. Since the change is additive (pre_run only, prompts and criteria unchanged), the risk is low — but please edit the PR description to add a line like:

Ran skill-flow-generic-dynamic-node and skill-flow-outlook-trigger-inbox locally and they passed (or: pre_run-only change; tasks pass when tenant is healthy, verified by prior nightly).

---

tests/tasks/uipath-maestro-flow/connector_features/generic_dynamic_node/generic_dynamic_node.yaml — verdict: OK

No issues. The added pre_run block correctly gates on uipath-servicenow-servicenow connectivity. Existing criteria (validate + debug + checker script at weight 6.0) provide strong meaningful coverage with cross-checks.

---

tests/tasks/uipath-maestro-flow/single_node/outlook_trigger_inbox/outlook_trigger_inbox.yaml — verdict: Low

Issues:

• [Low] Validate-only flow tests miss correctness: integration-tier task with no flow debug. Downgraded from Medium → Low per description-rationale carve-out (line 9–13): "A flow debug step is intentionally omitted — seeding a self-addressed email through the test-tenant Outlook connection is not reliable … Adding a debug E2E requires a dedicated service mailbox with no rules; tracked as follow-up." Pre-existing — not introduced by this PR.

Note: The added pre_run block correctly gates on uipath-microsoft-outlook365 connectivity. No new issues introduced.

---

Within-PR duplicates

No duplicate clusters detected. (Both modifications target existing, materially distinct tasks — one exercises ServiceNow generic-dynamic-node e2e, the other exercises Outlook trigger parentFolderId resolution.)

---

Conclusion

⚠️ 1 task has a pre-existing Low finding (validate-only, documented carve-out). The actual PR change — adding pre_run preflight gates — is clean across all rubric axes. Advisory only — not blocking merge. Please add a passing-run claim to the PR description.

[Copilot]

🟡 Changes recommended

The preflight can accept unhealthy or wrong-folder connections, use stale results, and lacks committed regression tests.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds tenant-connection preflight checks so two Flow evaluations report infrastructure outages as ERROR instead of agent failures.

Changes:

• Adds a shared Integration Service connection checker.
• Wires it into the Outlook and ServiceNow Flow tasks.
• Provides actionable tenant-remediation output.

File summaries

File	Description
_shared/preflight_connections.py	Checks connection state before evaluation.
outlook_trigger_inbox.yaml	Adds Outlook preflight.
generic_dynamic_node.yaml	Adds ServiceNow preflight.

Review details

Suppressed comments (1)

tests/tasks/uipath-maestro-flow/_shared/preflight_connections.py:60

• Do not report success from list state alone. Integration Service discovery requires uip is connections ping <id> after selecting an Enabled candidate to establish current health; otherwise this preflight can print OK without testing the revoked grant/provider availability it was introduced to detect. Ping the eligible candidates and pass only when at least one health check succeeds and reports Enabled.

        print(f"OK: {key} — {len(enabled)}/{len(conns)} connection(s) Enabled")

• Files reviewed: 3/3 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.