flightcheck: re-point ENV-004 to Declarative Agent components API

[Dawn Jeong (daeunJe0ng)]

Summary

Re-points the FlightCheck check ENV-004 (connections & connection references) from the Custom Agent Dataverse connectionreferences query to the Declarative Agent (DA) minimalBots components API.

Dependencies: the shared _da_connection_refs.py reader is provided by #326 (merge first). This PR is also stacked on #304 (DV-CONN-001 re-point), so merge order is #326 -> #304 -> this PR. Until #326 and #304 merge, this PR's diff also shows their commits; it reduces to the ENV-004-only diff once they land. This PR targets main.

FlightCheck is the ESS (Employee Self-Service) pre-flight validator a maker runs before shipping the agent. ENV-004 reports whether the agent's connection references are bound to real connections. On a Declarative Agent General Availability (DA-GA) install the agent's authored state no longer lives in Dataverse, so the old Dataverse read returned nothing and the check could not run. This change reads the same connection-reference data from the API the DA actually exposes, the same validated surface DV-CONN-001 now uses.

Work items: AB#7852495 (ENV-004 re-point), parent AB#7865450 (ESS DA GA FlightCheck).

What changed

• Uses the shared checks/_da_connection_refs.py reader (provided by flightcheck: add canonical DA connection-reference reader + contract test #326): fetches and normalizes the minimalBots components connectionReferenceChanges block into {connectionreferencelogicalname, connectorid, connectionid} rows. Single source for both DV-CONN-001 (single active agent) and ENV-004 (every configured agent, unioned and de-duped by logical name) so the two checks cannot drift. Fail-loudly: missing changeset key -> [] (genuine absence); present-but-not-a-list -> ValueError (dispatcher degrades to WARNING); no AgentBuilder client or no configured botId -> None (SKIP).
• checks/workday_extension.py: DV-CONN-001's _query_connection_references now delegates to the shared reader (behavior-preserving; the ValueError message and WARNING degrade path are unchanged).
• checks/environment.py: _check_connections_and_refs (ENV-004) rewritten to the DA model. Reads every configured agent's references via the shared reader; classifies each connectionId present = bound (PASS), absent = unbound (FAIL). Emits a summary row plus one ENV-004-UR-{i:03d} detail row per unbound reference. The env-wide orphan-reference / missing-reference / unbound-connection branches are dropped, because the DA components changeset carries no environment-wide connection inventory to compute them from. Removed the dead Dataverse helpers (_resolve_ref_solutions, _solution_link_parts) and their imports.
• New sub-check ENV-004-GRS (_env004_grs_commit_pin_result): opt-in commit pin that verifies the deployed agent's minimalBots ALM commit (GET .../alm/{id}/configure?realm={int} -> commitSha) matches an expected GRS commit SHA recorded in config. SKIP when no expected SHA is configured (inert until an operator opts in). Its verdict folds into the ENV-004 summary status so a targeted --check ENV-004 run (which filters detail rows out) still surfaces a GRS failure.
• registry.py: added the ENV-004 CheckpointSpec (a standalone checkpoint, not part of a checkpoint group), clients {AGENTBUILDER}, requires_dataverse_endpoint False, requires_config True, roles Power Platform admin + ESS maker. ENV-004 is deliberately not added to OWNED_PREFIXES, so its ENV-004-GRS / ENV-004-UR-* detail rows do not resolve as standalone checkpoints.
• Retired checks/_agent_connection_refs.py (the Dataverse topic->flow->reference scoper) and its test; updated the three cassette-catalog (INDEX.md) clauses that referenced it.
• Tests: replaced the ENV-004 Dataverse test file with a DA suite; extended the validated agentbuilder_connectivity mock with a commitSha builder; added a registry assertion.

Design rationale

• Same endpoint, already validated. ENV-004 reads connectionReferenceChanges from POST /copilotstudio/minimalBots/api/{agent_id}/components, the exact validated-tier surface the shipped native DA-CONN-001 and the stacked DV-CONN-001 read. The GRS pin reads .../alm/{id}/configure, whose commitSha is captured in the validated cassette agentbuilder_readiness.yaml (line 82). Per scripts/flightcheck/AGENTS.md no new cassette is required.
• No guessed fields, minimal verdict. The DA components connectionReference shape has no statuscode and no env-wide connection list, so the inactive-status, orphan, missing, and unbound-connection verdicts were dropped rather than faked. ENV-004 now asserts only what the API returns: is each declared reference bound.
• GRS pin is opt-in and inert by default. It SKIPs unless expectedGrsCommitSha (or an alias) is recorded in .local/config.json, so it never blocks a run that does not use commit pinning. It is a new local config key (see validation checklist).

Testing

• python -m pytest tests/flightcheck -> 1091 passed.
• python -m ruff check on all changed files -> clean.
• ENV-004 scenarios covered (each PASS/FAIL/WARN asserts a phrase from both result and remediation): all references bound (PASSED); one/more unbound (FAILED + ENV-004-UR-001 detail row); references unioned + de-duped across multiple configured agents; no AgentBuilder client (SKIPPED); no configured botId (SKIPPED); malformed changeset (WARNING); failed summary deep-links to the Power Apps solutions pane + connection-reference doc; unbound without env id falls back to prose. GRS pin: no expected SHA (SKIPPED + omitted from summary); matching commit (PASSED + folded into summary); mismatch (FAILED, forces summary FAIL); missing commitSha in configure (FAILED); configure read error (WARNING); invalid realm (FAILED); Dev realm = 0 regression guard (PASSED).
• Live run status: executed end to end against a real test-ring DA env (see the live-validation section below). All-bound PASS and multi-agent union are confirmed live; the unbound-reference FAIL path and the GRS pin were not reproducible because no reachable agent had an unbound reference and no deployed GRS commit was configured.

Risks

• Behavior change: ENV-004 no longer reports env-wide orphan / missing / unbound-connection findings. Those depended on a Dataverse connection inventory that does not exist in the DA components changeset. If that inventory is still wanted, it needs a separate env-wide source (open question).
• Like DV-CONN-001, ENV-004 builds the AgentBuilder client only via a plan or a targeted --check ENV-004 run; a broad scope run does not yet build it, so ENV-004 SKIPs there. Documented as a known limitation, mirroring DV-CONN-001.
• expectedGrsCommitSha is a new opt-in local config key. Absent it, the GRS pin is inert (SKIP).

Live DA-environment validation (partially complete)

Executed live against real test-ring Declarative Agent (DA) agents on 2026-09-22, authenticating as lmoulet@employeehub.onmicrosoft.com with the cached AgentBuilder token. This does NOT depend on the TIP2 (Test-In-Production ring 2) auth issue: ENV-004 reads the minimalBots components + ALM configure APIs, which the prod-audience token reaches. TIP2 only gates the separate Workday live "litmus" active probe (WD-RUN-001), which is not part of this PR. The check ran against the "ESS GA Sep15" env (env id 258bd301-3bcc-ef24-b832-867283451031) configured with all three of its agents.

Validated live:

• fetch_components on a real DA agent returns connectionReferenceChanges, with connectionId present for bound references. Confirmed: the Sep15 HR agent returned a bound shared_service-now reference.
• All-bound multi-agent config: ENV-004 -> PASSED; summary reported "2 reference(s) declared by the agent(s) | 2 bound".
• References unioned + de-duped across all configured agents: ran with three agents in agents[]; the summary collapsed to the 2 distinct bound references.
• Standalone --checkpoint ENV-004 run: cli.py built runner.agentbuilder and read the configured agent botIds with no false SKIP.

Not yet validated:

• Agent with an unbound reference: ENV-004 -> FAILED with an ENV-004-UR-001 detail row. No reachable test-ring agent currently has an unbound reference (all observed references were bound).
• Remediation deep link resolves: the Power Apps Solutions URL for the real env_id opens the Objects -> Connection references pane.
• SKIPPED path on a real run with no active-agent botId / no AgentBuilder client -> SKIPPED, no crash.
• Failure propagation: a real AgentBuilder auth/permission error surfaces as a dispatcher WARNING for ENV-004 (fail loudly), not a false PASS/SKIPPED.
• GRS pin end-to-end: record the real deployed commit as expectedGrsCommitSha in .local/config.json; confirm .../alm/{id}/configure?realm={int} returns a commitSha; ENV-004-GRS -> PASSED on match, FAILED on a deliberately wrong SHA.
• GRS realm: confirm the config realm string (Dev/Test/Prod) maps to the numeric realm the real ALM configure API expects, and that Dev (0) is accepted.

To reproduce the unbound-reference FAIL and the Workday-specific cases, a Workday DA agent must be provisioned in the dedicated Workday test env lmoulet-2026-09-21-da-ga-workday (env id a53fe556-24e9-e3ae-a008-9e8e82974f47, owner Laurent Moulet), which currently exposes no published DA agent.

---

Sync update (2026-09-24)

• Merged latest upstream/main into this branch (no rebase, no force-push).
• Resolved one import-block conflict in checks/workday_extension.py: main independently added a Dataverse query_all-based _query_connection_references helper; this branch (stacked on the DV-CONN-001 re-point, PR flightcheck: fail loud on malformed DA connection-ref payloads (reader hardening) #304) supersedes it with the reader-based read_active_agent_connection_references, so the now-dead query_all import was dropped. Kept main's agent_scope imports.
• Verified: ruff clean; targeted tests (test_workday_extension.py, test_da_connection_refs.py) 50 passed; full tests/flightcheck suite 1175 passed.
• Mergeable, blocked only on required review.