Skip to content

flightcheck: re-point WD-REST-001 + WD-ENV-001 to Declarative Agent components API - #317

Open
Dawn Jeong (daeunJe0ng) wants to merge 12 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-wd-scp-7852511
Open

Dawn Jeong (daeunJe0ng) wants to merge 12 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-wd-scp-7852511

Conversation

@daeunJe0ng

@daeunJe0ng Dawn Jeong (daeunJe0ng) commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Re-points WD-REST-001 (AB#7852511) and WD-ENV-001 (AB#7852509) to the Declarative Agent components API.

This PR stacks on the shared reader in #326 and on #304. Merge order: #326 -> #304 -> this PR. Until they merge, GitHub shows their commits in this diff.

What changed

  • Uses the shared flightcheck/checks/_da_connection_refs.py reader (provided by flightcheck: add canonical DA connection-reference reader + contract test #326) for Declarative Agent connection references.
  • Relies on the reader's two scoped entry points:
    • read_active_agent_connection_references returns only the active agent's references (config agent.botId). DV-CONN-001 consumes this.
    • _all_agents_connection_references returns every configured agent's references. The Workday shared-parameter sweep (WD-REST-001 / WD-ENV-001) consumes this, because it must inspect each configured agent's own Workday reference.
  • Re-pointed WD-REST-001 to read sharedConnectionParameters.values.restBaseUri from the Workday connection reference in the components payload.
  • Re-pointed WD-ENV-001 to read the Workday sharedConnectionParameters.values keys:
    • tenantName
    • token:ResourceUri
    • token:WorkdayTokenUri
    • token:WorkdayClientId
  • Updated the single-checkpoint registry so WD-REST-001 and WD-ENV-001 require AGENTBUILDER and do not require a Dataverse endpoint.
  • Added a documented-tier Workday sharedConnectionParameters builder: wrapper shape from tools/ess-ca-to-da/reference/hr/agent.yml, Workday keys from the Workday connector docs. No live components cassette captures these keys yet. Added PASS / FAIL / SKIP / WARNING coverage for both checks.

Design rationale

DV-CONN-001 and ENV-004 already use agentbuilder.AgentBuilderClient.fetch_components(agent_id) against the Declarative Agent components endpoint.

This PR uses the same endpoint and shared reader pattern instead of introducing a parallel minimalBots reader. The enclosing connectionReferenceChanges[].connectionReference shape is cassette-backed, but nested sharedConnectionParameters.values is documented-tier only: wrapper shape from the reference agent.yml, Workday keys from the connector docs, and no live components response captured yet. Workday PASS paths remain unverified live until a Workday-bound DA agent is available.

DV-CONN-001 scoping: an earlier revision of the shared reader iterated every configured agent, so DV-CONN-001 (a HIGH-priority "Workday SOAP connection reference bound to a connection you own" check) could PASS or FAIL on a Workday reference belonging to a non-active agent in a multi-agent config - a false verdict on the agent under check. The reader now returns only the active agent's references, matching the check's intent and its function name. The Workday shared-parameter sweep keeps the every-agent behavior it needs via the separate _all_agents_connection_references. Single-agent configs are unaffected; the change only matters for multi-agent configs.

The implementation does not guess alternate API fields. If the components payload shape is malformed, the reader raises ValueError, and the owning check reports a WARNING instead of returning a false PASS or FAIL.

Testing

Run from C:\Users\dawnjeong\worktrees\fc-pr-g2 with:

$env:PYTHONPATH="C:\Users\dawnjeong\worktrees\fc-pr-g2\solutions\ess-maker-skills\scripts"
python -m pytest tests\flightcheck -q
python -m ruff check solutions\ess-maker-skills\scripts

Results:

  • python -m pytest tests\flightcheck -q -> 1135 passed in 8.45s
  • python -m ruff check solutions\ess-maker-skills\scripts -> All checks passed!

Added a multi-agent regression test in tests/flightcheck/checks/test_workday_extension.py: the active agent has no Workday reference while another configured agent does; DV-CONN-001 must FAIL "not found" rather than pass on the other agent's binding. This pins the single-active scoping described above.

Update 2026-09-23: parse JSON-string sharedConnectionParameters

Live AgentBuilder returns connectionReference.sharedConnectionParameters as a JSON string, not a nested object. The reader required a dict and raised ValueError on the string shape, which would degrade WD-REST-001 and WD-ENV-001 to WARNING instead of a clean PASS against a real Workday-bound agent. The reader now json.loads the string before validating the shape. Added a live-shape test (test_shared_parameters_json_string_shape_passes, WD-ENV-001 passes on the string form) and a mock helper shared_connection_parameters_json_string. Re-run: python -m pytest tests\flightcheck -q -> 1136 passed; ruff clean.

Live DA-environment validation (partially complete)

Live run 2026-09-22 against ESS GA env 258bd301 (prod ring; non-Workday agent): both checks executed against a real DA agent and read connectionReferenceChanges. With no Workday connection reference present, WD-REST-001 returned the documented restBaseUri-missing FAIL with the reconnect remediation (checked below). WD-ENV-001 also FAILED and listed the missing keys, but because no Workday connref exists the 'connref present, one key missing' sub-case was not isolated, so that box stays unchecked. All Workday PASS paths need a Workday-bound DA agent (none reachable this run).

To be checked once a Declarative Agent that exposes Workday sharedConnectionParameters in its components payload is provisioned. Blocked on a published Workday-bound DA agent in the test env; these components reads use the prod-audience AgentBuilder token and do NOT depend on the TIP2 (Test-In-Production ring 2) auth issue (TIP2 only gates the separate WD-RUN-001 active probe, which is not in this PR). Each box is marked when the scenario is confirmed on a live DA env.

  • WD-REST-001 against a DA Workday agent where restBaseUri is present and trimmed to /api -> PASSED.
  • WD-REST-001 where restBaseUri is missing or empty -> FAILED with reconnect-the-Workday-connection remediation.
  • WD-ENV-001 where tenantName, token:ResourceUri, token:WorkdayTokenUri, and token:WorkdayClientId are all present -> PASSED.
  • WD-ENV-001 missing one required token key -> FAILED listing the missing key.
  • WD-ENV-001 is emitted once on a real run (no double emission with the legacy env-vars path).
  • No-AgentBuilder-client and no-active-agent-botId paths SKIP instead of failing.
  • Malformed connectionReferenceChanges / sharedConnectionParameters.values shape -> WARNING (fail loudly).

Sync 2026-09-24: Merged latest upstream/main (merge 9c487083). Two conflicts, both import-block collisions in flightcheck/checks/: workday.py and workday_extension.py. This PR (stacked on the DV-CONN-001 re-point) imports the components reader from _da_connection_refs; main added agent_scope helpers + auth.query_all. Kept BOTH import groups. In workday_extension.py the merged body uses the reader path, so main's query_all import was dead (ruff F401) and was removed with its stale comment - same supersession #314 handled. Ruff clean; tests/flightcheck 1220 passed. Net delta over main is ~674 lines re-pointing WD-REST-001 + WD-ENV-001 to the Declarative Agent components API - not superseded by main's independent Workday DA work.

@daeunJe0ng
Dawn Jeong (daeunJe0ng) marked this pull request as ready for review September 23, 2026 05:26
…test (AB#7852506, AB#7852495, AB#7852511)

Extract the shared minimalBots components connection-reference reader into
checks/_da_connection_refs.py as the single source that DV-CONN-001 (active
agent), ENV-004 (environment-wide, de-duped by logical name) and the Workday
shared-parameter checks (WD-ENV-001/WD-REST-001) all read through, so the
per-check re-point PRs stop adding divergent copies of it.

The module is the superset of PR microsoft#317's reader plus a public
read_all_agents_connection_references() that de-dupes by connection-reference
logical name for ENV-004. Brings the validated/documented mock builders the
checks need and a direct pure-logic contract test (18 cases) covering
normalization, None-gating (SKIP), fail-loudly ValueError, JSON-string
sharedConnectionParameters parsing, and the Workday shared-parameter sweep.

Foundation for the 6 DA re-point PRs to rebase onto (single shared reader,
no add/add collision). Full flightcheck suite: 1142 passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
… (7852506)

Read the Workday SOAP connection reference from the Declarative Agent
minimalBots components API (runner.agentbuilder.fetch_components) instead of the
Custom Agent Dataverse connectionreferences query, so DV-CONN-001 works against
DA-GA agents that no longer expose Dataverse. Verdict simplifies to found +
bound (connectionId present); the Dataverse-only statuscode and multiple-ref
branches are dropped since the components shape carries neither. BAP owner echo
is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Distinguish a genuinely-absent connectionReferenceChanges (missing key -> no
references -> FAILED not-found) from a present-but-non-list payload (a shape we
do not understand -> raise ValueError so the dispatcher degrades DV-CONN-001 to
a WARNING). Mirrors native_agent._connection_references, the shipped precedent
this check re-uses; the prior code collapsed both to [] and could report a
confident "reference not found" on an unparseable 200 response. Adds a
malformed-changeset test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…list)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
…7852511)

Re-point WD-REST-001 and WD-ENV-001 to AgentBuilder components sharedConnectionParameters so Workday REST and token configuration no longer depend on Dataverse/minimalbots legacy reads.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
@daeunJe0ng
Dawn Jeong (daeunJe0ng) force-pushed the users/dawnjeong/flightcheck-da-wd-scp-7852511 branch from c04f2cd to c1bd684 Compare September 24, 2026 00:16
…selection (WD-REST-001/WD-ENV-001)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
…rams sweep across all agents (AB#7852511)

The shared _da_connection_refs reader iterated every configured agent, so
DV-CONN-001 (a HIGH-priority "Workday SOAP connection reference bound to a
connection you own" check) could PASS/FAIL on a Workday reference belonging
to a non-active agent in a multi-agent config - a false verdict on the agent
under check.

Split the reader in two:
  * read_active_agent_connection_references now returns only the active agent's
    references (config agent.botId), matching DV-CONN-001's intent and its
    function name.
  * _all_agents_connection_references keeps the every-agent behavior the
    Workday shared-parameter sweep (WD-REST-001 / WD-ENV-001) needs.

Row shape is unchanged (still carries botid + sharedconnectionparameters), so
no caller changes. Single-agent configs are unaffected. Added a multi-agent
regression test: the active agent has no Workday reference while another
configured agent does; DV-CONN-001 must FAIL "not found" rather than pass on
the other agent's binding.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
…nents reader (WD-REST-001/WD-ENV-001)

Live AgentBuilder returns connectionReference.sharedConnectionParameters as a
JSON string, not a nested object. The reader required a dict and raised
ValueError on the string shape, degrading WD-REST-001 and WD-ENV-001 to WARNING
instead of a clean PASS against a real Workday-bound agent. json.loads the
string before validating, and add a live-shape test plus mock helper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
@daeunJe0ng Dawn Jeong (daeunJe0ng) changed the title flightcheck: re-point WD-REST-001 + WD-ENV-001 to Declarative Agent components API (AB#7852511, AB#7852509) flightcheck: re-point WD-REST-001 + WD-ENV-001 to Declarative Agent components API Sep 24, 2026
Dawn Jeong (daeunJe0ng) added a commit that referenced this pull request Sep 24, 2026
…test (#326)

Extract the shared minimalBots components connection-reference reader into
checks/_da_connection_refs.py as the single source that DV-CONN-001 (active
agent), ENV-004 (environment-wide, de-duped by logical name) and the Workday
shared-parameter checks (WD-ENV-001/WD-REST-001) all read through, so the
per-check re-point PRs stop adding divergent copies of it.

The module is the superset of PR #317's reader plus a public
read_all_agents_connection_references() that de-dupes by connection-reference
logical name for ENV-004. Brings the validated/documented mock builders the
checks need and a direct pure-logic contract test (18 cases) covering
normalization, None-gating (SKIP), fail-loudly ValueError, JSON-string
sharedConnectionParameters parsing, and the Workday shared-parameter sweep.

Foundation for the 6 DA re-point PRs to rebase onto (single shared reader,
no add/add collision). Full flightcheck suite: 1142 passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
…ghtcheck-da-wd-scp-7852511

# Conflicts:
#	solutions/ess-maker-skills/scripts/flightcheck/checks/workday.py
#	solutions/ess-maker-skills/scripts/flightcheck/checks/workday_extension.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant