Skip to content

flightcheck: re-point WD-WF-CAT-001 + WD-REF-001 to Declarative Agent components API - #318

Closed
Dawn Jeong (daeunJe0ng) wants to merge 10 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-wd-botcomp-7852513
Closed

Dawn Jeong (daeunJe0ng) wants to merge 10 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-wd-botcomp-7852513

Conversation

@daeunJe0ng

@daeunJe0ng Dawn Jeong (daeunJe0ng) commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Re-points two Workday FlightCheck checks to the Declarative Agent components API:

  • AB#7852505 / WD-WF-CAT-001: enumerate Workday topic components from botComponentChanges by matching schema names that contain .topic.Workday.
  • AB#7852513 / WD-REF-001: enumerate Workday reference-data structure from botComponentChanges by matching .variable.*LookupTable variables plus WorkdaySystemGetReferenceData and WorkdaySystemRefreshReferenceData topics.

This PR stacks on the shared reader in #326 and on #304. Merge order: #326 -> #304 -> this PR. Until they merge, the diff shows their commits.

What changed

  • Reads the top-level botComponentChanges list from the validated components payload via runner.agentbuilder.fetch_components(agent_id).
  • Updated WD-REF-001 to produce a deterministic structural verdict without Dataverse:
    • Passed when LookupTable variables and both reference-data topics are present.
    • Failed when the structural components are absent.
    • Skipped when AgentBuilder or the active botId is unavailable.
    • Warning when the components payload has a malformed botComponentChanges shape.
  • Updated WD-WF-CAT-001 to produce a deterministic Workday topic inventory from Declarative Agent topic component schema names.
  • Registered WD-REF-001 and WD-WF-CAT-001 for single-checkpoint execution with AGENTBUILDER only and requires_dataverse_endpoint=False.
  • Rewrote the two checks' tests against tests/mocks/agentbuilder_connectivity.py and extended that mock with a botComponentChanges builder.
  • Removed 8 now-dead module-level items left by the re-point (_WD_REF_KEY_LABELS, _WD_REF_SUPPORTED_RE, _WD_REF_REQUESTED_RE, _ref_key_label, _extract_requested_reference_keys, _extract_supported_reference_keys, _format_unknown_scenarios, _WD_WF_CAT_CHECKLIST): the old Dataverse reference-key reconciliation and catalog-checklist output that the structural DA verdict no longer calls (git history retains them if US 7792327 restores semantic classification).

Design rationale

The source of truth is the AgentBuilder components payload returned by AgentBuilderClient.fetch_components(agent_id). botComponentChanges is a top-level list in the same validated payload used by #304 for connectionReferenceChanges.

These checks are intentionally structural and flowless. They verify that the Declarative Agent contains the expected Workday topic and reference-data component shapes without depending on the old Dataverse botcomponents table, workspace-local extracted files, or cloud flow inventory.

The shipped-vs-custom classification catalog and final reference-data key rules are external inputs owned by Skantha's follow-up work, US 7792327. This PR does not block on that catalog. Classification remains gated out, and the result text explicitly notes that the current verdict is structural enumeration only pending US 7792327.

Testing

  • python -m pytest tests\flightcheck -q
    • Result: 1114 passed in 5.62s
  • python -m ruff check solutions\ess-maker-skills\scripts
    • Result: All checks passed!

Live DA-environment validation (partially complete)

Live run 2026-09-22 against ESS GA env 258bd301 (prod ring; non-Workday agent): the shared DA component reader fetched botComponentChanges from a real DA agent; with no Workday components present, WD-REF-001 returned the documented FAIL (checked below) and WD-WF-CAT-001 produced a deterministic empty Workday-topic inventory (0 .topic.Workday*). The Workday-present PASS paths need a Workday-bound DA agent (none reachable this run).

To be checked once a published Workday-bound Declarative Agent is provisioned in the test env. Blocked on that agent; these are read-only components reads on the prod-audience AgentBuilder token and do NOT depend on the TIP2 (Test-In-Production ring 2) auth issue. Structural enumeration only until the classification catalog lands (US 7792327). Each box is marked when the scenario is confirmed on a live DA env.

  • WD-REF-001 against a DA Workday agent whose components include .variable.*LookupTable variables plus WorkdaySystemGetReferenceData and WorkdaySystemRefreshReferenceData topics -> PASSED.
  • WD-REF-001 against a payload missing those structural components -> FAILED.
  • WD-WF-CAT-001 against a DA Workday agent -> deterministic Workday topic inventory from .topic.Workday* schema names in botComponentChanges.
  • No-AgentBuilder-client / no-active-botId -> SKIPPED, no crash.
  • Malformed botComponentChanges shape -> WARNING (fail loudly).

Follow-up

@daeunJe0ng
Dawn Jeong (daeunJe0ng) marked this pull request as ready for review September 23, 2026 05:26
…test (AB#7852506, AB#7852495, AB#7852511)

Extract the shared minimalBots components connection-reference reader into
checks/_da_connection_refs.py as the single source that DV-CONN-001 (active
agent), ENV-004 (environment-wide, de-duped by logical name) and the Workday
shared-parameter checks (WD-ENV-001/WD-REST-001) all read through, so the
per-check re-point PRs stop adding divergent copies of it.

The module is the superset of PR microsoft#317's reader plus a public
read_all_agents_connection_references() that de-dupes by connection-reference
logical name for ENV-004. Brings the validated/documented mock builders the
checks need and a direct pure-logic contract test (18 cases) covering
normalization, None-gating (SKIP), fail-loudly ValueError, JSON-string
sharedConnectionParameters parsing, and the Workday shared-parameter sweep.

Foundation for the 6 DA re-point PRs to rebase onto (single shared reader,
no add/add collision). Full flightcheck suite: 1142 passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
… (7852506)

Read the Workday SOAP connection reference from the Declarative Agent
minimalBots components API (runner.agentbuilder.fetch_components) instead of the
Custom Agent Dataverse connectionreferences query, so DV-CONN-001 works against
DA-GA agents that no longer expose Dataverse. Verdict simplifies to found +
bound (connectionId present); the Dataverse-only statuscode and multiple-ref
branches are dropped since the components shape carries neither. BAP owner echo
is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Distinguish a genuinely-absent connectionReferenceChanges (missing key -> no
references -> FAILED not-found) from a present-but-non-list payload (a shape we
do not understand -> raise ValueError so the dispatcher degrades DV-CONN-001 to
a WARNING). Mirrors native_agent._connection_references, the shipped precedent
this check re-uses; the prior code collapsed both to [] and could report a
confident "reference not found" on an unparseable 200 response. Adds a
malformed-changeset test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…list)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
@daeunJe0ng
Dawn Jeong (daeunJe0ng) force-pushed the users/dawnjeong/flightcheck-da-wd-botcomp-7852513 branch from e0a173f to cc0895f Compare September 24, 2026 00:14
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
…r DA re-point (WD-REF-001, WD-WF-CAT-001)

The WD-REF-001 and WD-WF-CAT-001 rewrites to DA structural enumeration
left 8 module-level items with no remaining call sites: _WD_REF_KEY_LABELS,
_WD_REF_SUPPORTED_RE, _WD_REF_REQUESTED_RE, _ref_key_label,
_extract_requested_reference_keys, _extract_supported_reference_keys
(old Dataverse reference-key reconciliation), plus _format_unknown_scenarios
and _WD_WF_CAT_CHECKLIST (old catalog checklist output). Removing them
keeps the check surface honest; git history retains them if the pending
catalog work (US 7792327) restores semantic classification.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
Update the WD-WF-CAT-LINK remediation to point at the current botComponentChanges-based topic inventory and refresh the validation matrix for WD-WF-CAT-001 plus the missing WD-REF-001 structural row.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
@daeunJe0ng Dawn Jeong (daeunJe0ng) changed the title flightcheck: re-point WD-WF-CAT-001 + WD-REF-001 to Declarative Agent components API (AB#7852505, AB#7852513) flightcheck: re-point WD-WF-CAT-001 + WD-REF-001 to Declarative Agent components API Sep 24, 2026
@daeunJe0ng

Copy link
Copy Markdown
Contributor Author

Closing as superseded.

While this PR was queued, main independently re-implemented WD-WF-CAT-001 and WD-REF-001. This PR re-points those checks to read Workday from the DA bot components API (_da_components.read_component_schema_names -> _workday_topic_schema_names). That is the wrong layer: Workday DA is flow/solution-scoped (connection ref new_sharedworkdaysoap_ff0df) and does not surface in botComponentChanges, so the re-pointed checks would look for components that structurally cannot appear.

main now handles Workday DA at the correct layer via checks/workday_da.py (WD-DA-PKG-001, _check_workday_da_package_installed), verifying the Workday DA solution/package is installed, and keeps the topic + Dataverse OOTB catalog reconciliation for WD-WF-CAT-001 / WD-REF-001.

This PR's unique deltas (_da_components.py reader, the agentbuilder connectivity mock, and ~1,765 lines of removed topic/catalog test scaffolding) all serve the superseded bot-components approach and have no consumer once main's approach is in place. Nothing to salvage.

Same disposition as #328.

Dawn Jeong (daeunJe0ng) added a commit to daeunJe0ng/Employee-Self-Service-Agent-Developer-Kit that referenced this pull request Sep 25, 2026
…icrosoft#304 review nit)

Resolves the PR microsoft#304 blocker by restoring DV-CONN-001 to its original
Dataverse meaning (it now equals main) and dropping the wrong-layer DA
Workday SOAP connection check that microsoft#304 had bolted onto that ID.

The Workday shared_workdaysoap reference is flow/solution-scoped and never
surfaces in the DA bot-components (connectionReferenceChanges) layer that the
dropped check read, so it would have FAILED "not found" on every real DA GA
agent -- the same wrong-layer defect that closed microsoft#318 and microsoft#328. The correct-
layer DA Workday-connection check is deferred to follow-up work.

Kept: harden _bot_connection_references to raise on a genuinely malformed
connectionReferenceChanges shape instead of silently skipping it. Its live
consumer, ENV-004, wraps the read in except Exception and degrades to a
warning row, so this is fail-loud without a regression. Absent/null
connectionReference entries (non-connection changes) are still tolerated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d47b6fbe-fc1f-402d-aadb-df1a6695da56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant