flightcheck: re-point WD-WF-CAT-001 + WD-REF-001 to Declarative Agent components API - #318
Dawn Jeong (daeunJe0ng) wants to merge 10 commits into
Conversation
…test (AB#7852506, AB#7852495, AB#7852511) Extract the shared minimalBots components connection-reference reader into checks/_da_connection_refs.py as the single source that DV-CONN-001 (active agent), ENV-004 (environment-wide, de-duped by logical name) and the Workday shared-parameter checks (WD-ENV-001/WD-REST-001) all read through, so the per-check re-point PRs stop adding divergent copies of it. The module is the superset of PR microsoft#317's reader plus a public read_all_agents_connection_references() that de-dupes by connection-reference logical name for ENV-004. Brings the validated/documented mock builders the checks need and a direct pure-logic contract test (18 cases) covering normalization, None-gating (SKIP), fail-loudly ValueError, JSON-string sharedConnectionParameters parsing, and the Workday shared-parameter sweep. Foundation for the 6 DA re-point PRs to rebase onto (single shared reader, no add/add collision). Full flightcheck suite: 1142 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
… (7852506) Read the Workday SOAP connection reference from the Declarative Agent minimalBots components API (runner.agentbuilder.fetch_components) instead of the Custom Agent Dataverse connectionreferences query, so DV-CONN-001 works against DA-GA agents that no longer expose Dataverse. Verdict simplifies to found + bound (connectionId present); the Dataverse-only statuscode and multiple-ref branches are dropped since the components shape carries neither. BAP owner echo is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Distinguish a genuinely-absent connectionReferenceChanges (missing key -> no references -> FAILED not-found) from a present-but-non-list payload (a shape we do not understand -> raise ValueError so the dispatcher degrades DV-CONN-001 to a WARNING). Mirrors native_agent._connection_references, the shipped precedent this check re-uses; the prior code collapsed both to [] and could report a confident "reference not found" on an unparseable 200 response. Adds a malformed-changeset test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…list) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
e0a173f to
cc0895f
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
…r DA re-point (WD-REF-001, WD-WF-CAT-001) The WD-REF-001 and WD-WF-CAT-001 rewrites to DA structural enumeration left 8 module-level items with no remaining call sites: _WD_REF_KEY_LABELS, _WD_REF_SUPPORTED_RE, _WD_REF_REQUESTED_RE, _ref_key_label, _extract_requested_reference_keys, _extract_supported_reference_keys (old Dataverse reference-key reconciliation), plus _format_unknown_scenarios and _WD_WF_CAT_CHECKLIST (old catalog checklist output). Removing them keeps the check surface honest; git history retains them if the pending catalog work (US 7792327) restores semantic classification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
Update the WD-WF-CAT-LINK remediation to point at the current botComponentChanges-based topic inventory and refresh the validation matrix for WD-WF-CAT-001 plus the missing WD-REF-001 structural row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
# Conflicts: # tests/mocks/agentbuilder_connectivity.py
|
Closing as superseded. While this PR was queued,
This PR's unique deltas ( Same disposition as #328. |
…icrosoft#304 review nit) Resolves the PR microsoft#304 blocker by restoring DV-CONN-001 to its original Dataverse meaning (it now equals main) and dropping the wrong-layer DA Workday SOAP connection check that microsoft#304 had bolted onto that ID. The Workday shared_workdaysoap reference is flow/solution-scoped and never surfaces in the DA bot-components (connectionReferenceChanges) layer that the dropped check read, so it would have FAILED "not found" on every real DA GA agent -- the same wrong-layer defect that closed microsoft#318 and microsoft#328. The correct- layer DA Workday-connection check is deferred to follow-up work. Kept: harden _bot_connection_references to raise on a genuinely malformed connectionReferenceChanges shape instead of silently skipping it. Its live consumer, ENV-004, wraps the read in except Exception and degrades to a warning row, so this is fail-loud without a regression. Absent/null connectionReference entries (non-connection changes) are still tolerated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d47b6fbe-fc1f-402d-aadb-df1a6695da56
Summary
Re-points two Workday FlightCheck checks to the Declarative Agent components API:
WD-WF-CAT-001: enumerate Workday topic components frombotComponentChangesby matching schema names that contain.topic.Workday.WD-REF-001: enumerate Workday reference-data structure frombotComponentChangesby matching.variable.*LookupTablevariables plusWorkdaySystemGetReferenceDataandWorkdaySystemRefreshReferenceDatatopics.This PR stacks on the shared reader in #326 and on #304. Merge order: #326 -> #304 -> this PR. Until they merge, the diff shows their commits.
What changed
botComponentChangeslist from the validated components payload viarunner.agentbuilder.fetch_components(agent_id).WD-REF-001to produce a deterministic structural verdict without Dataverse:botComponentChangesshape.WD-WF-CAT-001to produce a deterministic Workday topic inventory from Declarative Agent topic component schema names.WD-REF-001andWD-WF-CAT-001for single-checkpoint execution withAGENTBUILDERonly andrequires_dataverse_endpoint=False.tests/mocks/agentbuilder_connectivity.pyand extended that mock with abotComponentChangesbuilder._WD_REF_KEY_LABELS,_WD_REF_SUPPORTED_RE,_WD_REF_REQUESTED_RE,_ref_key_label,_extract_requested_reference_keys,_extract_supported_reference_keys,_format_unknown_scenarios,_WD_WF_CAT_CHECKLIST): the old Dataverse reference-key reconciliation and catalog-checklist output that the structural DA verdict no longer calls (git history retains them if US 7792327 restores semantic classification).Design rationale
The source of truth is the AgentBuilder components payload returned by
AgentBuilderClient.fetch_components(agent_id).botComponentChangesis a top-level list in the same validated payload used by #304 forconnectionReferenceChanges.These checks are intentionally structural and flowless. They verify that the Declarative Agent contains the expected Workday topic and reference-data component shapes without depending on the old Dataverse
botcomponentstable, workspace-local extracted files, or cloud flow inventory.The shipped-vs-custom classification catalog and final reference-data key rules are external inputs owned by Skantha's follow-up work, US 7792327. This PR does not block on that catalog. Classification remains gated out, and the result text explicitly notes that the current verdict is structural enumeration only pending US 7792327.
Testing
python -m pytest tests\flightcheck -q1114 passed in 5.62spython -m ruff check solutions\ess-maker-skills\scriptsAll checks passed!Live DA-environment validation (partially complete)
Live run 2026-09-22 against ESS GA env
258bd301(prod ring; non-Workday agent): the shared DA component reader fetchedbotComponentChangesfrom a real DA agent; with no Workday components present, WD-REF-001 returned the documented FAIL (checked below) and WD-WF-CAT-001 produced a deterministic empty Workday-topic inventory (0.topic.Workday*). The Workday-present PASS paths need a Workday-bound DA agent (none reachable this run).To be checked once a published Workday-bound Declarative Agent is provisioned in the test env. Blocked on that agent; these are read-only components reads on the prod-audience AgentBuilder token and do NOT depend on the TIP2 (Test-In-Production ring 2) auth issue. Structural enumeration only until the classification catalog lands (US 7792327). Each box is marked when the scenario is confirmed on a live DA env.
.variable.*LookupTablevariables plusWorkdaySystemGetReferenceDataandWorkdaySystemRefreshReferenceDatatopics -> PASSED..topic.Workday*schema names inbotComponentChanges.botComponentChangesshape -> WARNING (fail loudly).Follow-up