Skip to content

flightcheck: re-point PUB-001 + PUB-002 to Declarative Agent ALM export/import API - #315

Open
Dawn Jeong (daeunJe0ng) wants to merge 10 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-pub-7852507
Open

Dawn Jeong (daeunJe0ng) wants to merge 10 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-pub-7852507

Conversation

@daeunJe0ng

@daeunJe0ng Dawn Jeong (daeunJe0ng) commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Re-points FlightCheck publishing checks PUB-001 (AB#7852507) and PUB-002 (AB#7852508) from the old managed-solution/manual framing to the Declarative Agent AgentBuilder ALM export/import APIs.

This stacks on the shared reader in #326 and on #304. Merge order: #326 -> #304 -> this PR.

What changed

  • PUB-001 now calls AgentBuilderClient.export_package(...) and validates the returned package as a real zip archive by reading the central directory and running ZipFile.testzip() CRC validation.
  • PUB-002 now uses AgentBuilderClient.export_package(...) followed by AgentBuilderClient.import_package(...) when the caller explicitly enables the import probe.
  • 4003 ALM-not-enrolled responses are reported as defined FlightCheck failures with remediation instead of crashing.
  • Non-4003 export/import HTTP errors are reported as warnings with actionable remediation instead of crashing.
  • PUB-001 and PUB-002 are registered as AgentBuilder-backed single checkpoints with requires_dataverse_endpoint=False.
  • Native no-Dataverse FlightCheck now wires the publishing scope to AgentBuilder authentication.
  • Tests extend the AgentBuilder mock helper with a real in-memory zip export package and a valid import result shape.

Design rationale

PUB-001 is safe to run as a read-only export check, so it runs when the AgentBuilder client and botId are available.

PUB-002 is a mutating import path. It is intentionally gated behind an explicit runner opt-in so a normal FlightCheck run stays read-only and cannot create an agent in the current environment by surprise. The mocked tests cover the import behavior without running any live import.

The export archive validation deliberately does not use a ZIP magic-byte prefix check. It opens the archive through zipfile, verifies the central directory is readable, requires at least one entry, rejects unsafe archive entry paths, and runs CRC validation with testzip().

Testing

  • python -m pytest tests\flightcheck -q - 1137 passed.
  • python -m ruff check solutions\ess-maker-skills\scripts - passed.

Live DA-environment validation (partially complete)

Live run 2026-09-22 against ESS GA env 258bd301 (prod ring; agent 06ae0814 is not ALM-enrolled): the not-opted-in FAIL path is confirmed against a real DA agent (checked below). The opted-in PASS path, PUB-002 import, and the non-4003 WARNING path remain unconfirmed (no reachable ALM-opted-in / throwaway DA agent this run).

To be checked once a real Application Lifecycle Management (ALM) opted-in Declarative Agent is reachable. Blocked on a published DA agent enrolled in ALM; these ALM export/import reads use the prod-audience AgentBuilder token and do NOT depend on the TIP2 (Test-In-Production ring 2) auth issue. Each box is marked when the scenario is confirmed on a live DA env.

  • PUB-001 export on an ALM-opted-in DA agent: export_package returns a real zip; central directory readable, at least one entry, testzip() CRC passes -> PASSED.
  • PUB-001 on an agent not opted into ALM (export returns 4003) -> defined FAILED verdict with ALM-enrollment remediation, no crash.
  • PUB-001 on a non-4003 export HTTP error -> WARNING with actionable remediation (fail loudly, no false PASS).
  • PUB-002 with the import probe explicitly enabled on a throwaway ALM-enabled env: export_package then import_package succeeds -> PASSED.
  • PUB-002 import maps a 409 / DuplicateItem response to the defined FAILED verdict.
  • PUB-002 stays SKIPPED on a normal run (import probe not opted in), so a default run cannot create an agent in the target env by surprise.
  • No-AgentBuilder-client and no-active-botId paths SKIP instead of failing.

Work items

  • AB#7852507
  • AB#7852508

Sync update (2026-09-24)

  • Merged latest upstream/main into this branch (no rebase, no force-push). Clean auto-merge, no conflicts.
  • Verified: ruff clean across scripts/flightcheck; full tests/flightcheck suite 1223 passed.
  • Mergeable, blocked only on required review.

@daeunJe0ng
Dawn Jeong (daeunJe0ng) marked this pull request as ready for review September 23, 2026 05:26
…test (AB#7852506, AB#7852495, AB#7852511)

Extract the shared minimalBots components connection-reference reader into
checks/_da_connection_refs.py as the single source that DV-CONN-001 (active
agent), ENV-004 (environment-wide, de-duped by logical name) and the Workday
shared-parameter checks (WD-ENV-001/WD-REST-001) all read through, so the
per-check re-point PRs stop adding divergent copies of it.

The module is the superset of PR microsoft#317's reader plus a public
read_all_agents_connection_references() that de-dupes by connection-reference
logical name for ENV-004. Brings the validated/documented mock builders the
checks need and a direct pure-logic contract test (18 cases) covering
normalization, None-gating (SKIP), fail-loudly ValueError, JSON-string
sharedConnectionParameters parsing, and the Workday shared-parameter sweep.

Foundation for the 6 DA re-point PRs to rebase onto (single shared reader,
no add/add collision). Full flightcheck suite: 1142 passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
… (7852506)

Read the Workday SOAP connection reference from the Declarative Agent
minimalBots components API (runner.agentbuilder.fetch_components) instead of the
Custom Agent Dataverse connectionreferences query, so DV-CONN-001 works against
DA-GA agents that no longer expose Dataverse. Verdict simplifies to found +
bound (connectionId present); the Dataverse-only statuscode and multiple-ref
branches are dropped since the components shape carries neither. BAP owner echo
is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Distinguish a genuinely-absent connectionReferenceChanges (missing key -> no
references -> FAILED not-found) from a present-but-non-list payload (a shape we
do not understand -> raise ValueError so the dispatcher degrades DV-CONN-001 to
a WARNING). Mirrors native_agent._connection_references, the shipped precedent
this check re-uses; the prior code collapsed both to [] and could report a
confident "reference not found" on an unparseable 200 response. Adds a
malformed-changeset test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…list)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
@daeunJe0ng
Dawn Jeong (daeunJe0ng) force-pushed the users/dawnjeong/flightcheck-da-pub-7852507 branch from 3b3c2e9 to 283a248 Compare September 24, 2026 00:12
…B#7852508)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
…afety-gate test (PUB-001/PUB-002)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8246cd2c-37d0-4000-8fe3-fa9b082669e0
@daeunJe0ng Dawn Jeong (daeunJe0ng) changed the title flightcheck: re-point PUB-001 + PUB-002 to Declarative Agent ALM export/import API (AB#7852507, AB#7852508) flightcheck: re-point PUB-001 + PUB-002 to Declarative Agent ALM export/import API Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant