Skip to content

flightcheck: add Workday DA profile + versioned result contract - #327

Open
Dawn Jeong (daeunJe0ng) wants to merge 9 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-familyb-7865455
Open

Dawn Jeong (daeunJe0ng) wants to merge 9 commits into
microsoft:mainfrom
daeunJe0ng:users/dawnjeong/flightcheck-da-familyb-7865455

Conversation

@daeunJe0ng

@daeunJe0ng Dawn Jeong (daeunJe0ng) commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the Workday Declarative Agent (DA) validation contract layer that the re-point PRs (#304, #314, #315, #316, #317, #318) plug into. Where those PRs re-point individual checks onto the DA components/ALM APIs, this PR gives Connect a callable profile + versioned, stable result contract so it can run a scoped set of checkpoints and consume the outcome without scraping the HTML report.

Tracks user story AB#7865450 (tasks AB#7865455 profile + result contract, AB#7865460 fixtures + contract tests).

Why this is needed

The re-point PRs make each check talk to the DA components/ALM APIs, but Connect still had no supported way to:

  • ask FlightCheck for just the checkpoints relevant to one Workday DA stage (setup, dataverse, post-runtime, etc.), or
  • read results as a stable machine contract instead of parsing report HTML, or
  • distinguish "an essential platform capability was unavailable" (a release gate) from a benign "not applicable" skip.

What changed

  • runner.py — new Status.BLOCKED (distinct from Skipped): an unavailable essential capability now buckets into Action required and forces overall = NOT_READY, so it can never render as a success-shaped no-op (AC3). Added stable per-result contract fields (severity, automation_type, remediation_id, non-secret evidence) with backward-compatible defaults, and a versioned flightcheck.result.v1 payload (versioned_run_result_to_dict) nested in results.json alongside the existing snake_case keys.
  • registry.py — the eight callable profiles from the story (workday-da:setup-readiness, :dataverse-ready, :external-prerequisites, :post-runtime, :post-connection, :post-agent-wiring, :final, workday-legacy:diagnostic) plus profile resolution/planning helpers. WD-CONN-013 (agent connection OBO parameter sharing) resolves through the WD-CONN checkpoint group to the already-implemented check in checks/workday.py — no placeholder stub. profile_requirements now raises RegistryError on a profile that declares no checkpoints (instead of an IndexError).
  • cli.py — --profile and --validation-realm flags that run a profile and emit the versioned contract. ValidationContext.realm is required so a Dev run can't be mistaken for Test/Prod readiness; --validation-realm help + the runtime error hint now state the realm can also come from a "realm" key in .local/config.json.
  • tests/flightcheck/contract/ — seven non-secret connection-state fixtures (absent, partial, legacy, healthy, degraded, invoker, embedded) + contract tests covering versioned JSON, blocked bucketing, profile resolution, WD-CONN-013 resolving to the real check, fixture coverage, and empty-profile rejection.

Design notes / scope

Testing

  • Full suite after changes: pytest tests/flightcheck -q = 1232 passed.
  • Contract tests: pytest tests/flightcheck/contract/test_workday_da_profiles.py -q passes (incl. the new test_profile_requirements_rejects_empty_profile).
  • ruff check on touched files = passed.
  • Pre-existing failures in tests/scripts (installer) and tests/setup/test_da_setup_router.py reproduce on a clean upstream/main, so they are unrelated to this PR.

Live E2E validation (read-only)

Ran the workday-da:post-connection profile end to end against the real flow-based Workday DA environment (WD-Validate, orgba4e06cd, env 59e5511c, realm=prod), which carries a correctly configured Workday connection.

  • The versioned flightcheck.result.v1 contract emitted correctly: profile, profileCheckpoints, validationContext (non-secret identifiers only), camelCase counts, and per-row severity/automationType/remediationId/evidence.
  • The Dataverse-read Workday checks PASS on a correctly configured env: WD-PKG-001 (package fingerprint), WD-CONN-012, WD-CONN-013 (agent connection OBO parameter sharing), and DV-CONN-001.
  • Result: 4 passed / 1 fail / 1 manual / 1 not-configured. The non-passes reflect env/local-config state, not code defects: WD-REST-001 (Workday REST base URL not yet captured), WD-CONN-AUTH-001 (manual portal confirm of Entra SSO vs basic auth), and WD-REST-002 (reads the local .local/agents/ topic folder, which is bound to the local checkout's agent config).

This exercises the full profile path (auth → scoped checkpoints → bucketing → versioned results.json + report.html) against live Workday connection data.

Stacking note

Branched off the foundation reader PR (#326, now merged). This PR's net delta over main is the profile + versioned contract layer across cli.py/registry.py/runner.py plus the contract tests.


Sync 2026-09-25: Merged latest upstream/main (merge a0ddbf1f). One conflict in flightcheck/cli.py (both sides added a helper at the same spot): kept BOTH _validation_context_from_args (this PR) and _resolve_environment_ring (main). Follow-up commit 15039c0c addresses review nits (empty-profile guard + --validation-realm help/error text) with a regression test. tests/flightcheck = 1232 passed; ruff clean. #327 is MERGEABLE (only the review gate remains).

…test (AB#7852506, AB#7852495, AB#7852511)

Extract the shared minimalBots components connection-reference reader into
checks/_da_connection_refs.py as the single source that DV-CONN-001 (active
agent), ENV-004 (environment-wide, de-duped by logical name) and the Workday
shared-parameter checks (WD-ENV-001/WD-REST-001) all read through, so the
per-check re-point PRs stop adding divergent copies of it.

The module is the superset of PR microsoft#317's reader plus a public
read_all_agents_connection_references() that de-dupes by connection-reference
logical name for ENV-004. Brings the validated/documented mock builders the
checks need and a direct pure-logic contract test (18 cases) covering
normalization, None-gating (SKIP), fail-loudly ValueError, JSON-string
sharedConnectionParameters parsing, and the Workday shared-parameter sweep.

Foundation for the 6 DA re-point PRs to rebase onto (single shared reader,
no add/add collision). Full flightcheck suite: 1142 passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
…eholder stub (already implemented)

WD-CONN-013 (agent connection OBO parameter sharing) already exists as a
fully implemented, tested check in checks/workday.py (emitted by
run_workday_checks). The initial Family B draft registered it as a
'Profile Stubs' placeholder that double-emitted a not-implemented Manual
row and shadowed the real result. Removed the stub mechanism entirely;
WD-CONN-013 now resolves via the WD-CONN family to the real Workday
category. Updated contract tests to assert real resolution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
@daeunJe0ng Dawn Jeong (daeunJe0ng) changed the title flightcheck: add Workday DA profile + versioned result contract (foundation for Connect, AB#7865455, AB#7865460) flightcheck: add Workday DA profile + versioned result contract Sep 24, 2026
…ghtcheck-da-familyb-7865455

# Conflicts:
#	solutions/ess-maker-skills/scripts/flightcheck/cli.py
…ghtcheck-da-familyb-7865455

# Conflicts:
#	solutions/ess-maker-skills/scripts/flightcheck/cli.py
… (address PR microsoft#327 review nits)

profile_requirements now raises RegistryError for a profile with no checkpoint members instead of an opaque IndexError on checkpoint_ids[0]; adds a regression test. --validation-realm help/error text now states the .local/config.json realm fallback so it no longer claims the flag is strictly required.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d47b6fbe-fc1f-402d-aadb-df1a6695da56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant