flightcheck: add Workday DA profile + versioned result contract - #327
Open
Dawn Jeong (daeunJe0ng) wants to merge 9 commits into
Open
Dawn Jeong (daeunJe0ng) wants to merge 9 commits into
Dawn Jeong (daeunJe0ng) wants to merge 9 commits into
Conversation
…test (AB#7852506, AB#7852495, AB#7852511) Extract the shared minimalBots components connection-reference reader into checks/_da_connection_refs.py as the single source that DV-CONN-001 (active agent), ENV-004 (environment-wide, de-duped by logical name) and the Workday shared-parameter checks (WD-ENV-001/WD-REST-001) all read through, so the per-check re-point PRs stop adding divergent copies of it. The module is the superset of PR microsoft#317's reader plus a public read_all_agents_connection_references() that de-dupes by connection-reference logical name for ENV-004. Brings the validated/documented mock builders the checks need and a direct pure-logic contract test (18 cases) covering normalization, None-gating (SKIP), fail-loudly ValueError, JSON-string sharedConnectionParameters parsing, and the Workday shared-parameter sweep. Foundation for the 6 DA re-point PRs to rebase onto (single shared reader, no add/add collision). Full flightcheck suite: 1142 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
…eholder stub (already implemented) WD-CONN-013 (agent connection OBO parameter sharing) already exists as a fully implemented, tested check in checks/workday.py (emitted by run_workday_checks). The initial Family B draft registered it as a 'Profile Stubs' placeholder that double-emitted a not-implemented Manual row and shadowed the real result. Removed the stub mechanism entirely; WD-CONN-013 now resolves via the WD-CONN family to the real Workday category. Updated contract tests to assert real resolution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e25cf992-09c9-49b8-9971-2a19cb2fcb05
…ghtcheck-da-familyb-7865455 # Conflicts: # solutions/ess-maker-skills/scripts/flightcheck/cli.py
…ghtcheck-da-familyb-7865455 # Conflicts: # solutions/ess-maker-skills/scripts/flightcheck/cli.py
… (address PR microsoft#327 review nits) profile_requirements now raises RegistryError for a profile with no checkpoint members instead of an opaque IndexError on checkpoint_ids[0]; adds a regression test. --validation-realm help/error text now states the .local/config.json realm fallback so it no longer claims the flag is strictly required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d47b6fbe-fc1f-402d-aadb-df1a6695da56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Workday Declarative Agent (DA) validation contract layer that the re-point PRs (#304, #314, #315, #316, #317, #318) plug into. Where those PRs re-point individual checks onto the DA components/ALM APIs, this PR gives Connect a callable profile + versioned, stable result contract so it can run a scoped set of checkpoints and consume the outcome without scraping the HTML report.
Tracks user story AB#7865450 (tasks AB#7865455 profile + result contract, AB#7865460 fixtures + contract tests).
Why this is needed
The re-point PRs make each check talk to the DA components/ALM APIs, but Connect still had no supported way to:
What changed
runner.py— newStatus.BLOCKED(distinct fromSkipped): an unavailable essential capability now buckets into Action required and forcesoverall = NOT_READY, so it can never render as a success-shaped no-op (AC3). Added stable per-result contract fields (severity,automation_type,remediation_id, non-secretevidence) with backward-compatible defaults, and a versionedflightcheck.result.v1payload (versioned_run_result_to_dict) nested inresults.jsonalongside the existing snake_case keys.registry.py— the eight callable profiles from the story (workday-da:setup-readiness,:dataverse-ready,:external-prerequisites,:post-runtime,:post-connection,:post-agent-wiring,:final,workday-legacy:diagnostic) plus profile resolution/planning helpers.WD-CONN-013(agent connection OBO parameter sharing) resolves through theWD-CONNcheckpoint group to the already-implemented check inchecks/workday.py— no placeholder stub.profile_requirementsnow raisesRegistryErroron a profile that declares no checkpoints (instead of anIndexError).cli.py—--profileand--validation-realmflags that run a profile and emit the versioned contract.ValidationContext.realmis required so a Dev run can't be mistaken for Test/Prod readiness;--validation-realmhelp + the runtime error hint now state the realm can also come from a"realm"key in.local/config.json.tests/flightcheck/contract/— seven non-secret connection-state fixtures (absent, partial, legacy, healthy, degraded, invoker, embedded) + contract tests covering versioned JSON, blocked bucketing, profile resolution, WD-CONN-013 resolving to the real check, fixture coverage, and empty-profile rejection.Design notes / scope
Skippedbehavior is intentionally unchanged; only newly essential-unavailable rows useBlocked. Existing report UX and tests stay valid.contractblock with aschemaVersionto negotiate future additive changes.WD-REST/WD-ENV/WD-WF-CAT/WD-REFre-points (flightcheck: fail loud on malformed DA connection-ref payloads (reader hardening) #304, flightcheck: re-point ENV-004 to Declarative Agent components API #314, flightcheck: re-point PUB-001 + PUB-002 to Declarative Agent ALM export/import API #315, flightcheck: re-point ESS-SOLN-001 to Declarative Agent GRS/ALM configure API #316, flightcheck: re-point WD-REST-001 + WD-ENV-001 to Declarative Agent components API #317, flightcheck: re-point WD-WF-CAT-001 + WD-REF-001 to Declarative Agent components API #318), the WD-PKG-001 fingerprint fix (flightcheck: exclude per-agent DA connection refs from WD-PKG-001 fingerprint #328), and the additional checkpoint-group work under AB#7865456 / AB#7865457 / AB#7865458.Testing
pytest tests/flightcheck -q= 1232 passed.pytest tests/flightcheck/contract/test_workday_da_profiles.py -qpasses (incl. the newtest_profile_requirements_rejects_empty_profile).ruff checkon touched files = passed.tests/scripts(installer) andtests/setup/test_da_setup_router.pyreproduce on a cleanupstream/main, so they are unrelated to this PR.Live E2E validation (read-only)
Ran the
workday-da:post-connectionprofile end to end against the real flow-based Workday DA environment (WD-Validate,orgba4e06cd, env59e5511c, realm=prod), which carries a correctly configured Workday connection.flightcheck.result.v1contract emitted correctly:profile,profileCheckpoints,validationContext(non-secret identifiers only), camelCasecounts, and per-rowseverity/automationType/remediationId/evidence.WD-PKG-001(package fingerprint),WD-CONN-012,WD-CONN-013(agent connection OBO parameter sharing), andDV-CONN-001.WD-REST-001(Workday REST base URL not yet captured),WD-CONN-AUTH-001(manual portal confirm of Entra SSO vs basic auth), andWD-REST-002(reads the local.local/agents/topic folder, which is bound to the local checkout's agent config).This exercises the full profile path (auth → scoped checkpoints → bucketing → versioned
results.json+report.html) against live Workday connection data.Stacking note
Branched off the foundation reader PR (#326, now merged). This PR's net delta over
mainis the profile + versioned contract layer acrosscli.py/registry.py/runner.pyplus the contract tests.Sync 2026-09-25: Merged latest
upstream/main(mergea0ddbf1f). One conflict inflightcheck/cli.py(both sides added a helper at the same spot): kept BOTH_validation_context_from_args(this PR) and_resolve_environment_ring(main). Follow-up commit15039c0caddresses review nits (empty-profile guard +--validation-realmhelp/error text) with a regression test.tests/flightcheck= 1232 passed; ruff clean. #327 isMERGEABLE(only the review gate remains).