Skip to content

fix(flightcheck): explain incomplete setup readiness - #351

Merged
nkemms merged 6 commits into
mainfrom
users/nkemd/7947735-strict-setup-flightcheck-routing
Sep 26, 2026
Merged

nkemms merged 6 commits into
mainfrom
users/nkemd/7947735-strict-setup-flightcheck-routing

Conversation

@nkemms

@nkemms nkemms commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

FlightCheck remains gated until canonical setup readiness is complete. When setup is incomplete, routing now preserves the recorded readiness blockers instead of replacing them with generic /setup guidance. Foundation Setup also reconciles selected agents across native MinimalBot and Dataverse storage before entering DA-only work.

Description

The global router defines one ordered FlightCheck entry contract for standalone installations, completed canonical setup, materialized workspaces with readiness outstanding, and workspaces that still require preparation. The FlightCheck prompt and skill delegate to that contract rather than carrying separate gates.

For materialized workspaces, routing collects and deduplicates recorded failure_causes, maps internal steps to maker-facing readiness labels, and preserves service status, error codes, request IDs, and remediation. A neutral readiness template covers incomplete steps without a recorded cause.

Foundation Setup independently probes the native MinimalBot and Dataverse exact-ID endpoints. The URL backend only determines probe order. Product family comes from supported schema prefixes, while existence, backend, access, supportability, and ALM enrollment remain separate decisions.

ADO Bug 7947735.

Fixes

  • Preserve all recorded setup blockers instead of replacing them with a generic /setup redirect.
  • Centralize FlightCheck setup-first routing across the global router, prompt, and skill.
  • Probe both native MinimalBot and Dataverse identity stores, preserving dual-store and backend-mismatch evidence.
  • Recognize classic CA, DA-Preview, and DA-GA suffix variants through the two supported ESS schema prefixes.
  • Route legacy agents to the compatible main-ca kit and native DA-GA agents to the existing DA setup path.
  • Stop safely for custom agents, dual-store ambiguity, and Dataverse-backed DA local authoring.
  • Distinguish exact dual-store absence from authentication, authorization, service uncertainty, and missing ALM enrollment.
  • Treat a native realms 404 as not-enrolled, not as a missing agent.
  • Preserve service error details and request IDs in reconciliation results.
  • Harden recovery UX with exact message blocks, back-safe choices, no operational narration, and no preselected compatible-kit action.

Outcome

FlightCheck requests follow the strict setup-first model while keeping readiness failures visible and actionable. Foundation Setup identifies the selected agent without assuming its storage backend, enters only supported authoring paths, and stops with explicit recovery choices when evidence is ambiguous or unsupported.

Testing

Test flightcheck-strict-setup-routing

Assertion Baseline Patched
Incomplete setup with recorded blockers Generic /setup welcome replaces blocker details All recorded causes appear under maker-facing readiness labels
Incomplete setup without a recorded cause Redirect provides no current readiness item First incomplete item uses a neutral templated explanation
Routing ownership Global gate, prompt, and skill can diverge Prompt and skill delegate to one ordered entry contract
Agent backend lookup URL hint or one storage path determines identity Both exact-ID stores are probed and authoritative evidence is preserved
Missing agent detection Lookup failures can become not-found Only exact 404 from both accessible stores is definitive absence
ALM enrollment Realms 404 can imply a missing agent Existing agent is reported as not-enrolled
Product-family routing Exact schemas or native assumptions drive setup Supported prefixes route legacy and DA-GA variants explicitly
Unsupported recovery Unsupported agents can fall through DA setup Setup stops with explicit agent, environment, and back choices
Compatible-kit prompt Operational narration and apparent default selection Standalone exact message and initially unset choice control
  • 108 focused agent-reconciliation and setup-routing tests pass.
  • Ruff and Git diff validation pass for the changed implementation and tests.

Nkem Dockery and others added 5 commits September 25, 2026 11:35
Keep FlightCheck gated on canonical setup readiness while surfacing the recorded setup blockers that require attention. Centralize the ordered entry contract and use explicit maker-facing message templates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: eff50b3c-427f-453a-8f4a-49848ad02602
Carry confirmed setup intent through FlightCheck recovery and remove redundant confirmation prompts. End completed setup with stable landing-page, connection, and menu entry points.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace nested setup cancellation dead ends with explicit back routes while retaining cancellation on entry menus. Preserve or reset account context according to observed access evidence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Probe native MinimalBot and Dataverse identities independently, classify supported ESS families by schema prefix, and keep ALM enrollment distinct from agent existence.

Stop unsupported or ambiguous agents with explicit recovery UX while preserving authoritative access and failure evidence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c820c60f-79a3-4b3d-ac32-3a309689c1ca

@amilandi amilandi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Traced the dual-store probe, router contract, decision matrix, and UX changes — no issues at any severity.

Confirmed:

  • probe_native_identity and probe_dataverse_identity are fully independent CLI entries in a mutually-exclusive group; regression test asserts the native probe never touches the dataverse resolver.
  • HTTP failure taxonomy (401 → auth-required, 403 → access-denied, 404 → not-found, other → uncertain) mapped uniformly for both AgentBuilderHTTPError and requests.HTTPError; parametrized coverage present.
  • _exception_evidence preserves status_code / error_code / request_id through a bounded __cause__ / __context__ chain; no bare-except swallows evidence.
  • _resolve_dataverse_url sentinel is now raised as APIError and caught into a stage: "environment-resolution" failure — 401/403 are preserved (asserted by test_environment_resolution_preserves_permission_failure). Dropped SystemExit / AuthExpiredError are safe on this path (MSAL sys.exit is import-time only; AuthExpiredError is only raised by dataverse_get / query_all, neither of which runs here).
  • The four FlightCheck router states in copilot-instructions.md are gated by "Apply the first matching state" and are exhaustive; flightcheck.prompt.md and flightcheck/SKILL.md delegate to the entry contract rather than carrying parallel gates.
  • setup_existing_da.inspect_agent_route converts only 404 → almEnrollment: "not-enrolled"; 401/403/5xx re-raise. STUDIO_RING_BY_HOST untouched.
  • Every da-*.md recovery prompt replaces **Cancel setup** with **Go back** and documents the back-target; destructive actions and the compatible-kit selection retain explicit "do not preselect" clauses.
  • Schema classification splits msdyn_... (compatible-kit) vs gptagent_... (DA-GA), casefold-normalized and tested.

Thanks for the strict-setup rework — the "one ordered entry contract" refactor is a nice cleanup.

@nkemms
nkemms merged commit bf9ff8e into main Sep 26, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants