From 86765cb871350b7536985ca23411b9bd31b94fbb Mon Sep 17 00:00:00 2001 From: Nkem Dockery Date: Fri, 25 Sep 2026 11:35:30 -0700 Subject: [PATCH 1/4] fix(flightcheck): explain incomplete setup readiness Keep FlightCheck gated on canonical setup readiness while surfacing the recorded setup blockers that require attention. Centralize the ordered entry contract and use explicit maker-facing message templates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eff50b3c-427f-453a-8f4a-49848ad02602 --- .../.github/copilot-instructions.md | 93 +++++++++++++++++-- .../.github/prompts/flightcheck.prompt.md | 18 +--- .../src/skills/flightcheck/SKILL.md | 18 ++-- tests/setup/test_da_setup_router.py | 65 +++++++++++-- 4 files changed, 149 insertions(+), 45 deletions(-) diff --git a/solutions/ess-maker-skills/.github/copilot-instructions.md b/solutions/ess-maker-skills/.github/copilot-instructions.md index bb50773e6..fc8fbdacc 100644 --- a/solutions/ess-maker-skills/.github/copilot-instructions.md +++ b/solutions/ess-maker-skills/.github/copilot-instructions.md @@ -21,10 +21,14 @@ files. Do not attempt any customization work. Do not answer questions about ESS. Do not list your capabilities. Do not greet the user with a menu of options. Do not say "hello" or introduce yourself. -Respond with ONLY this exact message and nothing else: +Use this setup-required Message block and finish the request: -> Hey! Welcome to the ESS Maker Kit. Before we dive in, I need to set up -> your environment. Type `/setup` to get started — it only takes a couple minutes. +**Message:** + +Hey! Welcome to the ESS Maker Kit. Before we dive in, I need to set up your +environment. Type `/setup` to get started — it only takes a couple minutes. + +**End message.** **Exceptions:** @@ -46,10 +50,8 @@ Respond with ONLY this exact message and nothing else: requests such as "edit the testsets" and "change an expected response." That skill discovers workspace-level sets without setup and agent-owned sets when configuration is available. Deleting deployed sets still requires setup. -- If the user typed `/flightcheck`, read `.local/config.json`. If it has - `flightCheckOnly: true`, proceed with `src/skills/flightcheck/SKILL.md`. - This exception applies only to `/flightcheck`; every other command remains - gated. +- If the user typed `/flightcheck` or explicitly asked to validate setup or + environment readiness, follow the **FlightCheck entry contract** below. - If the user typed `/connect` or `/connect-workday`, allow the command after **local workspace materialization**, even when runtime `connect_ready` is false. Require @@ -58,13 +60,84 @@ Respond with ONLY this exact message and nothing else: workspace evidence plus `steps.SETUP-07.state: "done"`. Connector readiness is intentionally not a prerequisite because `/connect` is the workflow that resolves product-extension connection gaps. If materialization is incomplete, - show the setup message above and stop. + use the setup-required Message block above and finish the request. **Except for the cases above, this gate applies to ALL user messages** — including "hello", "hi", "help", "what can you do", "I need a topic", "create a workflow", or any other request. -If foundation setup isn't ready, and the user didn't say `/setup`, -show ONLY the welcome message above. No other text. No capabilities list. No greeting. +If foundation setup isn't ready and no exception above applies, +use the setup-required Message block above and finish the request. + +#### FlightCheck entry contract + +Read `.local/config.json` and resolve its `activeAgent` to the canonical agent +whose `agent.workspace_slug` matches. Apply the first matching state: +Message blocks contain the exact maker-facing copy. Replace their template +values with resolved evidence and render only the block contents. + +1. **Standalone FlightCheck:** local config has `flightCheckOnly: true`. + Proceed with `src/skills/flightcheck/SKILL.md`. +2. **Canonical setup ready:** canonical state has `schema_version: 4`, the + matching agent exists, and its `connect_ready` is `true`. Proceed with + `src/skills/flightcheck/SKILL.md`. +3. **Setup readiness outstanding:** canonical state has `schema_version: 4`, + the matching agent has `workspace.folder`, `workspace.agent_path`, and + `steps.SETUP-07.state: "done"`, and canonical readiness is outstanding + (`connect_ready` is false or absent). Build `{READINESS_ISSUES}` from + canonical setup evidence and use the readiness-outstanding Message block + below. +4. **Workspace preparation required:** use the workspace-preparation Message + block below for every remaining state. + +Build `{READINESS_ISSUES}` as follows: + +1. Collect every non-empty `failure_causes` entry from every step whose `state` + is `blocked`, combining exact duplicates. +2. Preserve the recorded service or readiness check, exception cause, status, + error code, request ID, and supported remediation. Translate + implementation-specific exception names into plain language while retaining + their diagnostic meaning. +3. Map each step to its maker-facing label: + - `SETUP-01` — **Target environment** + - `SETUP-02.1` — **Agent access** + - `SETUP-02.2` — **Environment capacity** + - `SETUP-03` — **Editable Dev agent** + - `SETUP-05` — **Connections** + - `SETUP-06` — **Agent content** + - `SETUP-07` — **Local workspace** + Present the bold labels in maker-facing text and keep the canonical setup + IDs internal. +4. Render each collected cause with this readiness issue template: + + - **{READINESS_ITEM}:** {READINESS_DETAIL} + +5. When the recorded blocked-cause collection is empty, render one line for + the first incomplete step. Set `{READINESS_DETAIL}` to + `/setup` will run this readiness check. + +**Message:** + +FlightCheck is available when setup readiness is complete. + +Setup readiness requires attention: + +{READINESS_ISSUES} + +Type `/setup` to refresh the readiness checks and work through each item. + +**End message.** + +Use the following Message block for workspace preparation: + +**Message:** + +FlightCheck is available when setup prepares the local agent workspace. Type +`/setup` to prepare the workspace. + +**End message.** + +After rendering either Message block, the request is complete. `/setup` +performs the next readiness run. ### If canonical setup is ready diff --git a/solutions/ess-maker-skills/.github/prompts/flightcheck.prompt.md b/solutions/ess-maker-skills/.github/prompts/flightcheck.prompt.md index 572416d20..35be8e674 100644 --- a/solutions/ess-maker-skills/.github/prompts/flightcheck.prompt.md +++ b/solutions/ess-maker-skills/.github/prompts/flightcheck.prompt.md @@ -5,20 +5,12 @@ description: "Type Enter to run a pre-deployment readiness check on your ESS age # FlightCheck -**Setup-state check.** Read `.local/setup/config.json` and `.local/config.json`. -If canonical state does not have `schema_version: 4` and an `agents` entry -matching the active workspace slug with `connect_ready: true`, check whether -local config has `flightCheckOnly: true`. If so, proceed without canonical setup -state. Otherwise, show: +**Setup-state check.** Follow the **FlightCheck entry contract** in +`.github/copilot-instructions.md`. Continue here only for its **Standalone +FlightCheck** and **Canonical setup ready** states. -> Welcome to the ESS Maker Kit. Before running this command, type `/setup` -> to set up your environment. - -and STOP. Otherwise proceed. - -This DA-only release supports only the local-files FlightCheck scope. Follow -the skill with scope fixed to `local`. Do not offer or run Dataverse, -integration, prerequisite, or publishing checks. +For an accepted entry state, use the scope selection supported by the active +configuration in `src/skills/flightcheck/SKILL.md`. You are a script executor. Read `src/skills/flightcheck/SKILL.md` and follow it. It will tell you what to do. diff --git a/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md b/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md index abfa5be08..6ade70cc3 100644 --- a/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md +++ b/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md @@ -11,19 +11,13 @@ Do not rephrase, add commentary, or tell the user what tools you are calling. ## Start -Read `.local/config.json` to confirm setup is complete and get the agent context. +The calling prompt applies the **FlightCheck entry contract** in +`.github/copilot-instructions.md`. When this skill is invoked directly, apply +that contract before continuing. -If setup is not complete, show: - -**Message:** - -You need to run `/setup` first before running a readiness check. - -**End message.** - -Stop here. - -If setup is complete, proceed. +Continue with this skill for the contract's **Standalone FlightCheck** and +**Canonical setup ready** states. The contract supplies the complete +maker-facing response for its other states. Treat a config with `releaseLine: "da"` and no `dataverseEndpoint` as a native no-Dataverse agent. Its supported scopes are `full`, `environment`, diff --git a/tests/setup/test_da_setup_router.py b/tests/setup/test_da_setup_router.py index cbfacd8ce..3b1ec3d9a 100644 --- a/tests/setup/test_da_setup_router.py +++ b/tests/setup/test_da_setup_router.py @@ -311,7 +311,6 @@ def test_global_and_command_gates_require_canonical_da_completion() -> None: "create.prompt.md", "delete.prompt.md", "evaluate.prompt.md", - "flightcheck.prompt.md", "push.prompt.md", "restore-template-configs.prompt.md", "review.prompt.md", @@ -343,16 +342,58 @@ def test_global_and_command_gates_require_canonical_da_completion() -> None: assert "Do not require\n`connect_ready: true`" in connect_prompt assert "workspace evidence" in connect_prompt + flightcheck_prompt = (_PROMPTS / "flightcheck.prompt.md").read_text( + encoding="utf-8" + ) + assert "FlightCheck entry contract" in flightcheck_prompt + assert "Standalone FlightCheck" in flightcheck_prompt + assert "Canonical setup ready" in flightcheck_prompt + assert "`.local/config.json`'s" in instructions -def test_global_gate_preserves_flightcheck_only_mode() -> None: +def test_global_gate_routes_flightcheck_through_setup_evidence() -> None: instructions = _INSTRUCTIONS.read_text(encoding="utf-8") normalized = " ".join(instructions.split()) + prompt = (_PROMPTS / "flightcheck.prompt.md").read_text(encoding="utf-8") + normalized_prompt = " ".join(prompt.split()) + skill = ( + _SOLUTION / "src" / "skills" / "flightcheck" / "SKILL.md" + ).read_text(encoding="utf-8") + normalized_skill = " ".join(skill.split()) - assert "typed `/flightcheck`" in normalized + assert "typed `/flightcheck` or explicitly asked to validate setup" in normalized + assert "FlightCheck entry contract" in instructions assert "`flightCheckOnly: true`" in normalized - assert "This exception applies only to `/flightcheck`" in normalized + assert "Apply the first matching state" in normalized + assert "**Standalone FlightCheck:**" in instructions + assert "**Canonical setup ready:**" in instructions + assert "**Setup readiness outstanding:**" in instructions + assert "**Workspace preparation required:**" in instructions + assert "exact maker-facing copy" in normalized + assert "FlightCheck is available when setup prepares the local agent workspace" in ( + normalized + ) + assert "Collect every non-empty `failure_causes` entry" in normalized + assert "**Environment capacity**" in instructions + assert "**Connections**" in instructions + assert "**Agent content**" in instructions + assert "keep the canonical setup IDs internal" in normalized + assert "{READINESS_ISSUES}" in instructions + assert "{READINESS_ITEM}" in instructions + assert "{READINESS_DETAIL}" in instructions + assert "Setup readiness requires attention:" in instructions + assert "performs the next readiness run" in normalized + + assert "Follow the **FlightCheck entry contract**" in normalized_prompt + assert "Standalone FlightCheck" in normalized_prompt + assert "Canonical setup ready" in normalized_prompt + assert "complete maker-facing response" in normalized_skill + + entry_contract = instructions.split("#### FlightCheck entry contract", 1)[1] + entry_contract = entry_contract.split("### If canonical setup is ready", 1)[0] + assert entry_contract.count("**Message:**") == 2 + assert entry_contract.count("**End message.**") == 2 def test_maker_profile_requires_only_canonical_completion() -> None: @@ -1298,11 +1339,15 @@ def test_da_local_capabilities_remain_available() -> None: assert "unchanged deployed" in normalized, name -def test_flightcheck_preserves_standalone_and_local_only_modes() -> None: +def test_flightcheck_preserves_standalone_and_configured_scope_modes() -> None: + instructions = _INSTRUCTIONS.read_text(encoding="utf-8") prompt = (_PROMPTS / "flightcheck.prompt.md").read_text(encoding="utf-8") normalized = " ".join(prompt.split()) - - assert "flightCheckOnly: true" in normalized - assert "proceed without canonical setup state" in normalized - assert "only the local-files FlightCheck scope" in normalized - assert "scope fixed to `local`" in normalized + skill = ( + _SOLUTION / "src" / "skills" / "flightcheck" / "SKILL.md" + ).read_text(encoding="utf-8") + + assert "flightCheckOnly: true" in instructions + assert "scope selection supported by the active configuration" in normalized + assert "supported scopes are `full`, `environment`" in skill + assert "scope fixed to `local`" not in normalized From dd1fa4d9fbd1de39a1b79812f5e7558d760252e5 Mon Sep 17 00:00:00 2001 From: Nkem Dockery Date: Fri, 25 Sep 2026 16:06:55 -0700 Subject: [PATCH 2/4] fix(setup): streamline setup handoffs Carry confirmed setup intent through FlightCheck recovery and remove redundant confirmation prompts. End completed setup with stable landing-page, connection, and menu entry points. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../.github/copilot-instructions.md | 69 ++++++++++++-- .../src/skills/flightcheck/SKILL.md | 4 +- .../src/skills/foundation-setup/SKILL.md | 69 +++++++++----- tests/setup/test_da_setup_router.py | 94 +++++++++++++------ 4 files changed, 171 insertions(+), 65 deletions(-) diff --git a/solutions/ess-maker-skills/.github/copilot-instructions.md b/solutions/ess-maker-skills/.github/copilot-instructions.md index fc8fbdacc..d34962891 100644 --- a/solutions/ess-maker-skills/.github/copilot-instructions.md +++ b/solutions/ess-maker-skills/.github/copilot-instructions.md @@ -117,27 +117,78 @@ Build `{READINESS_ISSUES}` as follows: **Message:** -FlightCheck is available when setup readiness is complete. - -Setup readiness requires attention: +Running FlightCheck requires Setup to be complete. Some setup items still +require attention: {READINESS_ISSUES} -Type `/setup` to refresh the readiness checks and work through each item. - **End message.** +Then use `vscode_askQuestions` with this exact question: + +```json +[ + { + "header": "Setup", + "question": "Would you like to return to Setup now?", + "options": [ + { + "label": "Return to Setup", + "description": "Continue Setup and work through these readiness items", + "recommended": true + }, + { + "label": "Not now", + "description": "Close this request and keep the current setup state" + } + ], + "allowFreeformInput": false + } +] +``` + Use the following Message block for workspace preparation: **Message:** -FlightCheck is available when setup prepares the local agent workspace. Type -`/setup` to prepare the workspace. +Running FlightCheck requires Setup to be complete. Setup needs to prepare the +local agent workspace. **End message.** -After rendering either Message block, the request is complete. `/setup` -performs the next readiness run. +Then use `vscode_askQuestions` with this exact question: + +```json +[ + { + "header": "Setup", + "question": "Would you like to run Setup now?", + "options": [ + { + "label": "Run Setup", + "description": "Prepare the local agent workspace", + "recommended": true + }, + { + "label": "Not now", + "description": "Close this request and keep the current setup state" + } + ], + "allowFreeformInput": false + } +] +``` + +When the maker selects **Return to Setup** or **Run Setup**, treat the selection +as a `/setup` invocation and read `src/skills/foundation-setup/SKILL.md`. The +selection already confirms setup intent. Carry forward the active agent, +environment, and canonical setup state resolved by this contract. Do not ask +the maker to select **Resume setup for this agent** or otherwise reconfirm the +same known target. Start at the first setup decision or operation not already +established by that context. If no usable target identity was resolved, follow +the normal target-selection flow. When the maker selects **Not now** or +dismisses the question, finish the request with canonical setup state +unchanged. ### If canonical setup is ready diff --git a/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md b/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md index 6ade70cc3..8d53bc90a 100644 --- a/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md +++ b/solutions/ess-maker-skills/src/skills/flightcheck/SKILL.md @@ -16,8 +16,8 @@ The calling prompt applies the **FlightCheck entry contract** in that contract before continuing. Continue with this skill for the contract's **Standalone FlightCheck** and -**Canonical setup ready** states. The contract supplies the complete -maker-facing response for its other states. +**Canonical setup ready** states. The contract owns the maker interaction and +next route for its other states. Treat a config with `releaseLine: "da"` and no `dataverseEndpoint` as a native no-Dataverse agent. Its supported scopes are `full`, `environment`, diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md b/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md index b883df878..cc5b75558 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md @@ -115,32 +115,29 @@ Do not describe an authorization wait as service processing, start a second comm Establish a working Python invocation before running setup commands. -Before checking the available Python invocation, render this exact message as a -completed response: +Before checking the available Python invocation, render this exact Message +block as a completed response. This is an informational disclosure, not another +setup confirmation. The current `/setup` request or the explicit choice that +entered this skill already confirms the maker's intent. After rendering the +message, proceed directly with runtime discovery: -> **Setup command approvals** -> -> VS Code will ask you to approve commands that: -> -> - check Python and prepare the required local tools; -> - sign you in and inspect the selected environment and agent; -> - perform the setup actions you confirm and prepare the local workspace; -> - download required Microsoft components when needed. -> -> To avoid repeated prompts, open the permissions menu below the chat input and -> select **Allow all** for this chat session. This applies to every tool used in -> the session, not only setup. Provide a screenshot of your chat input if you -> need guidance finding the setting. -> -> When you're ready, choose: +**Message:** -Offer exactly: +**Setup command approvals** -- **Continue setup** -- **Cancel setup** +VS Code will ask you to approve commands that: + +- check Python and prepare the required local tools; +- sign you in and inspect the selected environment and agent; +- perform the setup actions you confirm and prepare the local workspace; +- download required Microsoft components when needed. + +To avoid repeated prompts, open the permissions menu below the chat input and +select **Allow all** for this chat session. This applies to every tool used in +the session, not only setup. Provide a screenshot of your chat input if you +need guidance finding the setting. -Do not preselect a choice. For **Continue setup**, proceed with runtime -discovery. For **Cancel setup**, run no commands and stop. +**End message.** - Run setup commands from the current ESS Maker Skills workspace folder. - From the kit root, check each candidate with @@ -229,14 +226,31 @@ Parse `DA_ACTIVE_AGENT_JSON:`. Continue setup for that agent when its `connectRe The final handoff is the sole completion summary. After it, offer exactly these context-appropriate choices: -- **Continue customizing this agent** +- **Finish setup** - **Switch to another configured agent** -- only when another configured agent exists. - **Install another product in this environment** - **Reset and use this workspace** - **Create and open a new workspace** -- **Finish for now** -Do not preselect a choice. **Install another product in this environment** begins `da-mos-starter.md` at its first product-installation decision surface with the recorded environment and ring. **Finish for now** ends immediately. Every other selected follow-up begins at that follow-up's first decision surface rather than rendering another completion summary. +Do not preselect a choice. **Finish setup** closes the setup flow. Do not render +another setup completion summary. After the maker selects it, show: + +**Message:** + +The {agent display name} agent is now active. + +- Run `/landing-page` to configure branding and the content employees see. +- Run `/connect` to choose an integration. +- Type `/menu` to see all available capabilities. + +**End message.** + +Then end the request. + +**Install another product in this environment** begins `da-mos-starter.md` at +its first product-installation decision surface with the recorded environment +and ring. Every other selected follow-up begins at that follow-up's first +decision surface rather than rendering another completion summary. ## Start @@ -360,4 +374,7 @@ Offer exactly: - **Yes, I have an agent** — ask for its Copilot Studio URL. - **No, I need a fresh agent** — follow `src/skills/foundation-setup/da-mos-starter.md`. -Do not run Dataverse foundation or onboarding playbooks. Never route from `/setup` into an integration or topic playbook. +Do not run Dataverse foundation or onboarding playbooks. Never route from +`/setup` into an integration or topic playbook. **Finish setup** advertises +separate commands and ends the current request; a command selected afterward +begins its own prompt flow. diff --git a/tests/setup/test_da_setup_router.py b/tests/setup/test_da_setup_router.py index 14c4f2506..ff1b0e6c5 100644 --- a/tests/setup/test_da_setup_router.py +++ b/tests/setup/test_da_setup_router.py @@ -167,21 +167,23 @@ def test_public_setup_resolves_python_before_bootstrap_commands() -> None: foundation = _FOUNDATION.read_text(encoding="utf-8") normalized_prompt = " ".join(prompt.split()) normalized_foundation = " ".join(foundation.split()) - runtime_message = """> **Setup command approvals** -> -> VS Code will ask you to approve commands that: -> -> - check Python and prepare the required local tools; -> - sign you in and inspect the selected environment and agent; -> - perform the setup actions you confirm and prepare the local workspace; -> - download required Microsoft components when needed. -> -> To avoid repeated prompts, open the permissions menu below the chat input and -> select **Allow all** for this chat session. This applies to every tool used in -> the session, not only setup. Provide a screenshot of your chat input if you -> need guidance finding the setting. -> -> When you're ready, choose:""" + runtime_message = """**Message:** + +**Setup command approvals** + +VS Code will ask you to approve commands that: + +- check Python and prepare the required local tools; +- sign you in and inspect the selected environment and agent; +- perform the setup actions you confirm and prepare the local workspace; +- download required Microsoft components when needed. + +To avoid repeated prompts, open the permissions menu below the chat input and +select **Allow all** for this chat session. This applies to every tool used in +the session, not only setup. Provide a screenshot of your chat input if you +need guidance finding the setting. + +**End message.**""" declined_command_message = """> **Run this command manually** > > Setup paused before running this command: @@ -200,11 +202,14 @@ def test_public_setup_resolves_python_before_bootstrap_commands() -> None: assert foundation.index(runtime_message) < foundation.index( '{PYTHON} -c "import sys; print(sys.executable)"' ) - assert foundation.index("- **Continue setup**") < foundation.index( - "- **Cancel setup**" + command_runtime = foundation.split("## Command runtime", 1)[1].split( + "## Shared workspace choices", 1 + )[0] + assert "informational disclosure, not another setup confirmation" in ( + " ".join(command_runtime.split()) ) - assert "Do not preselect a choice." in foundation - assert "For **Cancel setup**, run no commands and stop." in foundation + assert "**Continue setup**" not in command_runtime + assert "**Cancel setup**" not in command_runtime assert "### When command approval is declined" in foundation assert declined_command_message in foundation assert ( @@ -374,9 +379,9 @@ def test_global_gate_routes_flightcheck_through_setup_evidence() -> None: assert "**Setup readiness outstanding:**" in instructions assert "**Workspace preparation required:**" in instructions assert "exact maker-facing copy" in normalized - assert "FlightCheck is available when setup prepares the local agent workspace" in ( - normalized - ) + assert "Running FlightCheck requires Setup to be complete." in instructions + assert "Some setup items still require attention:" in normalized + assert "Setup needs to prepare the local agent workspace." in normalized assert "Collect every non-empty `failure_causes` entry" in normalized assert "**Environment capacity**" in instructions assert "**Connections**" in instructions @@ -385,18 +390,31 @@ def test_global_gate_routes_flightcheck_through_setup_evidence() -> None: assert "{READINESS_ISSUES}" in instructions assert "{READINESS_ITEM}" in instructions assert "{READINESS_DETAIL}" in instructions - assert "Setup readiness requires attention:" in instructions - assert "performs the next readiness run" in normalized + assert '"question": "Would you like to return to Setup now?"' in instructions + assert '"label": "Return to Setup"' in instructions + assert '"question": "Would you like to run Setup now?"' in instructions + assert '"label": "Run Setup"' in instructions + assert instructions.count('"label": "Not now"') == 2 + assert instructions.count('"allowFreeformInput": false') >= 2 + assert "treat the selection as a `/setup` invocation" in normalized + assert "src/skills/foundation-setup/SKILL.md" in instructions + assert "selection already confirms setup intent" in normalized + assert "Carry forward the active agent" in instructions + assert "Do not ask the maker to select **Resume setup for this agent**" in normalized + assert "first setup decision or operation not already established" in normalized + assert "normal target-selection flow" in normalized + assert "canonical setup state unchanged" in normalized assert "Follow the **FlightCheck entry contract**" in normalized_prompt assert "Standalone FlightCheck" in normalized_prompt assert "Canonical setup ready" in normalized_prompt - assert "complete maker-facing response" in normalized_skill + assert "owns the maker interaction and next route" in normalized_skill entry_contract = instructions.split("#### FlightCheck entry contract", 1)[1] entry_contract = entry_contract.split("### If canonical setup is ready", 1)[0] assert entry_contract.count("**Message:**") == 2 assert entry_contract.count("**End message.**") == 2 + assert entry_contract.count("```json") == 2 def test_maker_profile_requires_only_canonical_completion() -> None: @@ -445,6 +463,12 @@ def test_foundation_routes_supported_da_setup_paths() -> None: "src/skills/foundation-setup/product-line-reconciliation.md", } assert "not a setup option to advertise or recommend" in normalized + assert ( + "Never route from `/setup` into an integration or topic playbook" in normalized + ) + assert "**Finish setup** advertises separate commands and ends the current request" in ( + normalized + ) assert "src/reference/native-alm-import.md" in import_text assert "DA_ALM_IMPORT_JSON:" in import_text assert "setup_existing_da.py validate-agent" in import_text @@ -995,10 +1019,24 @@ def test_foundation_exposes_multi_agent_entry_and_completion_choices() -> None: "Reset and use this workspace", "Create and open a new workspace", "Cancel setup", - "Continue customizing this agent", - "Finish for now", + "Finish setup", ): assert f"**{choice}**" in text + completion_choices = text.split( + "The final handoff is the sole completion summary.", 1 + )[1].split("## Start", 1)[0] + assert "- **Continue customizing this agent**" not in completion_choices + assert "- **Finish for now**" not in completion_choices + assert "The {agent display name} agent is now active." in completion_choices + assert ( + "- Run `/landing-page` to configure branding and the content employees see." + in completion_choices + ) + assert "- Run `/connect` to choose an integration." in completion_choices + assert "- Type `/menu` to see all available capabilities." in completion_choices + assert "What would you like to customize?" not in completion_choices + assert "**Create a topic**" not in completion_choices + assert "Then end the request." in completion_choices assert "Create a new workspace without opening it" not in text assert "setup_existing_da.py select-agent" in text assert "one Power Platform environment" in normalized @@ -1044,7 +1082,7 @@ def test_foundation_uses_maker_facing_progress_without_duplicate_state() -> None assert "a blocked state that requires maker action" in normalized assert "A sequence of setup operations that retains the same markers" in normalized assert "The final handoff is the sole completion summary" in normalized - assert "**Finish for now** ends immediately" in normalized + assert "**Finish setup** closes the setup flow" in normalized assert "first decision surface rather than rendering another completion summary" in ( normalized ) From 5cb2c69f8cad22bf2ec9fc70337a7acdb8c635a1 Mon Sep 17 00:00:00 2001 From: Nkem Dockery Date: Fri, 25 Sep 2026 17:04:20 -0700 Subject: [PATCH 3/4] fix(setup): route nested choices back safely Replace nested setup cancellation dead ends with explicit back routes while retaining cancellation on entry menus. Preserve or reset account context according to observed access evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../src/skills/foundation-setup/SKILL.md | 7 +- .../skills/foundation-setup/da-alm-import.md | 15 ++- .../foundation-setup/da-environment-target.md | 8 +- .../skills/foundation-setup/da-mos-starter.md | 7 +- .../skills/foundation-setup/da-prod-to-dev.md | 8 +- .../product-line-reconciliation.md | 28 +++-- tests/setup/test_da_setup_router.py | 107 ++++++++++++++++-- 7 files changed, 144 insertions(+), 36 deletions(-) diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md b/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md index cc5b75558..f00cffb34 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md @@ -185,9 +185,9 @@ When an occupied workspace needs a new environment, offer **Create and open a ne - **Use suggested location -- {suggested absolute sibling-folder path}** - **Choose another location** -- **Cancel setup** +- **Go back** -Do not ask the maker to type a path unless they select **Choose another location**. The destination must be a new absolute sibling-folder path outside the current Developer Kit repository. Then run: +For **Go back**, make no changes and return to the Setup choice surface that offered **Create and open a new workspace**. When explicit fresh-install intent for another environment entered this path directly, render the active-agent choice surface for the occupied workspace. Do not ask the maker to type a path unless they select **Choose another location**. The destination must be a new absolute sibling-folder path outside the current Developer Kit repository. Continue only after the maker selects a destination, then run: ```text python scripts/prepare_fresh_workspace.py \ @@ -205,9 +205,8 @@ Offer exactly: - **Reset workspace** - **Go back** -- **Cancel setup** -Do not preselect **Reset workspace**. Continue only when the maker selects it, then run: +Do not preselect **Reset workspace**. For **Go back**, make no changes and return to the choice surface that offered **Reset and use this workspace**. Continue only when the maker selects **Reset workspace**, then run: ```text python scripts/reset_local_workspace.py --confirm-reset diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md index 1eb295269..262f94bea 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md @@ -109,10 +109,12 @@ Offer exactly: - **Choose an existing agent in this environment** - **Replace an existing agent with this package** -- **Cancel setup** +- **Go back** Do not preselect a choice or recommend replacement. For either existing-agent choice, run `setup_existing_da.py list-agents` for the target environment, show the visible Dev agent names, and let the maker choose one exact agent. The native candidate result is authoritative DA-GA evidence; run the parent's selected-agent product-line reconciliation with `--native-da-ga` before validation or replacement. For **Choose an existing agent in this environment**, validate the selected agent and continue through `da-existing-dev.md`. +For **Go back**, retain the current account, environment, and ring and return to the parent skill's **What would you like to set up in this environment?** choice surface. Clear package-import and replacement intent, but preserve the import receipt and collision evidence. + Before replacement, validate the exact selected Dev agent: ```text @@ -130,12 +132,15 @@ Parse `DA_AGENT_VALIDATION_JSON:`. Show its display name, then ask: Offer exactly: - **Continue replacement** -- **Choose the existing agent without replacement** -- **Cancel setup** +- **Go back** Never preselect or recommend **Continue replacement**. Continue only after the -maker explicitly selects it. Pass the same validated internal ID in both -confirmation arguments: +maker explicitly selects it. For **Go back**, make no changes and return to +**Handle a collision**. Reuse the latest successful visible-agent list instead +of rerunning `list-agents` solely because the maker went back. Clear the +replacement intent and selected replacement candidate; require an exact agent +selection for whichever route the maker chooses next. Pass the same validated +internal ID in both confirmation arguments: ```text python scripts/setup_alm_import.py \ diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-environment-target.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-environment-target.md index 9a9ead6b3..1a554654e 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-environment-target.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-environment-target.md @@ -44,7 +44,7 @@ Use the host's interactive single-selection control and offer exactly: - **Use another account** - **Use an environment URL** - **Create a Power Platform environment** -- **Cancel setup** +- **Go back** Do not preselect a choice. @@ -56,7 +56,7 @@ Do not preselect a choice. Stop until the maker confirms that the environment is ready or supplies its URL. Do not route into a Dataverse provisioning skill. -- For **Cancel setup**, make no changes and stop. +- For **Go back**, retain the selected account and return to **Resolve the service ring**. Do not rerun environment discovery until the maker explicitly selects a ring. **Use another account** remains the account-switch route on this surface. When environment discovery fails, parse `DA_ENVIRONMENT_LIST_ERROR_JSON:` and preserve it with `DA_ENVIRONMENT_LIST_ERROR_RESPONSE_JSON:` or `DA_ENVIRONMENT_LIST_ERROR_RESPONSE_TEXT:` as diagnostic evidence. When `authorizationFailure` is `true`, say that the selected account could not list its Power Platform environments and offer the same four choices. Do not convert an authorization failure into an empty environment list. For any other failure, report the observed blocker and stop. @@ -72,12 +72,12 @@ Use the host's interactive single-selection control and present these labels unc - **Try with a different user** - **Try a different environment** -- **Cancel setup** +- **Go back** For **Try with a different user**, return to **Choose the sign-in account** in `SKILL.md`. After the maker selects or supplies a different user, rerun environment discovery for that account and continue from its environment picker. For **Try a different environment**, retain the current account and ring, rerun `list-environments`, and continue from the returned environment picker. -For **Cancel setup**, make no changes and stop. +For **Go back**, retain the current account, ring, and environment, then return to the parent skill's **What would you like to set up in this environment?** choice surface. Do not rerun the failed environment-scoped operation. When direct agent lookup remains available after an empty visible-agent list, also offer **Use an agent URL**. Place it before the three shared choices, ask for the exact Copilot Studio agent URL when selected, and continue through direct inspection in `da-existing-dev.md`. diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md index 842d95ed5..3643a14d5 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md @@ -83,9 +83,10 @@ Offer exactly: - **Create agent** - **Choose a different product** -- **Cancel setup** +- **Go back** Do not preselect **Create agent**. After the maker explicitly selects **Create agent** for the displayed product and target, begin the create operation immediately; the confirmation surface already communicates the selected product and environment. +For **Choose a different product**, return to the valid rows from the latest successful catalog result and continue through this confirmation surface for the new selection. For **Go back**, retain the current account, environment, and ring and return to the parent skill's **What would you like to set up in this environment?** choice surface. ## Create @@ -119,12 +120,14 @@ When the annotations report `outcome: collision`, do not infer which visible age - **Choose an existing agent in this environment** - **Choose a different catalog product** -- **Cancel setup** +- **Go back** Do not preselect a choice. For **Choose an existing agent in this environment**, show the returned names, let the maker select one exact agent, run the parent's selected-agent product-line reconciliation with `--native-da-ga`, and continue through `da-existing-dev.md`. The selected agent is maker-supplied intent, not proof of package identity. This path does not replace an agent. For **Choose a different catalog product**, present the valid rows from the latest successful catalog result and let the maker select another exact product. Continue through **Confirm the exact product and target** for that selection. A new create request becomes available only after the maker confirms the new product and uses a new client request UUID. +For **Go back**, retain the current account, environment, and ring and return to the parent skill's **What would you like to set up in this environment?** choice surface. Preserve the collided create result and its client request UUID; do not repeat that create request. + ## Enable ALM After a successful create, say: diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md index 8cd2f20ca..080f9faba 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md @@ -94,7 +94,7 @@ python scripts/setup_alm_export.py export \ --ring "{SOURCE_RING}" ``` -If the maker cancels after export, run `setup_alm_export.py cleanup`. A later +If the maker goes back after export, run `setup_alm_export.py cleanup`. A later fresh inspect or export also removes a recorded package left by an interrupted run before continuing. @@ -109,9 +109,9 @@ Immediately before import, ask: Offer exactly: - **Create editable Dev agent** -- **Cancel setup** +- **Go back** -Do not preselect **Create editable Dev agent**. Continue only after the maker selects it. +Do not preselect **Create editable Dev agent**. Continue only after the maker selects it. For **Go back**, run `setup_alm_export.py cleanup` before routing anywhere else. When the target environment is the source environment, retain the selected account and return to the choice surface that supplied the Prod agent: the active-agent choice surface for a recorded local target, or **Do you already have an ESS agent in Copilot Studio?** for a supplied URL. When the maker selected a different target environment, clear that target and return to **Choose the sign-in account** in the parent skill. After the maker selects an account, rerun source inspection under that account and resume target-environment selection before creating another export. Never reuse the cleaned-up package. When the maker explicitly selected another environment, use its friendly display name when an authoritative operation returned one in this invocation. Otherwise say "the selected Power Platform environment" without showing internal IDs or URLs. Run no other operation between the maker's confirmation and the create-only import: @@ -146,7 +146,7 @@ inspection, show: > A related editable Dev agent now exists: **{agent display name}**. Use it for > this workspace? -Offer exactly **Use related Dev agent** and **Cancel setup**, and offer to attach it through [Attach and complete](#attach-and-complete). Continue only when the maker selects **Use related Dev agent**. If no related Dev is returned or the family cannot be proven, stop with the safe conflict outcome from `src/reference/native-alm-import.md`. Do not recover a collision or offer replacement. +Offer exactly **Use related Dev agent** and **Go back**, and offer to attach it through [Attach and complete](#attach-and-complete). Continue only when the maker selects **Use related Dev agent**. For **Go back**, retain the selected account and durable operation evidence and return to the choice surface that supplied the Prod agent: the active-agent choice surface for a recorded local target, or **Do you already have an ESS agent in Copilot Studio?** for a supplied URL. Do not rerun export or import. If no related Dev is returned or the family cannot be proven, stop with the safe conflict outcome from `src/reference/native-alm-import.md`. Do not recover a collision or offer replacement. For any other result besides `kind: success`, stop and use the outcome guidance in `src/reference/native-alm-import.md`; do not retry or replace an agent. diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md b/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md index ee595cda2..1c584e08d 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md @@ -33,21 +33,35 @@ Parse `DA_SETUP_PRODUCT_RECONCILIATION_JSON:`. - For `action: continue-da-ga-setup`, continue at the next DA-GA setup operation. `classification: unknown` is deliberately fail-open; do not persist it or tell the maker that DA-GA was proven. - For `action: stop-and-use-cea-kit`, stop before inspection, validation, attachment, import, Object Model installation, or other DA-GA-only work. Render **Choose the starting point and target environment** as complete, **Verify access and agent identity** as blocked, **Establish an editable Dev agent** and **Materialize the local workspace** as pending, and **Review the setup handoff** as in progress while the kit-switch choice is pending. - When `recoveryUnavailable` is `true`, say that the compatible pinned installer could not be resolved, render **Review the setup handoff** as blocked, make no changes, and stop without guessing a branch or release. + When `recoveryUnavailable` is `true`, say that the compatible pinned installer could not be resolved and render **Review the setup handoff** as blocked. Do not guess a branch or release. Use the host's interactive single-selection control, ask **How would you like to continue setup?**, and offer exactly: + - **Choose a different agent** + - **Choose a different environment** + - **Go back** + + Do not offer **Install and open the compatible kit** when its pinned installer is unavailable. Follow the corresponding Setup routes below. Otherwise say: > **{agent display name or Selected agent}** belongs to the classic Employee Self-Service agent product line, so this Developer Kit cannot safely continue its setup. - > - > Want me to install the compatible CEA kit in a separate folder and open it now? - - When the maker accepts, run `recoveryCommand` in a terminal using `recoveryShell`. On success, render **Review the setup handoff** as complete and say: + Then use the host's interactive single-selection control and ask exactly: - > Continue setup in the workspace opened by the compatible installer. No Copilot Studio agent or setup state was changed by this product-line check. + > How would you like to continue setup? + + Offer exactly: + - **Install and open the compatible kit** + - **Choose a different agent** + - **Choose a different environment** + - **Go back** - - When the maker declines, render **Review the setup handoff** as complete, show `recoveryCommand` in a fenced block whose language is `recoveryShell`, and say: + Do not preselect a choice. + - For **Install and open the compatible kit**, run `recoveryCommand` in a terminal using `recoveryShell`. On success, render **Review the setup handoff** as complete and say: + + > Continue setup in the workspace opened by the compatible installer. No Copilot Studio agent or setup state was changed by this product-line check. - > Run this pinned command later to install the compatible kit in a separate folder. No Copilot Studio agent or setup state was changed by this product-line check. + - For **Choose a different agent**, retain the current account, environment, and ring. Read `src/skills/foundation-setup/da-existing-dev.md` and continue from its environment-scoped `list-agents` candidate-selection path. Do not persist the rejected agent or change the active local agent before another exact candidate passes product-line reconciliation. + - For **Choose a different environment**, retain the current account and ring. Read `src/skills/foundation-setup/da-environment-target.md`, rerun `list-environments`, and continue from its environment picker. A target in another environment follows the parent skill's new-workspace contract. + - For **Go back**, do not persist the rejected agent. Return to **Choose the sign-in account** in the parent skill. After the maker selects an account, read `src/skills/foundation-setup/da-environment-target.md`, list that account's environments, and continue from the selected environment through the parent skill's create-or-connect choice. The direct different-agent and different-environment routes above retain the current account; **Go back** is the account-reset route. - When command execution fails, render **Review the setup handoff** as blocked and state the observed installer failure. Do not label the unchanged command as a retry or reinterpret installer failure as a successful handoff. When the output identifies an existing installation, verify that its checkout matches `releaseTag` and offer to open its `solutions/ess-maker-skills` workspace directly. Otherwise provide one recovery action grounded in the observed failure. diff --git a/tests/setup/test_da_setup_router.py b/tests/setup/test_da_setup_router.py index ff1b0e6c5..17496bde6 100644 --- a/tests/setup/test_da_setup_router.py +++ b/tests/setup/test_da_setup_router.py @@ -126,19 +126,46 @@ def test_setup_reconciles_every_selected_agent_before_da_only_work() -> None: "**Review the setup handoff** as in progress", ): assert mismatch_state in normalized_reconciliation + assert "How would you like to continue setup?" in reconciliation + for recovery_choice in ( + "Install and open the compatible kit", + "Choose a different agent", + "Choose a different environment", + "Go back", + ): + assert f"**{recovery_choice}**" in reconciliation + assert "**Not now**" not in reconciliation + assert "**Cancel setup**" not in reconciliation + assert "For **Install and open the compatible kit**, run `recoveryCommand`" in ( + reconciliation + ) + assert "continue from its environment-scoped `list-agents`" in reconciliation + assert "rerun `list-environments`" in reconciliation + assert "Return to **Choose the sign-in account**" in reconciliation + assert "**Go back** is the account-reset route" in reconciliation + unavailable_recovery = reconciliation[ + reconciliation.index("When `recoveryUnavailable` is `true`") : + reconciliation.index("Otherwise say:") + ] + for recovery_choice in ( + "Choose a different agent", + "Choose a different environment", + "Go back", + ): + assert f"**{recovery_choice}**" in unavailable_recovery assert ( - "Want me to install the compatible CEA kit in a separate folder and open " - "it now?" - ) in normalized_reconciliation - assert "When the maker accepts, run `recoveryCommand`" in reconciliation - assert "When the maker declines" in reconciliation + "Do not offer **Install and open the compatible kit**" + in unavailable_recovery + ) + assert "Follow the corresponding Setup routes below." in unavailable_recovery + assert "stop without guessing" not in reconciliation assert "When command execution fails" in reconciliation assert "Do not label the unchanged command as a retry" in reconciliation assert "verify that its checkout matches `releaseTag`" in reconciliation assert "offer to open its `solutions/ess-maker-skills` workspace directly" in ( reconciliation ) - assert reconciliation.count("**Review the setup handoff** as complete") == 2 + assert reconciliation.count("**Review the setup handoff** as complete") == 1 assert reconciliation.count("**Review the setup handoff** as blocked") == 2 assert "No Copilot Studio agent or setup state was changed" in reconciliation assert "installation was not changed" not in reconciliation @@ -668,6 +695,16 @@ def test_prod_to_dev_reference_composes_durable_boundaries() -> None: assert "Create editable Dev agent" in text assert "Do not preselect **Create editable Dev agent**" in text assert "Use related Dev agent" in text + assert "**Cancel setup**" not in text + assert "If the maker goes back after export" in normalized + assert ( + "When the maker selected a different target environment, clear that " + "target and return to **Choose the sign-in account**" + ) in normalized + assert "Never reuse the cleaned-up package" in normalized + assert ( + "Do not rerun export or import" in normalized + ) assert "render the factual workspace and runtime-readiness report there" in normalized assert "Existing Prod agent; related Dev reused" in normalized assert "Existing Prod agent; new Dev created" in normalized @@ -731,7 +768,7 @@ def test_mos_starter_reference_composes_durable_boundaries() -> None: "Use another account", "Use an environment URL", "Create a Power Platform environment", - "Cancel setup", + "Go back", ): assert f"**{empty_environment_choice}**" in environment_target assert "does not require a Dataverse database" in environment_target @@ -749,7 +786,7 @@ def test_mos_starter_reference_composes_durable_boundaries() -> None: for retry_choice in ( "Try with a different user", "Try a different environment", - "Cancel setup", + "Go back", ): assert f"**{retry_choice}**" in environment_target assert "present these labels unchanged with the selection initially unset" in ( @@ -759,6 +796,13 @@ def test_mos_starter_reference_composes_durable_boundaries() -> None: normalized_environment_target ) assert "retain the current account and ring" in normalized_environment_target + assert "**Use another account** remains the account-switch route" in ( + normalized_environment_target + ) + assert ( + "return to the parent skill's **What would you like to set up in this " + "environment?** choice surface" + ) in normalized_environment_target assert "also offer **Use an agent URL**" in environment_target assert "setup_mos_starter.py list" in text assert "DA_MOS_STARTER_PACKAGES_JSON:" in text @@ -842,12 +886,26 @@ def test_mos_starter_reference_composes_durable_boundaries() -> None: assert "Create a new ESS agent" in normalized assert "**{selected product label}**" in text assert "Choose a different product" in text + assert ( + "For **Choose a different product**, return to the valid rows from the " + "latest successful catalog result" + ) in normalized assert _PREPARE_FRESH_WORKSPACE.is_file() assert "Create and open a new workspace" in foundation assert "Create a new workspace without opening it" not in foundation assert "Use suggested location -- {suggested absolute sibling-folder path}" in foundation assert "Choose another location" in foundation assert "Do not ask the maker to type a path unless" in normalized_foundation + workspace_location = foundation[ + foundation.index("When an occupied workspace needs a new environment") : + foundation.index("For **Reset and use this workspace**") + ] + assert "**Go back**" in workspace_location + assert "**Cancel setup**" not in workspace_location + assert ( + "return to the Setup choice surface that offered **Create and open a new " + "workspace**" + ) in " ".join(workspace_location.split()) assert "scripts/prepare_fresh_workspace.py" in foundation assert "--open-vscode" in foundation assert "DA_PREPARED_WORKSPACE_JSON:" in foundation @@ -974,7 +1032,9 @@ def test_mos_starter_reference_composes_durable_boundaries() -> None: assert "uses a new client request UUID" in normalized collision_choices = text[text.index("When the annotations report `outcome: collision`") :] collision_choices = collision_choices[: collision_choices.index("## Enable ALM")] - assert "**Go back**" not in collision_choices + assert "**Go back**" in collision_choices + assert "**Cancel setup**" not in collision_choices + assert "do not repeat that create request" in " ".join(collision_choices.split()) assert "does not identify the corresponding agent" in reference assert "createFromStarterPackage" in reference @@ -1006,6 +1066,15 @@ def test_foundation_offers_local_workspace_reset() -> None: assert "report its `ERROR:` and `NOTE:` output" in normalized assert "will not change or delete any agent in Copilot Studio" in normalized assert "archive its current local agent files" in normalized + assert ( + "For **Go back**, make no changes and return to the choice surface that " + "offered **Reset and use this workspace**." + ) in normalized + reset_confirmation = text[ + text.index("For **Reset and use this workspace**") : + text.index("For **Switch to another configured agent**") + ] + assert "**Cancel setup**" not in reset_confirmation def test_foundation_exposes_multi_agent_entry_and_completion_choices() -> None: @@ -1191,15 +1260,33 @@ def test_alm_import_collision_and_retry_require_separate_choices() -> None: for choice in ( "Choose an existing agent in this environment", "Replace an existing agent with this package", - "Cancel setup", "Continue replacement", + "Go back", "Retry import", "Stop without retrying", ): assert choice in text + assert "**Cancel setup**" not in text + assert "Choose the existing agent without replacement" not in text assert "Do not preselect a choice or recommend replacement" in normalized assert "setup_existing_da.py list-agents" in text assert "Never preselect or recommend **Continue replacement**" in text + assert ( + "For **Go back**, make no changes and return to **Handle a collision**." + in normalized + ) + assert ( + "Reuse the latest successful visible-agent list instead of rerunning " + "`list-agents` solely because the maker went back." + ) in normalized + assert ( + "Clear the replacement intent and selected replacement candidate" + in normalized + ) + assert ( + "Clear package-import and replacement intent, but preserve the import " + "receipt and collision evidence." + ) in normalized assert "could not be proven" in normalized assert "avoid creating or replacing the agent twice" in normalized assert "only after the maker selects **Retry import**" in normalized From 546075546f0943aeae6e5133aca9e3b07db2ca68 Mon Sep 17 00:00:00 2001 From: Nkem Dockery Date: Fri, 25 Sep 2026 18:11:36 -0700 Subject: [PATCH 4/4] fix(setup): reconcile agents across storage backends Probe native MinimalBot and Dataverse identities independently, classify supported ESS families by schema prefix, and keep ALM enrollment distinct from agent existence. Stop unsupported or ambiguous agents with explicit recovery UX while preserving authoritative access and failure evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c820c60f-79a3-4b3d-ac32-3a309689c1ca --- .../scripts/reconcile_setup_agent.py | 380 ++++++++++----- .../scripts/setup_existing_da.py | 30 +- .../src/skills/foundation-setup/SKILL.md | 17 +- .../skills/foundation-setup/da-alm-import.md | 6 +- .../foundation-setup/da-existing-dev.md | 9 +- .../skills/foundation-setup/da-mos-starter.md | 4 +- .../skills/foundation-setup/da-prod-to-dev.md | 5 +- .../product-line-reconciliation.md | 158 ++++-- tests/scripts/test_reconcile_setup_agent.py | 455 ++++++++++-------- tests/scripts/test_setup_existing_da.py | 27 ++ tests/setup/test_da_setup_router.py | 85 +++- 11 files changed, 804 insertions(+), 372 deletions(-) diff --git a/solutions/ess-maker-skills/scripts/reconcile_setup_agent.py b/solutions/ess-maker-skills/scripts/reconcile_setup_agent.py index 66c668ac5..89771bad3 100644 --- a/solutions/ess-maker-skills/scripts/reconcile_setup_agent.py +++ b/solutions/ess-maker-skills/scripts/reconcile_setup_agent.py @@ -23,27 +23,13 @@ derive_environment_host, validate_environment_host, ) -from auth import AuthExpiredError, authenticate, query_all +from auth import AuthExpiredError, authenticate, dataverse_get from flightcheck.powerplatform_client import PowerPlatformClient from http_errors import APIError -CEA_SCHEMA_NAMES = frozenset( - { - "msdyn_copilotforemployeeselfservice", - "msdyn_copilotforemployeeselfservicecore", - "msdyn_copilotforemployeeselfservicehr", - "msdyn_copilotforemployeeselfserviceit", - } -) -DA_SCHEMA_NAMES = frozenset( - { - "gptagent_copilotforemployeeselfservice", - "gptagent_copilotforemployeeselfservicecore", - "gptagent_copilotforemployeeselfservicehr", - "gptagent_copilotforemployeeselfserviceit", - } -) +SOLUTION_BACKED_ESS_PREFIX = "msdyn_copilotforemployeeselfservice" +DA_GA_PREFIX = "gptagent_copilotforemployeeselfservice" RESULT_PREFIX = "DA_SETUP_PRODUCT_RECONCILIATION_JSON:" _RELEASE_MANIFEST = ( Path(__file__).resolve().parents[1] @@ -67,12 +53,12 @@ def _normalize_guid(value: str, label: str) -> str: def classify_schema_name(schema_name: str | None) -> str: - """Classify only exact, recognized solution-backed bot schema names.""" + """Classify the two supported ESS schema families by stable prefix.""" normalized = str(schema_name or "").strip().casefold() - if normalized in CEA_SCHEMA_NAMES: - return "cea" - if normalized in DA_SCHEMA_NAMES: - return "da" + if normalized.startswith(SOLUTION_BACKED_ESS_PREFIX): + return "solution-backed-ess" + if normalized.startswith(DA_GA_PREFIX): + return "da-ga" return "unknown" @@ -170,7 +156,11 @@ def _resolve_dataverse_url( client.authenticate(preferred_username=account) raw_environments = client.list_environments_for_user() if isinstance(raw_environments, dict) and "_error" in raw_environments: - return None + raise APIError( + status_code=int(raw_environments.get("_status") or 0), + resource_name="environment", + operation="read", + ) target = _normalize_guid(environment_id, "Environment ID") for environment in raw_environments: @@ -194,29 +184,109 @@ def _read_dataverse_agent( env_url: str, agent_id: str, account: str | None, -) -> dict[str, Any] | None: +) -> dict[str, Any]: token = authenticate(env_url, preferred_username=account) - records = query_all( + record = dataverse_get( env_url, token, - entity_set="bots", - select="botid,name,schemaname,ismanaged", - filter_expr=f"botid eq {agent_id}", + f"bots({agent_id})", + {"$select": "botid,name,schemaname,ismanaged"}, ) - if len(records) > 1: - raise ValueError("Dataverse returned duplicate records for one agent ID.") - return records[0] if records else None + if not isinstance(record, dict): + raise ValueError("Dataverse agent lookup returned an invalid shape.") + return record + + +def _exception_evidence(exc: BaseException) -> dict[str, Any]: + chain: list[dict[str, str]] = [] + current: BaseException | None = exc + seen: set[int] = set() + while current is not None and id(current) not in seen and len(chain) < 4: + seen.add(id(current)) + chain.append( + { + "type": type(current).__name__, + "message": str(current), + } + ) + current = current.__cause__ or current.__context__ + + result: dict[str, Any] = {"causes": chain} + status_code = getattr(exc, "status_code", None) + if status_code is None: + response = getattr(exc, "response", None) + status_code = getattr(response, "status_code", None) + if isinstance(status_code, int): + result["statusCode"] = status_code + error_code = getattr(exc, "error_code", None) + if error_code: + result["errorCode"] = str(error_code) + request_id = getattr(exc, "request_id", None) + if request_id: + result["requestId"] = str(request_id) + return result + + +def _failure_result( + backend: str, + exc: BaseException, + *, + stage: str = "agent-lookup", +) -> dict[str, Any]: + error = _exception_evidence(exc) + status_code = error.get("statusCode") + if status_code == 404: + outcome = "not-found" + elif status_code == 401: + outcome = "authentication-required" + elif status_code == 403: + outcome = "access-denied" + else: + outcome = "uncertain" + return { + "backend": backend, + "outcome": outcome, + "stage": stage, + "error": error, + } -def _continue_result(classification: str, evidence: str) -> dict[str, Any]: +def _identity_summary( + backend: str, + agent: dict[str, Any], + *, + evidence: str, +) -> dict[str, Any]: + schema_name = str( + agent.get("schemaName") or agent.get("schemaname") or "" + ).strip() + display_name = str( + agent.get("fullBotName") + or agent.get("displayName") + or agent.get("name") + or "" + ).strip() + managed = agent.get("ismanaged") + if managed is None: + managed = agent.get("isManaged") + if managed is None and isinstance(agent.get("managedProperties"), dict): + managed = agent["managedProperties"].get("isManaged") + + identity: dict[str, Any] = {"schemaName": schema_name} + if display_name: + identity["displayName"] = display_name + if isinstance(managed, bool): + identity["isManaged"] = managed return { - "action": "continue-da-ga-setup", - "classification": classification, + "backend": backend, + "outcome": "found", "evidence": evidence, + "productFamily": classify_schema_name(schema_name), + "identity": identity, } -def reconcile_selected_agent( +def probe_native_identity( *, environment_id: str, agent_id: str, @@ -225,20 +295,11 @@ def reconcile_selected_agent( kit_root: Path = Path("."), host: str | None = None, api_version: str = "2024-10-01", - native_da_ga: bool = False, - dataverse_url: str | None = None, - operating_system: str | None = None, ) -> dict[str, Any]: - """Return a bounded routing result; unknown evidence always fails open.""" normalized_environment = _normalize_guid(environment_id, "Environment ID") normalized_agent = _normalize_guid(agent_id, "Agent ID") - - if native_da_ga: - return _continue_result("da-ga", "provided-native") - - cea_evidence = "dataverse-schema" try: - native_agent = _probe_native_agent( + agent = _probe_native_agent( normalized_environment, normalized_agent, ring, @@ -247,93 +308,164 @@ def reconcile_selected_agent( host, api_version, ) - native_classification = classify_schema_name( - native_agent.get("schemaName") - ) - if native_classification == "cea": - classification = "cea" - cea_evidence = "native-schema" - else: - return _continue_result("da-ga", "native-minimal-bot") - except (AgentBuilderError, OSError, ValueError, requests.RequestException): - classification = "unknown" - - if classification != "cea": - resolved_url = dataverse_url - if not resolved_url: - try: - resolved_url = _resolve_dataverse_url( - normalized_environment, - account, - ) - except ( - OSError, - RuntimeError, - SystemExit, - ValueError, - requests.RequestException, - ): - return _continue_result("unknown", "not-classified") - if not resolved_url: - return _continue_result("unknown", "not-classified") + except ( + AgentBuilderError, + OSError, + ValueError, + requests.RequestException, + ) as exc: + return _failure_result("native", exc) + return _identity_summary( + "native", + agent, + evidence="minimalbot-direct", + ) + +def probe_dataverse_identity( + *, + environment_id: str, + agent_id: str, + account: str | None = None, + dataverse_url: str | None = None, +) -> dict[str, Any]: + normalized_environment = _normalize_guid(environment_id, "Environment ID") + normalized_agent = _normalize_guid(agent_id, "Agent ID") + resolved_url = dataverse_url + if not resolved_url: try: - record = _read_dataverse_agent( - resolved_url.rstrip("/"), - normalized_agent, + resolved_url = _resolve_dataverse_url( + normalized_environment, account, ) except ( APIError, - AuthExpiredError, OSError, - SystemExit, + RuntimeError, ValueError, requests.RequestException, - ): - return _continue_result("unknown", "not-classified") + ) as exc: + return _failure_result( + "dataverse", + exc, + stage="environment-resolution", + ) + if not resolved_url: + return { + "backend": "dataverse", + "outcome": "uncertain", + "stage": "environment-resolution", + "error": { + "causes": [ + { + "type": "EnvironmentUrlNotResolved", + "message": ( + "The exact Dataverse URL was not available from " + "the user-scoped environment list." + ), + } + ] + }, + } - if not record: - return _continue_result("unknown", "not-classified") - classification = classify_schema_name( - str(record.get("schemaname") or "") + try: + agent = _read_dataverse_agent( + resolved_url.rstrip("/"), + normalized_agent, + account, ) - if classification != "cea": - return _continue_result(classification, "dataverse-schema") + except ( + APIError, + AuthExpiredError, + OSError, + ValueError, + requests.RequestException, + ) as exc: + return _failure_result("dataverse", exc) + return _identity_summary( + "dataverse", + agent, + evidence="dataverse-direct", + ) - result = { - "action": "stop-and-use-cea-kit", - "classification": "cea", - "evidence": cea_evidence, - } + +def reconcile_selected_agent( + *, + environment_id: str, + agent_id: str, + ring: str, + account: str | None = None, + kit_root: Path = Path("."), + host: str | None = None, + api_version: str = "2024-10-01", + dataverse_url: str | None = None, + probe: str, +) -> dict[str, Any]: + """Run one independent, read-only identity probe.""" + if probe == "native": + return probe_native_identity( + environment_id=environment_id, + agent_id=agent_id, + ring=ring, + account=account, + kit_root=kit_root, + host=host, + api_version=api_version, + ) + if probe == "dataverse": + return probe_dataverse_identity( + environment_id=environment_id, + agent_id=agent_id, + account=account, + dataverse_url=dataverse_url, + ) + raise ValueError("Probe must be 'native' or 'dataverse'.") + + +def known_native_identity(schema_name: str) -> dict[str, Any]: + """Classify identity already proven by a native MinimalBot operation.""" + return _identity_summary( + "native", + {"schemaName": schema_name}, + evidence="provided-native", + ) + + +def compatible_kit_result( + operating_system: str | None = None, +) -> dict[str, Any]: + """Return the pinned compatible-kit installation operation.""" try: command, shell = build_recovery_command(operating_system) release = _load_release() - except ValueError: - result["recoveryUnavailable"] = True - return result - result.update( - { - "releaseTag": release["releaseTag"], - "recoveryCommand": command, - "recoveryShell": shell, + except ValueError as exc: + return { + "outcome": "unavailable", + "error": _exception_evidence(exc), } - ) - return result + return { + "outcome": "available", + "releaseTag": release["releaseTag"], + "recoveryCommand": command, + "recoveryShell": shell, + } def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( - description="Reconcile one selected agent before DA-GA setup." + description="Probe one selected agent before DA-GA setup." ) - parser.add_argument("--environment-id", required=True) - parser.add_argument("--agent-id", required=True) - parser.add_argument("--ring", required=True) + operation = parser.add_mutually_exclusive_group(required=True) + operation.add_argument("--probe", choices=("native", "dataverse")) + operation.add_argument("--known-native-schema") + operation.add_argument("--compatible-kit", action="store_true") + parser.add_argument("--environment-id") + parser.add_argument("--agent-id") + parser.add_argument("--ring") parser.add_argument("--account") parser.add_argument("--kit-root", type=Path, default=Path(".")) parser.add_argument("--host") parser.add_argument("--api-version", default="2024-10-01") - parser.add_argument("--native-da-ga", action="store_true") parser.add_argument("--dataverse-url") parser.add_argument( "--operating-system", @@ -346,18 +478,28 @@ def build_parser() -> argparse.ArgumentParser: def main(argv: list[str] | None = None) -> int: args = build_parser().parse_args(argv) try: - result = reconcile_selected_agent( - environment_id=args.environment_id, - agent_id=args.agent_id, - ring=args.ring, - account=args.account, - kit_root=args.kit_root, - host=args.host, - api_version=args.api_version, - native_da_ga=args.native_da_ga, - dataverse_url=args.dataverse_url, - operating_system=args.operating_system, - ) + if args.compatible_kit: + result = compatible_kit_result(args.operating_system) + elif args.known_native_schema is not None: + result = known_native_identity(args.known_native_schema) + else: + if not args.environment_id or not args.agent_id: + raise ValueError( + "Environment ID and agent ID are required for a probe." + ) + if args.probe == "native" and not args.ring: + raise ValueError("Ring is required for a native probe.") + result = reconcile_selected_agent( + environment_id=args.environment_id, + agent_id=args.agent_id, + ring=args.ring or "", + account=args.account, + kit_root=args.kit_root, + host=args.host, + api_version=args.api_version, + dataverse_url=args.dataverse_url, + probe=args.probe or "", + ) except ValueError as exc: print(f"ERROR: {exc}") return 2 diff --git a/solutions/ess-maker-skills/scripts/setup_existing_da.py b/solutions/ess-maker-skills/scripts/setup_existing_da.py index f5de406f4..8dab047c0 100644 --- a/solutions/ess-maker-skills/scripts/setup_existing_da.py +++ b/solutions/ess-maker-skills/scripts/setup_existing_da.py @@ -191,7 +191,27 @@ def inspect_agent_route( """Return the service-owned route realm for one exact agent.""" normalized_environment_id = _normalize_environment_id(environment_id) normalized_agent_id = _normalize_guid(agent_id, "Agent ID") - realms = client.get_realms(normalized_agent_id) + result = { + "environmentId": normalized_environment_id, + "tenantId": client.tenant_id, + "host": client.host, + "ring": client.ring, + "apiVersion": client.api_version, + "agentId": normalized_agent_id, + } + try: + realms = client.get_realms(normalized_agent_id) + except AgentBuilderHTTPError as exc: + if exc.status_code != 404: + raise + return { + **result, + "realm": None, + "almEnrollment": "not-enrolled", + "statusCode": exc.status_code, + "errorCode": exc.error_code, + "requestId": exc.request_id, + } route_realm = realms.get("routeRealm") realm_name = next( ( @@ -210,13 +230,9 @@ def inspect_agent_route( "Agent realm discovery did not return a recognized route realm." ) return { - "environmentId": normalized_environment_id, - "tenantId": client.tenant_id, - "host": client.host, - "ring": client.ring, - "apiVersion": client.api_version, - "agentId": normalized_agent_id, + **result, "realm": realm_name, + "almEnrollment": "enrolled", } diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md b/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md index f00cffb34..1f857a483 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/SKILL.md @@ -95,6 +95,8 @@ Present account confirmation once per setup invocation. Do not repeat it before Whenever one exact environment and agent has been selected, read `src/skills/foundation-setup/product-line-reconciliation.md` and complete that handoff before the next DA-GA-only operation. This applies regardless of whether the identity came from a supplied URL, active local setup state, a configured-agent switch, environment candidate selection, MOS creation, or ALM import. Run it once per selected identity in this invocation and again only when the selection changes. +For a supplied Copilot Studio URL, retain its exact `agentBackend` query value as an ordering hint for that handoff. Do not infer existence, product family, support, or ALM enrollment from the hint. + ## Shared authorization message The account question is the confirmation for a selected account. When the maker chose the Microsoft account picker, show: @@ -307,10 +309,11 @@ already targets its related Dev agent. When the maker supplies a Copilot Studio URL that identifies an agent and has not explicitly selected package import, infer its environment ID, agent ID, -and service ring. When the URL does not identify the ring, use **Resolve the -service ring** in `src/skills/foundation-setup/da-environment-target.md` -exactly. Ask only when the environment ID or agent ID is unclear. Complete the -selected-agent product-line reconciliation before running: +service ring, and optional `agentBackend` ordering hint. When the URL does not +identify the ring, use **Resolve the service ring** in +`src/skills/foundation-setup/da-environment-target.md` exactly. Ask only when +the environment ID or agent ID is unclear. Complete the selected-agent +product-line reconciliation before running: ```text python scripts/setup_existing_da.py inspect-agent \ @@ -322,6 +325,12 @@ python scripts/setup_existing_da.py inspect-agent \ Parse `DA_AGENT_ROUTE_JSON:`. Do not infer the realm from names, URLs, or environment metadata. +- When `almEnrollment` is `not-enrolled`, the earlier native identity probe + still proves that the agent exists. Say that the agent is not enrolled in + the ALM family required for local authoring, then offer **Choose a different + agent**, **Choose a different environment**, and **Go back** using the exact + recovery routes in `product-line-reconciliation.md`. Do not call the agent + missing and do not continue to validation or attachment. - When `realm` is `prod`, read `src/skills/foundation-setup/da-prod-to-dev.md` and follow it, passing the inspection's internal tenant, environment, host, ring, API version, and agent diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md index 262f94bea..bcdbdab84 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-alm-import.md @@ -54,8 +54,8 @@ When `kind` is `success`, use the returned environment, tenant, host, ring, API version, and agent identity only as internal command inputs. Do not display those identifiers. The successful native import verification is authoritative DA-GA evidence, so run the parent's selected-agent product-line reconciliation -with `--native-da-ga`. Mark **Verify access and agent identity** and **Establish -an editable Dev agent** complete, then show: +with `--known-native-schema "{RETURNED_SCHEMA_NAME}"`. Mark **Verify access and +agent identity** and **Establish an editable Dev agent** complete, then show: > Agent package imported and verified as an editable Dev agent. Preparing its > local authoring workspace... @@ -111,7 +111,7 @@ Offer exactly: - **Replace an existing agent with this package** - **Go back** -Do not preselect a choice or recommend replacement. For either existing-agent choice, run `setup_existing_da.py list-agents` for the target environment, show the visible Dev agent names, and let the maker choose one exact agent. The native candidate result is authoritative DA-GA evidence; run the parent's selected-agent product-line reconciliation with `--native-da-ga` before validation or replacement. For **Choose an existing agent in this environment**, validate the selected agent and continue through `da-existing-dev.md`. +Do not preselect a choice or recommend replacement. For either existing-agent choice, run `setup_existing_da.py list-agents` for the target environment, show the visible Dev agent names, and let the maker choose one exact agent. The native candidate result is authoritative identity evidence; run the parent's selected-agent product-line reconciliation with `--known-native-schema "{RETURNED_SCHEMA_NAME}"` before validation or replacement. For **Choose an existing agent in this environment**, validate the selected agent and continue through `da-existing-dev.md`. For **Go back**, retain the current account, environment, and ring and return to the parent skill's **What would you like to set up in this environment?** choice surface. Clear package-import and replacement intent, but preserve the import receipt and collision evidence. diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-existing-dev.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-existing-dev.md index b38d16d7e..eea85054b 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-existing-dev.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-existing-dev.md @@ -29,7 +29,10 @@ python scripts/setup_existing_da.py inspect-agent \ Parse `DA_AGENT_ROUTE_JSON:`. Continue only when the current invocation has a service result reporting `realm: dev`. Do not infer the realm from the URL, agent name, environment metadata, canonical setup state, or conversation -history. If the service reports another realm, explain that this setup path +history. When `almEnrollment` is `not-enrolled`, preserve that the native agent +exists, explain that it is not enrolled for ALM-based local authoring, and use +the parent's alternate-agent, alternate-environment, and **Go back** recovery +routes. If the service reports another realm, explain that this setup path requires an editable Dev agent and stop. After a Dev result, show: @@ -86,13 +89,13 @@ python scripts/setup_existing_da.py list-agents \ --ring "{RING}" ``` -When candidates are returned, show their display names and ask the maker to choose one. For an identity returned by this native list, run the parent's selected-agent product-line reconciliation with `--native-da-ga` before validation or attachment. Validate only the selected candidate through `validate-agent` or `attach`. +When candidates are returned, show their display names and ask the maker to choose one. For an identity returned by this native list, run the parent's selected-agent product-line reconciliation with `--known-native-schema "{RETURNED_SCHEMA_NAME}"` before validation or attachment. Validate only the selected candidate through `validate-agent` or `attach`. When the list is empty, say: > No visible editable Dev agents were listed in this environment. A directly addressable agent may still be available. -Present **Retry setup with another target** from `da-environment-target.md`, including its **Use an agent URL** choice. For an exact agent selected through that URL, run the full selected-agent product-line reconciliation without `--native-da-ga` before validating it directly. +Present **Retry setup with another target** from `da-environment-target.md`, including its **Use an agent URL** choice. For an exact agent selected through that URL, run both independent identity probes in the full selected-agent product-line reconciliation before validating it directly. ## Maintain native FlightCheck evidence diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md index 3643a14d5..90fca52be 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-mos-starter.md @@ -114,7 +114,7 @@ The response, outcome label, fuse disposition, HTTP status, and request details When the annotations report `outcome: created`, keep the distinction between `catalogPackageVersion` and `templateVersion` in diagnostic evidence; do not explain those internal version concepts to the maker. Say that the new agent was created and setup is not complete. -The successful native create result is authoritative DA-GA evidence. Run the parent's selected-agent product-line reconciliation with the returned identity and `--native-da-ga` before the enable-ALM operation. +The successful native create result is authoritative identity evidence. Run the parent's selected-agent product-line reconciliation with the returned identity and `--known-native-schema "{RETURNED_SCHEMA_NAME}"` before the enable-ALM operation. When the annotations report `outcome: collision`, do not infer which visible agent corresponds to the package. List visible Dev agents in the same environment through `setup_existing_da.py list-agents`, then offer exactly: @@ -122,7 +122,7 @@ When the annotations report `outcome: collision`, do not infer which visible age - **Choose a different catalog product** - **Go back** -Do not preselect a choice. For **Choose an existing agent in this environment**, show the returned names, let the maker select one exact agent, run the parent's selected-agent product-line reconciliation with `--native-da-ga`, and continue through `da-existing-dev.md`. The selected agent is maker-supplied intent, not proof of package identity. This path does not replace an agent. +Do not preselect a choice. For **Choose an existing agent in this environment**, show the returned names, let the maker select one exact agent, run the parent's selected-agent product-line reconciliation with `--known-native-schema "{RETURNED_SCHEMA_NAME}"`, and continue through `da-existing-dev.md`. The selected agent is maker-supplied intent, not proof of package identity. This path does not replace an agent. For **Choose a different catalog product**, present the valid rows from the latest successful catalog result and let the maker select another exact product. Continue through **Confirm the exact product and target** for that selection. A new create request becomes available only after the maker confirms the new product and uses a new client request UUID. diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md b/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md index 080f9faba..943eed328 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/da-prod-to-dev.md @@ -159,8 +159,9 @@ relationship from the failed Prod operation. ## Attach and complete On a validated related-Dev path or successful import result, run the parent's -selected-agent product-line reconciliation with `--native-da-ga` for the -returned identity, then attach it: +selected-agent product-line reconciliation with +`--known-native-schema "{RETURNED_SCHEMA_NAME}"` for the returned identity, +then attach it: ```text python scripts/setup_existing_da.py attach \ diff --git a/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md b/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md index 1c584e08d..2ca3ef950 100644 --- a/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md +++ b/solutions/ess-maker-skills/src/skills/foundation-setup/product-line-reconciliation.md @@ -1,68 +1,162 @@ + # Reconcile the Selected Agent Product Line Run this handoff whenever setup has selected one exact environment and agent, regardless of whether the identity came from a supplied Copilot Studio URL, active local setup state, a configured-agent switch, environment candidate selection, MOS creation, or ALM import. Run it once per `(environmentId, agentId)` in one setup invocation and run it again only when that selected identity changes. -This check is read-only. It may prove that the selected agent belongs to the native DA-GA service or to a recognized solution-backed product. URL shape, query parameters, display names, generic API failures, and environment-level solution presence are not product evidence. +This check is read-only. The script exposes independent identity probes and preserves each service result. This skill owns probe ordering, combines the observations, interprets product support, and presents recovery choices. URL shape, query parameters, display names, generic API failures, and environment-level solution presence are not product evidence. + +## Use authoritative native evidence + +When a current-invocation MinimalBot operation already returned and validated the exact identity, classify its returned schema without repeating either remote lookup: + +```text +python scripts/reconcile_setup_agent.py \ + --known-native-schema "{RETURNED_SCHEMA_NAME}" +``` + +Parse `DA_SETUP_PRODUCT_RECONCILIATION_JSON:` as one native `found` observation, then apply **Interpret the observations** below. + +## Probe an externally supplied identity + +For an identity that was not already proven by a native operation, run both independent probes. The Copilot Studio URL's `agentBackend` query value is an ordering hint only: -## Run the reconciliation +- `dataverse` means run the Dataverse probe first. +- `cosmos` means run the native probe first. +- A missing or unrecognized value means run the Dataverse probe first. -For an identity already returned and validated by a native MinimalBot operation, run: +Do not infer product family, existence, support, or ALM enrollment from this hint. Run the second probe even when the first probe returns `found`; two `found` observations must remain distinguishable. + +Run the Dataverse probe: ```text python scripts/reconcile_setup_agent.py \ + --probe dataverse \ --environment-id "{ENVIRONMENT_ID}" \ - --agent-id "{AGENT_ID}" \ - --ring "{RING}" \ - --native-da-ga + --agent-id "{AGENT_ID}" ``` -Otherwise run: +Append the parent setup skill's confirmed `--account` when available. Append `--dataverse-url` only when the selected environment's exact Dataverse URL is already authoritative setup input. Do not ask the maker for a Dataverse URL solely for this check. + +Run the native MinimalBot probe: ```text python scripts/reconcile_setup_agent.py \ + --probe native \ --environment-id "{ENVIRONMENT_ID}" \ --agent-id "{AGENT_ID}" \ --ring "{RING}" ``` -Append the parent setup skill's confirmed `--account`, validated `--host`, and `--api-version` values when available. Append `--dataverse-url` only when the selected environment's exact Dataverse URL is already authoritative setup input. Do not ask the maker for a Dataverse URL solely for this check. +Append the parent setup skill's confirmed `--account`, validated `--host`, and `--api-version` values when available. + +Parse one `DA_SETUP_PRODUCT_RECONCILIATION_JSON:` result from each command. Preserve `backend`, `outcome`, `stage`, `productFamily`, `identity`, and `error` as internal evidence. The probe outcomes are: + +- `found` — that identity store returned the exact agent. +- `not-found` — that exact identity endpoint returned HTTP 404. +- `authentication-required` — the endpoint returned HTTP 401. +- `access-denied` — the endpoint returned HTTP 403. +- `uncertain` — transport, service, response-shape, environment-resolution, or other evidence could not establish the result. + +Do not convert `authentication-required`, `access-denied`, or `uncertain` into `not-found`. + +## Interpret the observations + +Schema classification is case-insensitive and prefix-based: + +- `msdyn_copilotforemployeeselfservice*` is the solution-backed Employee Self-Service family supported by the compatible `main-ca` kit. +- `gptagent_copilotforemployeeselfservice*` is the DA-GA family. +- Any other or missing schema is custom or unknown. + +Apply the first matching rule: + +1. When both probes returned `found`, stop before any realm or ALM operation. Say that both supported identity stores returned an agent for the same ID, so setup cannot safely choose a backend. Preserve both observations; do not select one from the URL hint. +2. When exactly one probe returned `found`, that observation proves existence and backend even if the other probe returned `not-found`, an access outcome, or an uncertainty outcome. Preserve a backend-hint mismatch as internal evidence, use the backend that actually returned the agent, and do not call the agent missing. +3. For a native `found` DA-GA observation, continue at the next DA-GA setup operation. Only this result makes native ALM realm inspection applicable. +4. For a Dataverse-only `found` DA-GA observation, stop before native inspection. Say that the agent belongs to the Employee Self-Service DA family but was found only in Dataverse, and this native setup path cannot safely prepare it for local authoring. +5. For any `found` solution-backed Employee Self-Service observation, follow **Use the compatible kit** below. This includes classic CA and DA-Preview variants. +6. For any `found` custom or unknown observation, follow **Unsupported agent** below. +7. When both probes returned `not-found`, say that the exact agent was not found in either accessible identity store. +8. When neither probe returned `found` and at least one returned `authentication-required` or `access-denied`, state the endpoint-specific sign-in or permission blocker. Do not claim the agent is missing. +9. Otherwise state that setup could not establish the agent's identity because one or more lookups were uncertain. Do not continue to realm inspection. + +Rules 1, 4, 6, 7, 8, and 9 use the same recovery choices and routes defined under **Unsupported agent**. For every stopped result, render **Choose the starting point and target environment** as complete, **Verify access and agent identity** as blocked, **Establish an editable Dev agent** and **Materialize the local workspace** as pending, and **Review the setup handoff** as in progress while the recovery choice is pending. + +## Use the compatible kit + +Before offering installation, obtain the pinned compatible-kit operation: + +```text +python scripts/reconcile_setup_agent.py --compatible-kit +``` Parse `DA_SETUP_PRODUCT_RECONCILIATION_JSON:`. -- For `action: continue-da-ga-setup`, continue at the next DA-GA setup operation. `classification: unknown` is deliberately fail-open; do not persist it or tell the maker that DA-GA was proven. -- For `action: stop-and-use-cea-kit`, stop before inspection, validation, attachment, import, Object Model installation, or other DA-GA-only work. Render **Choose the starting point and target environment** as complete, **Verify access and agent identity** as blocked, **Establish an editable Dev agent** and **Materialize the local workspace** as pending, and **Review the setup handoff** as in progress while the kit-switch choice is pending. +After the command returns, proceed directly to the required setup checklist render and the applicable exact maker-facing message below. Do not emit an operational progress line between them. Do not emit **Verified replacement agent identities and resolved compatible ESS kit** or narrate completed probes, schema classification, or compatible-kit resolution. + +When `outcome` is `unavailable`, say that the compatible pinned installer could not be resolved and render **Review the setup handoff** as blocked. Do not guess a branch or release. Ask **How would you like to continue setup?** and offer exactly: + +- **Choose a different agent** +- **Choose a different environment** +- **Go back** + +Do not offer **Install and open the compatible kit**. Follow the shared recovery routes under **Unsupported agent**. + +When `outcome` is `available`, send this exact Message block as its own completed chat message. Substitute the display name without adding Markdown emphasis around it: + +**Message:** + +{agent display name or Selected agent} belongs to a legacy agent family, so this Developer Kit cannot safely continue its setup. + +**End message.** + +After completing that message, use the `vscode_askQuestions` tool with this exact single-selection control: + +```json +[ + { + "header": "Continue setup", + "question": "How would you like to continue setup?", + "options": [ + { "label": "Install and open the compatible kit" }, + { "label": "Choose a different agent" }, + { "label": "Choose a different environment" }, + { "label": "Go back" } + ], + "allowFreeformInput": false + } +] +``` + +Leave the selection initially unset. Do not add `recommended`, `default`, or any equivalent preselection to an option. Keyboard focus or a visual highlight is not a selected value; wait for the maker to submit an explicit choice before continuing setup. + +- For **Install and open the compatible kit**, run `recoveryCommand` in a terminal using `recoveryShell`. On success, render **Review the setup handoff** as complete and say: - When `recoveryUnavailable` is `true`, say that the compatible pinned installer could not be resolved and render **Review the setup handoff** as blocked. Do not guess a branch or release. Use the host's interactive single-selection control, ask **How would you like to continue setup?**, and offer exactly: - - **Choose a different agent** - - **Choose a different environment** - - **Go back** + > Continue setup in the workspace opened by the compatible installer. No Copilot Studio agent or setup state was changed by this product-line check. - Do not offer **Install and open the compatible kit** when its pinned installer is unavailable. Follow the corresponding Setup routes below. +- Apply the three shared recovery routes below for the other choices. +- When command execution fails, render **Review the setup handoff** as blocked and state the observed installer failure. Do not label the unchanged command as a retry or reinterpret installer failure as a successful handoff. When the output identifies an existing installation, verify that its checkout matches `releaseTag` and offer to open its `solutions/ess-maker-skills` workspace directly. Otherwise provide one recovery action grounded in the observed failure. - Otherwise say: +## Unsupported agent - > **{agent display name or Selected agent}** belongs to the classic Employee Self-Service agent product line, so this Developer Kit cannot safely continue its setup. +For a custom or unknown `found` observation, say: - Then use the host's interactive single-selection control and ask exactly: +> **{agent display name or Selected agent}** was found, but it is not part of a supported Employee Self-Service agent family for this Developer Kit. - > How would you like to continue setup? +Use the host's interactive single-selection control and ask exactly: - Offer exactly: - - **Install and open the compatible kit** - - **Choose a different agent** - - **Choose a different environment** - - **Go back** +> How would you like to continue setup? - Do not preselect a choice. - - For **Install and open the compatible kit**, run `recoveryCommand` in a terminal using `recoveryShell`. On success, render **Review the setup handoff** as complete and say: +Offer exactly: - > Continue setup in the workspace opened by the compatible installer. No Copilot Studio agent or setup state was changed by this product-line check. +- **Choose a different agent** +- **Choose a different environment** +- **Go back** - - For **Choose a different agent**, retain the current account, environment, and ring. Read `src/skills/foundation-setup/da-existing-dev.md` and continue from its environment-scoped `list-agents` candidate-selection path. Do not persist the rejected agent or change the active local agent before another exact candidate passes product-line reconciliation. - - For **Choose a different environment**, retain the current account and ring. Read `src/skills/foundation-setup/da-environment-target.md`, rerun `list-environments`, and continue from its environment picker. A target in another environment follows the parent skill's new-workspace contract. - - For **Go back**, do not persist the rejected agent. Return to **Choose the sign-in account** in the parent skill. After the maker selects an account, read `src/skills/foundation-setup/da-environment-target.md`, list that account's environments, and continue from the selected environment through the parent skill's create-or-connect choice. The direct different-agent and different-environment routes above retain the current account; **Go back** is the account-reset route. +Do not preselect a choice. - - When command execution fails, render **Review the setup handoff** as blocked and state the observed installer failure. Do not label the unchanged command as a retry or reinterpret installer failure as a successful handoff. When the output identifies an existing installation, verify that its checkout matches `releaseTag` and offer to open its `solutions/ess-maker-skills` workspace directly. Otherwise provide one recovery action grounded in the observed failure. +- For **Choose a different agent**, retain the current account, environment, and ring. Read `src/skills/foundation-setup/da-existing-dev.md` and continue from its environment-scoped `list-agents` candidate-selection path. Do not persist the rejected agent or change the active local agent before another exact candidate passes product-line reconciliation. +- For **Choose a different environment**, retain the current account and ring. Read `src/skills/foundation-setup/da-environment-target.md`, rerun `list-environments`, and continue from its environment picker. A target in another environment follows the parent skill's new-workspace contract. +- For **Go back**, do not persist the rejected agent. Return to **Choose the sign-in account** in the parent skill. After the maker selects an account, read `src/skills/foundation-setup/da-environment-target.md`, list that account's environments, and continue from the selected environment through the parent skill's create-or-connect choice. The direct different-agent and different-environment routes above retain the current account; **Go back** is the account-reset route. -Do not expose the schema name, agent ID, environment ID, API host, or raw classifier failures. Do not reinterpret a missing or failed lookup as classic CEA evidence. +Do not expose the schema name, agent ID, environment ID, API host, raw probe failure, or URL backend hint. Do not reinterpret a missing or failed lookup as product evidence. diff --git a/tests/scripts/test_reconcile_setup_agent.py b/tests/scripts/test_reconcile_setup_agent.py index 56ee610fa..1aa28e68f 100644 --- a/tests/scripts/test_reconcile_setup_agent.py +++ b/tests/scripts/test_reconcile_setup_agent.py @@ -9,7 +9,8 @@ import pytest import reconcile_setup_agent as reconcile -from agentbuilder import AgentBuilderError +from agentbuilder import AgentBuilderError, AgentBuilderHTTPError +from http_errors import APIError ENVIRONMENT_ID = "00000000-0000-4000-8000-000000001111" @@ -20,203 +21,174 @@ @pytest.mark.parametrize( ("schema_name", "expected"), [ - ("msdyn_copilotforemployeeselfservice", "cea"), - ("msdyn_copilotforemployeeselfservicecore", "cea"), - ("msdyn_copilotforemployeeselfservicehr", "cea"), - ("msdyn_copilotforemployeeselfserviceit", "cea"), - ("gptagent_copilotforemployeeselfservice", "da"), - ("gptagent_copilotforemployeeselfservicecore", "da"), - ("gptagent_copilotforemployeeselfservicehr", "da"), - ("gptagent_copilotforemployeeselfserviceit", "da"), + ( + "msdyn_copilotforemployeeselfservice", + "solution-backed-ess", + ), + ( + "msdyn_copilotforemployeeselfservicecore", + "solution-backed-ess", + ), + ( + "msdyn_copilotforemployeeselfservicedahr", + "solution-backed-ess", + ), + ( + "msdyn_copilotforemployeeselfservicecustomsuffix", + "solution-backed-ess", + ), + ( + "gptagent_copilotforemployeeselfservice", + "da-ga", + ), + ( + "gptagent_copilotforemployeeselfservicehr", + "da-ga", + ), + ( + "gptagent_copilotforemployeeselfservicecustomsuffix", + "da-ga", + ), ("gptagent_employee_self_service", "unknown"), - ("MSDYN_COPILOTFOREMPLOYEESELFSERVICEHR", "cea"), + ( + "MSDYN_COPILOTFOREMPLOYEESELFSERVICEDAHR", + "solution-backed-ess", + ), + ( + "GPTAGENT_COPILOTFOREMPLOYEESELFSERVICEIT", + "da-ga", + ), (None, "unknown"), ], ) -def test_classifies_only_exact_recognized_schema_names( +def test_classifies_supported_schema_prefixes( schema_name: str | None, expected: str, ) -> None: assert reconcile.classify_schema_name(schema_name) == expected -def test_authoritative_native_evidence_short_circuits_remote_probes( - monkeypatch: pytest.MonkeyPatch, -) -> None: - monkeypatch.setattr( - reconcile, - "_probe_native_agent", - lambda *_args, **_kwargs: pytest.fail("native lookup should be skipped"), - ) - monkeypatch.setattr( - reconcile, - "_resolve_dataverse_url", - lambda *_args, **_kwargs: pytest.fail("Dataverse lookup should be skipped"), - ) - - result = reconcile.reconcile_selected_agent( - environment_id=ENVIRONMENT_ID, - agent_id=AGENT_ID, - ring="prod", - native_da_ga=True, +def test_known_native_identity_preserves_product_evidence() -> None: + result = reconcile.known_native_identity( + "gptagent_copilotforemployeeselfservicehr" ) assert result == { - "action": "continue-da-ga-setup", - "classification": "da-ga", + "backend": "native", + "outcome": "found", "evidence": "provided-native", + "productFamily": "da-ga", + "identity": { + "schemaName": "gptagent_copilotforemployeeselfservicehr", + }, } -def test_live_native_identity_is_authoritative( +def test_native_probe_returns_identity_without_dataverse_probe( monkeypatch: pytest.MonkeyPatch, ) -> None: monkeypatch.setattr( reconcile, "_probe_native_agent", lambda *_args, **_kwargs: { - "schemaName": "gptagent_copilotforemployeeselfservicehr" + "fullBotName": "Employee Self-Service HR", + "schemaName": "gptagent_copilotforemployeeselfservicehr", + "managedProperties": {"isManaged": True}, }, ) monkeypatch.setattr( reconcile, "_resolve_dataverse_url", - lambda *_args, **_kwargs: pytest.fail("Dataverse lookup should be skipped"), + lambda *_args, **_kwargs: pytest.fail( + "Each probe must remain independent." + ), ) result = reconcile.reconcile_selected_agent( environment_id=ENVIRONMENT_ID, agent_id=AGENT_ID, ring="prod", + probe="native", ) - assert result["classification"] == "da-ga" - assert result["evidence"] == "native-minimal-bot" + assert result == { + "backend": "native", + "outcome": "found", + "evidence": "minimalbot-direct", + "productFamily": "da-ga", + "identity": { + "displayName": "Employee Self-Service HR", + "schemaName": "gptagent_copilotforemployeeselfservicehr", + "isManaged": True, + }, + } -def test_positive_cea_schema_stops_with_release_pinned_recovery( +@pytest.mark.parametrize( + ("status_code", "outcome"), + [ + (401, "authentication-required"), + (403, "access-denied"), + (404, "not-found"), + (500, "uncertain"), + ], +) +def test_native_probe_preserves_http_failure_class( monkeypatch: pytest.MonkeyPatch, + status_code: int, + outcome: str, ) -> None: def fail_native(*_args, **_kwargs): - raise AgentBuilderError("not available through the native service") + raise AgentBuilderHTTPError( + "Direct agent lookup", + status_code, + error_code="ObjectNotFound" if status_code == 404 else "Failure", + request_id="request-123", + ) monkeypatch.setattr(reconcile, "_probe_native_agent", fail_native) - monkeypatch.setattr( - reconcile, - "_resolve_dataverse_url", - lambda *_args, **_kwargs: DATAVERSE_URL, - ) - monkeypatch.setattr( - reconcile, - "_read_dataverse_agent", - lambda *_args, **_kwargs: { - "schemaname": "msdyn_copilotforemployeeselfserviceit" - }, - ) - result = reconcile.reconcile_selected_agent( + result = reconcile.probe_native_identity( environment_id=ENVIRONMENT_ID, agent_id=AGENT_ID, ring="prod", - operating_system="Windows", ) - assert result["action"] == "stop-and-use-cea-kit" - assert result["classification"] == "cea" - assert result["releaseTag"] == "cea-v1.0.0-rc.1" - assert "/cea-v1.0.0-rc.1/setup/bootstrap.ps1" in result["recoveryCommand"] - assert "-Branch cea-v1.0.0-rc.1" in result["recoveryCommand"] - assert ( - "-SourceBaseUrl https://raw.githubusercontent.com/" - "microsoft/Employee-Self-Service-Agent-Developer-Kit/" - "cea-v1.0.0-rc.1/setup" - ) in result["recoveryCommand"] - assert ( - "-InstallRoot (Join-Path $env:USERPROFILE 'source-cea')" - in result["recoveryCommand"] - ) - assert result["recoveryShell"] == "powershell" + assert result["backend"] == "native" + assert result["outcome"] == outcome + assert result["stage"] == "agent-lookup" + assert result["error"]["statusCode"] == status_code + assert result["error"]["errorCode"] + assert result["error"]["requestId"] == "request-123" + assert result["error"]["causes"][0]["type"] == "AgentBuilderHTTPError" -def test_non_cea_and_unknown_evidence_continue( +def test_native_transport_failure_is_uncertain_and_keeps_cause_chain( monkeypatch: pytest.MonkeyPatch, ) -> None: - monkeypatch.setattr( - reconcile, - "_probe_native_agent", - lambda *_args, **_kwargs: (_ for _ in ()).throw( - AgentBuilderError("native lookup failed") - ), - ) - monkeypatch.setattr( - reconcile, - "_resolve_dataverse_url", - lambda *_args, **_kwargs: DATAVERSE_URL, - ) - records = iter( - [ - {"schemaname": "gptagent_copilotforemployeeselfservicehr"}, - {"schemaname": "contoso_unrecognized"}, - ] - ) - monkeypatch.setattr( - reconcile, - "_read_dataverse_agent", - lambda *_args, **_kwargs: next(records), - ) - - da_schema = reconcile.reconcile_selected_agent( - environment_id=ENVIRONMENT_ID, - agent_id=AGENT_ID, - ring="prod", - ) - unknown = reconcile.reconcile_selected_agent( - environment_id=ENVIRONMENT_ID, - agent_id=AGENT_ID, - ring="prod", - ) - - assert da_schema == { - "action": "continue-da-ga-setup", - "classification": "da", - "evidence": "dataverse-schema", - } - assert unknown == { - "action": "continue-da-ga-setup", - "classification": "unknown", - "evidence": "dataverse-schema", - } + try: + raise OSError("socket closed") + except OSError as cause: + failure = AgentBuilderError("native lookup unavailable") + failure.__cause__ = cause - -def test_lookup_failures_are_not_cea_evidence( - monkeypatch: pytest.MonkeyPatch, -) -> None: monkeypatch.setattr( reconcile, "_probe_native_agent", - lambda *_args, **_kwargs: (_ for _ in ()).throw( - AgentBuilderError("service unavailable") - ), - ) - monkeypatch.setattr( - reconcile, - "_resolve_dataverse_url", - lambda *_args, **_kwargs: (_ for _ in ()).throw( - RuntimeError("authentication unavailable") - ), + lambda *_args, **_kwargs: (_ for _ in ()).throw(failure), ) - result = reconcile.reconcile_selected_agent( + result = reconcile.probe_native_identity( environment_id=ENVIRONMENT_ID, agent_id=AGENT_ID, ring="prod", ) - assert result == { - "action": "continue-da-ga-setup", - "classification": "unknown", - "evidence": "not-classified", - } + assert result["outcome"] == "uncertain" + assert [item["type"] for item in result["error"]["causes"]] == [ + "AgentBuilderError", + "OSError", + ] def test_environment_resolution_uses_user_scoped_api_and_selected_account( @@ -254,7 +226,63 @@ def list_environments_for_user(self): } -def test_dataverse_lookup_is_bounded_to_exact_agent_and_account( +@pytest.mark.parametrize( + ("status_code", "outcome"), + [(401, "authentication-required"), (403, "access-denied")], +) +def test_environment_resolution_preserves_permission_failure( + monkeypatch: pytest.MonkeyPatch, + status_code: int, + outcome: str, +) -> None: + class FakeClient: + def __init__(self, _tenant: str) -> None: + pass + + def authenticate(self, preferred_username=None): + return "token" + + def list_environments_for_user(self): + return { + "_error": "insufficient_permissions", + "_status": status_code, + } + + monkeypatch.setattr(reconcile, "PowerPlatformClient", FakeClient) + + result = reconcile.probe_dataverse_identity( + environment_id=ENVIRONMENT_ID, + agent_id=AGENT_ID, + ) + + assert result["outcome"] == outcome + assert result["stage"] == "environment-resolution" + assert result["error"]["statusCode"] == status_code + + +def test_missing_dataverse_url_is_uncertain_not_not_found( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + reconcile, + "_resolve_dataverse_url", + lambda *_args, **_kwargs: None, + ) + + result = reconcile.probe_dataverse_identity( + environment_id=ENVIRONMENT_ID, + agent_id=AGENT_ID, + ) + + assert result["outcome"] == "uncertain" + assert result["stage"] == "environment-resolution" + assert ( + result["error"]["causes"][0]["type"] + == "EnvironmentUrlNotResolved" + ) + + +def test_dataverse_lookup_is_exact_and_preserves_selected_fields( monkeypatch: pytest.MonkeyPatch, ) -> None: observed: dict[str, object] = {} @@ -263,47 +291,100 @@ def fake_authenticate(env_url, preferred_username=None): observed["auth"] = (env_url, preferred_username) return "token" - def fake_query_all(env_url, token, **kwargs): - observed["query"] = (env_url, token, kwargs) - return [ - { - "botid": AGENT_ID, - "name": "ESS", - "schemaname": "msdyn_CopilotForEmployeeSelfServiceIT", - "ismanaged": True, - } - ] + def fake_dataverse_get(env_url, token, path, params): + observed["query"] = (env_url, token, path, params) + return { + "botid": AGENT_ID, + "name": "Employee Self-Service IT", + "schemaname": "msdyn_CopilotForEmployeeSelfServiceDAIT", + "ismanaged": True, + } monkeypatch.setattr(reconcile, "authenticate", fake_authenticate) - monkeypatch.setattr(reconcile, "query_all", fake_query_all) + monkeypatch.setattr(reconcile, "dataverse_get", fake_dataverse_get) - record = reconcile._read_dataverse_agent( - DATAVERSE_URL, - AGENT_ID, - "maker@example.com", + result = reconcile.probe_dataverse_identity( + environment_id=ENVIRONMENT_ID, + agent_id=AGENT_ID, + account="maker@example.com", + dataverse_url=DATAVERSE_URL, ) - assert record is not None assert observed["auth"] == (DATAVERSE_URL, "maker@example.com") assert observed["query"] == ( DATAVERSE_URL, "token", - { - "entity_set": "bots", - "select": "botid,name,schemaname,ismanaged", - "filter_expr": f"botid eq {AGENT_ID}", + f"bots({AGENT_ID})", + {"$select": "botid,name,schemaname,ismanaged"}, + ) + assert result == { + "backend": "dataverse", + "outcome": "found", + "evidence": "dataverse-direct", + "productFamily": "solution-backed-ess", + "identity": { + "displayName": "Employee Self-Service IT", + "schemaName": "msdyn_CopilotForEmployeeSelfServiceDAIT", + "isManaged": True, }, + } + + +@pytest.mark.parametrize( + ("status_code", "outcome"), + [ + (401, "authentication-required"), + (403, "access-denied"), + (404, "not-found"), + (503, "uncertain"), + ], +) +def test_dataverse_probe_preserves_http_failure_class( + monkeypatch: pytest.MonkeyPatch, + status_code: int, + outcome: str, +) -> None: + monkeypatch.setattr( + reconcile, + "_read_dataverse_agent", + lambda *_args, **_kwargs: (_ for _ in ()).throw( + APIError( + status_code=status_code, + message="Dataverse lookup failed", + tip="", + ) + ), ) + result = reconcile.probe_dataverse_identity( + environment_id=ENVIRONMENT_ID, + agent_id=AGENT_ID, + dataverse_url=DATAVERSE_URL, + ) -def test_posix_recovery_is_release_pinned_and_side_by_side() -> None: - command, shell = reconcile.build_recovery_command("Darwin") + assert result["backend"] == "dataverse" + assert result["outcome"] == outcome + assert result["error"]["statusCode"] == status_code - assert shell == "bash" - assert 'ESS_ADK_INSTALL_ROOT="$HOME/source-cea"' in command - assert "/cea-v1.0.0-rc.1/setup/bootstrap-mac.sh" in command - assert "--branch cea-v1.0.0-rc.1" in command - assert "--source-base-url" in command + +def test_compatible_kit_result_is_release_pinned_and_side_by_side() -> None: + windows = reconcile.compatible_kit_result("Windows") + posix = reconcile.compatible_kit_result("Darwin") + + assert windows["outcome"] == "available" + assert windows["releaseTag"] == "cea-v1.0.0-rc.1" + assert "/cea-v1.0.0-rc.1/setup/bootstrap.ps1" in windows[ + "recoveryCommand" + ] + assert "-Branch cea-v1.0.0-rc.1" in windows["recoveryCommand"] + assert ( + "-InstallRoot (Join-Path $env:USERPROFILE 'source-cea')" + in windows["recoveryCommand"] + ) + assert posix["recoveryShell"] == "bash" + assert 'ESS_ADK_INSTALL_ROOT="$HOME/source-cea"' in posix[ + "recoveryCommand" + ] def test_release_manifest_rejects_a_moving_branch(tmp_path: Path) -> None: @@ -329,18 +410,13 @@ def test_release_manifest_rejects_a_moving_branch(tmp_path: Path) -> None: reconcile.build_recovery_command("Windows", manifest) -def test_cli_emits_bounded_json_for_native_evidence( +def test_cli_emits_one_bounded_known_native_result( capsys: pytest.CaptureFixture[str], ) -> None: exit_code = reconcile.main( [ - "--environment-id", - ENVIRONMENT_ID, - "--agent-id", - AGENT_ID, - "--ring", - "prod", - "--native-da-ga", + "--known-native-schema", + "gptagent_copilotforemployeeselfservicehr", ] ) @@ -348,21 +424,23 @@ def test_cli_emits_bounded_json_for_native_evidence( assert exit_code == 0 assert output.startswith(reconcile.RESULT_PREFIX) result = json.loads(output.removeprefix(reconcile.RESULT_PREFIX)) - assert result["action"] == "continue-da-ga-setup" - assert ENVIRONMENT_ID not in output - assert AGENT_ID not in output + assert result["backend"] == "native" + assert result["outcome"] == "found" + assert result["productFamily"] == "da-ga" -def test_positive_cea_result_still_stops_when_recovery_is_unavailable( +def test_cli_rejects_probe_without_exact_identity( + capsys: pytest.CaptureFixture[str], +) -> None: + exit_code = reconcile.main(["--probe", "native", "--ring", "prod"]) + + assert exit_code == 2 + assert "Environment ID and agent ID are required" in capsys.readouterr().out + + +def test_compatible_kit_failure_is_not_success_shaped( monkeypatch: pytest.MonkeyPatch, ) -> None: - monkeypatch.setattr( - reconcile, - "_probe_native_agent", - lambda *_args, **_kwargs: { - "schemaName": "msdyn_copilotforemployeeselfservicehr" - }, - ) monkeypatch.setattr( reconcile, "build_recovery_command", @@ -371,15 +449,8 @@ def test_positive_cea_result_still_stops_when_recovery_is_unavailable( ), ) - result = reconcile.reconcile_selected_agent( - environment_id=ENVIRONMENT_ID, - agent_id=AGENT_ID, - ring="prod", - ) + result = reconcile.compatible_kit_result("Windows") - assert result == { - "action": "stop-and-use-cea-kit", - "classification": "cea", - "evidence": "native-schema", - "recoveryUnavailable": True, - } + assert result["outcome"] == "unavailable" + assert "recoveryCommand" not in result + assert result["error"]["causes"][0]["type"] == "ValueError" diff --git a/tests/scripts/test_setup_existing_da.py b/tests/scripts/test_setup_existing_da.py index a2f7f244b..a74b1e087 100644 --- a/tests/scripts/test_setup_existing_da.py +++ b/tests/scripts/test_setup_existing_da.py @@ -494,10 +494,37 @@ def test_inspect_agent_route_returns_service_realm( ) assert result["realm"] == expected + assert result["almEnrollment"] == "enrolled" assert result["agentId"] == AGENT_ID assert result["tenantId"] == TENANT_ID +def test_inspect_agent_route_distinguishes_missing_alm_enrollment() -> None: + client = FakeClient() + + def missing_realms(_agent_id: str) -> dict[str, Any]: + raise setup_existing_da.AgentBuilderHTTPError( + "Agent realm family", + 404, + error_code="ObjectNotFound", + request_id="request-123", + ) + + client.get_realms = missing_realms # type: ignore[method-assign] + + result = setup_existing_da.inspect_agent_route( + client, + environment_id=ENVIRONMENT_ID, + agent_id=AGENT_ID, + ) + + assert result["realm"] is None + assert result["almEnrollment"] == "not-enrolled" + assert result["statusCode"] == 404 + assert result["errorCode"] == "ObjectNotFound" + assert result["requestId"] == "request-123" + + def test_inspect_agent_route_rejects_unknown_realm() -> None: with pytest.raises( setup_existing_da.ExistingDASetupError, diff --git a/tests/setup/test_da_setup_router.py b/tests/setup/test_da_setup_router.py index 17496bde6..98c7cd142 100644 --- a/tests/setup/test_da_setup_router.py +++ b/tests/setup/test_da_setup_router.py @@ -112,12 +112,35 @@ def test_setup_reconciles_every_selected_agent_before_da_only_work() -> None: ) assert "complete its kit-switch handoff" in normalized_prompt assert "python scripts/reconcile_setup_agent.py" in reconciliation - assert "--native-da-ga" in reconciliation + assert '--known-native-schema "{RETURNED_SCHEMA_NAME}"' in reconciliation + assert "--probe dataverse" in reconciliation + assert "--probe native" in reconciliation + assert reconciliation.index("Run the Dataverse probe") < ( + reconciliation.index("Run the native MinimalBot probe") + ) + assert "`agentBackend` query value is an ordering hint only" in reconciliation + assert "`dataverse` means run the Dataverse probe first" in reconciliation + assert "`cosmos` means run the native probe first" in reconciliation + assert "Run the second probe even when the first probe returns `found`" in ( + reconciliation + ) assert "DA_SETUP_PRODUCT_RECONCILIATION_JSON:" in reconciliation - assert "action: stop-and-use-cea-kit" in reconciliation - assert "action: continue-da-ga-setup" in reconciliation - assert "deliberately fail-open" in normalized_reconciliation - assert "generic API failures" in normalized_reconciliation + assert "`authentication-required`" in reconciliation + assert "`access-denied`" in reconciliation + assert "`not-found`" in reconciliation + assert "`uncertain`" in reconciliation + assert ( + "Do not convert `authentication-required`, `access-denied`, or " + "`uncertain` into `not-found`." + ) in normalized_reconciliation + assert "msdyn_copilotforemployeeselfservice*" in reconciliation + assert "gptagent_copilotforemployeeselfservice*" in reconciliation + assert "classic CA and DA-Preview variants" in reconciliation + assert "When both probes returned `found`" in reconciliation + assert "When both probes returned `not-found`" in reconciliation + assert "Preserve a backend-hint mismatch as internal evidence" in reconciliation + assert "For a native `found` DA-GA observation" in reconciliation + assert "For a Dataverse-only `found` DA-GA observation" in reconciliation for mismatch_state in ( "**Choose the starting point and target environment** as complete", "**Verify access and agent identity** as blocked", @@ -139,13 +162,50 @@ def test_setup_reconciles_every_selected_agent_before_da_only_work() -> None: assert "For **Install and open the compatible kit**, run `recoveryCommand`" in ( reconciliation ) + legacy_message = """**Message:** + +{agent display name or Selected agent} belongs to a legacy agent family, so this Developer Kit cannot safely continue its setup. + +**End message.**""" + assert legacy_message in reconciliation + assert "belongs to the solution-backed Employee Self-Service" not in reconciliation + assert "classic Employee Self-Service agent product line" not in reconciliation + compatible_handoff = reconciliation.split( + "When `outcome` is `available`", 1 + )[1].split("## Unsupported agent", 1)[0] + assert '"header": "Continue setup"' in compatible_handoff + assert ( + '"question": "How would you like to continue setup?"' + in compatible_handoff + ) + assert '"allowFreeformInput": false' in compatible_handoff + assert '"recommended"' not in compatible_handoff + assert '"default"' not in compatible_handoff + assert "Leave the selection initially unset." in compatible_handoff + assert "Keyboard focus or a visual highlight is not a selected value" in ( + compatible_handoff + ) + assert "wait for the maker to submit an explicit choice" in compatible_handoff + assert "without adding Markdown emphasis" in compatible_handoff + assert ( + 'Do not emit an operational progress line between them.' + in reconciliation + ) + assert ( + "Do not emit **Verified replacement agent identities and resolved " + "compatible ESS kit**" + in reconciliation + ) + assert "it is not part of a supported Employee Self-Service agent family" in ( + reconciliation + ) assert "continue from its environment-scoped `list-agents`" in reconciliation assert "rerun `list-environments`" in reconciliation assert "Return to **Choose the sign-in account**" in reconciliation assert "**Go back** is the account-reset route" in reconciliation unavailable_recovery = reconciliation[ - reconciliation.index("When `recoveryUnavailable` is `true`") : - reconciliation.index("Otherwise say:") + reconciliation.index("When `outcome` is `unavailable`") : + reconciliation.index("When `outcome` is `available`") ] for recovery_choice in ( "Choose a different agent", @@ -157,7 +217,10 @@ def test_setup_reconciles_every_selected_agent_before_da_only_work() -> None: "Do not offer **Install and open the compatible kit**" in unavailable_recovery ) - assert "Follow the corresponding Setup routes below." in unavailable_recovery + assert ( + "Follow the shared recovery routes under **Unsupported agent**." + in unavailable_recovery + ) assert "stop without guessing" not in reconciliation assert "When command execution fails" in reconciliation assert "Do not label the unchanged command as a retry" in reconciliation @@ -170,9 +233,15 @@ def test_setup_reconciles_every_selected_agent_before_da_only_work() -> None: assert "No Copilot Studio agent or setup state was changed" in reconciliation assert "installation was not changed" not in reconciliation assert "identity returned by this native list" in existing_dev + assert "--known-native-schema" in existing_dev assert mos_starter.count("selected-agent product-line reconciliation") >= 2 + assert mos_starter.count("--known-native-schema") >= 2 assert alm_import.count("selected-agent product-line reconciliation") >= 2 + assert alm_import.count("--known-native-schema") >= 2 assert "selected-agent product-line reconciliation" in prod_to_dev + assert "--known-native-schema" in prod_to_dev + assert "`agentBackend` query value as an ordering hint" in foundation + assert "`almEnrollment` is `not-enrolled`" in foundation def test_foundation_defines_setup_state_sources() -> None: