Skip to content

Add provider-scoped FlightCheck execution - #301

Merged
Surendra Goutham (is-goutham) merged 12 commits into
mainfrom
users/gouthams/pr290-02-flightcheck-cli
Sep 25, 2026
Merged

Surendra Goutham (is-goutham) merged 12 commits into
mainfrom
users/gouthams/pr290-02-flightcheck-cli

Conversation

@is-goutham

@is-goutham Surendra Goutham (is-goutham) commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds reusable execution context for provider-specific FlightCheck runs:

  • merges an explicit provider config without overwriting foundation identity
  • accepts a specific agent workspace slug
  • uses a sidecar Dataverse endpoint only when foundation config has none
  • reports skipped checks separately from passed checks

This is replacement PR 2 of the PR #290 split and is stacked on #300.

Validation

  • python -m pytest tests/flightcheck/test_cli.py tests/flightcheck/test_cli_single_checkpoint.py tests/flightcheck/test_check_roles.py -q — 49 passed
  • git diff --check

@nkemms

nkemms commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Verdict: REQUEST CHANGES

  1. MAJOR — --agent-slug is only assigned onto runner.agent_slug in solutions/ess-maker-skills/scripts/flightcheck/cli.py:954 and :1570, but nothing in the Workday checks reads that attribute. The actual consumers still derive the agent from runner.config["activeAgent"] / agent.slug (solutions/ess-maker-skills/scripts/flightcheck/checks/workday.py:415-430) and the workflow inventory still walks every workspace/agents/* directory (solutions/ess-maker-skills/scripts/flightcheck/checks/workday.py:5090-5114), so the new flag does not actually scope the run.

@is-goutham

Copy link
Copy Markdown
Contributor Author

Fixed in ccbdd69 and propagated through the stack. Workday checks now resolve runner.agent_slug first: Copilot Studio links use the selected agent, and local Workday workflow discovery scans only workspace/agents/ when scoped. Added a two-agent regression proving the other agent is excluded.

@nkemms

nkemms commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Updated review findings — merge blockers remain

  • Major — malformed explicit agent scope can become a successful skip. Validation based on Path(agent_slug).name == agent_slug accepts ..; invalid or unresolved explicit scopes can then produce empty discovery and a successful SKIPPED result. Require a strict single path component and reject an invalid explicit --agent-slug as an input error.
  • Major — provider config can overwrite foundation-owned state. _merge_connect_config copies arbitrary top-level provider fields except a partial denylist, allowing fields such as releaseLine to replace foundation run-mode identity. Use an explicit allowlist or namespace provider-owned fields rather than relying on a protected-key denylist.

@nkemms

nkemms commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Re-reviewed the current head (926abc0) against its actual stacked base. Two blockers remain:

  • --agent-slug .. escapes the selected-agent scope — solutions/ess-maker-skills/scripts/flightcheck/checks/workday.py:5096-5114: Path("..").name == "..", so the current basename check accepts the parent segment and scans above workspace/agents. Require a single valid slug segment and verify the resolved path remains a direct child of workspace/agents.
  • Provider config can still override foundation-owned execution settings — solutions/ess-maker-skills/scripts/flightcheck/cli.py:710-745: the partial denylist permits fields such as releaseLine, powerPlatformApiEndpoint, and Workday probe configuration to change dispatch or retarget requests. Replace the denylist with an explicit provider-owned allowlist or namespace.

Selected-agent propagation for the ordinary valid-slug path is improved, but these input and ownership boundaries still prevent merge.

@is-goutham

Copy link
Copy Markdown
Contributor Author

Addressed the latest blockers in 933bbb4: explicit agent slugs now use strict single-segment validation plus resolved direct-child containment, and provider config uses an explicit provider-owned allowlist. Validation: 774 FlightCheck check tests passed.

@nkemms

nkemms commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Re-reviewed the latest head (933bbb4). The unsafe agent-slug handling and provider-overlay ownership issues are fixed through strict slug validation, direct-child path resolution, and an explicit provider-owned config allowlist.

I found no remaining critical blocker. Good to merge.

@nkemms

nkemms commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Follow-up PR: When no active or explicitly selected agent exists, avoid falling back to the first agents[] entry for Workday custom-workflow inventory. Preserve the broad inventory behavior or require an explicit selection. This is not a blocker for this PR.

nkemms
nkemms previously approved these changes Sep 24, 2026
@is-goutham
Surendra Goutham (is-goutham) changed the base branch from users/gouthams/pr290-01-checkpoint to main September 24, 2026 19:43
@is-goutham
Surendra Goutham (is-goutham) dismissed nkemms’s stale review September 24, 2026 19:43

The base branch was changed.

@is-goutham

Copy link
Copy Markdown
Contributor Author

PR #300 is merged. I retargeted this PR to main, merged the updated main into the branch, and reran the FlightCheck regression scope: 831 tests passed. The repository dismissed the prior approval after the base/head update; please re-review this now-current diff.

@is-goutham

Copy link
Copy Markdown
Contributor Author

Main advanced with #326 after the prior approval. I merged the new main into this branch; the merge was clean and the expanded FlightCheck scope now passes 849 tests. Re-review requested for the current head.

@is-goutham
Surendra Goutham (is-goutham) merged commit 603f2da into main Sep 25, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants