feat(uipath-maestro-flow): state headlessness in the flow eval config, not in 68 task prompts by rockymadden · Pull Request #3086 · UiPath/skills

rockymadden · 2026-09-04T15:39:38Z

Replaces #3079, which split these changes across two PRs. That was wrong: #3079 removed the (default) marking that rule #5's fallback consumes, while the replacement instruction lived elsewhere. Merged alone it would have left headless runs stopping on decisions they should have made.

Problem

In the 2026-09-04 nightly, 5 of 8 skill-flow-* tasks built and validated a flow, reported success, and never executed it. The checker then ran flow debug and found a null End-node output mapping, a faulted script, and an empty result. flow validate had passed on all of them.

Every one of those prompts contained "Do NOT ask for approval, confirmation, or feedback." That phrasing forbids asking. It does not say nobody is there to ask, so an agent can honor it and still stop at a consent gate waiting for a reply that never arrives.

Measured across the 128-task flow suite:

	count
Tasks stating the run is headless	0
Non-simulated tasks	119
...saying nothing about autonomy	51
...saying something, across 8 wording variants	68

Split

The fact goes to the harness config. The domain behavior stays in the skill.

Eval side

tests/experiments/flow.yaml — nightly's runtime with a system prompt stating the run is headless and what to do about it. Derived from nightly.yaml so the runtime cannot drift.
run-coder-eval.yml — an experiment input, defaulting to experiments/nightly.yaml. A config's defaults apply to whatever its run selects, so scoping is task_globs plus -e, not a fork of the workflow.
tests/Makefile — make flow runs the suite with that config and excludes interactive/, whose 9 tasks have a simulated user and must never be told nobody is present. The exclusion is the config's contract, so it lives with the target rather than in a comment someone has to find.
69 task files — hand-copied autonomy lines removed. Task-specific text that shared those sentences ("Do NOT substitute a mock", "Do NOT run or debug the flow — the grader executes it") is preserved.
test-task-template.yaml — stop telling authors to add the line.

Skill side

Both independent of headlessness, and true with a user watching:

flow debug consent comes from the mandate. A request to build something that does X is a request for it to work; building plus validating does not discharge it. Debug also overwrites the Studio Web solution behind the local .uipx SolutionId — confirmed in 3 of 8 runs and by the eval suite's own cleanup_solutions.py ("Delete Studio Web solutions uploaded by uip maestro flow debug") — so never debug a solution this run did not scaffold.
"Publish to Studio Web" is no longer marked (default) in either What's next dropdown. Rule feat(CodedAgents): group skill by functionality #5's non-interactive fallback takes the marked option, which would have auto-published to a tenant.

Modifies Critical Rules 2 and 5, called out per CONTRIBUTING.

Why the behavior is not in the skill

An earlier revision put it there as a Working style bullet. Removed, because three of its four branches (finite choice, missing value, irreversible action) only fire when nobody can be asked, and the fourth already exists as an anti-pattern here and in connector/impl.md. The only documented way to consume this skill is uip skills install into a developer's local Claude Code, Codex, Cursor, or Gemini. Nothing ships it to a headless surface, so that bullet was carrying eval-only behavior in a skill only interactive users install.

Per thread feedback, the alternative of putting this in the coder_eval orchestrator was rejected (UiPath/coder_eval#156, closed) because it would apply to simulated tasks — 319 of them repo-wide, not the 9 the flow suite suggested.

Verification

All 9 .maintenance/check-all.sh checkers: clean apart from one anchor that pre-exists on main.
Both flavors compose; 128 flow task YAMLs parse; every changed prompt is non-empty and retains its task-specific text.
make flow selects 119 tasks, 0 from interactive/.
The 9 simulated tasks are untouched; the only task-level system_prompt override is one of them, and it correctly shadows the config.
No eval run. This changes agent behavior on the same task set, so a skill-flow-* nightly is the real check.

Follow-ups (separate)

greenfield.md Step 2: assert one .flow + one project.uiproj after scaffold. bindings-no-duplicates passed its actual regression and failed only on a stray duplicate project.
script/impl.md: extend the node -e "new Function(...)" check from syntax to a value test. Would have caught the off-by-one that shipped "Suite 2000" as a city name.
variables-and-expressions.md: document the =js: boundary — an End-node object spread returned null with a clean validate. Needs a repro.
HITL node-selection: a quick-form is not a way to "display" a value; dice_roller never terminates.
Eval tenant credential preflight. A revoked Outlook grant and a hibernating ServiceNow instance currently look identical to agent error.

🤖 Generated with Claude Code

…, not in 68 task prompts In the 2026-09-04 nightly, 5 of 8 `skill-flow-*` tasks built and validated a flow, reported success, and never executed it. The checker then ran `flow debug` and found a null End-node output mapping, a faulted script, and an empty result. `flow validate` had passed on all of them. Every one of those prompts contained "Do NOT ask for approval, confirmation, or feedback". That phrasing forbids asking. It does not say nobody is there to ask, so an agent can honor it and still stop at a consent gate waiting for a reply that never arrives. Measured across the 128-task flow suite: 0 tasks stated the run was headless, 51 of the 119 non-simulated tasks said nothing about autonomy at all, and the 68 that did were spread across 8 wording variants. The fact belongs in the harness config, the domain behaviour in the skill. Eval side: - tests/experiments/flow.yaml — nightly's runtime with a system prompt that states the run is headless and what to do about it. Derived from nightly.yaml so the runtime cannot drift. - run-coder-eval.yml — `experiment` input, defaulting to nightly.yaml. A config's defaults apply to whatever its run selects, so scoping is task_globs plus -e, not a fork of the workflow. - tests/Makefile — `make flow` runs the suite with that config and excludes interactive/, whose 9 tasks have a simulated user and must never be told nobody is present. The exclusion is the config's contract, so it lives with the target. - 69 task files — the hand-copied autonomy lines removed. Task-specific text that shared those sentences ("Do NOT substitute a mock", "Do NOT run or debug the flow") is preserved. - test-task-template.yaml — stop telling authors to add the line. Skill side, both independent of headlessness and true with a user watching: - `flow debug` consent comes from the mandate. A request to build something that does X is a request for it to work, and building plus validating does not discharge that. Debug also overwrites the Studio Web solution behind the local .uipx SolutionId, so never debug a solution this run did not scaffold. - "Publish to Studio Web" is no longer marked `(default)` in either What's next dropdown. Rule #5's non-interactive fallback takes the marked option, which would have auto-published to a tenant. Modifies Critical Rules 2 and 5, per CONTRIBUTING. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…locking debug Three problems with the first wording. "Stop only when proceeding would be unsafe or irreversible" collided with this PR's own rule #2, which tells the agent that `flow debug` overwrites the Studio Web solution behind the local .uipx SolutionId. An agent reading both had every reason to treat debug as off-limits, which is the exact 5-of-8 stall this PR exists to fix. "Unsafe" was also broad enough to make an agent balk at the e2e tasks whose entire point is creating a Jira issue or posting to Slack. The stripped task lines carried "Build the complete flow end-to-end in a single pass", which is a completeness instruction, not an autonomy one. "Keep going" did not replace it; "complete the whole task in one pass" does. "Could not resolve a fact -> stop" fired too early. slack-channel- description gave up when the first page of a lookup came back empty; slack-weather-pipeline proved the same fact was resolvable by paginating five pages. The bar is now exhausting the documented resolution path. Replaces the two vague stop conditions with two concrete ones: do not delete or overwrite what this run did not create, and do not invent a value for a lookup that genuinely failed. Everything else the task implies is authorized, including tenant writes and real messages. The task-precedence line stays, and carries skill-flow-eval-no-auto- upload, which deliberately asserts a refusal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`escalation_jira_ticket` read "seeded inputs.Before starting" and `escalation_orchestrator_paths` had a double space in the same spot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ift risk Review of the PR against how it would actually be run turned up four problems, one of which made the change a net regression. The experiment input defaults to nightly.yaml, and run-coder-eval.yml is dispatch-only with per-skill globs. Anyone dispatching a flow run without setting the new input would get task prompts with the autonomy lines removed and a config that never states the run is headless — strictly worse than main. The partition step now refuses that dispatch, and refuses the mirror mistake of running uipath-maestro-flow/interactive/ under flow.yaml, where the task has a simulated user and the prompt would tell it nobody is there. Both cases were table-tested across seven dispatch shapes. flow.yaml is a snapshot of nightly.yaml, not a derivation — the experiment schema has no `extends`, so 55 lines are duplicated and will drift. An earlier commit message claimed the opposite. A parity test now asserts every substantive line of nightly.yaml still appears in flow.yaml, verified by adding a fake secret to nightly's passthrough list and watching it fail. The system prompt relied on "instructions in the task take precedence" to protect skill-flow-eval-no-auto-upload, which deliberately asserts a refusal. A system prompt outranking a user prompt is the usual direction, so that was load-bearing on an assumption. Uploading or publishing to a shared destination unless the task asks for it is now a hold-back in its own right, and no longer depends on precedence holding. Also drops EXPERIMENT_YAML from the sandbox env passthrough; it is used host-side to build the CLI args and the agent has no use for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rockymadden · 2026-09-04T15:58:46Z

Review fixes

Reviewed the PR against how it would actually be dispatched. Four problems, one of which made it a net regression.

🔴 The default path was worse than `main`

run-coder-eval.yml is dispatch-only and people run it per-skill (tasks/uipath-maestro-flow/connector_features/enum.yaml · codex in the recent run list). The experiment input defaults to nightly.yaml, so anyone dispatching a flow run without setting it would get prompts with the autonomy lines removed and a config that never states the run is headless. Strictly worse than today.

The partition step now refuses that dispatch, and refuses the mirror mistake — running uipath-maestro-flow/interactive/ under flow.yaml, where the task has a simulated user and the prompt would tell it nobody is present. Table-tested:

Selection	Config	Result
flow task	nightly.yaml	BLOCK
flow task	flow.yaml	allow
simulated flow task	flow.yaml	BLOCK
simulated flow task	nightly.yaml	allow
non-flow skill	nightly.yaml	allow
mixed flow + rpa	nightly.yaml	BLOCK
flow + its interactive dir	flow.yaml	BLOCK

🟠 The simulated-task guard was only in the Makefile

make flow excluded interactive/; CI had nothing. Now enforced at dispatch, per the table above.

🟠 I overclaimed drift protection

An earlier commit message said flow.yaml was "derived from nightly.yaml so the runtime cannot drift." False — it is a snapshot, the experiment schema has no extends, and 55 lines are duplicated (docker image, env_passthrough_extra, mounts, checker_context, post_run, run_limits).

_shared/test_flow_experiment_parity.py now asserts every substantive line of nightly.yaml still appears in flow.yaml. Verified by adding a fake secret to nightly's passthrough list and watching it fail with the exact missing line. Regex rather than PyYAML, since CI installs only pytest.

🟠 `no_auto_upload` rested on an assumption

The prompt relied on "instructions in the task take precedence" to protect skill-flow-eval-no-auto-upload, which deliberately asserts a refusal. A system prompt outranking a user prompt is the usual direction, so that was load-bearing on something that may not hold. Uploading or publishing to a shared destination unless the task asks for it is now its own hold-back.

🟡 Also

EXPERIMENT_YAML dropped from the sandbox env passthrough; it is host-side only.
Two spacing artifacts my regex left in prompts: "seeded inputs.Before starting" and a double space in the same position.

Verification

774 pytest tests pass across tests/tasks/uipath-maestro-flow/, including the two new parity tests.
Guard logic table-tested across seven dispatch shapes.
All maintenance checkers clean apart from the anchor pre-existing on main; both flavors compose.

Still no eval run. That remains the only real check on whether the prompt changes agent behavior the way I expect.

Copilot

🟡 Changes recommended

Flow guidance remains internally contradictory, generic template assumptions are false, and four prompts contain dangling bullets.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Centralizes headless Flow evaluation behavior in a dedicated experiment configuration while updating Flow execution guidance.

Changes:

Adds Flow-specific eval configuration and workflow/Makefile selection.
Removes duplicated autonomy wording from 69 Flow tasks.
Revises debug consent and removes automatic Studio Web publishing defaults.

File summaries

File	Description
`.github/workflows/run-coder-eval.yml`	Adds experiment selection.
`tests/Makefile`	Adds the non-interactive Flow target.
`tests/experiments/flow.yaml`	Defines headless Flow defaults.
`tests/templates/test-task-template.yaml`	Revises prompt-authoring guidance.
`skills/uipath-maestro-flow/SKILL.md`	Changes consent and decision rules.
`skills/uipath-maestro-flow/references/operate/run.md`	Documents mandate-derived debug consent.
`skills/uipath-maestro-flow/references/author/greenfield.md`	Removes the publish default.
`skills/uipath-maestro-flow/references/author/brownfield.md`	Removes the publish default.
`tests/tasks/uipath-maestro-flow/smoke/scheduled_trigger.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/smoke/merge_parallel_sync.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/transform_map/transform_map.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/transform_group_by/transform_group_by.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/transform_filter/transform_filter.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/terminate/terminate.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/switch/switch.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/subflow/subflow.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/rpa/rpa.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/outlook_waitfor_email/outlook_waitfor_email.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/outlook_trigger_inbox/outlook_trigger_inbox.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/openmeteo_weather/openmeteo_weather.yaml`	Preserves connector-specific guidance.
`tests/tasks/uipath-maestro-flow/single_node/lowcode_agent/lowcode_agent.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/file_attachment/file_attachment.yaml`	Preserves attachment preflight guidance.
`tests/tasks/uipath-maestro-flow/single_node/delay/delay.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/decision/decision.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/coded_agent/coded_agent.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/single_node/api_workflow/api_workflow.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/wiki_pageviews/wiki_pageviews.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/slack_weather_pipeline/slack_weather_pipeline.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/slack_channel_description/slack_channel_description.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/reading_list/reading_list.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/multi_city_weather/multi_city_weather.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/loop_multiply/loop_multiply.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/feet_inches/feet_inches.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/dice_roller/dice_roller.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/customer_escalation/customer_escalation.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/calculator/calculator.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/multi_node/bellevue_weather/bellevue_weather.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/ixp/e2e_03_project_creation_handoff/e2e_03_project_creation_handoff.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/ixp/e2e_01_invoice_extraction_greenfield.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/hitl/smoke_01_hitl_node_placed.yaml`	Removes a redundant scalar line.
`tests/tasks/uipath-maestro-flow/evaluate/inline_agent_eval/inline_agent_eval.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/edit/update_node/update_node.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/edit/remove_node/remove_node.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/edit/move_node/move_node.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/edit/group_to_subflow/group_to_subflow.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/edit/add_output/add_output.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/edit/add_node/add_node.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/e2e/escalation_slack_alert/escalation_slack_alert.yaml`	Preserves grader-execution guidance.
`tests/tasks/uipath-maestro-flow/e2e/escalation_orchestrator_paths/escalation_orchestrator_paths.yaml`	Preserves grader-execution guidance.
`tests/tasks/uipath-maestro-flow/e2e/escalation_jira_ticket/escalation_jira_ticket.yaml`	Preserves grader-execution guidance.
`tests/tasks/uipath-maestro-flow/context-grounding/summarize/summarize.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/context-grounding/batch_transform/batch_transform.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_trigger/webhook_waitfor_parallel.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_trigger/trigger_with_filter.yaml`	Removes a redundant scalar line.
`tests/tasks/uipath-maestro-flow/connector_features/testmanager_testset_lifecycle/testmanager_testset_lifecycle.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/testmanager_testcase_lifecycle/testmanager_testcase_lifecycle.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/testmanager_requirement_lifecycle/testmanager_requirement_lifecycle.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/testmanager_generic_records/testmanager_generic_records.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/testmanager_execution_results/testmanager_execution_results.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/testmanager_crud_grounded/testmanager_crud_grounded.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/testmanager_attachments/testmanager_attachments.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/slack-http-fallback/slack_http_fallback.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/paginated_reference_lookup.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/non-catalog-http-fallback/non_catalog_http_fallback.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/jdbc_databricks_query/jdbc_databricks_query.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/generic_dynamic_node/generic_dynamic_node.yaml`	Preserves no-mock guidance.
`tests/tasks/uipath-maestro-flow/connector_features/drive_to_slack.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/trigger_lifecycle.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/smoke_update.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/smoke_update_existing_flow.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/smoke_query.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/smoke_file_activities.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/smoke_error.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/smoke_create_all_types.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/integration_create_get.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/e2e_contract_intake_pipeline.yaml`	Removes autonomy wording.
`tests/tasks/uipath-maestro-flow/connector_features/datafabric_connector/contractregistry_crud_filters.yaml`	Removes autonomy wording.

Review details

Files reviewed: 78/78 changed files
Comments generated: 8
Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

The previous commit claimed a guard that is not in the file. The patch script raised before its write, so nothing landed, and I table-tested a standalone copy of the block rather than the workflow. The commit message and the PR comment both asserted behaviour that did not exist. The guard is now in the split step, and the test harness extracts the block from the shipping YAML rather than retyping it, so this cannot recur. Seven dispatch shapes pass, including the empty selection. Its first draft also had a real defect: `ALL=("${LINUX[@]}" "${WINDOWS[@]}")` is an unbound-variable error under `set -u` when an array is empty, which is the normal case for one of the two platforms. Reproduced on bash 3.2. Now word-splits the same `${LINUX[*]:-}` form the surrounding code uses. Parity test no longer re-reads and re-parses flow.yaml once per line of nightly.yaml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rockymadden · 2026-09-04T16:03:09Z

Correction: the guard I described in the last comment was not in the file

My patch script raised before its write, so nothing landed. I then table-tested a standalone copy of the block rather than the workflow, saw it pass, and reported it as done. The commit message and the previous comment both asserted behavior that did not exist.

It is in now, and the harness extracts the block from the shipping YAML instead of retyping it, so the test cannot pass while the file lacks the code.

Its first draft also had a real defect:

ALL=("${LINUX[@]}" "${WINDOWS[@]}")   # unbound variable under `set -u` when empty

One of the two platform arrays is normally empty, so this would have failed the partition step on most dispatches. Reproduced on bash 3.2; now uses the same ${LINUX[*]:-} word-split form as the surrounding code. Seven dispatch shapes pass, including the empty selection.

Open finding I did not fix: a second regression path

smoke-skills.yml runs flow tasks with -e experiments/smoke.yaml, hardcoded, on every PR touching skills. 13 flow tasks are tagged smoke, and their autonomy lines are stripped by this PR while smoke.yaml has no headless statement. Same regression as the dispatch path, on an automatic trigger, and my guard does not cover it. tests/Makefile all / e2e / tags have the same gap under default.yaml.

Three ways, and the choice is a scope decision rather than a technical one:

Add the headless paragraph to smoke.yaml. Closes the automatic path, but it applies to every skill's smoke tasks, not just flow. Needs a task-level system_prompt override on interactive/solution_select.yaml, the one simulated task tagged smoke, whose whole point is that the agent asks. Also creates a second copy of the paragraph, so the parity test would need to cover it.
Keep the autonomy line on the 13 smoke-tagged flow tasks. Stays in flow scope, no other skill affected, but leaves the suite inconsistent.
Accept it. Those tasks are structural (init_validate, registry_discovery, scheduled_trigger), and the two that run debug ask for it explicitly in their prompts, so the gates they hit are minimal.

I lean 1, but it widens the blast radius past what this PR was scoped to, so I would rather you pick.

Everything else re-verified: 774 pytest pass, all maintenance checkers clean apart from the anchor pre-existing on main, both flavors compose.

… gates, orphaned bullets Four valid findings. Rule #5's non-interactive fallback was deleted as collateral when the rule-13 experiment was reverted. The intended edit was only dropping `flow debug` from its consent-gate list. Rule #4 and ixp/impl.md:356 both route folder selection through "rule #5 (its non-interactive fallback applies)", so both pointed at behaviour that no longer existed. Restored, minus `flow debug`. Both authoring handoffs still gated Debug on consent ("Confirm consent first", "see the consent-before-debug rule"), which is the exact gate rule #2 stops applying. An agent following greenfield or brownfield to its What's next menu could still stall there. Both now say consent comes from the mandate and that selecting the option is the ask. The prompt strip left a bare list marker in four tasks, so every run sent an empty bullet: inline_agent_eval, e2e_03_project_creation_handoff, merge_parallel_sync, scheduled_trigger. Swept the whole suite; none left. The template's "do not add autonomy language" note was blanket, but only experiments/flow.yaml carries the headless system prompt. A future non-flow e2e task following it would have shipped with no autonomy instruction at all. Scoped per experiment. Not changed: flow.yaml being a copy of nightly.yaml rather than a derived config. There is no `extends` in the experiment schema, so the parity test added earlier is the remedy Copilot's own comment suggests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rockymadden · 2026-09-04T16:17:45Z

Correction to the smoke-tests assessment above

I said the 13 smoke tasks probably would not fail, on the grounds that none of their checkers runs flow debug. That reasoning was too narrow — debug is only one of the gates the skill can stall on, and it is the one I could rule out cleanly, so I stopped there.

Gates these tasks can reach without debug or a connector

Tool-ladder rung 4. editing-operations.md, and the self-checks in both greenfield.md and brownfield.md: "name the tool. If it is not Edit, Write, or uip maestro flow ..., STOP and ask the user." Any agent reaching for python / jq / sed on a .flow edit hits this.

smoke_update_existing_flow.yaml is exactly that shape: it pre-seeds a flow, then asks for a filter added, an ordering added, and both reverted. That is the kind of fiddly JSON surgery that invites a script. With no user, that is a stall, and debug is nowhere near it.

The What's next dropdown. Every authoring task ends at "Do not run any of these actions without explicit user selection." This one resolves safely — rule #5's restored fallback finds no marked option, so it stops and reports, which is the correct terminal behavior and the checker inspects files regardless. But it is a gate on all 13, not zero.

Also correcting myself

I claimed the solution-discovery gate cannot fire because sandboxes are fresh. Six of the 13 pre-seed a project through pre_run (scaffold_movie_report_flow.py). Rule #6 is scoped to scaffolding rather than editing so it likely still does not fire, but "fresh sandbox" was wrong as stated.

What settles it

Not more reading. smoke-skills.yml has no paths: filter and is a required status check, so all 13 of these tasks run under smoke.yaml — without the headless prompt — as part of merging this PR.

That is the experiment, already wired, at no extra cost. If the stripping breaks them, this PR's own CI fails and blocks itself, which is the right outcome. If they pass, the question is answered empirically rather than by my estimate.

So I am not expanding scope to smoke.yaml preemptively. If the smoke job fails, we will know which of the three options in my earlier comment is actually warranted instead of guessing.

One conflict, in SKILL.md rule #5. Both sides edited the same sentence: - this branch dropped `flow debug` from the consent-gate list, because rule #2 now derives its consent from the mandate - main (#2930, studioweb publish destination) appended the `user-question-options-extra` flavor markers to the end of the rule Kept both. The studioweb flavor's content for those markers ends with "Everything else in this rule still applies — the enumerated options, the numbered-list fallback, the non-interactive behaviour, and the consent gates", which stays accurate with `flow debug` out of that list. Verified after the merge: both flavors compose (182 studioweb replacements, up from 178 with the new marker), 773 pytest pass, all maintenance checkers clean apart from the anchor pre-existing on main, and this PR's four skill edits are intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both were missing from the README table of experiments and the list of make targets, which is where anyone looks to find out how to run a suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rockymadden · 2026-09-04T16:24:40Z

Third review: the nightly is still exposed, and my guard cannot see it

tests/README.md documents where each experiment is used:

Experiment	Used by
`nightly.yaml`	Nightly cron (`daily.sh`)
`smoke.yaml`	PR-gate smoke (Linux)

daily.sh runs on a VM outside this repo. It is the runner that produced the 2026-09-04 failures this PR exists to fix, and it does not go through run-coder-eval.yml, so the dispatch guard I added has no reach over it.

After this PR, that nightly runs flow tasks with:

autonomy lines removed (this PR stripped all 68)
nightly.yaml, which has no headless statement

The nightly gets the regression, and it is the one run we actually care about. I fixed the manual-dispatch path and left the real one open. This is the same structural problem for the third time: the config is scoped to flow, the task edits are global, and there are more runners than this repo can guard.

What I would do

Split the PR. Land the config, the workflow input, the guard, the skill fixes and the docs. Hold the 68 deletions until every runner that executes flow tasks uses flow.yaml — which means a daily.sh change outside this repo, and a decision on smoke.yaml.

That is the recommendation I made two rounds ago, backed away from when asked to do it in one PR, and have now had confirmed by a third independent path. The deletions are the only part that carries risk; everything else is additive and safe to merge today.

Also fixed this round

flow.yaml and make flow were missing from the README — the experiments table and the make-target list are exactly where someone looks to find out how to run a suite, and neither mentioned them.

Verified clean

Only SKILL.md overlapped between this branch and main, and that conflict was resolved by hand. No task file was silently auto-merged.
0 flow tasks carry autonomy language after the merge, so main did not reintroduce any.
21 checks pass, 0 fail. Run skill smoke tests and the Windows RPA job are still pending — the Linux smoke job is the one carrying the answer on the 13 smoke tasks.

rockymadden · 2026-09-04T16:32:57Z

Closing in favour of a simpler approach that does not need a second experiment config.

Three reviews turned up the same structural problem from different angles: the config is scoped to flow, the task edits are global, and there are more runners than this repo can guard.

The experiment input defaults to nightly.yaml, so a normal dispatch was a regression
smoke-skills.yml hardcodes smoke.yaml and runs 13 flow tasks on every PR
The real nightly runs from daily.sh in coder_eval_uipath, which this repo cannot guard at all
Its skills suite globs tests/tasks/**/*.yaml, so pointing it at flow.yaml would hit all 319 simulated tasks repo-wide — the thing the Slack thread vetoed

coder_eval has no pattern-scoped defaults — defaults and variants both apply run-wide — so any config approach has to solve task selection, which means a new suite in another repo before this one is safe to merge.

The replacement keeps the text in the task prompts, where it already works under every runner and config, and fixes what was actually wrong with it: the wording did not say nobody is there, it existed in 8 variants, and it was missing from 51 of 119 tasks. One canonical paragraph, applied to every non-simulated flow task, enforced by a test keyed off the simulation field so it cannot drift and cannot reach a task that has a user.

That drops the second config, the workflow input, the dispatch guard, the make target, the parity test, and the cross-repo dependency — and with them every risk found in review.

The two skill fixes carry over unchanged: flow debug consent from the mandate, and unmarking the publish default. Neither depends on where the headless text lives.

rockymadden and others added 2 commits September 4, 2026 09:48

fix(tests): repair two spacing artifacts left by the prompt strip

18d100a

`escalation_jira_ticket` read "seeded inputs.Before starting" and `escalation_orchestrator_paths` had a double space in the same spot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rockymadden requested a balanced review from Copilot September 4, 2026 15:56

Copilot started reviewing on behalf of rockymadden September 4, 2026 15:56 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

rockymadden and others added 2 commits September 4, 2026 10:21

docs(tests): document experiments/flow.yaml and the make flow target

8f2b4b8

Both were missing from the README table of experiments and the list of make targets, which is where anyone looks to find out how to run a suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rockymadden closed this Sep 4, 2026

rockymadden mentioned this pull request Sep 4, 2026

feat(uipath-maestro-flow): one canonical headless preamble on every zero-shot flow task #3088

Merged

Navigation Menu

Uh oh!

feat(uipath-maestro-flow): state headlessness in the flow eval config, not in 68 task prompts - #3086

feat(uipath-maestro-flow): state headlessness in the flow eval config, not in 68 task prompts#3086
rockymadden wants to merge 8 commits into
mainfrom
feat/flow-headless-system-prompt

rockymadden commented Sep 4, 2026

Uh oh!

rockymadden commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rockymadden commented Sep 4, 2026

Uh oh!

rockymadden commented Sep 4, 2026

Uh oh!

rockymadden commented Sep 4, 2026

Uh oh!

rockymadden commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rockymadden commented Sep 4, 2026

Problem

Split

Eval side

Skill side

Why the behavior is not in the skill

Verification

Follow-ups (separate)

Uh oh!

rockymadden commented Sep 4, 2026

Review fixes

🔴 The default path was worse than main

🟠 The simulated-task guard was only in the Makefile

🟠 I overclaimed drift protection

🟠 no_auto_upload rested on an assumption

🟡 Also

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rockymadden commented Sep 4, 2026

Correction: the guard I described in the last comment was not in the file

Open finding I did not fix: a second regression path

Uh oh!

rockymadden commented Sep 4, 2026

Correction to the smoke-tests assessment above

Gates these tasks can reach without debug or a connector

Also correcting myself

What settles it

Uh oh!

rockymadden commented Sep 4, 2026

Third review: the nightly is still exposed, and my guard cannot see it

What I would do

Also fixed this round

Verified clean

Uh oh!

rockymadden commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

🔴 The default path was worse than `main`

🟠 `no_auto_upload` rested on an assumption