[FEATURE]: Ordered cross-pipeline deploy/run from control lineage (package phase)

[matthewmoorcroft]

TL;DR

flowx emits one DAB per ADF pipeline with no notion of ordering across them. When one pipeline calls another via ExecutePipeline, the generated run_job_task points at a job living in a sibling bundle — so bundle deploy breaks, and today's fix (FLOWX-7) only patches it with a manual ${var.X} placeholder the operator fills in by hand. That's deploy-valid but records no ordering, and is wired one-bundle-at-a-time with no global view of the call graph.

This is the "2b" follow-up #23 deferred: consume #23's deterministic lineage.control_edges in the package phase to emit real, ordered cross-pipeline references. Recommended: generate a top-level orchestration bundle (Option B) whose run_job_task DAG encodes depends_on straight from the control graph, plus deploy-ordering (Option C) so callee bundles register before callers. Depends on #23.

---

The gap

Verified against source — four facts:

1. One bundle per pipeline. dab_writer.main writes each workflow to its own dir and calls write_bundle per workflow (dab_writer.py:434-447); each emits an independent databricks.yml + resources/*.yml (:81-159). Documented in the package skill (flowx-package/SKILL.md:247). This is FLOWX-7 (KNOWN_ISSUES_RCA.md:142-160).
2. ExecutePipeline → a cross-bundle run_job_task. execute_pipeline.prepare lowers the call to run_job_task.job_id = ${resources.jobs.<callee>.id} (execute_pipeline.py:49-52). That reference only resolves inside this bundle, but the callee lives in a sibling bundle → bundle deploy fails with no such node "resources.jobs.X".
3. The FLOWX-7 fix is deploy-valid but not ordered. _rewrite_cross_bundle_run_job_refs rewrites the unresolved ref to ${var.X} (dab_writer.py:1209-1248) and surfaces a manual SETUP.md row telling the operator to pass --var "X=<job_id>" (prereqs_writer.py:509-525). It records no deploy/run ordering, works one bundle at a time (_cross_bundle_variables resets per write_bundle), and leaves an exactly-knowable relationship to be reconstructed by hand.
4. No ordering logic exists anywhere. Nothing orders bundles, generates an orchestration job, or sequences deploys; main simply loops workflows in report order.

What #23 gives us. #23 adds a deterministic lineage block to inventory.json with control_edges (caller→callee ExecutePipeline edges: caller_pipeline, callee_pipeline, activity_name, wait_on_completion). That's the global input this issue needs — it names, before packaging runs, exactly which callee bundles each caller depends on. This issue consumes it.

---

Proposed solution

Consume control_edges in the package layer to replace manual ${var.X} wiring with real, ordered references. Repo conventions (AGENTS.md): @dataclass(slots=True, kw_only=True) models, deterministic bundler logic, golden-file tests.

Option B — top-level orchestration bundle (RECOMMENDED). Emit one additional generated bundle with a single orchestration job: one run_job_task per migrated pipeline, depends_on derived directly from control_edges (caller depends on callee, honoring wait_on_completion). A single deployable artifact expressing the whole call graph as a real Databricks Jobs DAG — a shape the codebase already models (RunJobActivity ir.py:328-340; the parent_child_orchestration motif motifs.py:120-131). It captures ordering as data the platform enforces, needs no per-bundle rewrite gymnastics, and composes with the existing per-pipeline bundles (still the deploy unit for the child jobs).

Option C — ordered deploys (COMPANION). Topologically sort the per-pipeline bundles from control_edges so callees deploy/register before callers, and emit the ordered databricks bundle deploy sequence into SETUP.md. Cheap; ship alongside B so the numeric job IDs a caller needs exist by deploy time.

Option A — real cross-bundle run_job_task wiring (rejected). Replace ${var.X} with the callee's real reference. Weakest: DAB ${resources.jobs.X.id} only resolves within one bundle, so a genuine symbolic cross-bundle reference isn't confirmable (see open questions), and without a shared bundle it degrades to the same manual numeric ID — the same step, relocated.

Recommendation: B (primary) + C (companion). B produces the enforceable ordering artifact; C ensures the underlying per-pipeline bundles register in dependency order.

Cycle handling. ADF permits A→B and B→A; a Jobs DAG cannot be cyclic. The builder must detect cycles (deterministic DFS/SCC) and, on detection, break the back-edge, emit the tasks without the offending depends_on, and record a WARNINGS.md/SETUP.md note naming the cycle members so the operator decides the real ordering. Never emit an invalid DAG.

---

Touch points

• New consumer — read inventory.json's lineage.control_edges ([FEATURE]: Deterministic lineage & dependency tracking, surfaced in the inventory #23) and build a caller→callee graph keyed by normalize_task_key(pipeline_name) to align with the resource keys execute_pipeline.prepare already emits (execute_pipeline.py:49).
• dab_writer.py — new write_orchestration_bundle, invoked from main (:434-459) after the per-pipeline bundles, reusing _build_databricks_yml (:576) and the run_job_task shape. Deploy-ordering (C) is computed here from the same graph.
• prereqs_writer.py — when the orchestration job resolves a caller's edge, drop/downgrade the manual "Cross-bundle job references" SETUP.md row (:509-525); keep it as the fallback for unresolved callees (partial exports).

FLOWX-7 interaction. The ${var.X} placeholder stays as the backward-compatible fallback for callees absent from the current export or when the orchestration bundle is disabled. When #23's graph resolves a callee, the orchestration job's depends_on becomes the source of ordering truth and the placeholder becomes a safety net, not the primary mechanism.

---

Testing

Repo conventions (AGENTS.md Testing Standards; tests/unit/, tests/integration/, golden-file):

• Unit (extending test_bundler.py / test_prereqs_writer.py): given a synthetic control_edges list (parent→several children; a diamond; a cycle), assert the orchestration job's run_job_tasks + depends_on are correct, a cycle is broken with a recorded warning, and a resolved edge suppresses the FLOWX-7 SETUP.md row while an unresolved callee keeps it. Closes the FLOWX-7 test gap (KNOWN_ISSUES_RCA.md:160).
• Integration / golden (test_golden_output.py style): a multi-pipeline fixture (a parent calling child pipelines) through load→translate→prepare→write; assert the orchestration bundle YAML is deterministic/byte-stable and the computed deploy order lists callees before callers.

---

Scope

• In scope: consuming [FEATURE]: Deterministic lineage & dependency tracking, surfaced in the inventory #23's control_edges to emit cross-pipeline ordering — the orchestration bundle (B), deploy-ordering (C), cycle handling, and the FLOWX-7 fallback interaction.
• Out of scope: the lineage extraction itself ([FEATURE]: Deterministic lineage & dependency tracking, surfaced in the inventory #23, done); the agentic-enrichment "holistic view" ([FEATURE]: Agentic insights in inventory.json (discover phase) #25); ordering derived from data_edges (this issue uses control edges only — data-edge ordering deferred unless a concrete need emerges); column-level lineage.

---

Dependencies & open questions

• Depends on [FEATURE]: Deterministic lineage & dependency tracking, surfaced in the inventory #23 — consumes its lineage.control_edges block; [FEATURE]: Deterministic lineage & dependency tracking, surfaced in the inventory #23 explicitly names this follow-up "2b" and defers ordering-consumption here.
• DAB cross-bundle references (needs on-platform confirmation). The repo only shows run_job_task.job_id = ${resources.jobs.X.id} resolving within a single bundle, plus the numeric ${var.X} fallback across bundles. Whether DAB supports a symbolic cross-bundle job reference isn't confirmable from the codebase. Option B is chosen partly to avoid depending on this: within one orchestration bundle, targets are either its own resources or numeric IDs (both known to work). Confirm whether the orchestration job references the per-pipeline jobs by numeric deployed ID (→ child bundles deploy first, Option C) or whether a single combined bundle owning all jobs is preferable.
• Cycles. The control graph may be cyclic; the Jobs DAG can't be. Needs deterministic detection + back-edge break + operator-facing note.
• Partial exports / unresolved callees. A control_edge may point at a callee_pipeline not in the current export ([FEATURE]: Deterministic lineage & dependency tracking, surfaced in the inventory #23 records rather than drops these). Fall back to the FLOWX-7 placeholder for unresolved callees; never emit a dangling dependency.
• Backward compatibility. Existing single-pipeline migrations and the current ${var.X} + SETUP.md contract must keep working; the orchestration bundle should be additive (ideally opt-in-able) so output stays byte-stable where no cross-pipeline edges exist.