Bundle packaging modes + ordered multi-bundle deploy

[matthewmoorcroft]

⚠️ Draft — initial port. This change is rebuilt onto the unified base as-is. It predates the engine → sources/adf module restructure, so it still needs adaptation to the new layout (and a green test run) before it is mergeable.

Implements the bundle-packaging design from Greg Hansen's proposal in the classification & single-bundle PR (#39), plus the ordered auto-deploy from Lorenzo Rubio / Alex Nastetsky's follow-up idea.

Relationship to the classification & single-bundle PR

the classification & single-bundle PR (fix/azure-function-activity-classification) carries the classification + initial single-bundle work and hosts the packaging design discussion. This PR is the packaging implementation, kept as a separate reviewable unit so classification and packaging review independently.

Greg's requirements

Must Implement

• Users can specify their packaging preference — --packaging-mode flag + adapter prompt
• Single, bundle-per-pipeline, and bundle-per-group all supported — PACKAGING_MODES = ("per-pipeline", "single", "per-group"); per-group supports both user-specified (--group-by spec) and inferred-from-Run-Pipeline-deps (--group-by inferred)
• Run Pipeline dependencies correctly tracked — bundler/pipeline_graph.py
• A single top-level DEPLOY.md — bundler/deploy_writer.py

Should Implement

• Generated code reused within each bundle — _dedupe_notebooks + _namespace_bundle_artifacts
• Deployment scripts consolidated per bundle — setup notebooks unioned/de-duped across the group in write_bundle_group
• A single SETUP.md per bundle

Ordered auto-deploy

Per Lorenzo/Alex's follow-up, bundler/deployer.py (exposed as the flowx-deploy skill and adapter deploy) discovers bundles, topologically orders them callees-first, deploys each with databricks bundle deploy, reads each deployed job's real numeric id from bundle summary, and injects it into callers via --var — no manual job-id wiring. Verified end-to-end against a live workspace.

Testing

• Unit tests: test_packaging_modes.py, test_pipeline_graph.py, test_deployer.py, test_deploy_writer.py
• Live deploy validated against a real multi-pipeline migration (correct order, cross-bundle id capture/injection, stop-on-failure)

Not included here

Two independent packager correctness bugs found while deploy-testing (bare {{input}} in inner jobs; templated existing_cluster_id) are split into fix/foreach-inner-job-packaging since they're unrelated to packaging design.

[matthewmoorcroft]

Closing — this was a port of another author's work and is being withdrawn from the public repo.