Skip to content

Fix three ADF -> Databricks control-flow / codegen bugs - #40

Closed
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/adf-control-flow-codegen-bugs
Closed

Fix three ADF -> Databricks control-flow / codegen bugs#40
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/adf-control-flow-codegen-bugs

Conversation

@matthewmoorcroft

Copy link
Copy Markdown
Member

⚠️ Draft — initial port. This change is rebuilt onto the unified base as-is. It predates the engine → sources/adf module restructure, so it still needs adaptation to the new layout (and a green test run) before it is mergeable.

  1. ForEach inputs-bridge notebooks now carry the # Databricks notebook source marker so bundle validate accepts the notebook_task.
    • Already fixed on main by the render_bridge_notebook consolidation (the bridge-notebook header fix). No source change here — this PR only adds a guard/characterization test for it. So the source diff shows two behavior changes, not three; a reviewer isn't left hunting for a missing third source fix.
  2. A join after an IfCondition uses run_if: NONE_FAILED so a failure in the taken branch propagates (AT_LEAST_ONE_SUCCESS masked it).
  3. Synthesised _init_<var> tasks that nothing references are pruned. The keep/drop decision matches structured references (the anchored tasks.<key>.values task-value token and depends_on edges) rather than a bare substring, so a dead _init_x is never kept alive by an unrelated key/path such as _init_x_2 or /Shared/_init_x_report.

Co-authored-by: Isaac

Changes

Tests

  • manually tested
  • added unit tests
  • added integration tests

@matthewmoorcroft
matthewmoorcroft marked this pull request as ready for review September 1, 2026 12:17
Rewrite internal package-proxy URLs (pypi-proxy.dev.databricks.com) to
pypi.org / files.pythonhosted.org so public CI resolves deps. Same pinned
versions and hashes; matches main.

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft

Copy link
Copy Markdown
Member Author

Closing — this was a port of another author's work and is being withdrawn from the public repo.

@matthewmoorcroft
matthewmoorcroft deleted the pr/adf-control-flow-codegen-bugs branch September 1, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant