Skip to content

Fix ADF→Lakeflow classification + single-bundle packaging for multi-pipeline factories - #39

Closed
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/azure-function-activity-classification
Closed

Fix ADF→Lakeflow classification + single-bundle packaging for multi-pipeline factories#39
matthewmoorcroft wants to merge 2 commits into
unify/reconcile-internalfrom
pr/azure-function-activity-classification

Conversation

@matthewmoorcroft

Copy link
Copy Markdown
Member

⚠️ Draft — initial port. This change is rebuilt onto the unified base as-is. It predates the engine → sources/adf module restructure, so it still needs adaptation to the new layout (and a green test run) before it is mergeable.

Fixes three flowx bugs surfaced while migrating 6 real Telefónica ADF/Synapse exports end-to-end through discover → convert → package → validate. With these changes, all 6 exports classify at 100% coverage
(0 unsupported) and package into a single deployable bundle each that passes databricks bundle validate with 0 errors.

This branch already carries the fixes from PR the multi-pipeline report fix and PR the bridge-notebook header fix.

Changes:

  • ADF exports the activity type as AzureFunctionActivity, but AGENTIC_TYPES only listed AzureFunction, so every Azure Function activity fell through to unsupported (no translation path) instead of agentic. Add "AzureFunctionActivity" to AGENTIC_TYPES.
  • Packaging emitted one databricks.yml bundle per pipeline. New _combine_into_single_bundle() folds every pipeline into one workflow tree (reusing the existing inner_workflows machinery write_bundle already uses to emit a parent plus its children as separate job resources). Each pipeline keeps its own job name, parameters, and schedule.
  • DAB job parameters require a default; flowx omitted the field for ADF params that had none, so bundle deploy failed with "The argument 'default' is required.". Emit default: "" for params with no source default.

@matthewmoorcroft
matthewmoorcroft marked this pull request as ready for review September 1, 2026 12:17
Rewrite internal package-proxy URLs (pypi-proxy.dev.databricks.com) to
pypi.org / files.pythonhosted.org so public CI resolves deps. Same pinned
versions and hashes; matches main.

Co-authored-by: Isaac <no-reply@databricks.com>
@matthewmoorcroft

Copy link
Copy Markdown
Member Author

Closing — this was a port of another author's work and is being withdrawn from the public repo.

@matthewmoorcroft
matthewmoorcroft deleted the pr/azure-function-activity-classification branch September 1, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant