Summary
The CDC and SCD Type 2 motifs emit a stub whose commented skeleton and labels reference apply_changes / APPLY CHANGES and "DLT" / "Delta Live Tables". Databricks has since renamed the product to Lakeflow (Declarative) Pipelines and, per the current docs, "the AUTO CDC APIs replace the APPLY CHANGES APIs… Databricks recommends using the AUTO CDC APIs in their place." The guidance flowx hands a human is therefore one generation behind the recommended API.
Both motifs (cdc_change_tracking and scd_type_2) share databricks_replacement="dlt_apply_changes" and route to _dlt_apply_changes_motif_body, which currently raises with a legacy skeleton.
Current behaviour (verified locations)
src/flowx/preparer/code_generator.py:1647 — _dlt_apply_changes_motif_body():
:1649 # DLT APPLY CHANGES -- replaces Copy/DataFlow SCD or CDC chain:1653-1663 commented import dlt / @dlt.table / dlt.apply_changes(...) skeleton:1665 raise NotImplementedError('Motif {motif_id}: implement as DLT pipeline')
src/flowx/models/motifs.py:70-74 — MOTIF_CDC_CHANGE_TRACKING description says "a DLT pipeline with APPLY CHANGES"src/flowx/models/motifs.py:152-155 — MOTIF_SCD_TYPE_2 description says "a DLT pipeline with APPLY CHANGES INTO"
Proposed change
Refresh the stub skeleton and descriptions to the current API. No behavioural change to detection/collapsing — this updates the human-facing guidance the stub points at.
- Python skeleton →
from pyspark import pipelines as dp / dp.create_auto_cdc_flow(...) (SCD Type 2 via stored_as_scd_type=2, which auto-manages __START_AT / __END_AT). Mention create_auto_cdc_from_snapshot_flow() for snapshot sources.
- SQL alternative →
AUTO CDC INTO (formerly APPLY CHANGES INTO).
- Naming → replace "DLT" / "Delta Live Tables" with "Lakeflow (Declarative) Pipelines" across the two motif descriptions and the notebook comments.
- Keep the
raise NotImplementedError (the semantic gap is still real — keys / sequence column can't be inferred deterministically), but update its message and the doc link.
- Note the caveat from the docs:
AUTO CDC requires serverless Lakeflow pipelines or Pro/Advanced editions; it is not supported by Apache Spark Declarative Pipelines.
Notes
- CDC (
cdc_change_tracking) and SCD 2 (scd_type_2) both map to the same modern construct, so a single refreshed body covers both.
- Per
FIXES_AND_LESSONS.md, changes must land in the git checkout (src/flowx/...) to be durable — the plugin cache copy is wiped on plugin update.
- Reference: https://docs.databricks.com/aws/en/dlt/cdc
Filed from analysis of a real ADF migration output, where the SCD/CDC motifs surfaced as stubs.
Summary
The CDC and SCD Type 2 motifs emit a stub whose commented skeleton and labels reference
apply_changes/APPLY CHANGESand "DLT" / "Delta Live Tables". Databricks has since renamed the product to Lakeflow (Declarative) Pipelines and, per the current docs, "theAUTO CDCAPIs replace theAPPLY CHANGESAPIs… Databricks recommends using theAUTO CDCAPIs in their place." The guidance flowx hands a human is therefore one generation behind the recommended API.Both motifs (
cdc_change_trackingandscd_type_2) sharedatabricks_replacement="dlt_apply_changes"and route to_dlt_apply_changes_motif_body, which currentlyraises with a legacy skeleton.Current behaviour (verified locations)
src/flowx/preparer/code_generator.py:1647—_dlt_apply_changes_motif_body()::1649# DLT APPLY CHANGES -- replaces Copy/DataFlow SCD or CDC chain:1653-1663commentedimport dlt/@dlt.table/dlt.apply_changes(...)skeleton:1665raise NotImplementedError('Motif {motif_id}: implement as DLT pipeline')src/flowx/models/motifs.py:70-74—MOTIF_CDC_CHANGE_TRACKINGdescription says "a DLT pipeline with APPLY CHANGES"src/flowx/models/motifs.py:152-155—MOTIF_SCD_TYPE_2description says "a DLT pipeline with APPLY CHANGES INTO"Proposed change
Refresh the stub skeleton and descriptions to the current API. No behavioural change to detection/collapsing — this updates the human-facing guidance the stub points at.
from pyspark import pipelines as dp/dp.create_auto_cdc_flow(...)(SCD Type 2 viastored_as_scd_type=2, which auto-manages__START_AT/__END_AT). Mentioncreate_auto_cdc_from_snapshot_flow()for snapshot sources.AUTO CDC INTO(formerlyAPPLY CHANGES INTO).raise NotImplementedError(the semantic gap is still real — keys / sequence column can't be inferred deterministically), but update its message and the doc link.AUTO CDCrequires serverless Lakeflow pipelines or Pro/Advanced editions; it is not supported by Apache Spark Declarative Pipelines.Notes
cdc_change_tracking) and SCD 2 (scd_type_2) both map to the same modern construct, so a single refreshed body covers both.FIXES_AND_LESSONS.md, changes must land in the git checkout (src/flowx/...) to be durable — the plugin cache copy is wiped on plugin update.Filed from analysis of a real ADF migration output, where the SCD/CDC motifs surfaced as stubs.