Skip to content

Add experimental ingestion module for dlt (data load tool) - #592

Open
RonaldHensbergen wants to merge 1 commit into
mainfrom
feat/589-dlt-ingestion-module
Open

Add experimental ingestion module for dlt (data load tool)#592
RonaldHensbergen wants to merge 1 commit into
mainfrom
feat/589-dlt-ingestion-module

Conversation

@RonaldHensbergen

Copy link
Copy Markdown
Owner

Summary

Implements #589: adds an experimental ingestion module for dlt (data load tool), following the same pattern the dbt transformation module established for experimental, one-shot-job modules.

Module (modules-experimental/ingestion/dlt/)

  • module.yaml: metadata.category: ingestion, productionSuitable: false. configSchema covers pipeline.{hostPath,containerPath}, entrypointScript (default pipeline.py), destinationDatabase (contract binding, required), destinationDataset (default raw), pipelineName (default cds_dlt_pipeline).
  • consumes: destination-database (kind sql-database, required). No provides — like dbt, this is a one-shot job with no live service to expose.
  • implementation.compose.services.dlt-run: one-shot (restart: "no", healthcheck disabled), read_only: true, cap_drop: [ALL], security_opt: [no-new-privileges:true], tmpfs for /tmp and /home/dlt/.dlt, read-only bind mount of the pipeline directory, and a named volume (dlt-state) for persisted pipeline state.
  • Maps ${bindings.destination-database.connectionUri} directly onto DESTINATION__POSTGRES__CREDENTIALS — dlt's own env-var convention for a full postgresql:// URI, so no CDS-side transformation is needed.

Image (images/dlt/)

  • Multi-stage build on python:3.14-slim (same digest pin as Dagster/dbt). Unlike dbt, dlt supports Python 3.10–3.14, so no base-image downgrade was needed.
  • Pinned to dlt[postgres]==1.30.0 (latest stable, verified via PyPI).
  • Non-root dlt user, pip self-uninstalled in both build stages (matches the Dagster/dbt CVE/SBOM-noise avoidance pattern).

Example pipeline (workdirs/dlt/pipeline.py)

Minimal @dlt.resource-based example meant to be replaced by real pipelines.

Workflow wiring (mirrors the dbt module exactly)

  • .github/workflows/publish-images.yml: context="." for both GHCR and Docker Hub publish jobs, plus a dlt) case in "Determine version" that extracts the version from images/dlt/requirements.txt.
  • .github/workflows/docker-smoke-test.yml, .github/workflows/image-security-scan.yml: context="." for images/dlt/*.
  • tests/fixtures/signed-images.json: placeholder cds-dlt entry (non-placeholder digest); CI's update-fixture job will fill in the real digest after first publish to main, same as it did for dbt.

Docs

  • docs/architecture.md: new Ingestion layer row.
  • docs/image-signing.md / docs/image-scanning.md: dlt tag/versioning notes.
  • docs/roadmap.md: dlt added to Experimental Components.
  • modules-experimental/ingestion/dlt/README.md / images/dlt/README.md: module and image docs, mirroring the dbt module's README structure.

Out of scope (left for follow-up, matching dbt's own precedent)

  • No demo profile wiring ingestion/dlt into a stable profiles/ stack.
  • No Dagster-triggered execution — dlt currently only runs as a standalone one-shot job via docker compose run. Follow-up work could add a contract letting Dagster trigger dlt runs directly.

Testing

  • make check — 614 tests, ruff, yamllint, markdownlint all pass.
  • yamllint . — clean (only unrelated warnings in vendored .venv packages).
  • Manually validated via a scratch profile wiring ingestion/dltwarehouse/postgres: cds validate, cds render, and cds test (plan/render stages) all pass; confirmed the rendered Compose service correctly resolves the sql-database contract binding into DESTINATION__POSTGRES__CREDENTIALS.
  • python -m unittest tests.test_standalone_module_profile tests.test_modules_no_committed_secrets -v — pass (the generic coverage new experimental modules get for free, matching dbt's own test footprint of zero bespoke test files).

Resolves #589

Resolves #589

Adds modules-experimental/ingestion/dlt/, a one-shot job module
following the existing dbt transformation module precedent: no
provides contract (there is no live service), productionSuitable:
false, read-only root filesystem, dropped capabilities, tmpfs scratch
dirs, and a bind-mounted project directory plus a named volume for
persisted pipeline state.

The module consumes a sql-database contract (destinationDatabase) and
maps ${bindings.destination-database.connectionUri} directly onto
dlt's DESTINATION__POSTGRES__CREDENTIALS env var, which is dlt's own
convention for a full postgresql:// URI - no CDS-side transformation
needed.

Also adds:
- images/dlt/{Dockerfile,entrypoint.sh,requirements.txt,README.md}:
  multi-stage build on python:3.14-slim (dlt supports 3.10-3.14, so no
  downgrade needed unlike dbt), pinned to dlt[postgres]==1.30.0.
- workdirs/dlt/pipeline.py: minimal example pipeline demonstrating the
  extract-load pattern, meant to be replaced by real pipelines.
- Workflow wiring mirroring the dbt module: publish-images.yml (build
  context + version extraction from requirements.txt),
  docker-smoke-test.yml, image-security-scan.yml, and a placeholder
  signed-images.json fixture entry (real digest filled in by CI's
  update-fixture job after first publish to main).
- docs/architecture.md: new Ingestion layer row; docs/image-signing.md
  and docs/image-scanning.md: dlt tag/versioning notes; docs/roadmap.md:
  dlt added to Experimental Components.

Out of scope for this change (left for follow-up work, matching the
dbt module's own precedent):
- No demo profile wiring dlt into a stable profiles/ stack.
- No Dagster-triggered execution (dlt currently only runs as a
  standalone one-shot job via `docker compose run`).

Testing:
- make check (614 tests, ruff, yamllint, markdownlint) passes.
- Manually validated via a scratch profile wiring
  ingestion/dlt -> warehouse/postgres: `cds validate`, `cds render`,
  and `cds test` (plan/render stages) all pass; confirmed the rendered
  Compose service correctly resolves the sql-database contract binding
  into DESTINATION__POSTGRES__CREDENTIALS.
- tests/test_standalone_module_profile.py and
  tests/test_modules_no_committed_secrets.py pass for the new module
  (the generic coverage new experimental modules get, matching the
  dbt module's own test footprint of zero bespoke test files).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@RonaldHensbergen RonaldHensbergen added area:configuration Configuration, wiring, and settings behavior enhancement New feature or request labels Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:configuration Configuration, wiring, and settings behavior enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ingestion module for dlt (data load tool)

1 participant