Add experimental ingestion module for dlt (data load tool) - #592
Open
RonaldHensbergen wants to merge 1 commit into
Open
Add experimental ingestion module for dlt (data load tool)#592RonaldHensbergen wants to merge 1 commit into
RonaldHensbergen wants to merge 1 commit into
Conversation
Resolves #589 Adds modules-experimental/ingestion/dlt/, a one-shot job module following the existing dbt transformation module precedent: no provides contract (there is no live service), productionSuitable: false, read-only root filesystem, dropped capabilities, tmpfs scratch dirs, and a bind-mounted project directory plus a named volume for persisted pipeline state. The module consumes a sql-database contract (destinationDatabase) and maps ${bindings.destination-database.connectionUri} directly onto dlt's DESTINATION__POSTGRES__CREDENTIALS env var, which is dlt's own convention for a full postgresql:// URI - no CDS-side transformation needed. Also adds: - images/dlt/{Dockerfile,entrypoint.sh,requirements.txt,README.md}: multi-stage build on python:3.14-slim (dlt supports 3.10-3.14, so no downgrade needed unlike dbt), pinned to dlt[postgres]==1.30.0. - workdirs/dlt/pipeline.py: minimal example pipeline demonstrating the extract-load pattern, meant to be replaced by real pipelines. - Workflow wiring mirroring the dbt module: publish-images.yml (build context + version extraction from requirements.txt), docker-smoke-test.yml, image-security-scan.yml, and a placeholder signed-images.json fixture entry (real digest filled in by CI's update-fixture job after first publish to main). - docs/architecture.md: new Ingestion layer row; docs/image-signing.md and docs/image-scanning.md: dlt tag/versioning notes; docs/roadmap.md: dlt added to Experimental Components. Out of scope for this change (left for follow-up work, matching the dbt module's own precedent): - No demo profile wiring dlt into a stable profiles/ stack. - No Dagster-triggered execution (dlt currently only runs as a standalone one-shot job via `docker compose run`). Testing: - make check (614 tests, ruff, yamllint, markdownlint) passes. - Manually validated via a scratch profile wiring ingestion/dlt -> warehouse/postgres: `cds validate`, `cds render`, and `cds test` (plan/render stages) all pass; confirmed the rendered Compose service correctly resolves the sql-database contract binding into DESTINATION__POSTGRES__CREDENTIALS. - tests/test_standalone_module_profile.py and tests/test_modules_no_committed_secrets.py pass for the new module (the generic coverage new experimental modules get, matching the dbt module's own test footprint of zero bespoke test files). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #589: adds an experimental ingestion module for dlt (data load tool), following the same pattern the
dbttransformation module established for experimental, one-shot-job modules.Module (
modules-experimental/ingestion/dlt/)module.yaml:metadata.category: ingestion,productionSuitable: false.configSchemacoverspipeline.{hostPath,containerPath},entrypointScript(defaultpipeline.py),destinationDatabase(contract binding, required),destinationDataset(defaultraw),pipelineName(defaultcds_dlt_pipeline).consumes: destination-database(kindsql-database, required). Noprovides— like dbt, this is a one-shot job with no live service to expose.implementation.compose.services.dlt-run: one-shot (restart: "no", healthcheck disabled),read_only: true,cap_drop: [ALL],security_opt: [no-new-privileges:true], tmpfs for/tmpand/home/dlt/.dlt, read-only bind mount of the pipeline directory, and a named volume (dlt-state) for persisted pipeline state.${bindings.destination-database.connectionUri}directly ontoDESTINATION__POSTGRES__CREDENTIALS— dlt's own env-var convention for a fullpostgresql://URI, so no CDS-side transformation is needed.Image (
images/dlt/)python:3.14-slim(same digest pin as Dagster/dbt). Unlike dbt, dlt supports Python 3.10–3.14, so no base-image downgrade was needed.dlt[postgres]==1.30.0(latest stable, verified via PyPI).dltuser, pip self-uninstalled in both build stages (matches the Dagster/dbt CVE/SBOM-noise avoidance pattern).Example pipeline (
workdirs/dlt/pipeline.py)Minimal
@dlt.resource-based example meant to be replaced by real pipelines.Workflow wiring (mirrors the dbt module exactly)
.github/workflows/publish-images.yml:context="."for both GHCR and Docker Hub publish jobs, plus adlt)case in "Determine version" that extracts the version fromimages/dlt/requirements.txt..github/workflows/docker-smoke-test.yml,.github/workflows/image-security-scan.yml:context="."forimages/dlt/*.tests/fixtures/signed-images.json: placeholdercds-dltentry (non-placeholder digest); CI'supdate-fixturejob will fill in the real digest after first publish tomain, same as it did for dbt.Docs
docs/architecture.md: new Ingestion layer row.docs/image-signing.md/docs/image-scanning.md: dlt tag/versioning notes.docs/roadmap.md: dlt added to Experimental Components.modules-experimental/ingestion/dlt/README.md/images/dlt/README.md: module and image docs, mirroring the dbt module's README structure.Out of scope (left for follow-up, matching dbt's own precedent)
ingestion/dltinto a stableprofiles/stack.docker compose run. Follow-up work could add a contract letting Dagster trigger dlt runs directly.Testing
make check— 614 tests, ruff, yamllint, markdownlint all pass.yamllint .— clean (only unrelated warnings in vendored.venvpackages).ingestion/dlt→warehouse/postgres:cds validate,cds render, andcds test(plan/render stages) all pass; confirmed the rendered Compose service correctly resolves thesql-databasecontract binding intoDESTINATION__POSTGRES__CREDENTIALS.python -m unittest tests.test_standalone_module_profile tests.test_modules_no_committed_secrets -v— pass (the generic coverage new experimental modules get for free, matching dbt's own test footprint of zero bespoke test files).Resolves #589