Wire dbt module to optionally target DuckDB (#593) - #599
Open
RonaldHensbergen wants to merge 2 commits into
Open
Wire dbt module to optionally target DuckDB (#593)#599RonaldHensbergen wants to merge 2 commits into
RonaldHensbergen wants to merge 2 commits into
Conversation
Introduces a new file-database contract and a modules-experimental/warehouse/duckdb/ module providing DuckDB as an embedded, file-based analytical warehouse alongside postgres. - New shared/contracts/file-database.yaml contract (hostDirectory, filename, path, readOnly) for embedded databases that have no network protocol, distinct from sql-database. - New duckdb-init one-shot compose service that bind-mounts a shared host directory and prepares the database file with permissive file modes so non-root consumer containers (dlt, dbt) can open it directly with their own DuckDB client library. - Digest-pinned, hardened per repo conventions (read_only, cap_drop, no-new-privileges, disabled healthcheck). - tests/test_duckdb_hardening.py covers module metadata, config schema, contract shape, and service hardening. - docs/architecture.md and docs/roadmap.md updated to list DuckDB as an experimental storage/compute option. DuckDB's experimental label reflects that the file-database contract shape is new to CDS and not yet exercised by real consumers, not any immaturity in DuckDB itself -- it is a fast, production-grade, vectorized/columnar OLAP engine competitive with distributed engines like Spark for single-node analytics. Wiring dlt/dbt to consume this contract and adding a demo profile are deferred to a follow-up PR to keep this change reviewable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extends the experimental dbt module to support two mutually exclusive targets, selected by a new warehouseType config field: - postgres (existing): consumes sql-database, unchanged behavior. - duckdb (new): consumes the file-database contract from the new DuckDB warehouse module, mounting its shared data directory into the dbt container and pointing dbt-duckdb at the shared .duckdb file. Renderer changes needed to support this cleanly: - Individual bind-mount entries in a service's volumes list can now carry an enabledFrom guard, mirroring the existing service-level and named-volume-level mechanism, so a DuckDB-only mount never reaches rendered output when Postgres is the active target. - _resolve_expr gained a "<path>==<value>" equality expression form (renderer only), since the prior enabledFrom only supported boolean flags, not string/enum comparisons like warehouseType==duckdb. images/dbt/profiles.yml, entrypoint.sh, requirements.txt (adds dbt-duckdb==1.11.0), and README.md updated accordingly. Verified manually with a scratch profile wiring one dbt instance per target plus the postgres and duckdb modules: both render correctly, and the DuckDB instance's bind-mount source matches the duckdb module's own data directory, confirming real cross-module file sharing via the file-database contract. dlt wiring and a combined demo profile remain deferred to a follow-up PR, pending the dlt module's own PR merging to main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the experimental
dbtmodule so it can optionally target the new DuckDB warehouse module (introduced in #596) instead of Postgres, selected by a newwarehouseTypeconfig field. This is the dbt half of the remaining #593 checklist.Changes
modules-experimental/transformation/dbt/module.yamlwarehouseTypeconfig field (postgres|duckdb, defaultpostgres).targetDatabase(sql-database) and the newtargetWarehouseFile(file-database) consumes entries are now optional; target selection and required-env enforcement moved to the container entrypoint.dbt-rungains a DuckDB bind-mount, guarded by a newenabledFrom: "config.warehouseType==duckdb"per-entry condition, plusDBT_WAREHOUSE_TYPE/DBT_DUCKDB_PATHenv vars; existing Postgres env vars are nowifNonempty:-wrapped so they resolve safely when Postgres isn't the active target.cli/renderer.pyvolumes:list can now carry anenabledFromguard (previously only whole services and named top-level volumes supported this), so a DuckDB-only mount never leaks into rendered output when Postgres is selected._resolve_exprgained a"<path>==<value>"equality expression form (renderer only), since the existingenabledFromonly supported boolean flags, not string/enum comparisons.images/dbt/:profiles.ymlnow defines two named outputs (postgres,duckdb) selected viaDBT_WAREHOUSE_TYPE;entrypoint.shvalidates required env vars per target;requirements.txtaddsdbt-duckdb==1.11.0;README.mddocuments the dual-target env vars and volumes.docs/architecture.md/docs/roadmap.md: reflect dbt's new dual-target capability.enabledFromvolume filtering and the==equality expression.Testing
make lint— clean.python -m unittest discover -s tests -p "test_*.py"— 629 tests pass.postgres,duckdb, and twodbtinstances (one per target): both render correctly, and the DuckDB-targeting instance's bind-mount source matches theduckdbmodule's own data directory — confirming real cross-module file sharing via thefile-databasecontract.Out of scope / follow-up
main.Part of #593 (see also #596). dlt wiring and a combined demo profile remain outstanding, so #593 is not fully resolved yet.