Add experimental DuckDB warehouse module - #596
Merged
Merged
Conversation
Introduces a new file-database contract and a modules-experimental/warehouse/duckdb/ module providing DuckDB as an embedded, file-based analytical warehouse alongside postgres. - New shared/contracts/file-database.yaml contract (hostDirectory, filename, path, readOnly) for embedded databases that have no network protocol, distinct from sql-database. - New duckdb-init one-shot compose service that bind-mounts a shared host directory and prepares the database file with permissive file modes so non-root consumer containers (dlt, dbt) can open it directly with their own DuckDB client library. - Digest-pinned, hardened per repo conventions (read_only, cap_drop, no-new-privileges, disabled healthcheck). - tests/test_duckdb_hardening.py covers module metadata, config schema, contract shape, and service hardening. - docs/architecture.md and docs/roadmap.md updated to list DuckDB as an experimental storage/compute option. DuckDB's experimental label reflects that the file-database contract shape is new to CDS and not yet exercised by real consumers, not any immaturity in DuckDB itself -- it is a fast, production-grade, vectorized/columnar OLAP engine competitive with distributed engines like Spark for single-node analytics. Wiring dlt/dbt to consume this contract and adding a demo profile are deferred to a follow-up PR to keep this change reviewable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SemTiOne
approved these changes
Sep 4, 2026
SemTiOne
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. I have a couple of nits below.
Co-authored-by: Dane Parin <emphyst80@gmail.com>
Co-authored-by: Dane Parin <emphyst80@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the first half of #593: adds an experimental
file-databasecontract and amodules-experimental/warehouse/duckdb/module providing DuckDB as an embedded, file-based analytical warehouse alongsidemodules/warehouse/postgres/. DuckDB is a fast, vectorized/columnar OLAP engine competitive with distributed engines like Spark for single-node analytics — the "experimental" label here reflects that thefile-databasecontract shape is new to CDS and not yet exercised by real consumers, not any immaturity in DuckDB itself.Contract (
shared/contracts/file-database.yaml)DuckDB is embedded/in-process — it has no network protocol, so it can't provide
sql-database(which assumeshost/port/connectionUri). Newfile-databasecontract instead:hostDirectory(shared host path),filename,path(convenience combination),readOnly. All fields required.Module (
modules-experimental/warehouse/duckdb/)module.yaml:metadata.category: warehouse,productionSuitable: false.configSchemacovershostDirectory(required),filename(defaultwarehouse.duckdb, pattern-constrained against path traversal),readOnly(defaultfalse).provides.file-database. Noconsumes— like dbt/dlt, this is a one-shot job, not a live service.implementation.compose.services.duckdb-init: one-shot (restart: "no", healthcheck disabled),read_only: true,cap_drop: [ALL],security_opt: [no-new-privileges:true], digest-pinnedalpine:3.20. Bind-mounts${config.hostDirectory}to/dataand prepares the shared file with permissive modes (chmod 0777dir,chmod 0666file) so consumer containers running as arbitrary non-root UIDs (e.g. dbt's uid 999) can subsequently open it themselves via their own DuckDB client library — verified against a live container running as uid 999 successfully writing to the prepared file.consumes: {contract: {kind: file-database}}, bind-mountinghostDirectorythemselves, and opening<mount>/<filename>directly (no connection string/driver handshake).README.mddocuments the contract fields, consumption pattern, and known limitations vs. postgres (no BI tool connectivity out of the box, no concurrent multi-writer support, no multi-node clustering — DuckDB targets the same single-node analytical use cases Spark's local mode does).Docs
docs/architecture.md: Storage/compute row now listsPostgres, DuckDB (experimental).docs/roadmap.md: DuckDB added to Experimental Components.Out of scope (deferred to a follow-up PR)
dlt/dbtwiring to consumefile-databaseas a destination/target.This split keeps the PR small and reviewable (6 files changed) per the plan discussed on #593.
Testing
make check— 627 tests, ruff, yamllint, markdownlint all pass.tests/test_duckdb_hardening.py(new, 13 tests): module metadata/experimental labeling, config schema validity and required/default fields,file-databasecontract shape, compose service hardening (digest pin,restart: no,read_only,cap_drop,no-new-privileges, disabled healthcheck), bind-mount wiring, init command permissions, no network ports declared.test_validator.ContractSchemaValidationTest::test_all_repo_contract_files_are_schema_validcovers the newfile-database.yamlcontract againstcontract.schema.json.cds validateandcds renderboth pass; confirmed the rendered Compose service correctly digest-pins the image, hardens the service, and resolves the bind-mount source path consistently against the repo root.duckdb-initcommand sequence directly against a bind-mounted host directory, then confirmed a second container running as uid 999 could write to the prepared file without any added capabilities.Part of #593. dbt wiring is picked up in #599; dlt wiring and a combined demo profile are still outstanding, so #593 is not fully resolved yet.