Skip to content

Add experimental DuckDB warehouse module - #596

Merged
RonaldHensbergen merged 4 commits into
mainfrom
feat/593-duckdb-warehouse-module
Sep 4, 2026
Merged

Add experimental DuckDB warehouse module#596
RonaldHensbergen merged 4 commits into
mainfrom
feat/593-duckdb-warehouse-module

Conversation

@RonaldHensbergen

@RonaldHensbergen RonaldHensbergen commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

Implements the first half of #593: adds an experimental file-database contract and a modules-experimental/warehouse/duckdb/ module providing DuckDB as an embedded, file-based analytical warehouse alongside modules/warehouse/postgres/. DuckDB is a fast, vectorized/columnar OLAP engine competitive with distributed engines like Spark for single-node analytics — the "experimental" label here reflects that the file-database contract shape is new to CDS and not yet exercised by real consumers, not any immaturity in DuckDB itself.

Contract (shared/contracts/file-database.yaml)

DuckDB is embedded/in-process — it has no network protocol, so it can't provide sql-database (which assumes host/port/connectionUri). New file-database contract instead: hostDirectory (shared host path), filename, path (convenience combination), readOnly. All fields required.

Module (modules-experimental/warehouse/duckdb/)

  • module.yaml: metadata.category: warehouse, productionSuitable: false. configSchema covers hostDirectory (required), filename (default warehouse.duckdb, pattern-constrained against path traversal), readOnly (default false). provides.file-database. No consumes — like dbt/dlt, this is a one-shot job, not a live service.
  • implementation.compose.services.duckdb-init: one-shot (restart: "no", healthcheck disabled), read_only: true, cap_drop: [ALL], security_opt: [no-new-privileges:true], digest-pinned alpine:3.20. Bind-mounts ${config.hostDirectory} to /data and prepares the shared file with permissive modes (chmod 0777 dir, chmod 0666 file) so consumer containers running as arbitrary non-root UIDs (e.g. dbt's uid 999) can subsequently open it themselves via their own DuckDB client library — verified against a live container running as uid 999 successfully writing to the prepared file.
  • Consumers wire this by declaring consumes: {contract: {kind: file-database}}, bind-mounting hostDirectory themselves, and opening <mount>/<filename> directly (no connection string/driver handshake).
  • Cross-module relative-path sharing works reliably because the renderer resolves relative bind-mount sources against the project root first — verified experimentally with a scratch profile inside the real repo tree.
  • README.md documents the contract fields, consumption pattern, and known limitations vs. postgres (no BI tool connectivity out of the box, no concurrent multi-writer support, no multi-node clustering — DuckDB targets the same single-node analytical use cases Spark's local mode does).

Docs

  • docs/architecture.md: Storage/compute row now lists Postgres, DuckDB (experimental).
  • docs/roadmap.md: DuckDB added to Experimental Components.

Out of scope (deferred to a follow-up PR)

  • No dlt/dbt wiring to consume file-database as a destination/target.
  • No demo profile combining dagster+dbt+dlt+duckdb.

This split keeps the PR small and reviewable (6 files changed) per the plan discussed on #593.

Testing

  • make check — 627 tests, ruff, yamllint, markdownlint all pass.
  • tests/test_duckdb_hardening.py (new, 13 tests): module metadata/experimental labeling, config schema validity and required/default fields, file-database contract shape, compose service hardening (digest pin, restart: no, read_only, cap_drop, no-new-privileges, disabled healthcheck), bind-mount wiring, init command permissions, no network ports declared.
  • test_validator.ContractSchemaValidationTest::test_all_repo_contract_files_are_schema_valid covers the new file-database.yaml contract against contract.schema.json.
  • Manually validated via a scratch profile: cds validate and cds render both pass; confirmed the rendered Compose service correctly digest-pins the image, hardens the service, and resolves the bind-mount source path consistently against the repo root.
  • Live Docker smoke test: ran the duckdb-init command sequence directly against a bind-mounted host directory, then confirmed a second container running as uid 999 could write to the prepared file without any added capabilities.

Part of #593. dbt wiring is picked up in #599; dlt wiring and a combined demo profile are still outstanding, so #593 is not fully resolved yet.

Introduces a new file-database contract and a
modules-experimental/warehouse/duckdb/ module providing DuckDB as an
embedded, file-based analytical warehouse alongside postgres.

- New shared/contracts/file-database.yaml contract (hostDirectory,
  filename, path, readOnly) for embedded databases that have no
  network protocol, distinct from sql-database.
- New duckdb-init one-shot compose service that bind-mounts a shared
  host directory and prepares the database file with permissive
  file modes so non-root consumer containers (dlt, dbt) can open it
  directly with their own DuckDB client library.
- Digest-pinned, hardened per repo conventions (read_only, cap_drop,
  no-new-privileges, disabled healthcheck).
- tests/test_duckdb_hardening.py covers module metadata, config
  schema, contract shape, and service hardening.
- docs/architecture.md and docs/roadmap.md updated to list DuckDB as
  an experimental storage/compute option.

DuckDB's experimental label reflects that the file-database contract
shape is new to CDS and not yet exercised by real consumers, not any
immaturity in DuckDB itself -- it is a fast, production-grade,
vectorized/columnar OLAP engine competitive with distributed engines
like Spark for single-node analytics.

Wiring dlt/dbt to consume this contract and adding a demo profile are
deferred to a follow-up PR to keep this change reviewable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@SemTiOne SemTiOne left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I have a couple of nits below.

Comment thread modules-experimental/warehouse/duckdb/module.yaml
Comment thread modules-experimental/warehouse/duckdb/module.yaml Outdated
RonaldHensbergen and others added 2 commits September 4, 2026 07:32
Co-authored-by: Dane Parin <emphyst80@gmail.com>
Co-authored-by: Dane Parin <emphyst80@gmail.com>
@RonaldHensbergen
RonaldHensbergen merged commit 2f08767 into main Sep 4, 2026
11 checks passed
@RonaldHensbergen
RonaldHensbergen deleted the feat/593-duckdb-warehouse-module branch September 4, 2026 05:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants