Add experimental DuckDB warehouse module

docs/architecture.md

@@ -39,7 +39,7 @@ The platform is divided into logical layers.
 | --- | --- | --- |
 | **Secrets** | credentials, secret injection, key management | Vault |
 | **Infrastructure services** | service coordination backends and support systems | KeyDB |
-| **Storage / compute** | databases, warehouses, and processing engines | Postgres |
+| **Storage / compute** | databases, warehouses, and processing engines | Postgres, DuckDB (experimental) |
 | **Orchestration** | workflow scheduling and task execution | Dagster |
 | **Transformation** | data modeling and transformation | (planned) |
 | **Quality** | data validation and testing | (planned) |

docs/roadmap.md

@@ -27,6 +27,7 @@ These are considered production-ready in the current release (v0.4.0):
 These work but may have breaking changes in upcoming releases:
 
 - Module: Airflow (`modules-experimental/orchestration/airflow/`) — not yet integrated into a stable profile
+- Module: DuckDB (`modules-experimental/warehouse/duckdb/`) — embedded/file-based warehouse via the new `file-database` contract; not yet wired into dlt/dbt or a demo profile (#593)
 - `cds test` — implemented; not yet exercised in CI or real contributor usage
 
 ---

modules-experimental/warehouse/duckdb/README.md

@@ -0,0 +1,70 @@
+# DuckDB (experimental)
+
+Warehouse module providing an embedded, file-based
+[DuckDB](https://duckdb.org/) database as an alternative to
+`modules/warehouse/postgres/` for single-node analytical workloads.
+DuckDB is a fast, vectorized/columnar OLAP engine that is widely regarded
+as production-grade and competitive with distributed engines like Spark
+for single-node analytics -- it is not a "toy" database. The
+**experimental** label on this module reflects that the `file-database`
+contract shape is new to CDS and not yet exercised by real consumers
+(dlt/dbt wiring lands in a follow-up PR), not any immaturity in DuckDB
+itself.
+
+## How this differs from postgres
+
+`postgres` is a client-server database: it runs as a long-lived network
+service and provides a `sql-database` contract (`host`/`port`/`username`/
+`connectionUri`) that any consumer can connect to over the network.
+
+DuckDB is embedded/in-process: it has no server, no network protocol, and
+no built-in authentication. The "database" is a single file, opened
+directly by whichever process reads or writes it. This module therefore
+provides a **`file-database`** contract instead of `sql-database`:
+
+| Field | Description |
+| --- | --- |
+| `hostDirectory` | Host path holding the DuckDB file. Consumers must bind-mount this exact directory into their own container -- there is no network address to connect to. A relative path resolves consistently against the repository root for every module that references it, so any consumer can reuse the same value as-is. |
+| `filename` | File name of the database within `hostDirectory`. |
+| `path` | Convenience field combining `hostDirectory` and `filename`. |
+| `readOnly` | Whether consumers should treat the file as read-only. |
+
+## What this module does (and doesn't do)
+
+This module does **not** run DuckDB itself -- there is no DuckDB server
+process to run. Its `duckdb-init` service is a one-shot job (`restart: no`)
+that only prepares the shared `hostDirectory`/`filename` with permissive
+file permissions so consumer containers running as different, non-root
+users can subsequently create/open the file themselves via the DuckDB
+client library embedded in their own process (e.g. Python's `duckdb`
+package used by `dlt` and `dbt`'s `dbt-duckdb` adapter).
+
+Consumers wire this contract into their own module by:
+
+1. Declaring a `consumes` entry with `contract.kind: file-database`.
+2. Bind-mounting the contract's `hostDirectory` value into their own
+   container at whatever path suits them.
+3. Opening `<their mount path>/<filename>` with their own DuckDB client
+   library -- there is no connection string/driver handshake beyond
+   opening the file.
+
+See #593 for the design discussion that led to this contract shape, and
+the tracking issue for wiring `dlt`/`dbt` to consume it as their
+destination/target.
+
+## Known limitations vs. postgres
+
+- **No BI tool connectivity out of the box.** Superset and other
+  `sql-database` consumers expect a network-reachable connection; they
+  cannot use this module unless they also bind-mount `hostDirectory` and
+  use a DuckDB-specific driver, which most BI tools don't ship by default.
+- **No concurrent multi-writer support.** DuckDB allows only one
+  read-write connection to a given file at a time; profiles combining
+  multiple writers against the same file will need to coordinate access
+  (e.g. via `dependsOn` ordering) rather than relying on database-level
+  locking guarantees the way postgres provides.
+- **Not suitable for multi-node clustering.** DuckDB runs in-process on a
+  single node by design (like Spark's single-node/local mode); this module
+  targets the same single-node analytical use cases DuckDB itself is built
+  for, and complements rather than replaces postgres for workloads that
+  need a networked, multi-writer, client-server database.

modules-experimental/warehouse/duckdb/module.yaml

@@ -0,0 +1,96 @@
+# yaml-language-server: $schema=../../../cli/resources/module.schema.json
+apiVersion: cds/v1alpha1
+kind: Module
+
+metadata:
+  name: duckdb
+  category: warehouse
+  version: "0.1.0"
+  productionSuitable: false
+  displayName: DuckDB (experimental)
+  description: >
+    Fast, in-process analytical (OLAP) warehouse based on DuckDB. Unlike
+    postgres, DuckDB is embedded and has no network protocol -- it provides
+    a file-database contract (a shared host directory holding the database
+    file) instead of a sql-database contract. This module only prepares
+    that shared file with the correct permissions; the actual database is
+    opened directly by whichever consumer module (e.g. dlt, dbt) reads or
+    writes it. This module is marked experimental because the file-database
+    contract shape is new to CDS and unproven across multiple consumers, not
+    because DuckDB itself is immature -- see README.md for consumption
+    details and current limitations (e.g. BI tools cannot connect to it like
+    they do postgres, and it has no multi-node clustering).
+
+spec:
+  runtime:
+    type: container
+    service:
+      name: duckdb-init
+      ports: []
+
+  configSchema:
+    type: object
+    additionalProperties: false
+    required:
+      - hostDirectory
+    properties:
+      hostDirectory:
+        type: string
+        minLength: 1
+        description: >
+          Host path (relative to the profile's rendered docker-compose.yml,
+          or absolute) of the directory that will hold the DuckDB database
+          file. Consumers bind-mount this same directory to share the file.
+          Use a project-local data directory, not a system path.
+
+      filename:
+        type: string
+        minLength: 1
+        pattern: "^(?!\\.\\.?$)[A-Za-z0-9_.-]+$"
+        default: warehouse.duckdb
+
+      readOnly:
+        type: boolean
+        default: false
+
+  provides:
+    - name: file-database
+      contract:
+        kind: file-database
+        spec:
+          hostDirectory: ${config.hostDirectory}
+          filename: ${config.filename}
+          path: ${config.hostDirectory}/${config.filename}
+          readOnly: ${config.readOnly}
+
+  implementation:
+    kind: docker-compose
+    compose:
+      services:
+        duckdb-init:
+          image: alpine:3.20@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc
+          restart: "no"
+          init: true
+          read_only: true
+          cap_drop:
+            - ALL
+          security_opt:
+            - no-new-privileges:true
+          pids_limit: 64
+          healthcheck:
+            disable: true
+          tmpfs:
+            - /tmp:rw,noexec,nosuid,nodev,mode=1777
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              mkdir -p /data
+              touch "/data/${config.filename}"
+              chmod 0777 /data
+              chmod 0666 "/data/${config.filename}"
+          volumes:
+            - type: bind
+              source: "${config.hostDirectory}"
+              target: /data

shared/contracts/file-database.yaml

@@ -0,0 +1,43 @@
+# yaml-language-server: $schema=../../cli/resources/contract.schema.json
+apiVersion: cds/v1alpha1
+kind: Contract
+
+metadata:
+  name: file-database
+  category: shared
+  version: "0.1.0"
+  description: >-
+    Shared contract describing an embedded, file-based database (e.g.
+    DuckDB) that consumers access by bind-mounting the same host directory,
+    rather than connecting over a network like sql-database. There is no
+    host/port/username -- the "connection" is opening the shared file
+    directly from within the consumer's own process.
+
+spec:
+  fields:
+    hostDirectory:
+      type: string
+      required: true
+      description: >-
+        Host filesystem path (relative to the profile's rendered
+        docker-compose.yml, or absolute) of the directory holding the
+        database file. Consumers must bind-mount this exact path into
+        their own container to access the same file; it is not a
+        network-reachable address.
+    filename:
+      type: string
+      required: true
+      description: File name of the database file within hostDirectory.
+    path:
+      type: string
+      required: true
+      description: Convenience field combining hostDirectory and filename.
+    readOnly:
+      type: boolean
+      required: true
+      description: Whether consumers should treat the file as read-only.
+  examples:
+    - hostDirectory: ./data/duckdb
+      filename: warehouse.duckdb
+      path: ./data/duckdb/warehouse.duckdb
+      readOnly: false

tests/test_duckdb_hardening.py

@@ -0,0 +1,123 @@
+import unittest
+from pathlib import Path
+
+import yaml
+from jsonschema import Draft202012Validator
+
+
+class DuckdbHardeningTest(unittest.TestCase):
+    def setUp(self) -> None:
+        self.repo_root = Path(__file__).resolve().parent.parent
+        module = yaml.safe_load(
+            (
+                self.repo_root
+                / "modules-experimental"
+                / "warehouse"
+                / "duckdb"
+                / "module.yaml"
+            ).read_text(encoding="utf-8")
+        )
+        self.module = module
+        self.config_schema = module["spec"]["configSchema"]
+        self.services = module["spec"]["implementation"]["compose"]["services"]
+        self.contract = yaml.safe_load(
+            (self.repo_root / "shared" / "contracts" / "file-database.yaml").read_text(encoding="utf-8")
+        )
+
+    def test_module_is_experimental(self) -> None:
+        # DuckDB's contract shape (file-database, not sql-database) is a new
+        # design this repo hasn't validated in production yet, so the
+        # module must declare itself unproven, matching the
+        # modules-experimental/ convention used by dlt and (previously) dbt.
+        self.assertFalse(self.module["metadata"]["productionSuitable"])
+        self.assertIn("experimental", self.module["metadata"]["displayName"].lower())
+
+    def test_module_schema_is_valid_json_schema(self) -> None:
+        Draft202012Validator.check_schema(self.config_schema)
+
+    def test_host_directory_is_required(self) -> None:
+        self.assertIn("hostDirectory", self.config_schema["required"])
+        self.assertEqual(self.config_schema["additionalProperties"], False)
+
+    def test_filename_defaults_and_is_constrained_to_safe_characters(self) -> None:
+        filename_schema = self.config_schema["properties"]["filename"]
+        self.assertEqual(filename_schema["default"], "warehouse.duckdb")
+        self.assertRegex(filename_schema["default"], filename_schema["pattern"])
+        # A path traversal attempt must not satisfy the filename pattern --
+        # this field is spliced directly into a shell command and a bind
+        # mount target in the compose template.
+        self.assertNotRegex("../../etc/passwd", filename_schema["pattern"])
+
+    def test_provides_file_database_contract_with_all_documented_fields(self) -> None:
+        provided = self.module["spec"]["provides"]
+        file_database = next(entry for entry in provided if entry["name"] == "file-database")
+        self.assertEqual(file_database["contract"]["kind"], "file-database")
+
+        contract_fields = set(self.contract["spec"]["fields"])
+        provided_fields = set(file_database["contract"]["spec"])
+        self.assertEqual(contract_fields, provided_fields)
+
+    def test_contract_fields_are_all_required(self) -> None:
+        # Unlike sql-database, file-database has no optional fields --
+        # every consumer needs the full hostDirectory/filename/path/readOnly
+        # set to correctly bind-mount and open the shared file.
+        for field_name, field_def in self.contract["spec"]["fields"].items():
+            with self.subTest(field=field_name):
+                self.assertTrue(field_def["required"])
+
+    def test_init_service_is_a_hardened_one_shot_job(self) -> None:
+        service = self.services["duckdb-init"]
+
+        self.assertEqual(service["restart"], "no")
+        self.assertTrue(service["read_only"])
+        self.assertEqual(service["cap_drop"], ["ALL"])
+        self.assertEqual(service["security_opt"], ["no-new-privileges:true"])
+        self.assertTrue(service["healthcheck"]["disable"])
+
+    def test_base_image_is_digest_pinned(self) -> None:
+        image = self.services["duckdb-init"]["image"]
+        self.assertRegex(image, r"^[^@]+@sha256:[0-9a-f]{64}$")
+
+    def test_shared_directory_bind_mount_matches_configured_host_directory(self) -> None:
+        volumes = self.services["duckdb-init"]["volumes"]
+        bind_mount = next(
+            volume
+            for volume in volumes
+            if isinstance(volume, dict) and volume.get("target") == "/data"
+        )
+        self.assertEqual(bind_mount["type"], "bind")
+        self.assertEqual(bind_mount["source"], "${config.hostDirectory}")
+
+    def test_init_command_prepares_the_shared_file_permissively(self) -> None:
+        command = "\n".join(self.services["duckdb-init"]["command"])
+        self.assertIn("mkdir -p /data", command)
+        self.assertIn('touch "/data/${config.filename}"', command)
+        self.assertIn("chmod 0777 /data", command)
+        self.assertIn('chmod 0666 "/data/${config.filename}"', command)
+
+    def test_runtime_declares_no_network_ports(self) -> None:
+        # DuckDB is embedded -- there is nothing listening on the network,
+        # unlike every sql-database-providing module (e.g. postgres).
+        self.assertEqual(self.module["spec"]["runtime"]["service"]["ports"], [])
+
+
+class FileDatabaseContractTest(unittest.TestCase):
+    def setUp(self) -> None:
+        self.repo_root = Path(__file__).resolve().parent.parent
+        self.contract = yaml.safe_load(
+            (self.repo_root / "shared" / "contracts" / "file-database.yaml").read_text(encoding="utf-8")
+        )
+
+    def test_contract_has_no_network_fields(self) -> None:
+        # Regression guard distinguishing file-database from sql-database:
+        # a file-database contract must never grow host/port/username
+        # fields, since that would misrepresent DuckDB as network-reachable.
+        fields = set(self.contract["spec"]["fields"])
+        self.assertFalse(fields & {"host", "port", "username", "password"})
+
+    def test_contract_kind_matches_metadata_name(self) -> None:
+        self.assertEqual(self.contract["metadata"]["name"], "file-database")
+
+
+if __name__ == "__main__":
+    unittest.main()