diff --git a/docs/architecture.md b/docs/architecture.md index 0b6c2a13..9d236153 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -39,7 +39,7 @@ The platform is divided into logical layers. | --- | --- | --- | | **Secrets** | credentials, secret injection, key management | Vault | | **Infrastructure services** | service coordination backends and support systems | KeyDB | -| **Storage / compute** | databases, warehouses, and processing engines | Postgres | +| **Storage / compute** | databases, warehouses, and processing engines | Postgres, DuckDB (experimental) | | **Orchestration** | workflow scheduling and task execution | Dagster | | **Transformation** | data modeling and transformation | (planned) | | **Quality** | data validation and testing | (planned) | diff --git a/docs/roadmap.md b/docs/roadmap.md index e9247525..206eec58 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -27,6 +27,7 @@ These are considered production-ready in the current release (v0.4.0): These work but may have breaking changes in upcoming releases: - Module: Airflow (`modules-experimental/orchestration/airflow/`) — not yet integrated into a stable profile +- Module: DuckDB (`modules-experimental/warehouse/duckdb/`) — embedded/file-based warehouse via the new `file-database` contract; not yet wired into dlt/dbt or a demo profile (#593) - `cds test` — implemented; not yet exercised in CI or real contributor usage --- diff --git a/modules-experimental/warehouse/duckdb/README.md b/modules-experimental/warehouse/duckdb/README.md new file mode 100644 index 00000000..30806449 --- /dev/null +++ b/modules-experimental/warehouse/duckdb/README.md @@ -0,0 +1,70 @@ +# DuckDB (experimental) + +Warehouse module providing an embedded, file-based +[DuckDB](https://duckdb.org/) database as an alternative to +`modules/warehouse/postgres/` for single-node analytical workloads. +DuckDB is a fast, vectorized/columnar OLAP engine that is widely regarded +as production-grade and competitive with distributed engines like Spark +for single-node analytics -- it is not a "toy" database. The +**experimental** label on this module reflects that the `file-database` +contract shape is new to CDS and not yet exercised by real consumers +(dlt/dbt wiring lands in a follow-up PR), not any immaturity in DuckDB +itself. + +## How this differs from postgres + +`postgres` is a client-server database: it runs as a long-lived network +service and provides a `sql-database` contract (`host`/`port`/`username`/ +`connectionUri`) that any consumer can connect to over the network. + +DuckDB is embedded/in-process: it has no server, no network protocol, and +no built-in authentication. The "database" is a single file, opened +directly by whichever process reads or writes it. This module therefore +provides a **`file-database`** contract instead of `sql-database`: + +| Field | Description | +| --- | --- | +| `hostDirectory` | Host path holding the DuckDB file. Consumers must bind-mount this exact directory into their own container -- there is no network address to connect to. A relative path resolves consistently against the repository root for every module that references it, so any consumer can reuse the same value as-is. | +| `filename` | File name of the database within `hostDirectory`. | +| `path` | Convenience field combining `hostDirectory` and `filename`. | +| `readOnly` | Whether consumers should treat the file as read-only. | + +## What this module does (and doesn't do) + +This module does **not** run DuckDB itself -- there is no DuckDB server +process to run. Its `duckdb-init` service is a one-shot job (`restart: no`) +that only prepares the shared `hostDirectory`/`filename` with permissive +file permissions so consumer containers running as different, non-root +users can subsequently create/open the file themselves via the DuckDB +client library embedded in their own process (e.g. Python's `duckdb` +package used by `dlt` and `dbt`'s `dbt-duckdb` adapter). + +Consumers wire this contract into their own module by: + +1. Declaring a `consumes` entry with `contract.kind: file-database`. +2. Bind-mounting the contract's `hostDirectory` value into their own + container at whatever path suits them. +3. Opening `/` with their own DuckDB client + library -- there is no connection string/driver handshake beyond + opening the file. + +See #593 for the design discussion that led to this contract shape, and +the tracking issue for wiring `dlt`/`dbt` to consume it as their +destination/target. + +## Known limitations vs. postgres + +- **No BI tool connectivity out of the box.** Superset and other + `sql-database` consumers expect a network-reachable connection; they + cannot use this module unless they also bind-mount `hostDirectory` and + use a DuckDB-specific driver, which most BI tools don't ship by default. +- **No concurrent multi-writer support.** DuckDB allows only one + read-write connection to a given file at a time; profiles combining + multiple writers against the same file will need to coordinate access + (e.g. via `dependsOn` ordering) rather than relying on database-level + locking guarantees the way postgres provides. +- **Not suitable for multi-node clustering.** DuckDB runs in-process on a + single node by design (like Spark's single-node/local mode); this module + targets the same single-node analytical use cases DuckDB itself is built + for, and complements rather than replaces postgres for workloads that + need a networked, multi-writer, client-server database. diff --git a/modules-experimental/warehouse/duckdb/module.yaml b/modules-experimental/warehouse/duckdb/module.yaml new file mode 100644 index 00000000..7c9be5ba --- /dev/null +++ b/modules-experimental/warehouse/duckdb/module.yaml @@ -0,0 +1,96 @@ +# yaml-language-server: $schema=../../../cli/resources/module.schema.json +apiVersion: cds/v1alpha1 +kind: Module + +metadata: + name: duckdb + category: warehouse + version: "0.1.0" + productionSuitable: false + displayName: DuckDB (experimental) + description: > + Fast, in-process analytical (OLAP) warehouse based on DuckDB. Unlike + postgres, DuckDB is embedded and has no network protocol -- it provides + a file-database contract (a shared host directory holding the database + file) instead of a sql-database contract. This module only prepares + that shared file with the correct permissions; the actual database is + opened directly by whichever consumer module (e.g. dlt, dbt) reads or + writes it. This module is marked experimental because the file-database + contract shape is new to CDS and unproven across multiple consumers, not + because DuckDB itself is immature -- see README.md for consumption + details and current limitations (e.g. BI tools cannot connect to it like + they do postgres, and it has no multi-node clustering). + +spec: + runtime: + type: container + service: + name: duckdb-init + ports: [] + + configSchema: + type: object + additionalProperties: false + required: + - hostDirectory + properties: + hostDirectory: + type: string + minLength: 1 + description: > + Host path (relative to the profile's rendered docker-compose.yml, + or absolute) of the directory that will hold the DuckDB database + file. Consumers bind-mount this same directory to share the file. + Use a project-local data directory, not a system path. + + filename: + type: string + minLength: 1 + pattern: "^(?!\\.\\.?$)[A-Za-z0-9_.-]+$" + default: warehouse.duckdb + + readOnly: + type: boolean + default: false + + provides: + - name: file-database + contract: + kind: file-database + spec: + hostDirectory: ${config.hostDirectory} + filename: ${config.filename} + path: ${config.hostDirectory}/${config.filename} + readOnly: ${config.readOnly} + + implementation: + kind: docker-compose + compose: + services: + duckdb-init: + image: alpine:3.20@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc + restart: "no" + init: true + read_only: true + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + pids_limit: 64 + healthcheck: + disable: true + tmpfs: + - /tmp:rw,noexec,nosuid,nodev,mode=1777 + command: + - sh + - -c + - | + set -eu + mkdir -p /data + touch "/data/${config.filename}" + chmod 0777 /data + chmod 0666 "/data/${config.filename}" + volumes: + - type: bind + source: "${config.hostDirectory}" + target: /data diff --git a/shared/contracts/file-database.yaml b/shared/contracts/file-database.yaml new file mode 100644 index 00000000..67ee0150 --- /dev/null +++ b/shared/contracts/file-database.yaml @@ -0,0 +1,43 @@ +# yaml-language-server: $schema=../../cli/resources/contract.schema.json +apiVersion: cds/v1alpha1 +kind: Contract + +metadata: + name: file-database + category: shared + version: "0.1.0" + description: >- + Shared contract describing an embedded, file-based database (e.g. + DuckDB) that consumers access by bind-mounting the same host directory, + rather than connecting over a network like sql-database. There is no + host/port/username -- the "connection" is opening the shared file + directly from within the consumer's own process. + +spec: + fields: + hostDirectory: + type: string + required: true + description: >- + Host filesystem path (relative to the profile's rendered + docker-compose.yml, or absolute) of the directory holding the + database file. Consumers must bind-mount this exact path into + their own container to access the same file; it is not a + network-reachable address. + filename: + type: string + required: true + description: File name of the database file within hostDirectory. + path: + type: string + required: true + description: Convenience field combining hostDirectory and filename. + readOnly: + type: boolean + required: true + description: Whether consumers should treat the file as read-only. + examples: + - hostDirectory: ./data/duckdb + filename: warehouse.duckdb + path: ./data/duckdb/warehouse.duckdb + readOnly: false diff --git a/tests/test_duckdb_hardening.py b/tests/test_duckdb_hardening.py new file mode 100644 index 00000000..26fa838d --- /dev/null +++ b/tests/test_duckdb_hardening.py @@ -0,0 +1,123 @@ +import unittest +from pathlib import Path + +import yaml +from jsonschema import Draft202012Validator + + +class DuckdbHardeningTest(unittest.TestCase): + def setUp(self) -> None: + self.repo_root = Path(__file__).resolve().parent.parent + module = yaml.safe_load( + ( + self.repo_root + / "modules-experimental" + / "warehouse" + / "duckdb" + / "module.yaml" + ).read_text(encoding="utf-8") + ) + self.module = module + self.config_schema = module["spec"]["configSchema"] + self.services = module["spec"]["implementation"]["compose"]["services"] + self.contract = yaml.safe_load( + (self.repo_root / "shared" / "contracts" / "file-database.yaml").read_text(encoding="utf-8") + ) + + def test_module_is_experimental(self) -> None: + # DuckDB's contract shape (file-database, not sql-database) is a new + # design this repo hasn't validated in production yet, so the + # module must declare itself unproven, matching the + # modules-experimental/ convention used by dlt and (previously) dbt. + self.assertFalse(self.module["metadata"]["productionSuitable"]) + self.assertIn("experimental", self.module["metadata"]["displayName"].lower()) + + def test_module_schema_is_valid_json_schema(self) -> None: + Draft202012Validator.check_schema(self.config_schema) + + def test_host_directory_is_required(self) -> None: + self.assertIn("hostDirectory", self.config_schema["required"]) + self.assertEqual(self.config_schema["additionalProperties"], False) + + def test_filename_defaults_and_is_constrained_to_safe_characters(self) -> None: + filename_schema = self.config_schema["properties"]["filename"] + self.assertEqual(filename_schema["default"], "warehouse.duckdb") + self.assertRegex(filename_schema["default"], filename_schema["pattern"]) + # A path traversal attempt must not satisfy the filename pattern -- + # this field is spliced directly into a shell command and a bind + # mount target in the compose template. + self.assertNotRegex("../../etc/passwd", filename_schema["pattern"]) + + def test_provides_file_database_contract_with_all_documented_fields(self) -> None: + provided = self.module["spec"]["provides"] + file_database = next(entry for entry in provided if entry["name"] == "file-database") + self.assertEqual(file_database["contract"]["kind"], "file-database") + + contract_fields = set(self.contract["spec"]["fields"]) + provided_fields = set(file_database["contract"]["spec"]) + self.assertEqual(contract_fields, provided_fields) + + def test_contract_fields_are_all_required(self) -> None: + # Unlike sql-database, file-database has no optional fields -- + # every consumer needs the full hostDirectory/filename/path/readOnly + # set to correctly bind-mount and open the shared file. + for field_name, field_def in self.contract["spec"]["fields"].items(): + with self.subTest(field=field_name): + self.assertTrue(field_def["required"]) + + def test_init_service_is_a_hardened_one_shot_job(self) -> None: + service = self.services["duckdb-init"] + + self.assertEqual(service["restart"], "no") + self.assertTrue(service["read_only"]) + self.assertEqual(service["cap_drop"], ["ALL"]) + self.assertEqual(service["security_opt"], ["no-new-privileges:true"]) + self.assertTrue(service["healthcheck"]["disable"]) + + def test_base_image_is_digest_pinned(self) -> None: + image = self.services["duckdb-init"]["image"] + self.assertRegex(image, r"^[^@]+@sha256:[0-9a-f]{64}$") + + def test_shared_directory_bind_mount_matches_configured_host_directory(self) -> None: + volumes = self.services["duckdb-init"]["volumes"] + bind_mount = next( + volume + for volume in volumes + if isinstance(volume, dict) and volume.get("target") == "/data" + ) + self.assertEqual(bind_mount["type"], "bind") + self.assertEqual(bind_mount["source"], "${config.hostDirectory}") + + def test_init_command_prepares_the_shared_file_permissively(self) -> None: + command = "\n".join(self.services["duckdb-init"]["command"]) + self.assertIn("mkdir -p /data", command) + self.assertIn('touch "/data/${config.filename}"', command) + self.assertIn("chmod 0777 /data", command) + self.assertIn('chmod 0666 "/data/${config.filename}"', command) + + def test_runtime_declares_no_network_ports(self) -> None: + # DuckDB is embedded -- there is nothing listening on the network, + # unlike every sql-database-providing module (e.g. postgres). + self.assertEqual(self.module["spec"]["runtime"]["service"]["ports"], []) + + +class FileDatabaseContractTest(unittest.TestCase): + def setUp(self) -> None: + self.repo_root = Path(__file__).resolve().parent.parent + self.contract = yaml.safe_load( + (self.repo_root / "shared" / "contracts" / "file-database.yaml").read_text(encoding="utf-8") + ) + + def test_contract_has_no_network_fields(self) -> None: + # Regression guard distinguishing file-database from sql-database: + # a file-database contract must never grow host/port/username + # fields, since that would misrepresent DuckDB as network-reachable. + fields = set(self.contract["spec"]["fields"]) + self.assertFalse(fields & {"host", "port", "username", "password"}) + + def test_contract_kind_matches_metadata_name(self) -> None: + self.assertEqual(self.contract["metadata"]["name"], "file-database") + + +if __name__ == "__main__": + unittest.main()