d-migrate

Database-agnostic tool for schema migration and data management.

English | Deutsch

📖 Documentation (German): Anwenderhandbuch (task-oriented user guide) · Administrationshandbuch (deployment/ops) · CLI spec · MCP server spec — full index: docs/user/README.md. See "Documentation" below for the complete list.

d-migrate is a database-agnostic tool for schema migration and data management, usable as a CLI and as an MCP server (mcp serve --transport stdio|http, MCP 2025-11-25). You define your schema once in a neutral YAML format and then validate, compare, generate DDL and execute live diff-based migrations against PostgreSQL, MySQL, SQLite, SQL Server and Oracle. d-migrate also covers reverse engineering of existing databases, streaming-based data export/import/transfer between databases and export to existing migration toolchains (Flyway, Liquibase, Django, Knex).

Who is it for?

d-migrate targets database administrators, platform engineers, data teams and integrators who:

• need a dialect-agnostic schema artefact (PostgreSQL / MySQL / SQLite / SQL Server / Oracle from the same YAML source)
• want reproducible, signed migration plans with explicit rollback contracts, drift checks and per-statement metadata
• run schema and data operations against existing databases — including reverse engineering, comparison, transfer, and incremental export — without locking into a single vendor's tooling
• need an AI-agent-callable MCP server for read-only schema discovery (validate / compare / generate / reverse) plus policy-gated job workers for data import / transfer / profile

It is not (yet) an ETL platform, a streaming-CDC pipeline or a replacement for hand-tuned dialect-specific migrations — but it captures the schema and data work that's common across these stacks.

What can I run today?

d-migrate is a working production tool at version 1.7.1 (stable, released 2026-09-16).

The current capabilities:

• Schema model: neutral YAML schema with 19 types + Spatial Geometry; validator with 35+ error codes.
• Schema operations: validate, generate, compare, reverse, migrate, rollback for every dialect listed under Supported databases — file/file, file/db, db/db.
• Diff migrations: tables, columns, indexes, constraints incl. CHECK/EXCLUDE with live-data preflight, foreign keys, sequences, views, materialized views (PG), triggers, functions/procedures; signed migration-plan.v1 artefact via --plan-artefact.
• Renames for tables, columns, views, triggers, functions, procedures, sequences — native RENAME DDL or Drop+Create fallback per dialect; CLI shortcuts (--rename-table, --rename-column) or file overlay (--migration-overlay).
• Sequence pipeline: MySQL helper-table emulation (dmg_sequences) with live drift check; opt-in preserveCurrentValue on all five dialects — probe + restore folded into one window under a per-dialect lock (pg_advisory_xact_lock / SELECT FOR UPDATE / BEGIN IMMEDIATE / sp_getapplock / DBMS_LOCK). On Oracle that window is serialized rather than atomic and says so (W159); SQLite sequence emulation via --sqlite-named-sequences helper_table.
• Spatial DDL on all five dialects: PostGIS on PostgreSQL, native spatial types on MySQL, SQL Server and Oracle, SpatiaLite on SQLite (--spatial-profile); view-query transformation across dialects.
• Data operations: streaming data export / import / transfer (JSON / YAML / CSV / Parquet) with named connections, UPSERT, truncate, trigger handling, reseeding, incremental export (--since-column / --since); data profile for data statistics.
• Parquet & object storage: data export / import --format parquet for bundle (multi-table + manifest.yaml) and single-file (footer-KV) layouts with checkpoint/resume and --table-order; S3-compatible ArtifactStore (artifacts.store: s3, AWS SDK v2) for server-mode artefacts.
• Integrations: d-migrate export flyway|liquibase|django|knex.
• MCP server (mcp serve --transport stdio|http, MCP 2025-11-25): read-only tool surface (schema_validate, schema_compare, schema_generate, schema_reverse_start) plus policy-gated job workers (data_import_start, data_transfer_start, data_profile_start, procedure_transform_*, testdata_*) with idempotency and JDBC-backed state; auth via JWT-JWKS / RFC-7662 introspection / stdio token registry.
• CLI UX: i18n EN/DE with ICU4J, explicit time-zone / temporal policy, CSV / BOM encoding contract, phased DDL via --split pre-post.
• OCI image at ghcr.io/pt9912/d-migrate:<version> and :latest, mirrored to Docker Hub as pt9912/d-migrate. Each release also ships a <version>-native variant built from the GraalVM native binary — no JVM inside, starts in milliseconds.

The simplest way to try the tool is the published OCI image:

docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema validate --source /work/schema.yaml

See Quick start below for more concrete recipes.

What makes it trustworthy?

• ≥ 90 % line coverage per module, enforced by Kover (minBound(90) in every module's build.gradle.kts). The CI build fails if any module drops below.
• Doc-check gate: Markdown link targets in docs/, spec/ and root Markdown files (including both READMEs and CHANGELOG.md) are validated against the file system on every CI run via d-check (digest-pinned container image, configured in .d-check.yml); broken internal links and anchors break the build.
• Static-analysis gate: Detekt plus a SOLID-suppression-gate (scripts/solid-suppression-gate.sh) — @Suppress("LargeClass") and friends are tracked in a ledger and require structural fixes, not inline waivers.
• Cross-dialect test matrix: test/cross-dialect-matrix sweeps every workstream × dialect × test-kind cell, with a carve-out registry (carve-outs.yaml) that requires every non-pinned cell to declare reason + plan-doc reference; silent carve-outs are rejected at load time.
• Live-DB integration tests against Testcontainers PostgreSQL, MySQL, SQL Server, Oracle and file-backed SQLite — every diff, rename, sequence and atomic-preserve pipeline runs against real engines via scripts/test-integration-docker.sh.
• Reproducible builds: --deterministic plus SOURCE_DATE_EPOCH emit byte-identical DDL across timestamps and OS environments.
• Signed migration plans: --plan-artefact writes a canonical, signed migration-plan.v1 JSON with stable fingerprints, statement IDs and rollback metadata; tampered artefacts are rejected by the MigrationPlanArtifactValidator.
• Hexagonal architecture: pure-domain hexagon:core plus hexagon:ports-{common,read,write,execute} with driving adapters (CLI, MCP) and driven adapters (drivers, formats, persistence) isolated through explicit interfaces. Architectural decisions live as ADRs in docs/adr/.
• CI mirrors local: every gate make ci runs locally also runs in GitHub Actions on every push (.github/workflows/build.yml).

Status

The full release history lives in CHANGELOG.md.

• Current stable · 1.7.1 (2026-09-16) — what :latest, Homebrew and an unpinned docker pull give you. It closes a consumer's findings against 1.7.0. Oracle now counts the objects it drops — a UNIQUE or PRIMARY KEY on a large-object column (E057, the gap SQL Server closed in 1.5.1) and its index losses (W152, E052) — and its exit code follows, so a run that quietly lost a constraint no longer reports success. A schemaRef needs no format any more: the reader detects the artifact's encoding. schema compare no longer calls a view changed whose columns are merely read unevenly or whose body ends in a server-added semicolon, it names the field when a view really did change, and it canonicalises CHECK expressions in their identifier quoting too. The MySQL reader carries the FULLTEXT index kind instead of degrading it to an ordinary index. The lineage below is unchanged: the remaining gaps in column-generation changes are closed. PostgreSQL can now turn an ordinary column into an identity column (SET NOT NULL + ADD GENERATED … AS IDENTITY + a setval catch-up past the existing high value, empty tables and negative values included). MySQL, Oracle and SQL Server can now change a column between ordinary and computed — none of the three has an in-place path, so both directions go through a column swap instead of a named rejection: no copy needed going ordinary → computed, a frozen-value copy via a temporary column going the other way. Both directions are marked destructive and need --allow-destructive; the swap stays blocked, by name, when the column carries a primary key, UNIQUE, an index or a foreign-key reference. schema generate now exits 8 whenever an object is actually skipped, not just noted — a CHECK constraint or computed column with a non-portable expression, an EXCLUDE constraint or composite type the target dialect lacks, a partial index on MySQL, a foreign key on a partitioned MySQL table; --allow-incomplete keeps the previous exit-0 behaviour. --inline-foreign-keys auto|always|never and ddl.include_comments: false give direct control over two switches that existed but were only reachable indirectly. ddl.postgresql.default_schema qualifies every generated object with a schema other than public. 1.5.1 fixes two more object losses the exit-8 fix above did not yet cover: a UNIQUE or PRIMARY KEY constraint on a large-object column on SQL Server (E057), and a rejected raw-text function DEFAULT not valid on the target dialect (E053) — both now report as a skipped object and drive exit 8, like the six sites above already did. 1.5.2 fixes one more gap in the same area: schema compare (and the MCP schema_compare tool call) now report W137 when a change to a computed column's expression could not be decided — the comparator has always folded that case to equal (a false alarm there could trigger an expensive table rewrite), but said nothing about it before; two schemas with a genuinely different formula used to compare as identical without any note. 1.7.0 closes the rest of the audit a consumer project ran: schema compare canonicalises the dialect spelling of raw expressions (CHECK and view bodies — quoting, whitespace) without touching their meaning, MySQL keeps ON DELETE RESTRICT instead of folding it away, a SQL Server view carries its columns, and generated DDL that MySQL would reject is skipped and named instead (a UNIQUE on an unbounded TEXT column). 1.6.0 carries that into the MCP server: the schema_generate tool call now reports status (complete/incomplete) and skippedCount, so an agent learns from a field that objects were dropped instead of parsing a free-text summary or filtering findings by severity — additive to the wire contract, and the call stays a success (isError=false), because a failed call would replace the payload and lose the very findings and DDL it is about. The container image runs as non-root (uid 10001), so writing into a bind mount needs --user "$(id -u):$(id -g)". Native binaries ship for linux-x64 and windows-x64; on macOS use Homebrew, the JVM artefacts or the container image.

For per-milestone task tables and ADR pointers see the canonical roadmap at docs/planning/in-progress/roadmap.md. ADRs live under docs/adr/; the canonical index is docs/adr/README.md.

All releases and details: CHANGELOG.md | GitHub Releases.

Build, Test, Lint

Individual gates for fast feedback loops:

make help              # list all available targets
make ci                # Docker CI build + docs-check (full local gate)
make gates             # Docker check, coverage, docs and semgrep gates
make docker-build      # build the runtime image
make docker-check      # Gradle check inside the Dockerfile build stage
make docker-test       # Gradle test inside the Dockerfile build stage
make docker-detekt     # Detekt static analysis
make docker-coverage-gate  # Kover ≥ 90 % per module
make docs-check        # validate Markdown link targets + Kover-excludes ledger
make semgrep           # hermetic semgrep scan with pinned rules
make integration       # Testcontainers integration suite
make docker-full-gates # docker-gates plus Docker-backed integration tests
make docker-oci-build  # build the publishable OCI image (runtime stage)
make release-assets    # build ZIP, TAR, fat JAR, SHA256 release assets

Targeted module runs:

make docker-check MODULES=":hexagon:core :adapters:driven:driver-postgresql"
make docker-test  MODULES=":adapters:driving:mcp"

Quick start

Prerequisites

• Docker
• Optional for local development without containers: JDK 21 or newer

Use the published OCI image

No local JDK required — pull the image and run it:

# Validation
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema validate --source /work/schema.yaml

# Compare (file/file)
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema compare --source file:/work/schema.yaml --target file:/work/schema-new.yaml

# Generate DDL
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  schema generate --source /work/schema.yaml --target postgresql

# Reverse engineering
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  --config /work/.d-migrate.yaml schema reverse --source mydb --output /work/reverse.yaml

# DB-to-DB data transfer
docker run --rm -v $(pwd):/work ghcr.io/pt9912/d-migrate:latest \
  data transfer --source sourcedb --target targetdb --tables users,orders

Docker / Volumes — running as non-root

The published image runs as a non-root user (uid 10001). Read-only commands (validate, compare) work as shown above. Commands that write into a bind-mounted host directory (reverse --output, generate to a file, data transfer to file targets) need the mount to be writable by the container user — add --user "$(id -u):$(id -g)" so output lands with your host ownership:

docker run --rm --user "$(id -u):$(id -g)" -v $(pwd):/work \
  ghcr.io/pt9912/d-migrate:latest \
  schema reverse --source mydb --output /work/reverse.yaml

GitHub Release assets

Published releases ship ZIP, TAR, a fat JAR and native binaries that need no Java, on the Releases page.

# Launcher-based distribution
tar -xf d-migrate-<version>.tar
./d-migrate-<version>/bin/d-migrate --help

# Or run the fat JAR directly
java -jar d-migrate-<version>-all.jar --help

# Or the native binary — no JVM required, starts in ~15 ms
chmod +x d-migrate-<version>-linux-x64
./d-migrate-<version>-linux-x64 --help

Native binaries are published for linux-x64 and windows-x64 (each with a .sha256); linux-x64 is guaranteed per release, windows-x64 is best-effort. There is no native macOS binary — on macOS use Homebrew, the JVM artefacts or the container image (ADR 0044). They are dynamically linked against glibc — on Alpine/musl use the JVM artefacts or the container image.

Homebrew (macOS and Linux)

d-migrate lives in its own tap, not in homebrew-core, so brew install d-migrate alone will not find it. Recent Homebrew versions additionally refuse to load formulae from untrusted third-party taps — you need all three steps:

brew tap pt9912/d-migrate
brew trust pt9912/d-migrate
brew install d-migrate

openjdk@21 comes along as a dependency. The tap follows stable releases only — release candidates never move it.

On macOS this is the recommended path, because there is no native macOS binary (ADR 0044).

The formula is generated for the tap on every stable release and then verified by installing it from the published tap — against the version the tag names, not one written down somewhere.

Build from source

make ci-build

Minimal schema example

Create a file called schema.yaml:

schema_format: "1.0"
name: "My App"
version: "1.0.0"

tables:
  users:
    columns:
      id:
        type: identifier
        auto_increment: true
      email:
        type: text
        max_length: 254
        required: true
        unique: true
      created_at:
        type: datetime
        default: current_timestamp
    primary_key: [id]

Validate it like this:

make docker-build
docker run --rm -v $(pwd):/work d-migrate:dev schema validate --source /work/schema.yaml

And compare two versions like this:

docker run --rm -v $(pwd):/work d-migrate:dev \
  schema compare --source /work/schema.yaml --target /work/schema-v2.yaml

Build and test locally with the Dockerfile

The repository ships a multi-stage Dockerfile that builds and tests the project inside the container and then packages the CLI distribution into a slim JRE runtime image. This is the simplest way to run the full build without installing a local JDK.

Dockerfile stages — overview

• deps: Gradle dependency pre-warm.
• build: build, tests, coverage gate, distribution. Use --target build --build-arg GRADLE_TASKS="..." to scope to a specific task list.
• detekt: Detekt static analysis.
• coverage: aggregated Kover HTML report on port 8080. docker build --target coverage -t d-migrate:coverage . + docker run --rm -p 8080:8080 d-migrate:coverage.
• coverage-json: Kover JSON to stdout via ENTRYPOINT.
• coverage-verify: hard koverVerify (≥ 90 % per module).
• release-assets: ZIP / TAR / fat JAR / SHA256 (target of make release-assets).
• runtime (default): the image that gets published — slim eclipse-temurin:21-jre-noble, non-root (uid 10001), mod_spatialite included. Target of make docker-oci-build (ADR 0041).

Common Dockerfile recipes

# Full build incl. tests and coverage validation
docker build -t d-migrate:dev .

# Force a full test/coverage run (bypasses both the Docker layer cache and the Gradle cache)
docker build --no-cache --progress=plain \
  --build-arg GRADLE_TASKS="build :adapters:driving:cli:installDist --rerun-tasks" \
  -t d-migrate:dev .

# Skip tests — build only the CLI distribution
docker build --build-arg GRADLE_TASKS="assemble :adapters:driving:cli:installDist" \
  -t d-migrate:dev .

# Run only part of the build stage without producing the final runtime image
docker build --target build \
  --build-arg GRADLE_TASKS=":hexagon:core:test :adapters:driven:driver-common:test" \
  -t d-migrate:phase-a .

# Run the locally built CLI
docker run --rm -v $(pwd):/work d-migrate:dev schema validate --source /work/schema.yaml

# Run the testcontainers integration suite
./scripts/test-integration-docker.sh

# Or a subset of integration tests
./scripts/test-integration-docker.sh :adapters:driven:driver-postgresql:test

Supported databases

All five carry the same core: reverse engineering, DDL generation, comparison, schema migration, the data path (export/import/transfer), tool export and profiling. The table says what is specific to each.

Database	Dialect specifics
PostgreSQL	PostGIS, materialized views, partitioning, expression indexes, computed columns, CREATE INDEX CONCURRENTLY
MySQL	sequence emulation via dmg_sequences, native spatial types
SQLite	sequence emulation, SpatiaLite, FTS5
SQL Server	full-text search, partitioning, import via SQLServerBulkCopy (ADR 0047)
Oracle	SDO_GEOMETRY, Oracle Text, bitmap indexes, partitioning, materialized views (ADR 0052)

Project Structure

.
├── .github/workflows/             ← GitHub Actions: build, integration, demo/sample DB, release
├── CHANGELOG.md
├── Dockerfile                     ← multi-stage (deps, build, detekt, coverage, runtime, release-assets)
├── Makefile                       ← build/test gates per Dockerfile stage
├── README.md                      ← English main version (this document)
├── README.de.md                   ← German version
├── build.gradle.kts               ← root build config + module aggregation
├── settings.gradle.kts            ← Gradle multi-module declaration
├── gradle.properties              ← pinned dependency versions
├── config/                        ← detekt and semgrep configuration
├── hexagon/                       ← pure domain + ports (no driver dependencies)
│   ├── core/                      ← neutral schema model, diff core, validators
│   ├── ports-common/              ← cross-cutting port contracts
│   ├── ports-read/                ← read-side ports (DDL generation, reverse, capabilities)
│   ├── ports-write/               ← write-side ports (data import / transfer)
│   ├── ports-execute/             ← atomic-execution ports (preserve, lock contracts)
│   ├── ports/                     ← driver-registry port (DatabaseDriver, DatabaseDriverRegistry, PreGenerationValidator) + facade re-export of ports-{common,read,write,execute}
│   ├── application/               ← use-case orchestration + stage pipelines
│   └── profiling/                 ← perf measurement infrastructure
├── adapters/
│   ├── driven/                    ← outbound: driver-postgresql/-mysql/-sqlite (+ -profiling),
│   │                                formats + formats-parquet, integrations
│   │                                (Flyway/Liquibase/Django/Knex), persistence-jdbc,
│   │                                storage-file/-s3, streaming, text-icu,
│   │                                audit-logging, connection-config
│   └── driving/                   ← inbound: cli, mcp
├── examples/
│   ├── bi-demo/                   ← Compose demo for Parquet/S3/BI flows
│   └── sample-db/                 ← on-demand sample database harness
├── test/
│   ├── consumer-read-probe/       ← read-only consumer surface verification
│   ├── cross-dialect-matrix/      ← workstream × dialect × kind sweep + carve-out registry
│   ├── integration-postgresql/    ← Testcontainers PG live-DB tests
│   ├── integration-mysql/         ← Testcontainers MySQL live-DB tests
│   ├── integration-sqlite/        ← file-backed SQLite live-DB tests
│   ├── integration-mssql/         ← Testcontainers SQL Server live-DB tests
│   ├── integration-oracle/        ← Testcontainers Oracle live-DB tests
│   ├── integration-concurrency/   ← race-condition reproducers (sequence preserve, atomic locks)
│   ├── integration-integrations/  ← export integration contract tests
│   ├── integration-persistence-jdbc/ ← JDBC store + migration runner ITs
│   ├── integration-server-state/  ← MCP server state machine ITs
│   ├── integration-storage-s3/    ← S3-compatible artifact store ITs
│   ├── e2e-cli/                   ← end-to-end CLI + MCP harness scenarios
│   ├── test-images/               ← the container images, one place per dialect
│   ├── perf-data-path/            ← data-path throughput scales
│   └── perf-large-schema/         ← N = 100 / 1000 / 10000 perf scales
├── scripts/                       ← verify-doc-refs.sh, solid-suppression-gate.sh,
│                                    test-integration-docker.sh, kover utilities
├── ledger/                        ← suppression and quality ledgers
├── spec/                          ← normative specs (German): lastenheft, architecture,
│                                    design, cli-spec, neutral-model-spec,
│                                    ddl-generation-rules, mcp-server, schema-reference,
│                                    connection-config-spec
└── docs/
    ├── adr/                       ← Architecture Decision Records + index
    ├── planning/
    │   ├── open/                  ← trigger watch + open follow-ups
    │   ├── next/                  ← planned but not yet active
    │   ├── in-progress/           ← active roadmap + slice plans
    │   └── done/                  ← completed slices + closure notes
    └── user/                      ← user / operator facing (guide.md, releasing.md)

Note: the linked ADRs, slice plans and planning documents under docs/ and spec/ are written in German. The English README mirrors the structure and key facts; for deep-dive content, consult README.de.md or the German source documents.

Documentation

Detailed documentation lives in docs/ and spec/:

• Documentation overview (German)
  • Anwenderhandbuch
  • Administrationshandbuch
  • Migrations-Leitfaden
  • API-Referenz (CLI + MCP)
  • Best-Practices-Leitfaden
  • Troubleshooting-Leitfaden
• Quick Start Guide (German)
• Architecture
• Schema YAML reference
• Neutral model specification
• CLI specification
• MCP server
• DDL generation rules
• Connection and configuration specification
• Roadmap
• Release guide
• Requirements document (German)

Contributing

Contributions are welcome! Please open an issue or a pull request on GitHub.

1. Fork the repository
2. Create a feature branch off main
3. Write tests for your changes (≥ 90 % per-module Kover gate applies)
4. Make sure the Docker CI gates pass (make ci)
5. Submit a pull request against main

License

This project is licensed under the MIT License.