From 7d442b78a2eb286354e29edc8d30486552444fd8 Mon Sep 17 00:00:00 2001 From: Alan Cruz Date: Wed, 2 Sep 2026 14:54:41 -0400 Subject: [PATCH] docs(cluster): add the "Running HFS in a Cluster" operator chapter There was no operator-facing description of what happens when N hfs instances share a database: which subsystems are safe, which still need a pinned instance, what HFS_CLUSTER refuses, and how a cluster is verified. Book chapter 15 covers prerequisites, the one-switch setup with the refusal and warning tables as implemented, a per-subsystem 1-vs-N matrix that names what is safe today (CRUD with compare-and-swap version bumps, schema init, bulk data on PostgreSQL, SIGTERM drain, JWT validation) and what is still per-instance and planned (SQL-on-FHIR async export, reindex, subscriptions, HTS caches and bootstrap, composite async sync), load-balancer notes (sticky sessions required today for exports and subscriptions, and why they are not sufficient for subscriptions), and the two-instance smoke harness. The test-hfs skill gains the T1/T2/T3 cluster testing section in both skill trees. Tests: mdbook build succeeds and the chapter renders; no code changes. --- .agents/skills/test-hfs/SKILL.md | 37 +++++++ .claude/skills/test-hfs/SKILL.md | 37 +++++++ book/src/SUMMARY.md | 1 + book/src/ch15-cluster-deployment.md | 150 ++++++++++++++++++++++++++++ 4 files changed, 225 insertions(+) create mode 100644 book/src/ch15-cluster-deployment.md diff --git a/.agents/skills/test-hfs/SKILL.md b/.agents/skills/test-hfs/SKILL.md index 41310cde24..67c00fadc3 100644 --- a/.agents/skills/test-hfs/SKILL.md +++ b/.agents/skills/test-hfs/SKILL.md @@ -58,6 +58,43 @@ uv run pytest python-tests/ -v cargo test ``` +### Cluster testing (T1/T2/T3) + +Cluster-capable-state work (discussion #223) uses three tiers: + +- **T1 unit** — pure logic with no store: the `HFS_CLUSTER` refusal table in + `crates/hfs/src/cluster.rs`, state machines against in-memory reference + models, serde round-trips. +- **T2 two-handle** — two *independently constructed* backends over one + testcontainer database, raced from a barrier; never a cloned `Arc`, which + shares the in-process heap and proves nothing. Helpers live in + `crates/persistence/tests/common/cluster_harness.rs` (`two_handles`, + `race2`, `assert_exactly_one`, `assert_visible`, + `assert_wrong_tenant_hidden`) and are `#[path]`-included by each test binary + that needs them (nothing declares `tests/common` as a module). Every T2 suite + asserts the applicable definition-of-done rows — visibility, wrong-tenant + isolation (mandatory), claim exclusivity, fencing, durability across a + handle drop, cache invalidation — and, where a `memory` variant exists, that + two memory handles do *not* see each other. Suites that claim from a + cross-tenant queue take a per-suite `tokio::Mutex` so parallel tests cannot + steal each other's rows. +- **T3 two-process** — `.github/workflows/cluster-smoke.yml`: two `hfs` + binaries on one PostgreSQL behind nginx. Reserved for what T2 cannot reach: + a live WebSocket on the far instance and kill-9 recovery of durable job + paths (nightly tier). Dispatch on a branch with + `gh workflow run cluster-smoke.yml --ref ` (add `-f nightly=true` + for the nightly tier); scheduled runs execute `main`'s copy. + +Run the T2 suites that exist today: + +```bash +cargo test -p helios-persistence --features postgres --test postgres_tests -- cluster_ +cargo test -p helios-persistence --features mongodb --test mongodb_tests -- cluster_ +``` + +Every cluster PR writes its test first and quotes the failing run against +`main` in the PR body; a test that cannot go red is not evidence. + ## Test Data - FHIR examples live in `crates/fhir/tests/data/`. diff --git a/.claude/skills/test-hfs/SKILL.md b/.claude/skills/test-hfs/SKILL.md index 41310cde24..67c00fadc3 100644 --- a/.claude/skills/test-hfs/SKILL.md +++ b/.claude/skills/test-hfs/SKILL.md @@ -58,6 +58,43 @@ uv run pytest python-tests/ -v cargo test ``` +### Cluster testing (T1/T2/T3) + +Cluster-capable-state work (discussion #223) uses three tiers: + +- **T1 unit** — pure logic with no store: the `HFS_CLUSTER` refusal table in + `crates/hfs/src/cluster.rs`, state machines against in-memory reference + models, serde round-trips. +- **T2 two-handle** — two *independently constructed* backends over one + testcontainer database, raced from a barrier; never a cloned `Arc`, which + shares the in-process heap and proves nothing. Helpers live in + `crates/persistence/tests/common/cluster_harness.rs` (`two_handles`, + `race2`, `assert_exactly_one`, `assert_visible`, + `assert_wrong_tenant_hidden`) and are `#[path]`-included by each test binary + that needs them (nothing declares `tests/common` as a module). Every T2 suite + asserts the applicable definition-of-done rows — visibility, wrong-tenant + isolation (mandatory), claim exclusivity, fencing, durability across a + handle drop, cache invalidation — and, where a `memory` variant exists, that + two memory handles do *not* see each other. Suites that claim from a + cross-tenant queue take a per-suite `tokio::Mutex` so parallel tests cannot + steal each other's rows. +- **T3 two-process** — `.github/workflows/cluster-smoke.yml`: two `hfs` + binaries on one PostgreSQL behind nginx. Reserved for what T2 cannot reach: + a live WebSocket on the far instance and kill-9 recovery of durable job + paths (nightly tier). Dispatch on a branch with + `gh workflow run cluster-smoke.yml --ref ` (add `-f nightly=true` + for the nightly tier); scheduled runs execute `main`'s copy. + +Run the T2 suites that exist today: + +```bash +cargo test -p helios-persistence --features postgres --test postgres_tests -- cluster_ +cargo test -p helios-persistence --features mongodb --test mongodb_tests -- cluster_ +``` + +Every cluster PR writes its test first and quotes the failing run against +`main` in the PR body; a test that cannot go red is not evidence. + ## Test Data - FHIR examples live in `crates/fhir/tests/data/`. diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 8431d7b074..31e8c6aa2c 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -16,6 +16,7 @@ - [Embedding HFS as a Library](ch12-embedding.md) - [Development Setup](ch13-dev-setup.md) - [Contributing Guidelines](ch14-contributing.md) +- [Running HFS in a Cluster](ch15-cluster-deployment.md) --- diff --git a/book/src/ch15-cluster-deployment.md b/book/src/ch15-cluster-deployment.md new file mode 100644 index 0000000000..140dfb4872 --- /dev/null +++ b/book/src/ch15-cluster-deployment.md @@ -0,0 +1,150 @@ +# Running HFS in a Cluster + +This chapter describes what happens when you run more than one `hfs` instance +behind a load balancer, which subsystems are cluster-safe today, which are +still single-instance, and how the `HFS_CLUSTER` switch enforces a safe +configuration at boot. + +See discussion [#223](https://github.com/HeliosSoftware/hfs/discussions/223) +for the design background. This chapter is the operator-facing summary and is +updated as each cluster-capable subsystem lands. + +--- + +## Prerequisites + +A cluster is only as shared as its slowest-moving state. Before starting a +second instance, you need: + +1. **A shared primary database** — PostgreSQL, MongoDB, or an + Elasticsearch-backed mode over them (`HFS_STORAGE_BACKEND=postgres`, + `postgres-elasticsearch`, `mongodb`, …). **SQLite cannot be clustered**: it + is a single-writer local file, and `HFS_CLUSTER=true` refuses to boot on it. + Prefer PostgreSQL: it is the primary every cluster-capable subsystem is + built on first, and bulk export needs it (see below). +2. **Shared object storage for bulk output** — set + `HFS_BULK_EXPORT_OUTPUT_BACKEND=s3` and `HFS_BULK_SUBMIT_OUTPUT_BACKEND=s3`. + The default `local-fs` writes to node-local disk, so a download routed to a + different node than the writer returns 404. +3. **A shared audit sink** — `HFS_AUDIT_BACKEND=database` or `cloudwatch`. + The `file` sink writes node-local NDJSON and produces a fragmented, + restart-lossy audit trail. +4. **A load balancer that forwards SIGTERM-friendly deploys.** Every instance + drains in-flight requests and flushes its audit sink on SIGTERM as well as + Ctrl-C, so a rolling restart can stop an instance the way container + runtimes and process supervisors do. + +--- + +## The one-switch setup: `HFS_CLUSTER=true` + +Setting `HFS_CLUSTER=true` declares "this process is one of N". It does two +things: + +1. **Flips cluster-safe defaults** — `HFS_JOB_STORE_BACKEND` defaults to + `database` instead of `memory`. (The unified job store that reads it is + the next subsystem to land; until then the selector is parsed and + validated but not consumed.) +2. **Fails fast on unsafe combinations** instead of silently degrading. The + server refuses to boot, printing one `Configuration error:` line per + violation so a deployment can be fixed in one pass: + +| Check | Refused when clustered | Fix | +|-------|------------------------|-----| +| Primary backend | `HFS_STORAGE_BACKEND=sqlite` (or `sqlite-elasticsearch`) | Use `postgres`, `mongodb`, or an `*-elasticsearch` mode over them | +| Bulk export output | `HFS_BULK_EXPORT_OUTPUT_BACKEND=local-fs` (with bulk export enabled) | Use `s3` | +| Bulk submit output | `HFS_BULK_SUBMIT_OUTPUT_BACKEND=local-fs` (with bulk submit enabled) | Use `s3` | +| Bulk export job store | bulk export enabled on a `mongodb` or `s3` primary — its job store is a node-local SQLite sidecar (worst case a per-process temp file), so job state is invisible to other instances | Use a `postgres` primary, or `HFS_BULK_EXPORT_ENABLED=false` | +| Audit sink | `HFS_AUDIT_BACKEND=file` | Use `database` or `cloudwatch` | +| Job store | explicit `HFS_JOB_STORE_BACKEND=memory` | Remove it (defaults to `database` when clustered) or set `database` | + +Two configurations run but log a warning at boot: + +| Warning | Why | What to do | +|---------|-----|------------| +| SQL-on-FHIR `$sql-export` enabled | Async export job state is held in-process by the instance that accepted the export, so a poll, cancel, or download routed elsewhere returns 404 | Pin export clients to one instance or use sticky sessions until the database-backed controller lands; or `HFS_SOF_ENABLED=false` | +| `HFS_EXPORT_SINK=fs` | Export output under `HFS_EXPORT_DIR` is only reachable from every instance if that directory is shared (e.g. NFS) | Share the directory, or use `s3` | + +With `HFS_CLUSTER` unset (the default), nothing changes: a single instance +keeps its zero-configuration SQLite defaults. + +A minimal two-instance setup on one shared PostgreSQL: + +```bash +export HFS_CLUSTER=true +export HFS_STORAGE_BACKEND=postgres +export HFS_DATABASE_URL=postgresql://… # same URL on every instance +export HFS_BULK_EXPORT_OUTPUT_BACKEND=s3 +export HFS_BULK_SUBMIT_OUTPUT_BACKEND=s3 +export HFS_AUDIT_BACKEND=database + +hfs --port 8080 & # instance A +hfs --port 8081 & # instance B (in practice: another machine/pod) +``` + +--- + +## Per-subsystem behavior with 1 vs N instances + +The cluster-capable-state work lands one subsystem at a time; this matrix is +the current state and is updated as each one ships. + +| Subsystem | 1 instance | N instances today | Status | +|-----------|------------|-------------------|--------| +| FHIR CRUD, search, history | ✅ | ✅ shared-database backends are stateless per request; every version bump is a version-guarded compare-and-swap, so two instances racing an unconditional `PUT` (including onto a deleted id) both succeed with distinct versions and a complete history | safe | +| Concurrent cold-start schema init | ✅ | ✅ serialized by a PostgreSQL advisory lock | safe | +| Bulk Data `$export` / `$bulk-submit` (jobs) | ✅ | ✅ DB-leased jobs with fencing tokens on a PostgreSQL primary; every instance runs competing workers. ❌ bulk export on a MongoDB/S3 primary uses a node-local sidecar job store — refused by `HFS_CLUSTER` | safe on PostgreSQL | +| Bulk output download | ✅ | ✅ with `s3` output; ❌ with `local-fs` — refused by `HFS_CLUSTER` | safe with `s3` | +| Graceful shutdown | ✅ | ✅ SIGTERM and Ctrl-C both drain and flush | safe | +| JWT validation | ✅ | ✅ stateless per-request validation; auth holds no cross-instance state. Each instance refreshes the IdP's JWKS independently (N fetches per boot or key rotation — a cost, not a correctness issue) | safe | +| SQL-on-FHIR `$sql-export` (async jobs) | ✅ | ⚠️ job state is per-instance: poll, cancel and download must reach the instance that accepted the export (`HFS_CLUSTER` warns) | **planned** — unified job store + database controller (#169) | +| Search `$reindex` jobs | ✅ | ⚠️ job state is per-instance: `$reindex-status` and cancel must reach the instance that started the job | **planned** — unified job store | +| Subscriptions (topics, delivery, WebSockets) | ✅ | ⚠️ single-instance: a Subscription created on instance A fires only for writes served by A until B restarts (registries hydrate at boot only); a WebSocket must be bound to the instance that minted its token and only receives notifications for writes served there; `eventNumber` and failure counters are per-instance; pending retries are lost on redeploy | **planned** — shared state + fan-out (#170) | +| Terminology (HTS) response caches | ✅ | ⚠️ `hts` is a separate binary; its response caches are per-instance and cleared only by writes served locally, so an import on one instance leaves the others stale | **planned** — HTS opt-in epoch invalidation | +| HTS bootstrap import | ✅ | ⚠️ N cold-starting `hts` instances all run the same bootstrap import (idempotent, but N× the cost) | **planned** — advisory lock | +| Composite async search sync (`*-elasticsearch`) | ⚠️ in-process queue; a crash loses queued secondary writes | ⚠️ same, per instance | **planned** — durable outbox | +| Audit, metrics, health | ✅ | ✅ with shared sinks; `/metrics` is per-instance (aggregate in your scraper) | safe | + +--- + +## Load-balancer notes + +- **Round-robin is fine for CRUD** — any instance answers any request against + the shared database. +- **Sticky sessions are required today for SQL-on-FHIR async exports and for + Subscriptions.** An export's status, cancel, and download URLs must reach + the instance that accepted it, and a WebSocket subscriber must bind to the + instance that minted its token. Note that sticky sessions are not + sufficient for Subscriptions: the write that triggers a notification may be + served by any instance, and only the instance holding the socket can + deliver it. Until the fan-out lands, run Subscriptions on one instance. +- **WebSocket upgrades** need the usual `Upgrade`/`Connection` header + pass-through on the proxy. +- **Health checks**: probe each instance's `/health` directly, not through + the balancer, so one bad instance can't mask another. +- **Rolling deploys**: send SIGTERM and wait for the instance to exit; it + drains in-flight requests and flushes the audit sink before it does. + +--- + +## Verifying a cluster + +The repository ships a two-instance smoke harness — two `hfs` processes +sharing one PostgreSQL behind an nginx round-robin front +(`.github/workflows/cluster-smoke.yml`, driven by +`crates/hfs/tests/cluster/run_external_cluster_smoke.sh`). It asserts health +on both instances and the front, real round-robin distribution +(`X-Hfs-Upstream` alternates), cross-instance visibility (write on A, read on +B), and that both instances drain on SIGTERM. Each subsystem that becomes +cluster-safe adds its own cross-instance check to the same harness. + +The same checks are easy to run by hand against any deployment: create a +resource through the balancer, then `GET` it repeatedly and confirm every +instance serves it; stop one instance with SIGTERM and confirm it logs +`Shutdown signal received` and exits cleanly. + +How the cluster-capable work is tested is described in the `test-hfs` +project skill: pure unit tables for the boot-time validation, two-handle +tests that run two independently constructed backends over one database to +prove a protocol across instances, and this two-process smoke for what only a +real socket or a real process can show.