feat(hfs): HFS_CLUSTER fail-fast validation and the job-store selector (Phase 0.2)

[aacruzgon]

Summary

Phase 0.2 of the cluster-capable-state rebuild (discussion #223). Adds the HFS_CLUSTER master switch and a fail-fast validator: under HFS_CLUSTER=true the server refuses to boot on configurations that cannot run as one of N instances, printing one Configuration error: line per violation that names the variable and the safe value, and logs a warning for the configurations that merely run per-instance today. Also parses and validates HFS_JOB_STORE_BACKEND ahead of the subsystems that will consume it.

Independent of #905 (no shared files); branched from main.

Changes

• crates/rest/src/config.rs — cluster: bool (HFS_CLUSTER, default false) and job_store_backend: String (HFS_JOB_STORE_BACKEND, default unset) on ServerConfig; JobStoreBackend { Memory, Database } with job_store_backend_mode() (unset → Database under cluster, else Memory; explicit values win; db accepted). An invalid value is a validate() error naming the variable, so it fails at parse time like every other config error.

• crates/hfs/src/cluster.rs (new) — ClusterConfigView assembled from the rest and audit configs (they only meet in the binary), and validate_cluster_config → ClusterVerdict { errors, warnings }. A pure function, so the table is unit-tested without env mutation. Refusals under HFS_CLUSTER=true:

  • F1 SQLite primary (HFS_STORAGE_BACKEND)
  • F2 local-fs bulk export / bulk submit output (each only when that subsystem is enabled)
  • F3 bulk export on a MongoDB or S3 primary — its job store is the node-local SQLite sidecar built in main.rs, worst case a per-process temp file
  • F4 HFS_AUDIT_BACKEND=file
  • explicit HFS_JOB_STORE_BACKEND=memory

  Warnings (boot proceeds): SQL-on-FHIR $sql-export job state is per-instance until a database controller exists, and HFS_EXPORT_SINK=fs needs a shared directory. A shared NFS export directory is a legitimate deployment, so the sink is a warning, not a refusal.

• crates/hfs/src/main.rs — wires the check right after the storage mode is resolved and before any subsystem starts; warnings via warn!, errors via the existing Configuration error: + exit(1) convention.

• Skills — run-hfs-server gains a Clustering section (both .claude/skills and .agents/skills).

Testing

Red first: the six validator tests were written against a stub returning an empty verdict; four failed (cluster_on_refuses_each_unsafe_dimension, cluster_on_collects_all_violations_at_once, cluster_on_accepts_a_fully_shared_configuration, fs_export_sink_warns_not_refuses), the two "accepts" cases passed trivially. All six pass with the implementation.

• cargo test -p helios-hfs --no-default-features --features R4,sqlite,postgres cluster:: — 6 passed (refusal table of 7 cases, collect-all = 5, disabled subsystems exempt, cluster-off accepts every single-instance default, fs sink warns).
• cargo test -p helios-rest --lib config::tests — 61 passed, including the two new job-store tests.
• Manual boot: HFS_CLUSTER=true HFS_STORAGE_BACKEND=sqlite HFS_JOB_STORE_BACKEND=memory HFS_AUDIT_BACKEND=file … prints five Configuration error: lines and exits 1; the same config without HFS_CLUSTER boots and answers /health 200 with no cluster output; HFS_JOB_STORE_BACKEND=kafka is rejected at parse time.
• cargo fmt --all; CI-exact clippy clean.

Notes

• No migration, no behaviour change unless HFS_CLUSTER=true is set.
• HFS_JOB_STORE_BACKEND is parsed and validated but not yet consumed; the unified job store (Phase 1) reads it.
• The operator chapter (book ch. 15) is a separate PR (0.5); this PR's skill section is the interim reference.

[codecov]

Codecov Report

❌ Patch coverage is 99.61832% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/hfs/src/cluster.rs	99.54%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!