ci(cluster): harden the two-instance smoke — nginx probe, SIGTERM drain, nightly tier (Phase 0.4)

[aacruzgon]

Stacked on #907 (the stop step asserts the SIGTERM drain that PR adds). Base branch is cluster/p0.3-sigterm; GitHub retargets to main when #907 merges.

Summary

Phase 0.4 of the cluster-capable-state rebuild (discussion #223): hardens the two-instance smoke harness (cluster-smoke.yml) so later cluster cases can add checks without touching the workflow's plumbing, and fixes a readiness-probe bug that made the skeleton flaky.

Changes

• nginx readiness no longer poisons the upstreams. The skeleton probed nginx by proxying /health to instances that were not started yet; with nginx's default max_fails=1 fail_timeout=10s, those refused connects blacklisted both upstreams for 10 s, and the smoke's first front request landed inside that window as 502 no live upstreams. Both server lines now carry max_fails=0, and readiness probes an nginx-local location = /nginx-health with curl -sf.
• Stop step sends SIGTERM and asserts the drain. Each instance must exit within 10 s of kill -TERM and its log must contain the drain line; needing SIGKILL fails the step (the process is still killed so nothing leaks). This is the CI proof for fix(hfs): drain on SIGTERM as well as Ctrl-C (Phase 0.3) #907.
• Nightly tier trigger. workflow_dispatch gains a boolean nightly input and a schedule (0 6 * * *); a workflow-level NIGHTLY env is true on either, so the kill-9 recovery steps that later PRs add can gate on it. Scheduled runs execute main's copy of the file.
• Build features. The hfs binary is built with R4,postgres,subscriptions,elasticsearch so the WebSocket fan-out and composite-sync cases do not have to change this job; the R4/postgres smoke ignores the extra features. A second build-hts job builds helios-hts --features R4,postgres into hts-cluster-smoke-binary for the terminology cache-coherency case.
• Header comment updated to describe what the base checks cover now.

The smoke script itself is unchanged: health, round-robin, and CRUD A→B.

Testing

• YAML parses; jobs build, build-hts, cluster-smoke; triggers workflow_dispatch, schedule.
• Dispatched on this branch (gh workflow run cluster-smoke.yml --ref cluster/p0.4-smoke-hardening): see the run linked in the PR conversation. Expected: the three skeleton checks pass, the nginx readiness step no longer needs a 502 window, and the stop step reports both instances drained on SIGTERM.

Notes

• No product code; the only Rust this depends on is fix(hfs): drain on SIGTERM as well as Ctrl-C (Phase 0.3) #907's shutdown_signal().
• Building subscriptions and elasticsearch into the smoke binary lengthens the build job somewhat; it is a one-time cost that saves a workflow change per later phase.

[aacruzgon]

Smoke dispatched on this branch: https://github.com/HeliosSoftware/hfs/actions/runs/33669817656 — green on the first run. Both build jobs (including the new build-hts) succeeded, the nginx readiness step passed without a 502 window, the three base checks passed, and the new stop step reported both instances drained on SIGTERM (this is the CI proof for #907).