Add read-only HA replication checks

[viliakov]

Adds sts-backup replication check for HDFS, Elasticsearch, Kafka, ClickHouse and ZooKeeper, with table/JSON output and stability waiting. Checks follow StatefulSet configuration without Helm release records or Secret access.

Run against the namespace containing the SUSE Observability instance (no release name required):

sts-backup replication check --namespace observability
sts-backup replication check --namespace observability --wait --stable-for 30s --timeout 10m --output json

For testing directly from this branch's checkout:

go run . replication check --namespace observability --wait

Replace observability with the installation namespace. Uses the current kubeconfig context; --kubeconfig <path> selects another config. --wait requires sustained healthy observations before the final audit; timeout or unverified replication returns nonzero.

Wait mode ignores unrelated Kubernetes updates and resets on topology, readiness or database restarts. Before success, HDFS streams a read-only file/block audit and rechecks health. Audit timeouts and incomplete evidence fail; retries are paced. Active WAL pipeline membership is reported separately from durable-write guarantees.

This reports sampled replication, not node-removal safety. Controlled recovery validation remains outstanding.

Validation: full Go tests and lint pass, including large streamed reports, incomplete evidence and retry/cancellation cases. Live nightly HDFS wait passed with 3,427 completed block entries and nine under-construction blocks; other adapters were also exercised on nightly.

Scope decision: minor review findings 3–6 were deliberately skipped and will not be implemented: the NameNode port documentation note, replacing exec with port-forwarding for Elasticsearch/ClickHouse, cross-referencing docs PR #414, and creating a separate tracking issue.

Tracking: https://github.com/StackVista/stackstate/issues/501

test run on stackstate-rancher-hosted tenant, (The Kafka check fails because __transaction_state topic hasnt been migrated to replicationFactor 1 yet)

❯ mise exec go@1.25.13 -- go run main.go  replication check --namespace stackstate-rancher-hosted --wait
[2026-09-15T10:52:41Z] Waiting for all applicable checks to remain healthy for 30s (timeout 10m0s).
[2026-09-15T10:52:58Z] hdfs unknown: query pod stackstate-rancher-hosted/rancher-hosted-hbase-hdfs-nn-0: error dialing backend: write tcp 10.0.8.160:41002->10.0.4.150:10250: use of closed network connection
[2026-09-15T10:52:58Z] elasticsearch healthy: 125 indices have replica shards; all shards allocated with no recovery or relocation
[2026-09-15T10:52:58Z] kafka degraded: __transaction_state partition 0: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 1: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 2: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 3: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 4: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 5: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 6: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 7: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 8: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 9: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 10: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 11: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 12: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 13: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 14: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 15: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 16: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 17: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 18: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 19: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 20: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 21: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 22: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 23: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 24: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 25: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 26: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 27: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 28: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 29: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 30: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 31: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 32: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 33: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 34: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 35: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 36: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 37: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 38: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 39: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 40: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 41: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 42: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 43: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 44: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 45: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 46: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 47: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 48: fewer than two distinct assigned replicas or invalid assignment; __transaction_state partition 49: fewer than two distinct assigned replicas or invalid assignment
[2026-09-15T10:52:58Z] clickhouse healthy: 5 replicated table groups checked on every member; no pending data replication tasks
[2026-09-15T10:52:58Z] zookeeper healthy: 3 voting members available; one leader and 2 synchronized followers
[2026-09-15T10:52:58Z] Waiting for all selected checks to become healthy; stability period has not started.
[2026-09-15T10:53:24Z] hdfs healthy: 3 DataNodes live; no missing, corrupt, under-replicated or pending-replication blocks
[2026-09-15T10:53:24Z] elasticsearch healthy: 125 indices have replica shards; all shards allocated with no recovery or relocation

[ai-collaboration-app]

Read the branch and checked the chart-side assumptions against a rendered stable/suse-observability, plus the ZooKeeper/ClickHouse/Hadoop/Kafka sources for the metric semantics. The discovery contract holds up — labels, hdfs-nn/hdfs-dn, container names, the single ZK ensemble, mntr, plaintext 9200, the ClickHouse env vars, and the kafka-topics.sh parsing including the leaderless and empty-ISR paths. Three things look like real defects.

1. Default invocation fails on three of the four HA profiles. clickhouse.go:47 returns Degraded when a shard has fewer than two replicas, but clickhouse.replicaCount: 1 in 150-ha, 250-ha and 500-ha — only 4000-ha runs replicated ClickHouse. So replication check -n <ns> exits 1 out of the box on those profiles, permanently. Single-replica ClickHouse is by design there (redundancy is the 3-replica Longhorn class, per rancher/stackstate-product-docs#414), so this wants an outcome distinct from degraded rather than every user having to pass --components.

2. ClickHouse false positive on a never-written replicated table. StorageReplicatedMergeTree::getStatus initialises both log_max_index and log_pointer to 0, and the /log_pointer znode is created empty, so an idle replicated table yields 0 <= 0 and reports a backlog. log_max_index > 0 && guards it. The fixtures use 100/101 and 100, so the boundary is untested.

3. __transaction_state is created lazily, on the first FindCoordinator request for a transactional producer, never at broker startup. I did not find initTransactions, transactional.id or processing.guarantee in stackstate or stackgraph. It clearly existed in your Beest run, so something creates it — worth pinning down what before making it a hard requirement, since otherwise the Kafka check can sit at unknown forever on an install that never uses transactions.

Smaller points:

• fingerprint() invalidates the entire report on any pod or StatefulSet resourceVersion change, which is exactly what the churn window this tool exists for produces. Pod UID set + nodeName + ready condition + STS generation would be less brittle.
• HDFS dfs.replication is a per-file create-time default, so "configured >= 2 and no under-replicated blocks" does not prove existing blocks have two copies — a repl-1 file meets its own target and is never counted. The chart's dfs.namenode.replication.min=2 mitigates it in practice and hdfs fsck / is what would prove it; the caveat in docs/replication.md is close but not quite this.
• NameNode 50070 works only because the chart overrides the Hadoop 3 default of 9870. Brittle coupling, worth a line in the docs.
• pods/exec in the product namespace is effectively read-any-secret. Elasticsearch and ClickHouse could use the port-forward path this CLI already has; Kafka/HDFS/ZK genuinely need exec. Defensible either way, but some security teams will refuse the Job as written.
• Cross-reference rancher/stackstate-product-docs#414. It states the required checks in nearly these words and says the product ships no such automation, while this adds it — and this covers one of its five bullets, so saying which one helps.

For the tracker: StackVista/stackstate#501's acceptance criteria are about storage-class customization, so this probably deserves its own issue on the board.

[ai-collaboration-app]

Re-reviewed at 9812df2. All three blocking findings are fixed, and I verified the fixes rather than taking them on trust: the five new chart assumptions this code introduces — dfs.namenode.replication.min, the stackgraph mono component label and container, the single Elasticsearch StatefulSet and its container name, ClickHouse at 3 replicas on 4000-ha, and offsetsTopicReplicationFactor: 2 — all hold against a rendered stable/suse-observability. Build, tests and CI are green.

Two of the fixes went further than I asked, correctly. Querying system.zookeeper to separate a genuinely empty log from an unprocessed first entry is better than the bare log_max_index > 0 guard I suggested, and the fsck audit closes the per-file gap properly.

One latent defect — Kafka 4.1. The absence probe accepts only UnknownTopicOrPartitionException. That is what kafka-configs.sh --describe produces on 3.9.1, which the chart pins: describeResourceConfig does no existence check, the broker's ConfigHelper returns UNKNOWN_TOPIC_OR_PARTITION, and main prints it via printStackTrace, so the class name appears verbatim and the exit is non-zero. KAFKA-18904 (4.1) added a pre-check that instead prints The topic '<name>' doesn't exist and doesn't have dynamic config. and returns — exit 0, no exception. The probe then reports present, and Kafka goes permanently unknown on any install that never uses transactions, which is the exact failure mode this commit removed. A routine image bump triggers it, so it is worth matching that message in the success branch now.

Worth a line in the code, not just here. I went looking for a cheaper audit — summary-only hdfs fsck /, asserting Minimally replicated blocks == Total blocks — and it is wrong twice over. That counter is gated on liveReplicas + decommissioned + decommissioning + maintenance, so a block with one live copy plus one replica on a decommissioning node passes it, which is precisely the state this tool exists to catch; and totalBlocks includes the incomplete last block the audit loop skips, so the equality can never hold while HBase holds a WAL open. Your per-block Live_repl= parse is strictly stronger, and -files -blocks costs the NameNode nothing extra — collectBlocksSummary runs regardless, the flags only gate printing. One sentence in hdfs_audit.go on why the summary counters are unusable would stop someone reversing this later.

Smaller:

• A parser rejection reports unsupported fsck file or block record with no offending line, and stderr is discarded, so a field failure is undiagnosable. diagnosticPath already gives you safe truncation.
• In the probe path a real Degraded is replaced by unknown when absence cannot be verified, losing the actionable ISR message. Both exit 1, so it only costs diagnosis.
• evaluateClickHouse's needsLogVerification guard is unreachable: verifyEmptyLogs either clears every flag or returns an error first.

One consequence to be deliberate about. not_applicable now exits 0, and ClickHouse is single-replica on 150-ha, 250-ha and 500-ha — only 4000-ha runs three replicas. So on the three most common HA profiles this returns success for ClickHouse having checked availability alone, while what actually protects that data during a rotation is the 3-replica Longhorn class, which is out of scope. The maintenance-boundary section says as much, but an automated rotation loop gated on this exit code will step to the next node while a Longhorn volume is still rebuilding. That is a faithful implementation of the stated scope; it does mean this cannot be the only gate.

[viliakov]

Addressed in 45d3078:

• Kafka's transaction-topic probe now uses --describe --all to bypass Kafka 4.1's ACL-filtered topic-list precheck and distinguish missing topics from authorization failures.
• Failed absence verification preserves existing partition/ISR findings. Absence is reported only after verification succeeds.
• HDFS parser failures include a line number and a bounded, escaped record excerpt, retained alongside execution errors.
• Added a short comment explaining why summary counters cannot replace the per-block HDFS audit.

Full Go tests and lint pass. Regression coverage includes Kafka 3.9/4.1 shell fixtures, ISR diagnostic retention, and streamed HDFS error excerpts. CI is currently running.

The ClickHouse defensive guard and storage-check boundary remain unchanged. Previously excluded minor findings 3–6 (NameNode port note, replacing exec with port-forwarding, docs PR cross-reference, and a separate tracking issue) were deliberately skipped and will not be implemented.

[ai-collaboration-app]

Re-reviewed at 45d3078. Both actionable items and the comment request from the last round are addressed, and I checked the Kafka fix against Kafka's source rather than the stated rationale.

--all works, and for the reason given. KAFKA-18904's precheck is wrapped in if (!describeAll) — that gate was in the original commit and survives the Java rewrite on trunk — so --all goes straight to describeConfigs. On 4.1 the broker's ConfigHelper.describeConfigs still returns UNKNOWN_TOPIC_OR_PARTITION for an unknown topic rather than falling back to defaults, which was the failure mode that would have made this fix worse than the bug; main's printStackTrace puts the class name in stderr, so the probe classifies absent. No 3.9 regression: 3.9 has no precheck at all, and --all only widens the config-source filter, leaving the existence-failure path unchanged.

The ACL half of that rationale is worth more than a passing mention. Because the precheck resolves the name through an ACL-filtered listTopics, the pre---all form could not distinguish a topic the principal cannot list from one that does not exist. This closes a silent misclassification on any ACL-restricted 4.1 broker, not just the version bump.

Modelling the precheck in a shell fixture and asserting the bypass is the right test here, and pinning the branch order with "authorization overrides absence" covers the case where both class names appear in one error.

The parser diagnostics went further than I asked: %q means a crafted HDFS path cannot inject terminal control sequences into an operator's console, and the test asserts that. Chunk-size independence at 1/17/oversize is the part that would actually have caught a streaming bug.

One item from the last round is still open, trivially. evaluateClickHouse's needsLogVerification guard is unreachable: verifyEmptyLogs clears the flag on every row it handles and otherwise returns an error, so checkClickHouse never reaches the guard with it set — only that function's own tests can. Harmless either way.

Noted on scope: skipping findings 3–6 is reasonable, and recording the decision in the PR body is the right way to close them. No objection.

Local go build ./... and the replication, cmd and k8s client test packages pass; CI is green on this head.