docs: define rolling Kanidm database maintenance

[pando85]

Summary

Relates to #950.

This PR records the proposed architecture for Kanidm database maintenance and turns the existing maintenance design document into a concrete implementation plan.

The main decision is to preserve HA by treating maintenance as a rolling per-replica offline operation, rather than reusing restore's full-topology quiesce/PVC Job workflow.

Architecture

• add a single immutable KanidmMaintenance one-shot resource;
• maintain at most one replica at a time;
• require another suitable replica to remain serving unless downtime is explicitly allowed;
• restart exactly one StatefulSet-owned Pod instead of scaling its StatefulSet;
• run the offline Kanidm maintenance command in an init container of the replacement Pod, against the same PVC and using the exact Kanidm image;
• keep the normal maintenance init path as a no-op unless an optional operation plan targets that exact Pod;
• persist operation completion on the data PVC so controller/Pod restarts do not lose successful local completion state;
• stop progression immediately when one replica fails maintenance;
• treat Kubernetes readiness as an availability gate, not as proof of replication convergence;
• leave native replication fences as a future strengthening rather than a prerequisite for add database maintenance support #950.

Important safety boundary

Kubernetes cannot provide exactly-once execution if a node dies in the middle of kanidmd database <operation>. The implementation plan therefore makes interruption/retry qualification of each Kanidm maintenance command Phase 0.

Verify should be straightforward. Reindex must pass kill/reopen/retry tests before production enablement. Vacuum remains out unless it independently passes the same safety gate.

If an operation is not safe after arbitrary interruption, the plan is to leave it unsupported or require an upstream Kanidm primitive—not compensate with a more complicated Kubernetes workflow.

Implementation plan

The revised plan breaks work into reviewable phases:

1. qualify Kanidm reindex/verify/vacuum interruption semantics;
2. add CRD and exclusive-operation coordination;
3. add the small maintenance runner and normally-no-op Pod init plumbing;
4. implement a single-replica controller path;
5. extend it to deterministic rolling AllReplicas maintenance;
6. add restart/fault-injection E2E coverage;
7. finish Events, metrics and user documentation.

Estimated production-grade effort is roughly 9-14 engineering days if the upstream command-safety spike is positive.

Scope intentionally excluded

• maintenance schedules;
• log-triggered automatic reindex;
• arbitrary label selectors;
• maxUnavailable > 1;
• full-topology offline maintenance Jobs;
• a new server-side maintenance protocol;
• automatic backup before maintenance;
• a generic workflow engine/refactor as a prerequisite.

Files

• docs/adr/0002-rolling-kanidm-database-maintenance.md
• docs/plans/maintenance-operations-design.md
• ADR/plan indexes

[forkline-bot]

✅ Fix pushed:

Fixed the CI failure by adding trailing newlines to docs/adr/0002-rolling-kanidm-database-maintenance.md and docs/plans/maintenance-operations-design.md. The end-of-file-fixer pre-commit hook had flagged both files.

[forkline-bot]

⏸️ CI Auto-Fix paused

A user commit was detected after an automated fix. CI auto-fix is now paused
to prevent conflicts with your changes.

To resume CI auto-fix, use the /reset or /reset-ci-auto-fix command.

[forkline-bot]

Merge conflict resolution complete. Here's a summary:

Changes made:

1. Merged master into PR branch - Resolved 33 conflicting files
2. Preserved PR's unique contributions:
   • cmd/maintenance-runner/ (new maintenance runner binary)
   • docs/adr/0002-rolling-kanidm-database-maintenance.md (ADR document)
   • libs/operator/src/kanidm/maintenance.rs (maintenance controller)
3. Accepted master's versions for CI configs, code files, and docs (newer, more complete)
4. Fixed integration issues:
   • Added maintenance module import in kanidm/mod.rs
   • Added maintenance-runner to Cargo.toml workspace members
   • Made is_replication_enabled() pub(crate) for module access
   • Removed duplicate restore.rs (master uses directory-based module)
   • Fixed shadowed test function name in maintenance-runner
   • Merged maintenance-operations-design.md from both branches
5. Verified: cargo check and make lint both pass

Pushed: docs/adr-rolling-database-maintenance branch updated to 860ee534