fix: compare Keeper replica versions numerically, not lexicographically#260
Open
nahuel11500 wants to merge 1 commit into
Open
fix: compare Keeper replica versions numerically, not lexicographically#260nahuel11500 wants to merge 1 commit into
nahuel11500 wants to merge 1 commit into
Conversation
The observed cluster version was picked with a plain string comparison, so across a digit-count boundary (e.g. 25.9 -> 25.10) the older version won and Status.Version and the VersionInSync condition reported the already-upgraded replicas as out of sync. Reuse upgrade.ParseBareVersion/CompareVersions to pick the newest reported version, falling back to the previous behavior if a version does not parse.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
keeperReconciler.evaluateVersionConditionspicks the cluster's observed version by comparing the replicas' reported version strings with Go's<operator, which is lexicographic. Whenever a component crosses a digit-count boundary (e.g.25.9→25.10), the older version compares as "greater":"25.9.1.100" > "25.10.2.5".During a rolling upgrade across such a boundary this means:
KeeperCluster.Status.Versionkeeps reporting the old version even after replicas have upgraded, andGetVersionSyncConditionuses the old version as the reference, so theVersionInSynccondition flags the already-upgraded replicas as mismatched instead of the pending ones.The impact is limited to status/condition reporting, but ClickHouse crosses a
.9 → .10minor boundary every year, so any Keeper upgrade over that boundary hits it.What
Replaced the string comparison with a small
newerVersionhelper that reuses the existingupgrade.ParseBareVersion/upgrade.CompareVersions(already used for the same purpose elsewhere, e.g. in the upgrade checker and fetcher), falling back to the previous lexicographic behavior if a version doesn't parse.Added a unit test in the existing
Keeper version statusspec that reproduces the bug with two replicas on25.9.1.100and25.10.2.5: before the fixStatus.Versionreports25.9.1.100, after the fix it correctly reports25.10.2.5.The ClickHouse controller is not affected — it takes
Status.Versionfrom the version probe result directly rather than from a max over replica versions.Related Issues
None — found while reading the reconciler code.