Skip to content

release: in-cluster upgrade receipt and pre-upgrade validation hook #1977

Description

@kristinapathak

The cluster needs to record which stack version it is running, because nothing else does. Helm knows chart versions per release; there is no bundle-level version anywhere in cluster state.

  • A ConfigMap recording installed stack version, bundle digest, and per-migration completion status.
  • A pre-install/pre-upgrade hook that reads it, compares against the target bundle, and exits non-zero before any workload changes if the customer is skipping a required major or a required migration has not completed.

The check itself is small — comparing two parsed semver values plus a ConfigMap read. It runs as a Helm hook rather than a client-side check in Makefile.dist because a client-side check is bypassable by invoking helmfile apply directly, and the failure it prevents (a partially upgraded control plane over un-migrated data) is expensive.

Bootstrapping onto existing clusters

There are real 0.x installs in the field (the 0.x line spans v0.7.0 through v0.20.7), and nothing published so far writes a receipt — including 1.0.0. So a cluster arriving at 2.0.0 from 0.x and one arriving from 1.0.0 look identical to a preflight: no receipt at all.

That leaves a gate with two incoherent options. Fail closed on a missing receipt and no existing customer can ever adopt 2.0. Fail open and the gate is permanently useless, because a 1.x → 3.x jump also presents no receipt.

Ship the receipt writer in the 1.x line first. Only the writer, not the validation hook: 1.x has nothing before it to gate on, and a writer cannot block an upgrade whereas a misbehaving checker can. A customer who installs any 1.x carrying it gets a receipt recording their installed version, and from 2.0.0 onward the normal rule applies with no special cases.

The 2.0.0 preflight then treats an absent receipt as a refusal with an instruction the customer can actually follow: install the latest 1.x first, then retry. That works for the 0.x cohort too — they have to pass through 1.x anyway under "install each major in order" — so no version-inference or chart-fingerprinting fallback is needed.

Two notes on sequencing:

  • Adding a Job plus its RBAC is a new capability, so conventional commits make this a feat and a minor bump rather than a patch. A 1.0.x patch is not readily available in any case: deploy/stacks/self-managed/VERSION already reads 1.1.0 and there is no release-deploy/stacks/self-managed/v1.0 branch to cut one on, so 1.1.0 is where this lands.
  • This is cheapest to do before the first data backfill ships. No migration in the corpus performs a backfill today, so a receipt written at install time is trivially accurate. Once a backfill exists, a newly introduced receipt has to make a claim about work that may already have run, which is exactly the kind of guess release: migration tasks report their own completion status #1978 warns against.

Third-party subchart skew across a 0.x → 2.0 jump (Cassandra, NATS, openbao crossing their own majors) is a separate hazard that no receipt catches, and probably the more likely breaker in practice.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

cicdContinuous integration, build, and release pipeline work

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions