Resolve policy per pod and make it part of profile identity (#87)

CHANGELOG.md

@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **BREAKING (data): a `ResourcePolicy` in any namespace no longer outranks every `ClusterResourcePolicy` fleet-wide, and policies are no longer resolved differently at admission than during measurement and resize** ([#87](https://github.com/Tight-Line/ballast/issues/87)). Both were the same defect. `WorkloadProfile` is cluster-scoped and carries only its identity tuple, so the metrics collector and resource adjuster were resolving policy from an input with no namespace, no owner kind, and no annotations. `client.InNamespace("")` means *all namespaces*, and namespace-scoped policies rank above cluster-scoped ones regardless of priority, so one `ResourcePolicy` with a loose selector governed recommendations and resizes for every matching profile in the cluster, outranking policies that explicitly set a higher priority. In the same breath, any policy selecting on `kinds`, `annotations`, or `namespaces.include` matched at admission but could never match in the controllers, so pods were admitted with one policy's values and then resized toward another's, with nothing logging a conflict because each path resolved successfully on its own terms.
+
+  Policy is now resolved exactly once per pod, by the workloadwatcher, which is the only component holding a pod and therefore the only one that can evaluate a selector correctly. The result is recorded on `workloadprofile.status.policyRef`; the metrics collector and resource adjuster read it instead of resolving again, so all three paths agree by construction. A `Resolver` `Input` with no namespace can no longer match a `ResourcePolicy` at all.
+
+  **The governing policy is now part of a profile's identity.** A profile holds one set of recommendations per container, and the policy chooses the metrics sources, poll cadence, tracked resources, aggregation, and headroom that produce them, so pods resolving to different policies now belong to different profiles. Profile names gain a policy token (`checkout--server--fleet-a1b2c3d4`), and each profile owns its own Redis key namespace via `status.measurementHash`, so sibling profiles never write into one sample series and the profile finalizer purges only its own keys.
+
+  **Upgrade impact: every existing `WorkloadProfile` is replaced by a newly-named one, and the fleet accrues from zero for one `readiness.minTimeSpan` (24h by default) before resizes resume.** No resizes are issued during that window; admission continues to apply nothing until each new profile is ready, so pods keep whatever requests they were created with. The old profiles orphan, age out over `orphanTTL` (168h), and their Redis history is purged by the finalizer as they go. Nothing needs to be done by hand.
+
+### Added
+
+- **Policy changes now take effect on a running cluster without waiting for pod churn.** The workloadwatcher watches both policy kinds and re-evaluates every enrolled pod when a policy is created, deleted, or has its selector or priority changed. Edits to the rest of a policy's spec (aggregation, headroom, thresholds, sources, cadence) are deliberately *not* treated as identity changes: they are read live on the next collection or resize cycle, so they take effect without re-keying measurement history or forcing a fresh accrual.
+- **`status.profileDiscriminator` on `ClusterResourcePolicy` and `ResourcePolicy`**, shown as a `DISCRIMINATOR` column in `kubectl get`. It is the token that policy contributes to the names of the profiles it governs, so they can be listed without computing a hash by hand:
+  `kubectl get workloadprofiles | grep "$(kubectl get crp fleet -o jsonpath='{.status.profileDiscriminator}')"`. Derived from the policy's kind, namespace, and name, so two same-named `ResourcePolicies` in different namespaces stay distinct.
+- **`POLICY` column on `kubectl get workloadprofiles`**, showing the governing policy from `status.policyRef`.
+- The `policy-ref` pod annotation is now refreshed by the workloadwatcher when a policy change moves a pod. Previously it kept advertising whatever was resolved at admission.
+- Documented on the `PolicySelector` CRD type which selector fields can scope measurement and which cannot: `namespaces` and `labelSelector` are expressible as server-side pod queries, `annotations` is not, so annotation selectors govern admission but cannot narrow a profile to an annotated subset of its tuple.
+
 ## [0.5.0] - 2026-07-28
 
 ### Changed

README.md

@@ -100,6 +100,8 @@ Ballast groups pods into `WorkloadProfile` objects by matching a configurable se
 
 **WorkloadProfiles are cluster-scoped.** Every pod in every namespace that shares the same label values for the identity keys feeds measurements into the same profile. This is intentional: forty dev namespaces all running the same billing app produce one well-sampled `WorkloadProfile`, not forty thin ones.
 
+The one thing that splits an identity tuple is the policy governing it. A profile holds a single set of recommendations, and the policy is what decides how they are measured and sized, so pods that resolve to different policies get different profiles (see [policy precedence](#default-metricssource-and-clusterresourcepolicy)). With only the default cluster-wide policy in place, nothing splits and the tuple is the whole story.
+
 ### Default: `name` + `component`
 
 ```yaml
@@ -232,13 +234,15 @@ Two flags make rolling out enrollment across a large cluster quick, in two stage
 
 ## Verifying a WorkloadProfile
 
-Once a pod carrying the `ballast.tightlinesoftware.com/mode` label is running, Ballast creates a `WorkloadProfile` for its identity tuple. Check it with:
+Once a pod carrying the `ballast.tightlinesoftware.com/mode` label is running, Ballast creates a `WorkloadProfile` for its identity tuple and the policy governing it. Check it with:
 
 ```bash
 kubectl get workloadprofiles
-kubectl describe workloadprofile billing--api--prod
+kubectl describe workloadprofile <name-from-the-list-above>
 ```
 
+A profile's name is its identity-tuple values joined with `--`, followed by a token identifying its policy (`billing--api--prod--default-a1b2c3d4`), so list the profiles first rather than assuming a name. The `POLICY` column shows which policy governs each one.
+
 The profile status shows accumulated usage statistics and recommendations once the readiness threshold is met (default: 250 samples collected over 24 hours). CPU, memory, and ephemeral storage are all tracked and sized:
 
 ```yaml
@@ -419,7 +423,16 @@ This catch-all policy applies to every opted-in pod in the cluster. Key design d
 - **250 samples over 24 hours before acting.** At the 5-minute poll interval a single long-running pod accrues ~288 samples in 24h, so the 24h window — not the sample count — is the binding constraint. A high coefficient of variation (CV > 1.5) also blocks action — it means the workload is too spiky to size reliably. The CV check is skipped when mean usage sits below a tiny per-resource floor (`cvMeanFloor`, defaults: 25m CPU, 25Mi memory, 2Mi ephemeral-storage): CV divides by the mean, so near-idle workloads produce huge CVs from quantization noise and rare startup spikes alone, and without the floor a single near-idle resource would pin the whole profile at `Accruing` forever — blocking recommendations for every other resource. Usage below the floor is too small for a mis-sized recommendation to matter.
 - **10% drift threshold.** A resize only fires when the current resource value deviates from the recommendation by more than 10%. In-place resize is cheap and safe (a request/limit patch on a running pod, no restart), so the band is deliberately tight: a recommendation that has moved more than 10% reflects a real shift in observed usage worth acting on, not noise.
 - **50% max change per cycle.** Each resize moves at most half the remaining gap between the current value and the recommendation, giving workloads time to stabilize between adjustments. The first step makes most of the correction; once a step would land within the drift threshold, the recommendation is applied exactly, so convergence completes instead of stalling just inside the threshold.
-- **Priority 0.** This is the lowest possible priority. Any `ClusterResourcePolicy` or `ResourcePolicy` with `priority > 0` wins for matched workloads, so you can override specific namespaces or workload kinds without touching this default.
+- **Priority 0.** This is the lowest possible priority. Any `ClusterResourcePolicy` or `ResourcePolicy` with `priority > 0` wins for matched workloads, so you can override specific namespaces or workload kinds without touching this default. A `ResourcePolicy` also beats a `ClusterResourcePolicy` for pods in its own namespace regardless of priority, on the principle that the namespace owner's policy is the more specific match; it has no effect on pods in any other namespace.
+
+Each workload is governed by exactly one policy, and that policy is part of the workload's `WorkloadProfile` identity: pods that resolve to different policies get different profiles, because a profile holds one set of recommendations and the policy is what produces them. Profile names therefore carry a token identifying the policy (`checkout--server--default-a1b2c3d4`), and `kubectl get workloadprofiles` shows the governing policy in its own column. To list the profiles a policy governs:
+
+```sh
+kubectl get workloadprofiles | grep "$(kubectl get clusterresourcepolicy default \
+  -o jsonpath='{.status.profileDiscriminator}')"
+```
+
+Applying, editing, or deleting a policy takes effect on a running cluster within seconds; no pod restart is needed. Changing a policy's **selector or priority** re-scopes which workloads it governs, so affected pods move to a different profile and begin accruing history there. Changing anything else (aggregation, headroom, thresholds, sources, cadence) takes effect on the next collection or resize cycle with no loss of history.
 
 ### Policy presets
 

api/v1/clusterresourcepolicy_types.go

@@ -37,6 +37,19 @@ type ClusterResourcePolicySpec struct {
 }
 
 // PolicySelector filters which workloads a policy applies to.
+//
+// Every field is evaluated against a real pod, at admission and again whenever the
+// workloadwatcher reconciles that pod. Both use the same inputs, so all four fields
+// are honored consistently and a pod is measured and resized under the same policy
+// it was admitted with; the resolved policy is recorded on the WorkloadProfile's
+// status.policyRef, and the metrics collector and resource adjuster read it there
+// rather than re-deriving it from the profile.
+//
+// One asymmetry is inherent rather than incidental: a policy decides which pods it
+// governs, but the pods a WorkloadProfile *measures* are fetched with a
+// server-side label selector. Distinctions that Kubernetes can express in such a
+// query (namespace, labels) can therefore split measurement; annotations cannot.
+// See the note on Annotations below.
 type PolicySelector struct {
 	// Kinds lists the owner kinds this policy applies to (e.g. Deployment, StatefulSet).
 	// Empty means all kinds.
@@ -48,6 +61,13 @@ type PolicySelector struct {
 	Namespaces NamespaceSelector `json:"namespaces,omitempty"`
 
 	// Annotations maps annotation keys to regex patterns that must match on the pod.
+	//
+	// Annotation selectors decide which policy governs a pod, but they cannot
+	// scope *measurement*. A WorkloadProfile gathers its pods with a label
+	// selector served by the API server, and there is no equivalent query for
+	// annotations, so a profile cannot be narrowed to "the annotated subset" of
+	// its identity tuple. Where that distinction matters, use a label instead:
+	// labelSelector and namespaces are both expressible server-side.
 	// +optional
 	Annotations map[string]string `json:"annotations,omitempty"`
 
@@ -200,11 +220,33 @@ type ResizeConfig struct {
 }
 
 // ClusterResourcePolicyStatus defines the observed state of ClusterResourcePolicy.
-type ClusterResourcePolicyStatus struct{}
+type ClusterResourcePolicyStatus struct {
+	// ProfileDiscriminator is the token this policy contributes to the name of
+	// every WorkloadProfile it governs, in the form "<policy-name>-<hash>".
+	//
+	// A WorkloadProfile's identity includes the policy governing it, because the
+	// policy decides which metrics sources are polled, how samples are
+	// aggregated, and how much headroom is added; pods resolving to different
+	// policies therefore cannot share one set of recommendations. This field
+	// makes the resulting profile names traceable back to their policy without
+	// recomputing a hash by hand:
+	//
+	//	kubectl get workloadprofiles | grep "$(kubectl get crp fleet \
+	//	  -o jsonpath='{.status.profileDiscriminator}')"
+	//
+	// The token is derived from the policy's kind, namespace, and name, so it is
+	// stable for the life of the object and identical across every profile that
+	// resolves to it. Renaming a policy produces a different token, and hence
+	// new profiles.
+	// +optional
+	ProfileDiscriminator string `json:"profileDiscriminator,omitempty"`
+}
 
 // +kubebuilder:object:root=true
 // +kubebuilder:resource:scope=Cluster
+// +kubebuilder:subresource:status
 // +kubebuilder:printcolumn:name="Priority",type="integer",JSONPath=".spec.priority"
+// +kubebuilder:printcolumn:name="Discriminator",type="string",JSONPath=".status.profileDiscriminator"
 // +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
 
 // ClusterResourcePolicy is the Schema for the clusterresourcepolicies API

api/v1/resourcepolicy_types.go

@@ -16,11 +16,14 @@ import (
 type ResourcePolicySpec = ClusterResourcePolicySpec
 
 // ResourcePolicyStatus defines the observed state of ResourcePolicy.
-type ResourcePolicyStatus struct{}
+// It uses the same shape as ClusterResourcePolicyStatus.
+type ResourcePolicyStatus = ClusterResourcePolicyStatus
 
 // +kubebuilder:object:root=true
 // +kubebuilder:resource:scope=Namespaced
+// +kubebuilder:subresource:status
 // +kubebuilder:printcolumn:name="Priority",type="integer",JSONPath=".spec.priority"
+// +kubebuilder:printcolumn:name="Discriminator",type="string",JSONPath=".status.profileDiscriminator"
 // +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
 
 // ResourcePolicy is the Schema for the resourcepolicies API

api/v1/workloadprofile_types.go

@@ -18,6 +18,7 @@ import (
 // +kubebuilder:resource:scope=Cluster
 // +kubebuilder:subresource:status
 // +kubebuilder:printcolumn:name="ActiveWorkloads",type="integer",JSONPath=".status.activeWorkloads"
+// +kubebuilder:printcolumn:name="Policy",type="string",JSONPath=".status.policyRef.name"
 // +kubebuilder:printcolumn:name="State",type="string",JSONPath=".status.state"
 // +kubebuilder:printcolumn:name="Orphaned",type="string",JSONPath=".status.conditions[?(@.type=='Orphaned')].status"
 // +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
@@ -45,12 +46,69 @@ const (
 	WorkloadProfileStateSufficient WorkloadProfileState = "Sufficient"
 )
 
+// Policy object kinds, as recorded in PolicyReference.Kind.
+const (
+	// KindClusterResourcePolicy is the cluster-scoped policy kind.
+	KindClusterResourcePolicy = "ClusterResourcePolicy"
+	// KindResourcePolicy is the namespace-scoped policy kind.
+	KindResourcePolicy = "ResourcePolicy"
+)
+
+// PolicyReference identifies the policy that governs a WorkloadProfile.
+type PolicyReference struct {
+	// Kind is the referenced policy's kind.
+	// +kubebuilder:validation:Enum=ClusterResourcePolicy;ResourcePolicy
+	Kind string `json:"kind"`
+
+	// Namespace is the policy's namespace, empty for a ClusterResourcePolicy.
+	// +optional
+	Namespace string `json:"namespace,omitempty"`
+
+	// Name is the policy object's name.
+	Name string `json:"name"`
+}
+
+// Key returns the canonical identity of the referenced policy as
+// "kind/namespace/name". Two ResourcePolicies in different namespaces may share
+// a name, so namespace and kind both participate: the key is what distinguishes
+// genuinely different policies wherever a policy identity is hashed or compared.
+func (r PolicyReference) Key() string {
+	return r.Kind + "/" + r.Namespace + "/" + r.Name
+}
+
 // WorkloadProfileStatus holds the observed state of a WorkloadProfile.
 type WorkloadProfileStatus struct {
 	// TupleLabels are the identity labels that define this profile.
 	// +optional
 	TupleLabels map[string]string `json:"tupleLabels,omitempty"`
 
+	// PolicyRef identifies the policy governing this profile, or is unset when no
+	// policy currently matches (the metrics collector then has nothing to
+	// measure with and skips the profile).
+	//
+	// The governing policy is part of a profile's identity, not merely an
+	// observation about it: the policy chooses the metrics sources, the poll
+	// cadence, the tracked resources, the aggregation, and the headroom behind
+	// one set of recommendations. Pods that resolve to different policies
+	// therefore belong to different profiles. The workloadwatcher resolves policy
+	// per pod (the only place with a pod's namespace, labels, annotations, and
+	// owner kind) and records the result here; the metrics collector and resource
+	// adjuster read it rather than resolving again, so all three agree by
+	// construction.
+	// +optional
+	PolicyRef *PolicyReference `json:"policyRef,omitempty"`
+
+	// MeasurementHash identifies the Redis key namespace this profile owns. It
+	// covers both TupleLabels and PolicyRef, so profiles that share a tuple but
+	// resolve to different policies keep separate sample series and the profile
+	// finalizer can purge its own keys without reference counting.
+	//
+	// Recorded here rather than recomputed on demand so that the collector's
+	// writes and the finalizer's purge can never disagree about which keys belong
+	// to this profile.
+	// +optional
+	MeasurementHash string `json:"measurementHash,omitempty"`
+
 	// SelectorLabels are used to query pods from the metrics API.
 	// Keys absent from the originating pod carry the sentinel value "--missing--",
 	// which the metrics plugin translates to a Kubernetes "!key" (does-not-exist)

api/v1/workloadprofile_types_test.go

@@ -0,0 +1,56 @@
+/*
+Copyright 2026 Tight Line LLC.
+
+Licensed under the MIT License. See LICENSE for the full text.
+*/
+
+package v1_test
+
+import (
+	"testing"
+
+	ballastv1 "github.com/tight-line/ballast/api/v1"
+)
+
+// Key is what distinguishes genuinely different policies wherever a policy
+// identity is hashed or compared, so kind and namespace must both participate:
+// two ResourcePolicies in different namespaces may share a name.
+func TestPolicyReference_Key(t *testing.T) {
+	tests := []struct {
+		name string
+		ref  ballastv1.PolicyReference
+		want string
+	}{
+		{
+			name: "cluster-scoped policy has no namespace segment",
+			ref:  ballastv1.PolicyReference{Kind: ballastv1.KindClusterResourcePolicy, Name: "fleet"},
+			want: "ClusterResourcePolicy//fleet",
+		},
+		{
+			name: "namespaced policy carries its namespace",
+			ref: ballastv1.PolicyReference{
+				Kind:      ballastv1.KindResourcePolicy,
+				Namespace: "team-a",
+				Name:      "local",
+			},
+			want: "ResourcePolicy/team-a/local",
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := tc.ref.Key(); got != tc.want {
+				t.Errorf("Key() = %q, want %q", got, tc.want)
+			}
+		})
+	}
+}
+
+// Same name, different namespaces: different policies, so different keys.
+func TestPolicyReference_KeyDistinguishesNamespace(t *testing.T) {
+	a := ballastv1.PolicyReference{Kind: ballastv1.KindResourcePolicy, Namespace: "team-a", Name: "defaults"}
+	b := ballastv1.PolicyReference{Kind: ballastv1.KindResourcePolicy, Namespace: "team-b", Name: "defaults"}
+	if a.Key() == b.Key() {
+		t.Errorf("same key %q for policies in different namespaces", a.Key())
+	}
+}