From a26bfc002578f5814aa4ccb29117bf68adf1d215 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 16:43:09 +0800 Subject: [PATCH 01/25] docs: add WatchGCStates server design Signed-off-by: Wenxuan Zhang --- .../2026-09-03-watch-gc-states-design.md | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-03-watch-gc-states-design.md diff --git a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md new file mode 100644 index 00000000000..6bd4f2ce993 --- /dev/null +++ b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md @@ -0,0 +1,232 @@ +# WatchGCStates server design + +`WatchGCStates` provides an ordered stream of complete, effective GC states for keyspaces. This design adds the PD server implementation for the API introduced by [kvproto PR #1528](https://github.com/pingcap/kvproto/pull/1528), while keeping the watch mechanism isolated from the gRPC transport and from the legacy `WatchGCSafePointV2` API. + +## Context + +The original draft in [PD PR #10498](https://github.com/tikv/pd/pull/10498) established the motivation for a streaming API, but its implementation coupled initial loading and live delivery in ways that could reorder states, allowed a slow client to affect unrelated watchers, and included compatibility work that is no longer required. This design retains the useful product semantics from the original [WatchGCStates proposal](https://pingcap.feishu.cn/wiki/TNJSw3rWGiCwjJk8iOtcyTDrnSe) and incorporates the concerns from [review 5097472165](https://github.com/tikv/pd/pull/10498#pullrequestreview-5097472165). + +The merged protobuf API represents every notification as a `GCStateChange` containing either a complete `upsert` state or a `removed` keyspace scope. Consumers apply changes in stream order to maintain a materialized view. + +## Goals + +The implementation has a deliberately narrow set of goals: + +- Stream the initial effective GC state of every active keyspace when requested. +- Stream effective state changes that occur after a watcher is registered. +- Guarantee that a watcher never observes an older state for a keyspace after a newer live state for that keyspace. +- Prevent a slow watcher from blocking GC advancement or affecting another watcher. +- Terminate watchers promptly and predictably when PD loses leadership. +- Keep the watcher implementation independently testable in `pkg/gc`. +- Bound response sizes using the actual protobuf wire size. + +## Non-goals + +The first implementation intentionally excludes adjacent features that are not needed to deliver the API safely: + +- Compatibility with `WatchGCSafePointV2`. +- A Go PD client implementation. +- Streaming GC barriers or global barriers. +- A globally atomic snapshot across keyspaces. +- A revision, cursor, replay log, or resume-from-revision protocol. +- Sharing one initial scan among multiple watchers. +- Producing `removed` events from keyspace lifecycle changes. + +## Stream contract + +The stream is a sequence of self-contained changes. An `upsert` replaces the consumer's entire state for its scope, and a `removed` change deletes that scope from the consumer's materialized view. Upserts do not contain GC barriers; barrier-only mutations do not directly produce changes. + +For `skip_loading_initial=false`, PD registers the live listener before starting the initial scan. Initial and live changes may be interleaved, and the initial scan is not a cross-keyspace transaction. For each individual keyspace, however, the server suppresses an initial value if a post-registration live value for the same scope has already been emitted. The stream therefore cannot regress from a live value `v2` to an older initial value `v1`. + +For `skip_loading_initial=true`, PD sends only effective changes produced after registration. This mode does not provide continuity with an earlier stream and is unsuitable for constructing a complete view on its own. A client establishing its first complete view or recovering from a disconnected stream uses `skip_loading_initial=false`. + +Clients should clear their materialized GC-state view before an initial connection or reconnection with `skip_loading_initial=false`, unless they independently reconcile stale entries. This is a recommended client convention rather than a server-enforced requirement. The first server implementation does not yet produce lifecycle-driven `removed` events, so a client that retains an old view cannot otherwise guarantee removal of scopes deleted while it was disconnected. + +The protocol does not expose an initial-scan completion marker. Consumers continuously apply changes in arrival order; correctness does not depend on distinguishing initial changes from live changes. + +## Architecture + +The design separates state observation, ordered merging, and transport adaptation into three layers. The deeper watcher module owns concurrency and lifecycle policy so the gRPC handler only deals with request validation, protobuf conversion, and response delivery. + +### GC state manager + +`GCStateManager` owns a registry of active watchers. Registration, live publication, watcher removal caused by a full live queue, and the leader-to-follower transition are serialized by the manager's existing mutex. This makes watcher registration linearizable with GC-state mutations and leadership changes. + +The `pkg/gc` layer defines an internal `GCStateChange` representation with `upsert` and `removed` variants. The type is independent of protobuf. An upsert contains a complete effective GC state, including the scope, whether GC is managed at keyspace level, the transaction safe point, and the GC safe point. A removed change contains the affected scope. + +The initial implementation produces upserts from successful GC-state mutations. It supports removed changes throughout the watcher and transport pipeline so keyspace lifecycle integration can be added without redesigning the stream. The implementation leaves an explicit TODO at the keyspace lifecycle integration point rather than adding an incomplete lifecycle dependency in this change. + +### GC state watcher + +Watcher mechanics live in a focused file such as `pkg/gc/gc_state_watcher.go`. Each watcher owns the following state: + +- `initCh`, a buffered channel of initial-state batches. +- `liveCh`, a bounded channel of individual live changes. +- `initDone`, which records whether initial loading has completed or was skipped. +- A cause-aware cancellation mechanism used for both cleanup and error reporting. +- Merge state, including the set of scopes made dirty by live delivery while initial loading is active. + +Only the initial loader writes to and closes `initCh`. Publishers write to `liveCh` only while holding the manager mutex, but `liveCh` is not closed; watcher cancellation is the termination signal. This ownership rule avoids send-versus-close races. + +The watcher exposes a receive operation that returns at most a requested number of visible changes. `RecvBatch(1024)` blocks until at least one change or a terminal cause is available, then opportunistically collects already available changes without waiting to fill the batch. The watcher, rather than the gRPC handler, owns the initial/live merge and its ordering invariant. + +### GC service + +`server/gc_service.go` remains a thin adapter. Its public `WatchGCStates` method performs the rate-limit check directly, validates the request, registers a watcher, converts internal changes to protobuf, splits changes into wire-size-bounded responses, sends them, and closes the watcher on every return path. + +Keeping `rateLimitCheck` in the public handler preserves the externally visible method name `WatchGCStates` in the caller-derived rate-limit label. The rate-limit token is held for the lifetime of the stream and released when the handler returns. + +## Registration and initial loading + +Registration establishes the boundary between pre-existing state and live changes. The sequence is: + +1. Lock `GCStateManager.mu`. +2. Verify that this PD member is the GC-state leader. +3. Create the watcher and add it to the registry. +4. Unlock `GCStateManager.mu`. +5. If `skip_loading_initial=false`, start the initial loader. Otherwise, mark initial loading complete immediately. + +Registering before scanning ensures that every effective mutation after the registration point is either queued as live data or causes that watcher to terminate as a slow consumer. No mutation can fall into a gap between snapshot setup and live subscription. + +The initial loader reuses the manager's existing all-keyspace iteration behavior and requests states without barriers. It preserves the current handling of inactive keyspaces and unified GC mode. It does not hold `GCStateManager.mu` while reading storage, constructing batches, or waiting for `initCh` capacity. + +Initial states are accumulated into batches of at most 1024 changes and sent through `initCh`. The loader closes `initCh` after a successful scan. If iteration fails, it records an initialization error as the watcher cancellation cause; a consumer may therefore have received a partial initial view before the stream terminates. + +Cancellation of the RPC or removal of the watcher cancels the loader as well. Storage iteration and channel sends observe the watcher context so a disconnected client cannot leave an initial-scan goroutine behind. + +## Live publication + +Live changes are published only after a mutation has committed successfully and the manager cache reflects the resulting effective state. Publication occurs before releasing `GCStateManager.mu`, preserving the same order for all watchers and serializing it with follower transition and registration. + +`AdvanceGC` and `AdvanceTxnSafePoint` publish a complete upsert only when the effective state changes. Rejected, no-op, or failed mutations do not publish. Setting or deleting a barrier does not itself publish because barriers are excluded from the API; if a later advancement changes the effective safe point, that successful advancement publishes the new complete state. + +Publication to each `liveCh` is non-blocking. If a watcher's channel is full, the manager removes and cancels only that watcher with the slow-consumer cause and continues publishing to other watchers. There is no timeout, retry, or channel wait while holding the manager mutex. + +## Initial/live ordering + +A single consumer merges `initCh` and `liveCh`. While initial loading is active, it maintains `dirtyDuringInit`, a set keyed by keyspace scope: + +- When the consumer emits a live change, it marks that scope dirty. +- When it encounters an initial upsert whose scope is already dirty, it drops the initial upsert. +- When initial loading finishes, it releases the dirty set because every later change is live and already ordered by `liveCh`. + +There are two possible observations for an initial value `v1` and a later live value `v2`. If the consumer receives `v1` first, it emits `v1` followed by `v2`. If it receives `v2` first, it marks the scope dirty and suppresses `v1`. In neither case can it emit `v2` followed by `v1`. + +The same rule supports the future `removed` producer: a live removal marks the scope dirty, preventing an older initial upsert from recreating it. Live changes retain FIFO order because mutation publication is serialized by the manager mutex and each watcher has a single live channel and a single consumer. + +The implementation must explain this timing guarantee next to the merge logic, including the registration boundary and both possible delivery orders. A deterministic test pauses initial loading after reading `v1` but before placing it on `initCh`, advances the same keyspace to `v2`, observes `v2`, resumes initial loading, and verifies that `v1` is never emitted afterward. + +## Capacity and backpressure + +The default capacities balance burst tolerance against per-watcher memory consumption and are implementation constants so tests can exercise smaller limits deterministically. + +`liveCh` holds 1024 individual changes. A 300,000-keyspace deployment in which every keyspace changes during a ten-minute interval produces roughly 500 changes per second; allowing for both transaction and GC safe-point changes gives an order-of-magnitude estimate of 1,000 changes per second. A capacity of 1024 therefore absorbs approximately one second of scheduling or network jitter at that scale without pretending to support an instantaneous 300,000-keyspace burst. Sustained delivery slower than production intentionally terminates and reconnects the affected watcher. + +Initial batches contain at most 1024 changes, and `initCh` has capacity 1. This permits the RPC consumer to process the current batch while one completed batch waits and the loader constructs the next batch. Increasing the channel capacity would mainly increase per-watcher read-ahead and memory use because initial loading is allowed to backpressure storage iteration. + +The response wire-size limit and the internal change-count limit solve different problems. `RecvBatch(1024)` bounds merge work and internal allocations; the gRPC adapter may split that result into multiple responses to enforce the protobuf size limit. + +## Lifecycle and errors + +Every watcher has exactly one terminal cause. Removal from the manager registry and cancellation are idempotent so concurrent RPC cleanup, initialization failure, leadership loss, and slow-consumer detection cannot leak or double-close resources. + +The lifecycle cases are: + +| Event | Manager behavior | Stream result | +| --- | --- | --- | +| Caller cancellation or send failure | Remove the watcher and cancel its loader | Return the caller or send error; record `client_cancel` | +| Leader becomes follower | Remove and cancel all registered watchers while holding the manager mutex | Return the domain not-leader error, mapped to gRPC `Unavailable` | +| Initial scan fails | Remove and cancel the watcher with the initialization cause | End after any already-sent partial initial data; map to gRPC `Unavailable` | +| `liveCh` is full | Remove and cancel only that watcher | Return the slow-consumer error, mapped to gRPC `ResourceExhausted` | +| Initial scan completes | Close `initCh` and release merge-only initial state | Continue streaming live changes | + +The `pkg/gc` layer returns domain errors and does not depend on gRPC status codes. The service maps not-leader and storage/initialization failures to `Unavailable`, and a slow consumer to `ResourceExhausted`. Existing request-validation and rate-limit paths retain their current status semantics. + +The receive path checks the terminal cause before returning buffered work after cancellation. This prevents an already-cancelled watcher from deliberately draining stale queued data, although an RPC send already in progress when leadership changes cannot be recalled. Clients treat any terminated stream as requiring reconnection and reinitialization. + +## Protobuf conversion and response batching + +The gRPC layer uses a dedicated converter from the internal change type to `pdpb.GCStateChange`. An upsert populates the complete effective state and never populates barrier fields. A removed change populates its `KeyspaceScope`. Unsupported or structurally invalid internal variants fail explicitly rather than producing an empty protobuf change. + +Each `WatchGCStatesResponse` is limited to 1 MiB under normal operation. Batching accounts for the actual protobuf wire representation, including the repeated-field tag and the length-delimited message envelope. + +The adapter starts with the serialized size of the response's non-change fields. For each change, it computes the serialized size of a headerless response containing only that one change; this is the exact additive delta for the repeated embedded-message field. If adding the delta would exceed the limit and the current response is non-empty, the adapter sends the current response first. It never sends an empty response. + +If one change by itself exceeds 1 MiB, the adapter logs the anomaly and sends that single change. Dropping it would silently corrupt the consumer's materialized view, while repeatedly rejecting it would make progress impossible. The protobuf schema makes this case unexpected, but the behavior remains defined. + +This approach avoids manually reproducing protobuf varint rules and avoids repeatedly serializing a growing candidate response, which would make batch construction quadratic. + +## Leadership behavior + +The cluster lifecycle continues to call `GCStateManager.OnNodeBecomesLeader` and `OnNodeBecomesFollower` synchronously. No independent service-level leadership callback is introduced. + +Registration fails if the member is not the current GC-state leader. On transition to follower, the manager cancels every watcher under the same mutex used for registration and publication. A client reconnects to the newly advertised leader with `skip_loading_initial=false` and rebuilds its view according to the stream contract. + +## Removed-event integration + +The internal model, ordering logic, protobuf converter, and tests all accept `removed` changes, but the first PD implementation has no authoritative keyspace lifecycle hook that produces them. Adding a partial producer would create misleading convergence guarantees, so production is deferred. + +A future lifecycle integration must publish removal and recreation events through the same manager-serialized live path as safe-point changes. It must also define how lifecycle ownership interacts with GC leadership. The implementation records this location with a targeted TODO so the limitation is discoverable without expanding the present scope. + +## Observability + +Metrics are intentionally low-cardinality and focus on operational decisions rather than per-message detail: + +- A gauge reports the number of active GC-state watchers. +- A counter reports watcher terminations with the bounded reason label values `client_cancel`, `leader_lost`, `slow_consumer`, and `init_error`. +- A slow-consumer log records the watcher identifier, configured live capacity, and observed queue length. + +Watcher identifiers, keyspace identifiers, client addresses, and error strings are not metric labels. The implementation does not add a per-send queue-length histogram because it would instrument the hot path without a demonstrated operational need. + +## Test strategy + +The watcher and service layers are tested separately, with focused integration coverage for leadership transitions. Tests use controllable capacities or failpoints instead of timing-dependent sleeps. + +### `pkg/gc` tests + +The domain tests cover: + +- Registration succeeds only on the leader, and leadership loss terminates all current watchers. +- `skip_loading_initial=false` returns initial and subsequent live states; `true` returns only post-registration live changes. +- Successful effective state changes publish complete upserts, while no-op and failed mutations do not. +- Barrier-only mutations produce no change, while a later effective safe-point change does. +- An initial-first sequence emits `v1` followed by `v2`. +- A deterministic paused-initial sequence emits `v2` and suppresses the later initial `v1`. +- Filling watcher A's `liveCh` terminates A without delaying watcher B; A can reconnect and rebuild. +- Initial iteration failure, caller cancellation, and concurrent deregistration terminate without goroutine or registry leaks. +- A full `initCh` backpressures only that watcher's initial iterator. +- Upsert and removed changes use the same per-scope dirty ordering rule. + +### Server tests + +The transport tests cover: + +- Conversion of complete upsert and removed variants, including an empty barrier list. +- Response splitting at a reduced test limit, with assertions based on the serialized protobuf size on both sides of the boundary. +- A rate-limit capacity of one: the first active stream holds the token, the second is rejected, and a third succeeds after the first closes. +- Actual leader transfer terminates the old stream and permits a fresh initial stream on the new leader. +- Client cancellation and send failure remove the watcher and release the rate-limit token. +- Domain error causes map to the specified gRPC status codes. + +## Dependency and rollout + +The root, client, tools, and tests Go modules are updated to a kvproto revision containing the merged `WatchGCStates` API from PR #1528. No compatibility wrapper or implementation is added for `WatchGCSafePointV2`. + +The API can be rolled out server-first because existing clients do not call the new RPC. New consumers use an initial stream to construct their materialized view and use the same path after any disconnect. The absence of lifecycle-produced removals remains an explicit limitation until the future integration is implemented. + +## Acceptance criteria + +The implementation is complete when all of the following are true: + +- `WatchGCStates` serves initial and live effective state changes with the documented `skip_loading_initial` behavior. +- The deterministic ordering test proves that no older initial value follows a newer live value for the same scope. +- A full live queue terminates only the affected watcher without blocking GC-state mutation. +- Leadership loss terminates all active streams and reconnecting with initial loading rebuilds the view. +- Response batches observe the 1 MiB target using exact protobuf size accounting, except for the defined oversized-single-change case. +- Metrics and logs use only the bounded dimensions described above. +- Targeted package and server tests pass with no failpoints left enabled. +- The implementation contains no `WatchGCSafePointV2` compatibility path and no Go client work. + +## Next step + +After this design is reviewed, the implementation work is decomposed into a test-first plan covering the dependency update, watcher domain model, mutation publication, gRPC adaptation, observability, and focused verification. From b968d133f4f7ea9ba4ed5cd3a0d2dbe37cafc790 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 17:06:43 +0800 Subject: [PATCH 02/25] docs: refine WatchGCStates server design Signed-off-by: Wenxuan Zhang --- .../2026-09-03-watch-gc-states-design.md | 99 ++++++++++++------- 1 file changed, 65 insertions(+), 34 deletions(-) diff --git a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md index 6bd4f2ce993..94e8cb49da0 100644 --- a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md +++ b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md @@ -12,9 +12,9 @@ The merged protobuf API represents every notification as a `GCStateChange` conta The implementation has a deliberately narrow set of goals: -- Stream the initial effective GC state of every active keyspace when requested. -- Stream effective state changes that occur after a watcher is registered. -- Guarantee that a watcher never observes an older state for a keyspace after a newer live state for that keyspace. +- Stream the initial effective GC state of every active keyspace observed by the initial scan when requested. +- Stream effective safe-point changes committed through the local `GCStateManager` after a watcher is registered. +- Guarantee that a watcher never observes an older state for a keyspace after a newer live state for that keyspace on the same stream. - Prevent a slow watcher from blocking GC advancement or affecting another watcher. - Terminate watchers promptly and predictably when PD loses leadership. - Keep the watcher implementation independently testable in `pkg/gc`. @@ -30,7 +30,8 @@ The first implementation intentionally excludes adjacent features that are not n - A globally atomic snapshot across keyspaces. - A revision, cursor, replay log, or resume-from-revision protocol. - Sharing one initial scan among multiple watchers. -- Producing `removed` events from keyspace lifecycle changes. +- Producing events from keyspace creation, enablement, disablement, deletion, or GC-mode changes. +- Fencing GC-state write transactions with the PD leadership lease or a cluster-wide leadership epoch. ## Stream contract @@ -44,17 +45,21 @@ Clients should clear their materialized GC-state view before an initial connecti The protocol does not expose an initial-scan completion marker. Consumers continuously apply changes in arrival order; correctness does not depend on distinguishing initial changes from live changes. +The first implementation does not subscribe to keyspace metadata changes after registration. A connected client therefore is not guaranteed to learn that a keyspace was created, enabled, disabled, deleted, or switched between unified and keyspace-level GC until it reconnects and reloads, receives a later safe-point upsert that reflects the new metadata, or reconciles that metadata independently. + +Watch registration and publication are linearized only within one PD process and one local leadership generation. Existing GC-state write transactions are not fenced by the PD leadership lease. In the rare case that a transaction accepted by an old leader commits after the new leader has completed the corresponding initial read, the new stream might not observe that value until another effective mutation or another reconnection. This limitation does not permit an older value to follow a newer value on the same stream. Leadership-fenced GC-state transactions are deferred as a separate improvement because they require changes across every modern and legacy write path and the storage transaction layer. + ## Architecture The design separates state observation, ordered merging, and transport adaptation into three layers. The deeper watcher module owns concurrency and lifecycle policy so the gRPC handler only deals with request validation, protobuf conversion, and response delivery. ### GC state manager -`GCStateManager` owns a registry of active watchers. Registration, live publication, watcher removal caused by a full live queue, and the leader-to-follower transition are serialized by the manager's existing mutex. This makes watcher registration linearizable with GC-state mutations and leadership changes. +`GCStateManager` owns a registry of active watchers. Registration, live publication, watcher removal caused by a full live queue, and local leader-generation transitions are serialized by the manager's existing mutex. This makes watcher registration linearizable with GC-state mutations in the same manager and leadership generation; it does not provide cross-process transaction fencing. The `pkg/gc` layer defines an internal `GCStateChange` representation with `upsert` and `removed` variants. The type is independent of protobuf. An upsert contains a complete effective GC state, including the scope, whether GC is managed at keyspace level, the transaction safe point, and the GC safe point. A removed change contains the affected scope. -The initial implementation produces upserts from successful GC-state mutations. It supports removed changes throughout the watcher and transport pipeline so keyspace lifecycle integration can be added without redesigning the stream. The implementation leaves an explicit TODO at the keyspace lifecycle integration point rather than adding an incomplete lifecycle dependency in this change. +The initial implementation produces upserts from successful safe-point mutations. It supports removed changes throughout the watcher and transport pipeline so keyspace lifecycle integration can be added without redesigning the stream. The implementation leaves an explicit TODO at the keyspace lifecycle integration point for creation, state, deletion, and GC-mode changes rather than adding an incomplete lifecycle dependency in this change. ### GC state watcher @@ -62,7 +67,8 @@ Watcher mechanics live in a focused file such as `pkg/gc/gc_state_watcher.go`. E - `initCh`, a buffered channel of initial-state batches. - `liveCh`, a bounded channel of individual live changes. -- `initDone`, which records whether initial loading has completed or was skipped. +- `initDone`, which is owned by the merge consumer and becomes true when initial loading was skipped or after the closed `initCh` has been fully drained. +- The local leadership generation in which the watcher was registered. - A cause-aware cancellation mechanism used for both cleanup and error reporting. - Merge state, including the set of scopes made dirty by live delivery while initial loading is active. @@ -76,29 +82,45 @@ The watcher exposes a receive operation that returns at most a requested number Keeping `rateLimitCheck` in the public handler preserves the externally visible method name `WatchGCStates` in the caller-derived rate-limit label. The rate-limit token is held for the lifetime of the stream and released when the handler returns. +## RPC preflight + +The server-streaming handler performs a local preflight because it cannot reuse the unary forwarding callback to proxy a long-lived stream. The check order is: + +1. Acquire the `WatchGCStates` rate-limit token. +2. Validate the request header, cluster ID, and local serving role with the existing validation helpers. +3. Reject a request handled by a non-serving member with the existing not-leader `Unavailable` status instead of forwarding the stream. +4. Reject an unbootstrapped server with `Unavailable` before registering a watcher. +5. Register the watcher with the current local GC leadership generation. + +Every successful `WatchGCStatesResponse` contains `grpcutil.WrapHeader()`. Request validation and rate-limit failures retain their existing status codes. A structurally invalid internal change is a server bug: the handler logs it, closes the watcher, and returns gRPC `Internal` without assigning it a fabricated domain termination reason. + ## Registration and initial loading Registration establishes the boundary between pre-existing state and live changes. The sequence is: 1. Lock `GCStateManager.mu`. -2. Verify that this PD member is the GC-state leader. -3. Create the watcher and add it to the registry. +2. Verify that this PD member has an active local GC leadership generation. +3. Create the watcher, tag it with that generation, and add it to the registry. 4. Unlock `GCStateManager.mu`. 5. If `skip_loading_initial=false`, start the initial loader. Otherwise, mark initial loading complete immediately. -Registering before scanning ensures that every effective mutation after the registration point is either queued as live data or causes that watcher to terminate as a slow consumer. No mutation can fall into a gap between snapshot setup and live subscription. +Registering before scanning ensures that every effective mutation published by the same manager and leadership generation after the registration point is either queued as live data or causes that watcher to terminate as a slow consumer. No such mutation can fall into a gap between snapshot setup and live subscription. -The initial loader reuses the manager's existing all-keyspace iteration behavior and requests states without barriers. It preserves the current handling of inactive keyspaces and unified GC mode. It does not hold `GCStateManager.mu` while reading storage, constructing batches, or waiting for `initCh` capacity. +The initial loader calls the incremental `iterateAllKeyspacesGCStates` path rather than `GetAllKeyspacesGCStates`, which materializes the full result before returning. It requests states without barriers and preserves the current handling of inactive keyspaces and unified GC mode. + +The loader never holds `GCStateManager.mu` while constructing a batch or waiting for `initCh` capacity, so client backpressure cannot block mutation publication or follower cleanup. On a cache miss, the existing single-keyspace slow path may acquire `GCStateManager.mu.RLock()` while it reads storage; that lock is released before the iterator callback can block on `initCh`. This preserves the cache's current synchronization rule without holding one manager lock across the entire scan. Initial states are accumulated into batches of at most 1024 changes and sent through `initCh`. The loader closes `initCh` after a successful scan. If iteration fails, it records an initialization error as the watcher cancellation cause; a consumer may therefore have received a partial initial view before the stream terminates. -Cancellation of the RPC or removal of the watcher cancels the loader as well. Storage iteration and channel sends observe the watcher context so a disconnected client cannot leave an initial-scan goroutine behind. +Cancellation of the RPC or removal of the watcher cancels the loader as well. The loader checks the watcher context between iterator items and while waiting to send a batch. The current keyspace iterator and storage read interfaces do not accept a context, so the loader cannot observe cancellation until an invocation already inside `Iterator.Next` or a storage operation returns. This inherited non-cancellable I/O boundary is accepted; the watcher introduces no additional wait around it. ## Live publication -Live changes are published only after a mutation has committed successfully and the manager cache reflects the resulting effective state. Publication occurs before releasing `GCStateManager.mu`, preserving the same order for all watchers and serializing it with follower transition and registration. +Live changes are published only after a mutation has committed successfully and the manager cache reflects the resulting effective state. Publication occurs before releasing `GCStateManager.mu`, preserving the same order for all watchers in that manager and leadership generation and serializing it with local follower transition and registration. + +Publication is attached exactly once to the successful post-cache-update paths in `advanceGCSafePointImpl` and `advanceTxnSafePointImpl`. This covers `AdvanceGCSafePoint`, `AdvanceTxnSafePoint`, `CompatibleUpdateGCSafePoint`, and the `gc_worker` branch of `CompatibleUpdateServiceGCSafePoint` without duplicating events at public API entry points. Rejected, no-op, or failed mutations do not publish. -`AdvanceGC` and `AdvanceTxnSafePoint` publish a complete upsert only when the effective state changes. Rejected, no-op, or failed mutations do not publish. Setting or deleting a barrier does not itself publish because barriers are excluded from the API; if a later advancement changes the effective safe point, that successful advancement publishes the new complete state. +The current barrier setters and deleters do not alter the persisted effective safe points, so those operations produce no event. Barrier details remain excluded from the stream. If a present or future barrier operation changes an effective safe point, the operation publishes the resulting complete upsert rather than the barrier itself. Publication to each `liveCh` is non-blocking. If a watcher's channel is full, the manager removes and cancels only that watcher with the slow-consumer cause and continues publishing to other watchers. There is no timeout, retry, or channel wait while holding the manager mutex. @@ -108,7 +130,7 @@ A single consumer merges `initCh` and `liveCh`. While initial loading is active, - When the consumer emits a live change, it marks that scope dirty. - When it encounters an initial upsert whose scope is already dirty, it drops the initial upsert. -- When initial loading finishes, it releases the dirty set because every later change is live and already ordered by `liveCh`. +- After `initCh` is closed and all of its buffered batches are consumed, the consumer releases the dirty set and disables the closed channel because every later change is live and already ordered by `liveCh`. There are two possible observations for an initial value `v1` and a later live value `v2`. If the consumer receives `v1` first, it emits `v1` followed by `v2`. If it receives `v2` first, it marks the scope dirty and suppresses `v1`. In neither case can it emit `v2` followed by `v1`. @@ -118,7 +140,7 @@ The implementation must explain this timing guarantee next to the merge logic, i ## Capacity and backpressure -The default capacities balance burst tolerance against per-watcher memory consumption and are implementation constants so tests can exercise smaller limits deterministically. +The default capacities balance burst tolerance against per-watcher memory consumption. Production wrappers pass the defaults to unexported constructors or helpers that accept explicit capacities, which lets tests exercise smaller limits without mutable package globals. `liveCh` holds 1024 individual changes. A 300,000-keyspace deployment in which every keyspace changes during a ten-minute interval produces roughly 500 changes per second; allowing for both transaction and GC safe-point changes gives an order-of-magnitude estimate of 1,000 changes per second. A capacity of 1024 therefore absorbs approximately one second of scheduling or network jitter at that scale without pretending to support an instantaneous 300,000-keyspace burst. Sustained delivery slower than production intentionally terminates and reconnects the affected watcher. @@ -128,21 +150,21 @@ The response wire-size limit and the internal change-count limit solve different ## Lifecycle and errors -Every watcher has exactly one terminal cause. Removal from the manager registry and cancellation are idempotent so concurrent RPC cleanup, initialization failure, leadership loss, and slow-consumer detection cannot leak or double-close resources. +Every watcher has exactly one terminal cause, and the first successful cancellation wins. Removal from the manager registry and cancellation are idempotent so concurrent RPC cleanup, initialization failure, leadership loss, and slow-consumer detection cannot leak or double-close resources. Cancellation invoked while holding the manager mutex does not synchronously reacquire that mutex. The lifecycle cases are: | Event | Manager behavior | Stream result | | --- | --- | --- | | Caller cancellation or send failure | Remove the watcher and cancel its loader | Return the caller or send error; record `client_cancel` | -| Leader becomes follower | Remove and cancel all registered watchers while holding the manager mutex | Return the domain not-leader error, mapped to gRPC `Unavailable` | +| A local leadership generation ends or is superseded | Remove and cancel the watchers registered in that generation while holding the manager mutex | Return the domain not-leader error, mapped to gRPC `Unavailable` | | Initial scan fails | Remove and cancel the watcher with the initialization cause | End after any already-sent partial initial data; map to gRPC `Unavailable` | | `liveCh` is full | Remove and cancel only that watcher | Return the slow-consumer error, mapped to gRPC `ResourceExhausted` | | Initial scan completes | Close `initCh` and release merge-only initial state | Continue streaming live changes | The `pkg/gc` layer returns domain errors and does not depend on gRPC status codes. The service maps not-leader and storage/initialization failures to `Unavailable`, and a slow consumer to `ResourceExhausted`. Existing request-validation and rate-limit paths retain their current status semantics. -The receive path checks the terminal cause before returning buffered work after cancellation. This prevents an already-cancelled watcher from deliberately draining stale queued data, although an RPC send already in progress when leadership changes cannot be recalled. Clients treat any terminated stream as requiring reconnection and reinitialization. +The receive path checks the terminal cause before returning buffered work after cancellation. Because one `RecvBatch` can be split into multiple protobuf responses, the handler checks the same terminal cause again immediately before every `Send`. This prevents an already-cancelled watcher from deliberately draining stale queued data; only an RPC send already in progress when leadership changes cannot be recalled. Clients treat any terminated stream as requiring reconnection and reinitialization. ## Protobuf conversion and response batching @@ -150,7 +172,7 @@ The gRPC layer uses a dedicated converter from the internal change type to `pdpb Each `WatchGCStatesResponse` is limited to 1 MiB under normal operation. Batching accounts for the actual protobuf wire representation, including the repeated-field tag and the length-delimited message envelope. -The adapter starts with the serialized size of the response's non-change fields. For each change, it computes the serialized size of a headerless response containing only that one change; this is the exact additive delta for the repeated embedded-message field. If adding the delta would exceed the limit and the current response is non-empty, the adapter sends the current response first. It never sends an empty response. +The adapter starts with the serialized size of a response containing `grpcutil.WrapHeader()` and no changes. For each change, it computes the serialized size of a headerless response containing only that one change; this is the exact additive delta for the repeated embedded-message field. If adding the delta would exceed the limit and the current response is non-empty, the adapter sends the current response first. It never sends an empty response. If one change by itself exceeds 1 MiB, the adapter logs the anomaly and sends that single change. Dropping it would silently corrupt the consumer's materialized view, while repeatedly rejecting it would make progress impossible. The protobuf schema makes this case unexpected, but the behavior remains defined. @@ -158,13 +180,13 @@ This approach avoids manually reproducing protobuf varint rules and avoids repea ## Leadership behavior -The cluster lifecycle continues to call `GCStateManager.OnNodeBecomesLeader` and `OnNodeBecomesFollower` synchronously. No independent service-level leadership callback is introduced. +The cluster lifecycle continues to notify `GCStateManager` synchronously. Under the manager mutex, `OnNodeBecomesLeader` creates a new local generation, cancels watchers left from older generations, clears the cache, and returns a teardown closure that captures the new generation. The cluster stores and invokes that closure when the corresponding leadership ends. No independent service-level leadership callback is introduced. -Registration fails if the member is not the current GC-state leader. On transition to follower, the manager cancels every watcher under the same mutex used for registration and publication. A client reconnects to the newly advertised leader with `skip_loading_initial=false` and rebuilds its view according to the stream contract. +Registration fails if the member has no active local GC leadership generation. A generation's teardown removes and cancels only watchers registered in that generation under the same mutex used for registration and publication. It clears the active-generation marker and cache only if its captured generation is still current. A delayed teardown from an older generation therefore cannot cancel watchers or clear state belonging to a newer generation. A client reconnects to the newly advertised leader with `skip_loading_initial=false` and rebuilds its view according to the stream contract, subject to the documented cross-leader late-commit limitation. ## Removed-event integration -The internal model, ordering logic, protobuf converter, and tests all accept `removed` changes, but the first PD implementation has no authoritative keyspace lifecycle hook that produces them. Adding a partial producer would create misleading convergence guarantees, so production is deferred. +The internal model, ordering logic, protobuf converter, and tests all accept `removed` changes, but the first PD implementation has no authoritative keyspace lifecycle hook that produces them. It also does not produce metadata-driven upserts for keyspace creation, enablement, or GC-mode changes. Adding only part of these producers would create misleading convergence guarantees, so production is deferred. A future lifecycle integration must publish removal and recreation events through the same manager-serialized live path as safe-point changes. It must also define how lifecycle ownership interacts with GC leadership. The implementation records this location with a targeted TODO so the limitation is discoverable without expanding the present scope. @@ -176,7 +198,7 @@ Metrics are intentionally low-cardinality and focus on operational decisions rat - A counter reports watcher terminations with the bounded reason label values `client_cancel`, `leader_lost`, `slow_consumer`, and `init_error`. - A slow-consumer log records the watcher identifier, configured live capacity, and observed queue length. -Watcher identifiers, keyspace identifiers, client addresses, and error strings are not metric labels. The implementation does not add a per-send queue-length histogram because it would instrument the hot path without a demonstrated operational need. +Watcher identifiers, keyspace identifiers, client addresses, and error strings are not metric labels. The four termination values describe watcher-domain lifecycle causes; a protobuf conversion bug is logged and returned at the transport layer instead of adding an `internal_error` label. The implementation does not add a per-send queue-length histogram because it would instrument the hot path without a demonstrated operational need. ## Test strategy @@ -186,16 +208,19 @@ The watcher and service layers are tested separately, with focused integration c The domain tests cover: -- Registration succeeds only on the leader, and leadership loss terminates all current watchers. +- Registration succeeds only in an active local leadership generation, and ending or superseding that generation terminates its watchers. - `skip_loading_initial=false` returns initial and subsequent live states; `true` returns only post-registration live changes. -- Successful effective state changes publish complete upserts, while no-op and failed mutations do not. -- Barrier-only mutations produce no change, while a later effective safe-point change does. +- Successful effective state changes publish complete upserts from the shared internal mutation paths, while no-op and failed mutations do not. +- Modern and legacy API entry points that reach the same internal mutation produce exactly one equivalent change. +- Current barrier-only mutations produce no change, while any operation that changes an effective safe point does. - An initial-first sequence emits `v1` followed by `v2`. - A deterministic paused-initial sequence emits `v2` and suppresses the later initial `v1`. - Filling watcher A's `liveCh` terminates A without delaying watcher B; A can reconnect and rebuild. - Initial iteration failure, caller cancellation, and concurrent deregistration terminate without goroutine or registry leaks. -- A full `initCh` backpressures only that watcher's initial iterator. +- A full `initCh` backpressures only that watcher's initial iterator and never waits on the channel while holding `GCStateManager.mu`. - Upsert and removed changes use the same per-scope dirty ordering rule. +- Initial merge state remains active until a closed `initCh` is fully drained, and disabling the closed channel prevents select spinning. +- A delayed teardown callback for an old local leadership generation does not cancel watchers in a newer generation. ### Server tests @@ -203,14 +228,19 @@ The transport tests cover: - Conversion of complete upsert and removed variants, including an empty barrier list. - Response splitting at a reduced test limit, with assertions based on the serialized protobuf size on both sides of the boundary. +- A successful header appears in every split response, and the header contributes to the size boundary. +- Cancellation between two responses derived from one `RecvBatch` prevents the remaining response from being sent. +- An oversized single change is sent alone, dirty-only input does not produce an empty response, and an invalid internal change returns `Internal`. - A rate-limit capacity of one: the first active stream holds the token, the second is rejected, and a third succeeds after the first closes. +- Request preflight covers cluster-ID mismatch, a non-serving member, and an unbootstrapped server. - Actual leader transfer terminates the old stream and permits a fresh initial stream on the new leader. - Client cancellation and send failure remove the watcher and release the rate-limit token. - Domain error causes map to the specified gRPC status codes. +- The active-watcher gauge and each domain termination counter change exactly once across registration and cleanup. ## Dependency and rollout -The root, client, tools, and tests Go modules are updated to a kvproto revision containing the merged `WatchGCStates` API from PR #1528. No compatibility wrapper or implementation is added for `WatchGCSafePointV2`. +The root, `client`, `tools`, and `tests/integrations` Go modules are updated to a kvproto revision containing the merged `WatchGCStates` API from PR #1528. No compatibility wrapper or implementation is added for `WatchGCSafePointV2`. The API can be rolled out server-first because existing clients do not call the new RPC. New consumers use an initial stream to construct their materialized view and use the same path after any disconnect. The absence of lifecycle-produced removals remains an explicit limitation until the future integration is implemented. @@ -218,15 +248,16 @@ The API can be rolled out server-first because existing clients do not call the The implementation is complete when all of the following are true: -- `WatchGCStates` serves initial and live effective state changes with the documented `skip_loading_initial` behavior. -- The deterministic ordering test proves that no older initial value follows a newer live value for the same scope. +- `WatchGCStates` serves initial and local safe-point mutation changes with the documented `skip_loading_initial` behavior and metadata-change limitations. +- The deterministic ordering test proves that no older initial value follows a newer live value for the same scope on one stream. - A full live queue terminates only the affected watcher without blocking GC-state mutation. -- Leadership loss terminates all active streams and reconnecting with initial loading rebuilds the view. +- Ending or superseding a local leadership generation terminates its active streams, and reconnecting with initial loading rebuilds the view subject to the documented cross-PD late-commit limitation. - Response batches observe the 1 MiB target using exact protobuf size accounting, except for the defined oversized-single-change case. -- Metrics and logs use only the bounded dimensions described above. +- Metric labels use only the bounded dimensions described above. - Targeted package and server tests pass with no failpoints left enabled. - The implementation contains no `WatchGCSafePointV2` compatibility path and no Go client work. +- Cross-PD leadership fencing remains explicitly out of scope and is recorded as a follow-up TODO. -## Next step +## Next steps After this design is reviewed, the implementation work is decomposed into a test-first plan covering the dependency update, watcher domain model, mutation publication, gRPC adaptation, observability, and focused verification. From d42e8785e2f085589cf44ef28ed16c31c6c64761 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 17:18:40 +0800 Subject: [PATCH 03/25] docs: streamline WatchGCStates server design Signed-off-by: Wenxuan Zhang --- .../2026-09-03-watch-gc-states-design.md | 71 +++++++------------ 1 file changed, 24 insertions(+), 47 deletions(-) diff --git a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md index 94e8cb49da0..21e5dc210d3 100644 --- a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md +++ b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md @@ -39,15 +39,13 @@ The stream is a sequence of self-contained changes. An `upsert` replaces the con For `skip_loading_initial=false`, PD registers the live listener before starting the initial scan. Initial and live changes may be interleaved, and the initial scan is not a cross-keyspace transaction. For each individual keyspace, however, the server suppresses an initial value if a post-registration live value for the same scope has already been emitted. The stream therefore cannot regress from a live value `v2` to an older initial value `v1`. -For `skip_loading_initial=true`, PD sends only effective changes produced after registration. This mode does not provide continuity with an earlier stream and is unsuitable for constructing a complete view on its own. A client establishing its first complete view or recovering from a disconnected stream uses `skip_loading_initial=false`. +For `skip_loading_initial=true`, PD sends only effective safe-point changes produced after registration. This mode does not provide continuity with an earlier stream and is unsuitable for constructing a complete view on its own. A client establishing its first complete view or recovering from a disconnected stream uses `skip_loading_initial=false`. Clients should clear their materialized GC-state view before an initial connection or reconnection with `skip_loading_initial=false`, unless they independently reconcile stale entries. This is a recommended client convention rather than a server-enforced requirement. The first server implementation does not yet produce lifecycle-driven `removed` events, so a client that retains an old view cannot otherwise guarantee removal of scopes deleted while it was disconnected. The protocol does not expose an initial-scan completion marker. Consumers continuously apply changes in arrival order; correctness does not depend on distinguishing initial changes from live changes. -The first implementation does not subscribe to keyspace metadata changes after registration. A connected client therefore is not guaranteed to learn that a keyspace was created, enabled, disabled, deleted, or switched between unified and keyspace-level GC until it reconnects and reloads, receives a later safe-point upsert that reflects the new metadata, or reconciles that metadata independently. - -Watch registration and publication are linearized only within one PD process and one local leadership generation. Existing GC-state write transactions are not fenced by the PD leadership lease. In the rare case that a transaction accepted by an old leader commits after the new leader has completed the corresponding initial read, the new stream might not observe that value until another effective mutation or another reconnection. This limitation does not permit an older value to follow a newer value on the same stream. Leadership-fenced GC-state transactions are deferred as a separate improvement because they require changes across every modern and legacy write path and the storage transaction layer. +The first implementation does not subscribe to keyspace metadata changes after registration. A connected client might not learn that a keyspace was created, enabled, disabled, deleted, or switched between unified and keyspace-level GC until it reconnects, receives a later safe-point upsert that reflects the new metadata, or reconciles that metadata independently. ## Architecture @@ -55,7 +53,7 @@ The design separates state observation, ordered merging, and transport adaptatio ### GC state manager -`GCStateManager` owns a registry of active watchers. Registration, live publication, watcher removal caused by a full live queue, and local leader-generation transitions are serialized by the manager's existing mutex. This makes watcher registration linearizable with GC-state mutations in the same manager and leadership generation; it does not provide cross-process transaction fencing. +`GCStateManager` owns a registry of active watchers. Its existing mutex serializes watcher registration, publication of effective safe-point changes, removal of slow watchers, and local leadership transitions. These guarantees apply within one manager and local leadership generation. The `pkg/gc` layer defines an internal `GCStateChange` representation with `upsert` and `removed` variants. The type is independent of protobuf. An upsert contains a complete effective GC state, including the scope, whether GC is managed at keyspace level, the transaction safe point, and the GC safe point. A removed change contains the affected scope. @@ -68,7 +66,6 @@ Watcher mechanics live in a focused file such as `pkg/gc/gc_state_watcher.go`. E - `initCh`, a buffered channel of initial-state batches. - `liveCh`, a bounded channel of individual live changes. - `initDone`, which is owned by the merge consumer and becomes true when initial loading was skipped or after the closed `initCh` has been fully drained. -- The local leadership generation in which the watcher was registered. - A cause-aware cancellation mechanism used for both cleanup and error reporting. - Merge state, including the set of scopes made dirty by live delivery while initial loading is active. @@ -78,21 +75,9 @@ The watcher exposes a receive operation that returns at most a requested number ### GC service -`server/gc_service.go` remains a thin adapter. Its public `WatchGCStates` method performs the rate-limit check directly, validates the request, registers a watcher, converts internal changes to protobuf, splits changes into wire-size-bounded responses, sends them, and closes the watcher on every return path. - -Keeping `rateLimitCheck` in the public handler preserves the externally visible method name `WatchGCStates` in the caller-derived rate-limit label. The rate-limit token is held for the lifetime of the stream and released when the handler returns. - -## RPC preflight - -The server-streaming handler performs a local preflight because it cannot reuse the unary forwarding callback to proxy a long-lived stream. The check order is: - -1. Acquire the `WatchGCStates` rate-limit token. -2. Validate the request header, cluster ID, and local serving role with the existing validation helpers. -3. Reject a request handled by a non-serving member with the existing not-leader `Unavailable` status instead of forwarding the stream. -4. Reject an unbootstrapped server with `Unavailable` before registering a watcher. -5. Register the watcher with the current local GC leadership generation. +`server/gc_service.go` remains a thin adapter. Its public `WatchGCStates` method performs the rate-limit check directly, validates the request locally, registers a watcher, converts internal changes to protobuf, splits changes into wire-size-bounded responses, sends them, and closes the watcher on every return path. The handler does not proxy the long-lived stream: a non-serving or unbootstrapped member returns `Unavailable`, and existing header and cluster-ID validation semantics remain unchanged. -Every successful `WatchGCStatesResponse` contains `grpcutil.WrapHeader()`. Request validation and rate-limit failures retain their existing status codes. A structurally invalid internal change is a server bug: the handler logs it, closes the watcher, and returns gRPC `Internal` without assigning it a fabricated domain termination reason. +Keeping `rateLimitCheck` in the public handler preserves the externally visible method name `WatchGCStates` in the caller-derived rate-limit label. The rate-limit token is held for the lifetime of the stream and released when the handler returns. Every successful `WatchGCStatesResponse` contains `grpcutil.WrapHeader()`; a structurally invalid internal change is logged and returned as gRPC `Internal`. ## Registration and initial loading @@ -100,11 +85,11 @@ Registration establishes the boundary between pre-existing state and live change 1. Lock `GCStateManager.mu`. 2. Verify that this PD member has an active local GC leadership generation. -3. Create the watcher, tag it with that generation, and add it to the registry. +3. Create the watcher and add it to the registry. 4. Unlock `GCStateManager.mu`. 5. If `skip_loading_initial=false`, start the initial loader. Otherwise, mark initial loading complete immediately. -Registering before scanning ensures that every effective mutation published by the same manager and leadership generation after the registration point is either queued as live data or causes that watcher to terminate as a slow consumer. No such mutation can fall into a gap between snapshot setup and live subscription. +Registering before scanning ensures that every effective safe-point change published by the same manager and leadership generation after the registration point is either queued as live data or causes that watcher to terminate as a slow consumer. No such change can fall into a gap between snapshot setup and live subscription. The initial loader calls the incremental `iterateAllKeyspacesGCStates` path rather than `GetAllKeyspacesGCStates`, which materializes the full result before returning. It requests states without barriers and preserves the current handling of inactive keyspaces and unified GC mode. @@ -116,7 +101,7 @@ Cancellation of the RPC or removal of the watcher cancels the loader as well. Th ## Live publication -Live changes are published only after a mutation has committed successfully and the manager cache reflects the resulting effective state. Publication occurs before releasing `GCStateManager.mu`, preserving the same order for all watchers in that manager and leadership generation and serializing it with local follower transition and registration. +Live changes are published only after a safe-point mutation has committed successfully and the manager cache reflects the resulting complete effective GC state. Publication occurs before releasing `GCStateManager.mu`, preserving the same order for all watchers in that manager and leadership generation and serializing it with local follower transition and registration. Publication is attached exactly once to the successful post-cache-update paths in `advanceGCSafePointImpl` and `advanceTxnSafePointImpl`. This covers `AdvanceGCSafePoint`, `AdvanceTxnSafePoint`, `CompatibleUpdateGCSafePoint`, and the `gc_worker` branch of `CompatibleUpdateServiceGCSafePoint` without duplicating events at public API entry points. Rejected, no-op, or failed mutations do not publish. @@ -157,7 +142,7 @@ The lifecycle cases are: | Event | Manager behavior | Stream result | | --- | --- | --- | | Caller cancellation or send failure | Remove the watcher and cancel its loader | Return the caller or send error; record `client_cancel` | -| A local leadership generation ends or is superseded | Remove and cancel the watchers registered in that generation while holding the manager mutex | Return the domain not-leader error, mapped to gRPC `Unavailable` | +| A local leadership generation ends or is superseded | Remove and cancel all current watchers while holding the manager mutex | Return the domain not-leader error, mapped to gRPC `Unavailable` | | Initial scan fails | Remove and cancel the watcher with the initialization cause | End after any already-sent partial initial data; map to gRPC `Unavailable` | | `liveCh` is full | Remove and cancel only that watcher | Return the slow-consumer error, mapped to gRPC `ResourceExhausted` | | Initial scan completes | Close `initCh` and release merge-only initial state | Continue streaming live changes | @@ -180,15 +165,15 @@ This approach avoids manually reproducing protobuf varint rules and avoids repea ## Leadership behavior -The cluster lifecycle continues to notify `GCStateManager` synchronously. Under the manager mutex, `OnNodeBecomesLeader` creates a new local generation, cancels watchers left from older generations, clears the cache, and returns a teardown closure that captures the new generation. The cluster stores and invokes that closure when the corresponding leadership ends. No independent service-level leadership callback is introduced. +The cluster lifecycle continues to notify `GCStateManager` synchronously. Under the manager mutex, `OnNodeBecomesLeader` advances the manager's local generation, cancels existing watchers, clears the cache, and returns a teardown closure that captures the new generation. The teardown acts only if its generation is still current; it then marks the manager as a follower, cancels all current watchers, and clears the cache. A delayed teardown from an older generation is therefore a no-op. Registration fails when the manager has no active local leadership generation. -Registration fails if the member has no active local GC leadership generation. A generation's teardown removes and cancels only watchers registered in that generation under the same mutex used for registration and publication. It clears the active-generation marker and cache only if its captured generation is still current. A delayed teardown from an older generation therefore cannot cancel watchers or clear state belonging to a newer generation. A client reconnects to the newly advertised leader with `skip_loading_initial=false` and rebuilds its view according to the stream contract, subject to the documented cross-leader late-commit limitation. +This mechanism does not fence GC-state write transactions across PD processes. In the rare case that a transaction accepted by an old leader commits after the new leader has read that keyspace's initial state, the new stream might miss the value until another effective safe-point change or reconnection. The per-stream non-regression guarantee still holds. Leadership-fenced GC-state transactions remain an out-of-scope follow-up. ## Removed-event integration The internal model, ordering logic, protobuf converter, and tests all accept `removed` changes, but the first PD implementation has no authoritative keyspace lifecycle hook that produces them. It also does not produce metadata-driven upserts for keyspace creation, enablement, or GC-mode changes. Adding only part of these producers would create misleading convergence guarantees, so production is deferred. -A future lifecycle integration must publish removal and recreation events through the same manager-serialized live path as safe-point changes. It must also define how lifecycle ownership interacts with GC leadership. The implementation records this location with a targeted TODO so the limitation is discoverable without expanding the present scope. +A future lifecycle integration must publish metadata-driven upserts and removals through the same manager-serialized live path as safe-point changes. The implementation records this integration point with a targeted TODO. ## Observability @@ -198,7 +183,7 @@ Metrics are intentionally low-cardinality and focus on operational decisions rat - A counter reports watcher terminations with the bounded reason label values `client_cancel`, `leader_lost`, `slow_consumer`, and `init_error`. - A slow-consumer log records the watcher identifier, configured live capacity, and observed queue length. -Watcher identifiers, keyspace identifiers, client addresses, and error strings are not metric labels. The four termination values describe watcher-domain lifecycle causes; a protobuf conversion bug is logged and returned at the transport layer instead of adding an `internal_error` label. The implementation does not add a per-send queue-length histogram because it would instrument the hot path without a demonstrated operational need. +Watcher identifiers, keyspace identifiers, client addresses, and error strings are not metric labels. The four termination values describe watcher lifecycle causes; transport conversion errors are logged separately. The implementation does not add a per-send queue-length histogram because it would instrument the hot path without a demonstrated operational need. ## Test strategy @@ -208,55 +193,47 @@ The watcher and service layers are tested separately, with focused integration c The domain tests cover: -- Registration succeeds only in an active local leadership generation, and ending or superseding that generation terminates its watchers. +- Registration succeeds only in an active local leadership generation; ending or superseding that generation terminates its watchers, while a delayed older teardown does not affect a newer generation. - `skip_loading_initial=false` returns initial and subsequent live states; `true` returns only post-registration live changes. -- Successful effective state changes publish complete upserts from the shared internal mutation paths, while no-op and failed mutations do not. -- Modern and legacy API entry points that reach the same internal mutation produce exactly one equivalent change. +- Effective safe-point changes publish complete upserts exactly once from the shared modern and legacy mutation paths; no-op and failed mutations do not publish. - Current barrier-only mutations produce no change, while any operation that changes an effective safe point does. -- An initial-first sequence emits `v1` followed by `v2`. -- A deterministic paused-initial sequence emits `v2` and suppresses the later initial `v1`. +- Initial-first delivery emits `v1` followed by `v2`; a deterministic paused-initial test emits `v2` and suppresses the later initial `v1`. - Filling watcher A's `liveCh` terminates A without delaying watcher B; A can reconnect and rebuild. - Initial iteration failure, caller cancellation, and concurrent deregistration terminate without goroutine or registry leaks. - A full `initCh` backpressures only that watcher's initial iterator and never waits on the channel while holding `GCStateManager.mu`. - Upsert and removed changes use the same per-scope dirty ordering rule. - Initial merge state remains active until a closed `initCh` is fully drained, and disabling the closed channel prevents select spinning. -- A delayed teardown callback for an old local leadership generation does not cancel watchers in a newer generation. ### Server tests The transport tests cover: -- Conversion of complete upsert and removed variants, including an empty barrier list. -- Response splitting at a reduced test limit, with assertions based on the serialized protobuf size on both sides of the boundary. -- A successful header appears in every split response, and the header contributes to the size boundary. +- Conversion covers complete upserts with empty barriers, removals, and invalid internal changes. +- Response batching covers exact protobuf size boundaries, successful headers, oversized single changes, and suppression of empty responses. - Cancellation between two responses derived from one `RecvBatch` prevents the remaining response from being sent. -- An oversized single change is sent alone, dirty-only input does not produce an empty response, and an invalid internal change returns `Internal`. - A rate-limit capacity of one: the first active stream holds the token, the second is rejected, and a third succeeds after the first closes. -- Request preflight covers cluster-ID mismatch, a non-serving member, and an unbootstrapped server. -- Actual leader transfer terminates the old stream and permits a fresh initial stream on the new leader. -- Client cancellation and send failure remove the watcher and release the rate-limit token. -- Domain error causes map to the specified gRPC status codes. -- The active-watcher gauge and each domain termination counter change exactly once across registration and cleanup. +- Request preflight covers cluster-ID mismatch and unavailable members, and domain errors map to the specified gRPC status codes. +- Leader transfer, client cancellation, and send failure terminate the stream, clean up the watcher, and release the rate-limit token. +- Watcher metrics change exactly once across registration and cleanup. ## Dependency and rollout The root, `client`, `tools`, and `tests/integrations` Go modules are updated to a kvproto revision containing the merged `WatchGCStates` API from PR #1528. No compatibility wrapper or implementation is added for `WatchGCSafePointV2`. -The API can be rolled out server-first because existing clients do not call the new RPC. New consumers use an initial stream to construct their materialized view and use the same path after any disconnect. The absence of lifecycle-produced removals remains an explicit limitation until the future integration is implemented. +The API can be rolled out server-first because existing clients do not call the new RPC. New consumers use an initial stream to construct their materialized view and use the same path after any disconnect. ## Acceptance criteria The implementation is complete when all of the following are true: -- `WatchGCStates` serves initial and local safe-point mutation changes with the documented `skip_loading_initial` behavior and metadata-change limitations. +- `WatchGCStates` serves initial states and local effective safe-point changes with the documented `skip_loading_initial` behavior. - The deterministic ordering test proves that no older initial value follows a newer live value for the same scope on one stream. - A full live queue terminates only the affected watcher without blocking GC-state mutation. -- Ending or superseding a local leadership generation terminates its active streams, and reconnecting with initial loading rebuilds the view subject to the documented cross-PD late-commit limitation. +- Ending or superseding a local leadership generation terminates its active streams. - Response batches observe the 1 MiB target using exact protobuf size accounting, except for the defined oversized-single-change case. - Metric labels use only the bounded dimensions described above. - Targeted package and server tests pass with no failpoints left enabled. - The implementation contains no `WatchGCSafePointV2` compatibility path and no Go client work. -- Cross-PD leadership fencing remains explicitly out of scope and is recorded as a follow-up TODO. ## Next steps From 0b6e52a77c29e155532b8279cdeac0f33ebdd005 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 17:44:04 +0800 Subject: [PATCH 04/25] docs: add WatchGCStates implementation plan Signed-off-by: Wenxuan Zhang --- .../plans/2026-09-03-watch-gc-states.md | 1430 +++++++++++++++++ 1 file changed, 1430 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-03-watch-gc-states.md diff --git a/docs/superpowers/plans/2026-09-03-watch-gc-states.md b/docs/superpowers/plans/2026-09-03-watch-gc-states.md new file mode 100644 index 00000000000..bd29a3ee1a3 --- /dev/null +++ b/docs/superpowers/plans/2026-09-03-watch-gc-states.md @@ -0,0 +1,1430 @@ +# WatchGCStates server implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Implement the PD `WatchGCStates` server stream with ordered initial and live delivery, isolated backpressure, local leadership lifecycle handling, exact protobuf response sizing, and bounded observability. + +**Architecture:** `pkg/gc` owns the watcher registry, initial-state iteration, live publication, per-keyspace merge ordering, cancellation causes, and lifecycle metrics. `server/gc_service.go` validates the local streaming request, converts domain changes to protobuf, splits responses by exact wire size, and sends them. Focused package tests prove deterministic concurrency behavior, while `tests/server/gc` covers real gRPC and leader-transfer behavior. + +**Tech stack:** Go 1.25, gRPC server streaming, gogo/protobuf, PingCAP failpoints, Prometheus client_golang, testify, and the existing PD test cluster. + +**Spec:** [`../specs/2026-09-03-watch-gc-states-design.md`](../specs/2026-09-03-watch-gc-states-design.md) + +## Global constraints + +Every task inherits these requirements from the approved design and repository rules: + +- Do not add `WatchGCSafePointV2` compatibility or a Go PD client API. +- Emit complete effective GC-state upserts only after a successful safe-point mutation updates the manager cache; do not emit barrier-only or no-op changes. +- Keep `removed` in the internal model and transport converter, but do not add keyspace lifecycle producers in this change. +- Register live delivery before initial scanning and suppress an initial state after a live change for the same scope has been emitted. +- Use `liveCh` capacity 1024, `initCh` capacity 1, initial batches of 1024 changes, and `RecvBatch(1024)` in production; expose capacities only through unexported test seams. +- Never block on watcher channels while holding `GCStateManager.mu`; evict only the watcher whose `liveCh` is full. +- Keep response wire size at or below 1 MiB except when one change alone exceeds the target, in which case send that change alone. +- Hold the `WatchGCStates` rate-limit token for the complete stream lifetime. +- Keep Prometheus labels bounded and pre-bind all `WithLabelValues` handles outside hot paths. +- Use `make gotest` for tests that rely on failpoints, and verify that failpoints are disabled before committing. +- Do not hard-wrap Markdown prose. + +## File map + +The implementation uses focused files and avoids unrelated refactoring: + +- Create `pkg/gc/gc_state_watcher.go` for domain changes, watcher merge state, registration helpers, initial loading, cancellation, and live fan-out. +- Create `pkg/gc/gc_state_watcher_test.go` for deterministic watcher, leadership, initial-loading, backpressure, publication, and metric tests. +- Modify `pkg/gc/gc_state_manager.go` only where manager fields, leadership callbacks, safe-point mutation hooks, and the existing iterator are involved. +- Modify `pkg/gc/gc_state_manager_test.go` only to adapt existing leadership setup and reuse its embedded-etcd GC fixtures. +- Modify `pkg/gc/metrics.go` for the active watcher gauge and bounded termination counters. +- Modify `pkg/errs/errno.go` and `errors.toml` for the slow-consumer sentinel. +- Modify `server/cluster/cluster.go` so each leader startup stores its generation-aware teardown closure. +- Modify `server/gc_service.go` for protobuf conversion, exact response splitting, local request preflight, domain-error mapping, and streaming. +- Create `server/gc_service_test.go` for converter, batching, cancellation-before-send, and error-mapping unit tests. +- Modify `tests/server/gc/gc_test.go` for real RPC, rate-limit, validation, and leader-transfer coverage. +- Modify the root, `client`, `tools`, and `tests/integrations` `go.mod` and `go.sum` pairs to use kvproto commit `65b4e27a438de9274bf88c58e89e83749e62f646`. + +--- + +### Task 1: Add the watcher change model and merge state + +This task creates the transport-independent state machine that merges initial batches and individual live changes without regressing a keyspace. + +**Files:** + +- Create: `pkg/gc/gc_state_watcher.go` +- Create: `pkg/gc/gc_state_watcher_test.go` + +**Interfaces:** + +- Consumes: Existing `GCState` from `pkg/gc/gc_state_manager.go`. +- Produces: `GCStateChange`, `NewGCStateUpsert`, `NewGCStateRemoved`, `GCStateChange.Upsert`, `GCStateChange.RemovedKeyspaceID`, `GCStateChange.KeyspaceID`, `GCStateWatcher.RecvBatch`, and `GCStateWatcher.Err`. +- Produces for Task 2: `newGCStateWatcher`, `gcStateWatchConfig`, `GCStateWatcher.initCh`, `GCStateWatcher.liveCh`, and `GCStateWatcher.cancel`. + +- [ ] **Step 1: Write failing tests for both observable delivery orders** + +Create tests that drive the channels directly so selection is deterministic: send initial `v1` and receive it before sending live `v2` for the initial-first case; receive live `v2` first, then send initial `v1` for the live-first case. + +```go +func TestGCStateWatcherInitialThenLive(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 1})} + + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint64(1), mustUpsert(t, got[0]).TxnSafePoint) + + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 2}) + got, err = w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint64(2), mustUpsert(t, got[0]).TxnSafePoint) +} + +func TestGCStateWatcherLiveSuppressesOlderInitial(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 2}) + + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint64(2), mustUpsert(t, got[0]).TxnSafePoint) + + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 1})} + close(w.initCh) + _, ok, err := w.receiveOne(false) + require.NoError(t, err) + require.False(t, ok) + require.True(t, w.initDone) +} +``` + +- [ ] **Step 2: Write failing tests for removed changes, closed-channel draining, batch bounds, and cancellation priority** + +Use the same direct-channel seam and add these exact cases: + +```go +func TestGCStateWatcherRemovedSuppressesInitial(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + w.liveCh <- NewGCStateRemoved(7) + got, err := w.RecvBatch(1) + require.NoError(t, err) + removed, ok := got[0].RemovedKeyspaceID() + require.True(t, ok) + require.Equal(t, uint32(7), removed) + + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7})} + close(w.initCh) + _, ok, err = w.receiveOne(false) + require.NoError(t, err) + require.False(t, ok) + require.True(t, w.initDone) +} + +func TestGCStateWatcherDrainsBufferedInitBeforeReleasingDirtySet(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 1}, false) + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7}) + _, err := w.RecvBatch(1) + require.NoError(t, err) + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 8})} + close(w.initCh) + + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint32(8), mustUpsert(t, got[0]).KeyspaceID) + require.False(t, w.initDone) + require.NotNil(t, w.dirtyDuringInit) + + _, ok, err := w.receiveOne(false) + require.NoError(t, err) + require.False(t, ok) + require.True(t, w.initDone) + require.Nil(t, w.initCh) + require.Nil(t, w.dirtyDuringInit) +} + +func TestGCStateWatcherRecvBatchHonorsMaximum(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 3}, true) + for id := uint32(1); id <= 3; id++ { + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: id}) + } + got, err := w.RecvBatch(2) + require.NoError(t, err) + require.Len(t, got, 2) + got, err = w.RecvBatch(2) + require.NoError(t, err) + require.Len(t, got, 1) +} + +func TestGCStateWatcherCancellationDiscardsBufferedWork(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 1}, true) + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7}) + want := errors.New("watch terminated") + w.cancel(want) + got, err := w.RecvBatch(1) + require.ErrorIs(t, err, want) + require.Nil(t, got) +} + +func TestGCStateWatcherFirstCancellationCauseWins(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 1}, true) + first := errors.New("first") + w.cancel(first) + w.cancel(errors.New("second")) + require.ErrorIs(t, w.Err(), first) +} +``` + +Add `mustUpsert` as a test helper that calls `change.Upsert()`, requires the boolean result, and returns the state. + +- [ ] **Step 3: Run the focused tests and confirm the red state** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run ^TestGCStateWatcher -count=1' +``` + +Expected: compilation fails because the watcher types and constructors do not exist. + +- [ ] **Step 4: Implement the domain change type and test helper accessors** + +Use an unexported discriminator so the zero value remains structurally invalid for Task 5 converter tests. + +```go +type gcStateChangeKind uint8 + +const ( + gcStateChangeUnknown gcStateChangeKind = iota + gcStateChangeUpsert + gcStateChangeRemoved +) + +type GCStateChange struct { + kind gcStateChangeKind + upsert GCState + removedKeyspaceID uint32 +} + +func NewGCStateUpsert(state GCState) GCStateChange { + state.GCBarriers = nil + return GCStateChange{kind: gcStateChangeUpsert, upsert: state} +} + +func NewGCStateRemoved(keyspaceID uint32) GCStateChange { + return GCStateChange{kind: gcStateChangeRemoved, removedKeyspaceID: keyspaceID} +} + +func (c GCStateChange) Upsert() (GCState, bool) { + return c.upsert, c.kind == gcStateChangeUpsert +} + +func (c GCStateChange) RemovedKeyspaceID() (uint32, bool) { + return c.removedKeyspaceID, c.kind == gcStateChangeRemoved +} + +func (c GCStateChange) KeyspaceID() (uint32, bool) { + if state, ok := c.Upsert(); ok { + return state.KeyspaceID, true + } + return c.RemovedKeyspaceID() +} +``` + +Add GoDoc to every exported type and function. + +- [ ] **Step 5: Implement the single-consumer merge** + +Define the production defaults and the unexported configuration seam: + +```go +const ( + defaultGCStateWatchInitialBatchSize = 1024 + defaultGCStateWatchInitChannelCapacity = 1 + defaultGCStateWatchLiveChannelCapacity = 1024 +) + +type gcStateWatchConfig struct { + initialBatchSize int + initChannelCapacity int + liveChannelCapacity int +} + +type GCStateWatcher struct { + ctx context.Context + cancel context.CancelCauseFunc + initCh chan []GCStateChange + liveCh chan GCStateChange + initDone bool + pendingInit []GCStateChange + dirtyDuringInit map[uint32]struct{} +} + +func newGCStateWatcher(parent context.Context, cfg gcStateWatchConfig, skipLoadingInitial bool) *GCStateWatcher { + ctx, cancel := context.WithCancelCause(parent) + watcher := &GCStateWatcher{ + ctx: ctx, + cancel: cancel, + initCh: make(chan []GCStateChange, cfg.initChannelCapacity), + liveCh: make(chan GCStateChange, cfg.liveChannelCapacity), + initDone: skipLoadingInitial, + } + if !skipLoadingInitial { + watcher.dirtyDuringInit = make(map[uint32]struct{}) + } + return watcher +} +``` + +Implement `receiveOne(block bool)` as the only place that selects from the channels. It performs these branches in order: + +1. Return `context.Cause(w.ctx)` before inspecting buffered work. +2. Consume `pendingInit` first, dropping an initial change when its keyspace is in `dirtyDuringInit`. +3. If initial loading is complete, select only `ctx.Done()` and `liveCh`. +4. Otherwise, select `ctx.Done()`, `liveCh`, and `initCh`. Mark every emitted live scope dirty. Copy a received initial batch into `pendingInit`. When the closed `initCh` is observed after buffered batches are drained, set `initCh=nil`, `initDone=true`, and `dirtyDuringInit=nil`. +5. For opportunistic collection, add a `default` branch when `block=false`. + +Implement `RecvBatch` by blocking for its first visible change, calling `receiveOne(false)` until `maxChanges` is reached or no visible work is ready, and checking `Err()` again before returning the batch. Panic on a non-positive `maxChanges`, because this is an internal programmer error and production always passes 1024. + +```go +func (w *GCStateWatcher) Err() error { + return context.Cause(w.ctx) +} + +func (w *GCStateWatcher) RecvBatch(maxChanges int) ([]GCStateChange, error) { + if maxChanges <= 0 { + panic("GCStateWatcher.RecvBatch requires a positive maximum") + } + first, ok, err := w.receiveOne(true) + if err != nil { + return nil, err + } + if !ok { + panic("blocking watcher receive returned no result") + } + result := []GCStateChange{first} + for len(result) < maxChanges { + change, ok, err := w.receiveOne(false) + if err != nil { + return nil, err + } + if !ok { + break + } + result = append(result, change) + } + if err := w.Err(); err != nil { + return nil, err + } + return result, nil +} +``` + +Place a comment beside the dirty-set branches explaining the two valid orders, `v1 -> v2` and `v2` with `v1` suppressed, as required by the spec. + +- [ ] **Step 6: Run the watcher merge tests and confirm the green state** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run ^TestGCStateWatcher -count=1' +``` + +Expected: all watcher merge tests pass, including cancellation under `-count=1`. + +- [ ] **Step 7: Format and commit the watcher state machine** + +Run: + +```bash +gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go +git diff --check +git add pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go +git commit -s -m "gc: add GC state watcher merge model" +``` + +### Task 2: Register watchers and bind them to local leadership + +This task connects the watcher state machine to `GCStateManager`, starts incremental initial loading, and replaces the counter-style follower callback with a generation-aware teardown closure. + +**Files:** + +- Modify: `pkg/gc/gc_state_watcher.go` +- Modify: `pkg/gc/gc_state_watcher_test.go` +- Modify: `pkg/gc/gc_state_manager.go:188-266` +- Modify: `pkg/gc/gc_state_manager_test.go:116-262` +- Modify: `server/cluster/cluster.go:495-496` + +**Interfaces:** + +- Consumes: Task 1's watcher channels, `RecvBatch`, and configuration seam; existing `iterateAllKeyspacesGCStates`. +- Produces: `GCStateManager.WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error)`, `GCStateWatcher.Close()`, and `GCStateManager.OnNodeBecomesLeader() func()`. +- Produces for Tasks 3 and 4: `terminateGCStateWatcherLocked`, the watcher registry, watcher IDs, and bounded termination-reason constants. + +- [ ] **Step 1: Write failing tests for registration and generation-aware teardown** + +Add tests with these concrete sequences: + +```go +func (s *gcStateManagerTestSuite) TestGCStateWatchRequiresActiveLeadership() { + follower := NewGCStateManager(s.provider, s.manager.cfg, s.manager.keyspaceManager) + _, err := follower.WatchGCStates(context.Background(), true) + s.Require().ErrorIs(err, errs.ErrNotLeader) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchLeadershipGeneration() { + re := s.Require() + stopFirst := s.manager.OnNodeBecomesLeader() + first, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + + stopSecond := s.manager.OnNodeBecomesLeader() + re.ErrorIs(first.Err(), errs.ErrNotLeader) + second, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + + stopFirst() + re.NoError(second.Err()) + stopSecond() + re.ErrorIs(second.Err(), errs.ErrNotLeader) +} +``` + +Use the existing suite fixture rather than duplicating embedded-etcd setup. Construct the follower manager from the suite's provider, config, and keyspace manager so it has never received a leader callback. + +- [ ] **Step 2: Write failing tests for initial loading and cleanup** + +Add these exact tests: + +- `TestGCStateWatchLoadsInitialStatesIncrementally`: call the unexported configured registration helper with batch size 2 and assert active keyspaces arrive in batches with no barriers. +- `TestGCStateWatchSkipsInitialLoading`: register with `skipLoadingInitial=true`, assert `initDone`, and verify the initial channel remains unused. +- `TestGCStateWatchLiveSuppressesPausedInitial`: persist transaction safe point `v1` for keyspace 2, pause `watchGCStatesInitialStateLoaded` only when that scope is read, register with initial loading, advance the same keyspace to `v2`, consume the live `v2`, release the loader, drain initial work, and assert that no emitted change contains `v1` for keyspace 2. +- `TestGCStateWatchInitialFailureTerminatesWatcher`: enable `iterateAllKeyspacesGCStatesError`, assert `RecvBatch` returns the injected error, and assert the registry no longer contains the watcher. +- `TestGCStateWatchFullInitChannelDoesNotHoldManagerMutex`: use initial batch size 1 and `initCh` capacity 1, wait until the channel is full, run a manager mutation and the leadership teardown with bounded channels, then close the watcher and assert the loader exits. +- `TestGCStateWatchConcurrentCloseIsIdempotent`: call `Close`, initialization termination, and leadership teardown concurrently and assert no registry or goroutine leak. + +Implement the paused-initial case with this sequence; use a larger test-only `initCh` capacity so the loader can reach keyspace 2 before the consumer starts draining earlier keyspaces: + +```go +func (s *gcStateManagerTestSuite) TestGCStateWatchLiveSuppressesPausedInitial() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 10, time.Now()) + re.NoError(err) + + reached := make(chan struct{}) + release := make(chan struct{}) + var reachedOnce, releaseOnce sync.Once + releaseLoader := func() { releaseOnce.Do(func() { close(release) }) } + re.NoError(failpoint.EnableCall("github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded", func(id uint32) { + if id == keyspaceID { + reachedOnce.Do(func() { close(reached) }) + <-release + } + })) + defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded")) }() + defer releaseLoader() + + w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{initialBatchSize: 1, initChannelCapacity: 16, liveChannelCapacity: 4}) + re.NoError(err) + defer w.Close() + select { + case <-reached: + case <-time.After(5 * time.Second): + re.FailNow("initial loader did not reach keyspace 2") + } + + _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + for { + changes, err := w.RecvBatch(1) + re.NoError(err) + state, ok := changes[0].Upsert() + if ok && state.KeyspaceID == keyspaceID && state.TxnSafePoint == 20 { + break + } + } + releaseLoader() + + re.Eventually(func() bool { + for { + change, ok, err := w.receiveOne(false) + re.NoError(err) + if !ok { + return w.initDone + } + state, upsert := change.Upsert() + re.False(upsert && state.KeyspaceID == keyspaceID && state.TxnSafePoint == 10) + } + }, 5*time.Second, 10*time.Millisecond) +} +``` + +Add two failpoint call sites to make timing deterministic: `watchGCStatesRegistered` immediately after registration releases `GCStateManager.mu`, and `watchGCStatesInitialStateLoaded` after one state is read but before its batch can be sent. Neither call site can execute while holding the manager mutex. + +- [ ] **Step 3: Run the focused lifecycle tests and confirm the red state** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/TestGCStateWatch(Requires|Leadership|Loads|Skips|Live|Initial|Full|Concurrent)" -count=1' +``` + +Expected: compilation fails because manager registration, teardown closures, and watcher cleanup do not exist. + +- [ ] **Step 4: Replace leadership counting with an active generation and teardown closure** + +Keep lock-free leadership reads for existing cache fast paths while making generation creation manager-owned: + +```go +watchers map[uint64]*GCStateWatcher +nextWatcherID uint64 +nextLeadershipGeneration uint64 +activeLeadershipGeneration atomic.Uint64 + +func (m *GCStateManager) OnNodeBecomesLeader() func() { + m.mu.Lock() + m.nextLeadershipGeneration++ + generation := m.nextLeadershipGeneration + m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) + m.activeLeadershipGeneration.Store(generation) + m.gcStateCache.clearAll() + m.mu.Unlock() + + return func() { + m.mu.Lock() + defer m.mu.Unlock() + if m.activeLeadershipGeneration.Load() != generation { + return + } + m.activeLeadershipGeneration.Store(0) + m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) + m.gcStateCache.clearAll() + } +} + +func (m *GCStateManager) nodeIsLeader() bool { + return m.activeLeadershipGeneration.Load() != 0 +} +``` + +Initialize the watcher map in `NewGCStateManager`. Remove `OnNodeBecomesFollower`; change cluster startup to the following direct assignment so the closure captured for this exact generation is invoked by `RaftCluster.Stop`: + +```go +c.stopGCStateManager = s.GetGCStateManager().OnNodeBecomesLeader() +``` + +- [ ] **Step 5: Implement registration, idempotent removal, and initial loading** + +Use these exact public and test-only entry points: + +```go +func (m *GCStateManager) WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error) { + return m.watchGCStates(ctx, skipLoadingInitial, gcStateWatchConfig{ + initialBatchSize: defaultGCStateWatchInitialBatchSize, + initChannelCapacity: defaultGCStateWatchInitChannelCapacity, + liveChannelCapacity: defaultGCStateWatchLiveChannelCapacity, + }) +} + +func (m *GCStateManager) watchGCStates(ctx context.Context, skipLoadingInitial bool, cfg gcStateWatchConfig) (*GCStateWatcher, error) +func (m *GCStateManager) loadInitialGCStates(watcher *GCStateWatcher, batchSize int) +func (m *GCStateManager) terminateGCStateWatcher(watcher *GCStateWatcher, cause error, reason gcStateWatcherTerminationReason) +func (m *GCStateManager) terminateGCStateWatcherLocked(watcher *GCStateWatcher, cause error, reason gcStateWatcherTerminationReason) bool +func (m *GCStateManager) terminateAllGCStateWatchersLocked(cause error, reason gcStateWatcherTerminationReason) +func (w *GCStateWatcher) Close() +``` + +Add `manager *GCStateManager` and `id uint64` to `GCStateWatcher`. Define the bounded reasons now so Task 4 can attach metrics without changing lifecycle signatures: + +```go +type gcStateWatcherTerminationReason string + +const ( + watcherTerminationClientCancel gcStateWatcherTerminationReason = "client_cancel" + watcherTerminationLeaderLost gcStateWatcherTerminationReason = "leader_lost" + watcherTerminationSlowConsumer gcStateWatcherTerminationReason = "slow_consumer" + watcherTerminationInitError gcStateWatcherTerminationReason = "init_error" +) +``` + +Registration locks the manager, rejects generation 0, allocates an ID, assigns the manager and ID to the watcher, inserts it, unlocks, invokes `watchGCStatesRegistered`, and only then starts the loader. For skipped initial loading, construct the watcher with `initDone=true` and do not start a loader. + +The loader uses a local batch and a cancellation-aware blocking flush around the existing iterator: + +```go +batch := make([]GCStateChange, 0, batchSize) +stopped := false +flush := func() bool { + if len(batch) == 0 { + return true + } + ready := batch + batch = make([]GCStateChange, 0, batchSize) + select { + case watcher.initCh <- ready: + return true + case <-watcher.ctx.Done(): + return false + } +} + +err := m.iterateAllKeyspacesGCStates( + watcher.ctx, + true, + func(uint32) bool { return true }, + func(state GCState) { + if stopped { + return + } + failpoint.InjectCall("watchGCStatesInitialStateLoaded", state.KeyspaceID) + if watcher.Err() != nil { + stopped = true + return + } + batch = append(batch, NewGCStateUpsert(state)) + if len(batch) == batchSize { + stopped = !flush() + } + }, + nil, +) + +if stopped || watcher.Err() != nil { + return +} +if err != nil { + m.terminateGCStateWatcher(watcher, errors.Annotate(err, "load initial GC states"), watcherTerminationInitError) + return +} +if !flush() { + return +} +close(watcher.initCh) +``` + +The local `stopped` flag is required because the iterator callback cannot return an error. The failpoint runs before the state is appended, the flush replaces the batch backing slice before reuse, and only the loader closes `initCh` after successful iteration and final flush. If the watcher context already has a cause, the loader returns without replacing that cause. + +`terminateGCStateWatcherLocked` first verifies that the ID still maps to the same watcher, deletes it, and calls its `CancelCauseFunc` without invoking any callback that reacquires `GCStateManager.mu`. `Close` delegates to the manager with `context.Canceled` and `watcherTerminationClientCancel`. + +- [ ] **Step 6: Adapt existing manager tests to the teardown-returning callback** + +In `newGCStateManagerForTest`, retain the teardown and include it in the returned cleanup: + +```go +stopGCStateManager := gcStateManager.OnNodeBecomesLeader() +originalClean := clean +clean = func() { + stopGCStateManager() + originalClean() +} +``` + +Keep `ensureMarkedLeader` compatible by registering the returned closure with `s.T().Cleanup` whenever it creates a new leadership generation. Replace the one test that temporarily writes `nodeLeadership` with a save/set/restore of `activeLeadershipGeneration`, and update read-only assertions to call `nodeIsLeader()`. Search for every remaining `OnNodeBecomesLeader`, `OnNodeBecomesFollower`, and `nodeLeadership` reference so no test silently loses the teardown for the generation it creates. + +- [ ] **Step 7: Run lifecycle and existing cache tests** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatch|GetGCStateCache|GetAllKeyspacesGCStates)" -count=1' +go test ./server/cluster -run '^$' +``` + +Expected: watcher lifecycle tests pass, existing leader-gated cache behavior remains green, and the cluster package compiles with the new callback. + +- [ ] **Step 8: Format and commit manager integration** + +Run: + +```bash +gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/gc/gc_state_manager_test.go server/cluster/cluster.go +git diff --check +git add pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/gc/gc_state_manager_test.go server/cluster/cluster.go +git commit -s -m "gc: tie watchers to local leadership" +``` + +### Task 3: Publish effective safe-point changes without blocking mutations + +This task attaches live publication exactly once to the successful shared mutation paths and proves that a full watcher queue affects only that watcher. + +**Files:** + +- Modify: `pkg/gc/gc_state_watcher.go` +- Modify: `pkg/gc/gc_state_watcher_test.go` +- Modify: `pkg/gc/gc_state_manager.go:350-413` +- Modify: `pkg/gc/gc_state_manager.go:445-604` +- Modify: `pkg/errs/errno.go:551-554` +- Modify: `errors.toml` + +**Interfaces:** + +- Consumes: Task 2's registry and termination helpers. +- Produces: `publishGCStateChangeLocked`, `errs.ErrGCStateWatcherSlowConsumer`, and complete live upserts used by the server stream. + +- [ ] **Step 1: Write failing publication tests for modern, compatible, and barrier paths** + +Register watchers with `skipLoadingInitial=true` and assert these exact cases: + +```go +func (s *gcStateManagerTestSuite) TestGCStateWatchPublishesAdvanceGCSafePoint() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 10) + re.NoError(err) + changes, err := w.RecvBatch(1) + re.NoError(err) + state := mustUpsert(s.T(), changes[0]) + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20, GCSafePoint: 10}, state) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchPublishesAdvanceTxnSafePoint() { + re := s.Require() + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, err = s.manager.AdvanceTxnSafePoint(2, 20, time.Now()) + re.NoError(err) + changes, err := w.RecvBatch(1) + re.NoError(err) + re.Equal(uint64(20), mustUpsert(s.T(), changes[0]).TxnSafePoint) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchCompatiblePathsPublishOnce() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 30, time.Now()) + re.NoError(err) + + gcWatcher, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + _, _, err = s.manager.CompatibleUpdateGCSafePoint(keyspaceID, 10) + re.NoError(err) + changes, err := gcWatcher.RecvBatch(1) + re.NoError(err) + re.Len(changes, 1) + re.Empty(gcWatcher.liveCh) + gcWatcher.Close() + + txnWatcher, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + _, _, err = s.manager.CompatibleUpdateServiceGCSafePoint(keyspaceID, keypath.GCWorkerServiceSafePointID, 40, math.MaxInt64, time.Now()) + re.NoError(err) + changes, err = txnWatcher.RecvBatch(1) + re.NoError(err) + re.Len(changes, 1) + re.Empty(txnWatcher.liveCh) + txnWatcher.Close() +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchDoesNotPublishNoOpOrFailure() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 10) + re.NoError(err) + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + _, _, err = s.manager.CompatibleUpdateGCSafePoint(keyspaceID, 10) + re.NoError(err) + _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 9) + re.ErrorIs(err, errs.ErrDecreasingGCSafePoint) + re.Empty(w.liveCh) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchDoesNotPublishBarrierOnlyChanges() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, err = s.manager.SetGCBarrier(keyspaceID, "backup", 30, time.Hour, time.Now()) + re.NoError(err) + _, err = s.manager.DeleteGCBarrier(keyspaceID, "backup") + re.NoError(err) + re.Empty(w.liveCh) +} +``` + +For every complete upsert, assert `KeyspaceID`, `IsKeyspaceLevel`, `TxnSafePoint`, `GCSafePoint`, and an empty `GCBarriers` slice. + +- [ ] **Step 2: Write the failing slow-consumer isolation test** + +Create watcher A with `liveChannelCapacity=1` and watcher B with capacity 4. Publish two successful state changes without reading A, read B after each mutation, and assert: + +```go +re.ErrorIs(watcherA.Err(), errs.ErrGCStateWatcherSlowConsumer) +re.NoError(watcherB.Err()) +re.NotContains(s.manager.watchers, watcherA.id) +re.Contains(s.manager.watchers, watcherB.id) +``` + +Then reconnect A with initial loading enabled and assert its rebuilt state contains the latest safe points. + +- [ ] **Step 3: Run the publication tests and confirm the red state** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/TestGCStateWatch(Publishes|Compatible|DoesNotPublish|SlowConsumer)" -count=1' +``` + +Expected: tests fail because mutations do not publish and a full `liveCh` is not handled. + +- [ ] **Step 4: Add the slow-consumer sentinel and regenerate error documentation** + +Add the normalized error beside the existing GC errors: + +```go +ErrGCStateWatcherSlowConsumer = errors.Normalize("GC state watcher is too slow", errors.RFCCodeText("PD:gc:ErrGCStateWatcherSlowConsumer")) +``` + +Run: + +```bash +make generate-errdoc +``` + +Verify that `errors.toml` contains `PD:gc:ErrGCStateWatcherSlowConsumer` and no unrelated generated changes. + +- [ ] **Step 5: Implement non-blocking fan-out under the manager mutex** + +Add a locked helper that iterates the registry and uses a non-blocking send: + +```go +func (m *GCStateManager) publishGCStateChangeLocked(change GCStateChange) { + for _, watcher := range m.watchers { + select { + case watcher.liveCh <- change: + default: + log.Warn("GC state watcher is too slow", zap.Uint64("watcher-id", watcher.id), zap.Int("capacity", cap(watcher.liveCh)), zap.Int("queue-length", len(watcher.liveCh))) + m.terminateGCStateWatcherLocked(watcher, errs.ErrGCStateWatcherSlowConsumer, watcherTerminationSlowConsumer) + } + } + // TODO: Publish keyspace metadata upserts and removals through this same serialized path when an authoritative GC-leader-owned lifecycle hook exists. +} +``` + +Deleting the current watcher from a Go map during iteration is valid; do not collect a second removal list and do not wait, retry, or start one goroutine per watcher. + +- [ ] **Step 6: Publish from the two successful post-cache-update paths** + +Immediately after each existing `gcStateCache.store` call, publish only when the effective value changed: + +```go +if newGCSafePoint != oldGCSafePoint { + m.publishGCStateChangeLocked(NewGCStateUpsert(GCState{ + KeyspaceID: keyspaceID, + IsKeyspaceLevel: keyspaceID != constant.NullKeyspaceID, + TxnSafePoint: txnSafePoint, + GCSafePoint: newGCSafePoint, + })) +} +``` + +```go +if newTxnSafePoint != oldTxnSafePoint { + m.publishGCStateChangeLocked(NewGCStateUpsert(GCState{ + KeyspaceID: keyspaceID, + IsKeyspaceLevel: keyspaceID != constant.NullKeyspaceID, + TxnSafePoint: newTxnSafePoint, + GCSafePoint: gcSafePoint, + })) +} +``` + +Keep these calls in `advanceGCSafePointImpl` and `advanceTxnSafePointImpl`; do not add publication at public entry points. This gives modern and compatible callers exactly one event. + +- [ ] **Step 7: Run the publication and regression tests** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatch|Advance|Compatible|SetGCBarrier|DeleteGCBarrier)" -count=1' +``` + +Expected: complete upserts arrive once, no-op and barrier-only calls emit nothing, and only the slow watcher terminates. + +- [ ] **Step 8: Format and commit live publication** + +Run: + +```bash +gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/errs/errno.go +git diff --check +git add pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/errs/errno.go errors.toml +git commit -s -m "gc: publish effective safe point changes" +``` + +### Task 4: Instrument the watcher lifecycle + +This task adds low-cardinality metrics at the manager-owned registration and termination points, with pre-bound counter handles for every reason. + +**Files:** + +- Modify: `pkg/gc/metrics.go` +- Modify: `pkg/gc/gc_state_watcher.go` +- Modify: `pkg/gc/gc_state_watcher_test.go` + +**Interfaces:** + +- Consumes: Task 2's four termination-reason constants and the single idempotent termination helper. +- Produces: `pd_gc_watcher_count`, `pd_gc_watcher_termination_total{reason=...}`, and `recordGCStateWatcherTerminationMetrics`. + +- [ ] **Step 1: Write failing metric-delta tests** + +Use `prometheus/testutil.ToFloat64` and compare deltas so process-global counters do not make tests order-dependent: + +```go +func (s *gcStateManagerTestSuite) TestGCStateWatcherMetrics() { + re := s.Require() + activeBefore := promtestutil.ToFloat64(gcStateWatcherGauge) + leaderLostBefore := promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter) + + stop := s.manager.OnNodeBecomesLeader() + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + re.Equal(activeBefore+1, promtestutil.ToFloat64(gcStateWatcherGauge)) + + stop() + re.ErrorIs(w.Err(), errs.ErrNotLeader) + re.Equal(activeBefore, promtestutil.ToFloat64(gcStateWatcherGauge)) + re.Equal(leaderLostBefore+1, promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter)) + w.Close() + re.Equal(leaderLostBefore+1, promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter)) + stopRemainingCases := s.manager.OnNodeBecomesLeader() + defer stopRemainingCases() +} +``` + +In the same test, record the three remaining counters before their triggers. For `client_cancel`, register one watcher and call `Close`. For `slow_consumer`, register with live capacity 1 and perform two successful `AdvanceTxnSafePoint` calls without reading the watcher, so both publications run through the production path while `GCStateManager.mu` is held. For `init_error`, enable `iterateAllKeyspacesGCStatesError`, register with initial loading, and call `RecvBatch`. After each trigger, assert that only its expected counter increased by one and the active gauge returned to `activeBefore`; never mutate a metric directly. + +- [ ] **Step 2: Run the metric test and confirm the red state** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run TestGCStateManager/TestGCStateWatcherMetrics -count=1' +``` + +Expected: compilation fails because the watcher metrics do not exist. + +- [ ] **Step 3: Define and pre-bind the metrics** + +Add these definitions to `pkg/gc/metrics.go`: + +```go +gcStateWatcherGauge = prometheus.NewGauge(prometheus.GaugeOpts{ + Namespace: "pd", + Subsystem: "gc", + Name: "watcher_count", + Help: "Current number of active GC state watchers.", +}) +gcStateWatcherTerminationCounter = prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: "pd", + Subsystem: "gc", + Name: "watcher_termination_total", + Help: "Total number of GC state watcher terminations by reason.", +}, []string{"reason"}) + +gcStateWatcherTerminationClientCancelCounter = gcStateWatcherTerminationCounter.WithLabelValues("client_cancel") +gcStateWatcherTerminationLeaderLostCounter = gcStateWatcherTerminationCounter.WithLabelValues("leader_lost") +gcStateWatcherTerminationSlowConsumerCounter = gcStateWatcherTerminationCounter.WithLabelValues("slow_consumer") +gcStateWatcherTerminationInitErrorCounter = gcStateWatcherTerminationCounter.WithLabelValues("init_error") +``` + +Register the gauge and vector with `prometheus.MustRegister`. Do not call `WithLabelValues` in registration, publication, or cleanup paths. + +- [ ] **Step 4: Record metrics at the single lifecycle boundaries** + +Increment the gauge only after successful insertion into the registry. In `terminateGCStateWatcherLocked`, decrement the gauge and invoke this switch only after deletion succeeds: + +```go +func recordGCStateWatcherTerminationMetrics(reason gcStateWatcherTerminationReason) { + switch reason { + case watcherTerminationClientCancel: + gcStateWatcherTerminationClientCancelCounter.Inc() + case watcherTerminationLeaderLost: + gcStateWatcherTerminationLeaderLostCounter.Inc() + case watcherTerminationSlowConsumer: + gcStateWatcherTerminationSlowConsumerCounter.Inc() + case watcherTerminationInitError: + gcStateWatcherTerminationInitErrorCounter.Inc() + default: + panic("unknown GC state watcher termination reason") + } +} +``` + +No metric uses watcher IDs, keyspace IDs, client addresses, or error text as labels. The gauge has no labels, so teardown decrements it rather than deleting a label series. + +- [ ] **Step 5: Run metric and lifecycle tests** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatcherMetrics|GCStateWatch)" -count=1' +``` + +Expected: each lifecycle increments exactly one reason counter and returns the active gauge to its baseline. + +- [ ] **Step 6: Format and commit observability** + +Run: + +```bash +gofmt -w pkg/gc/metrics.go pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go +git diff --check +git add pkg/gc/metrics.go pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go +git commit -s -m "gc: add watcher lifecycle metrics" +``` + +### Task 5: Implement the gRPC transport adapter and upgrade kvproto + +This task adopts the merged protobuf API, converts domain changes, batches by exact wire size, and implements the local server-streaming handler through a small test seam. + +**Files:** + +- Modify: `go.mod` +- Modify: `go.sum` +- Modify: `client/go.mod` +- Modify: `client/go.sum` +- Modify: `tools/go.mod` +- Modify: `tools/go.sum` +- Modify: `tests/integrations/go.mod` +- Modify: `tests/integrations/go.sum` +- Modify: `server/gc_service.go` +- Create: `server/gc_service_test.go` + +**Interfaces:** + +- Consumes: Task 2's `GCStateManager.WatchGCStates` and `GCStateWatcher` API; Task 3's slow-consumer sentinel and domain changes; kvproto `WatchGCStatesRequest`, `WatchGCStatesResponse`, and `GCStateChange`. +- Produces: `GrpcServer.WatchGCStates`, `gcStateChangeToProto`, `splitWatchGCStatesResponses`, `watchGCStatesErrorToStatus`, and `serveWatchGCStates`. + +- [ ] **Step 1: Upgrade all four module scopes to the merged kvproto commit** + +Use the exact pseudo-version derived from merged commit `65b4e27a438de9274bf88c58e89e83749e62f646`: + +```bash +go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d +(cd client && go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d) +(cd tools && go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d) +(cd tests/integrations && go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d) +go mod tidy +(cd client && go mod tidy) +(cd tools && go mod tidy) +(cd tests/integrations && go mod tidy) +``` + +Run `git diff -- go.mod go.sum client/go.mod client/go.sum tools/go.mod tools/go.sum tests/integrations/go.mod tests/integrations/go.sum` and verify that the kvproto revision is the only dependency change. + +- [ ] **Step 2: Confirm that the upgraded server has a missing-method red state** + +Run: + +```bash +go test ./server -run '^$' +``` + +Expected: compilation reports that `*GrpcServer` does not implement `pdpb.PDServer` because `WatchGCStates` is missing. + +- [ ] **Step 3: Write failing converter and batching tests** + +Create table-driven converter coverage for a complete upsert, a removed scope, and zero-value invalid `gc.GCStateChange`. Assert that upserts have no barriers. Add boundary tests that calculate `base := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()})` and `delta := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}})`. + +```go +func TestSplitWatchGCStatesResponses(t *testing.T) { + change := &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{Upsert: &pdpb.GCState{ + KeyspaceScope: &pdpb.KeyspaceScope{Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: 7}}, + TxnSafePoint: 10, + GcSafePoint: 5, + }}} + base := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}) + delta := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}) + + exact := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta) + require.Len(t, exact, 1) + require.LessOrEqual(t, proto.Size(exact[0]), base+2*delta) + + split := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta-1) + require.Len(t, split, 2) + for _, response := range split { + require.NotNil(t, response.GetHeader()) + require.NotEmpty(t, response.GetChanges()) + require.LessOrEqual(t, proto.Size(response), base+2*delta-1) + } + + oversized := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change}, base+delta-1) + require.Len(t, oversized, 1) + require.Greater(t, proto.Size(oversized[0]), base+delta-1) + require.Empty(t, splitWatchGCStatesResponses(nil, base+delta)) +} +``` + +- [ ] **Step 4: Write failing stream-loop and error-mapping tests** + +Define a fake receiver implementing `RecvBatch(int) ([]gc.GCStateChange, error)` and `Err() error`, plus a fake `pdpb.PD_WatchGCStatesServer` whose `Send` callback can install a terminal cause. Cover these cases: + +- Two responses derived from one batch: after the first `Send`, set `errs.ErrNotLeader`; assert only one response is recorded and the function returns `Unavailable`. +- Invalid internal change: assert zero sends and gRPC `Internal`. +- `errs.ErrNotLeader` and an arbitrary initialization error: assert `Unavailable`. +- `errs.ErrGCStateWatcherSlowConsumer`: assert `ResourceExhausted`. +- `context.Canceled` and `context.DeadlineExceeded`: assert `Canceled` and `DeadlineExceeded`. + +Run: + +```bash +go test ./server -run 'Test(GCStateChangeToProto|SplitWatchGCStatesResponses|ServeWatchGCStates|WatchGCStatesErrorToStatus)' -count=1 +``` + +Expected: compilation fails because the adapter helpers do not exist. + +- [ ] **Step 5: Implement conversion and exact wire-size batching** + +Use these constants and receiver seam: + +```go +const ( + watchGCStatesRecvBatchSize = 1024 + maxWatchGCStatesResponseSize = 1 << 20 +) + +type gcStateChangeReceiver interface { + RecvBatch(maxChanges int) ([]gc.GCStateChange, error) + Err() error +} +``` + +Convert the internal discriminator into the generated oneof and reject the zero value: + +```go +func gcStateChangeToProto(change gc.GCStateChange) (*pdpb.GCStateChange, error) { + if state, ok := change.Upsert(); ok { + state.GCBarriers = nil + return &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{Upsert: gcStateToProto(state, time.Time{})}}, nil + } + if keyspaceID, ok := change.RemovedKeyspaceID(); ok { + return &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Removed{Removed: &pdpb.KeyspaceScope{Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: keyspaceID}}}}, nil + } + return nil, errors.New("invalid GC state change") +} +``` + +Implement `splitWatchGCStatesResponses(changes []*pdpb.GCStateChange, maxSize int)` with a fresh `grpcutil.WrapHeader()` response for each batch. Start `currentSize` with `proto.Size` of the header-only response. For each change, calculate the exact additive delta with `proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}})`. Flush a non-empty current response before an addition that exceeds `maxSize`; never append an empty response. If the first change itself exceeds the target with the header, keep it alone, log its serialized size, and start a fresh response for the next change. + +- [ ] **Step 6: Implement domain-error mapping and the send loop** + +Map only watcher-domain errors; return `Send` errors unchanged: + +```go +func watchGCStatesErrorToStatus(err error) error { + switch { + case errors.Is(err, errs.ErrGCStateWatcherSlowConsumer): + return status.Error(codes.ResourceExhausted, err.Error()) + case errors.Is(err, errs.ErrNotLeader): + return errs.ErrNotLeader + case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded): + return status.FromContextError(err).Err() + default: + return status.Error(codes.Unavailable, err.Error()) + } +} +``` + +`serveWatchGCStates` repeatedly receives at most 1024 changes, converts every change, splits them, checks `receiver.Err()` immediately before every `Send`, and returns the raw send error. Conversion failure logs the error and returns gRPC `Internal`; it does not create a watcher termination metric reason. + +```go +func serveWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error { + for { + changes, err := receiver.RecvBatch(watchGCStatesRecvBatchSize) + if err != nil { + return watchGCStatesErrorToStatus(err) + } + protoChanges := make([]*pdpb.GCStateChange, 0, len(changes)) + for _, change := range changes { + converted, err := gcStateChangeToProto(change) + if err != nil { + log.Error("failed to convert GC state change", zap.Error(err)) + return status.Error(codes.Internal, err.Error()) + } + protoChanges = append(protoChanges, converted) + } + for _, response := range splitWatchGCStatesResponses(protoChanges, maxResponseSize) { + if err := receiver.Err(); err != nil { + return watchGCStatesErrorToStatus(err) + } + if err := stream.Send(response); err != nil { + return err + } + } + } +} +``` + +- [ ] **Step 7: Implement local RPC preflight and stream ownership** + +Implement the generated method directly in `server/gc_service.go`: + +```go +func (s *GrpcServer) WatchGCStates(request *pdpb.WatchGCStatesRequest, stream pdpb.PD_WatchGCStatesServer) error { + done, err := s.rateLimitCheck() + if err != nil { + return err + } + if done != nil { + defer done() + } + if err := s.validateRequest(request.GetHeader()); err != nil { + return err + } + if s.GetRaftCluster() == nil { + return status.Error(codes.Unavailable, errs.ErrNotBootstrapped.FastGenByArgs().Error()) + } + + watcher, err := s.gcStateManager.WatchGCStates(stream.Context(), request.GetSkipLoadingInitial()) + if err != nil { + return watchGCStatesErrorToStatus(err) + } + defer watcher.Close() + return serveWatchGCStates(watcher, stream, maxWatchGCStatesResponseSize) +} +``` + +Do not call `unaryMiddleware` or create a delegate client. Calling `rateLimitCheck` in this public method preserves the `WatchGCStates` limiter label, and deferring `done` here holds the token until the stream exits. + +- [ ] **Step 8: Run transport tests and compile all kvproto consumers** + +Run: + +```bash +go test ./server -run 'Test(GCStateChangeToProto|SplitWatchGCStatesResponses|ServeWatchGCStates|WatchGCStatesErrorToStatus)' -count=1 +go test ./pkg/gc ./server -run '^$' +(cd client && go test ./... -run '^$') +(cd tools && go test ./... -run '^$') +(cd tests/integrations && go test ./... -run '^$') +``` + +Expected: the converter, batching, cancellation, and status tests pass, and every module compiles against the same kvproto revision. + +- [ ] **Step 9: Format and commit the transport implementation** + +Run: + +```bash +gofmt -w server/gc_service.go server/gc_service_test.go +git diff --check +git add go.mod go.sum client/go.mod client/go.sum tools/go.mod tools/go.sum tests/integrations/go.mod tests/integrations/go.sum server/gc_service.go server/gc_service_test.go +git commit -s -m "server: implement WatchGCStates stream" +``` + +### Task 6: Prove RPC behavior in a real PD cluster + +This task covers the complete server stream, request preflight, lifetime rate limiting, and leader transfer through real generated gRPC clients. + +**Files:** + +- Modify: `tests/server/gc/gc_test.go` +- Modify if a test exposes an implementation defect: `server/gc_service.go` +- Modify if a test exposes a domain defect: `pkg/gc/gc_state_watcher.go` + +**Interfaces:** + +- Consumes: The generated `pdpb.PDClient.WatchGCStates` client and all production behavior from Tasks 1 through 5. +- Produces: End-to-end evidence for initial loading, skip-initial registration, validation, rate-limit token lifetime, and leadership reconnection. + +- [ ] **Step 1: Add deterministic stream helpers** + +Add a failpoint name constant for `github.com/tikv/pd/pkg/gc/watchGCStatesRegistered` and helpers that use bounded contexts: + +```go +func recvWatchGCStateForKeyspace(t *testing.T, stream pdpb.PD_WatchGCStatesClient, keyspaceID uint32) *pdpb.GCState { + t.Helper() + for { + response, err := stream.Recv() + require.NoError(t, err) + require.NotNil(t, response.GetHeader()) + for _, change := range response.GetChanges() { + if state := change.GetUpsert(); state != nil && state.GetKeyspaceScope().GetKeyspaceId() == keyspaceID { + return state + } + } + } +} +``` + +Use `context.WithTimeout(..., 20*time.Second)` for every stream and clean up every connection, context, and failpoint with `t.Cleanup` or `defer`. + +- [ ] **Step 2: Write the failing initial and skip-initial RPC test** + +In a bootstrapped one-node cluster, create a keyspace-level GC keyspace and establish `skip_loading_initial=false`. Read until that keyspace's complete initial state arrives, advance its transaction safe point, and read the complete live state. + +For `skip_loading_initial=true`, enable `watchGCStatesRegistered` with a `sync.Once` callback that closes a channel, establish the stream, wait for the callback, advance the same keyspace again, and assert the first received state contains the post-registration value. The registration callback removes timing sleeps and proves that no initial value was sent. + +```go +stream, err := grpcPDClient.WatchGCStates(ctx, &pdpb.WatchGCStatesRequest{Header: header, SkipLoadingInitial: true}) +require.NoError(t, err) +select { +case <-registered: +case <-time.After(5 * time.Second): + require.FailNow(t, "WatchGCStates was not registered") +} +``` + +- [ ] **Step 3: Write failing request-preflight tests** + +Use table-driven subtests that call `Recv` to observe server-stream establishment failures: + +- A request with the wrong cluster ID returns `codes.FailedPrecondition`. +- A direct request to a follower returns `codes.Unavailable` and is not forwarded. +- A request to the elected leader before `BootstrapCluster` returns `codes.Unavailable`. + +Assert that none of these cases returns a response message. + +- [ ] **Step 4: Write the failing lifetime rate-limit test** + +Enable gRPC rate limiting through `GetServiceMiddlewarePersistOptions().SetGRPCRateLimitConfig`, then call `GetGRPCRateLimiter().Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(1))`. Use the registration failpoint to wait until stream 1 holds its token. Assert stream 2 returns `codes.ResourceExhausted`. Cancel stream 1, use `GetConcurrencyLimiterStatus("WatchGCStates")` with `testutil.Eventually` until current usage is zero, then establish stream 3 and receive a change after a safe-point advancement. Restore the prior rate-limit config and delete the test limiter with `ratelimit.UpdateConcurrencyLimiter(0)` during cleanup. + +- [ ] **Step 5: Write the failing leader-transfer and reinitialization test** + +Reuse `newGCStateLeaderTransitionCluster`. Start an initial watch directly against the old leader and receive its current null-keyspace state. Resign that leader, wait for a different leader, and drain the old stream until it returns `codes.Unavailable`. Advance the safe point on the new leader, connect to the new leader with `skip_loading_initial=false`, and assert its initial state contains the new value. + +- [ ] **Step 6: Run the real-cluster tests and confirm the red or green state** + +Run: + +```bash +make gotest GOTEST_ARGS='./tests/server/gc -run ^TestWatchGCStates -count=1' +``` + +Expected before any required correction: at least one new test fails if the production path does not meet its contract. If all tests pass immediately, retain the tests as integration coverage and do not alter production code. + +- [ ] **Step 7: Make only corrections demonstrated by the failing integration assertions** + +Keep corrections inside the approved interfaces. Typical permitted corrections are preflight ordering, status mapping, cleanup order, or an additional cancellation check before `Send`. Do not add metadata producers, cross-PD transaction fencing, a replay protocol, a Go client, or `WatchGCSafePointV2` behavior. + +After each correction, rerun: + +```bash +make gotest GOTEST_ARGS='./tests/server/gc -run ^TestWatchGCStates -count=1' +make gotest GOTEST_ARGS='./pkg/gc -run TestGCStateManager/TestGCStateWatch -count=1' +make gotest GOTEST_ARGS='./server -run WatchGCStates -count=1' +``` + +Expected: all focused domain, transport, and integration tests pass. + +- [ ] **Step 8: Format and commit real-cluster coverage** + +Run: + +```bash +gofmt -w tests/server/gc/gc_test.go server/gc_service.go pkg/gc/gc_state_watcher.go +git diff --check +git add tests/server/gc/gc_test.go server/gc_service.go pkg/gc/gc_state_watcher.go +git commit -s -m "tests: cover WatchGCStates lifecycle" +``` + +Omit unchanged production files from `git add`. If integration testing required no production correction, commit only `tests/server/gc/gc_test.go`. + +### Task 7: Run final verification + +This task verifies formatting, generated error documentation, module consistency, race safety, focused behavior, and the repository's required checks before handoff. + +**Files:** + +- Verify: all files changed in Tasks 1 through 6 +- Modify: none unless a verification command reports a concrete defect + +**Interfaces:** + +- Consumes: The complete implementation and tests. +- Produces: Fresh command output demonstrating that the branch is ready for review. + +- [ ] **Step 1: Ensure failpoints are disabled before non-test commands** + +Run: + +```bash +make failpoint-disable +``` + +Expected: failpoint-generated rewrites are removed before formatting or static analysis. + +- [ ] **Step 2: Verify formatting and module tidiness** + +Run: + +```bash +make fmt +make generate-errdoc +make tidy +git diff --check +``` + +Expected: `make tidy` and `git diff --check` exit successfully. Inspect any formatting or generated change and commit it with the task that introduced the affected file; do not create an unexplained cleanup commit. + +- [ ] **Step 3: Run focused package tests with failpoint handling** + +Run: + +```bash +make gotest GOTEST_ARGS='./pkg/gc -run ^TestGCStateWatcher -count=1' +make gotest GOTEST_ARGS='./pkg/gc -run TestGCStateManager/TestGCStateWatch -count=1' +make gotest GOTEST_ARGS='./server -run "Test(GCStateChangeToProto|SplitWatchGCStatesResponses|ServeWatchGCStates|WatchGCStatesErrorToStatus)" -count=1' +make gotest GOTEST_ARGS='./tests/server/gc -run ^TestWatchGCStates -count=1' +``` + +Expected: every focused test passes with zero failures. + +- [ ] **Step 4: Run the focused race check** + +Run: + +```bash +make gotest GOTEST_ARGS='-race ./pkg/gc -run ^TestGCStateWatcher -count=1' +make gotest GOTEST_ARGS='-race ./pkg/gc -run TestGCStateManager/TestGCStateWatch -count=1' +make gotest GOTEST_ARGS='-race ./server ./tests/server/gc -run WatchGCStates -count=1' +``` + +Expected: all focused tests pass under the race detector with no race report or goroutine leak. + +- [ ] **Step 5: Run repository-level checks** + +Run: + +```bash +make check +make basic-test +``` + +Expected: formatting, lint, leak checks, generated error documentation, and the basic unit-test suite pass. + +- [ ] **Step 6: Verify scope and repository hygiene** + +Run: + +```bash +make failpoint-disable +! rg -n 'WatchGCSafePointV2' pkg/gc/gc_state_watcher.go server/gc_service.go tests/server/gc/gc_test.go +git status --short +git log --oneline --decorate -10 +``` + +Expected: the `rg` command finds no new compatibility reference in the touched WatchGCStates paths, `git status --short` is empty, and the log shows the signed task commits in order. + +## Execution handoff + +The plan is complete when this document is reviewed and committed. Execute it with one of the required workflows: + +1. **Subagent-driven:** Use `superpowers:subagent-driven-development`, dispatch a fresh worker for each task, and perform spec and code-quality review between tasks. +2. **Inline execution:** Use `superpowers:executing-plans`, execute tasks in batches, and stop at its review checkpoints. From d5e30684024743607c1679048535852c77afbe51 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 18:03:07 +0800 Subject: [PATCH 05/25] docs: consolidate WatchGCStates implementation tasks Signed-off-by: Wenxuan Zhang --- .../plans/2026-09-03-watch-gc-states.md | 114 ++++++++---------- 1 file changed, 47 insertions(+), 67 deletions(-) diff --git a/docs/superpowers/plans/2026-09-03-watch-gc-states.md b/docs/superpowers/plans/2026-09-03-watch-gc-states.md index bd29a3ee1a3..62236f45811 100644 --- a/docs/superpowers/plans/2026-09-03-watch-gc-states.md +++ b/docs/superpowers/plans/2026-09-03-watch-gc-states.md @@ -44,20 +44,24 @@ The implementation uses focused files and avoids unrelated refactoring: --- -### Task 1: Add the watcher change model and merge state +### Task 1: Build the watcher core and local leadership lifecycle -This task creates the transport-independent state machine that merges initial batches and individual live changes without regressing a keyspace. +This task creates the transport-independent initial/live merge state, registers watchers with `GCStateManager`, incrementally loads initial state, and binds every watcher to one local leadership generation. **Files:** - Create: `pkg/gc/gc_state_watcher.go` - Create: `pkg/gc/gc_state_watcher_test.go` +- Modify: `pkg/gc/gc_state_manager.go:188-266` +- Modify: `pkg/gc/gc_state_manager_test.go:116-262` +- Modify: `server/cluster/cluster.go:495-496` **Interfaces:** - Consumes: Existing `GCState` from `pkg/gc/gc_state_manager.go`. - Produces: `GCStateChange`, `NewGCStateUpsert`, `NewGCStateRemoved`, `GCStateChange.Upsert`, `GCStateChange.RemovedKeyspaceID`, `GCStateChange.KeyspaceID`, `GCStateWatcher.RecvBatch`, and `GCStateWatcher.Err`. -- Produces for Task 2: `newGCStateWatcher`, `gcStateWatchConfig`, `GCStateWatcher.initCh`, `GCStateWatcher.liveCh`, and `GCStateWatcher.cancel`. +- Produces: `newGCStateWatcher`, `gcStateWatchConfig`, `GCStateWatcher.initCh`, `GCStateWatcher.liveCh`, `GCStateWatcher.cancel`, `GCStateManager.WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error)`, `GCStateWatcher.Close()`, and `GCStateManager.OnNodeBecomesLeader() func()`. +- Produces for Task 2: `terminateGCStateWatcherLocked`, the watcher registry, watcher IDs, and bounded termination-reason constants. - [ ] **Step 1: Write failing tests for both observable delivery orders** @@ -185,7 +189,7 @@ Expected: compilation fails because the watcher types and constructors do not ex - [ ] **Step 4: Implement the domain change type and test helper accessors** -Use an unexported discriminator so the zero value remains structurally invalid for Task 5 converter tests. +Use an unexported discriminator so the zero value remains structurally invalid for Task 3 converter tests. ```go type gcStateChangeKind uint8 @@ -328,36 +332,22 @@ make gotest GOTEST_ARGS='./pkg/gc -run ^TestGCStateWatcher -count=1' Expected: all watcher merge tests pass, including cancellation under `-count=1`. -- [ ] **Step 7: Format and commit the watcher state machine** +- [ ] **Step 7: Format and inspect the watcher state machine** Run: ```bash gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go git diff --check -git add pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go -git commit -s -m "gc: add GC state watcher merge model" ``` -### Task 2: Register watchers and bind them to local leadership - -This task connects the watcher state machine to `GCStateManager`, starts incremental initial loading, and replaces the counter-style follower callback with a generation-aware teardown closure. - -**Files:** - -- Modify: `pkg/gc/gc_state_watcher.go` -- Modify: `pkg/gc/gc_state_watcher_test.go` -- Modify: `pkg/gc/gc_state_manager.go:188-266` -- Modify: `pkg/gc/gc_state_manager_test.go:116-262` -- Modify: `server/cluster/cluster.go:495-496` +Expected: formatting and whitespace checks pass before manager integration begins. Do not commit yet because the watcher isn't usable until the remaining steps connect it to `GCStateManager`. -**Interfaces:** +#### Manager integration phase -- Consumes: Task 1's watcher channels, `RecvBatch`, and configuration seam; existing `iterateAllKeyspacesGCStates`. -- Produces: `GCStateManager.WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error)`, `GCStateWatcher.Close()`, and `GCStateManager.OnNodeBecomesLeader() func()`. -- Produces for Tasks 3 and 4: `terminateGCStateWatcherLocked`, the watcher registry, watcher IDs, and bounded termination-reason constants. +This phase connects the tested merge state to the manager, starts incremental loading, and replaces the counter-style follower callback with a generation-aware teardown closure. It remains part of Task 1 because neither half is independently usable. -- [ ] **Step 1: Write failing tests for registration and generation-aware teardown** +- [ ] **Step 8: Write failing tests for registration and generation-aware teardown** Add tests with these concrete sequences: @@ -388,7 +378,7 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchLeadershipGeneration() { Use the existing suite fixture rather than duplicating embedded-etcd setup. Construct the follower manager from the suite's provider, config, and keyspace manager so it has never received a leader callback. -- [ ] **Step 2: Write failing tests for initial loading and cleanup** +- [ ] **Step 9: Write failing tests for initial loading and cleanup** Add these exact tests: @@ -458,7 +448,7 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchLiveSuppressesPausedInitial() Add two failpoint call sites to make timing deterministic: `watchGCStatesRegistered` immediately after registration releases `GCStateManager.mu`, and `watchGCStatesInitialStateLoaded` after one state is read but before its batch can be sent. Neither call site can execute while holding the manager mutex. -- [ ] **Step 3: Run the focused lifecycle tests and confirm the red state** +- [ ] **Step 10: Run the focused lifecycle tests and confirm the red state** Run: @@ -468,7 +458,7 @@ make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/TestGCStateWatch(Requ Expected: compilation fails because manager registration, teardown closures, and watcher cleanup do not exist. -- [ ] **Step 4: Replace leadership counting with an active generation and teardown closure** +- [ ] **Step 11: Replace leadership counting with an active generation and teardown closure** Keep lock-free leadership reads for existing cache fast paths while making generation creation manager-owned: @@ -510,7 +500,7 @@ Initialize the watcher map in `NewGCStateManager`. Remove `OnNodeBecomesFollower c.stopGCStateManager = s.GetGCStateManager().OnNodeBecomesLeader() ``` -- [ ] **Step 5: Implement registration, idempotent removal, and initial loading** +- [ ] **Step 12: Implement registration, idempotent removal, and initial loading** Use these exact public and test-only entry points: @@ -531,7 +521,7 @@ func (m *GCStateManager) terminateAllGCStateWatchersLocked(cause error, reason g func (w *GCStateWatcher) Close() ``` -Add `manager *GCStateManager` and `id uint64` to `GCStateWatcher`. Define the bounded reasons now so Task 4 can attach metrics without changing lifecycle signatures: +Add `manager *GCStateManager` and `id uint64` to `GCStateWatcher`. Define the bounded reasons now so the lifecycle metrics phase can attach metrics without changing lifecycle signatures: ```go type gcStateWatcherTerminationReason string @@ -603,7 +593,7 @@ The local `stopped` flag is required because the iterator callback cannot return `terminateGCStateWatcherLocked` first verifies that the ID still maps to the same watcher, deletes it, and calls its `CancelCauseFunc` without invoking any callback that reacquires `GCStateManager.mu`. `Close` delegates to the manager with `context.Canceled` and `watcherTerminationClientCancel`. -- [ ] **Step 6: Adapt existing manager tests to the teardown-returning callback** +- [ ] **Step 13: Adapt existing manager tests to the teardown-returning callback** In `newGCStateManagerForTest`, retain the teardown and include it in the returned cleanup: @@ -618,7 +608,7 @@ clean = func() { Keep `ensureMarkedLeader` compatible by registering the returned closure with `s.T().Cleanup` whenever it creates a new leadership generation. Replace the one test that temporarily writes `nodeLeadership` with a save/set/restore of `activeLeadershipGeneration`, and update read-only assertions to call `nodeIsLeader()`. Search for every remaining `OnNodeBecomesLeader`, `OnNodeBecomesFollower`, and `nodeLeadership` reference so no test silently loses the teardown for the generation it creates. -- [ ] **Step 7: Run lifecycle and existing cache tests** +- [ ] **Step 14: Run lifecycle and existing cache tests** Run: @@ -629,7 +619,7 @@ go test ./server/cluster -run '^$' Expected: watcher lifecycle tests pass, existing leader-gated cache behavior remains green, and the cluster package compiles with the new callback. -- [ ] **Step 8: Format and commit manager integration** +- [ ] **Step 15: Format and commit the watcher core and lifecycle** Run: @@ -637,12 +627,12 @@ Run: gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/gc/gc_state_manager_test.go server/cluster/cluster.go git diff --check git add pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/gc/gc_state_manager_test.go server/cluster/cluster.go -git commit -s -m "gc: tie watchers to local leadership" +git commit -s -m "gc: add GC state watcher lifecycle" ``` -### Task 3: Publish effective safe-point changes without blocking mutations +### Task 2: Publish and observe effective safe-point changes -This task attaches live publication exactly once to the successful shared mutation paths and proves that a full watcher queue affects only that watcher. +This task attaches live publication exactly once to the successful shared mutation paths, isolates slow consumers, and records the resulting watcher lifecycle transitions with bounded metrics. **Files:** @@ -650,13 +640,14 @@ This task attaches live publication exactly once to the successful shared mutati - Modify: `pkg/gc/gc_state_watcher_test.go` - Modify: `pkg/gc/gc_state_manager.go:350-413` - Modify: `pkg/gc/gc_state_manager.go:445-604` +- Modify: `pkg/gc/metrics.go` - Modify: `pkg/errs/errno.go:551-554` - Modify: `errors.toml` **Interfaces:** -- Consumes: Task 2's registry and termination helpers. -- Produces: `publishGCStateChangeLocked`, `errs.ErrGCStateWatcherSlowConsumer`, and complete live upserts used by the server stream. +- Consumes: Task 1's registry, watcher IDs, and termination helpers. +- Produces: `publishGCStateChangeLocked`, `errs.ErrGCStateWatcherSlowConsumer`, complete live upserts used by the server stream, `pd_gc_watcher_count`, `pd_gc_watcher_termination_total{reason=...}`, and `recordGCStateWatcherTerminationMetrics`. - [ ] **Step 1: Write failing publication tests for modern, compatible, and barrier paths** @@ -856,33 +847,22 @@ make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatch|Adv Expected: complete upserts arrive once, no-op and barrier-only calls emit nothing, and only the slow watcher terminates. -- [ ] **Step 8: Format and commit live publication** +- [ ] **Step 8: Format and inspect live publication** Run: ```bash gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/errs/errno.go git diff --check -git add pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/errs/errno.go errors.toml -git commit -s -m "gc: publish effective safe point changes" ``` -### Task 4: Instrument the watcher lifecycle - -This task adds low-cardinality metrics at the manager-owned registration and termination points, with pre-bound counter handles for every reason. - -**Files:** +Expected: formatting and whitespace checks pass before metrics are added. Do not commit yet because publication and its lifecycle accounting form one reviewable change. -- Modify: `pkg/gc/metrics.go` -- Modify: `pkg/gc/gc_state_watcher.go` -- Modify: `pkg/gc/gc_state_watcher_test.go` - -**Interfaces:** +#### Lifecycle metrics phase -- Consumes: Task 2's four termination-reason constants and the single idempotent termination helper. -- Produces: `pd_gc_watcher_count`, `pd_gc_watcher_termination_total{reason=...}`, and `recordGCStateWatcherTerminationMetrics`. +This phase adds low-cardinality metrics at the manager-owned registration and termination points, with pre-bound counter handles for every reason. It remains part of Task 2 because the metric increments must share the same idempotent publication and cleanup boundaries they describe. -- [ ] **Step 1: Write failing metric-delta tests** +- [ ] **Step 9: Write failing metric-delta tests** Use `prometheus/testutil.ToFloat64` and compare deltas so process-global counters do not make tests order-dependent: @@ -910,7 +890,7 @@ func (s *gcStateManagerTestSuite) TestGCStateWatcherMetrics() { In the same test, record the three remaining counters before their triggers. For `client_cancel`, register one watcher and call `Close`. For `slow_consumer`, register with live capacity 1 and perform two successful `AdvanceTxnSafePoint` calls without reading the watcher, so both publications run through the production path while `GCStateManager.mu` is held. For `init_error`, enable `iterateAllKeyspacesGCStatesError`, register with initial loading, and call `RecvBatch`. After each trigger, assert that only its expected counter increased by one and the active gauge returned to `activeBefore`; never mutate a metric directly. -- [ ] **Step 2: Run the metric test and confirm the red state** +- [ ] **Step 10: Run the metric test and confirm the red state** Run: @@ -920,7 +900,7 @@ make gotest GOTEST_ARGS='./pkg/gc -run TestGCStateManager/TestGCStateWatcherMetr Expected: compilation fails because the watcher metrics do not exist. -- [ ] **Step 3: Define and pre-bind the metrics** +- [ ] **Step 11: Define and pre-bind the metrics** Add these definitions to `pkg/gc/metrics.go`: @@ -946,7 +926,7 @@ gcStateWatcherTerminationInitErrorCounter = gcStateWatcherTerminationCounter.Wit Register the gauge and vector with `prometheus.MustRegister`. Do not call `WithLabelValues` in registration, publication, or cleanup paths. -- [ ] **Step 4: Record metrics at the single lifecycle boundaries** +- [ ] **Step 12: Record metrics at the single lifecycle boundaries** Increment the gauge only after successful insertion into the registry. In `terminateGCStateWatcherLocked`, decrement the gauge and invoke this switch only after deletion succeeds: @@ -969,7 +949,7 @@ func recordGCStateWatcherTerminationMetrics(reason gcStateWatcherTerminationReas No metric uses watcher IDs, keyspace IDs, client addresses, or error text as labels. The gauge has no labels, so teardown decrements it rather than deleting a label series. -- [ ] **Step 5: Run metric and lifecycle tests** +- [ ] **Step 13: Run metric and lifecycle tests** Run: @@ -979,18 +959,18 @@ make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatcherMe Expected: each lifecycle increments exactly one reason counter and returns the active gauge to its baseline. -- [ ] **Step 6: Format and commit observability** +- [ ] **Step 14: Format and commit publication and observability** Run: ```bash gofmt -w pkg/gc/metrics.go pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go git diff --check -git add pkg/gc/metrics.go pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go -git commit -s -m "gc: add watcher lifecycle metrics" +git add pkg/gc/metrics.go pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/errs/errno.go errors.toml +git commit -s -m "gc: publish and observe GC state changes" ``` -### Task 5: Implement the gRPC transport adapter and upgrade kvproto +### Task 3: Implement the gRPC transport adapter and upgrade kvproto This task adopts the merged protobuf API, converts domain changes, batches by exact wire size, and implements the local server-streaming handler through a small test seam. @@ -1009,7 +989,7 @@ This task adopts the merged protobuf API, converts domain changes, batches by ex **Interfaces:** -- Consumes: Task 2's `GCStateManager.WatchGCStates` and `GCStateWatcher` API; Task 3's slow-consumer sentinel and domain changes; kvproto `WatchGCStatesRequest`, `WatchGCStatesResponse`, and `GCStateChange`. +- Consumes: Task 1's `GCStateManager.WatchGCStates` and `GCStateWatcher` API; Task 2's slow-consumer sentinel and domain changes; kvproto `WatchGCStatesRequest`, `WatchGCStatesResponse`, and `GCStateChange`. - Produces: `GrpcServer.WatchGCStates`, `gcStateChangeToProto`, `splitWatchGCStatesResponses`, `watchGCStatesErrorToStatus`, and `serveWatchGCStates`. - [ ] **Step 1: Upgrade all four module scopes to the merged kvproto commit** @@ -1228,7 +1208,7 @@ git add go.mod go.sum client/go.mod client/go.sum tools/go.mod tools/go.sum test git commit -s -m "server: implement WatchGCStates stream" ``` -### Task 6: Prove RPC behavior in a real PD cluster +### Task 4: Prove RPC behavior in a real PD cluster This task covers the complete server stream, request preflight, lifetime rate limiting, and leader transfer through real generated gRPC clients. @@ -1240,7 +1220,7 @@ This task covers the complete server stream, request preflight, lifetime rate li **Interfaces:** -- Consumes: The generated `pdpb.PDClient.WatchGCStates` client and all production behavior from Tasks 1 through 5. +- Consumes: The generated `pdpb.PDClient.WatchGCStates` client and all production behavior from Tasks 1 through 3. - Produces: End-to-end evidence for initial loading, skip-initial registration, validation, rate-limit token lifetime, and leadership reconnection. - [ ] **Step 1: Add deterministic stream helpers** @@ -1336,13 +1316,13 @@ git commit -s -m "tests: cover WatchGCStates lifecycle" Omit unchanged production files from `git add`. If integration testing required no production correction, commit only `tests/server/gc/gc_test.go`. -### Task 7: Run final verification +## Final verification checklist -This task verifies formatting, generated error documentation, module consistency, race safety, focused behavior, and the repository's required checks before handoff. +This checklist verifies formatting, generated error documentation, module consistency, race safety, focused behavior, and the repository's required checks after all four implementation tasks are complete. It is a handoff gate rather than a separate implementation task or commit. **Files:** -- Verify: all files changed in Tasks 1 through 6 +- Verify: all files changed in Tasks 1 through 4 - Modify: none unless a verification command reports a concrete defect **Interfaces:** @@ -1424,7 +1404,7 @@ Expected: the `rg` command finds no new compatibility reference in the touched W ## Execution handoff -The plan is complete when this document is reviewed and committed. Execute it with one of the required workflows: +The plan is complete when this document is reviewed and committed. Execute its four implementation tasks with one of the required workflows: 1. **Subagent-driven:** Use `superpowers:subagent-driven-development`, dispatch a fresh worker for each task, and perform spec and code-quality review between tasks. 2. **Inline execution:** Use `superpowers:executing-plans`, execute tasks in batches, and stop at its review checkpoints. From 6dd7d28088021ef3324e9e9bc65668568f361074 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 18:21:57 +0800 Subject: [PATCH 06/25] gc: add GC state watcher lifecycle Signed-off-by: Wenxuan Zhang --- pkg/gc/gc_state_manager.go | 48 ++--- pkg/gc/gc_state_manager_test.go | 230 +++++++++++++++++++- pkg/gc/gc_state_watcher.go | 365 ++++++++++++++++++++++++++++++++ pkg/gc/gc_state_watcher_test.go | 129 +++++++++++ pkg/gc/metrics_test.go | 14 +- server/cluster/cluster.go | 3 +- 6 files changed, 750 insertions(+), 39 deletions(-) create mode 100644 pkg/gc/gc_state_watcher.go create mode 100644 pkg/gc/gc_state_watcher_test.go diff --git a/pkg/gc/gc_state_manager.go b/pkg/gc/gc_state_manager.go index aa25bbd9fda..7ce53f6f613 100644 --- a/pkg/gc/gc_state_manager.go +++ b/pkg/gc/gc_state_manager.go @@ -203,10 +203,10 @@ type GCStateManager struct { allKeyspacesGCStatesSingleFlight *syncutil.OrderedSingleFlight[map[uint32]GCState] allKeyspacesGCStatesExcludeGCBarriersSingleFlight *syncutil.OrderedSingleFlight[map[uint32]GCState] - // Note that nodeLeadership is a counter instead of a bool. Theoretically, it's possible that an - // OnNodeBecomesFollower invocation of the previous lease is later than the OnNodeBecomesLeader call of the new - // lease during PD leader changes. Making this a counter helps in guaranteeing the eventual consistency. - nodeLeadership atomic.Int32 + watchers map[uint64]*GCStateWatcher + nextWatcherID uint64 + nextLeadershipGeneration uint64 + activeLeadershipGeneration atomic.Uint64 } // NewGCStateManager creates a GCStateManager of GC and services. @@ -216,6 +216,7 @@ func NewGCStateManager(store endpoint.GCStateProvider, cfg config.PDServerConfig cfg: cfg, keyspaceManager: keyspaceManager, gcStateCache: newGCStateCache(), + watchers: make(map[uint64]*GCStateWatcher), allKeyspacesGCStatesSingleFlight: syncutil.NewOrderedSingleFlight[map[uint32]GCState](), allKeyspacesGCStatesExcludeGCBarriersSingleFlight: syncutil.NewOrderedSingleFlight[map[uint32]GCState](), } @@ -244,37 +245,34 @@ func getKeyspaceNameFromCtx(ctx context.Context) string { return "" } -// OnNodeBecomesLeader marks the current PD node as leader for GC state watches. -func (m *GCStateManager) OnNodeBecomesLeader() { +// OnNodeBecomesLeader starts a local leadership generation and returns its teardown function. +func (m *GCStateManager) OnNodeBecomesLeader() func() { m.mu.Lock() - defer m.mu.Unlock() - - m.nodeLeadership.Add(1) - - // Also trigger cache invalidation even when transitioning from follower to leader, as a protection against - // potential inconsistent cache state left from the last leadership. + m.nextLeadershipGeneration++ + generation := m.nextLeadershipGeneration + m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) + m.activeLeadershipGeneration.Store(generation) m.gcStateCache.clearAll() m.barrierMetrics.clearMetrics() productionBarrierMetrics.current.Store(m.barrierMetrics) -} - -// OnNodeBecomesFollower marks the current PD node as follower and closes all existing GC state watches. -func (m *GCStateManager) OnNodeBecomesFollower() { - m.mu.Lock() - defer m.mu.Unlock() - - m.nodeLeadership.Add(-1) + m.mu.Unlock() - // Invalidate the cache. - m.gcStateCache.clearAll() - m.barrierMetrics.clearMetrics() - if !m.nodeIsLeader() { + return func() { + m.mu.Lock() + defer m.mu.Unlock() + if m.activeLeadershipGeneration.Load() != generation { + return + } + m.activeLeadershipGeneration.Store(0) + m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) + m.gcStateCache.clearAll() + m.barrierMetrics.clearMetrics() productionBarrierMetrics.current.CompareAndSwap(m.barrierMetrics, nil) } } func (m *GCStateManager) nodeIsLeader() bool { - return m.nodeLeadership.Load() > 0 + return m.activeLeadershipGeneration.Load() != 0 } // redirectKeyspace checks the given keyspaceID, and returns the actual keyspaceID to operate on. diff --git a/pkg/gc/gc_state_manager_test.go b/pkg/gc/gc_state_manager_test.go index ddcdd6af714..457e2b3643f 100644 --- a/pkg/gc/gc_state_manager_test.go +++ b/pkg/gc/gc_state_manager_test.go @@ -210,7 +210,12 @@ func newGCStateManagerForTest(t testing.TB, opt newGCStateManagerForTestOptions) } } - gcStateManager.OnNodeBecomesLeader() + stopGCStateManager := gcStateManager.OnNodeBecomesLeader() + originalClean := clean + clean = func() { + stopGCStateManager() + originalClean() + } return s, s.GetGCStateProvider(), gcStateManager, clean, cancel } @@ -259,8 +264,220 @@ type gcStateCacheAccessCounterSnapshot struct { func (s *gcStateManagerTestSuite) ensureMarkedLeader() { if !s.manager.nodeIsLeader() { - s.manager.OnNodeBecomesLeader() + stopGCStateManager := s.manager.OnNodeBecomesLeader() + s.T().Cleanup(stopGCStateManager) + } +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchRequiresActiveLeadership() { + follower := NewGCStateManager(s.provider, s.manager.cfg, s.manager.keyspaceManager) + _, err := follower.WatchGCStates(context.Background(), true) + s.Require().ErrorIs(err, errs.ErrNotLeader) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchLeadershipGeneration() { + re := s.Require() + stopFirst := s.manager.OnNodeBecomesLeader() + first, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + + stopSecond := s.manager.OnNodeBecomesLeader() + re.ErrorIs(first.Err(), errs.ErrNotLeader) + second, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + + stopFirst() + re.NoError(second.Err()) + stopSecond() + re.ErrorIs(second.Err(), errs.ErrNotLeader) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchLoadsInitialStatesIncrementally() { + re := s.Require() + w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{ + initialBatchSize: 2, + initChannelCapacity: 1, + liveChannelCapacity: 1, + }) + re.NoError(err) + defer w.Close() + + want := make(map[uint32]struct{}, len(s.keyspacePresets.all)) + for _, keyspaceID := range s.keyspacePresets.all { + want[keyspaceID] = struct{}{} + } + got := make(map[uint32]struct{}, len(want)) + var batchSizes []int + for len(got) < len(want) { + changes, err := w.RecvBatch(2) + re.NoError(err) + batchSizes = append(batchSizes, len(changes)) + for _, change := range changes { + state := mustUpsert(s.T(), change) + re.Nil(state.GCBarriers) + got[state.KeyspaceID] = struct{}{} + } + } + re.Equal([]int{2, 2, 1}, batchSizes) + re.Equal(want, got) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchSkipsInitialLoading() { + re := s.Require() + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + re.True(w.initDone) + select { + case batch := <-w.initCh: + re.Fail("initial channel was used", "batch: %+v", batch) + default: + } +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchLiveSuppressesPausedInitial() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 10, time.Now()) + re.NoError(err) + + reached := make(chan struct{}) + release := make(chan struct{}) + var reachedOnce, releaseOnce sync.Once + releaseLoader := func() { releaseOnce.Do(func() { close(release) }) } + re.NoError(failpoint.EnableCall("github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded", func(id uint32) { + if id == keyspaceID { + reachedOnce.Do(func() { close(reached) }) + <-release + } + })) + defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded")) }() + defer releaseLoader() + + w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{initialBatchSize: 1, initChannelCapacity: 16, liveChannelCapacity: 4}) + re.NoError(err) + defer w.Close() + select { + case <-reached: + case <-time.After(5 * time.Second): + re.FailNow("initial loader did not reach keyspace 2") + } + + _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + // Task 2 connects successful state mutations to the live channel. Inject the + // corresponding live change directly here so this task remains scoped to the + // watcher merge and lifecycle. + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20}) + for { + changes, err := w.RecvBatch(1) + re.NoError(err) + state, ok := changes[0].Upsert() + if ok && state.KeyspaceID == keyspaceID && state.TxnSafePoint == 20 { + break + } } + releaseLoader() + + re.Eventually(func() bool { + for { + change, ok, err := w.receiveOne(false) + re.NoError(err) + if !ok { + return w.initDone + } + state, upsert := change.Upsert() + re.False(upsert && state.KeyspaceID == keyspaceID && state.TxnSafePoint == 10) + } + }, 5*time.Second, 10*time.Millisecond) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchInitialFailureTerminatesWatcher() { + re := s.Require() + const errorMessage = "injected initial watch failure" + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/gc/iterateAllKeyspacesGCStatesError", fmt.Sprintf(`return(%q)`, errorMessage))) + defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/iterateAllKeyspacesGCStatesError")) }() + + w, err := s.manager.WatchGCStates(context.Background(), false) + re.NoError(err) + _, err = w.RecvBatch(1) + re.ErrorContains(err, errorMessage) + re.Eventually(func() bool { + s.manager.mu.RLock() + defer s.manager.mu.RUnlock() + _, ok := s.manager.watchers[w.id] + return !ok + }, 5*time.Second, 10*time.Millisecond) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchFullInitChannelDoesNotHoldManagerMutex() { + re := s.Require() + stop := s.manager.OnNodeBecomesLeader() + w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{ + initialBatchSize: 1, + initChannelCapacity: 1, + liveChannelCapacity: 1, + }) + re.NoError(err) + re.Eventually(func() bool { return len(w.initCh) == cap(w.initCh) }, 5*time.Second, 10*time.Millisecond) + + mutationDone := make(chan error, 1) + go func() { + _, err := s.manager.AdvanceTxnSafePoint(2, 1, time.Now()) + mutationDone <- err + }() + select { + case err := <-mutationDone: + re.NoError(err) + case <-time.After(5 * time.Second): + re.FailNow("manager mutation blocked behind the initial state loader") + } + + teardownDone := make(chan struct{}) + go func() { + stop() + close(teardownDone) + }() + select { + case <-teardownDone: + case <-time.After(5 * time.Second): + re.FailNow("leadership teardown blocked behind the initial state loader") + } + w.Close() + re.Eventually(func() bool { + s.manager.mu.RLock() + defer s.manager.mu.RUnlock() + _, ok := s.manager.watchers[w.id] + return !ok + }, 5*time.Second, 10*time.Millisecond) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchConcurrentCloseIsIdempotent() { + re := s.Require() + stop := s.manager.OnNodeBecomesLeader() + w, err := s.manager.WatchGCStates(context.Background(), false) + re.NoError(err) + + var wg sync.WaitGroup + wg.Add(3) + go func() { + defer wg.Done() + w.Close() + }() + go func() { + defer wg.Done() + s.manager.terminateGCStateWatcher(w, errors.New("initial load failed"), watcherTerminationInitError) + }() + go func() { + defer wg.Done() + stop() + }() + wg.Wait() + re.Error(w.Err()) + s.manager.mu.RLock() + _, ok := s.manager.watchers[w.id] + s.manager.mu.RUnlock() + re.False(ok) } func (s *gcStateManagerTestSuite) trackGCStateCacheAccessCounters() *gcStateCacheAccessCounters { @@ -571,9 +788,9 @@ func (s *gcStateManagerTestSuite) TestCompatibleUpdateGCSafePointSequentiallyWit return wb.SetGCSafePoint(keyspaceID, 101) }) re.NoError(err) - oldLeadership := s.manager.nodeLeadership.Load() - s.manager.nodeLeadership.Store(0) - defer s.manager.nodeLeadership.Store(oldLeadership) + oldLeadership := s.manager.activeLeadershipGeneration.Load() + s.manager.activeLeadershipGeneration.Store(0) + defer s.manager.activeLeadershipGeneration.Store(oldLeadership) gcSafePoint, err = s.manager.CompatibleLoadGCSafePoint(keyspaceID) re.NoError(err) @@ -2078,7 +2295,8 @@ func (s *gcStateManagerTestSuite) TestGetGCStateWithGlobalGCBarriersRejectsRevis s.manager.keyspaceManager, ) s.T().Cleanup(otherManager.CloseBarrierMetrics) - otherManager.OnNodeBecomesLeader() + stopOtherManager := otherManager.OnNodeBecomesLeader() + defer stopOtherManager() _, err = otherManager.SetGlobalGCBarrier( ctx, "snapshot", diff --git a/pkg/gc/gc_state_watcher.go b/pkg/gc/gc_state_watcher.go new file mode 100644 index 00000000000..9696ba80b03 --- /dev/null +++ b/pkg/gc/gc_state_watcher.go @@ -0,0 +1,365 @@ +// Copyright 2026 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package gc + +import ( + "context" + + "github.com/pingcap/errors" + "github.com/pingcap/failpoint" + + "github.com/tikv/pd/pkg/errs" +) + +type gcStateChangeKind uint8 + +const ( + gcStateChangeUnknown gcStateChangeKind = iota + gcStateChangeUpsert + gcStateChangeRemoved +) + +// GCStateChange describes one effective GC state change for a keyspace scope. +type GCStateChange struct { + kind gcStateChangeKind + upsert GCState + removedKeyspaceID uint32 +} + +// NewGCStateUpsert creates a change containing the complete effective GC state. +func NewGCStateUpsert(state GCState) GCStateChange { + state.GCBarriers = nil + return GCStateChange{kind: gcStateChangeUpsert, upsert: state} +} + +// NewGCStateRemoved creates a change that removes a keyspace scope. +func NewGCStateRemoved(keyspaceID uint32) GCStateChange { + return GCStateChange{kind: gcStateChangeRemoved, removedKeyspaceID: keyspaceID} +} + +// Upsert returns the effective GC state when the change is an upsert. +func (c GCStateChange) Upsert() (GCState, bool) { + return c.upsert, c.kind == gcStateChangeUpsert +} + +// RemovedKeyspaceID returns the removed keyspace ID when the change is a removal. +func (c GCStateChange) RemovedKeyspaceID() (uint32, bool) { + return c.removedKeyspaceID, c.kind == gcStateChangeRemoved +} + +// KeyspaceID returns the keyspace scope changed by this value. +func (c GCStateChange) KeyspaceID() (uint32, bool) { + if state, ok := c.Upsert(); ok { + return state.KeyspaceID, true + } + return c.RemovedKeyspaceID() +} + +const ( + defaultGCStateWatchInitialBatchSize = 1024 + defaultGCStateWatchInitChannelCapacity = 1 + defaultGCStateWatchLiveChannelCapacity = 1024 +) + +type gcStateWatchConfig struct { + initialBatchSize int + initChannelCapacity int + liveChannelCapacity int +} + +type gcStateWatcherTerminationReason string + +const ( + watcherTerminationClientCancel gcStateWatcherTerminationReason = "client_cancel" + watcherTerminationLeaderLost gcStateWatcherTerminationReason = "leader_lost" + watcherTerminationSlowConsumer gcStateWatcherTerminationReason = "slow_consumer" + watcherTerminationInitError gcStateWatcherTerminationReason = "init_error" +) + +// GCStateWatcher receives a consistent initial view followed by live GC state changes. +// +// A watcher supports one receiving goroutine. Close may be called concurrently with +// receiving and with manager-owned lifecycle operations. +type GCStateWatcher struct { + ctx context.Context + cancel context.CancelCauseFunc + manager *GCStateManager + id uint64 + initCh chan []GCStateChange + liveCh chan GCStateChange + initDone bool + pendingInit []GCStateChange + dirtyDuringInit map[uint32]struct{} +} + +func newGCStateWatcher(parent context.Context, cfg gcStateWatchConfig, skipLoadingInitial bool) *GCStateWatcher { + ctx, cancel := context.WithCancelCause(parent) + watcher := &GCStateWatcher{ + ctx: ctx, + cancel: cancel, + initCh: make(chan []GCStateChange, cfg.initChannelCapacity), + liveCh: make(chan GCStateChange, cfg.liveChannelCapacity), + initDone: skipLoadingInitial, + } + if !skipLoadingInitial { + watcher.dirtyDuringInit = make(map[uint32]struct{}) + } + return watcher +} + +func (w *GCStateWatcher) receiveOne(block bool) (GCStateChange, bool, error) { + for { + if err := w.Err(); err != nil { + return GCStateChange{}, false, err + } + + for len(w.pendingInit) > 0 { + change := w.pendingInit[0] + w.pendingInit = w.pendingInit[1:] + keyspaceID, ok := change.KeyspaceID() + if ok { + if _, dirty := w.dirtyDuringInit[keyspaceID]; dirty { + // For each scope, consumers observe either initial v1 followed by live v2, + // or live v2 with the later-arriving initial v1 suppressed. + continue + } + } + return change, true, nil + } + w.pendingInit = nil + + if w.initDone { + if block { + select { + case <-w.ctx.Done(): + return GCStateChange{}, false, w.Err() + case change := <-w.liveCh: + return change, true, nil + } + } + select { + case <-w.ctx.Done(): + return GCStateChange{}, false, w.Err() + case change := <-w.liveCh: + return change, true, nil + default: + return GCStateChange{}, false, nil + } + } + + var ( + change GCStateChange + batch []GCStateChange + ok bool + ) + if block { + select { + case <-w.ctx.Done(): + return GCStateChange{}, false, w.Err() + case change = <-w.liveCh: + if keyspaceID, valid := change.KeyspaceID(); valid { + // Marking live scopes dirty preserves the alternate v2-only order when + // initial v1 has not yet been delivered. + w.dirtyDuringInit[keyspaceID] = struct{}{} + } + return change, true, nil + case batch, ok = <-w.initCh: + } + } else { + select { + case <-w.ctx.Done(): + return GCStateChange{}, false, w.Err() + case change = <-w.liveCh: + if keyspaceID, valid := change.KeyspaceID(); valid { + w.dirtyDuringInit[keyspaceID] = struct{}{} + } + return change, true, nil + case batch, ok = <-w.initCh: + default: + return GCStateChange{}, false, nil + } + } + + if !ok { + w.initCh = nil + w.initDone = true + w.dirtyDuringInit = nil + continue + } + w.pendingInit = append([]GCStateChange(nil), batch...) + } +} + +// Err returns the first cause that terminated the watcher. +func (w *GCStateWatcher) Err() error { + return context.Cause(w.ctx) +} + +// RecvBatch waits for one visible change and opportunistically collects up to maxChanges. +func (w *GCStateWatcher) RecvBatch(maxChanges int) ([]GCStateChange, error) { + if maxChanges <= 0 { + panic("GCStateWatcher.RecvBatch requires a positive maximum") + } + first, ok, err := w.receiveOne(true) + if err != nil { + return nil, err + } + if !ok { + panic("blocking watcher receive returned no result") + } + result := []GCStateChange{first} + for len(result) < maxChanges { + change, ok, err := w.receiveOne(false) + if err != nil { + return nil, err + } + if !ok { + break + } + result = append(result, change) + } + if err := w.Err(); err != nil { + return nil, err + } + return result, nil +} + +// Close stops the watcher and removes it from its manager. +func (w *GCStateWatcher) Close() { + if w.manager == nil { + w.cancel(context.Canceled) + return + } + w.manager.terminateGCStateWatcher(w, context.Canceled, watcherTerminationClientCancel) +} + +// WatchGCStates registers a watcher in the current local leadership generation. +func (m *GCStateManager) WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error) { + return m.watchGCStates(ctx, skipLoadingInitial, gcStateWatchConfig{ + initialBatchSize: defaultGCStateWatchInitialBatchSize, + initChannelCapacity: defaultGCStateWatchInitChannelCapacity, + liveChannelCapacity: defaultGCStateWatchLiveChannelCapacity, + }) +} + +func (m *GCStateManager) watchGCStates( + ctx context.Context, + skipLoadingInitial bool, + cfg gcStateWatchConfig, +) (*GCStateWatcher, error) { + watcher := newGCStateWatcher(ctx, cfg, skipLoadingInitial) + + m.mu.Lock() + if m.activeLeadershipGeneration.Load() == 0 { + m.mu.Unlock() + watcher.cancel(errs.ErrNotLeader) + return nil, errs.ErrNotLeader + } + m.nextWatcherID++ + watcher.manager = m + watcher.id = m.nextWatcherID + m.watchers[watcher.id] = watcher + m.mu.Unlock() + + failpoint.InjectCall("watchGCStatesRegistered") + if !skipLoadingInitial { + go m.loadInitialGCStates(watcher, cfg.initialBatchSize) + } + return watcher, nil +} + +func (m *GCStateManager) loadInitialGCStates(watcher *GCStateWatcher, batchSize int) { + batch := make([]GCStateChange, 0, batchSize) + stopped := false + flush := func() bool { + if len(batch) == 0 { + return true + } + ready := batch + batch = make([]GCStateChange, 0, batchSize) + select { + case watcher.initCh <- ready: + return true + case <-watcher.ctx.Done(): + return false + } + } + + err := m.iterateAllKeyspacesGCStates( + watcher.ctx, + true, + func(uint32) bool { return true }, + func(state GCState) { + if stopped { + return + } + failpoint.InjectCall("watchGCStatesInitialStateLoaded", state.KeyspaceID) + if watcher.Err() != nil { + stopped = true + return + } + batch = append(batch, NewGCStateUpsert(state)) + if len(batch) == batchSize { + stopped = !flush() + } + }, + nil, + ) + + if stopped || watcher.Err() != nil { + return + } + if err != nil { + m.terminateGCStateWatcher(watcher, errors.Annotate(err, "load initial GC states"), watcherTerminationInitError) + return + } + if !flush() { + return + } + close(watcher.initCh) +} + +func (m *GCStateManager) terminateGCStateWatcher( + watcher *GCStateWatcher, + cause error, + reason gcStateWatcherTerminationReason, +) { + m.mu.Lock() + defer m.mu.Unlock() + m.terminateGCStateWatcherLocked(watcher, cause, reason) +} + +func (m *GCStateManager) terminateGCStateWatcherLocked( + watcher *GCStateWatcher, + cause error, + _ gcStateWatcherTerminationReason, +) bool { + registered, ok := m.watchers[watcher.id] + if !ok || registered != watcher { + return false + } + delete(m.watchers, watcher.id) + watcher.cancel(cause) + return true +} + +func (m *GCStateManager) terminateAllGCStateWatchersLocked( + cause error, + reason gcStateWatcherTerminationReason, +) { + for _, watcher := range m.watchers { + m.terminateGCStateWatcherLocked(watcher, cause, reason) + } +} diff --git a/pkg/gc/gc_state_watcher_test.go b/pkg/gc/gc_state_watcher_test.go new file mode 100644 index 00000000000..e8d413199b8 --- /dev/null +++ b/pkg/gc/gc_state_watcher_test.go @@ -0,0 +1,129 @@ +// Copyright 2026 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package gc + +import ( + "context" + "errors" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestGCStateWatcherInitialThenLive(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 1})} + + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint64(1), mustUpsert(t, got[0]).TxnSafePoint) + + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 2}) + got, err = w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint64(2), mustUpsert(t, got[0]).TxnSafePoint) +} + +func TestGCStateWatcherLiveSuppressesOlderInitial(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 2}) + + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint64(2), mustUpsert(t, got[0]).TxnSafePoint) + + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 1})} + close(w.initCh) + _, ok, err := w.receiveOne(false) + require.NoError(t, err) + require.False(t, ok) + require.True(t, w.initDone) +} + +func TestGCStateWatcherRemovedSuppressesInitial(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + w.liveCh <- NewGCStateRemoved(7) + got, err := w.RecvBatch(1) + require.NoError(t, err) + removed, ok := got[0].RemovedKeyspaceID() + require.True(t, ok) + require.Equal(t, uint32(7), removed) + + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7})} + close(w.initCh) + _, ok, err = w.receiveOne(false) + require.NoError(t, err) + require.False(t, ok) + require.True(t, w.initDone) +} + +func TestGCStateWatcherDrainsBufferedInitBeforeReleasingDirtySet(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 1}, false) + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7}) + _, err := w.RecvBatch(1) + require.NoError(t, err) + w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 8})} + close(w.initCh) + + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, uint32(8), mustUpsert(t, got[0]).KeyspaceID) + require.False(t, w.initDone) + require.NotNil(t, w.dirtyDuringInit) + + _, ok, err := w.receiveOne(false) + require.NoError(t, err) + require.False(t, ok) + require.True(t, w.initDone) + require.Nil(t, w.initCh) + require.Nil(t, w.dirtyDuringInit) +} + +func TestGCStateWatcherRecvBatchHonorsMaximum(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 3}, true) + for id := uint32(1); id <= 3; id++ { + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: id}) + } + got, err := w.RecvBatch(2) + require.NoError(t, err) + require.Len(t, got, 2) + got, err = w.RecvBatch(2) + require.NoError(t, err) + require.Len(t, got, 1) +} + +func TestGCStateWatcherCancellationDiscardsBufferedWork(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 1}, true) + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7}) + want := errors.New("watch terminated") + w.cancel(want) + got, err := w.RecvBatch(1) + require.ErrorIs(t, err, want) + require.Nil(t, got) +} + +func TestGCStateWatcherFirstCancellationCauseWins(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 1}, true) + first := errors.New("first") + w.cancel(first) + w.cancel(errors.New("second")) + require.ErrorIs(t, w.Err(), first) +} + +func mustUpsert(t testing.TB, change GCStateChange) GCState { + state, ok := change.Upsert() + require.True(t, ok) + return state +} diff --git a/pkg/gc/metrics_test.go b/pkg/gc/metrics_test.go index b218ab47d7b..012cb6d7bf0 100644 --- a/pkg/gc/metrics_test.go +++ b/pkg/gc/metrics_test.go @@ -286,22 +286,23 @@ func TestBarrierMetricsRegistrationAndLeadership(t *testing.T) { observe := func(m *GCStateManager, id uint32) []barrierWarning { return m.barrierMetrics.observeMetrics(m.barrierMetrics.generation(), id, "tenant", barriers, nil, now) } - first.OnNodeBecomesLeader() + stopFirst := first.OnNodeBecomesLeader() require.Len(t, observe(first, 42), 1) require.Contains(t, gatherBarrierMetrics(t, prometheus.DefaultGatherer), "keyspace/42/old") generation := first.barrierMetrics.generation() - first.OnNodeBecomesLeader() + stopReplacement := first.OnNodeBecomesLeader() + defer stopReplacement() require.Empty(t, gatherBarrierMetrics(t, prometheus.DefaultGatherer)) require.Empty(t, first.barrierMetrics.observeMetrics(generation, 42, "tenant-a", barriers, nil, now)) require.Empty(t, gatherBarrierMetrics(t, prometheus.DefaultGatherer), "stale pre-leadership read cannot publish") - first.OnNodeBecomesFollower() // Previous lease ends, one leadership remains. + stopFirst() // Previous lease ends, the replacement leadership remains. require.Len(t, observe(first, 42), 1) require.Contains(t, gatherBarrierMetrics(t, prometheus.DefaultGatherer), "keyspace/42/old") - second.OnNodeBecomesLeader() + stopSecond := second.OnNodeBecomesLeader() require.Len(t, observe(second, 43), 1) first.CloseBarrierMetrics() require.Equal(t, map[string]float64{"keyspace/43/old": 1_999_712_000}, gatherBarrierMetrics(t, prometheus.DefaultGatherer), "old owner cleanup must preserve replacement") - second.OnNodeBecomesFollower() + stopSecond() require.Empty(t, gatherBarrierMetrics(t, prometheus.DefaultGatherer)) require.Nil(t, productionBarrierMetrics.current.Load(), "registry must not retain a stopped manager") } @@ -811,6 +812,7 @@ func (s *gcStateManagerTestSuite) TestBarrierMetricsRemovalFencesInflightPublica re := s.Require() now := time.Unix(2_000_000_000, 0) m := s.manager + stop := m.OnNodeBecomesLeader() m.barrierMetrics.now = func() time.Time { return now } registry := prometheus.NewRegistry() registry.MustRegister(m.barrierMetrics) @@ -829,7 +831,7 @@ func (s *gcStateManagerTestSuite) TestBarrierMetricsRemovalFencesInflightPublica advance() re.Equal(expected, gatherBarrierMetrics(s.T(), registry), "every accepted metadata state remains observable") } - m.OnNodeBecomesFollower() + stop() re.Empty(gatherBarrierMetrics(s.T(), registry)) m.OnNodeBecomesLeader() re.Empty(gatherBarrierMetrics(s.T(), registry)) diff --git a/server/cluster/cluster.go b/server/cluster/cluster.go index a37c31a2c60..0192faabd62 100644 --- a/server/cluster/cluster.go +++ b/server/cluster/cluster.go @@ -492,8 +492,7 @@ func (c *RaftCluster) Start(s Server, bootstrap bool) (err error) { go c.startProgressGC() go c.runStorageSizeCollector(s.GetMeteringWriter(), c.regionLabeler, s.GetKeyspaceManager()) - s.GetGCStateManager().OnNodeBecomesLeader() - c.stopGCStateManager = s.GetGCStateManager().OnNodeBecomesFollower + c.stopGCStateManager = s.GetGCStateManager().OnNodeBecomesLeader() log.Info("start background jobs completed", zap.Duration("cost", time.Since(backgroundJobsStart))) runnersStart := time.Now() From 9a5d6458fa634f4b16bcda4d870161afa1a6fb41 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 18:34:42 +0800 Subject: [PATCH 07/25] gc: publish and observe GC state changes Signed-off-by: Wenxuan Zhang --- errors.toml | 5 + pkg/errs/errno.go | 1 + pkg/gc/gc_state_manager.go | 16 +++ pkg/gc/gc_state_manager_test.go | 4 - pkg/gc/gc_state_watcher.go | 37 +++++- pkg/gc/gc_state_watcher_test.go | 223 ++++++++++++++++++++++++++++++++ pkg/gc/metrics.go | 20 +++ 7 files changed, 301 insertions(+), 5 deletions(-) diff --git a/errors.toml b/errors.toml index 33acad6fbfb..4787c7f86fa 100644 --- a/errors.toml +++ b/errors.toml @@ -481,6 +481,11 @@ error = ''' trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v ''' +["PD:gc:ErrGCStateWatcherSlowConsumer"] +error = ''' +GC state watcher is too slow +''' + ["PD:gc:ErrGlobalGCBarrierTSBehindTxnSafePoint"] error = ''' trying to set a global GC barrier on ts %d which is already behind the txn safe point %d of keyspace %s diff --git a/pkg/errs/errno.go b/pkg/errs/errno.go index 5465e5dcf66..91ab73b924e 100644 --- a/pkg/errs/errno.go +++ b/pkg/errs/errno.go @@ -552,6 +552,7 @@ var ( ErrDecreasingGCSafePoint = errors.Normalize("trying to update GC safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingGCSafePoint")) ErrGCSafePointExceedsTxnSafePoint = errors.Normalize("trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v", errors.RFCCodeText("PD:gc:ErrGCSafePointExceedsTxnSafePoint")) ErrDecreasingTxnSafePoint = errors.Normalize("trying to update txn safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingTxnSafePoint")) + ErrGCStateWatcherSlowConsumer = errors.Normalize("GC state watcher is too slow", errors.RFCCodeText("PD:gc:ErrGCStateWatcherSlowConsumer")) ErrGCBarrierTSBehindTxnSafePoint = errors.Normalize("trying to set a GC barrier on ts %d which is already behind the txn safe point %d", errors.RFCCodeText("PD:gc:ErrGCBarrierTSBehindTxnSafePoint")) ErrReservedGCBarrierID = errors.Normalize("trying to set a GC barrier with a barrier ID that is reserved: %v", errors.RFCCodeText("PD:gc:ErrReservedGCBarrierID")) ErrGlobalGCBarrierTSBehindTxnSafePoint = errors.Normalize("trying to set a global GC barrier on ts %d which is already behind the txn safe point %d of keyspace %s", errors.RFCCodeText("PD:gc:ErrGlobalGCBarrierTSBehindTxnSafePoint")) diff --git a/pkg/gc/gc_state_manager.go b/pkg/gc/gc_state_manager.go index 7ce53f6f613..7382c6b665f 100644 --- a/pkg/gc/gc_state_manager.go +++ b/pkg/gc/gc_state_manager.go @@ -404,6 +404,14 @@ func (m *GCStateManager) advanceGCSafePointImpl(ctx context.Context, keyspaceID TxnSafePoint: txnSafePoint, GCSafePoint: newGCSafePoint, }) + if newGCSafePoint != oldGCSafePoint { + m.publishGCStateChangeLocked(NewGCStateUpsert(GCState{ + KeyspaceID: keyspaceID, + IsKeyspaceLevel: keyspaceID != constant.NullKeyspaceID, + TxnSafePoint: txnSafePoint, + GCSafePoint: newGCSafePoint, + })) + } if newGCSafePoint != oldGCSafePoint { log.Info("advanced GC safe point", @@ -591,6 +599,14 @@ func (m *GCStateManager) advanceTxnSafePointImpl(ctx context.Context, keyspaceID TxnSafePoint: newTxnSafePoint, GCSafePoint: gcSafePoint, }) + if newTxnSafePoint != oldTxnSafePoint { + m.publishGCStateChangeLocked(NewGCStateUpsert(GCState{ + KeyspaceID: keyspaceID, + IsKeyspaceLevel: keyspaceID != constant.NullKeyspaceID, + TxnSafePoint: newTxnSafePoint, + GCSafePoint: gcSafePoint, + })) + } blockerDesc := "" simulatedServiceID := "" diff --git a/pkg/gc/gc_state_manager_test.go b/pkg/gc/gc_state_manager_test.go index 457e2b3643f..2f84e454732 100644 --- a/pkg/gc/gc_state_manager_test.go +++ b/pkg/gc/gc_state_manager_test.go @@ -365,10 +365,6 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchLiveSuppressesPausedInitial() _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) re.NoError(err) - // Task 2 connects successful state mutations to the live channel. Inject the - // corresponding live change directly here so this task remains scoped to the - // watcher merge and lifecycle. - w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20}) for { changes, err := w.RecvBatch(1) re.NoError(err) diff --git a/pkg/gc/gc_state_watcher.go b/pkg/gc/gc_state_watcher.go index 9696ba80b03..10f24dbe626 100644 --- a/pkg/gc/gc_state_watcher.go +++ b/pkg/gc/gc_state_watcher.go @@ -19,6 +19,8 @@ import ( "github.com/pingcap/errors" "github.com/pingcap/failpoint" + "github.com/pingcap/log" + "go.uber.org/zap" "github.com/tikv/pd/pkg/errs" ) @@ -271,6 +273,7 @@ func (m *GCStateManager) watchGCStates( watcher.manager = m watcher.id = m.nextWatcherID m.watchers[watcher.id] = watcher + gcStateWatcherGauge.Inc() m.mu.Unlock() failpoint.InjectCall("watchGCStatesRegistered") @@ -341,20 +344,52 @@ func (m *GCStateManager) terminateGCStateWatcher( m.terminateGCStateWatcherLocked(watcher, cause, reason) } +func (m *GCStateManager) publishGCStateChangeLocked(change GCStateChange) { + for _, watcher := range m.watchers { + select { + case watcher.liveCh <- change: + default: + log.Warn("GC state watcher is too slow", + zap.Uint64("watcher-id", watcher.id), + zap.Int("capacity", cap(watcher.liveCh)), + zap.Int("queue-length", len(watcher.liveCh))) + m.terminateGCStateWatcherLocked(watcher, errs.ErrGCStateWatcherSlowConsumer, watcherTerminationSlowConsumer) + } + } + // TODO: Publish keyspace metadata upserts and removals through this same serialized path when an authoritative GC-leader-owned lifecycle hook exists. +} + func (m *GCStateManager) terminateGCStateWatcherLocked( watcher *GCStateWatcher, cause error, - _ gcStateWatcherTerminationReason, + reason gcStateWatcherTerminationReason, ) bool { registered, ok := m.watchers[watcher.id] if !ok || registered != watcher { return false } delete(m.watchers, watcher.id) + gcStateWatcherGauge.Dec() + recordGCStateWatcherTerminationMetrics(reason) watcher.cancel(cause) return true } +func recordGCStateWatcherTerminationMetrics(reason gcStateWatcherTerminationReason) { + switch reason { + case watcherTerminationClientCancel: + gcStateWatcherTerminationClientCancelCounter.Inc() + case watcherTerminationLeaderLost: + gcStateWatcherTerminationLeaderLostCounter.Inc() + case watcherTerminationSlowConsumer: + gcStateWatcherTerminationSlowConsumerCounter.Inc() + case watcherTerminationInitError: + gcStateWatcherTerminationInitErrorCounter.Inc() + default: + panic("unknown GC state watcher termination reason") + } +} + func (m *GCStateManager) terminateAllGCStateWatchersLocked( cause error, reason gcStateWatcherTerminationReason, diff --git a/pkg/gc/gc_state_watcher_test.go b/pkg/gc/gc_state_watcher_test.go index e8d413199b8..438e6083d63 100644 --- a/pkg/gc/gc_state_watcher_test.go +++ b/pkg/gc/gc_state_watcher_test.go @@ -17,9 +17,17 @@ package gc import ( "context" "errors" + "fmt" + "math" "testing" + "time" + "github.com/pingcap/failpoint" + promtestutil "github.com/prometheus/client_golang/prometheus/testutil" "github.com/stretchr/testify/require" + + "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/utils/keypath" ) func TestGCStateWatcherInitialThenLive(t *testing.T) { @@ -122,6 +130,221 @@ func TestGCStateWatcherFirstCancellationCauseWins(t *testing.T) { require.ErrorIs(t, w.Err(), first) } +func (s *gcStateManagerTestSuite) TestGCStateWatchPublishesAdvanceGCSafePoint() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 10) + re.NoError(err) + changes, err := w.RecvBatch(1) + re.NoError(err) + state := mustUpsert(s.T(), changes[0]) + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20, GCSafePoint: 10}, state) + re.Empty(state.GCBarriers) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchPublishesAdvanceTxnSafePoint() { + re := s.Require() + const keyspaceID = uint32(2) + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + changes, err := w.RecvBatch(1) + re.NoError(err) + state := mustUpsert(s.T(), changes[0]) + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20}, state) + re.Empty(state.GCBarriers) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchCompatiblePathsPublishOnce() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 30, time.Now()) + re.NoError(err) + + gcWatcher, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + _, _, err = s.manager.CompatibleUpdateGCSafePoint(keyspaceID, 10) + re.NoError(err) + changes, err := gcWatcher.RecvBatch(1) + re.NoError(err) + re.Len(changes, 1) + state := mustUpsert(s.T(), changes[0]) + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 30, GCSafePoint: 10}, state) + re.Empty(state.GCBarriers) + re.Empty(gcWatcher.liveCh) + gcWatcher.Close() + + txnWatcher, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + _, _, err = s.manager.CompatibleUpdateServiceGCSafePoint(keyspaceID, keypath.GCWorkerServiceSafePointID, 40, math.MaxInt64, time.Now()) + re.NoError(err) + changes, err = txnWatcher.RecvBatch(1) + re.NoError(err) + re.Len(changes, 1) + state = mustUpsert(s.T(), changes[0]) + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 40, GCSafePoint: 10}, state) + re.Empty(state.GCBarriers) + re.Empty(txnWatcher.liveCh) + txnWatcher.Close() +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchDoesNotPublishNoOpOrFailure() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 10) + re.NoError(err) + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + _, _, err = s.manager.CompatibleUpdateGCSafePoint(keyspaceID, 10) + re.NoError(err) + _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 9) + re.ErrorIs(err, errs.ErrDecreasingGCSafePoint) + re.Empty(w.liveCh) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchDoesNotPublishBarrierOnlyChanges() { + re := s.Require() + const keyspaceID = uint32(2) + _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + defer w.Close() + + _, err = s.manager.SetGCBarrier(keyspaceID, "backup", 30, time.Hour, time.Now()) + re.NoError(err) + _, err = s.manager.DeleteGCBarrier(keyspaceID, "backup") + re.NoError(err) + re.Empty(w.liveCh) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatchSlowConsumerIsolation() { + re := s.Require() + const keyspaceID = uint32(2) + watcherA, err := s.manager.watchGCStates(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 1}) + re.NoError(err) + defer watcherA.Close() + watcherB, err := s.manager.watchGCStates(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 4}) + re.NoError(err) + defer watcherB.Close() + + _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 10, time.Now()) + re.NoError(err) + changes, err := watcherB.RecvBatch(1) + re.NoError(err) + state := mustUpsert(s.T(), changes[0]) + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 10}, state) + re.Empty(state.GCBarriers) + + _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) + re.NoError(err) + changes, err = watcherB.RecvBatch(1) + re.NoError(err) + state = mustUpsert(s.T(), changes[0]) + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20}, state) + re.Empty(state.GCBarriers) + + re.ErrorIs(watcherA.Err(), errs.ErrGCStateWatcherSlowConsumer) + re.NoError(watcherB.Err()) + re.NotContains(s.manager.watchers, watcherA.id) + re.Contains(s.manager.watchers, watcherB.id) + + reconnected, err := s.manager.WatchGCStates(context.Background(), false) + re.NoError(err) + defer reconnected.Close() + for { + changes, err = reconnected.RecvBatch(1) + re.NoError(err) + state = mustUpsert(s.T(), changes[0]) + if state.KeyspaceID == keyspaceID { + break + } + } + re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20}, state) + re.Empty(state.GCBarriers) +} + +func (s *gcStateManagerTestSuite) TestGCStateWatcherMetrics() { + re := s.Require() + activeBefore := promtestutil.ToFloat64(gcStateWatcherGauge) + clientCancelBefore := promtestutil.ToFloat64(gcStateWatcherTerminationClientCancelCounter) + leaderLostBefore := promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter) + slowConsumerBefore := promtestutil.ToFloat64(gcStateWatcherTerminationSlowConsumerCounter) + initErrorBefore := promtestutil.ToFloat64(gcStateWatcherTerminationInitErrorCounter) + assertTerminationDeltas := func(clientCancel, leaderLost, slowConsumer, initError float64) { + re.Equal(clientCancelBefore+clientCancel, promtestutil.ToFloat64(gcStateWatcherTerminationClientCancelCounter)) + re.Equal(leaderLostBefore+leaderLost, promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter)) + re.Equal(slowConsumerBefore+slowConsumer, promtestutil.ToFloat64(gcStateWatcherTerminationSlowConsumerCounter)) + re.Equal(initErrorBefore+initError, promtestutil.ToFloat64(gcStateWatcherTerminationInitErrorCounter)) + } + + stop := s.manager.OnNodeBecomesLeader() + w, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + re.Equal(activeBefore+1, promtestutil.ToFloat64(gcStateWatcherGauge)) + + stop() + re.ErrorIs(w.Err(), errs.ErrNotLeader) + re.Equal(activeBefore, promtestutil.ToFloat64(gcStateWatcherGauge)) + assertTerminationDeltas(0, 1, 0, 0) + w.Close() + assertTerminationDeltas(0, 1, 0, 0) + + stopRemainingCases := s.manager.OnNodeBecomesLeader() + defer stopRemainingCases() + + clientCanceled, err := s.manager.WatchGCStates(context.Background(), true) + re.NoError(err) + re.Equal(activeBefore+1, promtestutil.ToFloat64(gcStateWatcherGauge)) + clientCanceled.Close() + clientCanceled.Close() + re.Equal(activeBefore, promtestutil.ToFloat64(gcStateWatcherGauge)) + assertTerminationDeltas(1, 1, 0, 0) + + slowConsumer, err := s.manager.watchGCStates(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 1}) + re.NoError(err) + re.Equal(activeBefore+1, promtestutil.ToFloat64(gcStateWatcherGauge)) + _, err = s.manager.AdvanceTxnSafePoint(2, 10, time.Now()) + re.NoError(err) + re.Equal(activeBefore+1, promtestutil.ToFloat64(gcStateWatcherGauge)) + _, err = s.manager.AdvanceTxnSafePoint(2, 20, time.Now()) + re.NoError(err) + re.ErrorIs(slowConsumer.Err(), errs.ErrGCStateWatcherSlowConsumer) + re.Equal(activeBefore, promtestutil.ToFloat64(gcStateWatcherGauge)) + assertTerminationDeltas(1, 1, 1, 0) + slowConsumer.Close() + assertTerminationDeltas(1, 1, 1, 0) + + const errorMessage = "injected initial watch failure" + func() { + re.NoError(failpoint.Enable("github.com/tikv/pd/pkg/gc/iterateAllKeyspacesGCStatesError", fmt.Sprintf(`return(%q)`, errorMessage))) + defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/iterateAllKeyspacesGCStatesError")) }() + initFailed, err := s.manager.WatchGCStates(context.Background(), false) + re.NoError(err) + _, err = initFailed.RecvBatch(1) + re.ErrorContains(err, errorMessage) + re.Equal(activeBefore, promtestutil.ToFloat64(gcStateWatcherGauge)) + assertTerminationDeltas(1, 1, 1, 1) + initFailed.Close() + assertTerminationDeltas(1, 1, 1, 1) + }() +} + func mustUpsert(t testing.TB, change GCStateChange) GCState { state, ok := change.Upsert() require.True(t, ok) diff --git a/pkg/gc/metrics.go b/pkg/gc/metrics.go index 86e4f088150..912d99a9575 100644 --- a/pkg/gc/metrics.go +++ b/pkg/gc/metrics.go @@ -78,12 +78,32 @@ var ( gcStateCacheAccessHitCounter = gcStateCacheAccessCounter.WithLabelValues("hit") gcStateCacheAccessSlowHitCounter = gcStateCacheAccessCounter.WithLabelValues("slow_hit") gcStateCacheAccessMissCounter = gcStateCacheAccessCounter.WithLabelValues("miss") + + gcStateWatcherGauge = prometheus.NewGauge(prometheus.GaugeOpts{ + Namespace: "pd", + Subsystem: "gc", + Name: "watcher_count", + Help: "Current number of active GC state watchers.", + }) + gcStateWatcherTerminationCounter = prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: "pd", + Subsystem: "gc", + Name: "watcher_termination_total", + Help: "Total number of GC state watcher terminations by reason.", + }, []string{"reason"}) + + gcStateWatcherTerminationClientCancelCounter = gcStateWatcherTerminationCounter.WithLabelValues("client_cancel") + gcStateWatcherTerminationLeaderLostCounter = gcStateWatcherTerminationCounter.WithLabelValues("leader_lost") + gcStateWatcherTerminationSlowConsumerCounter = gcStateWatcherTerminationCounter.WithLabelValues("slow_consumer") + gcStateWatcherTerminationInitErrorCounter = gcStateWatcherTerminationCounter.WithLabelValues("init_error") ) func init() { prometheus.MustRegister(productionBarrierMetrics) prometheus.MustRegister(gcSafePointGauge) prometheus.MustRegister(gcStateCacheAccessCounter) + prometheus.MustRegister(gcStateWatcherGauge) + prometheus.MustRegister(gcStateWatcherTerminationCounter) } type barrierMetricScope struct { From d418fe3aab5ebe8e5cfdd650ec8c6c0f984eff43 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 18:51:51 +0800 Subject: [PATCH 08/25] server: implement WatchGCStates stream Signed-off-by: Wenxuan Zhang --- client/go.mod | 2 +- client/go.sum | 4 +- go.mod | 2 +- go.sum | 4 +- server/gc_service.go | 127 ++++++++++++++++++++ server/gc_service_test.go | 235 ++++++++++++++++++++++++++++++++++++++ tests/integrations/go.mod | 2 +- tests/integrations/go.sum | 4 +- tools/go.mod | 2 +- tools/go.sum | 4 +- 10 files changed, 374 insertions(+), 12 deletions(-) create mode 100644 server/gc_service_test.go diff --git a/client/go.mod b/client/go.mod index e99a4eca9af..0c4442cfdb7 100644 --- a/client/go.mod +++ b/client/go.mod @@ -10,7 +10,7 @@ require ( github.com/opentracing/opentracing-go v1.2.0 github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 - github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 + github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 github.com/prometheus/client_golang v1.20.5 github.com/prometheus/client_model v0.6.1 diff --git a/client/go.sum b/client/go.sum index b7e014b2e72..6a5b30aed15 100644 --- a/client/go.sum +++ b/client/go.sum @@ -53,8 +53,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c h1:xpW9bvK+HuuTm github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ueLEUZgtx2cIogM0v4Zj5uvvzhuuiu7Pn8HzMPg= github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 h1:tdMsjOqUR7YXHoBitzdebTvOjs/swniBTOLy5XiMtuE= github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86/go.mod h1:exzhVYca3WRtd6gclGNErRWb1qEgff3LYta0LvRmON4= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 h1:6yUryXKVbKpCNdZWL58/OcZj8NPLUA/xsJYXSbsD59w= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d h1:KS3ekak/ljCj5xvkGqbwVLi2eL7B8GFSYzU9TOUUPPo= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= diff --git a/go.mod b/go.mod index 827bda4a3c0..6a011e018a0 100644 --- a/go.mod +++ b/go.mod @@ -35,7 +35,7 @@ require ( github.com/pingcap/errcode v0.3.0 github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 - github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 + github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 github.com/pingcap/metering_sdk v0.0.0-20260814062708-9e3b68cd9adf github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21 diff --git a/go.sum b/go.sum index b74b2ade8cc..978e20fa993 100644 --- a/go.sum +++ b/go.sum @@ -490,8 +490,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ue github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 h1:tdMsjOqUR7YXHoBitzdebTvOjs/swniBTOLy5XiMtuE= github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86/go.mod h1:exzhVYca3WRtd6gclGNErRWb1qEgff3LYta0LvRmON4= github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 h1:6yUryXKVbKpCNdZWL58/OcZj8NPLUA/xsJYXSbsD59w= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d h1:KS3ekak/ljCj5xvkGqbwVLi2eL7B8GFSYzU9TOUUPPo= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= diff --git a/server/gc_service.go b/server/gc_service.go index 122ddc9d5d2..1f10a0262f9 100644 --- a/server/gc_service.go +++ b/server/gc_service.go @@ -16,9 +16,11 @@ package server import ( "context" + "errors" "math" "time" + "github.com/golang/protobuf/proto" "go.uber.org/zap" "google.golang.org/grpc" "google.golang.org/grpc/codes" @@ -28,6 +30,7 @@ import ( "github.com/pingcap/kvproto/pkg/pdpb" "github.com/pingcap/log" + "github.com/tikv/pd/pkg/errs" "github.com/tikv/pd/pkg/gc" "github.com/tikv/pd/pkg/keyspace/constant" "github.com/tikv/pd/pkg/storage/endpoint" @@ -36,6 +39,16 @@ import ( "github.com/tikv/pd/pkg/utils/typeutil" ) +const ( + watchGCStatesRecvBatchSize = 1024 + maxWatchGCStatesResponseSize = 1 << 20 +) + +type gcStateChangeReceiver interface { + RecvBatch(maxChanges int) ([]gc.GCStateChange, error) + Err() error +} + // UpdateGCSafePoint implements gRPC PDServer. // // Deprecated: Use AdvanceGCSafePoint instead. Note that it's only for use of GC internal. @@ -533,6 +546,96 @@ func gcStateToProto(gcState gc.GCState, now time.Time) *pdpb.GCState { } } +func gcStateChangeToProto(change gc.GCStateChange) (*pdpb.GCStateChange, error) { + if state, ok := change.Upsert(); ok { + state.GCBarriers = nil + return &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{ + Upsert: gcStateToProto(state, time.Time{}), + }}, nil + } + if keyspaceID, ok := change.RemovedKeyspaceID(); ok { + return &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Removed{ + Removed: &pdpb.KeyspaceScope{ + Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: keyspaceID}, + }, + }}, nil + } + return nil, errors.New("invalid GC state change") +} + +func splitWatchGCStatesResponses(changes []*pdpb.GCStateChange, maxSize int) []*pdpb.WatchGCStatesResponse { + if len(changes) == 0 { + return nil + } + + responses := make([]*pdpb.WatchGCStatesResponse, 0, 1) + newResponse := func() (*pdpb.WatchGCStatesResponse, int) { + response := &pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()} + return response, proto.Size(response) + } + current, currentSize := newResponse() + for _, change := range changes { + changeSize := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}) + if len(current.Changes) > 0 && currentSize+changeSize > maxSize { + responses = append(responses, current) + current, currentSize = newResponse() + } + + current.Changes = append(current.Changes, change) + currentSize += changeSize + if len(current.Changes) == 1 && currentSize > maxSize { + log.Warn("GC state change exceeds WatchGCStates response size", + zap.Int("serialized-size", currentSize), + zap.Int("max-size", maxSize)) + responses = append(responses, current) + current, currentSize = newResponse() + } + } + if len(current.Changes) > 0 { + responses = append(responses, current) + } + return responses +} + +func watchGCStatesErrorToStatus(err error) error { + switch { + case errors.Is(err, errs.ErrGCStateWatcherSlowConsumer): + return status.Error(codes.ResourceExhausted, err.Error()) + case errors.Is(err, errs.ErrNotLeader): + return errs.ErrNotLeader + case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded): + return status.FromContextError(err).Err() + default: + return status.Error(codes.Unavailable, err.Error()) + } +} + +func serveWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error { + for { + changes, err := receiver.RecvBatch(watchGCStatesRecvBatchSize) + if err != nil { + return watchGCStatesErrorToStatus(err) + } + protoChanges := make([]*pdpb.GCStateChange, 0, len(changes)) + for _, change := range changes { + converted, err := gcStateChangeToProto(change) + if err != nil { + log.Error("failed to convert GC state change", zap.Error(err)) + return status.Error(codes.Internal, err.Error()) + } + protoChanges = append(protoChanges, converted) + } + for _, response := range splitWatchGCStatesResponses(protoChanges, maxResponseSize) { + if err := receiver.Err(); err != nil { + return watchGCStatesErrorToStatus(err) + } + if err := stream.Send(response); err != nil { + return err + } + } + } +} + // AdvanceGCSafePoint tries to advance the GC safe point. func (s *GrpcServer) AdvanceGCSafePoint(ctx context.Context, request *pdpb.AdvanceGCSafePointRequest) (*pdpb.AdvanceGCSafePointResponse, error) { done, err := s.rateLimitCheck() @@ -806,6 +909,30 @@ func (s *GrpcServer) GetAllKeyspacesGCStates(ctx context.Context, request *pdpb. }, nil } +// WatchGCStates streams effective GC state changes from this PD server. +func (s *GrpcServer) WatchGCStates(request *pdpb.WatchGCStatesRequest, stream pdpb.PD_WatchGCStatesServer) error { + done, err := s.rateLimitCheck() + if err != nil { + return err + } + if done != nil { + defer done() + } + if err := s.validateRequest(request.GetHeader()); err != nil { + return err + } + if s.GetRaftCluster() == nil { + return status.Error(codes.Unavailable, errs.ErrNotBootstrapped.FastGenByArgs().Error()) + } + + watcher, err := s.gcStateManager.WatchGCStates(stream.Context(), request.GetSkipLoadingInitial()) + if err != nil { + return watchGCStatesErrorToStatus(err) + } + defer watcher.Close() + return serveWatchGCStates(watcher, stream, maxWatchGCStatesResponseSize) +} + // SetGlobalGCBarrier sets a global GC barrier. func (s *GrpcServer) SetGlobalGCBarrier(ctx context.Context, request *pdpb.SetGlobalGCBarrierRequest) (*pdpb.SetGlobalGCBarrierResponse, error) { done, err := s.rateLimitCheck() diff --git a/server/gc_service_test.go b/server/gc_service_test.go new file mode 100644 index 00000000000..296abf83f26 --- /dev/null +++ b/server/gc_service_test.go @@ -0,0 +1,235 @@ +// Copyright 2026 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package server + +import ( + "context" + "errors" + "testing" + + "github.com/golang/protobuf/proto" + "github.com/stretchr/testify/require" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/metadata" + "google.golang.org/grpc/status" + + "github.com/pingcap/kvproto/pkg/pdpb" + + "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/gc" + "github.com/tikv/pd/pkg/storage/endpoint" + "github.com/tikv/pd/pkg/utils/grpcutil" +) + +func TestGCStateChangeToProto(t *testing.T) { + testCases := []struct { + name string + change gc.GCStateChange + want *pdpb.GCStateChange + wantErr bool + }{ + { + name: "complete upsert", + change: gc.NewGCStateUpsert(gc.GCState{ + KeyspaceID: 7, + IsKeyspaceLevel: true, + TxnSafePoint: 10, + GCSafePoint: 5, + GCBarriers: []*endpoint.GCBarrier{ + {BarrierID: "test-barrier", BarrierTS: 8}, + }, + }), + want: &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{Upsert: &pdpb.GCState{ + KeyspaceScope: &pdpb.KeyspaceScope{Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: 7}}, + IsKeyspaceLevelGc: true, + TxnSafePoint: 10, + GcSafePoint: 5, + }}}, + }, + { + name: "removed scope", + change: gc.NewGCStateRemoved(9), + want: &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Removed{Removed: &pdpb.KeyspaceScope{ + Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: 9}, + }}}, + }, + { + name: "invalid zero value", + wantErr: true, + }, + } + + for _, testCase := range testCases { + t.Run(testCase.name, func(t *testing.T) { + got, err := gcStateChangeToProto(testCase.change) + if testCase.wantErr { + require.Error(t, err) + require.Nil(t, got) + return + } + require.NoError(t, err) + require.True(t, proto.Equal(testCase.want, got), "expected %s, got %s", testCase.want, got) + if got.GetUpsert() != nil { + require.Empty(t, got.GetUpsert().GetGcBarriers()) + } + }) + } +} + +func TestSplitWatchGCStatesResponses(t *testing.T) { + change := &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{Upsert: &pdpb.GCState{ + KeyspaceScope: &pdpb.KeyspaceScope{Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: 7}}, + TxnSafePoint: 10, + GcSafePoint: 5, + }}} + base := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}) + delta := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}) + + exact := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta) + require.Len(t, exact, 1) + require.LessOrEqual(t, proto.Size(exact[0]), base+2*delta) + + split := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta-1) + require.Len(t, split, 2) + for _, response := range split { + require.NotNil(t, response.GetHeader()) + require.NotEmpty(t, response.GetChanges()) + require.LessOrEqual(t, proto.Size(response), base+2*delta-1) + } + + oversized := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change}, base+delta-1) + require.Len(t, oversized, 1) + require.Greater(t, proto.Size(oversized[0]), base+delta-1) + require.Empty(t, splitWatchGCStatesResponses(nil, base+delta)) +} + +type fakeGCStateChangeReceiver struct { + batches [][]gc.GCStateChange + receiveErr error + terminalErr error + receivedMaxes []int +} + +func (r *fakeGCStateChangeReceiver) RecvBatch(maxChanges int) ([]gc.GCStateChange, error) { + r.receivedMaxes = append(r.receivedMaxes, maxChanges) + if len(r.batches) == 0 { + return nil, r.receiveErr + } + batch := r.batches[0] + r.batches = r.batches[1:] + return batch, nil +} + +func (r *fakeGCStateChangeReceiver) Err() error { + return r.terminalErr +} + +type fakeWatchGCStatesServer struct { + ctx context.Context + sent []*pdpb.WatchGCStatesResponse + sendHook func(*pdpb.WatchGCStatesResponse) error +} + +func (s *fakeWatchGCStatesServer) Send(response *pdpb.WatchGCStatesResponse) error { + if s.sendHook != nil { + if err := s.sendHook(response); err != nil { + return err + } + } + s.sent = append(s.sent, response) + return nil +} + +func (*fakeWatchGCStatesServer) SetHeader(metadata.MD) error { return nil } +func (*fakeWatchGCStatesServer) SendHeader(metadata.MD) error { return nil } +func (*fakeWatchGCStatesServer) SetTrailer(metadata.MD) {} + +func (s *fakeWatchGCStatesServer) Context() context.Context { + if s.ctx == nil { + return context.Background() + } + return s.ctx +} + +func (*fakeWatchGCStatesServer) SendMsg(any) error { return nil } +func (*fakeWatchGCStatesServer) RecvMsg(any) error { return nil } + +func TestServeWatchGCStatesRechecksTerminalCauseBeforeEverySend(t *testing.T) { + state := gc.GCState{KeyspaceID: 7, TxnSafePoint: 10, GCSafePoint: 5} + receiver := &fakeGCStateChangeReceiver{ + batches: [][]gc.GCStateChange{{gc.NewGCStateUpsert(state), gc.NewGCStateUpsert(state)}}, + } + stream := &fakeWatchGCStatesServer{} + stream.sendHook = func(*pdpb.WatchGCStatesResponse) error { + receiver.terminalErr = errs.ErrNotLeader + return nil + } + protoChange := &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{Upsert: &pdpb.GCState{ + KeyspaceScope: &pdpb.KeyspaceScope{Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: 7}}, + TxnSafePoint: 10, + GcSafePoint: 5, + }}} + maxSize := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}) + + proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{protoChange}}) + + err := serveWatchGCStates(receiver, stream, maxSize) + require.ErrorIs(t, err, errs.ErrNotLeader) + require.Equal(t, codes.Unavailable, status.Code(err)) + require.Len(t, stream.sent, 1) + require.Equal(t, []int{1024}, receiver.receivedMaxes) +} + +func TestServeWatchGCStatesRejectsInvalidInternalChange(t *testing.T) { + receiver := &fakeGCStateChangeReceiver{batches: [][]gc.GCStateChange{{{}}}} + stream := &fakeWatchGCStatesServer{} + + err := serveWatchGCStates(receiver, stream, 1024) + require.Equal(t, codes.Internal, status.Code(err)) + require.Empty(t, stream.sent) +} + +func TestServeWatchGCStatesReturnsRawSendError(t *testing.T) { + sendErr := errors.New("send failed") + receiver := &fakeGCStateChangeReceiver{ + batches: [][]gc.GCStateChange{{gc.NewGCStateRemoved(9)}}, + } + stream := &fakeWatchGCStatesServer{sendHook: func(*pdpb.WatchGCStatesResponse) error { + return sendErr + }} + + err := serveWatchGCStates(receiver, stream, 1024) + require.Same(t, sendErr, err) +} + +func TestWatchGCStatesErrorToStatus(t *testing.T) { + testCases := []struct { + name string + err error + code codes.Code + }{ + {name: "not leader", err: errs.ErrNotLeader, code: codes.Unavailable}, + {name: "initialization failure", err: errors.New("load initial GC states"), code: codes.Unavailable}, + {name: "slow consumer", err: errs.ErrGCStateWatcherSlowConsumer, code: codes.ResourceExhausted}, + {name: "canceled", err: context.Canceled, code: codes.Canceled}, + {name: "deadline exceeded", err: context.DeadlineExceeded, code: codes.DeadlineExceeded}, + } + + for _, testCase := range testCases { + t.Run(testCase.name, func(t *testing.T) { + err := watchGCStatesErrorToStatus(testCase.err) + require.Equal(t, testCase.code, status.Code(err)) + }) + } +} diff --git a/tests/integrations/go.mod b/tests/integrations/go.mod index 00b9e075b4f..eb933b245ba 100644 --- a/tests/integrations/go.mod +++ b/tests/integrations/go.mod @@ -15,7 +15,7 @@ require ( github.com/golang/protobuf v1.5.4 github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 - github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 + github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 github.com/prometheus/client_golang v1.20.5 github.com/prometheus/client_model v0.6.1 diff --git a/tests/integrations/go.sum b/tests/integrations/go.sum index 5db28b57234..2a17320ad60 100644 --- a/tests/integrations/go.sum +++ b/tests/integrations/go.sum @@ -483,8 +483,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ue github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 h1:tdMsjOqUR7YXHoBitzdebTvOjs/swniBTOLy5XiMtuE= github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86/go.mod h1:exzhVYca3WRtd6gclGNErRWb1qEgff3LYta0LvRmON4= github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 h1:6yUryXKVbKpCNdZWL58/OcZj8NPLUA/xsJYXSbsD59w= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d h1:KS3ekak/ljCj5xvkGqbwVLi2eL7B8GFSYzU9TOUUPPo= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= diff --git a/tools/go.mod b/tools/go.mod index 747f9198561..7e0027895b5 100644 --- a/tools/go.mod +++ b/tools/go.mod @@ -23,7 +23,7 @@ require ( github.com/mattn/go-shellwords v1.0.12 github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 - github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 + github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 github.com/pmezard/go-difflib v1.0.0 github.com/prometheus/client_golang v1.20.5 diff --git a/tools/go.sum b/tools/go.sum index 769f9c9e60d..efdcb12f0d3 100644 --- a/tools/go.sum +++ b/tools/go.sum @@ -488,8 +488,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ue github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 h1:tdMsjOqUR7YXHoBitzdebTvOjs/swniBTOLy5XiMtuE= github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86/go.mod h1:exzhVYca3WRtd6gclGNErRWb1qEgff3LYta0LvRmON4= github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 h1:6yUryXKVbKpCNdZWL58/OcZj8NPLUA/xsJYXSbsD59w= -github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d h1:KS3ekak/ljCj5xvkGqbwVLi2eL7B8GFSYzU9TOUUPPo= +github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE= github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw= github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4= From 61104691de86aa43b9d625c8a83093d22ee566e1 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 19:07:37 +0800 Subject: [PATCH 09/25] tests: cover WatchGCStates lifecycle Signed-off-by: Wenxuan Zhang --- tests/server/gc/gc_test.go | 302 +++++++++++++++++++++++++++++++++++++ 1 file changed, 302 insertions(+) diff --git a/tests/server/gc/gc_test.go b/tests/server/gc/gc_test.go index d1d261ae203..8df3b7397f9 100644 --- a/tests/server/gc/gc_test.go +++ b/tests/server/gc/gc_test.go @@ -25,12 +25,15 @@ import ( "github.com/stretchr/testify/require" "go.uber.org/goleak" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" "github.com/pingcap/failpoint" "github.com/pingcap/kvproto/pkg/pdpb" "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/keyspace/constant" + "github.com/tikv/pd/pkg/ratelimit" "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/pkg/versioninfo/kerneltype" "github.com/tikv/pd/server/config" @@ -48,6 +51,7 @@ const ( postGetGCStateCallFailpoint = "github.com/tikv/pd/server/postGetGCStateCall" getGCStateBeforeSlowPathFailpoint = "github.com/tikv/pd/pkg/gc/getGCStateBeforeSlowPath" skipCampaignLeaderCheckFailpoint = "github.com/tikv/pd/pkg/member/skipCampaignLeaderCheck" + watchGCStatesRegisteredFailpoint = "github.com/tikv/pd/pkg/gc/watchGCStatesRegistered" ) func makeKeyspaceScope(keyspaceID uint32) *pdpb.KeyspaceScope { @@ -139,6 +143,122 @@ func (p *blockingFailpoint) releaseAndDisable(re *require.Assertions) { }) } +type watchGCStatesRegistrationPoint struct { + registered chan struct{} + registerOnce sync.Once + disableOnce sync.Once +} + +func enableWatchGCStatesRegistrationPoint(t *testing.T) *watchGCStatesRegistrationPoint { + t.Helper() + re := require.New(t) + point := &watchGCStatesRegistrationPoint{registered: make(chan struct{})} + re.NoError(failpoint.EnableCall(watchGCStatesRegisteredFailpoint, func() { + point.registerOnce.Do(func() { + close(point.registered) + }) + })) + t.Cleanup(func() { + point.disable(re) + }) + return point +} + +func (p *watchGCStatesRegistrationPoint) wait(t *testing.T) { + t.Helper() + select { + case <-p.registered: + case <-time.After(5 * time.Second): + require.FailNow(t, "WatchGCStates was not registered") + } +} + +func (p *watchGCStatesRegistrationPoint) disable(re *require.Assertions) { + p.disableOnce.Do(func() { + re.NoError(failpoint.Disable(watchGCStatesRegisteredFailpoint)) + }) +} + +func newWatchGCStatesCluster(t *testing.T, serverCount int, bootstrap bool) *tests.TestCluster { + t.Helper() + re := require.New(t) + ctx, cancel := context.WithCancel(context.Background()) + cluster, err := tests.NewTestCluster(ctx, serverCount, func(conf *config.Config, _ string) { + conf.Keyspace.WaitRegionSplit = false + }) + re.NoError(err) + t.Cleanup(func() { + cancel() + cluster.Destroy() + }) + re.NoError(cluster.RunInitialServers()) + re.NotEmpty(cluster.WaitLeader()) + if bootstrap { + re.NoError(cluster.GetLeaderServer().BootstrapCluster()) + } + return cluster +} + +func newWatchGCStatesClient(t *testing.T, addr string) pdpb.PDClient { + t.Helper() + re := require.New(t) + client, conn := testutil.MustNewGrpcClient(re, addr) + t.Cleanup(func() { + re.NoError(conn.Close()) + }) + return client +} + +func openWatchGCStates( + t *testing.T, + client pdpb.PDClient, + header *pdpb.RequestHeader, + skipLoadingInitial bool, +) (pdpb.PD_WatchGCStatesClient, context.CancelFunc) { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) + t.Cleanup(cancel) + stream, err := client.WatchGCStates(ctx, &pdpb.WatchGCStatesRequest{ + Header: header, + SkipLoadingInitial: skipLoadingInitial, + }) + require.NoError(t, err) + return stream, cancel +} + +func recvWatchGCStateForKeyspace(t *testing.T, stream pdpb.PD_WatchGCStatesClient, keyspaceID uint32) *pdpb.GCState { + t.Helper() + for { + response, err := stream.Recv() + require.NoError(t, err) + require.NotNil(t, response.GetHeader()) + for _, change := range response.GetChanges() { + if state := change.GetUpsert(); state != nil && state.GetKeyspaceScope().GetKeyspaceId() == keyspaceID { + return state + } + } + } +} + +func advanceWatchGCStatesTxnSafePoint( + t *testing.T, + client pdpb.PDClient, + header *pdpb.RequestHeader, + keyspaceID uint32, + target uint64, +) { + t.Helper() + response, err := client.AdvanceTxnSafePoint(context.Background(), &pdpb.AdvanceTxnSafePointRequest{ + Header: header, + KeyspaceScope: makeKeyspaceScope(keyspaceID), + Target: target, + }) + require.NoError(t, err) + require.NotNil(t, response.GetHeader()) + require.Nil(t, response.GetHeader().GetError()) + require.Equal(t, target, response.GetNewTxnSafePoint()) +} + func TestGCOperations(t *testing.T) { re := require.New(t) ctx, cancel := context.WithCancel(context.Background()) @@ -883,3 +1003,185 @@ func TestGetGCStateSlowPathReadsLatestStateIfLeaderLostBeforeRead(t *testing.T) re.Nil(res.resp.GetHeader().GetError()) re.Equal(uint64(20), res.resp.GetGcState().GetTxnSafePoint()) } + +func TestWatchGCStatesInitialAndSkipInitialRegistrationBoundary(t *testing.T) { + re := require.New(t) + cluster := newWatchGCStatesCluster(t, 1, true) + leaderServer := cluster.GetLeaderServer() + re.NotNil(leaderServer) + + ks, err := leaderServer.GetKeyspaceManager().CreateKeyspace(&keyspace.CreateKeyspaceRequest{ + Name: "watch-gc-states", + Config: map[string]string{keyspace.GCManagementType: keyspace.KeyspaceLevelGC}, + CreateTime: time.Now().Unix(), + }) + re.NoError(err) + + client := newWatchGCStatesClient(t, leaderServer.GetAddr()) + header := testutil.NewRequestHeader(leaderServer.GetClusterID()) + initialStream, cancelInitial := openWatchGCStates(t, client, header, false) + initial := recvWatchGCStateForKeyspace(t, initialStream, ks.GetId()) + re.True(initial.GetIsKeyspaceLevelGc()) + re.Zero(initial.GetTxnSafePoint()) + re.Zero(initial.GetGcSafePoint()) + re.Empty(initial.GetGcBarriers()) + + advanceWatchGCStatesTxnSafePoint(t, client, header, ks.GetId(), 10) + live := recvWatchGCStateForKeyspace(t, initialStream, ks.GetId()) + re.True(live.GetIsKeyspaceLevelGc()) + re.Equal(uint64(10), live.GetTxnSafePoint()) + re.Zero(live.GetGcSafePoint()) + re.Empty(live.GetGcBarriers()) + cancelInitial() + + registration := enableWatchGCStatesRegistrationPoint(t) + skipInitialStream, _ := openWatchGCStates(t, client, header, true) + registration.wait(t) + registration.disable(re) + + advanceWatchGCStatesTxnSafePoint(t, client, header, ks.GetId(), 20) + firstAfterRegistration := recvWatchGCStateForKeyspace(t, skipInitialStream, ks.GetId()) + re.True(firstAfterRegistration.GetIsKeyspaceLevelGc()) + re.Equal(uint64(20), firstAfterRegistration.GetTxnSafePoint()) + re.Zero(firstAfterRegistration.GetGcSafePoint()) + re.Empty(firstAfterRegistration.GetGcBarriers()) +} + +func TestWatchGCStatesRequestPreflight(t *testing.T) { + tests := []struct { + name string + setup func(*testing.T) (string, *pdpb.RequestHeader) + wantCode codes.Code + }{ + { + name: "wrong cluster ID", + setup: func(t *testing.T) (string, *pdpb.RequestHeader) { + cluster := newWatchGCStatesCluster(t, 1, true) + leader := cluster.GetLeaderServer() + return leader.GetAddr(), testutil.NewRequestHeader(leader.GetClusterID() + 1) + }, + wantCode: codes.FailedPrecondition, + }, + { + name: "direct follower", + setup: func(t *testing.T) (string, *pdpb.RequestHeader) { + cluster := newWatchGCStatesCluster(t, 2, true) + follower := cluster.GetServer(cluster.GetFollower()) + require.NotNil(t, follower) + return follower.GetAddr(), testutil.NewRequestHeader(follower.GetClusterID()) + }, + wantCode: codes.Unavailable, + }, + { + name: "unbootstrapped leader", + setup: func(t *testing.T) (string, *pdpb.RequestHeader) { + cluster := newWatchGCStatesCluster(t, 1, false) + leader := cluster.GetLeaderServer() + return leader.GetAddr(), testutil.NewRequestHeader(leader.GetClusterID()) + }, + wantCode: codes.Unavailable, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + addr, header := test.setup(t) + client := newWatchGCStatesClient(t, addr) + stream, _ := openWatchGCStates(t, client, header, true) + response, err := stream.Recv() + require.Nil(t, response) + require.Equal(t, test.wantCode, status.Code(err)) + }) + } +} + +func TestWatchGCStatesHoldsRateLimitTokenForStreamLifetime(t *testing.T) { + re := require.New(t) + cluster := newWatchGCStatesCluster(t, 1, true) + leaderServer := cluster.GetLeaderServer() + re.NotNil(leaderServer) + server := leaderServer.GetServer() + options := server.GetServiceMiddlewarePersistOptions() + previousConfig := options.GetGRPCRateLimitConfig().Clone() + enabledConfig := previousConfig.Clone() + enabledConfig.EnableRateLimit = true + options.SetGRPCRateLimitConfig(enabledConfig) + limiter := server.GetGRPCRateLimiter() + limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(1)) + t.Cleanup(func() { + limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(0)) + options.SetGRPCRateLimitConfig(previousConfig) + }) + + client := newWatchGCStatesClient(t, leaderServer.GetAddr()) + header := testutil.NewRequestHeader(leaderServer.GetClusterID()) + firstRegistration := enableWatchGCStatesRegistrationPoint(t) + _, cancelFirst := openWatchGCStates(t, client, header, true) + firstRegistration.wait(t) + firstRegistration.disable(re) + limit, current := limiter.GetConcurrencyLimiterStatus("WatchGCStates") + re.Equal(uint64(1), limit) + re.Equal(uint64(1), current) + + secondStream, _ := openWatchGCStates(t, client, header, true) + response, err := secondStream.Recv() + re.Nil(response) + re.Equal(codes.ResourceExhausted, status.Code(err)) + + cancelFirst() + testutil.Eventually(re, func() bool { + _, current := limiter.GetConcurrencyLimiterStatus("WatchGCStates") + return current == 0 + }, testutil.WithWaitFor(5*time.Second), testutil.WithTickInterval(10*time.Millisecond)) + + thirdRegistration := enableWatchGCStatesRegistrationPoint(t) + thirdStream, _ := openWatchGCStates(t, client, header, true) + thirdRegistration.wait(t) + thirdRegistration.disable(re) + advanceWatchGCStatesTxnSafePoint(t, client, header, constant.NullKeyspaceID, 10) + state := recvWatchGCStateForKeyspace(t, thirdStream, constant.NullKeyspaceID) + re.Equal(uint64(10), state.GetTxnSafePoint()) +} + +func TestWatchGCStatesTerminatesOnLeaderTransferAndReinitializes(t *testing.T) { + re := require.New(t) + cluster, req, cleanup := newGCStateLeaderTransitionCluster(t) + t.Cleanup(cleanup) + + oldLeader := cluster.GetLeader() + re.NotEmpty(oldLeader) + oldLeaderServer := cluster.GetServer(oldLeader) + re.NotNil(oldLeaderServer) + oldClient := newWatchGCStatesClient(t, oldLeaderServer.GetAddr()) + oldStream, _ := openWatchGCStates(t, oldClient, req.GetHeader(), false) + initial := recvWatchGCStateForKeyspace(t, oldStream, constant.NullKeyspaceID) + re.False(initial.GetIsKeyspaceLevelGc()) + re.Zero(initial.GetTxnSafePoint()) + re.Zero(initial.GetGcSafePoint()) + re.Empty(initial.GetGcBarriers()) + + re.NoError(oldLeaderServer.ResignLeaderWithRetry()) + newLeader := cluster.WaitLeader() + re.NotEmpty(newLeader) + re.NotEqual(oldLeader, newLeader) + for { + response, err := oldStream.Recv() + if err != nil { + re.Nil(response) + re.Equal(codes.Unavailable, status.Code(err)) + break + } + re.NotNil(response) + } + + newLeaderServer := cluster.GetServer(newLeader) + re.NotNil(newLeaderServer) + newClient := newWatchGCStatesClient(t, newLeaderServer.GetAddr()) + advanceWatchGCStatesTxnSafePoint(t, newClient, req.GetHeader(), constant.NullKeyspaceID, 10) + newStream, _ := openWatchGCStates(t, newClient, req.GetHeader(), false) + reinitialized := recvWatchGCStateForKeyspace(t, newStream, constant.NullKeyspaceID) + re.False(reinitialized.GetIsKeyspaceLevelGc()) + re.Equal(uint64(10), reinitialized.GetTxnSafePoint()) + re.Zero(reinitialized.GetGcSafePoint()) + re.Empty(reinitialized.GetGcBarriers()) +} From 2cbef2006f84c2eefa4bbc24d983387a07764af0 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 19:19:29 +0800 Subject: [PATCH 10/25] tests: tighten WatchGCStates integration coverage Signed-off-by: Wenxuan Zhang --- tests/server/gc/gc_test.go | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/tests/server/gc/gc_test.go b/tests/server/gc/gc_test.go index 8df3b7397f9..3f9b45669d4 100644 --- a/tests/server/gc/gc_test.go +++ b/tests/server/gc/gc_test.go @@ -248,7 +248,9 @@ func advanceWatchGCStatesTxnSafePoint( target uint64, ) { t.Helper() - response, err := client.AdvanceTxnSafePoint(context.Background(), &pdpb.AdvanceTxnSafePointRequest{ + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) + defer cancel() + response, err := client.AdvanceTxnSafePoint(ctx, &pdpb.AdvanceTxnSafePointRequest{ Header: header, KeyspaceScope: makeKeyspaceScope(keyspaceID), Target: target, @@ -1040,7 +1042,13 @@ func TestWatchGCStatesInitialAndSkipInitialRegistrationBoundary(t *testing.T) { registration.disable(re) advanceWatchGCStatesTxnSafePoint(t, client, header, ks.GetId(), 20) - firstAfterRegistration := recvWatchGCStateForKeyspace(t, skipInitialStream, ks.GetId()) + firstResponse, err := skipInitialStream.Recv() + re.NoError(err) + re.NotNil(firstResponse.GetHeader()) + re.Len(firstResponse.GetChanges(), 1) + firstAfterRegistration := firstResponse.GetChanges()[0].GetUpsert() + re.NotNil(firstAfterRegistration) + re.Equal(ks.GetId(), firstAfterRegistration.GetKeyspaceScope().GetKeyspaceId()) re.True(firstAfterRegistration.GetIsKeyspaceLevelGc()) re.Equal(uint64(20), firstAfterRegistration.GetTxnSafePoint()) re.Zero(firstAfterRegistration.GetGcSafePoint()) From 9d7f4b079981df75c6855d4619640a1c1b343c74 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 19:36:34 +0800 Subject: [PATCH 11/25] gc: satisfy WatchGCStates static checks Signed-off-by: Wenxuan Zhang --- pkg/gc/gc_state_manager_test.go | 6 +++--- pkg/gc/gc_state_watcher.go | 14 +++++++------ pkg/gc/gc_state_watcher_test.go | 9 +++++---- server/gc_service.go | 5 ++--- server/gc_service_test.go | 35 ++++++++++++++++++++++----------- 5 files changed, 42 insertions(+), 27 deletions(-) diff --git a/pkg/gc/gc_state_manager_test.go b/pkg/gc/gc_state_manager_test.go index 2f84e454732..738b01da1c8 100644 --- a/pkg/gc/gc_state_manager_test.go +++ b/pkg/gc/gc_state_manager_test.go @@ -294,7 +294,7 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchLeadershipGeneration() { func (s *gcStateManagerTestSuite) TestGCStateWatchLoadsInitialStatesIncrementally() { re := s.Require() - w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{ + w, err := s.manager.registerGCStateWatcher(context.Background(), false, gcStateWatchConfig{ initialBatchSize: 2, initChannelCapacity: 1, liveChannelCapacity: 1, @@ -354,7 +354,7 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchLiveSuppressesPausedInitial() defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded")) }() defer releaseLoader() - w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{initialBatchSize: 1, initChannelCapacity: 16, liveChannelCapacity: 4}) + w, err := s.manager.registerGCStateWatcher(context.Background(), false, gcStateWatchConfig{initialBatchSize: 1, initChannelCapacity: 16, liveChannelCapacity: 4}) re.NoError(err) defer w.Close() select { @@ -409,7 +409,7 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchInitialFailureTerminatesWatche func (s *gcStateManagerTestSuite) TestGCStateWatchFullInitChannelDoesNotHoldManagerMutex() { re := s.Require() stop := s.manager.OnNodeBecomesLeader() - w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{ + w, err := s.manager.registerGCStateWatcher(context.Background(), false, gcStateWatchConfig{ initialBatchSize: 1, initChannelCapacity: 1, liveChannelCapacity: 1, diff --git a/pkg/gc/gc_state_watcher.go b/pkg/gc/gc_state_watcher.go index 10f24dbe626..8184a2c1f92 100644 --- a/pkg/gc/gc_state_watcher.go +++ b/pkg/gc/gc_state_watcher.go @@ -17,10 +17,11 @@ package gc import ( "context" + "go.uber.org/zap" + "github.com/pingcap/errors" "github.com/pingcap/failpoint" "github.com/pingcap/log" - "go.uber.org/zap" "github.com/tikv/pd/pkg/errs" ) @@ -34,6 +35,7 @@ const ( ) // GCStateChange describes one effective GC state change for a keyspace scope. +// nolint:revive // Keep GC in the name to match the established GCState domain API. type GCStateChange struct { kind gcStateChangeKind upsert GCState @@ -94,6 +96,7 @@ const ( // // A watcher supports one receiving goroutine. Close may be called concurrently with // receiving and with manager-owned lifecycle operations. +// nolint:revive // Keep GC in the name to match the established GCState domain API. type GCStateWatcher struct { ctx context.Context cancel context.CancelCauseFunc @@ -249,14 +252,14 @@ func (w *GCStateWatcher) Close() { // WatchGCStates registers a watcher in the current local leadership generation. func (m *GCStateManager) WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error) { - return m.watchGCStates(ctx, skipLoadingInitial, gcStateWatchConfig{ + return m.registerGCStateWatcher(ctx, skipLoadingInitial, gcStateWatchConfig{ initialBatchSize: defaultGCStateWatchInitialBatchSize, initChannelCapacity: defaultGCStateWatchInitChannelCapacity, liveChannelCapacity: defaultGCStateWatchLiveChannelCapacity, }) } -func (m *GCStateManager) watchGCStates( +func (m *GCStateManager) registerGCStateWatcher( ctx context.Context, skipLoadingInitial bool, cfg gcStateWatchConfig, @@ -363,16 +366,15 @@ func (m *GCStateManager) terminateGCStateWatcherLocked( watcher *GCStateWatcher, cause error, reason gcStateWatcherTerminationReason, -) bool { +) { registered, ok := m.watchers[watcher.id] if !ok || registered != watcher { - return false + return } delete(m.watchers, watcher.id) gcStateWatcherGauge.Dec() recordGCStateWatcherTerminationMetrics(reason) watcher.cancel(cause) - return true } func recordGCStateWatcherTerminationMetrics(reason gcStateWatcherTerminationReason) { diff --git a/pkg/gc/gc_state_watcher_test.go b/pkg/gc/gc_state_watcher_test.go index 438e6083d63..e965e296ba1 100644 --- a/pkg/gc/gc_state_watcher_test.go +++ b/pkg/gc/gc_state_watcher_test.go @@ -22,10 +22,11 @@ import ( "testing" "time" - "github.com/pingcap/failpoint" promtestutil "github.com/prometheus/client_golang/prometheus/testutil" "github.com/stretchr/testify/require" + "github.com/pingcap/failpoint" + "github.com/tikv/pd/pkg/errs" "github.com/tikv/pd/pkg/utils/keypath" ) @@ -236,10 +237,10 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchDoesNotPublishBarrierOnlyChang func (s *gcStateManagerTestSuite) TestGCStateWatchSlowConsumerIsolation() { re := s.Require() const keyspaceID = uint32(2) - watcherA, err := s.manager.watchGCStates(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 1}) + watcherA, err := s.manager.registerGCStateWatcher(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 1}) re.NoError(err) defer watcherA.Close() - watcherB, err := s.manager.watchGCStates(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 4}) + watcherB, err := s.manager.registerGCStateWatcher(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 4}) re.NoError(err) defer watcherB.Close() @@ -316,7 +317,7 @@ func (s *gcStateManagerTestSuite) TestGCStateWatcherMetrics() { re.Equal(activeBefore, promtestutil.ToFloat64(gcStateWatcherGauge)) assertTerminationDeltas(1, 1, 0, 0) - slowConsumer, err := s.manager.watchGCStates(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 1}) + slowConsumer, err := s.manager.registerGCStateWatcher(context.Background(), true, gcStateWatchConfig{liveChannelCapacity: 1}) re.NoError(err) re.Equal(activeBefore+1, promtestutil.ToFloat64(gcStateWatcherGauge)) _, err = s.manager.AdvanceTxnSafePoint(2, 10, time.Now()) diff --git a/server/gc_service.go b/server/gc_service.go index 1f10a0262f9..8df99f9d436 100644 --- a/server/gc_service.go +++ b/server/gc_service.go @@ -20,7 +20,6 @@ import ( "math" "time" - "github.com/golang/protobuf/proto" "go.uber.org/zap" "google.golang.org/grpc" "google.golang.org/grpc/codes" @@ -571,11 +570,11 @@ func splitWatchGCStatesResponses(changes []*pdpb.GCStateChange, maxSize int) []* responses := make([]*pdpb.WatchGCStatesResponse, 0, 1) newResponse := func() (*pdpb.WatchGCStatesResponse, int) { response := &pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()} - return response, proto.Size(response) + return response, response.Size() } current, currentSize := newResponse() for _, change := range changes { - changeSize := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}) + changeSize := (&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}).Size() if len(current.Changes) > 0 && currentSize+changeSize > maxSize { responses = append(responses, current) current, currentSize = newResponse() diff --git a/server/gc_service_test.go b/server/gc_service_test.go index 296abf83f26..5f6d35f53c4 100644 --- a/server/gc_service_test.go +++ b/server/gc_service_test.go @@ -19,7 +19,6 @@ import ( "errors" "testing" - "github.com/golang/protobuf/proto" "github.com/stretchr/testify/require" "google.golang.org/grpc/codes" "google.golang.org/grpc/metadata" @@ -80,9 +79,23 @@ func TestGCStateChangeToProto(t *testing.T) { return } require.NoError(t, err) - require.True(t, proto.Equal(testCase.want, got), "expected %s, got %s", testCase.want, got) - if got.GetUpsert() != nil { - require.Empty(t, got.GetUpsert().GetGcBarriers()) + switch want := testCase.want.GetChange().(type) { + case *pdpb.GCStateChange_Upsert: + upsert := got.GetUpsert() + require.NotNil(t, upsert) + require.Equal(t, want.Upsert.GetKeyspaceScope().GetKeyspaceId(), upsert.GetKeyspaceScope().GetKeyspaceId()) + require.Equal(t, want.Upsert.GetIsKeyspaceLevelGc(), upsert.GetIsKeyspaceLevelGc()) + require.Equal(t, want.Upsert.GetTxnSafePoint(), upsert.GetTxnSafePoint()) + require.Equal(t, want.Upsert.GetGcSafePoint(), upsert.GetGcSafePoint()) + require.Empty(t, upsert.GetGcBarriers()) + require.Nil(t, got.GetRemoved()) + case *pdpb.GCStateChange_Removed: + removed := got.GetRemoved() + require.NotNil(t, removed) + require.Equal(t, want.Removed.GetKeyspaceId(), removed.GetKeyspaceId()) + require.Nil(t, got.GetUpsert()) + default: + require.FailNow(t, "unexpected expected GC state change type") } }) } @@ -94,24 +107,24 @@ func TestSplitWatchGCStatesResponses(t *testing.T) { TxnSafePoint: 10, GcSafePoint: 5, }}} - base := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}) - delta := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}) + base := (&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}).Size() + delta := (&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}).Size() exact := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta) require.Len(t, exact, 1) - require.LessOrEqual(t, proto.Size(exact[0]), base+2*delta) + require.LessOrEqual(t, exact[0].Size(), base+2*delta) split := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta-1) require.Len(t, split, 2) for _, response := range split { require.NotNil(t, response.GetHeader()) require.NotEmpty(t, response.GetChanges()) - require.LessOrEqual(t, proto.Size(response), base+2*delta-1) + require.LessOrEqual(t, response.Size(), base+2*delta-1) } oversized := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change}, base+delta-1) require.Len(t, oversized, 1) - require.Greater(t, proto.Size(oversized[0]), base+delta-1) + require.Greater(t, oversized[0].Size(), base+delta-1) require.Empty(t, splitWatchGCStatesResponses(nil, base+delta)) } @@ -181,8 +194,8 @@ func TestServeWatchGCStatesRechecksTerminalCauseBeforeEverySend(t *testing.T) { TxnSafePoint: 10, GcSafePoint: 5, }}} - maxSize := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}) + - proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{protoChange}}) + maxSize := (&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}).Size() + + (&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{protoChange}}).Size() err := serveWatchGCStates(receiver, stream, maxSize) require.ErrorIs(t, err, errs.ErrNotLeader) From 238ad1afec35949c099c395e47ff9c5269ef2cfc Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 3 Sep 2026 20:20:11 +0800 Subject: [PATCH 12/25] gc: tighten WatchGCStates lifecycle guarantees Avoid entering initial-state storage after cancellation and cover cleanup when stream delivery fails. Clarify the merge ordering contract and keep the slow-consumer error consistent with repository conventions. Signed-off-by: Wenxuan Zhang --- errors.toml | 2 +- pkg/errs/errno.go | 11 +-- pkg/gc/gc_state_manager_test.go | 24 +++++++ pkg/gc/gc_state_watcher.go | 15 +++-- tests/server/gc/gc_test.go | 115 ++++++++++++++++++++++++++++++++ 5 files changed, 157 insertions(+), 10 deletions(-) diff --git a/errors.toml b/errors.toml index 4787c7f86fa..6c6fb3126e3 100644 --- a/errors.toml +++ b/errors.toml @@ -483,7 +483,7 @@ trying to update GC safe point to a too large value that exceeds the txn safe po ["PD:gc:ErrGCStateWatcherSlowConsumer"] error = ''' -GC state watcher is too slow +gc state watcher is too slow ''' ["PD:gc:ErrGlobalGCBarrierTSBehindTxnSafePoint"] diff --git a/pkg/errs/errno.go b/pkg/errs/errno.go index 91ab73b924e..020f7dfc56b 100644 --- a/pkg/errs/errno.go +++ b/pkg/errs/errno.go @@ -548,11 +548,12 @@ var ( // GC errors var ( - ErrGCOnInvalidKeyspace = errors.Normalize("trying to manage GC in keyspace %v (id: %v) where keyspace level GC is not enabled", errors.RFCCodeText("PD:gc:ErrGCOnInvalidKeyspace")) - ErrDecreasingGCSafePoint = errors.Normalize("trying to update GC safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingGCSafePoint")) - ErrGCSafePointExceedsTxnSafePoint = errors.Normalize("trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v", errors.RFCCodeText("PD:gc:ErrGCSafePointExceedsTxnSafePoint")) - ErrDecreasingTxnSafePoint = errors.Normalize("trying to update txn safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingTxnSafePoint")) - ErrGCStateWatcherSlowConsumer = errors.Normalize("GC state watcher is too slow", errors.RFCCodeText("PD:gc:ErrGCStateWatcherSlowConsumer")) + ErrGCOnInvalidKeyspace = errors.Normalize("trying to manage GC in keyspace %v (id: %v) where keyspace level GC is not enabled", errors.RFCCodeText("PD:gc:ErrGCOnInvalidKeyspace")) + ErrDecreasingGCSafePoint = errors.Normalize("trying to update GC safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingGCSafePoint")) + ErrGCSafePointExceedsTxnSafePoint = errors.Normalize("trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v", errors.RFCCodeText("PD:gc:ErrGCSafePointExceedsTxnSafePoint")) + ErrDecreasingTxnSafePoint = errors.Normalize("trying to update txn safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingTxnSafePoint")) + // ErrGCStateWatcherSlowConsumer indicates that a watcher cannot keep up with live GC state changes. + ErrGCStateWatcherSlowConsumer = errors.Normalize("gc state watcher is too slow", errors.RFCCodeText("PD:gc:ErrGCStateWatcherSlowConsumer")) ErrGCBarrierTSBehindTxnSafePoint = errors.Normalize("trying to set a GC barrier on ts %d which is already behind the txn safe point %d", errors.RFCCodeText("PD:gc:ErrGCBarrierTSBehindTxnSafePoint")) ErrReservedGCBarrierID = errors.Normalize("trying to set a GC barrier with a barrier ID that is reserved: %v", errors.RFCCodeText("PD:gc:ErrReservedGCBarrierID")) ErrGlobalGCBarrierTSBehindTxnSafePoint = errors.Normalize("trying to set a global GC barrier on ts %d which is already behind the txn safe point %d of keyspace %s", errors.RFCCodeText("PD:gc:ErrGlobalGCBarrierTSBehindTxnSafePoint")) diff --git a/pkg/gc/gc_state_manager_test.go b/pkg/gc/gc_state_manager_test.go index 738b01da1c8..1cf86c5d372 100644 --- a/pkg/gc/gc_state_manager_test.go +++ b/pkg/gc/gc_state_manager_test.go @@ -335,6 +335,30 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchSkipsInitialLoading() { } } +func (s *gcStateManagerTestSuite) TestGCStateWatchCanceledBeforeInitialLoaderDoesNotIterate() { + re := s.Require() + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + var iterationStarted atomic.Bool + re.NoError(failpoint.EnableCall("github.com/tikv/pd/pkg/gc/onGetAllKeyspacesGCStatesStart", func() { + iterationStarted.Store(true) + })) + defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/onGetAllKeyspacesGCStatesStart")) }() + re.NoError(failpoint.EnableCall("github.com/tikv/pd/pkg/gc/watchGCStatesRegistered", cancel)) + defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/watchGCStatesRegistered")) }() + + // Suppress the automatic loader so this test can model a loader goroutine that + // starts only after registration has synchronously canceled its watcher. + w, err := s.manager.registerGCStateWatcher(ctx, true, gcStateWatchConfig{initialBatchSize: 1, liveChannelCapacity: 1}) + re.NoError(err) + defer w.Close() + re.ErrorIs(w.Err(), context.Canceled) + + s.manager.loadInitialGCStates(w, 1) + re.False(iterationStarted.Load()) +} + func (s *gcStateManagerTestSuite) TestGCStateWatchLiveSuppressesPausedInitial() { re := s.Require() const keyspaceID = uint32(2) diff --git a/pkg/gc/gc_state_watcher.go b/pkg/gc/gc_state_watcher.go index 8184a2c1f92..9f49d3a7f73 100644 --- a/pkg/gc/gc_state_watcher.go +++ b/pkg/gc/gc_state_watcher.go @@ -92,7 +92,9 @@ const ( watcherTerminationInitError gcStateWatcherTerminationReason = "init_error" ) -// GCStateWatcher receives a consistent initial view followed by live GC state changes. +// GCStateWatcher merges ordered initial batches and live GC state changes for one stream. +// The initial scan is not globally atomic and may interleave with live delivery. For each +// keyspace, the merge prevents an older initial state from following a newer live state. // // A watcher supports one receiving goroutine. Close may be called concurrently with // receiving and with manager-owned lifecycle operations. @@ -136,8 +138,9 @@ func (w *GCStateWatcher) receiveOne(block bool) (GCStateChange, bool, error) { keyspaceID, ok := change.KeyspaceID() if ok { if _, dirty := w.dirtyDuringInit[keyspaceID]; dirty { - // For each scope, consumers observe either initial v1 followed by live v2, - // or live v2 with the later-arriving initial v1 suppressed. + // Registration precedes the initial scan, so a post-registration live v2 may + // race with initial v1. If v1 is consumed first, delivery is v1 then v2; if + // v2 is consumed first, this later v1 is suppressed and delivery is v2 only. continue } } @@ -203,7 +206,7 @@ func (w *GCStateWatcher) receiveOne(block bool) (GCStateChange, bool, error) { w.dirtyDuringInit = nil continue } - w.pendingInit = append([]GCStateChange(nil), batch...) + w.pendingInit = batch } } @@ -287,6 +290,10 @@ func (m *GCStateManager) registerGCStateWatcher( } func (m *GCStateManager) loadInitialGCStates(watcher *GCStateWatcher, batchSize int) { + if watcher.Err() != nil { + return + } + batch := make([]GCStateChange, 0, batchSize) stopped := false flush := func() bool { diff --git a/tests/server/gc/gc_test.go b/tests/server/gc/gc_test.go index 3f9b45669d4..77e6194ada1 100644 --- a/tests/server/gc/gc_test.go +++ b/tests/server/gc/gc_test.go @@ -16,6 +16,7 @@ package gc import ( "context" + "errors" "math" "slices" "sync" @@ -23,9 +24,11 @@ import ( "testing" "time" + "github.com/prometheus/client_golang/prometheus" "github.com/stretchr/testify/require" "go.uber.org/goleak" "google.golang.org/grpc/codes" + "google.golang.org/grpc/metadata" "google.golang.org/grpc/status" "github.com/pingcap/failpoint" @@ -36,6 +39,7 @@ import ( "github.com/tikv/pd/pkg/ratelimit" "github.com/tikv/pd/pkg/utils/testutil" "github.com/tikv/pd/pkg/versioninfo/kerneltype" + "github.com/tikv/pd/server" "github.com/tikv/pd/server/config" "github.com/tikv/pd/tests" ) @@ -261,6 +265,61 @@ func advanceWatchGCStatesTxnSafePoint( require.Equal(t, target, response.GetNewTxnSafePoint()) } +type failingWatchGCStatesServer struct { + ctx context.Context + sendErr error +} + +func (s *failingWatchGCStatesServer) Send(*pdpb.WatchGCStatesResponse) error { + return s.sendErr +} + +func (*failingWatchGCStatesServer) SetHeader(metadata.MD) error { return nil } +func (*failingWatchGCStatesServer) SendHeader(metadata.MD) error { return nil } +func (*failingWatchGCStatesServer) SetTrailer(metadata.MD) {} + +func (s *failingWatchGCStatesServer) Context() context.Context { + return s.ctx +} + +func (*failingWatchGCStatesServer) SendMsg(any) error { return nil } +func (*failingWatchGCStatesServer) RecvMsg(any) error { return nil } + +func prometheusMetricValue(t *testing.T, name string, labels map[string]string) float64 { + t.Helper() + families, err := prometheus.DefaultGatherer.Gather() + require.NoError(t, err) + for _, family := range families { + if family.GetName() != name { + continue + } + for _, metric := range family.GetMetric() { + if len(metric.GetLabel()) != len(labels) { + continue + } + matches := true + for _, pair := range metric.GetLabel() { + if labels[pair.GetName()] != pair.GetValue() { + matches = false + break + } + } + if !matches { + continue + } + if gauge := metric.GetGauge(); gauge != nil { + return gauge.GetValue() + } + if counter := metric.GetCounter(); counter != nil { + return counter.GetValue() + } + require.FailNow(t, "metric has unsupported type", name) + } + } + require.FailNow(t, "metric not found", name) + return 0 +} + func TestGCOperations(t *testing.T) { re := require.New(t) ctx, cancel := context.WithCancel(context.Background()) @@ -1103,6 +1162,62 @@ func TestWatchGCStatesRequestPreflight(t *testing.T) { } } +func TestWatchGCStatesSendFailureCleansUpPublicHandler(t *testing.T) { + re := require.New(t) + cluster := newWatchGCStatesCluster(t, 1, true) + leaderServer := cluster.GetLeaderServer() + re.NotNil(leaderServer) + pdServer := leaderServer.GetServer() + + options := pdServer.GetServiceMiddlewarePersistOptions() + previousConfig := options.GetGRPCRateLimitConfig().Clone() + enabledConfig := previousConfig.Clone() + enabledConfig.EnableRateLimit = true + options.SetGRPCRateLimitConfig(enabledConfig) + limiter := pdServer.GetGRPCRateLimiter() + limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(1)) + t.Cleanup(func() { + limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(0)) + options.SetGRPCRateLimitConfig(previousConfig) + }) + + activeBefore := prometheusMetricValue(t, "pd_gc_watcher_count", nil) + clientCancelBefore := prometheusMetricValue(t, "pd_gc_watcher_termination_total", map[string]string{"reason": "client_cancel"}) + registration := enableWatchGCStatesRegistrationPoint(t) + sendErr := errors.New("send failed") + streamCtx, cancelStream := context.WithTimeout(context.Background(), 20*time.Second) + defer cancelStream() + stream := &failingWatchGCStatesServer{ctx: streamCtx, sendErr: sendErr} + handlerDone := make(chan error, 1) + go func() { + handlerDone <- (&server.GrpcServer{Server: pdServer}).WatchGCStates(&pdpb.WatchGCStatesRequest{ + Header: testutil.NewRequestHeader(leaderServer.GetClusterID()), + SkipLoadingInitial: true, + }, stream) + }() + + registration.wait(t) + registration.disable(re) + re.Equal(activeBefore+1, prometheusMetricValue(t, "pd_gc_watcher_count", nil)) + limit, current := limiter.GetConcurrencyLimiterStatus("WatchGCStates") + re.Equal(uint64(1), limit) + re.Equal(uint64(1), current) + + _, err := pdServer.GetGCStateManager().AdvanceTxnSafePoint(constant.NullKeyspaceID, 10, time.Now()) + re.NoError(err) + select { + case err := <-handlerDone: + re.Same(sendErr, err) + case <-time.After(5 * time.Second): + re.FailNow("WatchGCStates handler did not return after the send failure") + } + + re.Equal(activeBefore, prometheusMetricValue(t, "pd_gc_watcher_count", nil)) + re.Equal(clientCancelBefore+1, prometheusMetricValue(t, "pd_gc_watcher_termination_total", map[string]string{"reason": "client_cancel"})) + _, current = limiter.GetConcurrencyLimiterStatus("WatchGCStates") + re.Zero(current) +} + func TestWatchGCStatesHoldsRateLimitTokenForStreamLifetime(t *testing.T) { re := require.New(t) cluster := newWatchGCStatesCluster(t, 1, true) From f8396f12c9325a5a1b3013f9b703d5eb2bde778d Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 16 Sep 2026 13:36:07 +0800 Subject: [PATCH 13/25] gc: preserve watch ordering across initial and live states Prevent queued live updates from following newer initial states on a watch stream. Drain only the live prefix present when an initial batch is received so later arrivals cannot indefinitely delay initial data. Add deterministic coverage for ordering, bounded initial progress, removals, and cancellation. Signed-off-by: Wenxuan Zhang --- .../2026-09-03-watch-gc-states-design.md | 14 +- pkg/gc/gc_state_watcher.go | 47 +++- pkg/gc/gc_state_watcher_test.go | 158 +++++++++++ server/gc_service.go | 24 ++ server/gc_service_test.go | 250 ++++++++++++++++++ 5 files changed, 478 insertions(+), 15 deletions(-) diff --git a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md index 21e5dc210d3..dce723fa9c1 100644 --- a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md +++ b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md @@ -37,7 +37,7 @@ The first implementation intentionally excludes adjacent features that are not n The stream is a sequence of self-contained changes. An `upsert` replaces the consumer's entire state for its scope, and a `removed` change deletes that scope from the consumer's materialized view. Upserts do not contain GC barriers; barrier-only mutations do not directly produce changes. -For `skip_loading_initial=false`, PD registers the live listener before starting the initial scan. Initial and live changes may be interleaved, and the initial scan is not a cross-keyspace transaction. For each individual keyspace, however, the server suppresses an initial value if a post-registration live value for the same scope has already been emitted. The stream therefore cannot regress from a live value `v2` to an older initial value `v1`. +For `skip_loading_initial=false`, PD registers the live listener before starting the initial scan. Initial and live changes may be interleaved, and the initial scan is not a cross-keyspace transaction. Before emitting a newly acquired initial batch, the server drains the live changes already queued when it acquired that batch. For each individual keyspace, it suppresses an initial value if a post-registration live value for the same scope has already been emitted. These rules prevent regression both from a newer initial value to an older queued live value and from a newer live value to an older initial value. For `skip_loading_initial=true`, PD sends only effective safe-point changes produced after registration. This mode does not provide continuity with an earlier stream and is unsuitable for constructing a complete view on its own. A client establishing its first complete view or recovering from a disconnected stream uses `skip_loading_initial=false`. @@ -67,7 +67,7 @@ Watcher mechanics live in a focused file such as `pkg/gc/gc_state_watcher.go`. E - `liveCh`, a bounded channel of individual live changes. - `initDone`, which is owned by the merge consumer and becomes true when initial loading was skipped or after the closed `initCh` has been fully drained. - A cause-aware cancellation mechanism used for both cleanup and error reporting. -- Merge state, including the set of scopes made dirty by live delivery while initial loading is active. +- Merge state, including the pending initial batch, the remaining count of live changes that must precede it, and the set of scopes made dirty by live delivery while initial loading is active. Only the initial loader writes to and closes `initCh`. Publishers write to `liveCh` only while holding the manager mutex, but `liveCh` is not closed; watcher cancellation is the termination signal. This ownership rule avoids send-versus-close races. @@ -113,15 +113,18 @@ Publication to each `liveCh` is non-blocking. If a watcher's channel is full, th A single consumer merges `initCh` and `liveCh`. While initial loading is active, it maintains `dirtyDuringInit`, a set keyed by keyspace scope: +- After receiving an initial batch, the consumer snapshots `len(liveCh)` and drains exactly that queued FIFO prefix before emitting any change from the batch. The remaining prefix count persists across `RecvBatch` calls. Live changes arriving after the snapshot do not extend the prefix, so they cannot indefinitely postpone unrelated states in the pending initial batch. - When the consumer emits a live change, it marks that scope dirty. - When it encounters an initial upsert whose scope is already dirty, it drops the initial upsert. -- After `initCh` is closed and all of its buffered batches are consumed, the consumer releases the dirty set and disables the closed channel because every later change is live and already ordered by `liveCh`. +- After `initCh` is closed and all of its buffered batches, the pending batch, and its live prefix are consumed, the consumer releases the dirty set and disables the closed channel because every later change is live and already ordered by `liveCh`. There are two possible observations for an initial value `v1` and a later live value `v2`. If the consumer receives `v1` first, it emits `v1` followed by `v2`. If it receives `v2` first, it marks the scope dirty and suppresses `v1`. In neither case can it emit `v2` followed by `v1`. +The initial scan can also read a newer value than an already queued live change. For example, live values `10` and `20` can be queued before the loader reads initial `20`. If the merge selects that initial batch first, draining the captured live prefix emits `10, 20` and suppresses the initial duplicate, preventing `20, 10, 20`. Mutation cache updates and publication are serialized by `GCStateManager.mu`: every strictly older live change has been published before the loader can observe the newer cache value. That newer value's own live publication may still follow its cache store, which is safe because it cannot regress the initial value. Taking the queue-length snapshot after acquiring the batch therefore covers every older live change that has not already been consumed, without adding locks or another queue. + The same rule supports the future `removed` producer: a live removal marks the scope dirty, preventing an older initial upsert from recreating it. Live changes retain FIFO order because mutation publication is serialized by the manager mutex and each watcher has a single live channel and a single consumer. -The implementation must explain this timing guarantee next to the merge logic, including the registration boundary and both possible delivery orders. A deterministic test pauses initial loading after reading `v1` but before placing it on `initCh`, advances the same keyspace to `v2`, observes `v2`, resumes initial loading, and verifies that `v1` is never emitted afterward. +The implementation must explain these timing guarantees next to the merge logic, including the registration boundary and both directions of initial/live overlap. A deterministic test pauses initial loading after reading `v1` but before placing it on `initCh`, advances the same keyspace to `v2`, observes `v2`, resumes initial loading, and verifies that `v1` is never emitted afterward. Receiver tests also force initial-batch acquisition with older live changes queued, verify prefix ordering across response boundaries, and keep adding live changes to verify that the pending initial batch still makes progress. ## Capacity and backpressure @@ -198,6 +201,7 @@ The domain tests cover: - Effective safe-point changes publish complete upserts exactly once from the shared modern and legacy mutation paths; no-op and failed mutations do not publish. - Current barrier-only mutations produce no change, while any operation that changes an effective safe point does. - Initial-first delivery emits `v1` followed by `v2`; a deterministic paused-initial test emits `v2` and suppresses the later initial `v1`. +- Acquiring a newer initial batch drains the already queued live prefix first, retains prefix progress across `RecvBatch` calls, and does not let later live arrivals postpone the pending initial batch. - Filling watcher A's `liveCh` terminates A without delaying watcher B; A can reconnect and rebuild. - Initial iteration failure, caller cancellation, and concurrent deregistration terminate without goroutine or registry leaks. - A full `initCh` backpressures only that watcher's initial iterator and never waits on the channel while holding `GCStateManager.mu`. @@ -227,7 +231,7 @@ The API can be rolled out server-first because existing clients do not call the The implementation is complete when all of the following are true: - `WatchGCStates` serves initial states and local effective safe-point changes with the documented `skip_loading_initial` behavior. -- The deterministic ordering test proves that no older initial value follows a newer live value for the same scope on one stream. +- Deterministic ordering tests prove that neither an older initial value follows a newer live value nor an older queued live value follows a newer initial value for the same scope on one stream. - A full live queue terminates only the affected watcher without blocking GC-state mutation. - Ending or superseding a local leadership generation terminates its active streams. - Response batches observe the 1 MiB target using exact protobuf size accounting, except for the defined oversized-single-change case. diff --git a/pkg/gc/gc_state_watcher.go b/pkg/gc/gc_state_watcher.go index 9f49d3a7f73..57bd7217dcc 100644 --- a/pkg/gc/gc_state_watcher.go +++ b/pkg/gc/gc_state_watcher.go @@ -94,21 +94,22 @@ const ( // GCStateWatcher merges ordered initial batches and live GC state changes for one stream. // The initial scan is not globally atomic and may interleave with live delivery. For each -// keyspace, the merge prevents an older initial state from following a newer live state. +// keyspace, the merge prevents initial and live delivery from regressing to an older state. // // A watcher supports one receiving goroutine. Close may be called concurrently with // receiving and with manager-owned lifecycle operations. // nolint:revive // Keep GC in the name to match the established GCState domain API. type GCStateWatcher struct { - ctx context.Context - cancel context.CancelCauseFunc - manager *GCStateManager - id uint64 - initCh chan []GCStateChange - liveCh chan GCStateChange - initDone bool - pendingInit []GCStateChange - dirtyDuringInit map[uint32]struct{} + ctx context.Context + cancel context.CancelCauseFunc + manager *GCStateManager + id uint64 + initCh chan []GCStateChange + liveCh chan GCStateChange + initDone bool + pendingInit []GCStateChange + pendingLiveCount int + dirtyDuringInit map[uint32]struct{} } func newGCStateWatcher(parent context.Context, cfg gcStateWatchConfig, skipLoadingInitial bool) *GCStateWatcher { @@ -132,6 +133,17 @@ func (w *GCStateWatcher) receiveOne(block bool) (GCStateChange, bool, error) { return GCStateChange{}, false, err } + if w.pendingLiveCount > 0 { + // This watcher has one receiver, so every change counted when the + // initial batch was acquired is still queued until we consume it. + change := <-w.liveCh + w.pendingLiveCount-- + if keyspaceID, valid := change.KeyspaceID(); valid { + w.dirtyDuringInit[keyspaceID] = struct{}{} + } + return change, true, nil + } + for len(w.pendingInit) > 0 { change := w.pendingInit[0] w.pendingInit = w.pendingInit[1:] @@ -206,10 +218,25 @@ func (w *GCStateWatcher) receiveOne(block bool) (GCStateChange, bool, error) { w.dirtyDuringInit = nil continue } + failpoint.InjectCall("watchGCStatesInitialBatchReceived") w.pendingInit = batch + // Snapshot only after acquiring the initial batch. Registration precedes + // its scan, and mutations publish under the manager mutex before the next + // mutation can update the cache. Thus any live state older than this batch's + // initial state is already queued or consumed. The initial state's own live + // publication may still follow its cache store, but cannot cause regression. + // Drain this FIFO prefix first and suppress initial scopes it makes dirty. + // Keep the remaining count across RecvBatch calls; later arrivals must not + // extend the prefix and indefinitely postpone unrelated initial states. + w.pendingLiveCount = len(w.liveCh) } } +// Done returns a channel that is closed when the watcher terminates. +func (w *GCStateWatcher) Done() <-chan struct{} { + return w.ctx.Done() +} + // Err returns the first cause that terminated the watcher. func (w *GCStateWatcher) Err() error { return context.Cause(w.ctx) diff --git a/pkg/gc/gc_state_watcher_test.go b/pkg/gc/gc_state_watcher_test.go index e965e296ba1..9fd65012763 100644 --- a/pkg/gc/gc_state_watcher_test.go +++ b/pkg/gc/gc_state_watcher_test.go @@ -61,6 +61,140 @@ func TestGCStateWatcherLiveSuppressesOlderInitial(t *testing.T) { require.True(t, w.initDone) } +func TestGCStateWatcherQueuedLivePrecedesNewerInitial(t *testing.T) { + for _, maxChanges := range []int{1, 2, 4} { + t.Run(fmt.Sprintf("batch-size-%d", maxChanges), func(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 2, liveChannelCapacity: 2}, false) + t.Cleanup(w.Close) + w.initCh <- []GCStateChange{ + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 20}), + NewGCStateUpsert(GCState{KeyspaceID: 8, TxnSafePoint: 1}), + } + w.initCh <- []GCStateChange{ + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 20}), + NewGCStateUpsert(GCState{KeyspaceID: 9, TxnSafePoint: 1}), + } + close(w.initCh) + queueLiveWhenInitialReceived(t, w, + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 10}), + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 20}), + ) + + want := []GCState{ + {KeyspaceID: 7, TxnSafePoint: 10}, + {KeyspaceID: 7, TxnSafePoint: 20}, + {KeyspaceID: 8, TxnSafePoint: 1}, + {KeyspaceID: 9, TxnSafePoint: 1}, + } + for offset := 0; offset < len(want); { + got, err := w.RecvBatch(maxChanges) + require.NoError(t, err) + require.Len(t, got, min(maxChanges, len(want)-offset)) + for _, change := range got { + require.Equal(t, want[offset], mustUpsert(t, change)) + offset++ + } + } + _, ok, err := w.receiveOne(false) + require.NoError(t, err) + require.False(t, ok, "initial duplicates must remain suppressed through the closed channel's buffered batches") + }) + } +} + +func TestGCStateWatcherLaterLiveDoesNotPostponePendingInitial(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 4}, false) + t.Cleanup(w.Close) + w.initCh <- []GCStateChange{ + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 20}), + NewGCStateUpsert(GCState{KeyspaceID: 8, TxnSafePoint: 1}), + NewGCStateUpsert(GCState{KeyspaceID: 9, TxnSafePoint: 1}), + } + close(w.initCh) + queueLiveWhenInitialReceived(t, w, + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 10}), + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 20}), + ) + + for i, want := range []GCState{ + {KeyspaceID: 7, TxnSafePoint: 10}, + {KeyspaceID: 7, TxnSafePoint: 20}, + {KeyspaceID: 8, TxnSafePoint: 1}, + {KeyspaceID: 9, TxnSafePoint: 1}, + } { + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Len(t, got, 1) + require.Equal(t, want, mustUpsert(t, got[0])) + // Keep the live queue nonempty after the first receive. These arrivals + // must not postpone the unrelated states in the acquired initial batch. + w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: uint64(30 + i*10)}) + } + got, err := w.RecvBatch(4) + require.NoError(t, err) + require.Len(t, got, 4) + for i, want := range []uint64{30, 40, 50, 60} { + require.Equal(t, GCState{KeyspaceID: 7, TxnSafePoint: want}, mustUpsert(t, got[i])) + } +} + +func TestGCStateWatcherQueuedRemovalSuppressesInitial(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + t.Cleanup(w.Close) + w.initCh <- []GCStateChange{ + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 10}), + NewGCStateUpsert(GCState{KeyspaceID: 8, TxnSafePoint: 1}), + } + close(w.initCh) + queueLiveWhenInitialReceived(t, w, NewGCStateRemoved(7)) + + got, err := w.RecvBatch(3) + require.NoError(t, err) + require.Len(t, got, 2) + removed, ok := got[0].RemovedKeyspaceID() + require.True(t, ok) + require.Equal(t, uint32(7), removed) + require.Equal(t, GCState{KeyspaceID: 8, TxnSafePoint: 1}, mustUpsert(t, got[1])) +} + +func TestGCStateWatcherCancellationDiscardsPendingLivePrefix(t *testing.T) { + w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) + t.Cleanup(w.Close) + w.initCh <- []GCStateChange{ + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 20}), + NewGCStateUpsert(GCState{KeyspaceID: 8, TxnSafePoint: 1}), + } + close(w.initCh) + queueLiveWhenInitialReceived(t, w, + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 10}), + NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 20}), + ) + got, err := w.RecvBatch(1) + require.NoError(t, err) + require.Equal(t, GCState{KeyspaceID: 7, TxnSafePoint: 10}, mustUpsert(t, got[0])) + + want := errors.New("watch terminated with pending initial and live changes") + w.cancel(want) + got, err = w.RecvBatch(3) + require.ErrorIs(t, err, want) + require.Nil(t, got) +} + +func queueLiveWhenInitialReceived(t *testing.T, w *GCStateWatcher, changes ...GCStateChange) { + t.Helper() + const name = "github.com/tikv/pd/pkg/gc/watchGCStatesInitialBatchReceived" + // Recreate the reachable merge state after select picks an initial batch + // while older live changes are queued. Enqueue in the hook only to force + // that branch deterministically, without depending on select randomness. + require.NoError(t, failpoint.EnableCall(name, func() { + for _, change := range changes { + w.liveCh <- change + } + changes = nil + })) + t.Cleanup(func() { require.NoError(t, failpoint.Disable(name)) }) +} + func TestGCStateWatcherRemovedSuppressesInitial(t *testing.T) { w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) w.liveCh <- NewGCStateRemoved(7) @@ -351,3 +485,27 @@ func mustUpsert(t testing.TB, change GCStateChange) GCState { require.True(t, ok) return state } + +func TestGCStateWatcherDonePublishesFirstCause(t *testing.T) { + ctx, cancel := context.WithCancelCause(context.Background()) + defer cancel(context.Canceled) + w := newGCStateWatcher(ctx, gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 1}, true) + defer w.Close() + done := w.Done() + require.Equal(t, done, w.Done()) + select { + case <-done: + require.FailNow(t, "watcher terminated before cancellation") + default: + } + cancel(errs.ErrNotLeader) + select { + case <-done: + case <-time.After(5 * time.Second): + require.FailNow(t, "watcher termination was not notified") + } + require.ErrorIs(t, w.Err(), errs.ErrNotLeader) + w.Close() + require.ErrorIs(t, w.Err(), errs.ErrNotLeader) + require.Equal(t, done, w.Done()) +} diff --git a/server/gc_service.go b/server/gc_service.go index 8df99f9d436..f4df26e59ba 100644 --- a/server/gc_service.go +++ b/server/gc_service.go @@ -34,6 +34,7 @@ import ( "github.com/tikv/pd/pkg/keyspace/constant" "github.com/tikv/pd/pkg/storage/endpoint" "github.com/tikv/pd/pkg/utils/grpcutil" + "github.com/tikv/pd/pkg/utils/logutil" "github.com/tikv/pd/pkg/utils/tsoutil" "github.com/tikv/pd/pkg/utils/typeutil" ) @@ -45,6 +46,7 @@ const ( type gcStateChangeReceiver interface { RecvBatch(maxChanges int) ([]gc.GCStateChange, error) + Done() <-chan struct{} Err() error } @@ -610,6 +612,28 @@ func watchGCStatesErrorToStatus(err error) error { } func serveWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error { + if err := receiver.Err(); err != nil { + return watchGCStatesErrorToStatus(err) + } + resultCh := make(chan error, 1) + go func() { + defer logutil.LogPanic() + resultCh <- sendWatchGCStates(receiver, stream, maxResponseSize) + }() + // Do not join the worker here: returning lets gRPC tear down the transport + // stream, which interrupts a Send blocked on flow control. + select { + case err := <-resultCh: + if cause := receiver.Err(); cause != nil { + return watchGCStatesErrorToStatus(cause) + } + return err + case <-receiver.Done(): + return watchGCStatesErrorToStatus(receiver.Err()) + } +} + +func sendWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error { for { changes, err := receiver.RecvBatch(watchGCStatesRecvBatchSize) if err != nil { diff --git a/server/gc_service_test.go b/server/gc_service_test.go index 5f6d35f53c4..50f09f98113 100644 --- a/server/gc_service_test.go +++ b/server/gc_service_test.go @@ -17,12 +17,18 @@ package server import ( "context" "errors" + "math" + "net" "testing" + "time" "github.com/stretchr/testify/require" + "google.golang.org/grpc" "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" "google.golang.org/grpc/metadata" "google.golang.org/grpc/status" + "google.golang.org/grpc/test/bufconn" "github.com/pingcap/kvproto/pkg/pdpb" @@ -145,6 +151,8 @@ func (r *fakeGCStateChangeReceiver) RecvBatch(maxChanges int) ([]gc.GCStateChang return batch, nil } +func (*fakeGCStateChangeReceiver) Done() <-chan struct{} { return nil } + func (r *fakeGCStateChangeReceiver) Err() error { return r.terminalErr } @@ -246,3 +254,245 @@ func TestWatchGCStatesErrorToStatus(t *testing.T) { }) } } + +// cancelableGCStateReceiver keeps batch state in the sending worker and exposes +// the terminal cause through a context, which the supervisor can read safely. +type cancelableGCStateReceiver struct { + ctx context.Context + changes []gc.GCStateChange + receiveStarted chan struct{} + receiveExited chan struct{} +} + +func (r *cancelableGCStateReceiver) Done() <-chan struct{} { return r.ctx.Done() } +func (r *cancelableGCStateReceiver) Err() error { return context.Cause(r.ctx) } +func (r *cancelableGCStateReceiver) RecvBatch(maxChanges int) ([]gc.GCStateChange, error) { + if err := r.Err(); err != nil { + return nil, err + } + if len(r.changes) == 0 { + if r.receiveStarted != nil { + close(r.receiveStarted) + defer close(r.receiveExited) + } + <-r.Done() + return nil, r.Err() + } + n := min(maxChanges, len(r.changes)) + batch := r.changes[:n] + r.changes = r.changes[n:] + return batch, nil +} + +func waitWatchGCStatesSignal(t *testing.T, signal <-chan struct{}, message string) { + t.Helper() + select { + case <-signal: + case <-time.After(5 * time.Second): + require.FailNow(t, message) + } +} + +func TestServeWatchGCStatesCancellationUnblocksHandler(t *testing.T) { + for _, tc := range []struct { + name string + cause error + code codes.Code + }{ + {"leader loss", errs.ErrNotLeader, codes.Unavailable}, + {"slow consumer", errs.ErrGCStateWatcherSlowConsumer, codes.ResourceExhausted}, + } { + t.Run(tc.name, func(t *testing.T) { + streamCtx, cancelStream := context.WithCancel(context.Background()) + receiverCtx, cancelReceiver := context.WithCancelCause(streamCtx) + receiver := &cancelableGCStateReceiver{ctx: receiverCtx, changes: []gc.GCStateChange{gc.NewGCStateUpsert(gc.GCState{KeyspaceID: 7, TxnSafePoint: 10})}} + sendStarted, sendExited := make(chan struct{}), make(chan struct{}) + stream := &fakeWatchGCStatesServer{ctx: streamCtx, sendHook: func(*pdpb.WatchGCStatesResponse) error { + close(sendStarted) + defer close(sendExited) + <-streamCtx.Done() + return streamCtx.Err() + }} + handlerDone := make(chan struct{}) + var handlerErr error + t.Cleanup(func() { + cancelReceiver(context.Canceled) + cancelStream() + waitWatchGCStatesSignal(t, handlerDone, "handler did not clean up") + select { + case <-sendStarted: + waitWatchGCStatesSignal(t, sendExited, "send did not clean up") + default: + } + }) + go func() { defer close(handlerDone); handlerErr = serveWatchGCStates(receiver, stream, 1024) }() + waitWatchGCStatesSignal(t, sendStarted, "send did not start") + cancelReceiver(tc.cause) + waitWatchGCStatesSignal(t, handlerDone, "handler did not return while send was blocked") + require.Equal(t, tc.code, status.Code(handlerErr)) + require.NoError(t, streamCtx.Err()) + select { + case <-sendExited: + require.FailNow(t, "send exited before transport teardown") + default: + } + cancelStream() + waitWatchGCStatesSignal(t, sendExited, "send did not exit after transport teardown") + }) + } +} + +type observedWatchGCStatesStream struct { + pdpb.PD_WatchGCStatesServer + sentWireBytes int + observed bool + blockedSendStarted chan struct{} + blockedSendExited chan struct{} +} + +func (s *observedWatchGCStatesStream) Send(response *pdpb.WatchGCStatesResponse) error { + // grpc-go v1.82.1 starts with 64 KiB write quota. With the client's static + // 64 KiB receive window and no Recv calls, this next send cannot regain quota. + const receiveWindow = 64 << 10 + const writeQuota = 64 << 10 + if !s.observed && s.sentWireBytes >= receiveWindow+writeQuota { + s.observed = true + close(s.blockedSendStarted) + defer close(s.blockedSendExited) + } + err := s.PD_WatchGCStatesServer.Send(response) + if err == nil { + s.sentWireBytes += response.Size() + 5 + } + return err +} + +type watchGCStatesTransportServer struct { + pdpb.UnimplementedPDServer + changes []gc.GCStateChange + cancelReceiver chan context.CancelCauseFunc + handlerResult chan error + blockedSendStarted chan struct{} + blockedSendExited chan struct{} +} + +func (s *watchGCStatesTransportServer) WatchGCStates(_ *pdpb.WatchGCStatesRequest, stream pdpb.PD_WatchGCStatesServer) error { + ctx, cancel := context.WithCancelCause(stream.Context()) + defer cancel(context.Canceled) + s.cancelReceiver <- cancel + receiver := &cancelableGCStateReceiver{ctx: ctx, changes: s.changes} + observed := &observedWatchGCStatesStream{PD_WatchGCStatesServer: stream, blockedSendStarted: s.blockedSendStarted, blockedSendExited: s.blockedSendExited} + err := serveWatchGCStates(receiver, observed, maxWatchGCStatesResponseSize) + s.handlerResult <- err + return err +} + +func TestWatchGCStatesTransportCancellationUnblocksSend(t *testing.T) { + for _, tc := range []struct { + name string + cause error + code codes.Code + }{ + {"leader loss", errs.ErrNotLeader, codes.Unavailable}, + {"slow consumer", errs.ErrGCStateWatcherSlowConsumer, codes.ResourceExhausted}, + } { + t.Run(tc.name, func(t *testing.T) { + service := &watchGCStatesTransportServer{ + cancelReceiver: make(chan context.CancelCauseFunc, 1), handlerResult: make(chan error, 1), + blockedSendStarted: make(chan struct{}), blockedSendExited: make(chan struct{}), + } + for i := range 16 * 1024 { + service.changes = append(service.changes, gc.NewGCStateUpsert(gc.GCState{ + KeyspaceID: uint32(i), IsKeyspaceLevel: true, TxnSafePoint: math.MaxUint64, GCSafePoint: math.MaxUint64 - 1, + })) + } + listener := bufconn.Listen(1 << 20) + transport := grpc.NewServer() + pdpb.RegisterPDServer(transport, service) + serveErr := make(chan error, 1) + go func() { serveErr <- transport.Serve(listener) }() + t.Cleanup(func() { transport.Stop(); require.NoError(t, <-serveErr) }) + conn, err := grpc.NewClient("passthrough:///bufnet", + grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) { return listener.Dial() }), + grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithStaticStreamWindowSize(64<<10), grpc.WithStaticConnWindowSize(64<<10), + ) + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, conn.Close()) }) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + t.Cleanup(cancel) + stream, err := pdpb.NewPDClient(conn).WatchGCStates(ctx, &pdpb.WatchGCStatesRequest{}) + require.NoError(t, err) + waitWatchGCStatesSignal(t, service.blockedSendStarted, "transport send did not reach exhausted quota") + select { + case <-service.blockedSendExited: + require.FailNow(t, "transport send unexpectedly completed") + default: + } + cancelReceiver := <-service.cancelReceiver + cancelReceiver(tc.cause) + select { + case err := <-service.handlerResult: + require.Equal(t, tc.code, status.Code(err)) + case <-time.After(5 * time.Second): + require.FailNow(t, "handler did not return while transport send was blocked") + } + waitWatchGCStatesSignal(t, service.blockedSendExited, "transport teardown did not unblock send") + require.NoError(t, ctx.Err()) + for err == nil { + _, err = stream.Recv() + } + require.Equal(t, tc.code, status.Code(err)) + }) + } +} + +func TestServeWatchGCStatesCancellationBeforeServing(t *testing.T) { + ctx, cancel := context.WithCancelCause(context.Background()) + cancel(errs.ErrNotLeader) + receiver := &cancelableGCStateReceiver{ctx: ctx, changes: []gc.GCStateChange{gc.NewGCStateRemoved(7)}} + stream := &fakeWatchGCStatesServer{} + err := serveWatchGCStates(receiver, stream, 1024) + require.Equal(t, codes.Unavailable, status.Code(err)) + require.Empty(t, stream.sent) +} + +func TestServeWatchGCStatesCancellationWhileReceiving(t *testing.T) { + for _, parentCanceled := range []bool{false, true} { + name := "watcher" + if parentCanceled { + name = "parent stream" + } + t.Run(name, func(t *testing.T) { + streamCtx, cancelStream := context.WithCancel(context.Background()) + ctx, cancelReceiver := context.WithCancelCause(streamCtx) + receiver := &cancelableGCStateReceiver{ctx: ctx, receiveStarted: make(chan struct{}), receiveExited: make(chan struct{})} + stream := &fakeWatchGCStatesServer{ctx: streamCtx} + handlerDone := make(chan struct{}) + var handlerErr error + t.Cleanup(func() { + cancelReceiver(context.Canceled) + cancelStream() + waitWatchGCStatesSignal(t, handlerDone, "handler did not clean up") + select { + case <-receiver.receiveStarted: + waitWatchGCStatesSignal(t, receiver.receiveExited, "receive did not clean up") + default: + } + }) + go func() { defer close(handlerDone); handlerErr = serveWatchGCStates(receiver, stream, 1024) }() + waitWatchGCStatesSignal(t, receiver.receiveStarted, "receive did not start") + want := codes.Unavailable + if parentCanceled { + cancelStream() + want = codes.Canceled + } else { + cancelReceiver(errs.ErrNotLeader) + } + waitWatchGCStatesSignal(t, handlerDone, "handler did not return after cancellation") + waitWatchGCStatesSignal(t, receiver.receiveExited, "receive did not return after cancellation") + require.Equal(t, want, status.Code(handlerErr)) + require.Empty(t, stream.sent) + }) + } +} From 1c3044645fc5dc52f593df90e42649166e20f67c Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 16 Sep 2026 14:48:00 +0800 Subject: [PATCH 14/25] gc: interrupt blocked watch sends on watcher termination Let WatchGCStates return on watcher termination even when gRPC flow control blocks sending, so leadership loss and slow-consumer eviction release the stream's concurrency-limit token promptly. Keep sending in one worker per stream with a buffered result channel. The handler must return before joining the worker because gRPC teardown provides the transport cancellation that releases a blocked send. Cover both termination causes with real transport flow control and public-handler cleanup tests, preserving ordering and queue bounds. Signed-off-by: Wenxuan Zhang --- ...2026-09-16-watch-gc-states-blocked-send.md | 242 ++++++++++++++++++ .../2026-09-03-watch-gc-states-design.md | 8 +- tests/server/gc/gc_test.go | 150 ++++++++++- 3 files changed, 387 insertions(+), 13 deletions(-) create mode 100644 docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md diff --git a/docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md b/docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md new file mode 100644 index 00000000000..18a49b53d4b --- /dev/null +++ b/docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md @@ -0,0 +1,242 @@ +# WatchGCStates Blocked Send Cancellation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let `WatchGCStates` return and release its concurrency-limit token when its watcher terminates, even while gRPC flow control blocks a response send, and verify that the sending goroutine exits after RPC teardown. + +**Architecture:** Run the existing receive/convert/send loop in one worker goroutine per RPC. The handler supervises the worker through a buffered result channel and the watcher's termination channel. The handler returns on watcher termination; gRPC then tears down the transport stream and unblocks the worker. + +**Tech Stack:** Go 1.25+, the repository-pinned grpc-go v1.82.1, existing PD GC watchers, failpoint-aware Go tests, and `bufconn` for transport coverage. + +**Spec:** [WatchGCStates server design](../specs/2026-09-03-watch-gc-states-design.md), especially “Lifecycle and errors” and “GC service”, refined by the decisions below. This document is the implementation handoff for the second review finding on PR #11264. The first finding, initial/live ordering, was fixed in `65c7ca00b`; preserve that fix, including `pendingLiveCount`. + +**Status:** Implemented and verified against base `65c7ca00b`. Both fake-send and real-transport regressions failed against the original synchronous handler, then passed with the fix. Targeted race tests passed in `pkg/gc`, `server`, and `tests/server/gc`; the real-transport regression passed 20 repeated race runs, and affected-package basic tests passed. Independent review found no blocking issues. A full `make check` attempt passed module tidying and formatting but stopped because `golangci-lint` is unavailable in the local environment; the remaining checks still need CI or a fully provisioned environment. The task descriptions below retain the implementation and acceptance instructions for reference. + +## Global constraints + +- Limit production changes to `WatchGCStates` and its watcher termination notification. Other streaming RPCs, generic gRPC infrastructure, protobuf, and dependency versions are outside this task. +- Keep `rateLimitCheck()` and both deferred cleanup operations in the public `WatchGCStates` handler. Its caller-derived rate-limit label must remain `WatchGCStates`. +- Keep one receiving/sending worker per RPC and at most one `Send` in progress. Keep all existing per-response terminal checks, response-size limits, ordering, queue capacities, and manager lock scopes. +- Add no per-send goroutine, timer, response queue, or general send timeout. An open stream without a watcher termination event remains governed by the existing policy. +- The worker result channel has capacity **1**. The handler returns without joining the worker; RPC teardown supplies the cancellation needed to finish a blocked `Send`. +- Preserve error mappings and watcher termination metrics. Use `Unavailable` for leader loss, `ResourceExhausted` for slow consumers, and context status codes for caller cancellation/deadline. Preserve a raw send/conversion error when no watcher terminal cause has been recorded. +- Follow repository `AGENTS.md`. Run tests through failpoint-aware targets and restore failpoints before editing, reviewing diffs, or committing. If delegating, one owner coordinates test runs and edits in this shared worktree. + +## Decision and lifetime ownership + +`GCStateWatcher` owns a child context derived from `stream.Context()`. Canceling that child wakes `RecvBatch`, but does not cancel the parent transport context observed by gRPC `Send`. The existing synchronous handler cannot reach its next terminal check while sending is blocked. + +The chosen design makes the handler independently responsive to watcher termination: + +```text +watcher terminates + -> supervisor returns the mapped terminal cause + -> public handler runs watcher.Close() and releases its rate-limit token + -> gRPC processes the handler return and closes the transport stream + -> blocked Send returns + -> worker publishes its result into the buffered channel and exits +``` + +The worker may briefly outlive the handler. It retains the response it is sending until `Send` returns; no response is mutated or recycled during that interval. The worker must not wait for a consumer of its final result. Waiting for the worker inside a handler defer would reverse this dependency and deadlock the teardown path. + +The public handler remains the owner of watcher cleanup. The supervisor does not close the watcher itself. For the real caller, the watcher is always derived from the RPC context, so its `Done` notification also covers client cancellation. There is no need for a third, independently managed cancellation context or a redundant `stream.Context().Done()` select arm. + +When worker completion races with watcher cancellation, prefer an already-recorded watcher cause when choosing the return value. This makes a transport cancellation caused by server teardown less likely to obscure the domain reason. It does not establish a total order for events that have not yet been recorded; the watcher continues to own its first cancellation cause. + +### Verified transport assumption + +In the pinned grpc-go source, `serverStream.SendMsg` calls the transport write path, whose `writeQuota.get` waits on quota or the transport stream's cancellation. `Server.processStreamingRPC` handles the application handler's return via `WriteStatus`, and `http2Server.finishStream` cancels the transport stream before queuing its final trailers. Thus handler return can release the blocked send without waiting for the client to start consuming responses. + +Recheck these functions in the pinned module if its version changes: + +- `google.golang.org/grpc/stream.go`: `serverStream.SendMsg`. +- `google.golang.org/grpc/server.go`: `Server.processStreamingRPC`. +- `google.golang.org/grpc/internal/transport/http2_server.go`: `writeStatus`, `finishStream`, and `write`. +- `google.golang.org/grpc/internal/transport/flowcontrol.go`: `writeQuota.get`. +- `google.golang.org/grpc/internal/transport/controlbuf.go`: `loopyWriter.processData`, which replenishes write quota only for bytes allowed by stream flow control. + +This guarantees a local cancellation path, not immediate delivery of the final status to a client that refuses to read. Queued response data/trailers may still require client progress. Test handler return and worker exit before resuming client reads; inspect the client status afterward. + +## Files and interfaces + +| File | Responsibility | +| --- | --- | +| `pkg/gc/gc_state_watcher.go` | Add `Done() <-chan struct{}` exposing the existing watcher context's termination channel. | +| `pkg/gc/gc_state_watcher_test.go` | Verify notification, cause visibility, and existing lifecycle behavior. | +| `server/gc_service.go` | Add `Done` to `gcStateChangeReceiver`; make `serveWatchGCStates` the supervisor and move its existing loop unchanged into `sendWatchGCStates`. | +| `server/gc_service_test.go` | Adapt existing receiver fakes; add deterministic cancellation tests and real gRPC teardown coverage. | +| `tests/server/gc/gc_test.go` | Verify public-handler registration, cleanup, status, and concurrency-token release during blocked sending. | +| `docs/superpowers/specs/2026-09-03-watch-gc-states-design.md` | Document worker/supervisor ownership and distinguish unrecalled sends from handler lifetime. | + +`GCStateWatcher.Done` is concurrency-safe and returns the same channel on every call. Once that channel is closed, `Err()` returns the terminal cause. Reading `Done` must not allocate a goroutine, acquire a manager lock, or register a new callback. + +## Task 1: Add cancellation supervision and deterministic regressions + +**Interfaces produced:** `(*gc.GCStateWatcher).Done() <-chan struct{}`, the extended `gcStateChangeReceiver`, and the internal synchronous worker `sendWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error`. + +- [ ] Add a context-backed receiver fake for cancellation tests. Use `context.WithCancelCause(streamCtx)` and return its `Done`/`Cause` from the fake; keep mutable batch state exclusively in the receiving worker. Existing non-cancellation fakes may return a nil `Done` channel, which disables that select arm. Never write the existing fake's plain `terminalErr` concurrently with the supervisor reading it. + + A sufficient receiver for the blocked-send tests is: + + ```go + type cancelableGCStateReceiver struct { + ctx context.Context + changes []gc.GCStateChange + } + + func (r *cancelableGCStateReceiver) Done() <-chan struct{} { return r.ctx.Done() } + func (r *cancelableGCStateReceiver) Err() error { return context.Cause(r.ctx) } + + func (r *cancelableGCStateReceiver) RecvBatch(maxChanges int) ([]gc.GCStateChange, error) { + if err := r.Err(); err != nil { + return nil, err + } + if len(r.changes) == 0 { + <-r.Done() + return nil, r.Err() + } + n := min(maxChanges, len(r.changes)) + batch := r.changes[:n] + r.changes = r.changes[n:] + return batch, nil + } + ``` + +- [ ] Write `TestServeWatchGCStatesCancellationUnblocksHandler` as table-driven cases for `errs.ErrNotLeader`/`Unavailable` and `errs.ErrGCStateWatcherSlowConsumer`/`ResourceExhausted`. For each case, create one upsert and use `fakeWatchGCStatesServer.sendHook` with the following behavior: + + ```go + sendStarted := make(chan struct{}) + sendExited := make(chan struct{}) + stream.sendHook = func(*pdpb.WatchGCStatesResponse) error { + close(sendStarted) + defer close(sendExited) + <-streamCtx.Done() + return streamCtx.Err() + } + handlerDone := make(chan error, 1) + go func() { handlerDone <- serveWatchGCStates(receiver, stream, 1024) }() + ``` + + Wait for `sendStarted`, cancel only the receiver with the table's cause, then require `handlerDone` to return the expected status within **5 seconds**, while `streamCtx.Err()` remains nil and `sendExited` remains open. Only afterward call the stream cancel function to model gRPC teardown, and require `sendExited` to close. Register failure cleanup before starting the goroutine: cancel both contexts and release/wait for test goroutines with bounded waits even if an assertion fails. This fake deliberately models the transport cancellation boundary rather than pretending watcher cancellation directly ends `Send`. + +- [ ] Run the new test against the synchronous implementation and record the expected failure: the handler fails to return before the stream is canceled. The fake's extra `Done` method does not require changing the old production interface to compile this red test. Do not accept a test that first cancels the client/stream and only then checks handler completion. + +- [ ] Implement the watcher notification and supervisor as follows, retaining the public handler's existing `defer watcher.Close()` and `defer done()`: + + ```go + // Done returns a channel that is closed when the watcher terminates. + func (w *GCStateWatcher) Done() <-chan struct{} { + return w.ctx.Done() + } + + type gcStateChangeReceiver interface { + RecvBatch(maxChanges int) ([]gc.GCStateChange, error) + Done() <-chan struct{} + Err() error + } + + func serveWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error { + if err := receiver.Err(); err != nil { + return watchGCStatesErrorToStatus(err) + } + resultCh := make(chan error, 1) + go func() { + defer logutil.LogPanic() + resultCh <- sendWatchGCStates(receiver, stream, maxResponseSize) + }() + select { + case err := <-resultCh: + if cause := receiver.Err(); cause != nil { + return watchGCStatesErrorToStatus(cause) + } + return err + case <-receiver.Done(): + return watchGCStatesErrorToStatus(receiver.Err()) + } + } + ``` + + Import the repository's `pkg/utils/logutil`. Rename the old `serveWatchGCStates` body to `sendWatchGCStates` without changing its receive loop, conversion, splitting, per-response `Err` check, or raw error returns. `logutil.LogPanic` follows the repository's goroutine convention; it logs fatally rather than silently recovering and stranding the supervisor. + +- [ ] Add the following focused assertions alongside the regression, then run them under the race detector: + + | Case | Required observation | + | --- | --- | + | Watcher terminates before serving starts | Mapped error returns and the fake stream's send hook is never called. | + | Cancellation while waiting in `RecvBatch` | Handler and worker finish; no response is sent. | + | Parent stream context canceled | Watcher notification wakes the supervisor; cancellation status is preserved. | + | Worker send fails with no watcher cause | Existing `TestServeWatchGCStatesReturnsRawSendError` still returns the same error. | + | Invalid internal change | Existing `Internal` mapping remains unchanged. | + | Cancellation between split responses | Existing per-send terminal check prevents sending the remaining response. | + | Watcher `Done` notification | Notification closes on cancellation; `Err` exposes the first cause and repeated `Done` calls return the same channel. | + + The process-level goroutine leak checker must pass. Tests must wait for their own cleanup and worker-visible exit conditions; do not reuse fake state while a worker may still access it. + +**Completion:** The blocked-send red test passes without canceling its stream first, existing server adapter tests pass under `-race`, and the diff contains one worker/channel per RPC with no manager lock changes. + +## Task 2: Verify real transport teardown and public-handler cleanup + +**Interfaces consumed:** The Task 1 supervisor and existing public `WatchGCStates` handler. Test doubles must honor the receiver batch bound and first-cause contract. + +- [ ] Add `TestWatchGCStatesTransportCancellationUnblocksSend` in `server/gc_service_test.go`. Reuse the `bufconn` setup pattern in `server/grpc_service_test.go`. Register a test service embedding `pdpb.UnimplementedPDServer`; its `WatchGCStates` implementation creates a receiver derived from the real `stream.Context()`, invokes `serveWatchGCStates`, reports the returned error through a buffered test channel, and returns it to gRPC. Supply at least **16 batches of 1024 complete upserts** with large timestamp values and distinct keyspace IDs so the generated stream exceeds **256 KiB**; have `RecvBatch` wait on its context after exhausting the fixture. + + Configure the client with: + + ```go + grpc.WithStaticStreamWindowSize(64 << 10), + grpc.WithStaticConnWindowSize(64 << 10), + ``` + + Use the normal uncompressed protobuf codec and a `bufconn` listener capacity of **1 MiB**. Keep the client connection and RPC context alive, and do not call client `Recv` before triggering server-side watcher cancellation. A **30-second** RPC deadline is a failure guard, not the cancellation mechanism under test. + +- [ ] Make the blocked-send observation deterministic using a test-only wrapper around the real `PD_WatchGCStatesServer`. grpc-go v1.82.1 starts with a **64 KiB** stream write quota (`internal/transport/defaults.go`, `defaultWriteQuota`). With a static **64 KiB** client receive window and no application reads, cumulative successful send enqueues can exhaust that quota plus the receive window. Track successful wire bytes using `response.Size() + 5` for the gRPC message envelope: + + ```go + func (s *observedWatchGCStatesStream) Send(response *pdpb.WatchGCStatesResponse) error { + // These test-only constants match the pinned grpc-go implementation + // and the explicitly configured client receive window. + const receiveWindow = 64 << 10 + const writeQuota = 64 << 10 + if !s.observed && s.sentWireBytes >= receiveWindow+writeQuota { + s.observed = true + close(s.blockedSendStarted) + defer close(s.blockedSendExited) + } + err := s.PD_WatchGCStatesServer.Send(response) + if err == nil { + s.sentWireBytes += response.Size() + 5 + } + return err + } + ``` + + Define the wrapper with an embedded `pdpb.PD_WatchGCStatesServer`, `sentWireBytes int`, `observed bool`, and the two notification channels. Only the sending worker accesses its counter and flag. The next send after that cumulative threshold cannot regain quota while the client does not read. The marker fires immediately before that send, which also covers cancellation racing with entry into `Send`; the original synchronous implementation still cannot observe watcher cancellation there. Recheck the byte/quota argument if the transport version or compression settings change; do not replace this observation with a sleep or a fake blocked `Send`. + +- [ ] After the marker, cancel only the receiver with the domain cause. Require the test service's handler result and `blockedSendExited` within **5 seconds**, without canceling the client or stopping the gRPC server to make those assertions succeed. Then drain client responses to the terminal status and assert the expected code. Cover leader-loss and slow-consumer causes. Register unconditional cleanup so failed assertions close the client/server and release goroutines. Verify that the test fails against the old synchronous supervisor and passes after Task 1; run the focused transport test repeatedly with `-race`. + +- [ ] Extend public-handler tests in `tests/server/gc/gc_test.go` using the existing `TestWatchGCStatesSendFailureCleansUpPublicHandler` setup. Enable concurrency limit **1**, observe registration, and block a test stream's `Send` on its own context. For leader loss, supersede the local manager generation with `stop := manager.OnNodeBecomesLeader()` and register `stop` for cleanup; this is a controlled domain transition, while the existing real leader-transfer test remains part of regression coverage. For slow consumption, once `Send` is blocked, produce **1025** subsequent successful increasing txn-safe-point updates to overflow the default **1024** live queue. Use the null keyspace and increasing targets so writes are neither rejected nor no-ops. If needed, factor the existing test setup into a helper instead of duplicating all rate-limit configuration code. + + Before releasing the fake transport, require: the public handler returns the domain status, `pd_gc_watcher_count` returns to its baseline, the appropriate termination counter increases once, and `GetConcurrencyLimiterStatus("WatchGCStates")` reports **0** current streams. Confirm a subsequent registered watch is admitted. On the leader-generation test, the newly superseded generation remains active until cleanup, so this admission check can use the same one-node fixture. Only then cancel the fake stream context, wait for its blocked send to exit, and clean up the subsequent watch. The fake's context cancellation models the real teardown proved by the preceding transport test. + +**Completion:** Both domain causes return the expected status and free the limit token before a stalled client resumes, and the real transport test proves the blocked send exits after handler return. Existing real leader transfer and send-error cleanup tests still pass. No goroutine leak is hidden by premature client/server shutdown. + +## Task 3: Update the design and run final verification + +- [ ] Update the existing design document's “GC service” and “Lifecycle and errors” sections to describe the supervisor/worker split, the one-worker/one-result-channel cost, cleanup ownership, and the prohibition on joining the worker before returning. Replace the ambiguous in-progress-send sentence with this contract: + + > A response send already in progress cannot be recalled. Watcher termination nevertheless causes the RPC handler to return without waiting for that send. Handler return initiates gRPC transport teardown, which interrupts blocked sending; the worker then exits. The handler retains ownership of watcher cleanup and rate-limit token release. A client that is not reading may observe the terminal status only after draining already queued responses. + +- [ ] Format touched Go files, run `git diff --check`, and execute the narrow tests before broader validation: + + ```sh + make gotest GOTEST_ARGS='./pkg/gc ./server -run "TestGCStateWatcher|TestServeWatchGCStates|TestWatchGCStatesTransport" -count=1 -tags=without_dashboard,deadlock -race -timeout=5m' + make gotest GOTEST_ARGS='./server -run TestWatchGCStatesTransportCancellationUnblocksSend -count=20 -tags=without_dashboard,deadlock -race -timeout=5m' + make gotest GOTEST_ARGS='./tests/server/gc -run TestWatchGCStates -count=1 -tags=without_dashboard,deadlock -race -timeout=10m' + GOFLAGS='-tags=without_dashboard' make basic-test BASIC_TEST_PKGS='./pkg/gc ./server' + ``` + + In this worktree, the tools required by these targets are already installed; `make -o install-tools ...` can reuse them. The `without_dashboard` tag avoids dependence on generated Dashboard assets for the integration tests. Keep regexes free of unescaped trailing `$` inside `GOTEST_ARGS`: Make can interpret the following character as a variable reference and break the shell quoting. If a wrapper fails before its cleanup branch runs, immediately run `make failpoint-disable` (or the installed-tool equivalent) before further work. + +- [ ] Review the final diff against every global constraint and the acceptance cases above. Run required repository checks before PR submission, including `make check`. Confirm failpoint-generated files are absent and only intended source/test/documentation changes remain. Hand back the changed files, red/green evidence, real-transport evidence, and any unresolved failure; do not describe fake-send coverage as proof of transport worker cleanup. + +**Completion:** Review and verification are complete, the handoff report distinguishes local handler exit from client-visible status delivery, and the implementation introduces no per-response concurrency or broad streaming-RPC refactor. diff --git a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md index dce723fa9c1..1bc2d4f86dc 100644 --- a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md +++ b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md @@ -75,7 +75,7 @@ The watcher exposes a receive operation that returns at most a requested number ### GC service -`server/gc_service.go` remains a thin adapter. Its public `WatchGCStates` method performs the rate-limit check directly, validates the request locally, registers a watcher, converts internal changes to protobuf, splits changes into wire-size-bounded responses, sends them, and closes the watcher on every return path. The handler does not proxy the long-lived stream: a non-serving or unbootstrapped member returns `Unavailable`, and existing header and cluster-ID validation semantics remain unchanged. +`server/gc_service.go` remains a thin adapter. Its public `WatchGCStates` method performs the rate-limit check directly, validates the request locally, registers a watcher, supervises streaming, and closes the watcher on every return path. One worker goroutine per RPC receives changes, converts them to protobuf, splits them into wire-size-bounded responses, and sends them sequentially. A result channel with capacity one lets the worker finish even after the handler returns. The supervisor selects between that result and the watcher’s `Done` notification, preferring an already-recorded watcher cause when processing a worker result. The handler does not proxy the long-lived stream: a non-serving or unbootstrapped member returns `Unavailable`, and existing header and cluster-ID validation semantics remain unchanged. Keeping `rateLimitCheck` in the public handler preserves the externally visible method name `WatchGCStates` in the caller-derived rate-limit label. The rate-limit token is held for the lifetime of the stream and released when the handler returns. Every successful `WatchGCStatesResponse` contains `grpcutil.WrapHeader()`; a structurally invalid internal change is logged and returned as gRPC `Internal`. @@ -152,7 +152,11 @@ The lifecycle cases are: The `pkg/gc` layer returns domain errors and does not depend on gRPC status codes. The service maps not-leader and storage/initialization failures to `Unavailable`, and a slow consumer to `ResourceExhausted`. Existing request-validation and rate-limit paths retain their current status semantics. -The receive path checks the terminal cause before returning buffered work after cancellation. Because one `RecvBatch` can be split into multiple protobuf responses, the handler checks the same terminal cause again immediately before every `Send`. This prevents an already-cancelled watcher from deliberately draining stale queued data; only an RPC send already in progress when leadership changes cannot be recalled. Clients treat any terminated stream as requiring reconnection and reinitialization. +The receive path checks the terminal cause before returning buffered work after cancellation. Because one `RecvBatch` can be split into multiple protobuf responses, the worker checks the same terminal cause again immediately before every `Send`. This prevents an already-cancelled watcher from deliberately draining stale queued data. + +A response send already in progress cannot be recalled. Watcher termination nevertheless causes the RPC handler to return without waiting for that send. Handler return initiates gRPC transport teardown, which interrupts blocked sending; the worker then exits. The handler retains ownership of watcher cleanup and rate-limit token release. A client that is not reading may observe the terminal status only after draining already queued responses. + +The handler must not join the worker before returning: transport teardown depends on that return to unblock `Send`. The worker retains its response until sending finishes and publishes its result without waiting for a reader. There is at most one send in progress, with no additional response queue or per-send goroutine. Clients treat any terminated stream as requiring reconnection and reinitialization. ## Protobuf conversion and response batching diff --git a/tests/server/gc/gc_test.go b/tests/server/gc/gc_test.go index 77e6194ada1..a0c5293e849 100644 --- a/tests/server/gc/gc_test.go +++ b/tests/server/gc/gc_test.go @@ -1169,17 +1169,7 @@ func TestWatchGCStatesSendFailureCleansUpPublicHandler(t *testing.T) { re.NotNil(leaderServer) pdServer := leaderServer.GetServer() - options := pdServer.GetServiceMiddlewarePersistOptions() - previousConfig := options.GetGRPCRateLimitConfig().Clone() - enabledConfig := previousConfig.Clone() - enabledConfig.EnableRateLimit = true - options.SetGRPCRateLimitConfig(enabledConfig) - limiter := pdServer.GetGRPCRateLimiter() - limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(1)) - t.Cleanup(func() { - limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(0)) - options.SetGRPCRateLimitConfig(previousConfig) - }) + limiter := limitWatchGCStatesConcurrency(t, pdServer) activeBefore := prometheusMetricValue(t, "pd_gc_watcher_count", nil) clientCancelBefore := prometheusMetricValue(t, "pd_gc_watcher_termination_total", map[string]string{"reason": "client_cancel"}) @@ -1308,3 +1298,141 @@ func TestWatchGCStatesTerminatesOnLeaderTransferAndReinitializes(t *testing.T) { re.Zero(reinitialized.GetGcSafePoint()) re.Empty(reinitialized.GetGcBarriers()) } + +func limitWatchGCStatesConcurrency(t *testing.T, pdServer *server.Server) *ratelimit.Controller { + t.Helper() + options := pdServer.GetServiceMiddlewarePersistOptions() + previousConfig := options.GetGRPCRateLimitConfig().Clone() + enabledConfig := previousConfig.Clone() + enabledConfig.EnableRateLimit = true + options.SetGRPCRateLimitConfig(enabledConfig) + limiter := pdServer.GetGRPCRateLimiter() + limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(1)) + t.Cleanup(func() { + limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(0)) + options.SetGRPCRateLimitConfig(previousConfig) + }) + return limiter +} + +type blockedWatchGCStatesServer struct { + failingWatchGCStatesServer + sendStarted chan struct{} + sendExited chan struct{} +} + +func (s *blockedWatchGCStatesServer) Send(*pdpb.WatchGCStatesResponse) error { + close(s.sendStarted) + defer close(s.sendExited) + <-s.ctx.Done() + return s.ctx.Err() +} + +func waitWatchGCStatesSignal(t *testing.T, signal <-chan struct{}, message string) { + t.Helper() + select { + case <-signal: + case <-time.After(5 * time.Second): + require.FailNow(t, message) + } +} + +func TestWatchGCStatesBlockedSendCleansUpPublicHandler(t *testing.T) { + for _, tc := range []struct { + name string + reason string + code codes.Code + }{ + {"leader loss", "leader_lost", codes.Unavailable}, + {"slow consumer", "slow_consumer", codes.ResourceExhausted}, + } { + t.Run(tc.name, func(t *testing.T) { + re := require.New(t) + cluster := newWatchGCStatesCluster(t, 1, true) + leaderServer := cluster.GetLeaderServer() + re.NotNil(leaderServer) + pdServer := leaderServer.GetServer() + manager := pdServer.GetGCStateManager() + limiter := limitWatchGCStatesConcurrency(t, pdServer) + activeBefore := prometheusMetricValue(t, "pd_gc_watcher_count", nil) + labels := map[string]string{"reason": tc.reason} + terminatedBefore := prometheusMetricValue(t, "pd_gc_watcher_termination_total", labels) + registration := enableWatchGCStatesRegistrationPoint(t) + streamCtx, cancelStream := context.WithTimeout(context.Background(), 30*time.Second) + stream := &blockedWatchGCStatesServer{ + failingWatchGCStatesServer: failingWatchGCStatesServer{ctx: streamCtx}, + sendStarted: make(chan struct{}), sendExited: make(chan struct{}), + } + handlerDone := make(chan struct{}) + var handlerErr error + t.Cleanup(func() { + cancelStream() + waitWatchGCStatesSignal(t, handlerDone, "public handler did not clean up") + select { + case <-stream.sendStarted: + waitWatchGCStatesSignal(t, stream.sendExited, "send did not clean up") + default: + } + }) + go func() { + defer close(handlerDone) + handlerErr = (&server.GrpcServer{Server: pdServer}).WatchGCStates(&pdpb.WatchGCStatesRequest{ + Header: testutil.NewRequestHeader(leaderServer.GetClusterID()), SkipLoadingInitial: true, + }, stream) + }() + registration.wait(t) + registration.disable(re) + re.Equal(activeBefore+1, prometheusMetricValue(t, "pd_gc_watcher_count", nil)) + _, current := limiter.GetConcurrencyLimiterStatus("WatchGCStates") + re.Equal(uint64(1), current) + res, err := manager.AdvanceTxnSafePoint(constant.NullKeyspaceID, 10, time.Now()) + re.NoError(err) + re.Equal(uint64(10), res.NewTxnSafePoint) + waitWatchGCStatesSignal(t, stream.sendStarted, "send did not start") + if tc.reason == "leader_lost" { + stop := manager.OnNodeBecomesLeader() + t.Cleanup(stop) + } else { + // The sending worker cannot receive these 1025 updates, overflowing + // the default live queue's 1024 slots. + for target := uint64(11); target <= 1035; target++ { + res, err := manager.AdvanceTxnSafePoint(constant.NullKeyspaceID, target, time.Now()) + re.NoError(err) + re.Equal(target, res.NewTxnSafePoint) + } + } + waitWatchGCStatesSignal(t, handlerDone, "public handler did not return while send was blocked") + re.Equal(tc.code, status.Code(handlerErr)) + re.NoError(streamCtx.Err()) + select { + case <-stream.sendExited: + re.FailNow("send exited before transport teardown") + default: + } + re.Equal(activeBefore, prometheusMetricValue(t, "pd_gc_watcher_count", nil)) + re.Equal(terminatedBefore+1, prometheusMetricValue(t, "pd_gc_watcher_termination_total", labels)) + _, current = limiter.GetConcurrencyLimiterStatus("WatchGCStates") + re.Zero(current) + + // Admission while the old send is still blocked proves the public + // handler released its token independently of transport progress. + nextRegistration := enableWatchGCStatesRegistrationPoint(t) + client := newWatchGCStatesClient(t, leaderServer.GetAddr()) + nextStream, cancelNext := openWatchGCStates(t, client, testutil.NewRequestHeader(leaderServer.GetClusterID()), true) + nextRegistration.wait(t) + nextRegistration.disable(re) + _, current = limiter.GetConcurrencyLimiterStatus("WatchGCStates") + re.Equal(uint64(1), current) + cancelStream() + waitWatchGCStatesSignal(t, stream.sendExited, "send did not exit after transport teardown") + cancelNext() + _, err = nextStream.Recv() + re.Equal(codes.Canceled, status.Code(err)) + testutil.Eventually(re, func() bool { + _, current := limiter.GetConcurrencyLimiterStatus("WatchGCStates") + return current == 0 && prometheusMetricValue(t, "pd_gc_watcher_count", nil) == activeBefore + }) + re.Equal(terminatedBefore+1, prometheusMetricValue(t, "pd_gc_watcher_termination_total", labels)) + }) + } +} From 3e55ddc1acca41463541f820ebbec6637c19aef0 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 16 Sep 2026 15:04:51 +0800 Subject: [PATCH 15/25] docs: stop tracking local WatchGCStates design and plans Keep local design and implementation notes out of the PR diff. Signed-off-by: Wenxuan Zhang --- .../plans/2026-09-03-watch-gc-states.md | 1410 ----------------- ...2026-09-16-watch-gc-states-blocked-send.md | 242 --- .../2026-09-03-watch-gc-states-design.md | 248 --- 3 files changed, 1900 deletions(-) delete mode 100644 docs/superpowers/plans/2026-09-03-watch-gc-states.md delete mode 100644 docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md delete mode 100644 docs/superpowers/specs/2026-09-03-watch-gc-states-design.md diff --git a/docs/superpowers/plans/2026-09-03-watch-gc-states.md b/docs/superpowers/plans/2026-09-03-watch-gc-states.md deleted file mode 100644 index 62236f45811..00000000000 --- a/docs/superpowers/plans/2026-09-03-watch-gc-states.md +++ /dev/null @@ -1,1410 +0,0 @@ -# WatchGCStates server implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Implement the PD `WatchGCStates` server stream with ordered initial and live delivery, isolated backpressure, local leadership lifecycle handling, exact protobuf response sizing, and bounded observability. - -**Architecture:** `pkg/gc` owns the watcher registry, initial-state iteration, live publication, per-keyspace merge ordering, cancellation causes, and lifecycle metrics. `server/gc_service.go` validates the local streaming request, converts domain changes to protobuf, splits responses by exact wire size, and sends them. Focused package tests prove deterministic concurrency behavior, while `tests/server/gc` covers real gRPC and leader-transfer behavior. - -**Tech stack:** Go 1.25, gRPC server streaming, gogo/protobuf, PingCAP failpoints, Prometheus client_golang, testify, and the existing PD test cluster. - -**Spec:** [`../specs/2026-09-03-watch-gc-states-design.md`](../specs/2026-09-03-watch-gc-states-design.md) - -## Global constraints - -Every task inherits these requirements from the approved design and repository rules: - -- Do not add `WatchGCSafePointV2` compatibility or a Go PD client API. -- Emit complete effective GC-state upserts only after a successful safe-point mutation updates the manager cache; do not emit barrier-only or no-op changes. -- Keep `removed` in the internal model and transport converter, but do not add keyspace lifecycle producers in this change. -- Register live delivery before initial scanning and suppress an initial state after a live change for the same scope has been emitted. -- Use `liveCh` capacity 1024, `initCh` capacity 1, initial batches of 1024 changes, and `RecvBatch(1024)` in production; expose capacities only through unexported test seams. -- Never block on watcher channels while holding `GCStateManager.mu`; evict only the watcher whose `liveCh` is full. -- Keep response wire size at or below 1 MiB except when one change alone exceeds the target, in which case send that change alone. -- Hold the `WatchGCStates` rate-limit token for the complete stream lifetime. -- Keep Prometheus labels bounded and pre-bind all `WithLabelValues` handles outside hot paths. -- Use `make gotest` for tests that rely on failpoints, and verify that failpoints are disabled before committing. -- Do not hard-wrap Markdown prose. - -## File map - -The implementation uses focused files and avoids unrelated refactoring: - -- Create `pkg/gc/gc_state_watcher.go` for domain changes, watcher merge state, registration helpers, initial loading, cancellation, and live fan-out. -- Create `pkg/gc/gc_state_watcher_test.go` for deterministic watcher, leadership, initial-loading, backpressure, publication, and metric tests. -- Modify `pkg/gc/gc_state_manager.go` only where manager fields, leadership callbacks, safe-point mutation hooks, and the existing iterator are involved. -- Modify `pkg/gc/gc_state_manager_test.go` only to adapt existing leadership setup and reuse its embedded-etcd GC fixtures. -- Modify `pkg/gc/metrics.go` for the active watcher gauge and bounded termination counters. -- Modify `pkg/errs/errno.go` and `errors.toml` for the slow-consumer sentinel. -- Modify `server/cluster/cluster.go` so each leader startup stores its generation-aware teardown closure. -- Modify `server/gc_service.go` for protobuf conversion, exact response splitting, local request preflight, domain-error mapping, and streaming. -- Create `server/gc_service_test.go` for converter, batching, cancellation-before-send, and error-mapping unit tests. -- Modify `tests/server/gc/gc_test.go` for real RPC, rate-limit, validation, and leader-transfer coverage. -- Modify the root, `client`, `tools`, and `tests/integrations` `go.mod` and `go.sum` pairs to use kvproto commit `65b4e27a438de9274bf88c58e89e83749e62f646`. - ---- - -### Task 1: Build the watcher core and local leadership lifecycle - -This task creates the transport-independent initial/live merge state, registers watchers with `GCStateManager`, incrementally loads initial state, and binds every watcher to one local leadership generation. - -**Files:** - -- Create: `pkg/gc/gc_state_watcher.go` -- Create: `pkg/gc/gc_state_watcher_test.go` -- Modify: `pkg/gc/gc_state_manager.go:188-266` -- Modify: `pkg/gc/gc_state_manager_test.go:116-262` -- Modify: `server/cluster/cluster.go:495-496` - -**Interfaces:** - -- Consumes: Existing `GCState` from `pkg/gc/gc_state_manager.go`. -- Produces: `GCStateChange`, `NewGCStateUpsert`, `NewGCStateRemoved`, `GCStateChange.Upsert`, `GCStateChange.RemovedKeyspaceID`, `GCStateChange.KeyspaceID`, `GCStateWatcher.RecvBatch`, and `GCStateWatcher.Err`. -- Produces: `newGCStateWatcher`, `gcStateWatchConfig`, `GCStateWatcher.initCh`, `GCStateWatcher.liveCh`, `GCStateWatcher.cancel`, `GCStateManager.WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error)`, `GCStateWatcher.Close()`, and `GCStateManager.OnNodeBecomesLeader() func()`. -- Produces for Task 2: `terminateGCStateWatcherLocked`, the watcher registry, watcher IDs, and bounded termination-reason constants. - -- [ ] **Step 1: Write failing tests for both observable delivery orders** - -Create tests that drive the channels directly so selection is deterministic: send initial `v1` and receive it before sending live `v2` for the initial-first case; receive live `v2` first, then send initial `v1` for the live-first case. - -```go -func TestGCStateWatcherInitialThenLive(t *testing.T) { - w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) - w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 1})} - - got, err := w.RecvBatch(1) - require.NoError(t, err) - require.Equal(t, uint64(1), mustUpsert(t, got[0]).TxnSafePoint) - - w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 2}) - got, err = w.RecvBatch(1) - require.NoError(t, err) - require.Equal(t, uint64(2), mustUpsert(t, got[0]).TxnSafePoint) -} - -func TestGCStateWatcherLiveSuppressesOlderInitial(t *testing.T) { - w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) - w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 2}) - - got, err := w.RecvBatch(1) - require.NoError(t, err) - require.Equal(t, uint64(2), mustUpsert(t, got[0]).TxnSafePoint) - - w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7, TxnSafePoint: 1})} - close(w.initCh) - _, ok, err := w.receiveOne(false) - require.NoError(t, err) - require.False(t, ok) - require.True(t, w.initDone) -} -``` - -- [ ] **Step 2: Write failing tests for removed changes, closed-channel draining, batch bounds, and cancellation priority** - -Use the same direct-channel seam and add these exact cases: - -```go -func TestGCStateWatcherRemovedSuppressesInitial(t *testing.T) { - w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 2}, false) - w.liveCh <- NewGCStateRemoved(7) - got, err := w.RecvBatch(1) - require.NoError(t, err) - removed, ok := got[0].RemovedKeyspaceID() - require.True(t, ok) - require.Equal(t, uint32(7), removed) - - w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 7})} - close(w.initCh) - _, ok, err = w.receiveOne(false) - require.NoError(t, err) - require.False(t, ok) - require.True(t, w.initDone) -} - -func TestGCStateWatcherDrainsBufferedInitBeforeReleasingDirtySet(t *testing.T) { - w := newGCStateWatcher(context.Background(), gcStateWatchConfig{initChannelCapacity: 1, liveChannelCapacity: 1}, false) - w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7}) - _, err := w.RecvBatch(1) - require.NoError(t, err) - w.initCh <- []GCStateChange{NewGCStateUpsert(GCState{KeyspaceID: 8})} - close(w.initCh) - - got, err := w.RecvBatch(1) - require.NoError(t, err) - require.Equal(t, uint32(8), mustUpsert(t, got[0]).KeyspaceID) - require.False(t, w.initDone) - require.NotNil(t, w.dirtyDuringInit) - - _, ok, err := w.receiveOne(false) - require.NoError(t, err) - require.False(t, ok) - require.True(t, w.initDone) - require.Nil(t, w.initCh) - require.Nil(t, w.dirtyDuringInit) -} - -func TestGCStateWatcherRecvBatchHonorsMaximum(t *testing.T) { - w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 3}, true) - for id := uint32(1); id <= 3; id++ { - w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: id}) - } - got, err := w.RecvBatch(2) - require.NoError(t, err) - require.Len(t, got, 2) - got, err = w.RecvBatch(2) - require.NoError(t, err) - require.Len(t, got, 1) -} - -func TestGCStateWatcherCancellationDiscardsBufferedWork(t *testing.T) { - w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 1}, true) - w.liveCh <- NewGCStateUpsert(GCState{KeyspaceID: 7}) - want := errors.New("watch terminated") - w.cancel(want) - got, err := w.RecvBatch(1) - require.ErrorIs(t, err, want) - require.Nil(t, got) -} - -func TestGCStateWatcherFirstCancellationCauseWins(t *testing.T) { - w := newGCStateWatcher(context.Background(), gcStateWatchConfig{liveChannelCapacity: 1}, true) - first := errors.New("first") - w.cancel(first) - w.cancel(errors.New("second")) - require.ErrorIs(t, w.Err(), first) -} -``` - -Add `mustUpsert` as a test helper that calls `change.Upsert()`, requires the boolean result, and returns the state. - -- [ ] **Step 3: Run the focused tests and confirm the red state** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run ^TestGCStateWatcher -count=1' -``` - -Expected: compilation fails because the watcher types and constructors do not exist. - -- [ ] **Step 4: Implement the domain change type and test helper accessors** - -Use an unexported discriminator so the zero value remains structurally invalid for Task 3 converter tests. - -```go -type gcStateChangeKind uint8 - -const ( - gcStateChangeUnknown gcStateChangeKind = iota - gcStateChangeUpsert - gcStateChangeRemoved -) - -type GCStateChange struct { - kind gcStateChangeKind - upsert GCState - removedKeyspaceID uint32 -} - -func NewGCStateUpsert(state GCState) GCStateChange { - state.GCBarriers = nil - return GCStateChange{kind: gcStateChangeUpsert, upsert: state} -} - -func NewGCStateRemoved(keyspaceID uint32) GCStateChange { - return GCStateChange{kind: gcStateChangeRemoved, removedKeyspaceID: keyspaceID} -} - -func (c GCStateChange) Upsert() (GCState, bool) { - return c.upsert, c.kind == gcStateChangeUpsert -} - -func (c GCStateChange) RemovedKeyspaceID() (uint32, bool) { - return c.removedKeyspaceID, c.kind == gcStateChangeRemoved -} - -func (c GCStateChange) KeyspaceID() (uint32, bool) { - if state, ok := c.Upsert(); ok { - return state.KeyspaceID, true - } - return c.RemovedKeyspaceID() -} -``` - -Add GoDoc to every exported type and function. - -- [ ] **Step 5: Implement the single-consumer merge** - -Define the production defaults and the unexported configuration seam: - -```go -const ( - defaultGCStateWatchInitialBatchSize = 1024 - defaultGCStateWatchInitChannelCapacity = 1 - defaultGCStateWatchLiveChannelCapacity = 1024 -) - -type gcStateWatchConfig struct { - initialBatchSize int - initChannelCapacity int - liveChannelCapacity int -} - -type GCStateWatcher struct { - ctx context.Context - cancel context.CancelCauseFunc - initCh chan []GCStateChange - liveCh chan GCStateChange - initDone bool - pendingInit []GCStateChange - dirtyDuringInit map[uint32]struct{} -} - -func newGCStateWatcher(parent context.Context, cfg gcStateWatchConfig, skipLoadingInitial bool) *GCStateWatcher { - ctx, cancel := context.WithCancelCause(parent) - watcher := &GCStateWatcher{ - ctx: ctx, - cancel: cancel, - initCh: make(chan []GCStateChange, cfg.initChannelCapacity), - liveCh: make(chan GCStateChange, cfg.liveChannelCapacity), - initDone: skipLoadingInitial, - } - if !skipLoadingInitial { - watcher.dirtyDuringInit = make(map[uint32]struct{}) - } - return watcher -} -``` - -Implement `receiveOne(block bool)` as the only place that selects from the channels. It performs these branches in order: - -1. Return `context.Cause(w.ctx)` before inspecting buffered work. -2. Consume `pendingInit` first, dropping an initial change when its keyspace is in `dirtyDuringInit`. -3. If initial loading is complete, select only `ctx.Done()` and `liveCh`. -4. Otherwise, select `ctx.Done()`, `liveCh`, and `initCh`. Mark every emitted live scope dirty. Copy a received initial batch into `pendingInit`. When the closed `initCh` is observed after buffered batches are drained, set `initCh=nil`, `initDone=true`, and `dirtyDuringInit=nil`. -5. For opportunistic collection, add a `default` branch when `block=false`. - -Implement `RecvBatch` by blocking for its first visible change, calling `receiveOne(false)` until `maxChanges` is reached or no visible work is ready, and checking `Err()` again before returning the batch. Panic on a non-positive `maxChanges`, because this is an internal programmer error and production always passes 1024. - -```go -func (w *GCStateWatcher) Err() error { - return context.Cause(w.ctx) -} - -func (w *GCStateWatcher) RecvBatch(maxChanges int) ([]GCStateChange, error) { - if maxChanges <= 0 { - panic("GCStateWatcher.RecvBatch requires a positive maximum") - } - first, ok, err := w.receiveOne(true) - if err != nil { - return nil, err - } - if !ok { - panic("blocking watcher receive returned no result") - } - result := []GCStateChange{first} - for len(result) < maxChanges { - change, ok, err := w.receiveOne(false) - if err != nil { - return nil, err - } - if !ok { - break - } - result = append(result, change) - } - if err := w.Err(); err != nil { - return nil, err - } - return result, nil -} -``` - -Place a comment beside the dirty-set branches explaining the two valid orders, `v1 -> v2` and `v2` with `v1` suppressed, as required by the spec. - -- [ ] **Step 6: Run the watcher merge tests and confirm the green state** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run ^TestGCStateWatcher -count=1' -``` - -Expected: all watcher merge tests pass, including cancellation under `-count=1`. - -- [ ] **Step 7: Format and inspect the watcher state machine** - -Run: - -```bash -gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go -git diff --check -``` - -Expected: formatting and whitespace checks pass before manager integration begins. Do not commit yet because the watcher isn't usable until the remaining steps connect it to `GCStateManager`. - -#### Manager integration phase - -This phase connects the tested merge state to the manager, starts incremental loading, and replaces the counter-style follower callback with a generation-aware teardown closure. It remains part of Task 1 because neither half is independently usable. - -- [ ] **Step 8: Write failing tests for registration and generation-aware teardown** - -Add tests with these concrete sequences: - -```go -func (s *gcStateManagerTestSuite) TestGCStateWatchRequiresActiveLeadership() { - follower := NewGCStateManager(s.provider, s.manager.cfg, s.manager.keyspaceManager) - _, err := follower.WatchGCStates(context.Background(), true) - s.Require().ErrorIs(err, errs.ErrNotLeader) -} - -func (s *gcStateManagerTestSuite) TestGCStateWatchLeadershipGeneration() { - re := s.Require() - stopFirst := s.manager.OnNodeBecomesLeader() - first, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - - stopSecond := s.manager.OnNodeBecomesLeader() - re.ErrorIs(first.Err(), errs.ErrNotLeader) - second, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - - stopFirst() - re.NoError(second.Err()) - stopSecond() - re.ErrorIs(second.Err(), errs.ErrNotLeader) -} -``` - -Use the existing suite fixture rather than duplicating embedded-etcd setup. Construct the follower manager from the suite's provider, config, and keyspace manager so it has never received a leader callback. - -- [ ] **Step 9: Write failing tests for initial loading and cleanup** - -Add these exact tests: - -- `TestGCStateWatchLoadsInitialStatesIncrementally`: call the unexported configured registration helper with batch size 2 and assert active keyspaces arrive in batches with no barriers. -- `TestGCStateWatchSkipsInitialLoading`: register with `skipLoadingInitial=true`, assert `initDone`, and verify the initial channel remains unused. -- `TestGCStateWatchLiveSuppressesPausedInitial`: persist transaction safe point `v1` for keyspace 2, pause `watchGCStatesInitialStateLoaded` only when that scope is read, register with initial loading, advance the same keyspace to `v2`, consume the live `v2`, release the loader, drain initial work, and assert that no emitted change contains `v1` for keyspace 2. -- `TestGCStateWatchInitialFailureTerminatesWatcher`: enable `iterateAllKeyspacesGCStatesError`, assert `RecvBatch` returns the injected error, and assert the registry no longer contains the watcher. -- `TestGCStateWatchFullInitChannelDoesNotHoldManagerMutex`: use initial batch size 1 and `initCh` capacity 1, wait until the channel is full, run a manager mutation and the leadership teardown with bounded channels, then close the watcher and assert the loader exits. -- `TestGCStateWatchConcurrentCloseIsIdempotent`: call `Close`, initialization termination, and leadership teardown concurrently and assert no registry or goroutine leak. - -Implement the paused-initial case with this sequence; use a larger test-only `initCh` capacity so the loader can reach keyspace 2 before the consumer starts draining earlier keyspaces: - -```go -func (s *gcStateManagerTestSuite) TestGCStateWatchLiveSuppressesPausedInitial() { - re := s.Require() - const keyspaceID = uint32(2) - _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 10, time.Now()) - re.NoError(err) - - reached := make(chan struct{}) - release := make(chan struct{}) - var reachedOnce, releaseOnce sync.Once - releaseLoader := func() { releaseOnce.Do(func() { close(release) }) } - re.NoError(failpoint.EnableCall("github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded", func(id uint32) { - if id == keyspaceID { - reachedOnce.Do(func() { close(reached) }) - <-release - } - })) - defer func() { re.NoError(failpoint.Disable("github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded")) }() - defer releaseLoader() - - w, err := s.manager.watchGCStates(context.Background(), false, gcStateWatchConfig{initialBatchSize: 1, initChannelCapacity: 16, liveChannelCapacity: 4}) - re.NoError(err) - defer w.Close() - select { - case <-reached: - case <-time.After(5 * time.Second): - re.FailNow("initial loader did not reach keyspace 2") - } - - _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) - re.NoError(err) - for { - changes, err := w.RecvBatch(1) - re.NoError(err) - state, ok := changes[0].Upsert() - if ok && state.KeyspaceID == keyspaceID && state.TxnSafePoint == 20 { - break - } - } - releaseLoader() - - re.Eventually(func() bool { - for { - change, ok, err := w.receiveOne(false) - re.NoError(err) - if !ok { - return w.initDone - } - state, upsert := change.Upsert() - re.False(upsert && state.KeyspaceID == keyspaceID && state.TxnSafePoint == 10) - } - }, 5*time.Second, 10*time.Millisecond) -} -``` - -Add two failpoint call sites to make timing deterministic: `watchGCStatesRegistered` immediately after registration releases `GCStateManager.mu`, and `watchGCStatesInitialStateLoaded` after one state is read but before its batch can be sent. Neither call site can execute while holding the manager mutex. - -- [ ] **Step 10: Run the focused lifecycle tests and confirm the red state** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/TestGCStateWatch(Requires|Leadership|Loads|Skips|Live|Initial|Full|Concurrent)" -count=1' -``` - -Expected: compilation fails because manager registration, teardown closures, and watcher cleanup do not exist. - -- [ ] **Step 11: Replace leadership counting with an active generation and teardown closure** - -Keep lock-free leadership reads for existing cache fast paths while making generation creation manager-owned: - -```go -watchers map[uint64]*GCStateWatcher -nextWatcherID uint64 -nextLeadershipGeneration uint64 -activeLeadershipGeneration atomic.Uint64 - -func (m *GCStateManager) OnNodeBecomesLeader() func() { - m.mu.Lock() - m.nextLeadershipGeneration++ - generation := m.nextLeadershipGeneration - m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) - m.activeLeadershipGeneration.Store(generation) - m.gcStateCache.clearAll() - m.mu.Unlock() - - return func() { - m.mu.Lock() - defer m.mu.Unlock() - if m.activeLeadershipGeneration.Load() != generation { - return - } - m.activeLeadershipGeneration.Store(0) - m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) - m.gcStateCache.clearAll() - } -} - -func (m *GCStateManager) nodeIsLeader() bool { - return m.activeLeadershipGeneration.Load() != 0 -} -``` - -Initialize the watcher map in `NewGCStateManager`. Remove `OnNodeBecomesFollower`; change cluster startup to the following direct assignment so the closure captured for this exact generation is invoked by `RaftCluster.Stop`: - -```go -c.stopGCStateManager = s.GetGCStateManager().OnNodeBecomesLeader() -``` - -- [ ] **Step 12: Implement registration, idempotent removal, and initial loading** - -Use these exact public and test-only entry points: - -```go -func (m *GCStateManager) WatchGCStates(ctx context.Context, skipLoadingInitial bool) (*GCStateWatcher, error) { - return m.watchGCStates(ctx, skipLoadingInitial, gcStateWatchConfig{ - initialBatchSize: defaultGCStateWatchInitialBatchSize, - initChannelCapacity: defaultGCStateWatchInitChannelCapacity, - liveChannelCapacity: defaultGCStateWatchLiveChannelCapacity, - }) -} - -func (m *GCStateManager) watchGCStates(ctx context.Context, skipLoadingInitial bool, cfg gcStateWatchConfig) (*GCStateWatcher, error) -func (m *GCStateManager) loadInitialGCStates(watcher *GCStateWatcher, batchSize int) -func (m *GCStateManager) terminateGCStateWatcher(watcher *GCStateWatcher, cause error, reason gcStateWatcherTerminationReason) -func (m *GCStateManager) terminateGCStateWatcherLocked(watcher *GCStateWatcher, cause error, reason gcStateWatcherTerminationReason) bool -func (m *GCStateManager) terminateAllGCStateWatchersLocked(cause error, reason gcStateWatcherTerminationReason) -func (w *GCStateWatcher) Close() -``` - -Add `manager *GCStateManager` and `id uint64` to `GCStateWatcher`. Define the bounded reasons now so the lifecycle metrics phase can attach metrics without changing lifecycle signatures: - -```go -type gcStateWatcherTerminationReason string - -const ( - watcherTerminationClientCancel gcStateWatcherTerminationReason = "client_cancel" - watcherTerminationLeaderLost gcStateWatcherTerminationReason = "leader_lost" - watcherTerminationSlowConsumer gcStateWatcherTerminationReason = "slow_consumer" - watcherTerminationInitError gcStateWatcherTerminationReason = "init_error" -) -``` - -Registration locks the manager, rejects generation 0, allocates an ID, assigns the manager and ID to the watcher, inserts it, unlocks, invokes `watchGCStatesRegistered`, and only then starts the loader. For skipped initial loading, construct the watcher with `initDone=true` and do not start a loader. - -The loader uses a local batch and a cancellation-aware blocking flush around the existing iterator: - -```go -batch := make([]GCStateChange, 0, batchSize) -stopped := false -flush := func() bool { - if len(batch) == 0 { - return true - } - ready := batch - batch = make([]GCStateChange, 0, batchSize) - select { - case watcher.initCh <- ready: - return true - case <-watcher.ctx.Done(): - return false - } -} - -err := m.iterateAllKeyspacesGCStates( - watcher.ctx, - true, - func(uint32) bool { return true }, - func(state GCState) { - if stopped { - return - } - failpoint.InjectCall("watchGCStatesInitialStateLoaded", state.KeyspaceID) - if watcher.Err() != nil { - stopped = true - return - } - batch = append(batch, NewGCStateUpsert(state)) - if len(batch) == batchSize { - stopped = !flush() - } - }, - nil, -) - -if stopped || watcher.Err() != nil { - return -} -if err != nil { - m.terminateGCStateWatcher(watcher, errors.Annotate(err, "load initial GC states"), watcherTerminationInitError) - return -} -if !flush() { - return -} -close(watcher.initCh) -``` - -The local `stopped` flag is required because the iterator callback cannot return an error. The failpoint runs before the state is appended, the flush replaces the batch backing slice before reuse, and only the loader closes `initCh` after successful iteration and final flush. If the watcher context already has a cause, the loader returns without replacing that cause. - -`terminateGCStateWatcherLocked` first verifies that the ID still maps to the same watcher, deletes it, and calls its `CancelCauseFunc` without invoking any callback that reacquires `GCStateManager.mu`. `Close` delegates to the manager with `context.Canceled` and `watcherTerminationClientCancel`. - -- [ ] **Step 13: Adapt existing manager tests to the teardown-returning callback** - -In `newGCStateManagerForTest`, retain the teardown and include it in the returned cleanup: - -```go -stopGCStateManager := gcStateManager.OnNodeBecomesLeader() -originalClean := clean -clean = func() { - stopGCStateManager() - originalClean() -} -``` - -Keep `ensureMarkedLeader` compatible by registering the returned closure with `s.T().Cleanup` whenever it creates a new leadership generation. Replace the one test that temporarily writes `nodeLeadership` with a save/set/restore of `activeLeadershipGeneration`, and update read-only assertions to call `nodeIsLeader()`. Search for every remaining `OnNodeBecomesLeader`, `OnNodeBecomesFollower`, and `nodeLeadership` reference so no test silently loses the teardown for the generation it creates. - -- [ ] **Step 14: Run lifecycle and existing cache tests** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatch|GetGCStateCache|GetAllKeyspacesGCStates)" -count=1' -go test ./server/cluster -run '^$' -``` - -Expected: watcher lifecycle tests pass, existing leader-gated cache behavior remains green, and the cluster package compiles with the new callback. - -- [ ] **Step 15: Format and commit the watcher core and lifecycle** - -Run: - -```bash -gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/gc/gc_state_manager_test.go server/cluster/cluster.go -git diff --check -git add pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/gc/gc_state_manager_test.go server/cluster/cluster.go -git commit -s -m "gc: add GC state watcher lifecycle" -``` - -### Task 2: Publish and observe effective safe-point changes - -This task attaches live publication exactly once to the successful shared mutation paths, isolates slow consumers, and records the resulting watcher lifecycle transitions with bounded metrics. - -**Files:** - -- Modify: `pkg/gc/gc_state_watcher.go` -- Modify: `pkg/gc/gc_state_watcher_test.go` -- Modify: `pkg/gc/gc_state_manager.go:350-413` -- Modify: `pkg/gc/gc_state_manager.go:445-604` -- Modify: `pkg/gc/metrics.go` -- Modify: `pkg/errs/errno.go:551-554` -- Modify: `errors.toml` - -**Interfaces:** - -- Consumes: Task 1's registry, watcher IDs, and termination helpers. -- Produces: `publishGCStateChangeLocked`, `errs.ErrGCStateWatcherSlowConsumer`, complete live upserts used by the server stream, `pd_gc_watcher_count`, `pd_gc_watcher_termination_total{reason=...}`, and `recordGCStateWatcherTerminationMetrics`. - -- [ ] **Step 1: Write failing publication tests for modern, compatible, and barrier paths** - -Register watchers with `skipLoadingInitial=true` and assert these exact cases: - -```go -func (s *gcStateManagerTestSuite) TestGCStateWatchPublishesAdvanceGCSafePoint() { - re := s.Require() - const keyspaceID = uint32(2) - _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) - re.NoError(err) - w, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - defer w.Close() - - _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 10) - re.NoError(err) - changes, err := w.RecvBatch(1) - re.NoError(err) - state := mustUpsert(s.T(), changes[0]) - re.Equal(GCState{KeyspaceID: keyspaceID, IsKeyspaceLevel: true, TxnSafePoint: 20, GCSafePoint: 10}, state) -} - -func (s *gcStateManagerTestSuite) TestGCStateWatchPublishesAdvanceTxnSafePoint() { - re := s.Require() - w, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - defer w.Close() - - _, err = s.manager.AdvanceTxnSafePoint(2, 20, time.Now()) - re.NoError(err) - changes, err := w.RecvBatch(1) - re.NoError(err) - re.Equal(uint64(20), mustUpsert(s.T(), changes[0]).TxnSafePoint) -} - -func (s *gcStateManagerTestSuite) TestGCStateWatchCompatiblePathsPublishOnce() { - re := s.Require() - const keyspaceID = uint32(2) - _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 30, time.Now()) - re.NoError(err) - - gcWatcher, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - _, _, err = s.manager.CompatibleUpdateGCSafePoint(keyspaceID, 10) - re.NoError(err) - changes, err := gcWatcher.RecvBatch(1) - re.NoError(err) - re.Len(changes, 1) - re.Empty(gcWatcher.liveCh) - gcWatcher.Close() - - txnWatcher, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - _, _, err = s.manager.CompatibleUpdateServiceGCSafePoint(keyspaceID, keypath.GCWorkerServiceSafePointID, 40, math.MaxInt64, time.Now()) - re.NoError(err) - changes, err = txnWatcher.RecvBatch(1) - re.NoError(err) - re.Len(changes, 1) - re.Empty(txnWatcher.liveCh) - txnWatcher.Close() -} - -func (s *gcStateManagerTestSuite) TestGCStateWatchDoesNotPublishNoOpOrFailure() { - re := s.Require() - const keyspaceID = uint32(2) - _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) - re.NoError(err) - _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 10) - re.NoError(err) - w, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - defer w.Close() - - _, err = s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) - re.NoError(err) - _, _, err = s.manager.CompatibleUpdateGCSafePoint(keyspaceID, 10) - re.NoError(err) - _, _, err = s.manager.AdvanceGCSafePoint(keyspaceID, 9) - re.ErrorIs(err, errs.ErrDecreasingGCSafePoint) - re.Empty(w.liveCh) -} - -func (s *gcStateManagerTestSuite) TestGCStateWatchDoesNotPublishBarrierOnlyChanges() { - re := s.Require() - const keyspaceID = uint32(2) - _, err := s.manager.AdvanceTxnSafePoint(keyspaceID, 20, time.Now()) - re.NoError(err) - w, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - defer w.Close() - - _, err = s.manager.SetGCBarrier(keyspaceID, "backup", 30, time.Hour, time.Now()) - re.NoError(err) - _, err = s.manager.DeleteGCBarrier(keyspaceID, "backup") - re.NoError(err) - re.Empty(w.liveCh) -} -``` - -For every complete upsert, assert `KeyspaceID`, `IsKeyspaceLevel`, `TxnSafePoint`, `GCSafePoint`, and an empty `GCBarriers` slice. - -- [ ] **Step 2: Write the failing slow-consumer isolation test** - -Create watcher A with `liveChannelCapacity=1` and watcher B with capacity 4. Publish two successful state changes without reading A, read B after each mutation, and assert: - -```go -re.ErrorIs(watcherA.Err(), errs.ErrGCStateWatcherSlowConsumer) -re.NoError(watcherB.Err()) -re.NotContains(s.manager.watchers, watcherA.id) -re.Contains(s.manager.watchers, watcherB.id) -``` - -Then reconnect A with initial loading enabled and assert its rebuilt state contains the latest safe points. - -- [ ] **Step 3: Run the publication tests and confirm the red state** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/TestGCStateWatch(Publishes|Compatible|DoesNotPublish|SlowConsumer)" -count=1' -``` - -Expected: tests fail because mutations do not publish and a full `liveCh` is not handled. - -- [ ] **Step 4: Add the slow-consumer sentinel and regenerate error documentation** - -Add the normalized error beside the existing GC errors: - -```go -ErrGCStateWatcherSlowConsumer = errors.Normalize("GC state watcher is too slow", errors.RFCCodeText("PD:gc:ErrGCStateWatcherSlowConsumer")) -``` - -Run: - -```bash -make generate-errdoc -``` - -Verify that `errors.toml` contains `PD:gc:ErrGCStateWatcherSlowConsumer` and no unrelated generated changes. - -- [ ] **Step 5: Implement non-blocking fan-out under the manager mutex** - -Add a locked helper that iterates the registry and uses a non-blocking send: - -```go -func (m *GCStateManager) publishGCStateChangeLocked(change GCStateChange) { - for _, watcher := range m.watchers { - select { - case watcher.liveCh <- change: - default: - log.Warn("GC state watcher is too slow", zap.Uint64("watcher-id", watcher.id), zap.Int("capacity", cap(watcher.liveCh)), zap.Int("queue-length", len(watcher.liveCh))) - m.terminateGCStateWatcherLocked(watcher, errs.ErrGCStateWatcherSlowConsumer, watcherTerminationSlowConsumer) - } - } - // TODO: Publish keyspace metadata upserts and removals through this same serialized path when an authoritative GC-leader-owned lifecycle hook exists. -} -``` - -Deleting the current watcher from a Go map during iteration is valid; do not collect a second removal list and do not wait, retry, or start one goroutine per watcher. - -- [ ] **Step 6: Publish from the two successful post-cache-update paths** - -Immediately after each existing `gcStateCache.store` call, publish only when the effective value changed: - -```go -if newGCSafePoint != oldGCSafePoint { - m.publishGCStateChangeLocked(NewGCStateUpsert(GCState{ - KeyspaceID: keyspaceID, - IsKeyspaceLevel: keyspaceID != constant.NullKeyspaceID, - TxnSafePoint: txnSafePoint, - GCSafePoint: newGCSafePoint, - })) -} -``` - -```go -if newTxnSafePoint != oldTxnSafePoint { - m.publishGCStateChangeLocked(NewGCStateUpsert(GCState{ - KeyspaceID: keyspaceID, - IsKeyspaceLevel: keyspaceID != constant.NullKeyspaceID, - TxnSafePoint: newTxnSafePoint, - GCSafePoint: gcSafePoint, - })) -} -``` - -Keep these calls in `advanceGCSafePointImpl` and `advanceTxnSafePointImpl`; do not add publication at public entry points. This gives modern and compatible callers exactly one event. - -- [ ] **Step 7: Run the publication and regression tests** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatch|Advance|Compatible|SetGCBarrier|DeleteGCBarrier)" -count=1' -``` - -Expected: complete upserts arrive once, no-op and barrier-only calls emit nothing, and only the slow watcher terminates. - -- [ ] **Step 8: Format and inspect live publication** - -Run: - -```bash -gofmt -w pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/errs/errno.go -git diff --check -``` - -Expected: formatting and whitespace checks pass before metrics are added. Do not commit yet because publication and its lifecycle accounting form one reviewable change. - -#### Lifecycle metrics phase - -This phase adds low-cardinality metrics at the manager-owned registration and termination points, with pre-bound counter handles for every reason. It remains part of Task 2 because the metric increments must share the same idempotent publication and cleanup boundaries they describe. - -- [ ] **Step 9: Write failing metric-delta tests** - -Use `prometheus/testutil.ToFloat64` and compare deltas so process-global counters do not make tests order-dependent: - -```go -func (s *gcStateManagerTestSuite) TestGCStateWatcherMetrics() { - re := s.Require() - activeBefore := promtestutil.ToFloat64(gcStateWatcherGauge) - leaderLostBefore := promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter) - - stop := s.manager.OnNodeBecomesLeader() - w, err := s.manager.WatchGCStates(context.Background(), true) - re.NoError(err) - re.Equal(activeBefore+1, promtestutil.ToFloat64(gcStateWatcherGauge)) - - stop() - re.ErrorIs(w.Err(), errs.ErrNotLeader) - re.Equal(activeBefore, promtestutil.ToFloat64(gcStateWatcherGauge)) - re.Equal(leaderLostBefore+1, promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter)) - w.Close() - re.Equal(leaderLostBefore+1, promtestutil.ToFloat64(gcStateWatcherTerminationLeaderLostCounter)) - stopRemainingCases := s.manager.OnNodeBecomesLeader() - defer stopRemainingCases() -} -``` - -In the same test, record the three remaining counters before their triggers. For `client_cancel`, register one watcher and call `Close`. For `slow_consumer`, register with live capacity 1 and perform two successful `AdvanceTxnSafePoint` calls without reading the watcher, so both publications run through the production path while `GCStateManager.mu` is held. For `init_error`, enable `iterateAllKeyspacesGCStatesError`, register with initial loading, and call `RecvBatch`. After each trigger, assert that only its expected counter increased by one and the active gauge returned to `activeBefore`; never mutate a metric directly. - -- [ ] **Step 10: Run the metric test and confirm the red state** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run TestGCStateManager/TestGCStateWatcherMetrics -count=1' -``` - -Expected: compilation fails because the watcher metrics do not exist. - -- [ ] **Step 11: Define and pre-bind the metrics** - -Add these definitions to `pkg/gc/metrics.go`: - -```go -gcStateWatcherGauge = prometheus.NewGauge(prometheus.GaugeOpts{ - Namespace: "pd", - Subsystem: "gc", - Name: "watcher_count", - Help: "Current number of active GC state watchers.", -}) -gcStateWatcherTerminationCounter = prometheus.NewCounterVec(prometheus.CounterOpts{ - Namespace: "pd", - Subsystem: "gc", - Name: "watcher_termination_total", - Help: "Total number of GC state watcher terminations by reason.", -}, []string{"reason"}) - -gcStateWatcherTerminationClientCancelCounter = gcStateWatcherTerminationCounter.WithLabelValues("client_cancel") -gcStateWatcherTerminationLeaderLostCounter = gcStateWatcherTerminationCounter.WithLabelValues("leader_lost") -gcStateWatcherTerminationSlowConsumerCounter = gcStateWatcherTerminationCounter.WithLabelValues("slow_consumer") -gcStateWatcherTerminationInitErrorCounter = gcStateWatcherTerminationCounter.WithLabelValues("init_error") -``` - -Register the gauge and vector with `prometheus.MustRegister`. Do not call `WithLabelValues` in registration, publication, or cleanup paths. - -- [ ] **Step 12: Record metrics at the single lifecycle boundaries** - -Increment the gauge only after successful insertion into the registry. In `terminateGCStateWatcherLocked`, decrement the gauge and invoke this switch only after deletion succeeds: - -```go -func recordGCStateWatcherTerminationMetrics(reason gcStateWatcherTerminationReason) { - switch reason { - case watcherTerminationClientCancel: - gcStateWatcherTerminationClientCancelCounter.Inc() - case watcherTerminationLeaderLost: - gcStateWatcherTerminationLeaderLostCounter.Inc() - case watcherTerminationSlowConsumer: - gcStateWatcherTerminationSlowConsumerCounter.Inc() - case watcherTerminationInitError: - gcStateWatcherTerminationInitErrorCounter.Inc() - default: - panic("unknown GC state watcher termination reason") - } -} -``` - -No metric uses watcher IDs, keyspace IDs, client addresses, or error text as labels. The gauge has no labels, so teardown decrements it rather than deleting a label series. - -- [ ] **Step 13: Run metric and lifecycle tests** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run "TestGCStateManager/Test(GCStateWatcherMetrics|GCStateWatch)" -count=1' -``` - -Expected: each lifecycle increments exactly one reason counter and returns the active gauge to its baseline. - -- [ ] **Step 14: Format and commit publication and observability** - -Run: - -```bash -gofmt -w pkg/gc/metrics.go pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go -git diff --check -git add pkg/gc/metrics.go pkg/gc/gc_state_watcher.go pkg/gc/gc_state_watcher_test.go pkg/gc/gc_state_manager.go pkg/errs/errno.go errors.toml -git commit -s -m "gc: publish and observe GC state changes" -``` - -### Task 3: Implement the gRPC transport adapter and upgrade kvproto - -This task adopts the merged protobuf API, converts domain changes, batches by exact wire size, and implements the local server-streaming handler through a small test seam. - -**Files:** - -- Modify: `go.mod` -- Modify: `go.sum` -- Modify: `client/go.mod` -- Modify: `client/go.sum` -- Modify: `tools/go.mod` -- Modify: `tools/go.sum` -- Modify: `tests/integrations/go.mod` -- Modify: `tests/integrations/go.sum` -- Modify: `server/gc_service.go` -- Create: `server/gc_service_test.go` - -**Interfaces:** - -- Consumes: Task 1's `GCStateManager.WatchGCStates` and `GCStateWatcher` API; Task 2's slow-consumer sentinel and domain changes; kvproto `WatchGCStatesRequest`, `WatchGCStatesResponse`, and `GCStateChange`. -- Produces: `GrpcServer.WatchGCStates`, `gcStateChangeToProto`, `splitWatchGCStatesResponses`, `watchGCStatesErrorToStatus`, and `serveWatchGCStates`. - -- [ ] **Step 1: Upgrade all four module scopes to the merged kvproto commit** - -Use the exact pseudo-version derived from merged commit `65b4e27a438de9274bf88c58e89e83749e62f646`: - -```bash -go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d -(cd client && go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d) -(cd tools && go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d) -(cd tests/integrations && go get github.com/pingcap/kvproto@v0.0.0-20260903062353-65b4e27a438d) -go mod tidy -(cd client && go mod tidy) -(cd tools && go mod tidy) -(cd tests/integrations && go mod tidy) -``` - -Run `git diff -- go.mod go.sum client/go.mod client/go.sum tools/go.mod tools/go.sum tests/integrations/go.mod tests/integrations/go.sum` and verify that the kvproto revision is the only dependency change. - -- [ ] **Step 2: Confirm that the upgraded server has a missing-method red state** - -Run: - -```bash -go test ./server -run '^$' -``` - -Expected: compilation reports that `*GrpcServer` does not implement `pdpb.PDServer` because `WatchGCStates` is missing. - -- [ ] **Step 3: Write failing converter and batching tests** - -Create table-driven converter coverage for a complete upsert, a removed scope, and zero-value invalid `gc.GCStateChange`. Assert that upserts have no barriers. Add boundary tests that calculate `base := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()})` and `delta := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}})`. - -```go -func TestSplitWatchGCStatesResponses(t *testing.T) { - change := &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{Upsert: &pdpb.GCState{ - KeyspaceScope: &pdpb.KeyspaceScope{Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: 7}}, - TxnSafePoint: 10, - GcSafePoint: 5, - }}} - base := proto.Size(&pdpb.WatchGCStatesResponse{Header: grpcutil.WrapHeader()}) - delta := proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}}) - - exact := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta) - require.Len(t, exact, 1) - require.LessOrEqual(t, proto.Size(exact[0]), base+2*delta) - - split := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change, change}, base+2*delta-1) - require.Len(t, split, 2) - for _, response := range split { - require.NotNil(t, response.GetHeader()) - require.NotEmpty(t, response.GetChanges()) - require.LessOrEqual(t, proto.Size(response), base+2*delta-1) - } - - oversized := splitWatchGCStatesResponses([]*pdpb.GCStateChange{change}, base+delta-1) - require.Len(t, oversized, 1) - require.Greater(t, proto.Size(oversized[0]), base+delta-1) - require.Empty(t, splitWatchGCStatesResponses(nil, base+delta)) -} -``` - -- [ ] **Step 4: Write failing stream-loop and error-mapping tests** - -Define a fake receiver implementing `RecvBatch(int) ([]gc.GCStateChange, error)` and `Err() error`, plus a fake `pdpb.PD_WatchGCStatesServer` whose `Send` callback can install a terminal cause. Cover these cases: - -- Two responses derived from one batch: after the first `Send`, set `errs.ErrNotLeader`; assert only one response is recorded and the function returns `Unavailable`. -- Invalid internal change: assert zero sends and gRPC `Internal`. -- `errs.ErrNotLeader` and an arbitrary initialization error: assert `Unavailable`. -- `errs.ErrGCStateWatcherSlowConsumer`: assert `ResourceExhausted`. -- `context.Canceled` and `context.DeadlineExceeded`: assert `Canceled` and `DeadlineExceeded`. - -Run: - -```bash -go test ./server -run 'Test(GCStateChangeToProto|SplitWatchGCStatesResponses|ServeWatchGCStates|WatchGCStatesErrorToStatus)' -count=1 -``` - -Expected: compilation fails because the adapter helpers do not exist. - -- [ ] **Step 5: Implement conversion and exact wire-size batching** - -Use these constants and receiver seam: - -```go -const ( - watchGCStatesRecvBatchSize = 1024 - maxWatchGCStatesResponseSize = 1 << 20 -) - -type gcStateChangeReceiver interface { - RecvBatch(maxChanges int) ([]gc.GCStateChange, error) - Err() error -} -``` - -Convert the internal discriminator into the generated oneof and reject the zero value: - -```go -func gcStateChangeToProto(change gc.GCStateChange) (*pdpb.GCStateChange, error) { - if state, ok := change.Upsert(); ok { - state.GCBarriers = nil - return &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Upsert{Upsert: gcStateToProto(state, time.Time{})}}, nil - } - if keyspaceID, ok := change.RemovedKeyspaceID(); ok { - return &pdpb.GCStateChange{Change: &pdpb.GCStateChange_Removed{Removed: &pdpb.KeyspaceScope{Keyspace: &pdpb.KeyspaceScope_KeyspaceId{KeyspaceId: keyspaceID}}}}, nil - } - return nil, errors.New("invalid GC state change") -} -``` - -Implement `splitWatchGCStatesResponses(changes []*pdpb.GCStateChange, maxSize int)` with a fresh `grpcutil.WrapHeader()` response for each batch. Start `currentSize` with `proto.Size` of the header-only response. For each change, calculate the exact additive delta with `proto.Size(&pdpb.WatchGCStatesResponse{Changes: []*pdpb.GCStateChange{change}})`. Flush a non-empty current response before an addition that exceeds `maxSize`; never append an empty response. If the first change itself exceeds the target with the header, keep it alone, log its serialized size, and start a fresh response for the next change. - -- [ ] **Step 6: Implement domain-error mapping and the send loop** - -Map only watcher-domain errors; return `Send` errors unchanged: - -```go -func watchGCStatesErrorToStatus(err error) error { - switch { - case errors.Is(err, errs.ErrGCStateWatcherSlowConsumer): - return status.Error(codes.ResourceExhausted, err.Error()) - case errors.Is(err, errs.ErrNotLeader): - return errs.ErrNotLeader - case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded): - return status.FromContextError(err).Err() - default: - return status.Error(codes.Unavailable, err.Error()) - } -} -``` - -`serveWatchGCStates` repeatedly receives at most 1024 changes, converts every change, splits them, checks `receiver.Err()` immediately before every `Send`, and returns the raw send error. Conversion failure logs the error and returns gRPC `Internal`; it does not create a watcher termination metric reason. - -```go -func serveWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error { - for { - changes, err := receiver.RecvBatch(watchGCStatesRecvBatchSize) - if err != nil { - return watchGCStatesErrorToStatus(err) - } - protoChanges := make([]*pdpb.GCStateChange, 0, len(changes)) - for _, change := range changes { - converted, err := gcStateChangeToProto(change) - if err != nil { - log.Error("failed to convert GC state change", zap.Error(err)) - return status.Error(codes.Internal, err.Error()) - } - protoChanges = append(protoChanges, converted) - } - for _, response := range splitWatchGCStatesResponses(protoChanges, maxResponseSize) { - if err := receiver.Err(); err != nil { - return watchGCStatesErrorToStatus(err) - } - if err := stream.Send(response); err != nil { - return err - } - } - } -} -``` - -- [ ] **Step 7: Implement local RPC preflight and stream ownership** - -Implement the generated method directly in `server/gc_service.go`: - -```go -func (s *GrpcServer) WatchGCStates(request *pdpb.WatchGCStatesRequest, stream pdpb.PD_WatchGCStatesServer) error { - done, err := s.rateLimitCheck() - if err != nil { - return err - } - if done != nil { - defer done() - } - if err := s.validateRequest(request.GetHeader()); err != nil { - return err - } - if s.GetRaftCluster() == nil { - return status.Error(codes.Unavailable, errs.ErrNotBootstrapped.FastGenByArgs().Error()) - } - - watcher, err := s.gcStateManager.WatchGCStates(stream.Context(), request.GetSkipLoadingInitial()) - if err != nil { - return watchGCStatesErrorToStatus(err) - } - defer watcher.Close() - return serveWatchGCStates(watcher, stream, maxWatchGCStatesResponseSize) -} -``` - -Do not call `unaryMiddleware` or create a delegate client. Calling `rateLimitCheck` in this public method preserves the `WatchGCStates` limiter label, and deferring `done` here holds the token until the stream exits. - -- [ ] **Step 8: Run transport tests and compile all kvproto consumers** - -Run: - -```bash -go test ./server -run 'Test(GCStateChangeToProto|SplitWatchGCStatesResponses|ServeWatchGCStates|WatchGCStatesErrorToStatus)' -count=1 -go test ./pkg/gc ./server -run '^$' -(cd client && go test ./... -run '^$') -(cd tools && go test ./... -run '^$') -(cd tests/integrations && go test ./... -run '^$') -``` - -Expected: the converter, batching, cancellation, and status tests pass, and every module compiles against the same kvproto revision. - -- [ ] **Step 9: Format and commit the transport implementation** - -Run: - -```bash -gofmt -w server/gc_service.go server/gc_service_test.go -git diff --check -git add go.mod go.sum client/go.mod client/go.sum tools/go.mod tools/go.sum tests/integrations/go.mod tests/integrations/go.sum server/gc_service.go server/gc_service_test.go -git commit -s -m "server: implement WatchGCStates stream" -``` - -### Task 4: Prove RPC behavior in a real PD cluster - -This task covers the complete server stream, request preflight, lifetime rate limiting, and leader transfer through real generated gRPC clients. - -**Files:** - -- Modify: `tests/server/gc/gc_test.go` -- Modify if a test exposes an implementation defect: `server/gc_service.go` -- Modify if a test exposes a domain defect: `pkg/gc/gc_state_watcher.go` - -**Interfaces:** - -- Consumes: The generated `pdpb.PDClient.WatchGCStates` client and all production behavior from Tasks 1 through 3. -- Produces: End-to-end evidence for initial loading, skip-initial registration, validation, rate-limit token lifetime, and leadership reconnection. - -- [ ] **Step 1: Add deterministic stream helpers** - -Add a failpoint name constant for `github.com/tikv/pd/pkg/gc/watchGCStatesRegistered` and helpers that use bounded contexts: - -```go -func recvWatchGCStateForKeyspace(t *testing.T, stream pdpb.PD_WatchGCStatesClient, keyspaceID uint32) *pdpb.GCState { - t.Helper() - for { - response, err := stream.Recv() - require.NoError(t, err) - require.NotNil(t, response.GetHeader()) - for _, change := range response.GetChanges() { - if state := change.GetUpsert(); state != nil && state.GetKeyspaceScope().GetKeyspaceId() == keyspaceID { - return state - } - } - } -} -``` - -Use `context.WithTimeout(..., 20*time.Second)` for every stream and clean up every connection, context, and failpoint with `t.Cleanup` or `defer`. - -- [ ] **Step 2: Write the failing initial and skip-initial RPC test** - -In a bootstrapped one-node cluster, create a keyspace-level GC keyspace and establish `skip_loading_initial=false`. Read until that keyspace's complete initial state arrives, advance its transaction safe point, and read the complete live state. - -For `skip_loading_initial=true`, enable `watchGCStatesRegistered` with a `sync.Once` callback that closes a channel, establish the stream, wait for the callback, advance the same keyspace again, and assert the first received state contains the post-registration value. The registration callback removes timing sleeps and proves that no initial value was sent. - -```go -stream, err := grpcPDClient.WatchGCStates(ctx, &pdpb.WatchGCStatesRequest{Header: header, SkipLoadingInitial: true}) -require.NoError(t, err) -select { -case <-registered: -case <-time.After(5 * time.Second): - require.FailNow(t, "WatchGCStates was not registered") -} -``` - -- [ ] **Step 3: Write failing request-preflight tests** - -Use table-driven subtests that call `Recv` to observe server-stream establishment failures: - -- A request with the wrong cluster ID returns `codes.FailedPrecondition`. -- A direct request to a follower returns `codes.Unavailable` and is not forwarded. -- A request to the elected leader before `BootstrapCluster` returns `codes.Unavailable`. - -Assert that none of these cases returns a response message. - -- [ ] **Step 4: Write the failing lifetime rate-limit test** - -Enable gRPC rate limiting through `GetServiceMiddlewarePersistOptions().SetGRPCRateLimitConfig`, then call `GetGRPCRateLimiter().Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(1))`. Use the registration failpoint to wait until stream 1 holds its token. Assert stream 2 returns `codes.ResourceExhausted`. Cancel stream 1, use `GetConcurrencyLimiterStatus("WatchGCStates")` with `testutil.Eventually` until current usage is zero, then establish stream 3 and receive a change after a safe-point advancement. Restore the prior rate-limit config and delete the test limiter with `ratelimit.UpdateConcurrencyLimiter(0)` during cleanup. - -- [ ] **Step 5: Write the failing leader-transfer and reinitialization test** - -Reuse `newGCStateLeaderTransitionCluster`. Start an initial watch directly against the old leader and receive its current null-keyspace state. Resign that leader, wait for a different leader, and drain the old stream until it returns `codes.Unavailable`. Advance the safe point on the new leader, connect to the new leader with `skip_loading_initial=false`, and assert its initial state contains the new value. - -- [ ] **Step 6: Run the real-cluster tests and confirm the red or green state** - -Run: - -```bash -make gotest GOTEST_ARGS='./tests/server/gc -run ^TestWatchGCStates -count=1' -``` - -Expected before any required correction: at least one new test fails if the production path does not meet its contract. If all tests pass immediately, retain the tests as integration coverage and do not alter production code. - -- [ ] **Step 7: Make only corrections demonstrated by the failing integration assertions** - -Keep corrections inside the approved interfaces. Typical permitted corrections are preflight ordering, status mapping, cleanup order, or an additional cancellation check before `Send`. Do not add metadata producers, cross-PD transaction fencing, a replay protocol, a Go client, or `WatchGCSafePointV2` behavior. - -After each correction, rerun: - -```bash -make gotest GOTEST_ARGS='./tests/server/gc -run ^TestWatchGCStates -count=1' -make gotest GOTEST_ARGS='./pkg/gc -run TestGCStateManager/TestGCStateWatch -count=1' -make gotest GOTEST_ARGS='./server -run WatchGCStates -count=1' -``` - -Expected: all focused domain, transport, and integration tests pass. - -- [ ] **Step 8: Format and commit real-cluster coverage** - -Run: - -```bash -gofmt -w tests/server/gc/gc_test.go server/gc_service.go pkg/gc/gc_state_watcher.go -git diff --check -git add tests/server/gc/gc_test.go server/gc_service.go pkg/gc/gc_state_watcher.go -git commit -s -m "tests: cover WatchGCStates lifecycle" -``` - -Omit unchanged production files from `git add`. If integration testing required no production correction, commit only `tests/server/gc/gc_test.go`. - -## Final verification checklist - -This checklist verifies formatting, generated error documentation, module consistency, race safety, focused behavior, and the repository's required checks after all four implementation tasks are complete. It is a handoff gate rather than a separate implementation task or commit. - -**Files:** - -- Verify: all files changed in Tasks 1 through 4 -- Modify: none unless a verification command reports a concrete defect - -**Interfaces:** - -- Consumes: The complete implementation and tests. -- Produces: Fresh command output demonstrating that the branch is ready for review. - -- [ ] **Step 1: Ensure failpoints are disabled before non-test commands** - -Run: - -```bash -make failpoint-disable -``` - -Expected: failpoint-generated rewrites are removed before formatting or static analysis. - -- [ ] **Step 2: Verify formatting and module tidiness** - -Run: - -```bash -make fmt -make generate-errdoc -make tidy -git diff --check -``` - -Expected: `make tidy` and `git diff --check` exit successfully. Inspect any formatting or generated change and commit it with the task that introduced the affected file; do not create an unexplained cleanup commit. - -- [ ] **Step 3: Run focused package tests with failpoint handling** - -Run: - -```bash -make gotest GOTEST_ARGS='./pkg/gc -run ^TestGCStateWatcher -count=1' -make gotest GOTEST_ARGS='./pkg/gc -run TestGCStateManager/TestGCStateWatch -count=1' -make gotest GOTEST_ARGS='./server -run "Test(GCStateChangeToProto|SplitWatchGCStatesResponses|ServeWatchGCStates|WatchGCStatesErrorToStatus)" -count=1' -make gotest GOTEST_ARGS='./tests/server/gc -run ^TestWatchGCStates -count=1' -``` - -Expected: every focused test passes with zero failures. - -- [ ] **Step 4: Run the focused race check** - -Run: - -```bash -make gotest GOTEST_ARGS='-race ./pkg/gc -run ^TestGCStateWatcher -count=1' -make gotest GOTEST_ARGS='-race ./pkg/gc -run TestGCStateManager/TestGCStateWatch -count=1' -make gotest GOTEST_ARGS='-race ./server ./tests/server/gc -run WatchGCStates -count=1' -``` - -Expected: all focused tests pass under the race detector with no race report or goroutine leak. - -- [ ] **Step 5: Run repository-level checks** - -Run: - -```bash -make check -make basic-test -``` - -Expected: formatting, lint, leak checks, generated error documentation, and the basic unit-test suite pass. - -- [ ] **Step 6: Verify scope and repository hygiene** - -Run: - -```bash -make failpoint-disable -! rg -n 'WatchGCSafePointV2' pkg/gc/gc_state_watcher.go server/gc_service.go tests/server/gc/gc_test.go -git status --short -git log --oneline --decorate -10 -``` - -Expected: the `rg` command finds no new compatibility reference in the touched WatchGCStates paths, `git status --short` is empty, and the log shows the signed task commits in order. - -## Execution handoff - -The plan is complete when this document is reviewed and committed. Execute its four implementation tasks with one of the required workflows: - -1. **Subagent-driven:** Use `superpowers:subagent-driven-development`, dispatch a fresh worker for each task, and perform spec and code-quality review between tasks. -2. **Inline execution:** Use `superpowers:executing-plans`, execute tasks in batches, and stop at its review checkpoints. diff --git a/docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md b/docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md deleted file mode 100644 index 18a49b53d4b..00000000000 --- a/docs/superpowers/plans/2026-09-16-watch-gc-states-blocked-send.md +++ /dev/null @@ -1,242 +0,0 @@ -# WatchGCStates Blocked Send Cancellation Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let `WatchGCStates` return and release its concurrency-limit token when its watcher terminates, even while gRPC flow control blocks a response send, and verify that the sending goroutine exits after RPC teardown. - -**Architecture:** Run the existing receive/convert/send loop in one worker goroutine per RPC. The handler supervises the worker through a buffered result channel and the watcher's termination channel. The handler returns on watcher termination; gRPC then tears down the transport stream and unblocks the worker. - -**Tech Stack:** Go 1.25+, the repository-pinned grpc-go v1.82.1, existing PD GC watchers, failpoint-aware Go tests, and `bufconn` for transport coverage. - -**Spec:** [WatchGCStates server design](../specs/2026-09-03-watch-gc-states-design.md), especially “Lifecycle and errors” and “GC service”, refined by the decisions below. This document is the implementation handoff for the second review finding on PR #11264. The first finding, initial/live ordering, was fixed in `65c7ca00b`; preserve that fix, including `pendingLiveCount`. - -**Status:** Implemented and verified against base `65c7ca00b`. Both fake-send and real-transport regressions failed against the original synchronous handler, then passed with the fix. Targeted race tests passed in `pkg/gc`, `server`, and `tests/server/gc`; the real-transport regression passed 20 repeated race runs, and affected-package basic tests passed. Independent review found no blocking issues. A full `make check` attempt passed module tidying and formatting but stopped because `golangci-lint` is unavailable in the local environment; the remaining checks still need CI or a fully provisioned environment. The task descriptions below retain the implementation and acceptance instructions for reference. - -## Global constraints - -- Limit production changes to `WatchGCStates` and its watcher termination notification. Other streaming RPCs, generic gRPC infrastructure, protobuf, and dependency versions are outside this task. -- Keep `rateLimitCheck()` and both deferred cleanup operations in the public `WatchGCStates` handler. Its caller-derived rate-limit label must remain `WatchGCStates`. -- Keep one receiving/sending worker per RPC and at most one `Send` in progress. Keep all existing per-response terminal checks, response-size limits, ordering, queue capacities, and manager lock scopes. -- Add no per-send goroutine, timer, response queue, or general send timeout. An open stream without a watcher termination event remains governed by the existing policy. -- The worker result channel has capacity **1**. The handler returns without joining the worker; RPC teardown supplies the cancellation needed to finish a blocked `Send`. -- Preserve error mappings and watcher termination metrics. Use `Unavailable` for leader loss, `ResourceExhausted` for slow consumers, and context status codes for caller cancellation/deadline. Preserve a raw send/conversion error when no watcher terminal cause has been recorded. -- Follow repository `AGENTS.md`. Run tests through failpoint-aware targets and restore failpoints before editing, reviewing diffs, or committing. If delegating, one owner coordinates test runs and edits in this shared worktree. - -## Decision and lifetime ownership - -`GCStateWatcher` owns a child context derived from `stream.Context()`. Canceling that child wakes `RecvBatch`, but does not cancel the parent transport context observed by gRPC `Send`. The existing synchronous handler cannot reach its next terminal check while sending is blocked. - -The chosen design makes the handler independently responsive to watcher termination: - -```text -watcher terminates - -> supervisor returns the mapped terminal cause - -> public handler runs watcher.Close() and releases its rate-limit token - -> gRPC processes the handler return and closes the transport stream - -> blocked Send returns - -> worker publishes its result into the buffered channel and exits -``` - -The worker may briefly outlive the handler. It retains the response it is sending until `Send` returns; no response is mutated or recycled during that interval. The worker must not wait for a consumer of its final result. Waiting for the worker inside a handler defer would reverse this dependency and deadlock the teardown path. - -The public handler remains the owner of watcher cleanup. The supervisor does not close the watcher itself. For the real caller, the watcher is always derived from the RPC context, so its `Done` notification also covers client cancellation. There is no need for a third, independently managed cancellation context or a redundant `stream.Context().Done()` select arm. - -When worker completion races with watcher cancellation, prefer an already-recorded watcher cause when choosing the return value. This makes a transport cancellation caused by server teardown less likely to obscure the domain reason. It does not establish a total order for events that have not yet been recorded; the watcher continues to own its first cancellation cause. - -### Verified transport assumption - -In the pinned grpc-go source, `serverStream.SendMsg` calls the transport write path, whose `writeQuota.get` waits on quota or the transport stream's cancellation. `Server.processStreamingRPC` handles the application handler's return via `WriteStatus`, and `http2Server.finishStream` cancels the transport stream before queuing its final trailers. Thus handler return can release the blocked send without waiting for the client to start consuming responses. - -Recheck these functions in the pinned module if its version changes: - -- `google.golang.org/grpc/stream.go`: `serverStream.SendMsg`. -- `google.golang.org/grpc/server.go`: `Server.processStreamingRPC`. -- `google.golang.org/grpc/internal/transport/http2_server.go`: `writeStatus`, `finishStream`, and `write`. -- `google.golang.org/grpc/internal/transport/flowcontrol.go`: `writeQuota.get`. -- `google.golang.org/grpc/internal/transport/controlbuf.go`: `loopyWriter.processData`, which replenishes write quota only for bytes allowed by stream flow control. - -This guarantees a local cancellation path, not immediate delivery of the final status to a client that refuses to read. Queued response data/trailers may still require client progress. Test handler return and worker exit before resuming client reads; inspect the client status afterward. - -## Files and interfaces - -| File | Responsibility | -| --- | --- | -| `pkg/gc/gc_state_watcher.go` | Add `Done() <-chan struct{}` exposing the existing watcher context's termination channel. | -| `pkg/gc/gc_state_watcher_test.go` | Verify notification, cause visibility, and existing lifecycle behavior. | -| `server/gc_service.go` | Add `Done` to `gcStateChangeReceiver`; make `serveWatchGCStates` the supervisor and move its existing loop unchanged into `sendWatchGCStates`. | -| `server/gc_service_test.go` | Adapt existing receiver fakes; add deterministic cancellation tests and real gRPC teardown coverage. | -| `tests/server/gc/gc_test.go` | Verify public-handler registration, cleanup, status, and concurrency-token release during blocked sending. | -| `docs/superpowers/specs/2026-09-03-watch-gc-states-design.md` | Document worker/supervisor ownership and distinguish unrecalled sends from handler lifetime. | - -`GCStateWatcher.Done` is concurrency-safe and returns the same channel on every call. Once that channel is closed, `Err()` returns the terminal cause. Reading `Done` must not allocate a goroutine, acquire a manager lock, or register a new callback. - -## Task 1: Add cancellation supervision and deterministic regressions - -**Interfaces produced:** `(*gc.GCStateWatcher).Done() <-chan struct{}`, the extended `gcStateChangeReceiver`, and the internal synchronous worker `sendWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error`. - -- [ ] Add a context-backed receiver fake for cancellation tests. Use `context.WithCancelCause(streamCtx)` and return its `Done`/`Cause` from the fake; keep mutable batch state exclusively in the receiving worker. Existing non-cancellation fakes may return a nil `Done` channel, which disables that select arm. Never write the existing fake's plain `terminalErr` concurrently with the supervisor reading it. - - A sufficient receiver for the blocked-send tests is: - - ```go - type cancelableGCStateReceiver struct { - ctx context.Context - changes []gc.GCStateChange - } - - func (r *cancelableGCStateReceiver) Done() <-chan struct{} { return r.ctx.Done() } - func (r *cancelableGCStateReceiver) Err() error { return context.Cause(r.ctx) } - - func (r *cancelableGCStateReceiver) RecvBatch(maxChanges int) ([]gc.GCStateChange, error) { - if err := r.Err(); err != nil { - return nil, err - } - if len(r.changes) == 0 { - <-r.Done() - return nil, r.Err() - } - n := min(maxChanges, len(r.changes)) - batch := r.changes[:n] - r.changes = r.changes[n:] - return batch, nil - } - ``` - -- [ ] Write `TestServeWatchGCStatesCancellationUnblocksHandler` as table-driven cases for `errs.ErrNotLeader`/`Unavailable` and `errs.ErrGCStateWatcherSlowConsumer`/`ResourceExhausted`. For each case, create one upsert and use `fakeWatchGCStatesServer.sendHook` with the following behavior: - - ```go - sendStarted := make(chan struct{}) - sendExited := make(chan struct{}) - stream.sendHook = func(*pdpb.WatchGCStatesResponse) error { - close(sendStarted) - defer close(sendExited) - <-streamCtx.Done() - return streamCtx.Err() - } - handlerDone := make(chan error, 1) - go func() { handlerDone <- serveWatchGCStates(receiver, stream, 1024) }() - ``` - - Wait for `sendStarted`, cancel only the receiver with the table's cause, then require `handlerDone` to return the expected status within **5 seconds**, while `streamCtx.Err()` remains nil and `sendExited` remains open. Only afterward call the stream cancel function to model gRPC teardown, and require `sendExited` to close. Register failure cleanup before starting the goroutine: cancel both contexts and release/wait for test goroutines with bounded waits even if an assertion fails. This fake deliberately models the transport cancellation boundary rather than pretending watcher cancellation directly ends `Send`. - -- [ ] Run the new test against the synchronous implementation and record the expected failure: the handler fails to return before the stream is canceled. The fake's extra `Done` method does not require changing the old production interface to compile this red test. Do not accept a test that first cancels the client/stream and only then checks handler completion. - -- [ ] Implement the watcher notification and supervisor as follows, retaining the public handler's existing `defer watcher.Close()` and `defer done()`: - - ```go - // Done returns a channel that is closed when the watcher terminates. - func (w *GCStateWatcher) Done() <-chan struct{} { - return w.ctx.Done() - } - - type gcStateChangeReceiver interface { - RecvBatch(maxChanges int) ([]gc.GCStateChange, error) - Done() <-chan struct{} - Err() error - } - - func serveWatchGCStates(receiver gcStateChangeReceiver, stream pdpb.PD_WatchGCStatesServer, maxResponseSize int) error { - if err := receiver.Err(); err != nil { - return watchGCStatesErrorToStatus(err) - } - resultCh := make(chan error, 1) - go func() { - defer logutil.LogPanic() - resultCh <- sendWatchGCStates(receiver, stream, maxResponseSize) - }() - select { - case err := <-resultCh: - if cause := receiver.Err(); cause != nil { - return watchGCStatesErrorToStatus(cause) - } - return err - case <-receiver.Done(): - return watchGCStatesErrorToStatus(receiver.Err()) - } - } - ``` - - Import the repository's `pkg/utils/logutil`. Rename the old `serveWatchGCStates` body to `sendWatchGCStates` without changing its receive loop, conversion, splitting, per-response `Err` check, or raw error returns. `logutil.LogPanic` follows the repository's goroutine convention; it logs fatally rather than silently recovering and stranding the supervisor. - -- [ ] Add the following focused assertions alongside the regression, then run them under the race detector: - - | Case | Required observation | - | --- | --- | - | Watcher terminates before serving starts | Mapped error returns and the fake stream's send hook is never called. | - | Cancellation while waiting in `RecvBatch` | Handler and worker finish; no response is sent. | - | Parent stream context canceled | Watcher notification wakes the supervisor; cancellation status is preserved. | - | Worker send fails with no watcher cause | Existing `TestServeWatchGCStatesReturnsRawSendError` still returns the same error. | - | Invalid internal change | Existing `Internal` mapping remains unchanged. | - | Cancellation between split responses | Existing per-send terminal check prevents sending the remaining response. | - | Watcher `Done` notification | Notification closes on cancellation; `Err` exposes the first cause and repeated `Done` calls return the same channel. | - - The process-level goroutine leak checker must pass. Tests must wait for their own cleanup and worker-visible exit conditions; do not reuse fake state while a worker may still access it. - -**Completion:** The blocked-send red test passes without canceling its stream first, existing server adapter tests pass under `-race`, and the diff contains one worker/channel per RPC with no manager lock changes. - -## Task 2: Verify real transport teardown and public-handler cleanup - -**Interfaces consumed:** The Task 1 supervisor and existing public `WatchGCStates` handler. Test doubles must honor the receiver batch bound and first-cause contract. - -- [ ] Add `TestWatchGCStatesTransportCancellationUnblocksSend` in `server/gc_service_test.go`. Reuse the `bufconn` setup pattern in `server/grpc_service_test.go`. Register a test service embedding `pdpb.UnimplementedPDServer`; its `WatchGCStates` implementation creates a receiver derived from the real `stream.Context()`, invokes `serveWatchGCStates`, reports the returned error through a buffered test channel, and returns it to gRPC. Supply at least **16 batches of 1024 complete upserts** with large timestamp values and distinct keyspace IDs so the generated stream exceeds **256 KiB**; have `RecvBatch` wait on its context after exhausting the fixture. - - Configure the client with: - - ```go - grpc.WithStaticStreamWindowSize(64 << 10), - grpc.WithStaticConnWindowSize(64 << 10), - ``` - - Use the normal uncompressed protobuf codec and a `bufconn` listener capacity of **1 MiB**. Keep the client connection and RPC context alive, and do not call client `Recv` before triggering server-side watcher cancellation. A **30-second** RPC deadline is a failure guard, not the cancellation mechanism under test. - -- [ ] Make the blocked-send observation deterministic using a test-only wrapper around the real `PD_WatchGCStatesServer`. grpc-go v1.82.1 starts with a **64 KiB** stream write quota (`internal/transport/defaults.go`, `defaultWriteQuota`). With a static **64 KiB** client receive window and no application reads, cumulative successful send enqueues can exhaust that quota plus the receive window. Track successful wire bytes using `response.Size() + 5` for the gRPC message envelope: - - ```go - func (s *observedWatchGCStatesStream) Send(response *pdpb.WatchGCStatesResponse) error { - // These test-only constants match the pinned grpc-go implementation - // and the explicitly configured client receive window. - const receiveWindow = 64 << 10 - const writeQuota = 64 << 10 - if !s.observed && s.sentWireBytes >= receiveWindow+writeQuota { - s.observed = true - close(s.blockedSendStarted) - defer close(s.blockedSendExited) - } - err := s.PD_WatchGCStatesServer.Send(response) - if err == nil { - s.sentWireBytes += response.Size() + 5 - } - return err - } - ``` - - Define the wrapper with an embedded `pdpb.PD_WatchGCStatesServer`, `sentWireBytes int`, `observed bool`, and the two notification channels. Only the sending worker accesses its counter and flag. The next send after that cumulative threshold cannot regain quota while the client does not read. The marker fires immediately before that send, which also covers cancellation racing with entry into `Send`; the original synchronous implementation still cannot observe watcher cancellation there. Recheck the byte/quota argument if the transport version or compression settings change; do not replace this observation with a sleep or a fake blocked `Send`. - -- [ ] After the marker, cancel only the receiver with the domain cause. Require the test service's handler result and `blockedSendExited` within **5 seconds**, without canceling the client or stopping the gRPC server to make those assertions succeed. Then drain client responses to the terminal status and assert the expected code. Cover leader-loss and slow-consumer causes. Register unconditional cleanup so failed assertions close the client/server and release goroutines. Verify that the test fails against the old synchronous supervisor and passes after Task 1; run the focused transport test repeatedly with `-race`. - -- [ ] Extend public-handler tests in `tests/server/gc/gc_test.go` using the existing `TestWatchGCStatesSendFailureCleansUpPublicHandler` setup. Enable concurrency limit **1**, observe registration, and block a test stream's `Send` on its own context. For leader loss, supersede the local manager generation with `stop := manager.OnNodeBecomesLeader()` and register `stop` for cleanup; this is a controlled domain transition, while the existing real leader-transfer test remains part of regression coverage. For slow consumption, once `Send` is blocked, produce **1025** subsequent successful increasing txn-safe-point updates to overflow the default **1024** live queue. Use the null keyspace and increasing targets so writes are neither rejected nor no-ops. If needed, factor the existing test setup into a helper instead of duplicating all rate-limit configuration code. - - Before releasing the fake transport, require: the public handler returns the domain status, `pd_gc_watcher_count` returns to its baseline, the appropriate termination counter increases once, and `GetConcurrencyLimiterStatus("WatchGCStates")` reports **0** current streams. Confirm a subsequent registered watch is admitted. On the leader-generation test, the newly superseded generation remains active until cleanup, so this admission check can use the same one-node fixture. Only then cancel the fake stream context, wait for its blocked send to exit, and clean up the subsequent watch. The fake's context cancellation models the real teardown proved by the preceding transport test. - -**Completion:** Both domain causes return the expected status and free the limit token before a stalled client resumes, and the real transport test proves the blocked send exits after handler return. Existing real leader transfer and send-error cleanup tests still pass. No goroutine leak is hidden by premature client/server shutdown. - -## Task 3: Update the design and run final verification - -- [ ] Update the existing design document's “GC service” and “Lifecycle and errors” sections to describe the supervisor/worker split, the one-worker/one-result-channel cost, cleanup ownership, and the prohibition on joining the worker before returning. Replace the ambiguous in-progress-send sentence with this contract: - - > A response send already in progress cannot be recalled. Watcher termination nevertheless causes the RPC handler to return without waiting for that send. Handler return initiates gRPC transport teardown, which interrupts blocked sending; the worker then exits. The handler retains ownership of watcher cleanup and rate-limit token release. A client that is not reading may observe the terminal status only after draining already queued responses. - -- [ ] Format touched Go files, run `git diff --check`, and execute the narrow tests before broader validation: - - ```sh - make gotest GOTEST_ARGS='./pkg/gc ./server -run "TestGCStateWatcher|TestServeWatchGCStates|TestWatchGCStatesTransport" -count=1 -tags=without_dashboard,deadlock -race -timeout=5m' - make gotest GOTEST_ARGS='./server -run TestWatchGCStatesTransportCancellationUnblocksSend -count=20 -tags=without_dashboard,deadlock -race -timeout=5m' - make gotest GOTEST_ARGS='./tests/server/gc -run TestWatchGCStates -count=1 -tags=without_dashboard,deadlock -race -timeout=10m' - GOFLAGS='-tags=without_dashboard' make basic-test BASIC_TEST_PKGS='./pkg/gc ./server' - ``` - - In this worktree, the tools required by these targets are already installed; `make -o install-tools ...` can reuse them. The `without_dashboard` tag avoids dependence on generated Dashboard assets for the integration tests. Keep regexes free of unescaped trailing `$` inside `GOTEST_ARGS`: Make can interpret the following character as a variable reference and break the shell quoting. If a wrapper fails before its cleanup branch runs, immediately run `make failpoint-disable` (or the installed-tool equivalent) before further work. - -- [ ] Review the final diff against every global constraint and the acceptance cases above. Run required repository checks before PR submission, including `make check`. Confirm failpoint-generated files are absent and only intended source/test/documentation changes remain. Hand back the changed files, red/green evidence, real-transport evidence, and any unresolved failure; do not describe fake-send coverage as proof of transport worker cleanup. - -**Completion:** Review and verification are complete, the handoff report distinguishes local handler exit from client-visible status delivery, and the implementation introduces no per-response concurrency or broad streaming-RPC refactor. diff --git a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md b/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md deleted file mode 100644 index 1bc2d4f86dc..00000000000 --- a/docs/superpowers/specs/2026-09-03-watch-gc-states-design.md +++ /dev/null @@ -1,248 +0,0 @@ -# WatchGCStates server design - -`WatchGCStates` provides an ordered stream of complete, effective GC states for keyspaces. This design adds the PD server implementation for the API introduced by [kvproto PR #1528](https://github.com/pingcap/kvproto/pull/1528), while keeping the watch mechanism isolated from the gRPC transport and from the legacy `WatchGCSafePointV2` API. - -## Context - -The original draft in [PD PR #10498](https://github.com/tikv/pd/pull/10498) established the motivation for a streaming API, but its implementation coupled initial loading and live delivery in ways that could reorder states, allowed a slow client to affect unrelated watchers, and included compatibility work that is no longer required. This design retains the useful product semantics from the original [WatchGCStates proposal](https://pingcap.feishu.cn/wiki/TNJSw3rWGiCwjJk8iOtcyTDrnSe) and incorporates the concerns from [review 5097472165](https://github.com/tikv/pd/pull/10498#pullrequestreview-5097472165). - -The merged protobuf API represents every notification as a `GCStateChange` containing either a complete `upsert` state or a `removed` keyspace scope. Consumers apply changes in stream order to maintain a materialized view. - -## Goals - -The implementation has a deliberately narrow set of goals: - -- Stream the initial effective GC state of every active keyspace observed by the initial scan when requested. -- Stream effective safe-point changes committed through the local `GCStateManager` after a watcher is registered. -- Guarantee that a watcher never observes an older state for a keyspace after a newer live state for that keyspace on the same stream. -- Prevent a slow watcher from blocking GC advancement or affecting another watcher. -- Terminate watchers promptly and predictably when PD loses leadership. -- Keep the watcher implementation independently testable in `pkg/gc`. -- Bound response sizes using the actual protobuf wire size. - -## Non-goals - -The first implementation intentionally excludes adjacent features that are not needed to deliver the API safely: - -- Compatibility with `WatchGCSafePointV2`. -- A Go PD client implementation. -- Streaming GC barriers or global barriers. -- A globally atomic snapshot across keyspaces. -- A revision, cursor, replay log, or resume-from-revision protocol. -- Sharing one initial scan among multiple watchers. -- Producing events from keyspace creation, enablement, disablement, deletion, or GC-mode changes. -- Fencing GC-state write transactions with the PD leadership lease or a cluster-wide leadership epoch. - -## Stream contract - -The stream is a sequence of self-contained changes. An `upsert` replaces the consumer's entire state for its scope, and a `removed` change deletes that scope from the consumer's materialized view. Upserts do not contain GC barriers; barrier-only mutations do not directly produce changes. - -For `skip_loading_initial=false`, PD registers the live listener before starting the initial scan. Initial and live changes may be interleaved, and the initial scan is not a cross-keyspace transaction. Before emitting a newly acquired initial batch, the server drains the live changes already queued when it acquired that batch. For each individual keyspace, it suppresses an initial value if a post-registration live value for the same scope has already been emitted. These rules prevent regression both from a newer initial value to an older queued live value and from a newer live value to an older initial value. - -For `skip_loading_initial=true`, PD sends only effective safe-point changes produced after registration. This mode does not provide continuity with an earlier stream and is unsuitable for constructing a complete view on its own. A client establishing its first complete view or recovering from a disconnected stream uses `skip_loading_initial=false`. - -Clients should clear their materialized GC-state view before an initial connection or reconnection with `skip_loading_initial=false`, unless they independently reconcile stale entries. This is a recommended client convention rather than a server-enforced requirement. The first server implementation does not yet produce lifecycle-driven `removed` events, so a client that retains an old view cannot otherwise guarantee removal of scopes deleted while it was disconnected. - -The protocol does not expose an initial-scan completion marker. Consumers continuously apply changes in arrival order; correctness does not depend on distinguishing initial changes from live changes. - -The first implementation does not subscribe to keyspace metadata changes after registration. A connected client might not learn that a keyspace was created, enabled, disabled, deleted, or switched between unified and keyspace-level GC until it reconnects, receives a later safe-point upsert that reflects the new metadata, or reconciles that metadata independently. - -## Architecture - -The design separates state observation, ordered merging, and transport adaptation into three layers. The deeper watcher module owns concurrency and lifecycle policy so the gRPC handler only deals with request validation, protobuf conversion, and response delivery. - -### GC state manager - -`GCStateManager` owns a registry of active watchers. Its existing mutex serializes watcher registration, publication of effective safe-point changes, removal of slow watchers, and local leadership transitions. These guarantees apply within one manager and local leadership generation. - -The `pkg/gc` layer defines an internal `GCStateChange` representation with `upsert` and `removed` variants. The type is independent of protobuf. An upsert contains a complete effective GC state, including the scope, whether GC is managed at keyspace level, the transaction safe point, and the GC safe point. A removed change contains the affected scope. - -The initial implementation produces upserts from successful safe-point mutations. It supports removed changes throughout the watcher and transport pipeline so keyspace lifecycle integration can be added without redesigning the stream. The implementation leaves an explicit TODO at the keyspace lifecycle integration point for creation, state, deletion, and GC-mode changes rather than adding an incomplete lifecycle dependency in this change. - -### GC state watcher - -Watcher mechanics live in a focused file such as `pkg/gc/gc_state_watcher.go`. Each watcher owns the following state: - -- `initCh`, a buffered channel of initial-state batches. -- `liveCh`, a bounded channel of individual live changes. -- `initDone`, which is owned by the merge consumer and becomes true when initial loading was skipped or after the closed `initCh` has been fully drained. -- A cause-aware cancellation mechanism used for both cleanup and error reporting. -- Merge state, including the pending initial batch, the remaining count of live changes that must precede it, and the set of scopes made dirty by live delivery while initial loading is active. - -Only the initial loader writes to and closes `initCh`. Publishers write to `liveCh` only while holding the manager mutex, but `liveCh` is not closed; watcher cancellation is the termination signal. This ownership rule avoids send-versus-close races. - -The watcher exposes a receive operation that returns at most a requested number of visible changes. `RecvBatch(1024)` blocks until at least one change or a terminal cause is available, then opportunistically collects already available changes without waiting to fill the batch. The watcher, rather than the gRPC handler, owns the initial/live merge and its ordering invariant. - -### GC service - -`server/gc_service.go` remains a thin adapter. Its public `WatchGCStates` method performs the rate-limit check directly, validates the request locally, registers a watcher, supervises streaming, and closes the watcher on every return path. One worker goroutine per RPC receives changes, converts them to protobuf, splits them into wire-size-bounded responses, and sends them sequentially. A result channel with capacity one lets the worker finish even after the handler returns. The supervisor selects between that result and the watcher’s `Done` notification, preferring an already-recorded watcher cause when processing a worker result. The handler does not proxy the long-lived stream: a non-serving or unbootstrapped member returns `Unavailable`, and existing header and cluster-ID validation semantics remain unchanged. - -Keeping `rateLimitCheck` in the public handler preserves the externally visible method name `WatchGCStates` in the caller-derived rate-limit label. The rate-limit token is held for the lifetime of the stream and released when the handler returns. Every successful `WatchGCStatesResponse` contains `grpcutil.WrapHeader()`; a structurally invalid internal change is logged and returned as gRPC `Internal`. - -## Registration and initial loading - -Registration establishes the boundary between pre-existing state and live changes. The sequence is: - -1. Lock `GCStateManager.mu`. -2. Verify that this PD member has an active local GC leadership generation. -3. Create the watcher and add it to the registry. -4. Unlock `GCStateManager.mu`. -5. If `skip_loading_initial=false`, start the initial loader. Otherwise, mark initial loading complete immediately. - -Registering before scanning ensures that every effective safe-point change published by the same manager and leadership generation after the registration point is either queued as live data or causes that watcher to terminate as a slow consumer. No such change can fall into a gap between snapshot setup and live subscription. - -The initial loader calls the incremental `iterateAllKeyspacesGCStates` path rather than `GetAllKeyspacesGCStates`, which materializes the full result before returning. It requests states without barriers and preserves the current handling of inactive keyspaces and unified GC mode. - -The loader never holds `GCStateManager.mu` while constructing a batch or waiting for `initCh` capacity, so client backpressure cannot block mutation publication or follower cleanup. On a cache miss, the existing single-keyspace slow path may acquire `GCStateManager.mu.RLock()` while it reads storage; that lock is released before the iterator callback can block on `initCh`. This preserves the cache's current synchronization rule without holding one manager lock across the entire scan. - -Initial states are accumulated into batches of at most 1024 changes and sent through `initCh`. The loader closes `initCh` after a successful scan. If iteration fails, it records an initialization error as the watcher cancellation cause; a consumer may therefore have received a partial initial view before the stream terminates. - -Cancellation of the RPC or removal of the watcher cancels the loader as well. The loader checks the watcher context between iterator items and while waiting to send a batch. The current keyspace iterator and storage read interfaces do not accept a context, so the loader cannot observe cancellation until an invocation already inside `Iterator.Next` or a storage operation returns. This inherited non-cancellable I/O boundary is accepted; the watcher introduces no additional wait around it. - -## Live publication - -Live changes are published only after a safe-point mutation has committed successfully and the manager cache reflects the resulting complete effective GC state. Publication occurs before releasing `GCStateManager.mu`, preserving the same order for all watchers in that manager and leadership generation and serializing it with local follower transition and registration. - -Publication is attached exactly once to the successful post-cache-update paths in `advanceGCSafePointImpl` and `advanceTxnSafePointImpl`. This covers `AdvanceGCSafePoint`, `AdvanceTxnSafePoint`, `CompatibleUpdateGCSafePoint`, and the `gc_worker` branch of `CompatibleUpdateServiceGCSafePoint` without duplicating events at public API entry points. Rejected, no-op, or failed mutations do not publish. - -The current barrier setters and deleters do not alter the persisted effective safe points, so those operations produce no event. Barrier details remain excluded from the stream. If a present or future barrier operation changes an effective safe point, the operation publishes the resulting complete upsert rather than the barrier itself. - -Publication to each `liveCh` is non-blocking. If a watcher's channel is full, the manager removes and cancels only that watcher with the slow-consumer cause and continues publishing to other watchers. There is no timeout, retry, or channel wait while holding the manager mutex. - -## Initial/live ordering - -A single consumer merges `initCh` and `liveCh`. While initial loading is active, it maintains `dirtyDuringInit`, a set keyed by keyspace scope: - -- After receiving an initial batch, the consumer snapshots `len(liveCh)` and drains exactly that queued FIFO prefix before emitting any change from the batch. The remaining prefix count persists across `RecvBatch` calls. Live changes arriving after the snapshot do not extend the prefix, so they cannot indefinitely postpone unrelated states in the pending initial batch. -- When the consumer emits a live change, it marks that scope dirty. -- When it encounters an initial upsert whose scope is already dirty, it drops the initial upsert. -- After `initCh` is closed and all of its buffered batches, the pending batch, and its live prefix are consumed, the consumer releases the dirty set and disables the closed channel because every later change is live and already ordered by `liveCh`. - -There are two possible observations for an initial value `v1` and a later live value `v2`. If the consumer receives `v1` first, it emits `v1` followed by `v2`. If it receives `v2` first, it marks the scope dirty and suppresses `v1`. In neither case can it emit `v2` followed by `v1`. - -The initial scan can also read a newer value than an already queued live change. For example, live values `10` and `20` can be queued before the loader reads initial `20`. If the merge selects that initial batch first, draining the captured live prefix emits `10, 20` and suppresses the initial duplicate, preventing `20, 10, 20`. Mutation cache updates and publication are serialized by `GCStateManager.mu`: every strictly older live change has been published before the loader can observe the newer cache value. That newer value's own live publication may still follow its cache store, which is safe because it cannot regress the initial value. Taking the queue-length snapshot after acquiring the batch therefore covers every older live change that has not already been consumed, without adding locks or another queue. - -The same rule supports the future `removed` producer: a live removal marks the scope dirty, preventing an older initial upsert from recreating it. Live changes retain FIFO order because mutation publication is serialized by the manager mutex and each watcher has a single live channel and a single consumer. - -The implementation must explain these timing guarantees next to the merge logic, including the registration boundary and both directions of initial/live overlap. A deterministic test pauses initial loading after reading `v1` but before placing it on `initCh`, advances the same keyspace to `v2`, observes `v2`, resumes initial loading, and verifies that `v1` is never emitted afterward. Receiver tests also force initial-batch acquisition with older live changes queued, verify prefix ordering across response boundaries, and keep adding live changes to verify that the pending initial batch still makes progress. - -## Capacity and backpressure - -The default capacities balance burst tolerance against per-watcher memory consumption. Production wrappers pass the defaults to unexported constructors or helpers that accept explicit capacities, which lets tests exercise smaller limits without mutable package globals. - -`liveCh` holds 1024 individual changes. A 300,000-keyspace deployment in which every keyspace changes during a ten-minute interval produces roughly 500 changes per second; allowing for both transaction and GC safe-point changes gives an order-of-magnitude estimate of 1,000 changes per second. A capacity of 1024 therefore absorbs approximately one second of scheduling or network jitter at that scale without pretending to support an instantaneous 300,000-keyspace burst. Sustained delivery slower than production intentionally terminates and reconnects the affected watcher. - -Initial batches contain at most 1024 changes, and `initCh` has capacity 1. This permits the RPC consumer to process the current batch while one completed batch waits and the loader constructs the next batch. Increasing the channel capacity would mainly increase per-watcher read-ahead and memory use because initial loading is allowed to backpressure storage iteration. - -The response wire-size limit and the internal change-count limit solve different problems. `RecvBatch(1024)` bounds merge work and internal allocations; the gRPC adapter may split that result into multiple responses to enforce the protobuf size limit. - -## Lifecycle and errors - -Every watcher has exactly one terminal cause, and the first successful cancellation wins. Removal from the manager registry and cancellation are idempotent so concurrent RPC cleanup, initialization failure, leadership loss, and slow-consumer detection cannot leak or double-close resources. Cancellation invoked while holding the manager mutex does not synchronously reacquire that mutex. - -The lifecycle cases are: - -| Event | Manager behavior | Stream result | -| --- | --- | --- | -| Caller cancellation or send failure | Remove the watcher and cancel its loader | Return the caller or send error; record `client_cancel` | -| A local leadership generation ends or is superseded | Remove and cancel all current watchers while holding the manager mutex | Return the domain not-leader error, mapped to gRPC `Unavailable` | -| Initial scan fails | Remove and cancel the watcher with the initialization cause | End after any already-sent partial initial data; map to gRPC `Unavailable` | -| `liveCh` is full | Remove and cancel only that watcher | Return the slow-consumer error, mapped to gRPC `ResourceExhausted` | -| Initial scan completes | Close `initCh` and release merge-only initial state | Continue streaming live changes | - -The `pkg/gc` layer returns domain errors and does not depend on gRPC status codes. The service maps not-leader and storage/initialization failures to `Unavailable`, and a slow consumer to `ResourceExhausted`. Existing request-validation and rate-limit paths retain their current status semantics. - -The receive path checks the terminal cause before returning buffered work after cancellation. Because one `RecvBatch` can be split into multiple protobuf responses, the worker checks the same terminal cause again immediately before every `Send`. This prevents an already-cancelled watcher from deliberately draining stale queued data. - -A response send already in progress cannot be recalled. Watcher termination nevertheless causes the RPC handler to return without waiting for that send. Handler return initiates gRPC transport teardown, which interrupts blocked sending; the worker then exits. The handler retains ownership of watcher cleanup and rate-limit token release. A client that is not reading may observe the terminal status only after draining already queued responses. - -The handler must not join the worker before returning: transport teardown depends on that return to unblock `Send`. The worker retains its response until sending finishes and publishes its result without waiting for a reader. There is at most one send in progress, with no additional response queue or per-send goroutine. Clients treat any terminated stream as requiring reconnection and reinitialization. - -## Protobuf conversion and response batching - -The gRPC layer uses a dedicated converter from the internal change type to `pdpb.GCStateChange`. An upsert populates the complete effective state and never populates barrier fields. A removed change populates its `KeyspaceScope`. Unsupported or structurally invalid internal variants fail explicitly rather than producing an empty protobuf change. - -Each `WatchGCStatesResponse` is limited to 1 MiB under normal operation. Batching accounts for the actual protobuf wire representation, including the repeated-field tag and the length-delimited message envelope. - -The adapter starts with the serialized size of a response containing `grpcutil.WrapHeader()` and no changes. For each change, it computes the serialized size of a headerless response containing only that one change; this is the exact additive delta for the repeated embedded-message field. If adding the delta would exceed the limit and the current response is non-empty, the adapter sends the current response first. It never sends an empty response. - -If one change by itself exceeds 1 MiB, the adapter logs the anomaly and sends that single change. Dropping it would silently corrupt the consumer's materialized view, while repeatedly rejecting it would make progress impossible. The protobuf schema makes this case unexpected, but the behavior remains defined. - -This approach avoids manually reproducing protobuf varint rules and avoids repeatedly serializing a growing candidate response, which would make batch construction quadratic. - -## Leadership behavior - -The cluster lifecycle continues to notify `GCStateManager` synchronously. Under the manager mutex, `OnNodeBecomesLeader` advances the manager's local generation, cancels existing watchers, clears the cache, and returns a teardown closure that captures the new generation. The teardown acts only if its generation is still current; it then marks the manager as a follower, cancels all current watchers, and clears the cache. A delayed teardown from an older generation is therefore a no-op. Registration fails when the manager has no active local leadership generation. - -This mechanism does not fence GC-state write transactions across PD processes. In the rare case that a transaction accepted by an old leader commits after the new leader has read that keyspace's initial state, the new stream might miss the value until another effective safe-point change or reconnection. The per-stream non-regression guarantee still holds. Leadership-fenced GC-state transactions remain an out-of-scope follow-up. - -## Removed-event integration - -The internal model, ordering logic, protobuf converter, and tests all accept `removed` changes, but the first PD implementation has no authoritative keyspace lifecycle hook that produces them. It also does not produce metadata-driven upserts for keyspace creation, enablement, or GC-mode changes. Adding only part of these producers would create misleading convergence guarantees, so production is deferred. - -A future lifecycle integration must publish metadata-driven upserts and removals through the same manager-serialized live path as safe-point changes. The implementation records this integration point with a targeted TODO. - -## Observability - -Metrics are intentionally low-cardinality and focus on operational decisions rather than per-message detail: - -- A gauge reports the number of active GC-state watchers. -- A counter reports watcher terminations with the bounded reason label values `client_cancel`, `leader_lost`, `slow_consumer`, and `init_error`. -- A slow-consumer log records the watcher identifier, configured live capacity, and observed queue length. - -Watcher identifiers, keyspace identifiers, client addresses, and error strings are not metric labels. The four termination values describe watcher lifecycle causes; transport conversion errors are logged separately. The implementation does not add a per-send queue-length histogram because it would instrument the hot path without a demonstrated operational need. - -## Test strategy - -The watcher and service layers are tested separately, with focused integration coverage for leadership transitions. Tests use controllable capacities or failpoints instead of timing-dependent sleeps. - -### `pkg/gc` tests - -The domain tests cover: - -- Registration succeeds only in an active local leadership generation; ending or superseding that generation terminates its watchers, while a delayed older teardown does not affect a newer generation. -- `skip_loading_initial=false` returns initial and subsequent live states; `true` returns only post-registration live changes. -- Effective safe-point changes publish complete upserts exactly once from the shared modern and legacy mutation paths; no-op and failed mutations do not publish. -- Current barrier-only mutations produce no change, while any operation that changes an effective safe point does. -- Initial-first delivery emits `v1` followed by `v2`; a deterministic paused-initial test emits `v2` and suppresses the later initial `v1`. -- Acquiring a newer initial batch drains the already queued live prefix first, retains prefix progress across `RecvBatch` calls, and does not let later live arrivals postpone the pending initial batch. -- Filling watcher A's `liveCh` terminates A without delaying watcher B; A can reconnect and rebuild. -- Initial iteration failure, caller cancellation, and concurrent deregistration terminate without goroutine or registry leaks. -- A full `initCh` backpressures only that watcher's initial iterator and never waits on the channel while holding `GCStateManager.mu`. -- Upsert and removed changes use the same per-scope dirty ordering rule. -- Initial merge state remains active until a closed `initCh` is fully drained, and disabling the closed channel prevents select spinning. - -### Server tests - -The transport tests cover: - -- Conversion covers complete upserts with empty barriers, removals, and invalid internal changes. -- Response batching covers exact protobuf size boundaries, successful headers, oversized single changes, and suppression of empty responses. -- Cancellation between two responses derived from one `RecvBatch` prevents the remaining response from being sent. -- A rate-limit capacity of one: the first active stream holds the token, the second is rejected, and a third succeeds after the first closes. -- Request preflight covers cluster-ID mismatch and unavailable members, and domain errors map to the specified gRPC status codes. -- Leader transfer, client cancellation, and send failure terminate the stream, clean up the watcher, and release the rate-limit token. -- Watcher metrics change exactly once across registration and cleanup. - -## Dependency and rollout - -The root, `client`, `tools`, and `tests/integrations` Go modules are updated to a kvproto revision containing the merged `WatchGCStates` API from PR #1528. No compatibility wrapper or implementation is added for `WatchGCSafePointV2`. - -The API can be rolled out server-first because existing clients do not call the new RPC. New consumers use an initial stream to construct their materialized view and use the same path after any disconnect. - -## Acceptance criteria - -The implementation is complete when all of the following are true: - -- `WatchGCStates` serves initial states and local effective safe-point changes with the documented `skip_loading_initial` behavior. -- Deterministic ordering tests prove that neither an older initial value follows a newer live value nor an older queued live value follows a newer initial value for the same scope on one stream. -- A full live queue terminates only the affected watcher without blocking GC-state mutation. -- Ending or superseding a local leadership generation terminates its active streams. -- Response batches observe the 1 MiB target using exact protobuf size accounting, except for the defined oversized-single-change case. -- Metric labels use only the bounded dimensions described above. -- Targeted package and server tests pass with no failpoints left enabled. -- The implementation contains no `WatchGCSafePointV2` compatibility path and no Go client work. - -## Next steps - -After this design is reviewed, the implementation work is decomposed into a test-first plan covering the dependency update, watcher domain model, mutation publication, gRPC adaptation, observability, and focused verification. From faa4d8f3f84c21fd5281eb54f416931f34b30694 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Sun, 20 Sep 2026 16:40:44 +0800 Subject: [PATCH 16/25] tests: reuse WatchGCStates concurrency limit helper Keep rate limiter setup and cleanup consistent across watch tests. Signed-off-by: Wenxuan Zhang --- tests/server/gc/gc_test.go | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/tests/server/gc/gc_test.go b/tests/server/gc/gc_test.go index a0c5293e849..e324fc20b6a 100644 --- a/tests/server/gc/gc_test.go +++ b/tests/server/gc/gc_test.go @@ -1213,18 +1213,7 @@ func TestWatchGCStatesHoldsRateLimitTokenForStreamLifetime(t *testing.T) { cluster := newWatchGCStatesCluster(t, 1, true) leaderServer := cluster.GetLeaderServer() re.NotNil(leaderServer) - server := leaderServer.GetServer() - options := server.GetServiceMiddlewarePersistOptions() - previousConfig := options.GetGRPCRateLimitConfig().Clone() - enabledConfig := previousConfig.Clone() - enabledConfig.EnableRateLimit = true - options.SetGRPCRateLimitConfig(enabledConfig) - limiter := server.GetGRPCRateLimiter() - limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(1)) - t.Cleanup(func() { - limiter.Update("WatchGCStates", ratelimit.UpdateConcurrencyLimiter(0)) - options.SetGRPCRateLimitConfig(previousConfig) - }) + limiter := limitWatchGCStatesConcurrency(t, leaderServer.GetServer()) client := newWatchGCStatesClient(t, leaderServer.GetAddr()) header := testutil.NewRequestHeader(leaderServer.GetClusterID()) From 9b98acc69ff2a8eec73f85d92d5cec401be373db Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 23 Sep 2026 11:22:31 +0800 Subject: [PATCH 17/25] gc: cache enabled keyspaces at applied etcd revisions Keep a leader-local, complete ENABLED keyspace index for GC watcher initialization. Publish metadata changes only after watch progress confirms their revision, so readers never observe a partially applied list. Signed-off-by: Wenxuan Zhang --- pkg/gc/enabled_keyspace_cache.go | 339 ++++++++++++++++++++++++++ pkg/gc/enabled_keyspace_cache_test.go | 265 ++++++++++++++++++++ 2 files changed, 604 insertions(+) create mode 100644 pkg/gc/enabled_keyspace_cache.go create mode 100644 pkg/gc/enabled_keyspace_cache_test.go diff --git a/pkg/gc/enabled_keyspace_cache.go b/pkg/gc/enabled_keyspace_cache.go new file mode 100644 index 00000000000..8644499e9ca --- /dev/null +++ b/pkg/gc/enabled_keyspace_cache.go @@ -0,0 +1,339 @@ +// Copyright 2026 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package gc + +import ( + "context" + "fmt" + "slices" + "strconv" + "strings" + "sync" + "time" + + "github.com/gogo/protobuf/proto" + "go.etcd.io/etcd/api/v3/mvccpb" + clientv3 "go.etcd.io/etcd/client/v3" + + "github.com/pingcap/kvproto/pkg/keyspacepb" + + "github.com/tikv/pd/pkg/keyspace" + "github.com/tikv/pd/pkg/utils/etcdutil" +) + +const ( + enabledKeyspacePageSize = 256 + enabledKeyspaceRequestTimeout = 5 * time.Second + enabledKeyspaceRetryDelay = time.Second + enabledKeyspaceWatchTimeout = 10 * time.Second +) + +// enabledKeyspace is a value copy of the metadata needed by GC initialization. +type enabledKeyspace struct { + id uint32 + gcManagementType string +} + +// enabledKeyspaceCache belongs to one leadership term. The published map and +// its revision always describe one complete, successfully applied snapshot. +type enabledKeyspaceCache struct { + termCtx context.Context + client *clientv3.Client + prefix string + + mu sync.Mutex + entries map[uint32]enabledKeyspace + revision int64 + ready bool + changed chan struct{} +} + +func newEnabledKeyspaceCache(termCtx context.Context, client *clientv3.Client, prefix string) *enabledKeyspaceCache { + return &enabledKeyspaceCache{ + termCtx: termCtx, + client: client, + prefix: prefix, + changed: make(chan struct{}), + } +} + +// run blocks until the leadership term ends. A failed or compacted watch is +// followed by a complete reload, so no missing revision is silently skipped. +func (c *enabledKeyspaceCache) run() { + for c.termCtx.Err() == nil { + entries, revision, err := c.load() + if err == nil { + c.publish(entries, revision) + _ = c.watch(revision + 1) + } + if c.termCtx.Err() != nil { + return + } + select { + case <-c.termCtx.Done(): + return + case <-time.After(enabledKeyspaceRetryDelay): + } + } +} + +func (c *enabledKeyspaceCache) waitReady(ctx context.Context) error { + for { + c.mu.Lock() + if err := ctx.Err(); err != nil { + c.mu.Unlock() + return err + } + if c.ready { + c.mu.Unlock() + return c.termCtx.Err() + } + changed := c.changed + c.mu.Unlock() + select { + case <-ctx.Done(): + return ctx.Err() + case <-c.termCtx.Done(): + return c.termCtx.Err() + case <-changed: + } + } +} + +func (c *enabledKeyspaceCache) snapshotAtLeast(ctx context.Context, revision int64) ([]enabledKeyspace, int64, error) { + for { + c.mu.Lock() + if err := c.termCtx.Err(); err != nil { + c.mu.Unlock() + return nil, 0, err + } + if err := ctx.Err(); err != nil { + c.mu.Unlock() + return nil, 0, err + } + if c.ready && c.revision >= revision { + result := make([]enabledKeyspace, 0, len(c.entries)) + for _, entry := range c.entries { + result = append(result, entry) + } + applied := c.revision + c.mu.Unlock() + slices.SortFunc(result, func(a, b enabledKeyspace) int { + switch { + case a.id < b.id: + return -1 + case a.id > b.id: + return 1 + default: + return 0 + } + }) + return result, applied, nil + } + changed := c.changed + c.mu.Unlock() + select { + case <-ctx.Done(): + return nil, 0, ctx.Err() + case <-c.termCtx.Done(): + return nil, 0, c.termCtx.Err() + case <-changed: + } + } +} + +func (c *enabledKeyspaceCache) publish(entries map[uint32]enabledKeyspace, revision int64) { + c.mu.Lock() + defer c.mu.Unlock() + if c.termCtx.Err() != nil { + return + } + c.entries = entries + c.revision = revision + c.ready = true + close(c.changed) + c.changed = make(chan struct{}) +} + +func (c *enabledKeyspaceCache) publishProgress(changes map[uint32]*enabledKeyspace, revision int64) { + c.mu.Lock() + defer c.mu.Unlock() + if c.termCtx.Err() != nil || revision <= c.revision { + return + } + for id, entry := range changes { + if entry == nil { + delete(c.entries, id) + } else { + c.entries[id] = *entry + } + } + c.revision = revision + close(c.changed) + c.changed = make(chan struct{}) +} + +// load reads each page at the first page's revision and returns only after the +// entire prefix has been decoded. No partial page is ever published. +func (c *enabledKeyspaceCache) load() (map[uint32]enabledKeyspace, int64, error) { + entries := make(map[uint32]enabledKeyspace) + start := c.prefix + end := clientv3.GetPrefixRangeEnd(c.prefix) + var revision int64 + for { + ctx, cancel := context.WithTimeout(c.termCtx, enabledKeyspaceRequestTimeout) + opts := []clientv3.OpOption{clientv3.WithRange(end), clientv3.WithLimit(enabledKeyspacePageSize)} + if revision != 0 { + opts = append(opts, clientv3.WithRev(revision)) + } + resp, err := c.client.Get(ctx, start, opts...) + cancel() + if err != nil { + return nil, 0, err + } + if revision == 0 { + revision = resp.Header.Revision + } + for _, kv := range resp.Kvs { + id, entry, enabled, err := c.decode(kv.Key, kv.Value) + if err != nil { + return nil, 0, err + } + if enabled { + entries[id] = entry + } + } + if !resp.More { + return entries, revision, nil + } + if len(resp.Kvs) == 0 { + return nil, 0, fmt.Errorf("empty keyspace metadata page at revision %d", revision) + } + start = string(resp.Kvs[len(resp.Kvs)-1].Key) + "\x00" + } +} + +func (c *enabledKeyspaceCache) decode(rawKey, rawValue []byte) (uint32, enabledKeyspace, bool, error) { + key := string(rawKey) + if !strings.HasPrefix(key, c.prefix) { + return 0, enabledKeyspace{}, false, fmt.Errorf("keyspace metadata key %q is outside prefix", key) + } + id64, err := strconv.ParseUint(strings.TrimPrefix(key, c.prefix), 10, 32) + if err != nil { + return 0, enabledKeyspace{}, false, fmt.Errorf("invalid keyspace metadata key %q: %w", key, err) + } + id := uint32(id64) + meta := &keyspacepb.KeyspaceMeta{} + if err := proto.Unmarshal(rawValue, meta); err != nil { + return 0, enabledKeyspace{}, false, fmt.Errorf("decode keyspace metadata %q: %w", key, err) + } + if meta.GetId() != id { + return 0, enabledKeyspace{}, false, fmt.Errorf("keyspace metadata %q contains ID %d", key, meta.GetId()) + } + return id, enabledKeyspace{id: id, gcManagementType: meta.Config[keyspace.GCManagementType]}, meta.State == keyspacepb.KeyspaceState_ENABLED, nil +} + +// watch updates a private working set. Progress notifications are ordered after +// prior events on the same stream, so they prove a complete applied revision. +// Event response headers alone may be ahead of the delivered prefix events. +func (c *enabledKeyspaceCache) watch(nextRevision int64) error { + watcher := clientv3.NewWatcher(c.client) + defer watcher.Close() + watchCtx, cancel := context.WithCancel(clientv3.WithRequireLeader(c.termCtx)) + defer cancel() + watchCh := watcher.Watch(watchCtx, c.prefix, clientv3.WithPrefix(), clientv3.WithRev(nextRevision), clientv3.WithProgressNotify()) + ticker := time.NewTicker(etcdutil.RequestProgressInterval) + defer ticker.Stop() + pending := make(map[uint32]*enabledKeyspace) + publishedRevision := nextRevision - 1 + pendingRevision := publishedRevision + lastProgress := time.Now() + for { + select { + case <-c.termCtx.Done(): + return c.termCtx.Err() + case <-ticker.C: + if time.Since(lastProgress) >= enabledKeyspaceWatchTimeout { + return fmt.Errorf("keyspace metadata watch made no progress for %s", enabledKeyspaceWatchTimeout) + } + ctx, cancel := context.WithTimeout(watchCtx, enabledKeyspaceRequestTimeout) + err := watcher.RequestProgress(ctx) + cancel() + if err != nil { + return err + } + case resp, ok := <-watchCh: + if !ok { + return fmt.Errorf("keyspace metadata watch closed") + } + if err := resp.Err(); err != nil { + return err + } + if resp.IsProgressNotify() { + if resp.Header.Revision < pendingRevision || resp.Header.Revision < publishedRevision || + (resp.Header.Revision == publishedRevision && len(pending) != 0) { + return fmt.Errorf("keyspace metadata watch progress %d precedes applied events at %d", resp.Header.Revision, pendingRevision) + } + c.publishProgress(pending, resp.Header.Revision) + publishedRevision = resp.Header.Revision + pendingRevision = publishedRevision + lastProgress = time.Now() + clear(pending) + continue + } + if len(resp.Events) == 0 { + continue + } + // Decode the complete clientv3 response before changing the working + // set. clientv3 merges fragmented watch responses before delivery. + type change struct { + id uint32 + entry enabledKeyspace + enabled bool + } + changes := make([]change, 0, len(resp.Events)) + for _, event := range resp.Events { + if event.Kv == nil || event.Kv.ModRevision <= publishedRevision { + return fmt.Errorf("keyspace metadata watch received stale or missing event at revision %d", publishedRevision) + } + pendingRevision = max(pendingRevision, event.Kv.ModRevision) + switch event.Type { + case mvccpb.PUT: + id, entry, enabled, err := c.decode(event.Kv.Key, event.Kv.Value) + if err != nil { + return err + } + changes = append(changes, change{id: id, entry: entry, enabled: enabled}) + case mvccpb.DELETE: + id64, err := strconv.ParseUint(strings.TrimPrefix(string(event.Kv.Key), c.prefix), 10, 32) + if err != nil { + return fmt.Errorf("invalid deleted keyspace metadata key %q: %w", event.Kv.Key, err) + } + changes = append(changes, change{id: uint32(id64)}) + default: + return fmt.Errorf("unexpected keyspace metadata event type %v", event.Type) + } + } + for _, change := range changes { + if change.enabled { + entry := change.entry + pending[change.id] = &entry + } else { + pending[change.id] = nil + } + } + } + } +} diff --git a/pkg/gc/enabled_keyspace_cache_test.go b/pkg/gc/enabled_keyspace_cache_test.go new file mode 100644 index 00000000000..f6d5a3a1166 --- /dev/null +++ b/pkg/gc/enabled_keyspace_cache_test.go @@ -0,0 +1,265 @@ +// Copyright 2026 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package gc + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + "github.com/gogo/protobuf/proto" + "github.com/stretchr/testify/require" + clientv3 "go.etcd.io/etcd/client/v3" + + "github.com/pingcap/kvproto/pkg/keyspacepb" + + "github.com/tikv/pd/pkg/keyspace" + "github.com/tikv/pd/pkg/utils/etcdutil" +) + +const enabledKeyspaceTestPrefix = "/test/enabled-keyspaces/" + +func putEnabledKeyspaceTestMeta(t *testing.T, client *clientv3.Client, id uint32, state keyspacepb.KeyspaceState, gcType string) int64 { + t.Helper() + value, err := proto.Marshal(&keyspacepb.KeyspaceMeta{ + Keyspace: &keyspacepb.KeyspaceMeta_Id{Id: id}, + State: state, + Config: map[string]string{keyspace.GCManagementType: gcType}, + }) + require.NoError(t, err) + resp, err := client.Put(context.Background(), fmt.Sprintf("%s%08d", enabledKeyspaceTestPrefix, id), string(value)) + require.NoError(t, err) + return resp.Header.Revision +} + +func startEnabledKeyspaceTestCache(t *testing.T, client *clientv3.Client) (*enabledKeyspaceCache, <-chan struct{}) { + t.Helper() + termCtx, cancel := context.WithCancel(context.Background()) + cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) + done := make(chan struct{}) + go func() { + cache.run() + close(done) + }() + t.Cleanup(func() { + cancel() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Error("cache did not stop after term cancellation") + } + }) + return cache, done +} + +func TestEnabledKeyspaceCacheEmptySnapshotAndProgress(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + cache, _ := startEnabledKeyspaceTestCache(t, client) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + require.NoError(t, cache.waitReady(ctx)) + initial, revision, err := cache.snapshotAtLeast(ctx, 0) + require.NoError(t, err) + require.Empty(t, initial) + require.Positive(t, revision) + + resp, err := client.Put(ctx, "/test/unrelated", "changed") + require.NoError(t, err) + list, applied, err := cache.snapshotAtLeast(ctx, resp.Header.Revision) + require.NoError(t, err) + require.Empty(t, list) + require.GreaterOrEqual(t, applied, resp.Header.Revision) +} + +func TestEnabledKeyspaceCacheAppliesMetadataChanges(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + putEnabledKeyspaceTestMeta(t, client, 3, keyspacepb.KeyspaceState_ENABLED, keyspace.KeyspaceLevelGC) + putEnabledKeyspaceTestMeta(t, client, 2, keyspacepb.KeyspaceState_DISABLED, keyspace.UnifiedGC) + putEnabledKeyspaceTestMeta(t, client, 1, keyspacepb.KeyspaceState_ENABLED, keyspace.UnifiedGC) + cache, _ := startEnabledKeyspaceTestCache(t, client) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + require.NoError(t, cache.waitReady(ctx)) + list, _, err := cache.snapshotAtLeast(ctx, 0) + require.NoError(t, err) + require.Equal(t, []enabledKeyspace{ + {id: 1, gcManagementType: keyspace.UnifiedGC}, + {id: 3, gcManagementType: keyspace.KeyspaceLevelGC}, + }, list) + + rev := putEnabledKeyspaceTestMeta(t, client, 2, keyspacepb.KeyspaceState_ENABLED, keyspace.KeyspaceLevelGC) + list, _, err = cache.snapshotAtLeast(ctx, rev) + require.NoError(t, err) + require.Equal(t, []enabledKeyspace{ + {id: 1, gcManagementType: keyspace.UnifiedGC}, + {id: 2, gcManagementType: keyspace.KeyspaceLevelGC}, + {id: 3, gcManagementType: keyspace.KeyspaceLevelGC}, + }, list) + + rev = putEnabledKeyspaceTestMeta(t, client, 1, keyspacepb.KeyspaceState_DISABLED, keyspace.UnifiedGC) + list, _, err = cache.snapshotAtLeast(ctx, rev) + require.NoError(t, err) + require.Equal(t, []enabledKeyspace{ + {id: 2, gcManagementType: keyspace.KeyspaceLevelGC}, + {id: 3, gcManagementType: keyspace.KeyspaceLevelGC}, + }, list) + + resp, err := client.Delete(ctx, fmt.Sprintf("%s%08d", enabledKeyspaceTestPrefix, 3)) + require.NoError(t, err) + list, _, err = cache.snapshotAtLeast(ctx, resp.Header.Revision) + require.NoError(t, err) + require.Equal(t, []enabledKeyspace{{id: 2, gcManagementType: keyspace.KeyspaceLevelGC}}, list) +} + +func TestEnabledKeyspaceCacheLoadsAllPagesAtOneRevision(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + ops := make([]clientv3.Op, 0, enabledKeyspacePageSize+1) + var revision int64 + for id := uint32(1); id <= enabledKeyspacePageSize+1; id++ { + value, err := proto.Marshal(&keyspacepb.KeyspaceMeta{ + Keyspace: &keyspacepb.KeyspaceMeta_Id{Id: id}, + State: keyspacepb.KeyspaceState_ENABLED, + Config: map[string]string{keyspace.GCManagementType: keyspace.KeyspaceLevelGC}, + }) + require.NoError(t, err) + ops = append(ops, clientv3.OpPut(fmt.Sprintf("%s%08d", enabledKeyspaceTestPrefix, id), string(value))) + if len(ops) == 100 { + resp, err := client.Txn(ctx).Then(ops...).Commit() + require.NoError(t, err) + revision = resp.Header.Revision + ops = ops[:0] + } + } + resp, err := client.Txn(ctx).Then(ops...).Commit() + require.NoError(t, err) + revision = resp.Header.Revision + termCtx, stop := context.WithCancel(context.Background()) + defer stop() + cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) + entries, loadedRevision, err := cache.load() + require.NoError(t, err) + require.Equal(t, revision, loadedRevision) + require.Len(t, entries, enabledKeyspacePageSize+1) + require.Equal(t, enabledKeyspace{id: enabledKeyspacePageSize + 1, gcManagementType: keyspace.KeyspaceLevelGC}, entries[enabledKeyspacePageSize+1]) +} + +func TestEnabledKeyspaceCacheRejectsMalformedMetadataUntilReload(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + initial := putEnabledKeyspaceTestMeta(t, client, 1, keyspacepb.KeyspaceState_ENABLED, keyspace.KeyspaceLevelGC) + cache, _ := startEnabledKeyspaceTestCache(t, client) + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + require.NoError(t, cache.waitReady(ctx)) + + resp, err := client.Put(ctx, fmt.Sprintf("%s%08d", enabledKeyspaceTestPrefix, 2), "malformed protobuf") + require.NoError(t, err) + shortCtx, stop := context.WithTimeout(ctx, 300*time.Millisecond) + defer stop() + _, _, err = cache.snapshotAtLeast(shortCtx, resp.Header.Revision) + require.ErrorIs(t, err, context.DeadlineExceeded) + list, revision, err := cache.snapshotAtLeast(ctx, initial) + require.NoError(t, err) + require.Equal(t, initial, revision) + require.Equal(t, []enabledKeyspace{{id: 1, gcManagementType: keyspace.KeyspaceLevelGC}}, list) + + fixed := putEnabledKeyspaceTestMeta(t, client, 2, keyspacepb.KeyspaceState_ENABLED, keyspace.UnifiedGC) + list, revision, err = cache.snapshotAtLeast(ctx, fixed) + require.NoError(t, err) + require.GreaterOrEqual(t, revision, fixed) + require.Equal(t, []enabledKeyspace{ + {id: 1, gcManagementType: keyspace.KeyspaceLevelGC}, + {id: 2, gcManagementType: keyspace.UnifiedGC}, + }, list) +} + +func TestEnabledKeyspaceCacheReloadsAfterCompactedWatch(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + first := putEnabledKeyspaceTestMeta(t, client, 1, keyspacepb.KeyspaceState_ENABLED, keyspace.KeyspaceLevelGC) + termCtx, cancel := context.WithCancel(context.Background()) + defer cancel() + cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) + entries, revision, err := cache.load() + require.NoError(t, err) + require.Equal(t, first, revision) + cache.publish(entries, revision) + + ctx, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + _, err = client.Delete(ctx, fmt.Sprintf("%s%08d", enabledKeyspaceTestPrefix, 1)) + require.NoError(t, err) + latest := putEnabledKeyspaceTestMeta(t, client, 2, keyspacepb.KeyspaceState_ENABLED, keyspace.UnifiedGC) + _, err = client.Compact(ctx, latest, clientv3.WithCompactPhysical()) + require.NoError(t, err) + // The watch cannot replay the missing revisions. Its caller must reload + // the complete prefix before publishing a newer waterline. + require.Error(t, cache.watch(revision+1)) + list, applied, err := cache.snapshotAtLeast(ctx, revision) + require.NoError(t, err) + require.Equal(t, revision, applied) + require.Equal(t, []enabledKeyspace{{id: 1, gcManagementType: keyspace.KeyspaceLevelGC}}, list) + + entries, revision, err = cache.load() + require.NoError(t, err) + cache.publish(entries, revision) + list, applied, err = cache.snapshotAtLeast(ctx, latest) + require.NoError(t, err) + require.GreaterOrEqual(t, applied, latest) + require.Equal(t, []enabledKeyspace{{id: 2, gcManagementType: keyspace.UnifiedGC}}, list) +} + +func TestEnabledKeyspaceCacheTermCancellationUnblocksWaiters(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + termCtx, cancel := context.WithCancel(context.Background()) + cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) + done := make(chan struct{}) + go func() { + cache.run() + close(done) + }() + ctx, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + require.NoError(t, cache.waitReady(ctx)) + _, revision, err := cache.snapshotAtLeast(ctx, 0) + require.NoError(t, err) + waitResult := make(chan error, 1) + go func() { + _, _, err := cache.snapshotAtLeast(ctx, revision+100) + waitResult <- err + }() + cancel() + select { + case err := <-waitResult: + require.True(t, errors.Is(err, context.Canceled), "waiting snapshot error: %v", err) + case <-ctx.Done(): + t.Fatal("snapshot did not stop after term cancellation") + } + select { + case <-done: + case <-ctx.Done(): + t.Fatal("cache run did not stop after term cancellation") + } +} From 96bbf4987b92a0f17d802595fd52630deae3c1f9 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 23 Sep 2026 11:31:48 +0800 Subject: [PATCH 18/25] gc: make keyspace cache recovery observable and bounded Log synchronization failures with revision context and back off repeated reloads. Verify fixed-revision pagination and automatic recovery after a compacted watch using controlled embedded etcd tests. Signed-off-by: Wenxuan Zhang --- pkg/gc/enabled_keyspace_cache.go | 57 ++++++++-- pkg/gc/enabled_keyspace_cache_test.go | 143 ++++++++++++++++++++++---- 2 files changed, 168 insertions(+), 32 deletions(-) diff --git a/pkg/gc/enabled_keyspace_cache.go b/pkg/gc/enabled_keyspace_cache.go index 8644499e9ca..4b8e5fd5546 100644 --- a/pkg/gc/enabled_keyspace_cache.go +++ b/pkg/gc/enabled_keyspace_cache.go @@ -26,8 +26,10 @@ import ( "github.com/gogo/protobuf/proto" "go.etcd.io/etcd/api/v3/mvccpb" clientv3 "go.etcd.io/etcd/client/v3" + "go.uber.org/zap" "github.com/pingcap/kvproto/pkg/keyspacepb" + "github.com/pingcap/log" "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/utils/etcdutil" @@ -37,6 +39,8 @@ const ( enabledKeyspacePageSize = 256 enabledKeyspaceRequestTimeout = 5 * time.Second enabledKeyspaceRetryDelay = time.Second + enabledKeyspaceMaxRetryDelay = 30 * time.Second + enabledKeyspaceLogInterval = 30 * time.Second enabledKeyspaceWatchTimeout = 10 * time.Second ) @@ -49,9 +53,10 @@ type enabledKeyspace struct { // enabledKeyspaceCache belongs to one leadership term. The published map and // its revision always describe one complete, successfully applied snapshot. type enabledKeyspaceCache struct { - termCtx context.Context - client *clientv3.Client - prefix string + termCtx context.Context + client *clientv3.Client + prefix string + watcherFactory func(*clientv3.Client) clientv3.Watcher mu sync.Mutex entries map[uint32]enabledKeyspace @@ -62,33 +67,65 @@ type enabledKeyspaceCache struct { func newEnabledKeyspaceCache(termCtx context.Context, client *clientv3.Client, prefix string) *enabledKeyspaceCache { return &enabledKeyspaceCache{ - termCtx: termCtx, - client: client, - prefix: prefix, - changed: make(chan struct{}), + termCtx: termCtx, + client: client, + prefix: prefix, + watcherFactory: clientv3.NewWatcher, + changed: make(chan struct{}), } } // run blocks until the leadership term ends. A failed or compacted watch is // followed by a complete reload, so no missing revision is silently skipped. func (c *enabledKeyspaceCache) run() { + retryDelay := enabledKeyspaceRetryDelay + var lastLog time.Time + suppressedErrors := 0 for c.termCtx.Err() == nil { entries, revision, err := c.load() + phase := "load" if err == nil { c.publish(entries, revision) - _ = c.watch(revision + 1) + watchStarted := time.Now() + err = c.watch(revision + 1) + phase = "watch" + if time.Since(watchStarted) >= enabledKeyspaceMaxRetryDelay { + retryDelay = enabledKeyspaceRetryDelay + } } if c.termCtx.Err() != nil { return } + if time.Since(lastLog) >= enabledKeyspaceLogInterval { + log.Warn("failed to synchronize enabled keyspace cache", + zap.String("prefix", c.prefix), + zap.String("phase", phase), + zap.Int64("revision", c.appliedRevision()), + zap.Duration("retry-delay", retryDelay), + zap.Int("suppressed-errors", suppressedErrors), + zap.Error(err)) + lastLog = time.Now() + suppressedErrors = 0 + } else { + suppressedErrors++ + } + timer := time.NewTimer(retryDelay) select { case <-c.termCtx.Done(): + timer.Stop() return - case <-time.After(enabledKeyspaceRetryDelay): + case <-timer.C: } + retryDelay = min(retryDelay*2, enabledKeyspaceMaxRetryDelay) } } +func (c *enabledKeyspaceCache) appliedRevision() int64 { + c.mu.Lock() + defer c.mu.Unlock() + return c.revision +} + func (c *enabledKeyspaceCache) waitReady(ctx context.Context) error { for { c.mu.Lock() @@ -249,7 +286,7 @@ func (c *enabledKeyspaceCache) decode(rawKey, rawValue []byte) (uint32, enabledK // prior events on the same stream, so they prove a complete applied revision. // Event response headers alone may be ahead of the delivered prefix events. func (c *enabledKeyspaceCache) watch(nextRevision int64) error { - watcher := clientv3.NewWatcher(c.client) + watcher := c.watcherFactory(c.client) defer watcher.Close() watchCtx, cancel := context.WithCancel(clientv3.WithRequireLeader(c.termCtx)) defer cancel() diff --git a/pkg/gc/enabled_keyspace_cache_test.go b/pkg/gc/enabled_keyspace_cache_test.go index f6d5a3a1166..4dc4c4330fd 100644 --- a/pkg/gc/enabled_keyspace_cache_test.go +++ b/pkg/gc/enabled_keyspace_cache_test.go @@ -18,6 +18,7 @@ import ( "context" "errors" "fmt" + "sync" "testing" "time" @@ -33,6 +34,42 @@ import ( const enabledKeyspaceTestPrefix = "/test/enabled-keyspaces/" +type pauseAfterFirstPageKV struct { + clientv3.KV + firstPage chan<- int64 + release <-chan struct{} + once sync.Once +} + +func (kv *pauseAfterFirstPageKV) Get(ctx context.Context, key string, opts ...clientv3.OpOption) (*clientv3.GetResponse, error) { + resp, err := kv.KV.Get(ctx, key, opts...) + if err == nil { + kv.once.Do(func() { + kv.firstPage <- resp.Header.Revision + select { + case <-kv.release: + case <-ctx.Done(): + } + }) + } + return resp, err +} + +type pauseBeforeWatch struct { + clientv3.Watcher + started chan<- struct{} + release <-chan struct{} +} + +func (w *pauseBeforeWatch) Watch(ctx context.Context, key string, opts ...clientv3.OpOption) clientv3.WatchChan { + w.started <- struct{}{} + select { + case <-w.release: + case <-ctx.Done(): + } + return w.Watcher.Watch(ctx, key, opts...) +} + func putEnabledKeyspaceTestMeta(t *testing.T, client *clientv3.Client, id uint32, state keyspacepb.KeyspaceState, gcType string) int64 { t.Helper() value, err := proto.Marshal(&keyspacepb.KeyspaceMeta{ @@ -154,14 +191,60 @@ func TestEnabledKeyspaceCacheLoadsAllPagesAtOneRevision(t *testing.T) { resp, err := client.Txn(ctx).Then(ops...).Commit() require.NoError(t, err) revision = resp.Header.Revision + firstPage := make(chan int64, 1) + releasePage := make(chan struct{}) + clientWithPause := *client + clientWithPause.KV = &pauseAfterFirstPageKV{KV: client.KV, firstPage: firstPage, release: releasePage} termCtx, stop := context.WithCancel(context.Background()) defer stop() - cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) - entries, loadedRevision, err := cache.load() + cache := newEnabledKeyspaceCache(termCtx, &clientWithPause, enabledKeyspaceTestPrefix) + type loadedSnapshot struct { + entries map[uint32]enabledKeyspace + revision int64 + err error + } + loaded := make(chan loadedSnapshot, 1) + go func() { + entries, loadedRevision, err := cache.load() + loaded <- loadedSnapshot{entries: entries, revision: loadedRevision, err: err} + }() + select { + case firstRevision := <-firstPage: + require.Equal(t, revision, firstRevision) + case <-ctx.Done(): + t.Fatal("first metadata page was not read") + } + _, err = client.Delete(ctx, fmt.Sprintf("%s%08d", enabledKeyspaceTestPrefix, enabledKeyspacePageSize+1)) require.NoError(t, err) - require.Equal(t, revision, loadedRevision) - require.Len(t, entries, enabledKeyspacePageSize+1) - require.Equal(t, enabledKeyspace{id: enabledKeyspacePageSize + 1, gcManagementType: keyspace.KeyspaceLevelGC}, entries[enabledKeyspacePageSize+1]) + insertedRevision := putEnabledKeyspaceTestMeta(t, client, 300, keyspacepb.KeyspaceState_ENABLED, keyspace.UnifiedGC) + close(releasePage) + var initial loadedSnapshot + select { + case initial = <-loaded: + case <-ctx.Done(): + t.Fatal("fixed-revision metadata load did not finish") + } + require.NoError(t, initial.err) + require.Equal(t, revision, initial.revision) + require.Len(t, initial.entries, enabledKeyspacePageSize+1) + require.Contains(t, initial.entries, uint32(enabledKeyspacePageSize+1)) + require.NotContains(t, initial.entries, uint32(300)) + + cache.publish(initial.entries, initial.revision) + watchDone := make(chan error, 1) + go func() { watchDone <- cache.watch(initial.revision + 1) }() + list, applied, err := cache.snapshotAtLeast(ctx, insertedRevision) + require.NoError(t, err) + require.GreaterOrEqual(t, applied, insertedRevision) + require.Len(t, list, enabledKeyspacePageSize+1) + require.NotContains(t, list, enabledKeyspace{id: enabledKeyspacePageSize + 1, gcManagementType: keyspace.KeyspaceLevelGC}) + require.Equal(t, enabledKeyspace{id: 300, gcManagementType: keyspace.UnifiedGC}, list[len(list)-1]) + stop() + select { + case <-watchDone: + case <-ctx.Done(): + t.Fatal("metadata watch did not stop") + } } func TestEnabledKeyspaceCacheRejectsMalformedMetadataUntilReload(t *testing.T) { @@ -201,33 +284,49 @@ func TestEnabledKeyspaceCacheReloadsAfterCompactedWatch(t *testing.T) { termCtx, cancel := context.WithCancel(context.Background()) defer cancel() cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) - entries, revision, err := cache.load() - require.NoError(t, err) - require.Equal(t, first, revision) - cache.publish(entries, revision) - + watchStarting := make(chan struct{}, 1) + releaseWatch := make(chan struct{}) + firstWatch := true + cache.watcherFactory = func(client *clientv3.Client) clientv3.Watcher { + watcher := clientv3.NewWatcher(client) + if !firstWatch { + return watcher + } + firstWatch = false + return &pauseBeforeWatch{Watcher: watcher, started: watchStarting, release: releaseWatch} + } + done := make(chan struct{}) + go func() { + cache.run() + close(done) + }() ctx, stop := context.WithTimeout(context.Background(), 10*time.Second) defer stop() + select { + case <-watchStarting: + case <-ctx.Done(): + t.Fatal("initial cache load did not reach watch startup") + } + list, revision, err := cache.snapshotAtLeast(ctx, first) + require.NoError(t, err) + require.Equal(t, first, revision) + require.Equal(t, []enabledKeyspace{{id: 1, gcManagementType: keyspace.KeyspaceLevelGC}}, list) _, err = client.Delete(ctx, fmt.Sprintf("%s%08d", enabledKeyspaceTestPrefix, 1)) require.NoError(t, err) latest := putEnabledKeyspaceTestMeta(t, client, 2, keyspacepb.KeyspaceState_ENABLED, keyspace.UnifiedGC) _, err = client.Compact(ctx, latest, clientv3.WithCompactPhysical()) require.NoError(t, err) - // The watch cannot replay the missing revisions. Its caller must reload - // the complete prefix before publishing a newer waterline. - require.Error(t, cache.watch(revision+1)) - list, applied, err := cache.snapshotAtLeast(ctx, revision) - require.NoError(t, err) - require.Equal(t, revision, applied) - require.Equal(t, []enabledKeyspace{{id: 1, gcManagementType: keyspace.KeyspaceLevelGC}}, list) - - entries, revision, err = cache.load() - require.NoError(t, err) - cache.publish(entries, revision) - list, applied, err = cache.snapshotAtLeast(ctx, latest) + close(releaseWatch) + list, applied, err := cache.snapshotAtLeast(ctx, latest) require.NoError(t, err) require.GreaterOrEqual(t, applied, latest) require.Equal(t, []enabledKeyspace{{id: 2, gcManagementType: keyspace.UnifiedGC}}, list) + cancel() + select { + case <-done: + case <-ctx.Done(): + t.Fatal("cache run did not stop") + } } func TestEnabledKeyspaceCacheTermCancellationUnblocksWaiters(t *testing.T) { From 7ff7c9ad3c940da6acc3b53fe3c7d92268ec61c1 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 23 Sep 2026 11:42:44 +0800 Subject: [PATCH 19/25] gc: initialize watchers from enabled keyspace index Reuse one leader-local metadata index for full GC watch initialization. Register live delivery before probing the applied etcd revision so each watch starts from a complete ENABLED keyspace list. Signed-off-by: Wenxuan Zhang --- pkg/gc/gc_state_manager.go | 33 ++++++- pkg/gc/gc_state_manager_test.go | 8 ++ pkg/gc/gc_state_watcher.go | 117 +++++++++++++++++++----- pkg/gc/gc_state_watcher_test.go | 154 ++++++++++++++++++++++++++++++++ server/server.go | 1 + 5 files changed, 291 insertions(+), 22 deletions(-) diff --git a/pkg/gc/gc_state_manager.go b/pkg/gc/gc_state_manager.go index 7382c6b665f..ae3e24ed48f 100644 --- a/pkg/gc/gc_state_manager.go +++ b/pkg/gc/gc_state_manager.go @@ -23,6 +23,7 @@ import ( "sync/atomic" "time" + clientv3 "go.etcd.io/etcd/client/v3" "go.uber.org/zap" "github.com/pingcap/failpoint" @@ -198,7 +199,10 @@ type GCStateManager struct { // A read/write - update cache procedure must be done while holding the outer mutex `GCStateManager.mu`. // A read-only operation can be done on gcStateCache directly without locking `GCStateManager.mu`. - gcStateCache *gcStateCache + gcStateCache *gcStateCache + etcdClient *clientv3.Client + enabledKeyspaces *enabledKeyspaceCache + cancelEnabledKeyspaces context.CancelFunc allKeyspacesGCStatesSingleFlight *syncutil.OrderedSingleFlight[map[uint32]GCState] allKeyspacesGCStatesExcludeGCBarriersSingleFlight *syncutil.OrderedSingleFlight[map[uint32]GCState] @@ -227,6 +231,14 @@ func NewGCStateManager(store endpoint.GCStateProvider, cfg config.PDServerConfig return m } +// SetEtcdClient supplies the client used by the leader-local keyspace index. +// It must be called before the manager's first leadership generation starts. +func (m *GCStateManager) SetEtcdClient(client *clientv3.Client) { + m.mu.Lock() + defer m.mu.Unlock() + m.etcdClient = client +} + type keyspaceNameKeyType struct{} var keyspaceNameKey = keyspaceNameKeyType{} @@ -248,14 +260,28 @@ func getKeyspaceNameFromCtx(ctx context.Context) string { // OnNodeBecomesLeader starts a local leadership generation and returns its teardown function. func (m *GCStateManager) OnNodeBecomesLeader() func() { m.mu.Lock() + if m.cancelEnabledKeyspaces != nil { + m.cancelEnabledKeyspaces() + } m.nextLeadershipGeneration++ generation := m.nextLeadershipGeneration m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) m.activeLeadershipGeneration.Store(generation) m.gcStateCache.clearAll() + m.enabledKeyspaces = nil + m.cancelEnabledKeyspaces = nil + if m.etcdClient != nil { + termCtx, cancel := context.WithCancel(context.Background()) + m.cancelEnabledKeyspaces = cancel + m.enabledKeyspaces = newEnabledKeyspaceCache(termCtx, m.etcdClient, keypath.KeyspaceMetaPrefix()) + } + enabledKeyspaces := m.enabledKeyspaces m.barrierMetrics.clearMetrics() productionBarrierMetrics.current.Store(m.barrierMetrics) m.mu.Unlock() + if enabledKeyspaces != nil { + go enabledKeyspaces.run() + } return func() { m.mu.Lock() @@ -264,6 +290,11 @@ func (m *GCStateManager) OnNodeBecomesLeader() func() { return } m.activeLeadershipGeneration.Store(0) + if m.cancelEnabledKeyspaces != nil { + m.cancelEnabledKeyspaces() + m.cancelEnabledKeyspaces = nil + m.enabledKeyspaces = nil + } m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) m.gcStateCache.clearAll() m.barrierMetrics.clearMetrics() diff --git a/pkg/gc/gc_state_manager_test.go b/pkg/gc/gc_state_manager_test.go index 1cf86c5d372..9148a2d7744 100644 --- a/pkg/gc/gc_state_manager_test.go +++ b/pkg/gc/gc_state_manager_test.go @@ -98,6 +98,8 @@ type newGCStateManagerForTestOptions struct { serverNodes int etcdServerCfgModifier func(cfg *embed.Config) etcdClientCfgModifier etcdutil.CreateEtcdClientOpt + useEnabledKeyspaceCache bool + beforeLeader func(*clientv3.Client) } func (opt *newGCStateManagerForTestOptions) generateKeyspacesByCount(count int) { @@ -144,6 +146,9 @@ func newGCStateManagerForTest(t testing.TB, opt newGCStateManagerForTestOptions) kgm := keyspace.NewKeyspaceGroupManager(ctx, s, client) keyspaceManager := keyspace.NewKeyspaceManager(ctx, s, mockcluster.NewCluster(ctx, config.NewPersistOptions(cfg)), allocator, &config.KeyspaceConfig{}, kgm, nil) gcStateManager = NewGCStateManager(s.GetGCStateProvider(), cfg.PDServerCfg, keyspaceManager) + if opt.useEnabledKeyspaceCache { + gcStateManager.SetEtcdClient(client) + } t.Cleanup(gcStateManager.CloseBarrierMetrics) err = kgm.Bootstrap(ctx) @@ -210,6 +215,9 @@ func newGCStateManagerForTest(t testing.TB, opt newGCStateManagerForTestOptions) } } + if opt.beforeLeader != nil { + opt.beforeLeader(client) + } stopGCStateManager := gcStateManager.OnNodeBecomesLeader() originalClean := clean clean = func() { diff --git a/pkg/gc/gc_state_watcher.go b/pkg/gc/gc_state_watcher.go index 57bd7217dcc..690e7a8040f 100644 --- a/pkg/gc/gc_state_watcher.go +++ b/pkg/gc/gc_state_watcher.go @@ -16,6 +16,8 @@ package gc import ( "context" + "fmt" + "time" "go.uber.org/zap" @@ -24,6 +26,9 @@ import ( "github.com/pingcap/log" "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/keyspace" + "github.com/tikv/pd/pkg/keyspace/constant" + "github.com/tikv/pd/pkg/utils/keypath" ) type gcStateChangeKind uint8 @@ -75,6 +80,7 @@ const ( defaultGCStateWatchInitialBatchSize = 1024 defaultGCStateWatchInitChannelCapacity = 1 defaultGCStateWatchLiveChannelCapacity = 1024 + gcStateWatchMetadataWaitTimeout = 5 * time.Minute ) type gcStateWatchConfig struct { @@ -110,6 +116,7 @@ type GCStateWatcher struct { pendingInit []GCStateChange pendingLiveCount int dirtyDuringInit map[uint32]struct{} + enabledKeyspaces *enabledKeyspaceCache } func newGCStateWatcher(parent context.Context, cfg gcStateWatchConfig, skipLoadingInitial bool) *GCStateWatcher { @@ -295,9 +302,34 @@ func (m *GCStateManager) registerGCStateWatcher( cfg gcStateWatchConfig, ) (*GCStateWatcher, error) { watcher := newGCStateWatcher(ctx, cfg, skipLoadingInitial) + var cache *enabledKeyspaceCache + var generation uint64 + if !skipLoadingInitial { + m.mu.RLock() + generation = m.activeLeadershipGeneration.Load() + cache = m.enabledKeyspaces + m.mu.RUnlock() + if generation == 0 { + watcher.cancel(errs.ErrNotLeader) + return nil, errs.ErrNotLeader + } + if cache != nil { + failpoint.InjectCall("watchGCStatesBeforeCacheReady") + waitCtx, cancel := context.WithTimeout(ctx, gcStateWatchMetadataWaitTimeout) + err := cache.waitReady(waitCtx) + cancel() + if err != nil { + if ctx.Err() == nil && cache.termCtx.Err() != nil { + err = errs.ErrNotLeader + } + watcher.cancel(err) + return nil, err + } + } + } m.mu.Lock() - if m.activeLeadershipGeneration.Load() == 0 { + if m.activeLeadershipGeneration.Load() == 0 || (generation != 0 && m.activeLeadershipGeneration.Load() != generation) { m.mu.Unlock() watcher.cancel(errs.ErrNotLeader) return nil, errs.ErrNotLeader @@ -305,6 +337,7 @@ func (m *GCStateManager) registerGCStateWatcher( m.nextWatcherID++ watcher.manager = m watcher.id = m.nextWatcherID + watcher.enabledKeyspaces = cache m.watchers[watcher.id] = watcher gcStateWatcherGauge.Inc() m.mu.Unlock() @@ -337,26 +370,26 @@ func (m *GCStateManager) loadInitialGCStates(watcher *GCStateWatcher, batchSize } } - err := m.iterateAllKeyspacesGCStates( - watcher.ctx, - true, - func(uint32) bool { return true }, - func(state GCState) { - if stopped { - return - } - failpoint.InjectCall("watchGCStatesInitialStateLoaded", state.KeyspaceID) - if watcher.Err() != nil { - stopped = true - return - } - batch = append(batch, NewGCStateUpsert(state)) - if len(batch) == batchSize { - stopped = !flush() - } - }, - nil, - ) + addState := func(state GCState) { + if stopped { + return + } + failpoint.InjectCall("watchGCStatesInitialStateLoaded", state.KeyspaceID) + if watcher.Err() != nil { + stopped = true + return + } + batch = append(batch, NewGCStateUpsert(state)) + if len(batch) == batchSize { + stopped = !flush() + } + } + var err error + if watcher.enabledKeyspaces != nil { + err = m.iterateEnabledKeyspacesGCStates(watcher.ctx, watcher.enabledKeyspaces, addState) + } else { + err = m.iterateAllKeyspacesGCStates(watcher.ctx, true, func(uint32) bool { return true }, addState, nil) + } if stopped || watcher.Err() != nil { return @@ -371,6 +404,48 @@ func (m *GCStateManager) loadInitialGCStates(watcher *GCStateWatcher, batchSize close(watcher.initCh) } +func (m *GCStateManager) iterateEnabledKeyspacesGCStates( + ctx context.Context, + cache *enabledKeyspaceCache, + cb func(GCState), +) error { + // The default Get is linearizable. This exact key supplies only the global + // revision; metadata membership comes from the shared index. + probeCtx, cancel := context.WithTimeout(ctx, enabledKeyspaceRequestTimeout) + resp, err := cache.client.Get(probeCtx, keypath.KeyspaceMetaPrefix()) + cancel() + if err != nil { + return fmt.Errorf("probe keyspace metadata revision: %w", err) + } + waitCtx, cancel := context.WithTimeout(ctx, gcStateWatchMetadataWaitTimeout) + entries, _, err := cache.snapshotAtLeast(waitCtx, resp.Header.Revision) + cancel() + if err != nil { + return fmt.Errorf("wait for keyspace metadata revision %d: %w", resp.Header.Revision, err) + } + + nullState, err := m.getGCStateImpl(constant.NullKeyspaceID, true) + if err != nil { + return err + } + cb(nullState) + for _, entry := range entries { + if err := ctx.Err(); err != nil { + return err + } + if entry.gcManagementType != keyspace.KeyspaceLevelGC { + cb(GCState{KeyspaceID: entry.id, IsKeyspaceLevel: false}) + continue + } + state, err := m.getGCStateImpl(entry.id, true) + if err != nil { + return err + } + cb(state) + } + return nil +} + func (m *GCStateManager) terminateGCStateWatcher( watcher *GCStateWatcher, cause error, diff --git a/pkg/gc/gc_state_watcher_test.go b/pkg/gc/gc_state_watcher_test.go index 9fd65012763..0405d4f96ae 100644 --- a/pkg/gc/gc_state_watcher_test.go +++ b/pkg/gc/gc_state_watcher_test.go @@ -19,15 +19,18 @@ import ( "errors" "fmt" "math" + "sync" "testing" "time" promtestutil "github.com/prometheus/client_golang/prometheus/testutil" "github.com/stretchr/testify/require" + clientv3 "go.etcd.io/etcd/client/v3" "github.com/pingcap/failpoint" "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/utils/keypath" ) @@ -509,3 +512,154 @@ func TestGCStateWatcherDonePublishesFirstCause(t *testing.T) { require.ErrorIs(t, w.Err(), errs.ErrNotLeader) require.Equal(t, done, w.Done()) } + +func TestGCStateWatcherUsesEnabledMetadataCache(t *testing.T) { + _, _, manager, clean, cancel := newGCStateManagerForTest(t, newGCStateManagerForTestOptions{useEnabledKeyspaceCache: true}) + defer clean() + defer cancel() + ctx, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + require.NoError(t, manager.enabledKeyspaces.waitReady(ctx)) + + id := uint32(19) + _, err := manager.keyspaceManager.CreateKeyspaceByID(&keyspace.CreateKeyspaceByIDRequest{ + ID: &id, Name: "watch-cache-created", Config: map[string]string{keyspace.GCManagementType: keyspace.KeyspaceLevelGC}, CreateTime: time.Now().Unix(), + }) + require.NoError(t, err) + // A full watcher must use the index even when the legacy iterator fails. + const failpointName = "github.com/tikv/pd/pkg/gc/iterateAllKeyspacesGCStatesError" + require.NoError(t, failpoint.Enable(failpointName, `return("legacy iterator used")`)) + defer func() { require.NoError(t, failpoint.Disable(failpointName)) }() + w, err := manager.WatchGCStates(ctx, false) + require.NoError(t, err) + defer w.Close() + for { + changes, err := w.RecvBatch(16) + require.NoError(t, err) + for _, change := range changes { + if state := mustUpsert(t, change); state.KeyspaceID == id { + require.True(t, state.IsKeyspaceLevel) + return + } + } + } +} + +func TestGCStateWatcherWaitReadyStopsOnCancelAndLeaderLoss(t *testing.T) { + _, _, manager, clean, cancel := newGCStateManagerForTest(t, newGCStateManagerForTestOptions{ + useEnabledKeyspaceCache: true, + beforeLeader: func(c *clientv3.Client) { + _, err := c.Put(context.Background(), keypath.KeyspaceMetaPath(19), "invalid protobuf") + require.NoError(t, err) + }, + }) + defer clean() + defer cancel() + + ctx, stop := context.WithCancel(context.Background()) + result := make(chan error, 1) + go func() { + _, err := manager.WatchGCStates(ctx, false) + result <- err + }() + stop() + require.ErrorIs(t, <-result, context.Canceled) + require.Empty(t, manager.watchers) + + waiting := make(chan struct{}, 1) + const waitHook = "github.com/tikv/pd/pkg/gc/watchGCStatesBeforeCacheReady" + require.NoError(t, failpoint.EnableCall(waitHook, func() { waiting <- struct{}{} })) + defer func() { require.NoError(t, failpoint.Disable(waitHook)) }() + result = make(chan error, 1) + go func() { + _, err := manager.WatchGCStates(context.Background(), false) + result <- err + }() + select { + case <-waiting: + case <-time.After(5 * time.Second): + t.Fatal("watcher did not begin waiting for cache readiness") + } + stopTerm := manager.OnNodeBecomesLeader() + defer stopTerm() + select { + case err := <-result: + require.ErrorIs(t, err, errs.ErrNotLeader) + case <-time.After(5 * time.Second): + t.Fatal("watcher did not exit on leader change") + } + require.Empty(t, manager.watchers) +} + +func TestGCStateWatcherIndexedInitialMergesConcurrentGCWrite(t *testing.T) { + _, _, manager, clean, cancel := newGCStateManagerForTest(t, newGCStateManagerForTestOptions{useEnabledKeyspaceCache: true}) + defer clean() + defer cancel() + ctx, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + require.NoError(t, manager.enabledKeyspaces.waitReady(ctx)) + _, err := manager.AdvanceTxnSafePoint(2, 10, time.Now()) + require.NoError(t, err) + + reached := make(chan struct{}) + release := make(chan struct{}) + var once sync.Once + releaseLoader := func() { once.Do(func() { close(release) }) } + defer releaseLoader() + const hook = "github.com/tikv/pd/pkg/gc/watchGCStatesInitialStateLoaded" + require.NoError(t, failpoint.EnableCall(hook, func(id uint32) { + if id == 2 { + close(reached) + <-release + } + })) + defer func() { require.NoError(t, failpoint.Disable(hook)) }() + w, err := manager.WatchGCStates(ctx, false) + require.NoError(t, err) + defer w.Close() + select { + case <-reached: + case <-ctx.Done(): + t.Fatal("initial loader did not reach keyspace 2") + } + _, err = manager.AdvanceTxnSafePoint(2, 20, time.Now()) + require.NoError(t, err) + for { + changes, err := w.RecvBatch(1) + require.NoError(t, err) + if state := mustUpsert(t, changes[0]); state.KeyspaceID == 2 { + require.Equal(t, uint64(20), state.TxnSafePoint) + break + } + } + releaseLoader() + require.Eventually(t, func() bool { + for { + change, ok, err := w.receiveOne(false) + require.NoError(t, err) + if !ok { + return w.initDone + } + state := mustUpsert(t, change) + require.False(t, state.KeyspaceID == 2 && state.TxnSafePoint == 10) + } + }, 5*time.Second, 10*time.Millisecond) +} + +func TestGCStateWatcherIndexedPostRegistrationErrorCleansUp(t *testing.T) { + _, _, manager, clean, cancel := newGCStateManagerForTest(t, newGCStateManagerForTestOptions{useEnabledKeyspaceCache: true}) + defer clean() + defer cancel() + ctx, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + require.NoError(t, manager.enabledKeyspaces.waitReady(ctx)) + const hook = "github.com/tikv/pd/pkg/gc/watchGCStatesRegistered" + require.NoError(t, failpoint.EnableCall(hook, manager.cancelEnabledKeyspaces)) + defer func() { require.NoError(t, failpoint.Disable(hook)) }() + w, err := manager.WatchGCStates(ctx, false) + require.NoError(t, err) + defer w.Close() + _, err = w.RecvBatch(1) + require.Error(t, err) + require.NotContains(t, manager.watchers, w.id) +} diff --git a/server/server.go b/server/server.go index 20e0579dea9..2847ea1a549 100644 --- a/server/server.go +++ b/server/server.go @@ -563,6 +563,7 @@ func (s *Server) startServer(ctx context.Context) error { log.Info("no metering config provided, the metering writer will not be started") } s.gcStateManager = gc.NewGCStateManager(s.storage.GetGCStateProvider(), s.cfg.PDServerCfg, s.keyspaceManager) + s.gcStateManager.SetEtcdClient(s.client) s.hbStreams = hbstream.NewHeartbeatStreams(ctx, "", s.cluster) // initial hot_region_storage in here. From 6128cb7b1b927e3514515df99884e931d5315790 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 23 Sep 2026 11:55:33 +0800 Subject: [PATCH 20/25] gc: limit enabled keyspace index to NextGen watches Keep Classic full watch initialization on its existing iterator path. Verify production wiring and the post-registration revision barrier with real etcd watchers. Signed-off-by: Wenxuan Zhang --- pkg/gc/gc_state_watcher.go | 1 + pkg/gc/gc_state_watcher_test.go | 96 +++++++++++++++++++++++++++++++++ server/server.go | 5 +- tests/server/gc/gc_test.go | 22 ++++++++ 4 files changed, 123 insertions(+), 1 deletion(-) diff --git a/pkg/gc/gc_state_watcher.go b/pkg/gc/gc_state_watcher.go index 690e7a8040f..793db7f9759 100644 --- a/pkg/gc/gc_state_watcher.go +++ b/pkg/gc/gc_state_watcher.go @@ -417,6 +417,7 @@ func (m *GCStateManager) iterateEnabledKeyspacesGCStates( if err != nil { return fmt.Errorf("probe keyspace metadata revision: %w", err) } + failpoint.InjectCall("watchGCStatesTargetRevisionProbed", resp.Header.Revision) waitCtx, cancel := context.WithTimeout(ctx, gcStateWatchMetadataWaitTimeout) entries, _, err := cache.snapshotAtLeast(waitCtx, resp.Header.Revision) cancel() diff --git a/pkg/gc/gc_state_watcher_test.go b/pkg/gc/gc_state_watcher_test.go index 0405d4f96ae..861aa8b9b7a 100644 --- a/pkg/gc/gc_state_watcher_test.go +++ b/pkg/gc/gc_state_watcher_test.go @@ -545,6 +545,102 @@ func TestGCStateWatcherUsesEnabledMetadataCache(t *testing.T) { } } +func TestGCStateWatcherWaitsForCommittedMetadataRevision(t *testing.T) { + _, _, manager, clean, cancel := newGCStateManagerForTest(t, newGCStateManagerForTestOptions{useEnabledKeyspaceCache: true}) + defer clean() + defer cancel() + ctx, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + + // Replace the test term's cache with one whose real etcd watch cannot + // start until this test releases it. Its initial snapshot is ready, but + // the subsequent metadata commit remains unapplied at registration. + watchStarted := make(chan struct{}, 16) + releaseWatch := make(chan struct{}) + var releaseOnce sync.Once + release := func() { releaseOnce.Do(func() { close(releaseWatch) }) } + defer release() + manager.mu.Lock() + manager.cancelEnabledKeyspaces() + termCtx, termCancel := context.WithCancel(context.Background()) + cache := newEnabledKeyspaceCache(termCtx, manager.etcdClient, keypath.KeyspaceMetaPrefix()) + cache.watcherFactory = func(client *clientv3.Client) clientv3.Watcher { + return &pauseBeforeWatch{ + Watcher: clientv3.NewWatcher(client), + started: watchStarted, + release: releaseWatch, + } + } + manager.enabledKeyspaces = cache + manager.cancelEnabledKeyspaces = termCancel + manager.mu.Unlock() + cacheDone := make(chan struct{}) + go func() { + cache.run() + close(cacheDone) + }() + defer func() { + termCancel() + select { + case <-cacheDone: + case <-time.After(5 * time.Second): + t.Error("replacement cache did not stop") + } + }() + require.NoError(t, cache.waitReady(ctx)) + select { + case <-watchStarted: + case <-ctx.Done(): + t.Fatal("replacement cache did not reach the watch") + } + + id := uint32(19) + _, err := manager.keyspaceManager.CreateKeyspaceByID(&keyspace.CreateKeyspaceByIDRequest{ + ID: &id, Name: "watch-cache-lagged", Config: map[string]string{keyspace.GCManagementType: keyspace.KeyspaceLevelGC}, CreateTime: time.Now().Unix(), + }) + require.NoError(t, err) + commit, err := manager.etcdClient.Get(ctx, keypath.KeyspaceMetaPath(id)) + require.NoError(t, err) + require.Less(t, cache.appliedRevision(), commit.Header.Revision) + + probed := make(chan int64, 1) + const hook = "github.com/tikv/pd/pkg/gc/watchGCStatesTargetRevisionProbed" + require.NoError(t, failpoint.EnableCall(hook, func(revision int64) { probed <- revision })) + defer func() { require.NoError(t, failpoint.Disable(hook)) }() + w, err := manager.WatchGCStates(ctx, false) + require.NoError(t, err) + defer w.Close() + select { + case target := <-probed: + require.GreaterOrEqual(t, target, commit.Header.Revision) + case <-ctx.Done(): + t.Fatal("watcher did not probe the target revision") + } + received := make(chan error, 1) + go func() { + _, err := w.RecvBatch(1) + received <- err + }() + select { + case err := <-received: + t.Fatalf("initial state arrived before the metadata watch resumed: %v", err) + case <-time.After(200 * time.Millisecond): + } + + release() + require.NoError(t, <-received) + for { + changes, err := w.RecvBatch(16) + require.NoError(t, err) + for _, change := range changes { + if state := mustUpsert(t, change); state.KeyspaceID == id { + require.True(t, state.IsKeyspaceLevel) + return + } + } + } +} + func TestGCStateWatcherWaitReadyStopsOnCancelAndLeaderLoss(t *testing.T) { _, _, manager, clean, cancel := newGCStateManagerForTest(t, newGCStateManagerForTestOptions{ useEnabledKeyspaceCache: true, diff --git a/server/server.go b/server/server.go index 2847ea1a549..a3ba0ce3bf8 100644 --- a/server/server.go +++ b/server/server.go @@ -90,6 +90,7 @@ import ( "github.com/tikv/pd/pkg/utils/tsoutil" "github.com/tikv/pd/pkg/utils/typeutil" "github.com/tikv/pd/pkg/versioninfo" + "github.com/tikv/pd/pkg/versioninfo/kerneltype" "github.com/tikv/pd/server/cluster" "github.com/tikv/pd/server/config" @@ -563,7 +564,9 @@ func (s *Server) startServer(ctx context.Context) error { log.Info("no metering config provided, the metering writer will not be started") } s.gcStateManager = gc.NewGCStateManager(s.storage.GetGCStateProvider(), s.cfg.PDServerCfg, s.keyspaceManager) - s.gcStateManager.SetEtcdClient(s.client) + if kerneltype.IsNextGen() { + s.gcStateManager.SetEtcdClient(s.client) + } s.hbStreams = hbstream.NewHeartbeatStreams(ctx, "", s.cluster) // initial hot_region_storage in here. diff --git a/tests/server/gc/gc_test.go b/tests/server/gc/gc_test.go index e324fc20b6a..66601e94afd 100644 --- a/tests/server/gc/gc_test.go +++ b/tests/server/gc/gc_test.go @@ -1114,6 +1114,28 @@ func TestWatchGCStatesInitialAndSkipInitialRegistrationBoundary(t *testing.T) { re.Empty(firstAfterRegistration.GetGcBarriers()) } +func TestWatchGCStatesUsesIndexOnlyInNextGen(t *testing.T) { + cluster := newWatchGCStatesCluster(t, 1, true) + leader := cluster.GetLeaderServer() + require.NotNil(t, leader) + const legacyIterator = "github.com/tikv/pd/pkg/gc/iterateAllKeyspacesGCStatesError" + const legacyError = "legacy keyspace iterator reached" + require.NoError(t, failpoint.Enable(legacyIterator, `return("legacy keyspace iterator reached")`)) + defer func() { require.NoError(t, failpoint.Disable(legacyIterator)) }() + + client := newWatchGCStatesClient(t, leader.GetAddr()) + stream, _ := openWatchGCStates(t, client, testutil.NewRequestHeader(leader.GetClusterID()), false) + if kerneltype.IsNextGen() { + state := recvWatchGCStateForKeyspace(t, stream, constant.NullKeyspaceID) + require.False(t, state.GetIsKeyspaceLevelGc()) + return + } + response, err := stream.Recv() + require.Nil(t, response) + require.ErrorContains(t, err, legacyError) + require.Equal(t, codes.Unavailable, status.Code(err)) +} + func TestWatchGCStatesRequestPreflight(t *testing.T) { tests := []struct { name string From 674e5017f8f7908a764058a97e5404fa5ae2aeb5 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 23 Sep 2026 12:40:21 +0800 Subject: [PATCH 21/25] gc: log initial enabled keyspace cache load Record when the complete enabled keyspace index first becomes usable, including the total load time, entry count, and applied revision. Signed-off-by: Wenxuan Zhang --- pkg/gc/enabled_keyspace_cache.go | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/pkg/gc/enabled_keyspace_cache.go b/pkg/gc/enabled_keyspace_cache.go index 4b8e5fd5546..83d82eab199 100644 --- a/pkg/gc/enabled_keyspace_cache.go +++ b/pkg/gc/enabled_keyspace_cache.go @@ -78,6 +78,8 @@ func newEnabledKeyspaceCache(termCtx context.Context, client *clientv3.Client, p // run blocks until the leadership term ends. A failed or compacted watch is // followed by a complete reload, so no missing revision is silently skipped. func (c *enabledKeyspaceCache) run() { + initialLoadStartedAt := time.Now() + initialLoadLogged := false retryDelay := enabledKeyspaceRetryDelay var lastLog time.Time suppressedErrors := 0 @@ -85,7 +87,14 @@ func (c *enabledKeyspaceCache) run() { entries, revision, err := c.load() phase := "load" if err == nil { - c.publish(entries, revision) + if c.publish(entries, revision) && !initialLoadLogged { + log.Info("load enabled keyspace cache completed", + zap.String("prefix", c.prefix), + zap.Int64("revision", revision), + zap.Int("enabled-keyspace-count", len(entries)), + zap.Duration("cost", time.Since(initialLoadStartedAt))) + initialLoadLogged = true + } watchStarted := time.Now() err = c.watch(revision + 1) phase = "watch" @@ -191,17 +200,18 @@ func (c *enabledKeyspaceCache) snapshotAtLeast(ctx context.Context, revision int } } -func (c *enabledKeyspaceCache) publish(entries map[uint32]enabledKeyspace, revision int64) { +func (c *enabledKeyspaceCache) publish(entries map[uint32]enabledKeyspace, revision int64) bool { c.mu.Lock() defer c.mu.Unlock() if c.termCtx.Err() != nil { - return + return false } c.entries = entries c.revision = revision c.ready = true close(c.changed) c.changed = make(chan struct{}) + return true } func (c *enabledKeyspaceCache) publishProgress(changes map[uint32]*enabledKeyspace, revision int64) { From 7e39580850c17bf18c9993a7e9aac35098220088 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 23 Sep 2026 12:43:02 +0800 Subject: [PATCH 22/25] gc: log every full enabled keyspace cache load Report each successful full load and reload with its own elapsed time, so operators can observe recovery cost after watch resynchronization. Signed-off-by: Wenxuan Zhang --- pkg/gc/enabled_keyspace_cache.go | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/pkg/gc/enabled_keyspace_cache.go b/pkg/gc/enabled_keyspace_cache.go index 83d82eab199..d0ae01012be 100644 --- a/pkg/gc/enabled_keyspace_cache.go +++ b/pkg/gc/enabled_keyspace_cache.go @@ -78,22 +78,20 @@ func newEnabledKeyspaceCache(termCtx context.Context, client *clientv3.Client, p // run blocks until the leadership term ends. A failed or compacted watch is // followed by a complete reload, so no missing revision is silently skipped. func (c *enabledKeyspaceCache) run() { - initialLoadStartedAt := time.Now() - initialLoadLogged := false retryDelay := enabledKeyspaceRetryDelay var lastLog time.Time suppressedErrors := 0 for c.termCtx.Err() == nil { + loadStartedAt := time.Now() entries, revision, err := c.load() phase := "load" if err == nil { - if c.publish(entries, revision) && !initialLoadLogged { + if c.publish(entries, revision) { log.Info("load enabled keyspace cache completed", zap.String("prefix", c.prefix), zap.Int64("revision", revision), zap.Int("enabled-keyspace-count", len(entries)), - zap.Duration("cost", time.Since(initialLoadStartedAt))) - initialLoadLogged = true + zap.Duration("cost", time.Since(loadStartedAt))) } watchStarted := time.Now() err = c.watch(revision + 1) From 39aaee4e3118d4fa4a8546bd236b809d2052181f Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Wed, 23 Sep 2026 19:56:47 +0800 Subject: [PATCH 23/25] storage: linearize read-only GC state revision validation Compare-only etcd transactions can validate against a stale follower, causing false conflicts or missing concurrent GC state changes. Require linearizable validation without advancing the GC revision or changing write transaction atomicity. Add three-node regression coverage for lagging followers and verify that read-only transactions preserve both GC and etcd revisions. Signed-off-by: Wenxuan Zhang --- pkg/storage/endpoint/gc_states.go | 10 ++ pkg/storage/endpoint/gc_states_txn_test.go | 181 +++++++++++++++++++++ 2 files changed, 191 insertions(+) create mode 100644 pkg/storage/endpoint/gc_states_txn_test.go diff --git a/pkg/storage/endpoint/gc_states.go b/pkg/storage/endpoint/gc_states.go index 439bb0ca914..00ae63b56d8 100644 --- a/pkg/storage/endpoint/gc_states.go +++ b/pkg/storage/endpoint/gc_states.go @@ -472,6 +472,16 @@ func (p GCStateProvider) RunInGCStateTransaction(f func(wb *GCStateWriteBatch) e OpType: kv.RawTxnOpPut, Value: nextRevision, }) + } else { + // etcd treats a Compare-only transaction as serializable and may check + // the revision on a stale follower. A non-serializable Get makes etcd + // linearize the whole transaction before evaluating the comparison, + // including when it takes the empty Else branch. It does not write or + // advance the revision; its response is counted along with the ops below. + ops = append(ops, kv.RawTxnOp{ + Key: revisionKey, + OpType: kv.RawTxnOpGet, + }) } txn, err := p.storage.createRawTxn() diff --git a/pkg/storage/endpoint/gc_states_txn_test.go b/pkg/storage/endpoint/gc_states_txn_test.go new file mode 100644 index 00000000000..733f7e0b940 --- /dev/null +++ b/pkg/storage/endpoint/gc_states_txn_test.go @@ -0,0 +1,181 @@ +// Copyright 2026 TiKV Project Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package endpoint + +import ( + "context" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/require" + clientv3 "go.etcd.io/etcd/client/v3" + + "github.com/pingcap/errors" + + "github.com/tikv/pd/pkg/errs" + "github.com/tikv/pd/pkg/storage/kv" + "github.com/tikv/pd/pkg/utils/etcdutil" + "github.com/tikv/pd/pkg/utils/keypath" +) + +func TestGCStateReadOnlyTransactionLaggingFollower(t *testing.T) { + for _, concurrentWrite := range []bool{false, true} { + name := "completed-write" + if concurrentWrite { + name = "concurrent-write" + } + t.Run(name, func(t *testing.T) { + re := require.New(t) + servers, _, clean := etcdutil.NewTestEtcdCluster(t, 3, nil) + defer clean() + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) + defer cancel() + + leader, follower := servers[0], servers[1] + for _, server := range servers { + if uint64(server.Server.ID()) == server.Server.Lead() { + leader = server + } else { + follower = server + } + } + re.NotEqual(leader.Server.ID(), follower.Server.ID()) + newClient := func(endpoint string) *clientv3.Client { + client, err := clientv3.New(clientv3.Config{Endpoints: []string{endpoint}, Context: ctx}) + re.NoError(err) + t.Cleanup(func() { re.NoError(client.Close()) }) + return client + } + freshClient := newClient(leader.Config().ListenClientUrls[0].String()) + staleClient := newClient(follower.Config().ListenClientUrls[0].String()) + freshKV, staleKV := kv.NewEtcdKVBase(freshClient), kv.NewEtcdKVBase(staleClient) + writer := NewStorageEndpoint(freshKV, nil).GetGCStateProvider() + // Pin ordinary reads to the leader and validation to the follower. This + // reproduces the problematic round-robin routing without relying on chance. + reader := NewStorageEndpoint(struct { + kv.Base + kv.RawTxnCapable + }{freshKV, staleKV}, nil).GetGCStateProvider() + write := func() error { + return writer.RunInGCStateTransaction(func(wb *GCStateWriteBatch) error { + return wb.SetGCSafePoint(0, 100) + }) + } + re.NoError(write()) + revisionKey := keypath.GCStateRevisionPath() + before, err := staleClient.Get(ctx, revisionKey) + re.NoError(err) + re.Len(before.Kvs, 1) + re.Equal("1", string(before.Kvs[0].Value)) + + // Isolate only the follower's Raft traffic; client RPCs still work and + // the other two members can commit the next (and final) write. + for _, server := range servers { + if server != follower { + server.Server.CutPeer(follower.Server.ID()) + follower.Server.CutPeer(server.Server.ID()) + } + } + // MendPeer restarts remote pipelines and must only be called once. + mend := sync.OnceFunc(func() { + for _, server := range servers { + if server != follower { + server.Server.MendPeer(follower.Server.ID()) + follower.Server.MendPeer(server.Server.ID()) + } + } + }) + defer mend() + if !concurrentWrite { + re.NoError(write()) + } + + ready := make(chan struct{}) + done := make(chan error, 1) + go func() { + done <- reader.RunInGCStateTransaction(func(_ *GCStateWriteBatch) error { + defer close(ready) + if concurrentWrite { + // Commit after the reader has sampled revision 1, so validation + // must reject it even though the follower still has revision 1. + return write() + } + return nil + }) + }() + select { + case <-ready: + case <-ctx.Done(): + t.Fatal(ctx.Err()) + } + fresh, err := freshClient.Get(ctx, revisionKey) + re.NoError(err) + re.Len(fresh.Kvs, 1) + re.Equal("2", string(fresh.Kvs[0].Value)) + stale, err := staleClient.Get(ctx, revisionKey, clientv3.WithSerializable()) + re.NoError(err) + re.Equal(before.Kvs, stale.Kvs, "the follower must still be behind after all writes finish") + + // With no new writes, validation must wait for the follower to catch up. + // An empty Compare-only transaction instead returns immediately using + // revision 1: a false conflict, or a missed real conflict, respectively. + var txnErr error + select { + case txnErr = <-done: + mend() + case <-time.After(500 * time.Millisecond): + mend() + select { + case txnErr = <-done: + case <-ctx.Done(): + t.Fatal(ctx.Err()) + } + } + if concurrentWrite { + re.True(errors.ErrorEqual(txnErr, errs.ErrEtcdTxnConflict), "got %v", txnErr) + } else { + re.NoError(txnErr) + } + after, err := freshClient.Get(ctx, revisionKey) + re.NoError(err) + re.Equal(fresh.Kvs, after.Kvs, "read-only validation must not write the GC revision") + re.Equal(fresh.Header.Revision, after.Header.Revision, "read-only validation must not advance etcd revision") + }) + } +} + +func TestGCStateReadOnlyTransactionRevision(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + defer clean() + re := require.New(t) + provider := NewStorageEndpoint(kv.NewEtcdKVBase(client), nil).GetGCStateProvider() + for _, initialized := range []bool{false, true} { + if initialized { + re.NoError(provider.RunInGCStateTransaction(func(wb *GCStateWriteBatch) error { + return wb.SetGCSafePoint(0, 100) + })) + } + before, err := client.Get(client.Ctx(), keypath.GCStateRevisionPath()) + re.NoError(err) + for range 2 { + re.NoError(provider.RunInGCStateTransaction(func(_ *GCStateWriteBatch) error { return nil })) + } + after, err := client.Get(client.Ctx(), keypath.GCStateRevisionPath()) + re.NoError(err) + re.Equal(before.Kvs, after.Kvs) + re.Equal(before.Header.Revision, after.Header.Revision) + } +} From 3b097f6d4781f0478c56ee03a0cb7525e6f6ae01 Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 24 Sep 2026 11:55:46 +0800 Subject: [PATCH 24/25] gc: bound enabled keyspace watch creation time Prevent stalled watch creation from blocking keyspace cache recovery for the entire leadership term. Retry when creation times out while allowing established watches to continue running. Cover stalled creation, recovery after timeout, and continued updates from healthy watches with regression tests. Signed-off-by: Wenxuan Zhang --- pkg/gc/enabled_keyspace_cache.go | 7 ++ pkg/gc/enabled_keyspace_cache_test.go | 152 ++++++++++++++++++++++++++ 2 files changed, 159 insertions(+) diff --git a/pkg/gc/enabled_keyspace_cache.go b/pkg/gc/enabled_keyspace_cache.go index d0ae01012be..04564a51d43 100644 --- a/pkg/gc/enabled_keyspace_cache.go +++ b/pkg/gc/enabled_keyspace_cache.go @@ -33,6 +33,7 @@ import ( "github.com/tikv/pd/pkg/keyspace" "github.com/tikv/pd/pkg/utils/etcdutil" + "github.com/tikv/pd/pkg/utils/grpcutil" ) const ( @@ -298,7 +299,13 @@ func (c *enabledKeyspaceCache) watch(nextRevision int64) error { defer watcher.Close() watchCtx, cancel := context.WithCancel(clientv3.WithRequireLeader(c.termCtx)) defer cancel() + done := make(chan struct{}) + go grpcutil.CheckStream(watchCtx, cancel, done) watchCh := watcher.Watch(watchCtx, c.prefix, clientv3.WithPrefix(), clientv3.WithRev(nextRevision), clientv3.WithProgressNotify()) + done <- struct{}{} + if err := watchCtx.Err(); err != nil { + return fmt.Errorf("keyspace metadata watch creation failed: %w", err) + } ticker := time.NewTicker(etcdutil.RequestProgressInterval) defer ticker.Stop() pending := make(map[uint32]*enabledKeyspace) diff --git a/pkg/gc/enabled_keyspace_cache_test.go b/pkg/gc/enabled_keyspace_cache_test.go index 4dc4c4330fd..a568021a985 100644 --- a/pkg/gc/enabled_keyspace_cache_test.go +++ b/pkg/gc/enabled_keyspace_cache_test.go @@ -18,13 +18,16 @@ import ( "context" "errors" "fmt" + "net" "sync" "testing" "time" "github.com/gogo/protobuf/proto" "github.com/stretchr/testify/require" + pb "go.etcd.io/etcd/api/v3/etcdserverpb" clientv3 "go.etcd.io/etcd/client/v3" + "google.golang.org/grpc" "github.com/pingcap/kvproto/pkg/keyspacepb" @@ -61,6 +64,31 @@ type pauseBeforeWatch struct { release <-chan struct{} } +type signalWatchCreated struct { + clientv3.Watcher + created chan<- struct{} +} + +func (w *signalWatchCreated) Watch(ctx context.Context, key string, opts ...clientv3.OpOption) clientv3.WatchChan { + ch := w.Watcher.Watch(ctx, key, opts...) + w.created <- struct{}{} + return ch +} + +type neverCreateWatchServer struct { + pb.UnimplementedWatchServer + started chan struct{} +} + +func (s *neverCreateWatchServer) Watch(stream pb.Watch_WatchServer) error { + if _, err := stream.Recv(); err != nil { + return err + } + close(s.started) + <-stream.Context().Done() + return stream.Context().Err() +} + func (w *pauseBeforeWatch) Watch(ctx context.Context, key string, opts ...clientv3.OpOption) clientv3.WatchChan { w.started <- struct{}{} select { @@ -362,3 +390,127 @@ func TestEnabledKeyspaceCacheTermCancellationUnblocksWaiters(t *testing.T) { t.Fatal("cache run did not stop after term cancellation") } } + +func TestEnabledKeyspaceCacheWatchCreationTimesOut(t *testing.T) { + listener, err := net.Listen("tcp", "127.0.0.1:0") + require.NoError(t, err) + server := grpc.NewServer() + backend := &neverCreateWatchServer{started: make(chan struct{})} + pb.RegisterWatchServer(server, backend) + go func() { _ = server.Serve(listener) }() + defer server.Stop() + client, err := clientv3.New(clientv3.Config{Endpoints: []string{listener.Addr().String()}, DialTimeout: time.Second}) + require.NoError(t, err) + defer client.Close() + termCtx, cancel := context.WithCancel(context.Background()) + defer cancel() + cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) + done := make(chan error, 1) + go func() { done <- cache.watch(1) }() + select { + case <-backend.started: + case <-time.After(3 * time.Second): + t.Fatal("watch create request did not reach server") + } + select { + case err := <-done: + require.Error(t, err) + require.NoError(t, termCtx.Err()) + case <-time.After(6 * time.Second): + cancel() + <-done + t.Fatal("watch creation did not time out") + } +} + +func TestEnabledKeyspaceCacheReloadsAfterWatchCreationTimeout(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + first := putEnabledKeyspaceTestMeta(t, client, 1, keyspacepb.KeyspaceState_ENABLED, keyspace.KeyspaceLevelGC) + termCtx, cancel := context.WithCancel(context.Background()) + cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) + watchStarted := make(chan struct{}, 1) + releaseWatch := make(chan struct{}) + firstWatch := true + cache.watcherFactory = func(client *clientv3.Client) clientv3.Watcher { + watcher := clientv3.NewWatcher(client) + if !firstWatch { + return watcher + } + firstWatch = false + return &pauseBeforeWatch{Watcher: watcher, started: watchStarted, release: releaseWatch} + } + done := make(chan struct{}) + go func() { + cache.run() + close(done) + }() + defer func() { + cancel() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Error("cache did not stop") + } + }() + ctx, stop := context.WithTimeout(context.Background(), 8*time.Second) + defer stop() + select { + case <-watchStarted: + case <-ctx.Done(): + t.Fatal("initial cache load did not reach watch creation") + } + initial, revision, err := cache.snapshotAtLeast(ctx, first) + require.NoError(t, err) + require.Equal(t, first, revision) + require.Equal(t, []enabledKeyspace{{id: 1, gcManagementType: keyspace.KeyspaceLevelGC}}, initial) + latest := putEnabledKeyspaceTestMeta(t, client, 2, keyspacepb.KeyspaceState_ENABLED, keyspace.UnifiedGC) + list, applied, err := cache.snapshotAtLeast(ctx, latest) + require.NoError(t, err) + require.GreaterOrEqual(t, applied, latest) + require.Equal(t, []enabledKeyspace{ + {id: 1, gcManagementType: keyspace.KeyspaceLevelGC}, + {id: 2, gcManagementType: keyspace.UnifiedGC}, + }, list) +} + +func TestEnabledKeyspaceCacheWatchSurvivesCreationTimeout(t *testing.T) { + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + t.Cleanup(clean) + termCtx, stop := context.WithCancel(context.Background()) + cache := newEnabledKeyspaceCache(termCtx, client, enabledKeyspaceTestPrefix) + created := make(chan struct{}, 1) + cache.watcherFactory = func(client *clientv3.Client) clientv3.Watcher { + return &signalWatchCreated{Watcher: clientv3.NewWatcher(client), created: created} + } + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + entries, revision, err := cache.load() + require.NoError(t, err) + require.True(t, cache.publish(entries, revision)) + done := make(chan error, 1) + go func() { done <- cache.watch(revision + 1) }() + defer func() { + stop() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Error("watch did not stop") + } + }() + select { + case <-created: + case <-ctx.Done(): + t.Fatal("watch was not created") + } + select { + case err := <-done: + t.Fatalf("watch ended after creation: %v", err) + case <-time.After(4 * time.Second): + } + latest := putEnabledKeyspaceTestMeta(t, client, 3, keyspacepb.KeyspaceState_ENABLED, keyspace.UnifiedGC) + list, applied, err := cache.snapshotAtLeast(ctx, latest) + require.NoError(t, err) + require.GreaterOrEqual(t, applied, latest) + require.Equal(t, []enabledKeyspace{{id: 3, gcManagementType: keyspace.UnifiedGC}}, list) +} From df470d9481f313daf023d1f88f25281ce549e1cc Mon Sep 17 00:00:00 2001 From: Wenxuan Zhang Date: Thu, 24 Sep 2026 13:51:43 +0800 Subject: [PATCH 25/25] gc: prevent stale cache reads during leadership changes Leadership was visible to lock-free GC readers before stale cache entries were cleared. Keep cache reads disabled until the new term is initialized, including when replacing an active generation. Cover follower promotion and generation replacement with concurrent reads against a newer persisted safe point. Signed-off-by: Wenxuan Zhang --- pkg/gc/gc_state_manager.go | 6 ++- pkg/gc/gc_state_manager_test.go | 79 +++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 1 deletion(-) diff --git a/pkg/gc/gc_state_manager.go b/pkg/gc/gc_state_manager.go index ae3e24ed48f..40897157ca4 100644 --- a/pkg/gc/gc_state_manager.go +++ b/pkg/gc/gc_state_manager.go @@ -260,13 +260,16 @@ func getKeyspaceNameFromCtx(ctx context.Context) string { // OnNodeBecomesLeader starts a local leadership generation and returns its teardown function. func (m *GCStateManager) OnNodeBecomesLeader() func() { m.mu.Lock() + // Disable lock-free cache reads throughout the reset, including when + // replacing an active leadership generation. + m.activeLeadershipGeneration.Store(0) if m.cancelEnabledKeyspaces != nil { m.cancelEnabledKeyspaces() } m.nextLeadershipGeneration++ generation := m.nextLeadershipGeneration m.terminateAllGCStateWatchersLocked(errs.ErrNotLeader, watcherTerminationLeaderLost) - m.activeLeadershipGeneration.Store(generation) + failpoint.InjectCall("beforeLeaderGCStateCacheReset") m.gcStateCache.clearAll() m.enabledKeyspaces = nil m.cancelEnabledKeyspaces = nil @@ -278,6 +281,7 @@ func (m *GCStateManager) OnNodeBecomesLeader() func() { enabledKeyspaces := m.enabledKeyspaces m.barrierMetrics.clearMetrics() productionBarrierMetrics.current.Store(m.barrierMetrics) + m.activeLeadershipGeneration.Store(generation) m.mu.Unlock() if enabledKeyspaces != nil { go enabledKeyspaces.run() diff --git a/pkg/gc/gc_state_manager_test.go b/pkg/gc/gc_state_manager_test.go index 9148a2d7744..ab54d4dac44 100644 --- a/pkg/gc/gc_state_manager_test.go +++ b/pkg/gc/gc_state_manager_test.go @@ -300,6 +300,85 @@ func (s *gcStateManagerTestSuite) TestGCStateWatchLeadershipGeneration() { re.ErrorIs(second.Err(), errs.ErrNotLeader) } +func TestGCStateLeadershipClearsCacheBeforeEnablingReads(t *testing.T) { + for _, replacingLeader := range []bool{false, true} { + name := "follower-promotion" + if replacingLeader { + name = "leader-replacement" + } + t.Run(name, func(t *testing.T) { + re := require.New(t) + _, client, clean := etcdutil.NewTestEtcdCluster(t, 1, nil) + defer clean() + provider := endpoint.NewStorageEndpoint(kv.NewEtcdKVBase(client), nil).GetGCStateProvider() + manager := NewGCStateManager(provider, config.PDServerConfig{}, nil) + defer manager.CloseBarrierMetrics() + if replacingLeader { + stop := manager.OnNodeBecomesLeader() + defer stop() + } + + writeSafePoint := func(safePoint uint64) { + re.NoError(provider.RunInGCStateTransaction(func(wb *endpoint.GCStateWriteBatch) error { + return wb.SetGCSafePoint(constant.NullKeyspaceID, safePoint) + })) + } + writeSafePoint(100) + // Reads can populate the cache even while this manager is a follower. + state, err := manager.GetGCState(constant.NullKeyspaceID, true) + re.NoError(err) + re.Equal(uint64(100), state.GCSafePoint) + // Another leader advances storage after the cached read. + writeSafePoint(200) + + resetReached := make(chan struct{}) + releaseReset := make(chan struct{}) + release := sync.OnceFunc(func() { close(releaseReset) }) + const hook = "github.com/tikv/pd/pkg/gc/beforeLeaderGCStateCacheReset" + re.NoError(failpoint.EnableCall(hook, func() { + close(resetReached) + <-releaseReset + })) + defer func() { re.NoError(failpoint.Disable(hook)) }() + leaderReady := make(chan struct{}) + var stopLeader func() + go func() { + stopLeader = manager.OnNodeBecomesLeader() + close(leaderReady) + }() + defer func() { + release() + select { + case <-leaderReady: + stopLeader() + case <-time.After(5 * time.Second): + t.Error("leadership initialization did not stop") + } + }() + select { + case <-resetReached: + case <-time.After(5 * time.Second): + t.Fatal("leadership initialization did not reach the cache reset") + } + + // Cache reads must remain disabled while the stale snapshot is present. + safePoint, err := manager.CompatibleLoadGCSafePoint(constant.NullKeyspaceID) + re.NoError(err) + re.Equal(uint64(200), safePoint) + release() + select { + case <-leaderReady: + case <-time.After(5 * time.Second): + t.Fatal("leadership initialization did not finish") + } + re.True(manager.nodeIsLeader()) + safePoint, err = manager.CompatibleLoadGCSafePoint(constant.NullKeyspaceID) + re.NoError(err) + re.Equal(uint64(200), safePoint) + }) + } +} + func (s *gcStateManagerTestSuite) TestGCStateWatchLoadsInitialStatesIncrementally() { re := s.Require() w, err := s.manager.registerGCStateWatcher(context.Background(), false, gcStateWatchConfig{