Skip to content

gc, server: implement WatchGCStates streaming API #11264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wfxr wants to merge 25 commits into
base: master
Choose a base branch
from
Open

gc, server: implement WatchGCStates streaming API #11264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a26bfc0
docs: add WatchGCStates server design
wfxr Sep 3, 2026
b968d13
docs: refine WatchGCStates server design
wfxr Sep 3, 2026
d42e878
docs: streamline WatchGCStates server design
wfxr Sep 3, 2026
0b6e52a
docs: add WatchGCStates implementation plan
wfxr Sep 3, 2026
d5e3068
docs: consolidate WatchGCStates implementation tasks
wfxr Sep 3, 2026
6dd7d28
gc: add GC state watcher lifecycle
wfxr Sep 3, 2026
9a5d645
gc: publish and observe GC state changes
wfxr Sep 3, 2026
d418fe3
server: implement WatchGCStates stream
wfxr Sep 3, 2026
6110469
tests: cover WatchGCStates lifecycle
wfxr Sep 3, 2026
2cbef20
tests: tighten WatchGCStates integration coverage
wfxr Sep 3, 2026
9d7f4b0
gc: satisfy WatchGCStates static checks
wfxr Sep 3, 2026
238ad1a
gc: tighten WatchGCStates lifecycle guarantees
wfxr Sep 3, 2026
f8396f1
gc: preserve watch ordering across initial and live states
wfxr Sep 16, 2026
1c30446
gc: interrupt blocked watch sends on watcher termination
wfxr Sep 16, 2026
3e55ddc
docs: stop tracking local WatchGCStates design and plans
wfxr Sep 16, 2026
faa4d8f
tests: reuse WatchGCStates concurrency limit helper
wfxr Sep 20, 2026
9b98acc
gc: cache enabled keyspaces at applied etcd revisions
wfxr Sep 23, 2026
96bbf49
gc: make keyspace cache recovery observable and bounded
wfxr Sep 23, 2026
7ff7c9a
gc: initialize watchers from enabled keyspace index
wfxr Sep 23, 2026
6128cb7
gc: limit enabled keyspace index to NextGen watches
wfxr Sep 23, 2026
674e501
gc: log initial enabled keyspace cache load
wfxr Sep 23, 2026
7e39580
gc: log every full enabled keyspace cache load
wfxr Sep 23, 2026
39aaee4
storage: linearize read-only GC state revision validation
wfxr Sep 23, 2026
3b097f6
gc: bound enabled keyspace watch creation time
wfxr Sep 24, 2026
df470d9
gc: prevent stale cache reads during leadership changes
wfxr Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion client/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ require (
github.com/opentracing/opentracing-go v1.2.0
github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86
github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250
github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3
github.com/prometheus/client_golang v1.20.5
github.com/prometheus/client_model v0.6.1
Expand Down
4 changes: 2 additions & 2 deletions client/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c h1:xpW9bvK+HuuTm
github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ueLEUZgtx2cIogM0v4Zj5uvvzhuuiu7Pn8HzMPg=
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 h1:tdMsjOqUR7YXHoBitzdebTvOjs/swniBTOLy5XiMtuE=
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86/go.mod h1:exzhVYca3WRtd6gclGNErRWb1qEgff3LYta0LvRmON4=
github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 h1:6yUryXKVbKpCNdZWL58/OcZj8NPLUA/xsJYXSbsD59w=
github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE=
github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d h1:KS3ekak/ljCj5xvkGqbwVLi2eL7B8GFSYzU9TOUUPPo=
github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE=
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw=
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
Expand Down
5 changes: 5 additions & 0 deletions errors.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,6 +481,11 @@ error = '''
trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v
'''

["PD:gc:ErrGCStateWatcherSlowConsumer"]
error = '''
gc state watcher is too slow
'''

["PD:gc:ErrGlobalGCBarrierTSBehindTxnSafePoint"]
error = '''
trying to set a global GC barrier on ts %d which is already behind the txn safe point %d of keyspace %s
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ require (
github.com/pingcap/errcode v0.3.0
github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86
github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250
github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3
github.com/pingcap/metering_sdk v0.0.0-20260814062708-9e3b68cd9adf
github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,8 +490,8 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ue
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 h1:tdMsjOqUR7YXHoBitzdebTvOjs/swniBTOLy5XiMtuE=
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86/go.mod h1:exzhVYca3WRtd6gclGNErRWb1qEgff3LYta0LvRmON4=
github.com/pingcap/kvproto v0.0.0-20191211054548-3c6b38ea5107/go.mod h1:WWLmULLO7l8IOcQG+t+ItJ3fEcrL5FxF0Wu+HrMy26w=
github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250 h1:6yUryXKVbKpCNdZWL58/OcZj8NPLUA/xsJYXSbsD59w=
github.com/pingcap/kvproto v0.0.0-20260903054228-107095f1d250/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE=
github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d h1:KS3ekak/ljCj5xvkGqbwVLi2eL7B8GFSYzU9TOUUPPo=
github.com/pingcap/kvproto v0.0.0-20260903062353-65b4e27a438d/go.mod h1:z6+aAHB7dBkA+LyinEX+48/ImRJ3jag0Hg0c7wkhEvE=
github.com/pingcap/log v0.0.0-20210625125904-98ed8e2eb1c7/go.mod h1:8AanEdAHATuRurdGxZXBz0At+9avep+ub7U1AGYLIMM=
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw=
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4=
Expand Down
10 changes: 6 additions & 4 deletions pkg/errs/errno.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -548,10 +548,12 @@ var (

// GC errors
var (
ErrGCOnInvalidKeyspace = errors.Normalize("trying to manage GC in keyspace %v (id: %v) where keyspace level GC is not enabled", errors.RFCCodeText("PD:gc:ErrGCOnInvalidKeyspace"))
ErrDecreasingGCSafePoint = errors.Normalize("trying to update GC safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingGCSafePoint"))
ErrGCSafePointExceedsTxnSafePoint = errors.Normalize("trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v", errors.RFCCodeText("PD:gc:ErrGCSafePointExceedsTxnSafePoint"))
ErrDecreasingTxnSafePoint = errors.Normalize("trying to update txn safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingTxnSafePoint"))
ErrGCOnInvalidKeyspace = errors.Normalize("trying to manage GC in keyspace %v (id: %v) where keyspace level GC is not enabled", errors.RFCCodeText("PD:gc:ErrGCOnInvalidKeyspace"))
ErrDecreasingGCSafePoint = errors.Normalize("trying to update GC safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingGCSafePoint"))
ErrGCSafePointExceedsTxnSafePoint = errors.Normalize("trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v", errors.RFCCodeText("PD:gc:ErrGCSafePointExceedsTxnSafePoint"))
ErrDecreasingTxnSafePoint = errors.Normalize("trying to update txn safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingTxnSafePoint"))
// ErrGCStateWatcherSlowConsumer indicates that a watcher cannot keep up with live GC state changes.
ErrGCStateWatcherSlowConsumer = errors.Normalize("gc state watcher is too slow", errors.RFCCodeText("PD:gc:ErrGCStateWatcherSlowConsumer"))
ErrGCBarrierTSBehindTxnSafePoint = errors.Normalize("trying to set a GC barrier on ts %d which is already behind the txn safe point %d", errors.RFCCodeText("PD:gc:ErrGCBarrierTSBehindTxnSafePoint"))
ErrReservedGCBarrierID = errors.Normalize("trying to set a GC barrier with a barrier ID that is reserved: %v", errors.RFCCodeText("PD:gc:ErrReservedGCBarrierID"))
ErrGlobalGCBarrierTSBehindTxnSafePoint = errors.Normalize("trying to set a global GC barrier on ts %d which is already behind the txn safe point %d of keyspace %s", errors.RFCCodeText("PD:gc:ErrGlobalGCBarrierTSBehindTxnSafePoint"))
Expand Down
Loading
Loading