Sharding - #1241
Open
CatherineF-dev wants to merge 4 commits into
Open
Conversation
The pod labels informer cached the full OwnerReferences of every pod, but getLabelsFromMeta only reads Kind and Name, and only for DaemonSet/StatefulSet/ReplicaSet/Job owners. Cache just those two fields for those kinds, drop owner references of other kinds, and nil out empty label maps. This shrinks the per-pod footprint of the cluster-wide pod metadata cache on large clusters.
Add --total-shards and --shard-id flags. When total-shards > 1, each replica exports only the events whose involved object UID hashes (FNV-1a) to its shard, so every event is exported by exactly one replica. shard-id defaults to -1, which derives the ID from the pod hostname ordinal suffix, matching StatefulSet replica names. The filter wraps the sink handler in createWatcher, so both the streaming path (handler called directly from streamingListEvents) and the reflector store path are covered. Filtered events are counted in the new sharding_filtered_events_count metric. Sharding by involved object UID rather than by namespace avoids namespace-size skew, and for pod events the involved object UID is the pod UID, which allows the pod label cache to be sharded by the same key in a follow-up.
When sharding is enabled, the pod labels informer transform drops pods whose UID belongs to another shard, using the same hash as the event filter. Since events are sharded by involved object UID and for pod events that is the pod UID, the shard that exports a pod's events always has that pod in its cache; owner label lookups keep working with no cross-shard misses. This divides the dominant steady-state memory consumer, the cluster-wide pod metadata cache, by the number of shards on each replica.
--sink=local writes each exported event as a JSON line to stdout (prefixed with EXPORTED_EVENT), including the pod owner labels looked up the same way the Stackdriver sink does. This allows running the exporter in clusters without Stackdriver access (e.g. kind) and verifying export behavior, such as exactly-once delivery under sharding, from pod logs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.