Skip to content

policy: Deduplicate SecretWatchers and TLS contexts - #2006

Merged
jrajahalme merged 1 commit into
mainfrom
policy-secret-sharing
Aug 22, 2026
Merged

policy: Deduplicate SecretWatchers and TLS contexts#2006
jrajahalme merged 1 commit into
mainfrom
policy-secret-sharing

Conversation

@jrajahalme

@jrajahalme jrajahalme commented Aug 20, 2026

Copy link
Copy Markdown
Member

Network policies may contain hundreds or thousands of HeaderMatch or TLS rules referring to a small set of Secrets. Each reference previously constructed a separate SecretWatcher or Envoy TLS ContextConfigImpl, including its SDS provider callbacks and other context state.

Add separate weak caches for SecretWatchers and upstream and downstream TLS contexts, keyed by the SDS name or Cilium TLSContext protobuf, respectively. Reuse a live context when an equivalent policy configuration is encountered.

Reset the caches together with the policy maps after an NPDS stream restart, preserving the existing agent-restart behavior. Prune expired cache entries asynchronously after policy updates and worker quiescence, keeping pruning off the policy-update hot path.

Benchmark an NPDS update containing 1,000 policies that reference the same SDS-derived CA validation context. The CA is delivered through a real dynamic SDS provider rather than embedded in the policy TLS contexts.

Optimized benchmark results (3 repetitions):

                                CPU time     Retained heap     Heap/policy
  TLS context cache enabled     1.47 ms      1.581 MiB         1.619 KiB
  TLS context cache disabled    3.95 ms      4.990 MiB         5.109 KiB

This reduces CPU time by approximately 63% and retained heap by 68%, saving about 3.41 MiB per 1,000-policy update and increasing throughput by 2.68x.

The benchmark uses Envoy's small, single-certificate test CA. Production SDS resources commonly contain larger CA bundles comparable to those shipped with operating systems, so the benchmark likely understates the memory impact of retaining duplicate certificate-related context state.

Benchmark results for deduplication of SecretWatchers on HeaderMatches are more modest (duplicating header sized secrets carries less overhead), but are still positive around 20-25% for CPU and memory.

@jrajahalme
jrajahalme requested a review from a team as a code owner August 20, 2026 09:01
@jrajahalme jrajahalme added the enhancement New feature or request label Aug 20, 2026
@jrajahalme
jrajahalme requested a review from sayboras August 20, 2026 09:01
@jrajahalme
jrajahalme force-pushed the policy-secret-sharing branch 3 times, most recently from 5782f11 to b96455e Compare August 20, 2026 13:05
@jrajahalme
jrajahalme requested a review from nezdolik August 20, 2026 13:06
@jrajahalme jrajahalme changed the title policy: Deduplicate TLS contexts policy: Deduplicate SecretWatchers and TLS contexts Aug 20, 2026
@jrajahalme jrajahalme removed the dont-merge/preview-only DON'T MERGE label Aug 20, 2026
@jrajahalme
jrajahalme force-pushed the policy-secret-sharing branch from b96455e to e0b9f37 Compare August 20, 2026 13:48
@jrajahalme

Copy link
Copy Markdown
Member Author

added a missing include for tidy

@jrajahalme

Copy link
Copy Markdown
Member Author

restarted integration test due to a flake fixed by #2007

Comment thread cilium/network_policy.cc Outdated
Comment thread cilium/secret_watcher.cc
Network policies may contain hundreds or thousands of HeaderMatch or TLS
rules referring to a small set of Secrets. Each reference previously
constructed a separate SecretWatcher or Envoy TLS ContextConfigImpl,
including its SDS provider callbacks and other context state.

Add separate weak caches for SecretWatchers and upstream and downstream
TLS contexts, keyed by the SDS name or Cilium TLSContext protobuf. Reuse
a live context when an equivalent policy configuration is encountered.

Reset the caches together with the policy maps after an NPDS stream restart,
preserving the existing agent-restart behavior. Prune expired cache entries
asynchronously after policy updates and worker quiescence, keeping pruning
off the policy-update hot path.

Benchmark an NPDS update containing 1,000 policies that reference the same
SDS-derived CA validation context. The CA is delivered through a real dynamic
SDS provider rather than embedded in the policy TLS contexts.

Optimized benchmark results (3 repetitions):

                            CPU time   Retained heap   Heap/policy
  TLS context cache enabled   1.47 ms      1.581 MiB      1.619 KiB
  TLS context cache disabled  3.95 ms      4.990 MiB      5.109 KiB

This reduces CPU time by approximately 63% and retained heap by 68%, saving
about 3.41 MiB per 1,000-policy update and increasing throughput by 2.68x.

The benchmark uses Envoy's small, single-certificate test CA. Production SDS
resources commonly contain larger CA bundles comparable to those shipped
with operating systems, so the benchmark likely understates the memory impact
of retaining duplicate certificate-related context state.

Benchmark results for deduplication of SecretWatchers on HeaderMatches
are more modest (duplicating header sized secrets carries less overhead),
but are still positive around 20-25% for CPU and memory.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
@jrajahalme
jrajahalme force-pushed the policy-secret-sharing branch from e0b9f37 to a3faf4d Compare August 21, 2026 13:47
@jrajahalme
jrajahalme requested a review from nezdolik August 21, 2026 13:48
@jrajahalme
jrajahalme enabled auto-merge (rebase) August 22, 2026 08:48
@jrajahalme
jrajahalme merged commit 7dfaf94 into main Aug 22, 2026
9 of 10 checks passed
@jrajahalme
jrajahalme deleted the policy-secret-sharing branch August 22, 2026 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants