Skip to content

fix(prometheus): guard family membership against metric-name prefix collisions - #309

Open
detail-app[bot] wants to merge 1 commit into
masterfrom
detail/bug-fix/fix-prometheus-guard-family-membership-against-met-9848c7
Open

detail-app[bot] wants to merge 1 commit into
masterfrom
detail/bug-fix/fix-prometheus-guard-family-membership-against-met-9848c7

Conversation

@detail-app

@detail-app detail-app Bot commented Sep 16, 2026

Copy link
Copy Markdown

Detail bug report: View on Detail

Fixes ENG-710

Bug

PolicyStreamingFilter (the streaming Prometheus policy proxy in src/signals/prometheus/streaming_filter.zig) associated each sample to the current metric family with a bare std.mem.startsWith(sample_name, current_metric_name) and never invalidated the stored family metadata (# HELP/# TYPE, description, type) when a family's samples were all policy-dropped. A later, different family whose sample name merely extended the dropped family's name (e.g. foo_extra after a fully-dropped foo) was treated as a member of the dropped family, causing:

  • Orphan # HELP foo/# TYPE foo lines emitted in front of the unrelated foo_extra sample.
  • Description/type policies evaluating the metadata-less foo_extra against foo's stale description/type — wrong-direction false drops (and, symmetrically, false keeps under KEEP policies).

Fix

  • Added belongsToCurrentFamily, a type-aware membership predicate: exact name match for counter/gauge/untyped/# TYPE-less families, plus <name>_bucket/_sum/_count suffixes for histograms and <name>_sum/_count for summaries. _total is treated as part of the family name (this repo's convention), not a suffix, so a hardcoded suffix whitelist can't re-introduce the bug.
  • maybeWriteMetadata and shouldKeepMetric now use this predicate, so a dropped family's metadata is never emitted for a non-member and description/type fall back to null for non-members during policy evaluation.
  • Added clearCurrentMetadata to invalidate stored metadata at a family boundary, an orthogonal safeguard against stale metadata leaking past a fully-dropped family.

Testing

  • 7 new inline regression tests (the two reported repros plus guards for the _total suffix footgun, histogram/summary suffix handling, and cross-family-boundary invalidation). Full unit suite passes (526 pass, 1 skipped).
  • Mutation check: reverting only the fix logic fails 5 of the 7 new tests; the 2 histogram/summary guards still pass, confirming they're positive guards for the suffix predicate rather than bug-catchers.
  • Build, zig fmt --check, ziglint, and task do (format + lint + ReleaseSafe tests) all pass.
  • End-to-end through the running edge-prometheus binary against a fake Prometheus upstream serving prefix-colliding exposition text, via GET /metrics:
    • name-DROP ^foo$ emits only foo_extra 2 with no orphan # HELP/# TYPE foo;
    • description-DROP matching foo's HELP still keeps the metadata-less foo_extra;
    • the realistic go_memstats_alloc_bytes gauge vs go_memstats_alloc_bytes_total counter under a description-DROP keeps the _total counter and a distinct bar family's own metadata;
    • a histogram label-DROP on the first le="0.1" bucket emits the family # HELP/# TYPE exactly once before the first kept sample.
    • Request-counter deltas on /_edge/metrics confirmed the requests traversed the filter path.

Not verified:

  • Per-test filtered execution: the Zig 0.16 test runner doesn't accept a --test-filter flag and the build defines no per-name test step, so per-test isolation is evidenced by the mutation check rather than per-name runs.
  • Per-request lines_kept/lines_dropped/bytes_forwarded stats over the wire: the server discards the filter's FilterStats return value, so these stats are only observable via the existing unit tests (which pass), not from a live response.

Automatic Fixes PRs can be configured here.

@macroscopeapp

macroscopeapp Bot commented Sep 16, 2026

Copy link
Copy Markdown

Approvability

Verdict: Approved at 6d0e0da

Macroscope's review found this PR approvable — This is a narrowly scoped Prometheus filtering bug fix that prevents stale HELP/TYPE and policy metadata from crossing metric-family boundaries while preserving histogram and summary suffix handling. The implementation is self-contained and backed by targeted regression tests, with no schema, infrastructure, or sensitive-area changes.

You can add or adjust custom eligibility rules. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant