fix(prometheus): gate metric metadata on family match in policy filter - #306
Open
detail-app[bot] wants to merge 1 commit into
Open
detail-app[bot] wants to merge 1 commit into
detail-app[bot] wants to merge 1 commit into
Conversation
ApprovabilityVerdict: Approved at Macroscope's review found this PR approvable — This is a contained Prometheus filtering bug fix that prevents stale HELP/TYPE metadata from affecting unrelated samples while preserving histogram and summary family handling. The implementation is limited to the existing filter and includes focused regression coverage. You can add or adjust custom eligibility rules. Learn more. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detail bug report: View on Detail
Summary
PolicyStreamingFilterevaluated Prometheus samples against the policy engine using stale# HELP/# TYPEmetadata from the most-recent prior typed metric family, so a metadata-less sample (no preceding# HELP/# TYPEof its own) could be incorrectly kept or dropped based on a different family's type/description. Roots:shouldKeepMetriccopied the stickycurrent_description/current_type_strinto the policy context without checking the sample's metric name belonged to that family, while the siblingmaybeWriteMetadataalready gated onstartsWith(metric_name, current_metric_name)— an asymmetry present since the filter was first added.Fixes ENG-707
Bug
# HELP/# TYPEline was evaluated with the most-recent prior typed family's description/type rather than null.metric_type exact "counter",b 2was incorrectly dropped because it inheriteda's"counter".metric_typeordescriptioncould be wrong for any metadata-less family immediately following a differently-named typed family (common in real scrapes from custom/federated exporters that omit# TYPEfor some families).shouldKeepMetricinstreaming_filter.zig;field_accessor.zigfaithfully surfaced whateverdescription/metric_typewas placed on the context, so no change was needed there.Fix
In
shouldKeepMetric, gatectx.description/ctx.metric_typeon whether the sample belongs to the current family, mirroring the guardmaybeWriteMetadataalready uses for emitting metadata:A metadata-less sample is now evaluated with null description/type. The
startsWithrule is intentionally aligned with the emission path so policy-evaluation and metadata-emission use the same family-membership heuristic, preserving histogram/summary_bucket/_sum/_countmembers (which share the family name prefix).Testing
streaming_filter.zig:metric_type exact "counter"rule (the original reproducer).startsWithheuristic against future over-tightening.zig build,zig build prometheus,zig fmt --check,ziglint, andzig build test --summary all(521 passed, 1 skipped, 0 failed) all pass.edge-prometheuswith a file policy (metric_type exact "counter", DROP) and a Python server serving the bug-triggering exposition.GET /metricsthrough the realhttpzfrontend returnedb 2(kept) witha 1dropped; rebuilding without the guard returned an empty body (b 2dropped, having inheritedcounter), confirming the fix resolves the bug end-to-end. Config/policies were restored afterward.std.testing.allocator); results are deterministic across repeated runs.Automatic Fixes PRs can be configured here.