Skip to content

fix(prometheus): gate metric metadata on family match in policy filter - #306

Open
detail-app[bot] wants to merge 1 commit into
masterfrom
detail/bug-fix/fix-prometheus-gate-metric-metadata-on-family-matc-370880
Open

detail-app[bot] wants to merge 1 commit into
masterfrom
detail/bug-fix/fix-prometheus-gate-metric-metadata-on-family-matc-370880

Conversation

@detail-app

@detail-app detail-app Bot commented Sep 16, 2026

Copy link
Copy Markdown

Detail bug report: View on Detail

Summary

PolicyStreamingFilter evaluated Prometheus samples against the policy engine using stale # HELP / # TYPE metadata from the most-recent prior typed metric family, so a metadata-less sample (no preceding # HELP/# TYPE of its own) could be incorrectly kept or dropped based on a different family's type/description. Roots: shouldKeepMetric copied the sticky current_description/current_type_str into the policy context without checking the sample's metric name belonged to that family, while the sibling maybeWriteMetadata already gated on startsWith(metric_name, current_metric_name) — an asymmetry present since the filter was first added.

Fixes ENG-707

Bug

  • A sample whose metric has no preceding # HELP/# TYPE line was evaluated with the most-recent prior typed family's description/type rather than null.
  • Observable with:
    # HELP a A metric
    # TYPE a counter
    a 1
    b 2
    
    Under a DROP policy metric_type exact "counter", b 2 was incorrectly dropped because it inherited a's "counter".
  • Impact: keep/drop decisions using metric_type or description could be wrong for any metadata-less family immediately following a differently-named typed family (common in real scrapes from custom/federated exporters that omit # TYPE for some families).
  • Producer was shouldKeepMetric in streaming_filter.zig; field_accessor.zig faithfully surfaced whatever description/metric_type was placed on the context, so no change was needed there.

Fix

In shouldKeepMetric, gate ctx.description / ctx.metric_type on whether the sample belongs to the current family, mirroring the guard maybeWriteMetadata already uses for emitting metadata:

const meta_matches = self.current_metric_name.len > 0 and
    std.mem.startsWith(u8, sample.metric_name, self.current_metric_name);

A metadata-less sample is now evaluated with null description/type. The startsWith rule is intentionally aligned with the emission path so policy-evaluation and metadata-emission use the same family-membership heuristic, preserving histogram/summary _bucket/_sum/_count members (which share the family name prefix).

Testing

  • Two regression tests added in streaming_filter.zig:
    • A metadata-less sample following a typed counter family is no longer dropped by a metric_type exact "counter" rule (the original reproducer).
    • Type-based policies still apply to histogram family members with suffixed sample names, locking in the startsWith heuristic against future over-tightening.
    • The redundant description-field variant was deliberately excluded to keep the suite lean; both fields share one guard.
  • Verified the new tests are discriminating: reverting only the production guard reproduces exactly the original failure (the metadata-less sample is dropped), and re-applying it returns the suite to green.
  • zig build, zig build prometheus, zig fmt --check, ziglint, and zig build test --summary all (521 passed, 1 skipped, 0 failed) all pass.
  • End-to-end verification against a mocked upstream: ran edge-prometheus with a file policy (metric_type exact "counter", DROP) and a Python server serving the bug-triggering exposition. GET /metrics through the real httpz frontend returned b 2 (kept) with a 1 dropped; rebuilding without the guard returned an empty body (b 2 dropped, having inherited counter), confirming the fix resolves the bug end-to-end. Config/policies were restored afterward.
  • No memory leaks in the new tests (std.testing.allocator); results are deterministic across repeated runs.

Automatic Fixes PRs can be configured here.

@macroscopeapp

macroscopeapp Bot commented Sep 16, 2026

Copy link
Copy Markdown

Approvability

Verdict: Approved at 4919eea

Macroscope's review found this PR approvable — This is a contained Prometheus filtering bug fix that prevents stale HELP/TYPE metadata from affecting unrelated samples while preserving histogram and summary family handling. The implementation is limited to the existing filter and includes focused regression coverage.

You can add or adjust custom eligibility rules. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant