Skip to content

feat(ingester): Add head-only queried series metrics#7500

Open
yeya24 wants to merge 1 commit intomasterfrom
head-queried-series-metrics
Open

feat(ingester): Add head-only queried series metrics#7500
yeya24 wants to merge 1 commit intomasterfrom
head-queried-series-metrics

Conversation

@yeya24
Copy link
Copy Markdown
Contributor

@yeya24 yeya24 commented May 10, 2026

What this PR does

This PR adds two new metrics that track query activity against the TSDB head only (in-memory, ~2h data):

  1. cortex_ingester_queried_series_from_head — Estimated unique series queried from head within a configurable time window (HLL-based, deduplicated across queries).

  2. cortex_ingester_queried_metric_series_in_head — Current head cardinality for each metric name that was queried within the configured window.

Motivation

The existing cortex_ingester_active_queried_series metric tracks all queried series regardless of whether they come from head or on-disk blocks. For understanding active query patterns and resource usage, we specifically need to know what's being queried from the head (the hot, recent data).

Implementation

  • Uses a BlockChunkQuerierFunc wrapper that intercepts Select calls only for head queriers (identified by RangeHead/Head ULIDs)
  • Reuses existing ActiveQueriedSeries HLL and ActiveQueriedSeriesService for the total series metric
  • Per-metric-name tracking uses a simple map[string]time.Time with lazy count lookup via seriesInMetric.getSeriesCountForMetric() on periodic collection
  • Only __name__= equality matchers trigger per-metric tracking
  • Sampling supported for the HLL metric to reduce overhead
  • Both metrics share the same configurable time window

New configuration flags (experimental)

Flag Default Description
-ingester.head-queried-series-metrics-enabled false Enable the feature
-ingester.head-queried-series-metrics-windows 2h Time windows to report
-ingester.head-queried-series-metrics-window-duration 15m HLL sub-window bucket size
-ingester.head-queried-series-metrics-sample-rate 1.0 Query sampling rate

Checklist

  • Tests updated
  • Documentation added
  • v1-guarantees.md updated (experimental feature)

@dosubot dosubot Bot added component/ingester type/feature type/observability To help know what is going on inside Cortex labels May 10, 2026
@yeya24 yeya24 force-pushed the head-queried-series-metrics branch from 13dd902 to aae20fa Compare May 11, 2026 01:03
Add two new metrics that track query activity against the TSDB head only:

1. cortex_ingester_queried_series_from_head: Estimated unique series
   queried from head within a configurable time window (HLL-based).

2. cortex_ingester_queried_metric_series_in_head: Current head cardinality
   for each metric name that was queried within the configured window.

Implementation uses a BlockChunkQuerierFunc wrapper that intercepts Select
calls only for head queriers (identified by RangeHead/Head ULIDs). The
wrapper collects series hashes for the HLL tracker and records metric names
for the per-metric cardinality tracker.

Key design decisions:
- Reuses existing ActiveQueriedSeries HLL and ActiveQueriedSeriesService
  for the total series metric (no new service needed)
- Per-metric-name tracking uses a simple map[string]time.Time with lazy
  count lookup via seriesInMetric.getSeriesCountForMetric() on collection
- Only __name__= equality matchers trigger per-metric tracking
- Sampling is supported for the HLL metric to reduce overhead
- Both metrics share the same configurable time window

New configuration flags (all experimental):
- ingester.head-queried-series-metrics-enabled
- ingester.head-queried-series-metrics-windows
- ingester.head-queried-series-metrics-window-duration
- ingester.head-queried-series-metrics-sample-rate

Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24 yeya24 force-pushed the head-queried-series-metrics branch from aae20fa to d79e35f Compare May 11, 2026 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant