feat(ingester): Add head-only queried series metrics#7500
Open
feat(ingester): Add head-only queried series metrics#7500
Conversation
13dd902 to
aae20fa
Compare
Add two new metrics that track query activity against the TSDB head only: 1. cortex_ingester_queried_series_from_head: Estimated unique series queried from head within a configurable time window (HLL-based). 2. cortex_ingester_queried_metric_series_in_head: Current head cardinality for each metric name that was queried within the configured window. Implementation uses a BlockChunkQuerierFunc wrapper that intercepts Select calls only for head queriers (identified by RangeHead/Head ULIDs). The wrapper collects series hashes for the HLL tracker and records metric names for the per-metric cardinality tracker. Key design decisions: - Reuses existing ActiveQueriedSeries HLL and ActiveQueriedSeriesService for the total series metric (no new service needed) - Per-metric-name tracking uses a simple map[string]time.Time with lazy count lookup via seriesInMetric.getSeriesCountForMetric() on collection - Only __name__= equality matchers trigger per-metric tracking - Sampling is supported for the HLL metric to reduce overhead - Both metrics share the same configurable time window New configuration flags (all experimental): - ingester.head-queried-series-metrics-enabled - ingester.head-queried-series-metrics-windows - ingester.head-queried-series-metrics-window-duration - ingester.head-queried-series-metrics-sample-rate Signed-off-by: Ben Ye <benye@amazon.com>
aae20fa to
d79e35f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR adds two new metrics that track query activity against the TSDB head only (in-memory, ~2h data):
cortex_ingester_queried_series_from_head— Estimated unique series queried from head within a configurable time window (HLL-based, deduplicated across queries).cortex_ingester_queried_metric_series_in_head— Current head cardinality for each metric name that was queried within the configured window.Motivation
The existing
cortex_ingester_active_queried_seriesmetric tracks all queried series regardless of whether they come from head or on-disk blocks. For understanding active query patterns and resource usage, we specifically need to know what's being queried from the head (the hot, recent data).Implementation
BlockChunkQuerierFuncwrapper that interceptsSelectcalls only for head queriers (identified by RangeHead/Head ULIDs)ActiveQueriedSeriesHLL andActiveQueriedSeriesServicefor the total series metricmap[string]time.Timewith lazy count lookup viaseriesInMetric.getSeriesCountForMetric()on periodic collection__name__=equality matchers trigger per-metric trackingNew configuration flags (experimental)
-ingester.head-queried-series-metrics-enabledfalse-ingester.head-queried-series-metrics-windows2h-ingester.head-queried-series-metrics-window-duration15m-ingester.head-queried-series-metrics-sample-rate1.0Checklist
v1-guarantees.mdupdated (experimental feature)