From 455d3c0eb9087fca5e3f6651a5762684b36f3c38 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Thu, 27 Aug 2026 13:50:00 +0200 Subject: [PATCH] docs: pipeline: outputs: prometheus_remote_write: document stale metric expiration Document the one-hour staleness cut-off added to the Prometheus remote write output plugin in fluent-bit d46750cbb. - Add a Stale metric expiration section explaining that each flush drops metrics with a timestamp more than one hour old before the remote write payload is built, because backends reject stale samples and one rejected sample fails the whole request - Note that expiration is evaluated per label set, so an idle time series is dropped while actively updated series of the same metric are still sent - Note that the cut-off is fixed at one hour and can't be configured, and that expired metrics are dropped without an error - Call out the practical consequence: a backlog recovered from filesystem buffering after an outage longer than an hour, or a replay from an archive, doesn't reach the backend - Clarify backlog expiration scope Signed-off-by: Eric D. Schabell --- pipeline/outputs/prometheus-remote-write.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/pipeline/outputs/prometheus-remote-write.md b/pipeline/outputs/prometheus-remote-write.md index 3acbdb4d4..f30ca748b 100644 --- a/pipeline/outputs/prometheus-remote-write.md +++ b/pipeline/outputs/prometheus-remote-write.md @@ -37,6 +37,16 @@ This plugin supports the following parameters: | `uri` | Specify an optional HTTP URI for the target web server, for example `/someuri`. | _none_ | | `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | +## Stale metric expiration + +Stale metric expiration is available in Fluent Bit version 5.1.2 and greater. + +On each flush, Fluent Bit drops any metric whose timestamp is more than one hour old before it builds the remote write payload. Remote write backends reject samples this old as stale, and a single rejected sample fails the whole request, which would also leave the current samples undelivered. + +Expiration applies to each set of labels separately, so an idle time series is dropped while the actively updated series of the same metric are still sent. + +This cut-off is fixed at one hour and can't be configured. Metrics older than the cut-off are dropped without an error, and the remaining metrics in the same flush are still sent. When a backlog recovered from [filesystem buffering](../buffering.md#filesystem-buffering-hybrid) or a replay from an archive spans more than an hour, only the samples that are already older than the cut-off when the flush runs are dropped. Newer samples remain eligible for delivery. Because the cut-off is evaluated against the current time on each flush, a slow replay can expire more of the backlog as it progresses. + ## Get started The Prometheus remote write plugin works only with metrics collected by one of the metric input plugins. In the following example, host metrics are collected by the node exporter metrics plugin and then delivered by the Prometheus remote write output plugin.