Skip to content

Backport(v1.19): fix(metrics): export-only clamp for buffer statistics (#5467) - #5488

Merged
Watson1978 merged 1 commit into
v1.19from
backport-to-v1.19/pr5467
Sep 7, 2026
Merged

Watson1978 merged 1 commit into
v1.19from
backport-to-v1.19/pr5467

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

Which issue(s) this PR fixes:
Backport #5467
Fixes #5303

What this PR does / why we need it:
Buffer size gauges (stage_byte_size, queue_byte_size, and derived total_queued_size) can go transiently negative when Fluentd core under/over-subtracts during concurrent stage/queue transitions (the deferred @stage_size_metrics.add after chunk unlock vs enqueue_chunk's sub — see #2712 / #2734). Those values are mirrored by the Prometheus plugin as fluentd_output_status_buffer_total_bytes / fluentd_output_status_buffer_stage_byte_size, which is what #5303 reports.

Clamping the gauge store on sub/dec is the wrong fix: it turns a self-correcting transient negative into a permanent over-count, so Buffer#storable? (which reads the raw gauge) eventually refuses every write. Thanks @Watson1978 for catching that.

This PR takes an export-only approach:

  1. Leave LocalMetrics gauge sub/dec/set semantics unchanged (negatives still allowed so the deferred-add race can self-heal).
  2. Clamp stage/queue sizes to >= 0 only when building statistics (the path Prometheus and the monitor agent consume).
  3. Clamp available_buffer_space_ratios to [0, 100] when counters overshoot total_limit_size, and treat total_limit_size == 0 as 0% free without dividing (no NaN / no dead NaN guard).

Docs Changes:
None

Release Note:

  • buffer: clamp exported buffer size metrics to non-negative values

General Checklist:

Tests:

  • test/plugin/test_buffer.rb #statistics: negative underlying gauges export as 0; overshoot clamps ratio to 0; total_limit_size == 0 does not raise and ratio stays finite.

**Which issue(s) this PR fixes**:
Fixes #5303

**What this PR does / why we need it**:
Buffer size gauges (`stage_byte_size`, `queue_byte_size`, and derived
`total_queued_size`) can go transiently negative when Fluentd core
under/over-subtracts during concurrent stage/queue transitions (the
deferred `@stage_size_metrics.add` after chunk unlock vs
`enqueue_chunk`'s `sub` — see #2712 / #2734). Those values are mirrored
by the Prometheus plugin as `fluentd_output_status_buffer_total_bytes` /
`fluentd_output_status_buffer_stage_byte_size`, which is what #5303
reports.

Clamping the **gauge store** on `sub`/`dec` is the wrong fix: it turns a
self-correcting transient negative into a **permanent over-count**, so
`Buffer#storable?` (which reads the raw gauge) eventually refuses every
write. Thanks @Watson1978 for catching that.

This PR takes an **export-only** approach:

1. Leave `LocalMetrics` gauge `sub`/`dec`/`set` semantics unchanged
(negatives still allowed so the deferred-add race can self-heal).
2. Clamp stage/queue sizes to `>= 0` only when building `statistics`
(the path Prometheus and the monitor agent consume).
3. Clamp `available_buffer_space_ratios` to `[0, 100]` when counters
overshoot `total_limit_size`, and treat `total_limit_size == 0` as 0%
free without dividing (no NaN / no dead NaN guard).

**Docs Changes**:
None

**Release Note**:
* buffer: clamp exported buffer size metrics to non-negative values

**General Checklist**:
- [x] I have read and followed
[CONTRIBUTING.md](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md)
- [x] tests pass (CI)
- [x] DCO signed-off

**Tests**:
- `test/plugin/test_buffer.rb` `#statistics`: negative underlying gauges
export as 0; overshoot clamps ratio to 0; `total_limit_size == 0` does
not raise and ratio stays finite.

---------

Signed-off-by: Vedant Madane <6527493+VedantMadane@users.noreply.github.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@Watson1978 Watson1978 added this to the v1.19.4 milestone Sep 6, 2026
@Watson1978
Watson1978 merged commit 0cf5f23 into v1.19 Sep 7, 2026
21 checks passed
@Watson1978
Watson1978 deleted the backport-to-v1.19/pr5467 branch September 7, 2026 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants