OTEL cumulative counters with labels cause OTLP payload to grow over time — what is the recommended mitigation?

[abhinavkumar9962]

Hi OTEL team,

I’m using the OpenTelemetry C++ SDK to export metrics in OTLP format to a time-series backend (VictoriaMetrics via vmagent).

I understand and agree with the OTEL metrics semantics, but I’m facing an operational issue and wanted to confirm the recommended industry-standard approach.

Setup

• SDK: OpenTelemetry C++
• Exporter: OTLP (push-based)
• Metric type: Cumulative counters
• Collection interval: periodic
• Metrics count: ~16k metric names
• Metrics have labels, some of which are high-cardinality (e.g., integer IDs)

Observed Behavior

With cumulative counters:

• Once a (metric + labelset) appears even once, it is exported at every collection interval
• This happens even when the counter value does not change
• Over time, as more label values appear, the OTLP payload size keeps increasing
• Eventually, large payloads cause ingestion-side failures (e.g., request size limits)

Example pattern:

REQUEST_COUNT{id=1234}

If id=1234 occurs once, that series is exported for the entire lifetime of the process, even if no further requests occur for that ID.

Understanding (to confirm)

I believe this is expected and correct behavior because:

• Cumulative counters must be continuously exported for correctness
• Skipping unchanged values would break rate() / increase() semantics
• Absence of samples is semantically different from a value of zero
• So I’m not looking to drop unchanged counters, as I understand that would violate the spec.

Problem Statement

In a push-based OTLP pipeline, cumulative counters with labels cause the export payload to grow monotonically over time, driven by label cardinality rather than actual activity.

This becomes a practical problem in large systems where:

• Labels may appear transiently
• Payload size eventually exceeds ingestion limits
• Increasing request size limits only delays the issue

Questions

1. Is this a known and expected trade-off of cumulative counters in push-based OTLP pipelines?
2. What is the recommended way to control OTLP payload size over long-running processes?
   3.Are there best practices from OTEL around:
   • metric modeling
   • label cardinality
   • export interval tuning
3. Can we configure OTEL to partition the payload and send in chunks?

I’m looking for guidance on how this problem is typically handled in production systems using OTEL.

[lalitb]

Is this a known and expected trade-off of cumulative counters in push-based OTLP pipelines?

Yeah, this is expected - you're hitting a known trade-off that the OTEL spec actually calls out explicitly.
Cumulative counters in a push pipeline will keep exporting every series you've ever seen. That's by design. The spec basically says "if you want cumulative + high cardinality, you're signing up for this pain."

Can we configure OTEL to partition the payload and send in chunks?

There's no "max payload size" setting in the SDKs - and it's not in the OTEL spec either. This just isn't something OTEL provides.

I’m looking for guidance on how this problem is typically handled in production systems using OTEL.

• Fix your metric modeling. Don't put IDs in metric attributes. That id=1234 label is the root cause - high-cardinality attributes belong in traces/logs, not metrics.
• Use cardinality limits. The otel-cpp default to 2000 series per metric with overflow bucketing. Tune this via Views if needed.
• Drop dimensions via Views. You can explicitly exclude id from your metrics while keeping other labels.
• Switch to Delta temporality. This lets you stop exporting inactive series. Convert back to cumulative downstream in a Collector or backend if needed.