Skip to content

Batch otel logs and disable daemon cluster metrics - #25

Open
ArnobKumarSaha wants to merge 1 commit into
masterfrom
arnob-otel-batch
Open

ArnobKumarSaha wants to merge 1 commit into
masterfrom
arnob-otel-batch

Conversation

@ArnobKumarSaha

@ArnobKumarSaha ArnobKumarSaha commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor
  • Gateway logs pipeline had no batch processor, so every small log chunk became its own ClickHouse INSERT (~2.5 inserts/s of ~7 rows on telecash-dbaas). This caused constant part creation and merges. Add batch/logs (5000 / 10000 / 5s).
  • The daemon inherited presets.clusterMetrics: true from the opentelemetry-kube-stack defaults. k8s_cluster metrics carry pod/namespace only as resource attributes, and prometheusremotewrite drops those, so every k8s_* metric collapsed to one series. Thanos then rejected the rest with duplicate sample for timestamp (~40 errors/min). kube-state-metrics already provides these metrics, so disable the preset.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant