Skip to content

Migration: add CDC resource utilization metrics for ClickPipes #330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mintlify wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Migration: add CDC resource utilization metrics for ClickPipes #330

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions integrations/clickpipes/monitoring.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,30 @@ ClickPipes_Latency{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickho
| `ClickPipes_Latency` | Gauge (_avg_) | Time in milliseconds between when a record was produced at the source and when it was written to ClickHouse. |
| `ClickPipes_SourceReplicationLatency_MiB` | Gauge (_avg_) | **CDC ClickPipes** Replication lag at the source in MiB. For Postgres ClickPipes, this reflects the replication slot lag. |

### Resource usage {#metrics-resources}
### Resource utilization {#metrics-resources}

#### Streaming and object storage ClickPipes {#metrics-resources-streaming-os}

`ClickPipes_Replica_*` metrics report resource utilization for streaming and object storage ClickPipes. Each ClickPipe has **dedicated compute**, so these values reflect utilization for individual ClickPipes.

| Metric | Type | Description |
|----------------------------------|----------------|-------------|
| `ClickPipes_Replica_CPUUsage` | Gauge (_avg_) | Average CPU usage of the ingestion replicas, in millicores. |
| `ClickPipes_Replica_CPULimit` | Gauge (_last_) | Configured CPU limit of the ingestion replicas, in millicores. |
| `ClickPipes_Replica_MemoryUsage` | Gauge (_avg_) | Average memory usage of the ingestion replicas, in bytes. |
| `ClickPipes_Replica_MemoryLimit` | Gauge (_last_) | Configured memory limit of the ingestion replicas, in bytes. |

#### CDC ClickPipes {#metrics-resources-cdc}

`ClickPipes_CDC_*` metrics report resource utilization for CDC ClickPipes. All CDC ClickPipes in the same service **share compute**, so these values reflect the combined usage of every CDC ClickPipe in the service.

| Metric | Type | Description |
|----------------------------------|----------------|-------------|
| `ClickPipes_Replica_CPUUsage` | Gauge (_avg_) | Average CPU usage of the ingestion replicas, in millicores. |
| `ClickPipes_Replica_CPULimit` | Gauge (_last_) | Configured CPU limit of the ingestion replicas, in millicores. |
| `ClickPipes_Replica_MemoryUsage` | Gauge (_avg_) | Average memory usage of the ingestion replicas, in bytes. |
| `ClickPipes_Replica_MemoryLimit` | Gauge (_last_) | Configured memory limit of the ingestion replicas, in bytes. |
| `ClickPipes_CDC_CPUUsage` | Gauge (_avg_) | Average CPU usage, in millicores. Shared across all CDC ClickPipes in the service. |
| `ClickPipes_CDC_CPULimit` | Gauge (_last_) | Configured CPU limit, in millicores. Shared across all CDC ClickPipes in the service. |
| `ClickPipes_CDC_MemoryUsage` | Gauge (_avg_) | Average memory usage, in bytes. Shared across all CDC ClickPipes in the service. |
| `ClickPipes_CDC_MemoryLimit` | Gauge (_last_) | Configured memory limit, in bytes. Shared across all CDC ClickPipes in the service. |
| `ClickPipes_CDC_NetworkReceiveBytes` | Gauge | Inbound network bandwidth, in bytes (delta per scrape window). Shared across all CDC ClickPipes in the service. |

### State and progress {#metrics-state}

Expand Down