Skip to content

Kafka Connect: recover from metadata commit stall after Kafka cluster recreation#16360

Open
Schiewatir wants to merge 2 commits into
apache:mainfrom
Schiewatir:fix/kafka-connect-cluster-recreation-metadata-commit
Open

Kafka Connect: recover from metadata commit stall after Kafka cluster recreation#16360
Schiewatir wants to merge 2 commits into
apache:mainfrom
Schiewatir:fix/kafka-connect-cluster-recreation-metadata-commit

Conversation

@Schiewatir
Copy link
Copy Markdown

Problem

After a Kafka cluster is recreated (old cluster deleted, new one started), the Apache Iceberg Kafka Connect sink stops committing metadata. Parquet files accumulate in object storage but no snapshots are created, and the logs show "Coordinator … found nothing to commit to table" on every commit cycle. (Reported in #15293.)

Root cause: The control topic resets to offset 0 on the new cluster, but the Iceberg snapshot still stores committed offsets from the old cluster (e.g. {"0":100}). The deduplication filter in Coordinator.commitToTable() rejects every DataWritten event whose offset is below the stored baseline, so all data files are silently dropped.

// before fix – stale minOffset = 100, envelope.offset() = 0..4 → all filtered
Long minOffset = committedOffsets.get(envelope.partition());
return minOffset == null || envelope.offset() >= minOffset;

Fix

When a partition's current control-topic offset is lower than its stored committed offset, the committed offset for that partition is reset to the current offset minus one (so the next write is accepted). This is detected per-partition during commitToTable(), so there is no behaviour change for the normal (non-reset) path.

After the reset, the filter accepts the new events, a snapshot is committed, and its stored offset is reset to the new baseline — replication resumes.

Fixes #15293

…ation

After a Kafka cluster is recreated the control topic resets to offset 0,
but the Iceberg snapshot still stores committed offsets from the old cluster
(e.g. {0:100}).  Every DataWritten event on the new cluster carries a low
offset (< 100) and the deduplication filter in commitToTable() silently
discards all of them, causing the coordinator to log "nothing to commit"
indefinitely while parquet files accumulate without a snapshot.

Detect this scenario by comparing the coordinator's observed control topic
offsets against the stored committed offsets.  When the current offsets are
lower, the control topic was likely reset; skip deduplication and store the
new (lower) baseline so subsequent commits behave correctly.

Fixes apache#15293
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metadata is not committed after Kafka cluster recreation (parquiet files are written successfully)

1 participant