Skip to content

fix(tail): gate truncation offset reset on prefix hash, not unconditionally - #297

Open
detail-app[bot] wants to merge 3 commits into
masterfrom
detail/bug-fix/fix-tail-gate-truncation-offset-reset-on-prefix-ha-7688c7
Open

detail-app[bot] wants to merge 3 commits into
masterfrom
detail/bug-fix/fix-tail-gate-truncation-offset-reset-on-prefix-ha-7688c7

Conversation

@detail-app

@detail-app detail-app Bot commented Sep 16, 2026

Copy link
Copy Markdown

Detail bug report: View on Detail

Fixes ENG-698

Bug

In processDirtyIndex (src/tail/watch.zig), a file shrinking below its tracked offset was unconditionally reset to 0 before maybeHandleContentRewrite ran. When the head-prefix hash was unchanged (a benign partial truncation that preserves the prefix, 0 < size < offset and size >= prefix_len), the hash check's early return left the offset at 0, so emitReadableRange re-emitted [0, size]. Under read_from=.head this duplicated already-delivered bytes; under the production default .tail it injected stale pre-existing content into the live stream; under .checkpoint the restored offset was discarded and the file replayed from byte 0. The unconditional reset pre-empted maybeHandleContentRewrite's prefix-gated offset decision, leaving its offset branches dead on every size < offset path. (Canonical logrotate copytruncate truncates to zero and was unaffected.)

Fix

Removed the unconditional reset and let maybeHandleContentRewrite own the offset decision for the no-checkpoint path:

  • For .head/.tail (no checkpoint lane), the prefix hash now gates the reset: unchanged prefix → advance offset to size and emit nothing; changed prefix → reset to 0 and re-emit. The size == 0 branch now also explicitly resets the offset, preserving copytruncate-to-zero behaviour the removed line used to handle.
  • For .checkpoint, the conservative reset + re-emit is retained. maybeHandleContentRewrite's freshly-recomputed prefix hash at resume cannot distinguish a benign truncation from an in-place rewrite, so a no-emit there would risk data loss (at-least-once violation) and would infinite-loop — applyCheckpointOffsetOne keeps resurrecting a stale offset above the file size, re-queueing the index forever. The emitted event lets the runtime reconcile the stale checkpoint. This gate is the key divergence from the naive "remove the line entirely" fix, which the regression test proves would hang the runner.

Branch A (size < prefix_len) and Branch C (prefix changed) are unchanged — the conservative reset is information-theoretically necessary when the prefix bytes are gone.

Testing

  • New unit tests in src/tail/watch.zig (inline, run via zig build test): the original .head repro, the .tail stale-injection case, copytruncate-to-zero-then-append (guards the new size==0 reset), and a .checkpoint resume case that asserts a single re-emission and no loop. All pass.
  • Routine checks pass: zig build, zig build test (Debug and ReleaseSafe — 523/524, 1 skip is an unrelated S3 e2e needing minio), zig fmt --check, and ziglint are clean.
  • End-to-end against the edge-tail binary: ran the partial-truncation/append/copytruncate-to-zero scenario with both the poll and uring (native Linux) io engines — no duplicate/stale lines, and fresh content streams from byte 0 after truncate-to-zero. A .tail run confirmed no stale pre-existing bytes are injected; a .checkpoint resume run confirmed exactly one re-emission of the surviving range with clean shutdown (no hang).
  • The kqueue backend is macOS-only and normalizes to poll on Linux; it shares the same markDirtyprocessDirtyIndexmaybeHandleContentRewrite path the fix changes, so it is covered by construction (could not run it live on this Linux host).
  • The bench/logging pytest integration harness was run for the rotation and checkpoint suites; results matched the baseline binary exactly (no regression). Some pre-existing timing-flaky tests (test_line_framing, test_error_handling) fail/hang on both baseline and fix and are unrelated to this change. The full integration suite includes long-running stress/benchmark tests that exceed the available runtime; the relevant rotation/checkpoint/functional suites were each run individually and match baseline.

Automatic Fixes PRs can be configured here.

Comment thread src/tail/watch.zig Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 16, 2026

Copy link
Copy Markdown

Approvability

Verdict: Approved at 82be1ac

Macroscope's review found this PR approvable — This is a narrowly scoped tail truncation bug fix with explicit handling for partial and zero-length truncation, checkpoint recovery, and asynchronous offset synchronization. The added regression tests cover the affected head, tail, and checkpoint paths without introducing a new capability or broader infrastructure change.

You can add or adjust custom eligibility rules. Learn more.

When a file is truncated but its first 64-byte prefix is unchanged,
resetting the offset to the new size would permanently skip any
rewritten data in the retained range. Reset to 0 instead so that
range is re-emitted.
Comment thread src/tail/watch.zig Outdated
…ion reset

When a checkpoint lane holds an offset above the live file size (file
was truncated after the checkpoint was saved), processDirtyIndex resets
offsets[i] to 0 and re-emits the surviving range conservatively.

Two bugs allowed the stale offset to be resurrected on the very next
collect call before the async checkpoint worker drained the new offset:

1. maybeHandleContentRewrite reset offsets[i] to 0 (not size) in the
   prefix-unchanged branch. For a partial truncation with no checkpoint
   lane, this caused already-delivered bytes to be re-emitted on every
   collect call. Fix: clamp to size so emitReadableRange finds nothing
   to deliver.

2. applyCheckpointOffsetOne only applies a checkpoint offset when it
   exceeds the current in-memory offset. After the reset+emit cycle,
   offsets[i]=size, but the stale checkpoint value (> size) was still
   in the lane's in-memory store, so the next applyCheckpointOffsetOne
   call resurrected it and triggered another reset+emit. Fix: call
   lane.resetOffset(id, 0) synchronously after the in-memory reset so
   getOffset returns the new value immediately without waiting for the
   async worker.

The second collect assertion added to the checkpoint resume test
exercises the resurrection scenario directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant