fix(tail): gate truncation offset reset on prefix hash, not unconditionally - #297
Open
detail-app[bot] wants to merge 3 commits into
Open
detail-app[bot] wants to merge 3 commits into
detail-app[bot] wants to merge 3 commits into
Conversation
ApprovabilityVerdict: Approved at Macroscope's review found this PR approvable — This is a narrowly scoped tail truncation bug fix with explicit handling for partial and zero-length truncation, checkpoint recovery, and asynchronous offset synchronization. The added regression tests cover the affected head, tail, and checkpoint paths without introducing a new capability or broader infrastructure change. You can add or adjust custom eligibility rules. Learn more. |
When a file is truncated but its first 64-byte prefix is unchanged, resetting the offset to the new size would permanently skip any rewritten data in the retained range. Reset to 0 instead so that range is re-emitted.
…ion reset When a checkpoint lane holds an offset above the live file size (file was truncated after the checkpoint was saved), processDirtyIndex resets offsets[i] to 0 and re-emits the surviving range conservatively. Two bugs allowed the stale offset to be resurrected on the very next collect call before the async checkpoint worker drained the new offset: 1. maybeHandleContentRewrite reset offsets[i] to 0 (not size) in the prefix-unchanged branch. For a partial truncation with no checkpoint lane, this caused already-delivered bytes to be re-emitted on every collect call. Fix: clamp to size so emitReadableRange finds nothing to deliver. 2. applyCheckpointOffsetOne only applies a checkpoint offset when it exceeds the current in-memory offset. After the reset+emit cycle, offsets[i]=size, but the stale checkpoint value (> size) was still in the lane's in-memory store, so the next applyCheckpointOffsetOne call resurrected it and triggered another reset+emit. Fix: call lane.resetOffset(id, 0) synchronously after the in-memory reset so getOffset returns the new value immediately without waiting for the async worker. The second collect assertion added to the checkpoint resume test exercises the resurrection scenario directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detail bug report: View on Detail
Fixes ENG-698
Bug
In
processDirtyIndex(src/tail/watch.zig), a file shrinking below its tracked offset was unconditionally reset to0beforemaybeHandleContentRewriteran. When the head-prefix hash was unchanged (a benign partial truncation that preserves the prefix,0 < size < offsetandsize >= prefix_len), the hash check's early return left the offset at0, soemitReadableRangere-emitted[0, size]. Underread_from=.headthis duplicated already-delivered bytes; under the production default.tailit injected stale pre-existing content into the live stream; under.checkpointthe restored offset was discarded and the file replayed from byte 0. The unconditional reset pre-emptedmaybeHandleContentRewrite's prefix-gated offset decision, leaving its offset branches dead on everysize < offsetpath. (Canonicallogrotate copytruncatetruncates to zero and was unaffected.)Fix
Removed the unconditional reset and let
maybeHandleContentRewriteown the offset decision for the no-checkpoint path:.head/.tail(no checkpoint lane), the prefix hash now gates the reset: unchanged prefix → advanceoffsettosizeand emit nothing; changed prefix → reset to0and re-emit. Thesize == 0branch now also explicitly resets the offset, preserving copytruncate-to-zero behaviour the removed line used to handle..checkpoint, the conservative reset + re-emit is retained.maybeHandleContentRewrite's freshly-recomputed prefix hash at resume cannot distinguish a benign truncation from an in-place rewrite, so a no-emit there would risk data loss (at-least-once violation) and would infinite-loop —applyCheckpointOffsetOnekeeps resurrecting a stale offset above the file size, re-queueing the index forever. The emitted event lets the runtime reconcile the stale checkpoint. This gate is the key divergence from the naive "remove the line entirely" fix, which the regression test proves would hang the runner.Branch A (
size < prefix_len) and Branch C (prefix changed) are unchanged — the conservative reset is information-theoretically necessary when the prefix bytes are gone.Testing
src/tail/watch.zig(inline, run viazig build test): the original.headrepro, the.tailstale-injection case, copytruncate-to-zero-then-append (guards the newsize==0reset), and a.checkpointresume case that asserts a single re-emission and no loop. All pass.zig build,zig build test(Debug and ReleaseSafe — 523/524, 1 skip is an unrelated S3 e2e needing minio),zig fmt --check, andziglintare clean.edge-tailbinary: ran the partial-truncation/append/copytruncate-to-zero scenario with both thepollanduring(native Linux) io engines — no duplicate/stale lines, and fresh content streams from byte 0 after truncate-to-zero. A.tailrun confirmed no stale pre-existing bytes are injected; a.checkpointresume run confirmed exactly one re-emission of the surviving range with clean shutdown (no hang).kqueuebackend is macOS-only and normalizes topollon Linux; it shares the samemarkDirty→processDirtyIndex→maybeHandleContentRewritepath the fix changes, so it is covered by construction (could not run it live on this Linux host).bench/loggingpytest integration harness was run for the rotation and checkpoint suites; results matched the baseline binary exactly (no regression). Some pre-existing timing-flaky tests (test_line_framing,test_error_handling) fail/hang on both baseline and fix and are unrelated to this change. The full integration suite includes long-running stress/benchmark tests that exceed the available runtime; the relevant rotation/checkpoint/functional suites were each run individually and match baseline.Automatic Fixes PRs can be configured here.