Skip to content

fix(tail): pre-size scratch before non-fixed io_uring read SQEs - #315

Open
detail-app[bot] wants to merge 3 commits into
masterfrom
detail/bug-fix/fix-tail-pre-size-scratch-before-non-fixed-io-urin-a8ef0e
Open

detail-app[bot] wants to merge 3 commits into
masterfrom
detail/bug-fix/fix-tail-pre-size-scratch-before-non-fixed-io-urin-a8ef0e

Conversation

@detail-app

@detail-app detail-app Bot commented Sep 16, 2026

Copy link
Copy Markdown

Detail bug report: View on Detail

Fixes ENG-716https://linear.app/usetero/issue/ENG-716/detail-bug-tail-io-uring-non-fixed-multi-read-batches-can-emit

Bug

In the io_uring Scheduler.processBatch non-fixed fallback (src/tail/read_scheduler/uring_linux.zig), self.scratch was grown per event inside the dispatch loop while self.ring.read had already pinned each event's slot pointer into sqe.addr. Reads are not submitted until submit_and_wait runs after the loop, so a mid-loop ArrayList relocation freed the backing buffer that earlier SQEs still referenced. The kernel then read file bytes into freed memory while the completion handler sliced the relocated scratch, feeding wrong/zero bytes to the framer — silent wrong output on the daemon's primary data path. This fallback runs on every cycle when fixed-buffer/fixed-file registration is unavailable (kernels lacking register_files_sparse, low RLIMIT_NOFILE, or ENOMEM). Introduced in a32652c (PR #148).

Fix

Pre-size self.scratch once for the whole batch before preparing any SQEs: a preview pass sums the per-event read sizes (mirroring the dispatch loop's skip/cap logic), then per-event slots are taken with addManyAsSliceAssumeCapacity, which only advances items.len and never reallocates. Every sqe.addr now stays live and at a stable address until its CQE is reaped — the stability the fixed path already had incidentally. The Debug assert in addManyAsSliceAssumeCapacity guards that the preview and dispatch passes agree.

Testing

  • Routine checks: zig fmt --check, ziglint, zig build, and zig build test (Debug and ReleaseSafe) all pass — 520 passed, 1 skipped (pre-existing s3-e2e), 0 failed.
  • Regression test (committed): added read scheduler uring non-fixed path: multi-event batch keeps read buffers stable, backed by a MovingAllocator that forces every grow to relocate while retaining freed memory, deterministically exercising the invariant regardless of allocator size-class luck. Verified it passes with the fix and fails when the bug is restored (the completion handler reads relocated/poison bytes instead of the file's bytes) in both Debug and ReleaseSafe — the production c_allocator config where the wrong-output symptom manifests.
  • End-to-end: ran the ReleaseSafe edge-tail binary against two growing files with --io-engine uring, appending distinct NDJSON lines across poll cycles; confirmed byte-exact emission on both the fixed path (default rlimit) and the non-fixed path, the latter forced via ulimit -n 96 (a probe confirmed this makes register_files_sparse fail with UserFdQuotaExceeded, flipping fixed_enabled = false — the exact production fallback the bug lives in). An initial e2e attempt produced no output because lines were written before the daemon started (--read-from defaults to tail/EOF) under an rlimit too low for the daemon to open its files; re-run with post-start appends and a sufficient rlimit passed.
  • Stress and edge coverage (run as throwaway tests, then removed): a 256-event × 64 KiB batch (16 MiB scratch with repeated forced relocations) completed with no addManyAsSliceAssumeCapacity assert trips; edge cases for zero-range/no-op events (0 ops, early return), batches exceeding the 256-slot ring cap (capped correctly), and events larger than read_buf (remainder funnelled through the scalar tail-continuation path) were all byte-exact in both build modes.

Automatic Fixes PRs can be configured here.

Comment thread src/tail/read_scheduler.zig Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 16, 2026

Copy link
Copy Markdown

Approvability

Verdict: Approved at 4be95de

Macroscope's review found this PR approvable — This is a narrowly scoped io_uring bug fix that stabilizes non-fixed read buffers without changing fixed or scalar paths. A forced-relocation regression test verifies byte-exact output and confirms the actual CQE path was exercised.

You can add or adjust custom eligibility rules. Learn more.

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 16, 2026
Comment thread src/tail/read_scheduler.zig
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant