fix(pipeline): do not treat a 0 stream() return as EOF in streamReaderToWriter

[detail-app]

Detail bug report: View on Detail

Fixes ENG-702

Bug

streamReaderToWriter (src/pipeline/pipeline.zig) — the bounded reader→writer copy used by exec.processBuffered to decode a compressed client body into an arena buffer for policy evaluation — treated a 0 return from Reader.stream as end-of-stream (if (bytes == 0) break;). This violates the std.Io.Reader.VTable contract: a 0 return "does not indicate end of stream" — only error.EndOfStream is EOF.

The std gzip and zstd decompressors select an indirect reader vtable when given a non-zero decode buffer (the production path always allocates one). That vtable's stream decodes a block into the reader's internal buffer and unconditionally returns 0; the bytes appear on the next stream call. So the loop broke immediately with total_bytes == 0, the trailing excess probe then read the buffered byte, and the function spuriously returned error.BodyTooLarge for every non-empty gzip/zstd body on the buffered path.

The bug is deterministic. Affected routes are the JSON variants that route to pipe_buffered (datadog /api/v2/series metrics JSON and OTLP JSON /v1/logs|metrics|traces), but only when policies are configured for the signal (the policiesActiveFor gate). On httpz, the spurious error surfaces to the client as 413; on stdio, it's caught by the existing fail-open branch which forwards the raw compressed body unevaluated (a silent policy bypass). The existing unit tests missed it because they used std.Io.Reader.fixed, whose vtable never returns 0 mid-stream.

Fix

Only terminate the loop on error.EndOfStream (now return total_bytes directly — EOF needs no excess probe). A 0 return no longer breaks the loop; it loops again to drain the reader's freshly-filled internal buffer. The trailing readSliceShort excess probe is retained for the total_bytes >= max_bytes exit, so the max_bytes bound is still enforced (genuine oversize still returns error.BodyTooLarge). This mirrors the correct pattern already used by encoding.zig's decodeAll.

Testing

• Added regression tests against a real encoding.Decoder reader (the uncovered production path) rather than std.Io.Reader.fixed: gzip and zstd each copy a 10 KB body verbatim with max_bytes well above the body, and a gzip oversize case confirms error.BodyTooLarge is still returned when the decoded body genuinely exceeds the bound. The decompressor-path tests fail with BodyTooLarge at the excess probe when the buggy if (bytes == 0) break; line is temporarily restored, confirming causation.
• Unit tests, zig fmt --check, ziglint, zig build, and zig build test -Doptimize=ReleaseSafe all pass (522/523 with the 1 pre-existing unrelated S3 e2e skip; no regressions).
• End-to-end on the wire, with a KEEP-all metric policy loaded so policiesActiveFor(.metric) is true: gzip and zstd compressed Datadog v2 Series JSON POSTs to /api/v2/series return 202 (was 413) and reach a local echo upstream; with max_decoded_bytes set below the decoded body the request still returns 413 (bound preserved); with no policy_providers the raw body passes through unchanged (173 raw bytes); and a corrupt gzip body still fails open with ReadFailed (not BodyTooLarge). The same compressed+policy request against the stdio frontend returns 202 with no buffered transform failed open warn and forwards a re-encoded body (161 bytes, not the raw 173), confirming valid compressed bodies no longer trigger the spurious fail-open bypass.

---

Automatic Fixes PRs can be configured here.

[macroscopeapp]

Approvability

Verdict: Approved at 42ac225

Macroscope's review found this PR approvable — This is a small, self-contained bug fix correcting zero-byte stream handling for compressed bodies while preserving oversized-body rejection. Regression tests cover the affected gzip/zstd paths and the existing size limit.

^{You can add or adjust custom eligibility rules. Learn more.}