test(matrix): fault suite for both frontends, and the defects it found

[jaronoff97]

Stack

	PR	What
1	#322	stdio inbound deadlines, and a log plus a status on every drop path
2	#323 (this one)	the fault matrix, telemetry expectations, and the fixes it found

Merge bottom to top. Based on stdio-deadlines-observability, so review #322 first.

What

A fault matrix under bench/matrix/: faults injected on both sides of the edge — the agent that sends to us, and the intake we send to — with every case run against both frontends.

Neither frontend is the oracle. My first draft treated httpz as correct and marked every stdio difference as acceptable, which hid four httpz defects. Each case now asserts what a customer needs, and both frontends are judged against it. Behaviour we have already found is declared, not excused:

class Pipelined(MatrixCase):
    DEFECTS = {"httpz": "answers 400 to a pipelined pair"}

The assertion still runs at full strength. The runner reports xfail with the note, and XPASS the day it starts passing, which means the note must go.

The intake is the real echo-server, which gains ten fault modes armed over HTTP (POST /fault?mode=&arg=&count=): status, slow, hang, close_early, reject_early, reset, garbage, truncate, oversize, stale_keepalive. count faults only the next N requests, so a case can break the first attempt and require the retry to succeed.

34 cases, one per file, one class per case, over three surfaces: sender framing and encodings (a*), the upstream leg (b*), and capacity (c*). Every case also asserts the invariants that hold whatever the fault was: the process lives, health still answers, connection slots return to baseline, and every 5xx left a log line.

$ ./bin/uv run --python ./bin/python3 --with requests --with zstandard bench/matrix/run.py
case                                                     stdio     httpz
a04_slow_drip_body.test_a_dribbling_sender_is_cut_off    pass      xfail
a09_corrupt_gzip.test_..._cannot_decode_fails_open       xfail     xfail
a12_pipelined.test_pipelined_requests_are_both_served    pass      xfail
b09_stale_keepalive.test_a_dead_pooled_connection...     pass      pass
c02_health_under_burst.test_health_stays_fast...         pass      xfail
34 case(s), 0 failure(s), 9 known defect(s)

Defects found and fixed here

1. stdio never retried any batch. Every body with a Content-Length was streamed, and a stream cannot be replayed, so one stale pooled connection lost it. Log intake clients do not retry, so this was silent data loss. Bodies below the streaming threshold are now resident, which is where httpz already drew the line. Caught by b04, b09, b10.
2. A header flood answered 502, sending an agent into a retry loop it could never win. Our cap and the sender's fault, so it answers 431 now. Caught by a14.
3. A retried dial was never counted, so a dial storm was invisible in edge_upstream_retries_total. Caught by b01.

Defects found and still open

Both frontends:

• An undecodable body is dropped when policies are loaded, though service/datadog documents fail-open. The edge becomes the reason the data disappears. (a09)
• A truncated intake response is reported as 202. An agent that reads 202 deletes its copy. (b11)
• An unsupported content encoding drops the batch: httpz answers 502 and blames the intake, stdio answers 400. (a10)

httpz only:

• A batch of up to 16 requests goes to one pool thread: a health probe that arrives with a burst waited 6.0 s against a 3 s intake. This is the reported ECS failure. (c02)
• Pipelined requests refused with 400. (a12)
• Headers above the cap dropped in silence, request still answers 202. (a14)
• A stalled sender gets no status: partial head, short body, dribbled body all close with nothing. (a02, a03, a04)

stdio only:

• A broken chunk waits for the request deadline instead of being refused at once, so bad framing holds a slot for 30 s. (a07)

IMPLEMENTATION-PROGRESS.md carries the full list and the task breakdown.

Notes

• zig-out-stdio/ is now in .gitignore; the previous commit captured its 3.2 MB binary, and this one untracks it.
• Python comes from hermit (3.12.3); requests and zstandard come from uv run --with. oha stays a brew dependency, as bench/perf/run.sh already assumes, and the load cases skip without it.
• zstd batches pass on both frontends, which matters because current Datadog agents send them.

🤖 Generated with Claude Code

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — Although the PR adds a valuable development-only fault matrix, it also changes production routing, retry and buffering semantics, overload handling, policy failure behavior, response relaying, and shutdown across both frontends. Its 101-file, 4,499-line scope and cross-component runtime impact require human review.

Not approved because:

• Credit balance exhausted. Approvability relies on correctness review in order to determine eligibility

^{Review your spending limits in Billing settings. You can add or adjust custom eligibility rules. Learn more.}