feat(frontend): bound stdio inbound reads and report every drop path

[jaronoff97]

Stack

	PR	What
1	#322 (this one)	stdio inbound deadlines, and a log plus a status on every drop path
2	#323	the fault matrix, telemetry expectations, and the fixes it found

Merge bottom to top.

Why

An ECS customer sees tero-edge marked UNHEALTHY for ~90 s at a time: flat CPU, flat memory, logs still flowing, and the task recovers on its own. Two defects explain the class, and neither was visible from outside the process.

Dispatch (httpz). httpz batches up to 16 ready requests and hands the whole batch to one pool thread. No other thread drains that queue, and stealing is one hop, so half the queues have no stealer at all. Measured on a 16-core host against a 3 s upstream: a /_health probe in that batch waited 3.0 s to 15.6 s with 15 requests in flight and 128 idle handler threads. The thread count is not the lever — 16 threads and 128 threads give the same distribution. bench/health_hol_repro.py reproduces it.

Inbound reads (stdio). std.Io.net.Stream.Reader never times out. 300 idle sockets took every connection slot and /_health never answered again, while httpz reclaimed all 256 slots within 15 s. bench/idle_conn_check.py reproduces it.

This PR fixes the second defect and makes the first one diagnosable. The dispatch fix needs a patch to httpz itself, which is a separate change.

What

src/frontend/stdio/deadline_reader.zig reads through the net_receive operation under Io.operateTimeout. On POSIX that is a non-blocking recvmsg plus poll with the deadline: no extra task, no unit of concurrency, and no std.posix outside core/io_select.zig. Two deadlines:

deadline	bounds	httpz equivalent
idle	one read while no request is in flight	timeout.keepalive
request	a whole request from its first byte	none — SO_RCVTIMEO restarts per read

Every drop path now logs and answers. Load shed 503, unparseable head 400, inbound deadline 408, a response that fails after its head is on the wire truncates with a warn, and a fixed error response we could not deliver is logged instead of ignored.

New series. edge_requests_in_flight, edge_upstream_timeouts_total, edge_connections_max on both frontends; edge_connections_total, edge_connections_active, edge_connections_shed_total{reason}, edge_inbound_timeouts_total{phase}, edge_requests_invalid_total on stdio. The connection series are gated to stdio because httpz gives the handler no accept hook, and a flat zero would read as "no connections" rather than "not measured". Duration buckets now reach 60 s; the old 5 s ceiling could not show a 30 s stall.

Verification

Idle socket wedge, 300 idle sockets against max_connections 256:

before   after 45 s   health NO 200   (never recovered)
after    after 30 s   health 1 ms     edge_inbound_timeouts_total{phase="idle"} 256

Stalled partial request:

after 30.0 s the sender got: HTTP/1.1 408 Request Timeout
[ERROR] request.failed method="POST" path="/api/v2/logs" err="InboundBodyTimeout"
[WARN]  request.slow  method="POST" path="/api/v2/logs" status=408 duration_ms=30002.44

Throughput: small bodies cost 1.8% (90.2k → 88.6k rps); 1 MB bodies gain 10% (3885 → 4287 rps) with p99.9 down from 35.9 ms to 21.0 ms.

zig build test passes on both frontends (527/528, 1 skipped), plus ziglint and zig fmt --check. Three new tests cover the reader over real loopback TCP.

Known gaps, unchanged here

• The upstream dial has no deadline; std.http.Client takes none.
• The write side has no deadline; std.Io 0.16 has no net_send operation. A client that stops reading still holds its slot. httpz has the same gap.
• All measurements are Darwin. Nothing has run on Linux yet.

🤖 Generated with Claude Code

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — The new stdio deadline reader changes default handling of existing connections and stalled requests, including connection reclamation and 408 responses. The accompanying telemetry spans both frontends and adds multiple new production metrics, making this broader than a small, isolated fix.

Not approved because:

• Credit balance exhausted. Approvability relies on correctness review in order to determine eligibility

^{Review your spending limits in Billing settings. You can add or adjust custom eligibility rules. Learn more.}