fix(ingress): eliminate 5s latency from HAProxy evidence routing#94
Merged
fix(ingress): eliminate 5s latency from HAProxy evidence routing#94
Conversation
The evidence-server routing used payload(0,0) to inspect request bytes. Per HAProxy docs, length=0 means "extract to end of buffer", which in TCP mode defers rule evaluation until the full inspect-delay (5s) expires — HAProxy cannot know when a raw TCP stream ends. This added a 5-second latency to every TLS connection, making the service appear broken for users. Fix: use payload(0,16) with a concrete byte count (16 = len of "HEAD /evidences", the longest prefix matched) and accept once req.len >= 16. After SSL termination a full TLS record is decrypted atomically, so these 16 bytes arrive instantly. Before: ~5.2s per request (TLS 0.2s + inspect-delay 5.0s) After: ~0.28s per request (TLS 0.2s + routing ~0.04s) Tested on a live Phala CVM with Route53 DNS — both normal requests and /evidences endpoint verified working. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
payload(0,0)deferring TCP content inspection until the fullinspect-delaytimer expirespayload(0,0)topayload(0,16)andreq.len gt 0toreq.len ge 16so HAProxy evaluates the evidence routing ACL as soon as 16 bytes arrive (enough for"HEAD /evidences")Root Cause
In TCP mode,
payload(0,0)with length 0 means "extract from offset 0 to the end of the buffer." Because HAProxy cannot know when a TCP stream ends, it waits for the fullinspect-delay(5s) before evaluating the rule — adding 5 seconds of latency to every proxied request, not just/evidences.The previous fix (commit 66abdd9) changed
WAIT_ENDtoreq.len gt 0, which correctly unblocked theacceptrule, but leftpayload(0,0)in the ACL conditions. Since theacceptrule fires first (any data present → accept → stop inspecting), thepayload(0,0)in the ACL lines was still evaluated and caused the same deferral.Fix
req.len ge 16: wait until at least 16 bytes are buffered (the longest prefix we match:"HEAD /evidences"= 16 chars), then accept — stops further inspection immediatelypayload(0,16): extract exactly 16 bytes from the start instead of the unboundedpayload(0,0), so the ACL can be evaluated without waiting for stream endTest Results
Verified on a live CVM deployment (Route53 + Phala gateway):
/evidencesroutingTest plan
haproxy -c/evidencesendpoint still routes correctly after fix🤖 Generated with Claude Code