Skip to content

fix(ingress): eliminate 5s latency from HAProxy evidence routing#94

Merged
h4x3rotab merged 1 commit intomainfrom
debug/route53-ssl-handshake
Apr 15, 2026
Merged

fix(ingress): eliminate 5s latency from HAProxy evidence routing#94
h4x3rotab merged 1 commit intomainfrom
debug/route53-ssl-handshake

Conversation

@h4x3rotab
Copy link
Copy Markdown
Contributor

Summary

  • Fix 5-second per-request latency caused by HAProxy's payload(0,0) deferring TCP content inspection until the full inspect-delay timer expires
  • Change payload(0,0) to payload(0,16) and req.len gt 0 to req.len ge 16 so HAProxy evaluates the evidence routing ACL as soon as 16 bytes arrive (enough for "HEAD /evidences")

Root Cause

In TCP mode, payload(0,0) with length 0 means "extract from offset 0 to the end of the buffer." Because HAProxy cannot know when a TCP stream ends, it waits for the full inspect-delay (5s) before evaluating the rule — adding 5 seconds of latency to every proxied request, not just /evidences.

The previous fix (commit 66abdd9) changed WAIT_END to req.len gt 0, which correctly unblocked the accept rule, but left payload(0,0) in the ACL conditions. Since the accept rule fires first (any data present → accept → stop inspecting), the payload(0,0) in the ACL lines was still evaluated and caused the same deferral.

Fix

-    tcp-request content accept if { req.len gt 0 }
-    acl is_evidence payload(0,0) -m beg "GET /evidences"
-    acl is_evidence payload(0,0) -m beg "HEAD /evidences"
+    tcp-request content accept if { req.len ge 16 }
+    acl is_evidence payload(0,16) -m beg "GET /evidences"
+    acl is_evidence payload(0,16) -m beg "HEAD /evidences"
  • req.len ge 16: wait until at least 16 bytes are buffered (the longest prefix we match: "HEAD /evidences" = 16 chars), then accept — stops further inspection immediately
  • payload(0,16): extract exactly 16 bytes from the start instead of the unbounded payload(0,0), so the ACL can be evaluated without waiting for stream end

Test Results

Verified on a live CVM deployment (Route53 + Phala gateway):

Metric Before After
Time to first byte ~5.2s ~0.28s
/evidences routing Working Working

Test plan

  • HAProxy config syntax validated with haproxy -c
  • Deployed to live CVM and confirmed latency drop (5.2s → 0.28s)
  • Verified /evidences endpoint still routes correctly after fix
  • Tested with both single-domain and evidence server modes

🤖 Generated with Claude Code

The evidence-server routing used payload(0,0) to inspect request bytes.
Per HAProxy docs, length=0 means "extract to end of buffer", which in
TCP mode defers rule evaluation until the full inspect-delay (5s)
expires — HAProxy cannot know when a raw TCP stream ends.

This added a 5-second latency to every TLS connection, making the
service appear broken for users.

Fix: use payload(0,16) with a concrete byte count (16 = len of
"HEAD /evidences", the longest prefix matched) and accept once
req.len >= 16.  After SSL termination a full TLS record is decrypted
atomically, so these 16 bytes arrive instantly.

Before: ~5.2s per request  (TLS 0.2s + inspect-delay 5.0s)
After:  ~0.28s per request (TLS 0.2s + routing ~0.04s)

Tested on a live Phala CVM with Route53 DNS — both normal requests and
/evidences endpoint verified working.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@h4x3rotab h4x3rotab merged commit ea4c114 into main Apr 15, 2026
6 checks passed
@h4x3rotab h4x3rotab deleted the debug/route53-ssl-handshake branch April 15, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant