grep -rn NODELAY src/ inc/ app/ finds nothing, and Linux leaves Nagle on by
default, so both the accepted client socket and the backend socket
create_connect() opens hold a small trailing segment until the peer
acknowledges what is already in flight. The peer's acknowledgement is itself
delayed, so the pair costs a delayed-ACK timer, which is 40 ms on Linux.
Measured with bench.sh -m message, 1000 connections held for the run, 200
byte messages each way against the echo sink, all three subjects on the same
Debian and libssl3, four workers on four physical cores pinned at base clock,
generator on the second host. Medians of three reps:
Every connection sending on a shared 33 ms tick, which is a 30 Hz game
server. Slots a connection was too late to use are counted as missed:
|
p50 |
p99 |
p999 |
max |
msg/s |
missed slots |
| tlsproxy |
0.43 ms |
42.08 ms |
57.88 ms |
84.0 ms |
29,316 |
4,534 |
| nginx stream |
0.44 ms |
0.86 ms |
78.94 ms |
142.0 ms |
29,664 |
2,087 |
| haproxy tcp |
0.43 ms |
0.88 ms |
53.88 ms |
71.8 ms |
29,688 |
1,292 |
The median is level with the other two and the p999 is better than nginx's, so
this is one distinct failure mode rather than general slowness: a little over 1%
of messages wait out a delayed-ACK timer, and a connection that waited 42 ms has
already missed its next 33 ms slot, which is where twice the missed slots come
from.
One message per second per connection, which is the MQTT and RPC case, does
not reproduce it, and there tlsproxy has the best tail of the three:
|
p50 |
p99 |
p999 |
max |
| tlsproxy |
0.33 ms |
1.25 ms |
57.94 ms |
61.8 ms |
| nginx stream |
0.35 ms |
5.83 ms |
112.12 ms |
133.4 ms |
| haproxy tcp |
0.34 ms |
3.05 ms |
69.68 ms |
78.0 ms |
The difference between the two is the regime rather than the load: a second of
silence leaves nothing unacknowledged for Nagle to wait on, while a message
every 33 ms can still have the previous one outstanding. A sequential
request/response probe over loopback puts the boundary at the message size as
well: at 1400 bytes, 34 of 400 round trips exceeded 5 ms and every one of them
landed between 40 and 47 ms, at 64 and 1024 bytes only the first message after
the handshake did, and at 8192 and 32768 bytes none did, which is what Nagle
predicts since it only holds a sub-MSS segment. nginx and haproxy show none at
any size.
Which of the two legs holds the segment is not established. Both lack the
option, and a capture on the bridge would say. The echo sink sets TCP_NODELAY
on the sockets it accepts, so the backend is not the one waiting.
TCP_NODELAY set on the listening socket is copied onto every fd accept()
returns, measured on 6.18.41 beside SO_KEEPALIVE, which behaves the same way,
so the client leg can be configured once in bind_listen_sock() where the
keepalive block already is and only the backend leg needs its own call in
create_connect().
None of the other four modes can see this. -m handshake and -m rate close
each connection as soon as the handshake completes, -m idle sends no
application data, and -m bulk runs with -n 0, streaming continuously so the
window is never small enough for Nagle to hold anything.
Release and Debug are affected identically, since no assert() is involved.
grep -rn NODELAY src/ inc/ app/finds nothing, and Linux leaves Nagle on bydefault, so both the accepted client socket and the backend socket
create_connect()opens hold a small trailing segment until the peeracknowledges what is already in flight. The peer's acknowledgement is itself
delayed, so the pair costs a delayed-ACK timer, which is 40 ms on Linux.
Measured with
bench.sh -m message, 1000 connections held for the run, 200byte messages each way against the echo sink, all three subjects on the same
Debian and libssl3, four workers on four physical cores pinned at base clock,
generator on the second host. Medians of three reps:
Every connection sending on a shared 33 ms tick, which is a 30 Hz game
server. Slots a connection was too late to use are counted as missed:
The median is level with the other two and the p999 is better than nginx's, so
this is one distinct failure mode rather than general slowness: a little over 1%
of messages wait out a delayed-ACK timer, and a connection that waited 42 ms has
already missed its next 33 ms slot, which is where twice the missed slots come
from.
One message per second per connection, which is the MQTT and RPC case, does
not reproduce it, and there tlsproxy has the best tail of the three:
The difference between the two is the regime rather than the load: a second of
silence leaves nothing unacknowledged for Nagle to wait on, while a message
every 33 ms can still have the previous one outstanding. A sequential
request/response probe over loopback puts the boundary at the message size as
well: at 1400 bytes, 34 of 400 round trips exceeded 5 ms and every one of them
landed between 40 and 47 ms, at 64 and 1024 bytes only the first message after
the handshake did, and at 8192 and 32768 bytes none did, which is what Nagle
predicts since it only holds a sub-MSS segment. nginx and haproxy show none at
any size.
Which of the two legs holds the segment is not established. Both lack the
option, and a capture on the bridge would say. The echo sink sets
TCP_NODELAYon the sockets it accepts, so the backend is not the one waiting.
TCP_NODELAYset on the listening socket is copied onto every fdaccept()returns, measured on 6.18.41 beside
SO_KEEPALIVE, which behaves the same way,so the client leg can be configured once in
bind_listen_sock()where thekeepalive block already is and only the backend leg needs its own call in
create_connect().None of the other four modes can see this.
-m handshakeand-m ratecloseeach connection as soon as the handshake completes,
-m idlesends noapplication data, and
-m bulkruns with-n 0, streaming continuously so thewindow is never small enough for Nagle to hold anything.
Release and Debug are affected identically, since no
assert()is involved.