Measured with benchmark/bench.sh -m idle at 6e847f2, holding connections
open rather than churning them, on four physical cores with the same certificate
and TLS parameters for all three subjects:
| connections held |
tlsproxy |
nginx stream |
haproxy tcp |
| 1,000 |
86 MB |
184 MB |
99 MB |
| 5,000 |
417 MB |
430 MB |
334 MB |
| 20,000 |
1,459 MB |
1,355 MB |
1,175 MB |
Taking the slope between 1k and 20k, so that the fixed footprint drops out, the
marginal cost of one held connection is 74 KB for tlsproxy, 63 KB for nginx
and 58 KB for haproxy. We are the most expensive of the three.
This only shows up when connections are held. A churn benchmark has
few live at any instant, and by that measure we looked several times better
than we are, so the honest number is this one.
The composition is worth separating, because the two halves point different
ways. Our fixed footprint is the smallest by a wide margin: 15 MB against
haproxy's 48 MB and nginx's 126 MB. It is the per-connection part that is
worst. 74 KB is roughly OpenSSL's read and write buffers, which default to
about 16 KB each plus overhead, on top of the two TPX_NET_BUFSIZE chunks in
the bufq pair.
SSL_MODE_RELEASE_BUFFERS tells OpenSSL to free the read and write buffers
whenever a connection has no data in flight, at the cost of reallocating them
when it next does. nginx sets it. For a proxy whose stated use is many
long-lived and mostly idle connections, an MQTT broker or a game server or an
RPC service, that trade is the right way round, since the buffers are idle
almost all the time.
Worth measuring rather than assuming, since it costs an allocation per wake-up
on a busy connection: bench.sh -m idle gives the memory side and bench.sh -m bulk and -m rate give whatever it costs the data path, so the trade is
directly observable before and after.
The other component is ours. Two 16 KB bufq chunks per proxy are allocated
whether or not the connection is carrying anything; allocating the first chunk
lazily on first use would take an idle pair down to the proxy_t itself.
Measured with
benchmark/bench.sh -m idleat6e847f2, holding connectionsopen rather than churning them, on four physical cores with the same certificate
and TLS parameters for all three subjects:
Taking the slope between 1k and 20k, so that the fixed footprint drops out, the
marginal cost of one held connection is 74 KB for tlsproxy, 63 KB for nginx
and 58 KB for haproxy. We are the most expensive of the three.
This only shows up when connections are held. A churn benchmark has
few live at any instant, and by that measure we looked several times better
than we are, so the honest number is this one.
The composition is worth separating, because the two halves point different
ways. Our fixed footprint is the smallest by a wide margin: 15 MB against
haproxy's 48 MB and nginx's 126 MB. It is the per-connection part that is
worst. 74 KB is roughly OpenSSL's read and write buffers, which default to
about 16 KB each plus overhead, on top of the two
TPX_NET_BUFSIZEchunks inthe
bufqpair.SSL_MODE_RELEASE_BUFFERStells OpenSSL to free the read and write bufferswhenever a connection has no data in flight, at the cost of reallocating them
when it next does. nginx sets it. For a proxy whose stated use is many
long-lived and mostly idle connections, an MQTT broker or a game server or an
RPC service, that trade is the right way round, since the buffers are idle
almost all the time.
Worth measuring rather than assuming, since it costs an allocation per wake-up
on a busy connection:
bench.sh -m idlegives the memory side andbench.sh -m bulkand-m rategive whatever it costs the data path, so the trade isdirectly observable before and after.
The other component is ours. Two 16 KB
bufqchunks per proxy are allocatedwhether or not the connection is carrying anything; allocating the first chunk
lazily on first use would take an idle pair down to the
proxy_titself.