Measured with benchmark/bench.sh -m bulk at 6e847f2: fixed concurrency over
persistent connections against an echo sink, same host on disjoint cpusets
because a gigabit link is far below what any of the three can move. Normalized
per core, since the subjects do not all draw the same CPU:
| payload |
tlsproxy |
nginx stream |
haproxy tcp |
| 64 B |
7 |
4 |
4 |
| 1 KB |
62 |
65 |
52 |
| 8 KB |
293 |
317 |
284 |
| 64 KB |
555 |
618 |
527 |
| 1 MB |
555 |
668 |
554 |
MB/s per core. We lead comfortably at 64 B and are level in the middle, so the
small-record path is in good shape. The interesting part is the top end: we
plateau at 555 MB/s per core from 64 KB upward and do not improve at all when
the payload goes to 1 MB, while nginx keeps climbing to 668.
haproxy plateaus at 554, within noise of us. Both of us use 16 KB buffers by
default, TPX_NET_BUFSIZE in inc/proxy.h and tune.bufsize in haproxy's
case, and nginx does not stall in the same place. That is circumstantial rather
than proof, but it is the obvious first thing to test: every payload above 16 KB
is being split into chunks, and whatever the per-chunk cost is, it is what stops
larger payloads paying off.
The experiment is cheap. Rebuild with TPX_NET_BUFSIZE at 32 KB and 64 KB and
re-run bench.sh -m bulk -x "64 1024 8192 65536 1048576". If the plateau moves
up with the constant, the cause is confirmed and the size becomes a trade
against the idle memory in the sibling issue, since the same constant sets how
much an idle connection holds. If it does not move, the cost is per-iteration in
the read/write path rather than per-chunk and the constant is innocent.
Note that the two issues pull in opposite directions: a larger chunk helps bulk
and hurts idle memory. That argues for making the size adaptive or the first
chunk lazy rather than simply raising the constant.
Measured with
benchmark/bench.sh -m bulkat6e847f2: fixed concurrency overpersistent connections against an echo sink, same host on disjoint cpusets
because a gigabit link is far below what any of the three can move. Normalized
per core, since the subjects do not all draw the same CPU:
MB/s per core. We lead comfortably at 64 B and are level in the middle, so the
small-record path is in good shape. The interesting part is the top end: we
plateau at 555 MB/s per core from 64 KB upward and do not improve at all when
the payload goes to 1 MB, while nginx keeps climbing to 668.
haproxy plateaus at 554, within noise of us. Both of us use 16 KB buffers by
default,
TPX_NET_BUFSIZEininc/proxy.handtune.bufsizein haproxy'scase, and nginx does not stall in the same place. That is circumstantial rather
than proof, but it is the obvious first thing to test: every payload above 16 KB
is being split into chunks, and whatever the per-chunk cost is, it is what stops
larger payloads paying off.
The experiment is cheap. Rebuild with
TPX_NET_BUFSIZEat 32 KB and 64 KB andre-run
bench.sh -m bulk -x "64 1024 8192 65536 1048576". If the plateau movesup with the constant, the cause is confirmed and the size becomes a trade
against the idle memory in the sibling issue, since the same constant sets how
much an idle connection holds. If it does not move, the cost is per-iteration in
the read/write path rather than per-chunk and the constant is innocent.
Note that the two issues pull in opposite directions: a larger chunk helps bulk
and hurts idle memory. That argues for making the size adaptive or the first
chunk lazy rather than simply raising the constant.