Skip to content

Make SSL release idle buffers, and update benchmark #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
novotimo merged 1 commit into from
Aug 11, 2026
Merged

Make SSL release idle buffers, and update benchmark #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions app/main.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -639,6 +639,7 @@ SSL_CTX *init_openssl(const tpx_listen_conf_t *config, int logfd) {
| SSL_OP_CIPHER_SERVER_PREFERENCE;

SSL_CTX_set_options(ctx, opts);
SSL_CTX_set_mode(ctx, SSL_MODE_RELEASE_BUFFERS);

if (config->cacerts != NULL) {
if (load_servcert(config, ctx, logfd) == 0)
Expand Down
66 changes: 63 additions & 3 deletions benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ and #44 means there are no TLS knobs to sweep.

## Measurements

Four modes, all driven by `bench.sh` onto one CSV schema. The harness emits the
Five modes, all driven by `bench.sh` onto one CSV schema. The harness emits the
CSV, a committed plotting script renders the images, and CSV, script and images
are all committed together.

Expand Down Expand Up @@ -314,6 +314,28 @@ are all committed together.
points are where a payload spans several queue chunks. Connections persist
for the whole run rather than churning, since at small payloads a churning
client measures its own ephemeral port table instead of the proxy.
5. **Message round trip on held connections,** `-m message`. N connections
established once and held, each sending one `-b` byte message every `-i`
milliseconds against the echo sink, swept over N. The other four modes all
miss this: handshake and rate close a connection the moment it is up, idle
sends nothing at all, and bulk streams continuously with `-n 0` so the
window is never small. That leaves the steady state of every workload the
proxy is actually for, an MQTT broker or a game server or an RPC service,
unmeasured, and it is where Nagle, a wake-up per message and any per-message
allocation show up. The latency columns for this mode are `msg_*` and
describe a round trip; `hs_*` still describes the handshakes that set the
connections up.

The send grid is absolute and fixed in advance, so a connection whose
response came back later than its own interval skips the slots it missed and
they are counted in `shed`. Advancing from completion instead would let a
slow subject quietly lower the offered rate until it matched whatever it
could serve, which is the same reason `-m rate` exists. Each connection
takes a phase fixed once at setup so a thread's sends spread evenly over one
interval; `-L` puts them all in phase instead, which is what a game tick
looks like. Run it over the wire like the first three modes rather than with
`GEN=local`, since a generator sharing the subject's CPU adds scheduler
noise to exactly the tail this mode exists to measure.

Two derived quantities normalize the rest and carry the comparison: **CPU per
handshake** and **MB/s per core**. A subject that wins on rate while spending a
Expand Down Expand Up @@ -343,6 +365,21 @@ that, `thermal_throttle/package_throttle_count` is read before and after every
run and any movement fails it, because a run that throttled part way through
reports a median that no configuration produced.

**Resumption actually happening.** With `-K`, `tlsload` prints `reused=` on the
`completed=` line of every raw file, counting the handshakes the server
resumed. A `-K` run reporting `reused=0` measured full handshakes in both arms
and says nothing about resumption. Two things made that the default outcome
until 2026-08-11 and both are worth knowing, since either one alone is enough to
turn the flag into a no-op: TLS 1.3 has no session to take when `SSL_connect()`
returns, because the server sends NewSessionTicket afterwards and only a read
processes it, so the session has to be collected from
`SSL_CTX_sess_set_new_cb()`; and `SSL_free()` on a connection closed without
`close_notify` runs `ssl_clear_bad_session()`, which calls
`SSL_CTX_remove_session()` and sets `not_resumable` on the session object
itself, so a session held by reference is spoiled by the connection that
supplied it and then by every connection that offers it. `tlsload` keeps a
copy, and hands out a copy per connection, for that reason.

**Ephemeral ports.** The client closes first in every churn test, so TIME_WAIT
accumulates on the generator. Linux holds TIME_WAIT for 60 seconds and exposes
no sysctl for the length, and the default `ip_local_port_range` of 32768 to
Expand Down Expand Up @@ -463,6 +500,23 @@ since repeatable runs matter more here than peak ones, and this chassis
otherwise throttles. To undo it without rebooting, write `0` to
`intel_pstate/no_turbo` and `powersave` back to every `scaling_governor`.

`bench.sh` brings each subject up with `compose up -d` and no `--build`, so it
runs whatever is already tagged `tlsproxy:bench-deb` and a change to the tree
does not reach it. Rebuild first, and note that a local `cmake --build build`
does nothing here since the subject is the image and the image compiles its own
copy from the build context:

```sh
docker compose --profile tlsproxy build tlsproxy
```

`provenance.txt` records `tlsproxy_sha` from `git rev-parse HEAD`, which is
blind to uncommitted work, so two runs either side of an unstaged change are
indistinguishable in it. When measuring something not yet committed, keep the
old image under a second tag (`docker tag tlsproxy:bench-deb
tlsproxy:bench-deb-<before>`) and retag to pick the arm, since `compose.yml`
names one image.

Then any of the four modes:

```sh
Expand All @@ -471,8 +525,14 @@ Then any of the four modes:
./bench.sh -m idle -x "1000 5000 20000" -d 30 -R 3
GEN=local ./bench.sh -m bulk -x "64 1024 8192 65536 1048576" \
-c 32 -n 0 -d 8 -R 3
./bench.sh -m message -x "1000 5000 20000" -b 200 -i 1000 -d 30 -R 3
./bench.sh -m message -x "1000 5000" -b 200 -i 33 -L -d 30 -R 3
```

The two message runs are the two shapes worth having: a message a second per
connection is the MQTT and RPC case, and 33 ms in lockstep is a 30 Hz game
tick. `-b` is the message size each way, since the sink echoes.

`bulk` runs with `GEN=local` because a gigabit link caps the data path far
below what the subjects can move, so across the wire it would measure the NIC.
It pins the generator to `6,7,14,15`, off the subject's cpuset. The subject
Expand All @@ -494,7 +554,7 @@ the underlying column goes up or down.
| Path | State |
| --- | --- |
| `README.md` | this file, the method |
| `bench.sh` | the entry point for all four modes, writing provenance and the git SHA into every result |
| `bench.sh` | the entry point for all five modes, writing provenance and the git SHA into every result |
| `report.py` | renders a results directory as markdown; `--against baseline/` adds deltas |
| `BASELINE.md` | the reference numbers as a report, with the caveats that stop them being a published result |
| `baseline/` | the CSVs and provenance behind it at `6e847f2`, tracked so a later run has something to diff against |
Expand All @@ -506,7 +566,7 @@ the underlying column goes up or down.
| `backend/nginx.conf` | the HTTP backend, for the tests that speak HTTP over the tunnel |
| `backend/sink/` | the discard/echo backend, static on `scratch` so its cgroup accounts for connection state and nothing else |
| `generators/build.sh` | builds `tlsload`, `tls-perf` and `wrk2` for the generator host and records what they are |
| `generators/tlsload/` | the open-model generator: handshake, hold and request modes, client-pinned TLS parameters, latency percentiles |
| `generators/tlsload/` | the open-model generator: handshake, hold, request and message modes, client-pinned TLS parameters, latency percentiles |
| `diag/` | per-second memory and socket-state time series for one subject under load |
| `monitoring.yml` | Prometheus, cAdvisor, node-exporter and Grafana, split out so bringing up the benchmark does not bring them up beside the subject |
| `prometheus.yml` | configuration for the above |
Expand Down
57 changes: 43 additions & 14 deletions benchmark/bench.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
#!/bin/sh
# Drives one benchmark mode across subjects and writes CSV, raw generator
# output and provenance into results/<run>. All four modes share one CSV
# output and provenance into results/<run>. All five modes share one CSV
# schema and one set of validity checks.
#
# bench.sh -m handshake -x "64 256 1024 4096" closed loop, axis = concurrency
# bench.sh -m rate -x "2000 4000 6000 8000" open loop, axis = offered rate
# bench.sh -m idle -x "1000 5000 20000" held open, axis = connections
# bench.sh -m bulk -x "64 1024 65536" echo, axis = payload bytes
# bench.sh -m message -x "1000 5000 20000" echo, axis = connections
#
# message holds its connections and sends one small message per connection every
# -i milliseconds, so its latency columns are msg_* and describe a round trip
# rather than a handshake. The hs_* columns still describe the handshakes that
# set the connections up.
set -eu

here=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
Expand Down Expand Up @@ -39,31 +45,36 @@ reps=6
dur=10
threads=4
resume=0
interval=1000
lockstep=0
backend=""

while getopts "m:s:e:x:c:r:b:n:R:d:t:B:K" o; do
while getopts "m:s:e:x:c:r:b:n:R:d:t:B:i:LK" o; do
case $o in
m) mode=$OPTARG ;; s) subjects=$OPTARG ;;
e) certs=$OPTARG ;; x) axis=$OPTARG ;;
c) conc=$OPTARG ;; r) rate=$OPTARG ;;
b) payload=$OPTARG ;; n) reqs=$OPTARG ;;
R) reps=$OPTARG ;; d) dur=$OPTARG ;;
t) threads=$OPTARG ;; B) backend=$OPTARG ;;
i) interval=$OPTARG ;; L) lockstep=1 ;;
K) resume=1 ;;
*) echo "usage: $0 -m handshake|rate|idle|bulk [-x values] [-s subjects]" >&2
*) echo "usage: $0 -m handshake|rate|idle|bulk|message [-x values]" \
"[-s subjects]" >&2
exit 2 ;;
esac
done

case $mode in
handshake|rate|idle|bulk) : ;;
*) echo "mode must be handshake, rate, idle or bulk" >&2; exit 2 ;;
handshake|rate|idle|bulk|message) : ;;
*) echo "mode must be handshake, rate, idle, bulk or message" >&2; exit 2 ;;
esac

# bulk needs the sink echoing; the rest only need it to hold and to close.
# bulk and message need the sink echoing; the rest only need it to hold and to
# close.
if [ -z "$backend" ]; then
backend=backend-sink
[ "$mode" = bulk ] && backend=backend-echo
case $mode in bulk|message) backend=backend-echo ;; esac
fi

run=$(date -u +%Y%m%dT%H%M%SZ)
Expand Down Expand Up @@ -130,6 +141,8 @@ wait_ready() {
echo "cipher $CIPHER"
echo "tls_version $TLSVER"
echo "resumption $resume"
[ "$mode" = message ] && echo "msg_interval_ms $interval"
[ "$mode" = message ] && echo "msg_lockstep $lockstep"
echo "duration_s $dur"
echo "reps $reps"
echo "threads $threads"
Expand All @@ -149,7 +162,7 @@ wait_ready() {
done
} > "$out/provenance.txt" 2>&1

echo "run,sha,mode,model,subject,cert,resume,axis,concurrency,offered_rate,payload,reqs,threads,duration,rep,completed,rate,errors,shed,hs_p50,hs_p95,hs_p99,hs_p999,hs_max,mb_per_sec,cpu_cores,ws_peak_bytes,anon_bytes,file_bytes,slab_bytes,mem_max_delta,throttle_delta,workers_stable,run_ok" > "$csv"
echo "run,sha,mode,model,subject,cert,resume,axis,concurrency,offered_rate,payload,reqs,threads,duration,rep,completed,rate,errors,shed,hs_p50,hs_p95,hs_p99,hs_p999,hs_max,msg_p50,msg_p95,msg_p99,msg_p999,msg_max,mb_per_sec,cpu_cores,ws_peak_bytes,anon_bytes,file_bytes,slab_bytes,mem_max_delta,throttle_delta,workers_stable,run_ok" > "$csv"

bport=8080
case $backend in backend-sink) bport=8081 ;; backend-echo) bport=8082 ;; esac
Expand All @@ -171,17 +184,19 @@ for cert in $certs; do
for x in $axis; do
a_conc=$conc; a_rate=$rate; a_pay=$payload
case $mode in
handshake|idle) a_conc=$x ;;
rate) a_rate=$x ;;
bulk) a_pay=$x ;;
handshake|idle|message) a_conc=$x ;;
rate) a_rate=$x ;;
bulk) a_pay=$x ;;
esac

case $mode in
handshake) margs="-m handshake -c $a_conc" ;;
idle) margs="-m hold -c $a_conc -H 0" ;;
rate) margs="-m handshake -r $a_rate -M 400000" ;;
bulk) margs="-m request -c $a_conc -b $a_pay -n $reqs" ;;
message) margs="-m message -c $a_conc -b $a_pay -I $interval" ;;
esac
[ "$mode" = message ] && [ "$lockstep" = 1 ] && margs="$margs -L"
[ "$resume" = 1 ] && margs="$margs -R"

for rep in $(seq 1 "$reps"); do
Expand Down Expand Up @@ -224,20 +239,34 @@ for cert in $certs; do
if (kv[1]=="p99") p99=kv[2]
if (kv[1]=="p999") p999=kv[2]
if (kv[1]=="max") mx=kv[2] } }
# Left empty rather than zeroed for the modes that send no
# messages, so a median never averages in a column that was
# never measured.
/^messages=/ {
for (i=1;i<=NF;i++) { split($i,kv,"=")
if (kv[1]=="msg_ms_p50") m50=kv[2]
if (kv[1]=="p95") m95=kv[2]
if (kv[1]=="p99") m99=kv[2]
if (kv[1]=="p999") m999=kv[2]
if (kv[1]=="max") mmx=kv[2] } }
/^bytes_up=/ {
for (i=1;i<=NF;i++) { split($i,kv,"=")
if (kv[1]=="mbytes_per_sec") mb=kv[2] } }
END { printf "%d,%d,%d,%d,%s,%s,%s,%s,%s,%s,%d",
END { printf "%d,%d,%d,%d,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%d",
comp+0, rt+0, er+0, sh+0, p50+0, p95+0, p99+0,
p999+0, mx+0, mb+0, ok+0 }
p999+0, mx+0, m50, m95, m99, m999, mmx, mb+0, ok+0 }
' "$raw")
ok=${stats##*,}; stats=${stats%,*}

cores=$(awk -v a="$pre_cpu" -v b="$post_cpu" -v d="$dur" \
'BEGIN{printf "%.3f",(b-a)/1000000/d}')
stable=0; [ "$pre_procs" = "$post_procs" ] && stable=1
tdelta=$((post_thr - pre_thr)); mmdelta=$((post_mm - pre_mm))
model=closed; [ "$mode" = rate ] && model=open
# message is open per message rather than per connection: the
# send grid is fixed in advance, so a subject that cannot keep
# up misses slots instead of being offered less.
model=closed
case $mode in rate|message) model=open ;; esac

echo "$run,$sha,$mode,$model,$subj,$cert,$resume,$x,$a_conc,$a_rate,$a_pay,$reqs,$threads,$dur,$rep,$stats,$cores,$ws,$anon,$file,$slab,$mmdelta,$tdelta,$stable,$ok" >> "$csv"

Expand Down
Loading
Loading