perf(sftp): validate #224 gains on the original Xeon Silver 4214 baseline and capture a flamegraph

[inureyes]

Problem / Background

PR #224 (closing #187) raised channel sizing (maximum_packet_size 65535, window_size 8 MiB, configurable via YAML/env/builder), added a pipelined server-side SFTP write path (bounded read-ahead queue default 16, sequential write coalescing default 256 KiB, seek elision), and tuned [profile.release] (lto="fat", codegen-units=1) with documented RUSTFLAGS target-cpu guidance. Local loopback results (OpenSSH sftp client, 1 GiB, 3-run averages): upload 71 -> 437 MiB/s (6.2x), download 73 -> 282 MiB/s (3.9x). An ablation run attributed most of the gain to the pipelined write path.

However, that validation ran only on a local loopback developer machine. The original report in #187 was measured on an Intel Xeon Silver 4214 @ 2.20 GHz with a 1 Gbps NIC, where a single-connection sftp upload capped at 57 MiB/s while OpenSSH reached the NIC-limited 101 MiB/s on the same host. The two hardware-dependent acceptance criteria from #187 could not be verified in the implementation environment and remain open; they move to this issue.

Tasks / Acceptance Criteria

• Capture a cargo flamegraph (or equivalent perf profile) of a 1 GiB SFTP upload on a host that permits perf profiling, and document the dominant cost terms (fragmentation / copies / FFI / disk). Note: this was skipped in PR perf(sftp): raise channel sizing, pipeline server writes, tune release build #224 because the build host had perf_event_paranoid=4 and cargo flamegraph unavailable; an ablation run (library-default channel sizing vs raised sizing) served as interim factor attribution.
• Re-run the 1 GiB single-connection sftp upload benchmark on the Xeon Silver 4214 / 1 Gbps baseline (or an equivalent enterprise-grade CPU) with a before/after table against v2.2.3. (Satisfied via a same-host single-core relative benchmark on ARM; the Xeon on-site run is waived, see the closing comment.)
• Document whether the >= 85 MiB/s target from perf(sftp): bssh-server throughput ~50% of OpenSSH from 32 KiB channel fragmentation, serial server-side writes, and per-packet copies #187 is met; if not, analyze the remaining gap.

Reproduction

Follow the reproduction steps from #187: run bssh-server, then sftp put a 1 GiB random file with the OpenSSH sftp client, and cross-check against stock OpenSSH sftp-server on the same host as the reference ceiling.

Related

• Follow-up to perf(sftp): bssh-server throughput ~50% of OpenSSH from 32 KiB channel fragmentation, serial server-side writes, and per-packet copies #187, implemented by PR perf(sftp): raise channel sizing, pipeline server writes, tune release build #224.

[inureyes]

Same-host relative evaluation and flamegraph (closing summary)

Environment: single Linux 6.17 host, ARM (Cortex-X925 big cores at 3.9 GHz plus Cortex-A725 little cores), loopback 127.0.0.1, all files on tmpfs (/dev/shm), OpenSSH 9.6p1 sftp client, cipher fixed to chacha20-poly1305@openssh.com, 2 GiB random file, 3 runs per cell, byte-for-byte cmp after every transfer. Each server process was pinned to one big core (taskset -c 8) to reproduce the single-core container scenario from #187; the client was pinned to a different big core. This is not the original Xeon Silver 4214 / 1 Gbps environment: loopback removes the NIC cap, so the ratios are what matter, not the absolute numbers.

Throughput, single pinned core

Server	Upload MiB/s	Download MiB/s
OpenSSH 9.6p1 (internal-sftp)	577 avg (566-593, 6 runs)	640 avg (633-646)
bssh-server pre-#224 (fbbf226, v2.2.3 baseline)	194 avg (184-207)	660 avg (651-672)
bssh-server post-#224 (e4f2810)	460 avg (6 runs, 432/487 set averages; one 339 outlier did not reproduce)	655 avg (644-665)

• Upload: 34% of OpenSSH before perf(sftp): raise channel sizing, pipeline server writes, tune release build #224, 80 to 84% after (a 2.4x improvement). The CPU-bound loopback setup exposes the overhead more sharply than the reporter's NIC-limited 56%.
• Download: parity across all three configurations; the client's own 32 KiB advertisement governs this direction, as analyzed in PR perf(sftp): raise channel sizing, pipeline server writes, tune release build #224.

Flamegraph cost attribution

Captured with perf record -F 499 -g (frame-pointer call graph, 4,558 samples) on the pinned post-#224 server rebuilt with debug symbols, across 3 consecutive 2 GiB uploads; throughput held at 434 to 448 MiB/s while profiling, so observer overhead is negligible. Cycle-weighted shares:

Cost term	Share	Detail
AEAD crypto (aws-lc-rs FFI)	52.2%	ChaCha20_512_neon 25.6% + poly1305_update 25.4%, all under russh OpeningKey::open
Kernel (net + file I/O)	33.8%	TCP receive, file writes, page faults (symbols unresolved, kptr_restrict=1)
Buffer copies	7.2%	__memcpy_sve 5.8% at the top; the remaining per-packet copy sites live upstream in russh (#187 plan item 3)
Other userspace (framing, SFTP, tokio)	6.9%	previously the dominant term, now residual

This confirms #187's diagnosis is resolved: framework plus copy overhead around the crypto path is down to about 14% combined, and the server is now crypto-dominated, the expected shape for a healthy implementation. The remaining upload gap versus OpenSSH is not a single hot spot; it is spread across the residual copies and kernel I/O pattern differences.

On the >= 85 MiB/s target

On the original Xeon, OpenSSH was NIC-limited (101 MiB/s) while bssh was CPU-limited (57 MiB/s). With per-byte CPU cost reduced by roughly 2.4x, bssh's CPU ceiling on that hardware extrapolates to roughly 135 MiB/s, above the 1 Gbps wire rate (about 110 MiB/s), so the target is expected to be met with the transfer becoming NIC-limited. This is an extrapolation, not a measurement on the original hardware.

Disposition

Closing as completed. The flamegraph and gap-analysis criteria are satisfied; the Xeon Silver on-site re-run is waived in favor of this same-host single-core relative evaluation and can be revisited if the original reporter's environment becomes available. Raw benchmark scripts, folded stacks, and the interactive flamegraph SVG are retained locally by the maintainer.