You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Under identical hardware and workload, bssh-server sustains roughly half the single-connection SFTP throughput of OpenSSH's sftp-server (e.g. 57 MiB/s vs. a NIC-limited 101 MiB/s on a Xeon Silver 4214). The reporter correctly isolated the cause to per-SSH-packet framework and protocol overhead around the crypto path, not the AEAD primitive itself. A code audit confirms and refines that diagnosis: the dominant costs are (1) 32 KiB SSH channel fragmentation of large SFTP writes, (2) a strictly request-blocked server-side SFTP loop, (3) several per-packet buffer copies and small heap allocations, and (4) an untuned release build. This is a blocker for using bssh-server as a drop-in replacement for OpenSSH in Backend.AI SFTP agent containers.
Background
The reporter's measurements show AEAD ciphers (AES-GCM, ChaCha20-Poly1305), which route through aws-lc-rs, capping at the same ~57 MiB/s on the Xeon even though the primitive can do multiple GB/s, with one tokio worker thread pegged at ~90% CPU. That points squarely at framing/protocol overhead rather than the cipher.
A read-only code audit against the current tree (v2.2.3) confirms the bottlenecks and corrects one of the reporter's hypotheses:
Note (2026-07-17): re-audited against current main after #212 replaced the vendored crates/bssh-russh fork with upstream russh 0.62.1. The russh-side references below now point into the upstream crate source (written as russh-0.62.1/src/...). All findings still hold; see the Refresh log at the bottom.
Hypothesis feat: add connection pooling infrastructure for future features #1 (per-packet AEAD context construction) is not what the code does. For GCM and ChaCha20-Poly1305 the sealing/opening key (the aws-lc-rs bound context) is built once per key exchange in russh-0.62.1/src/kex/mod.rs:450 / :460 and reused for every packet. The per-packet cost is one in-place FFI seal_in_place/open_in_place call plus a nonce advance() and two tiny (4-byte length, 16-byte tag) clone_from_slices (russh-0.62.1/src/cipher/gcm.rs:119,167, russh-0.62.1/src/cipher/chacha20poly1305.rs:81,130). The crypto itself is zero-copy. So the tax is the number of FFI calls, which the fragmentation below multiplies, not key setup.
Root-cause findings (code audit)
1. 32 KiB SSH channel fragmentation is the highest-leverage factor. russh defaults maximum_packet_size = 32768 and window_size = 2097152 (russh-0.62.1/src/server/mod.rs:115-116). bssh-server never overrides them: build_russh_config uses ..Default::default() (src/server/mod.rs:217). Meanwhile the SFTP layer accepts 256 KiB SFTP packets (max_client_packet_len = 262144, crates/bssh-russh-sftp/src/server/mod.rs:43). Net effect: every 256 KiB SFTP WRITE is chopped into 8 × 32 KiB CHANNEL_DATA packets, multiplying the per-packet AEAD FFI call, copies, and task hops by 8x. maximum_packet_size is capped at 65535 by a guard at russh-0.62.1/src/server/mod.rs:908.
2. The server-side SFTP loop is strictly request-blocked (no pipelining, no read-ahead).process_handler in crates/bssh-russh-sftp/src/server/mod.rs:79-96 does read_packet → process_request → write_all → flush serially, one request at a time, flushing before the next packet is even read. Each WRITE (src/server/sftp.rs:744-783) additionally takes a per-write async mutex on the handle map, seek(offset) even for sequential writes, then write_all(&data).await before replying. Disk-write latency and the status round-trip are fully serialized. Note: the fork's "pipelined File I/O" (write_all_pipelined / read_to_writer_pipelined in crates/bssh-russh-sftp/src/client/fs/file.rs) is client-side only and does not help the server upload path.
3. Per-packet copies and small allocations around the (zero-copy) crypto. On upload, each 32 KiB packet incurs: Bytes::decode alloc+copy plus a data.clone() per CHANNEL_DATA packet (russh-0.62.1/src/server/encrypted.rs:1019,1044), put_slice into the SFTP read buffer (russh-0.62.1/src/channels/io/rx.rs:51), a vec![0; length] per-SFTP-packet alloc (crates/bssh-russh-sftp/src/utils.rs:21), a serde Vec<u8> copy of the Write.data body, plus a per-packet vec![0; ..] for the transport length prefix (russh-0.62.1/src/cipher/mod.rs:301) and one tokio mpsc hop. On download, russh-0.62.1/src/channels/io/tx.rs:118 does Bytes::copy_from_slice per chunk and :86 takes an owned async-mutex lock on the window per poll_write.
4. AES-CTR is a secondary contributor. CTR modes use pure-Rust aes/ctr (russh-0.62.1/src/cipher/block.rs) plus a separate HMAC pass (two passes over each packet), and CtrWrapper re-creates the AES key schedule on every call (block.rs:75-93). CTR is not the primary bottleneck cipher (AEAD is), but it explains the lower CTR numbers in the report.
5. No release tuning. There is no [profile.release] anywhere in the workspace and no .cargo/config.toml with target-cpu. Release therefore builds with lto = false, codegen-units = 16, and baseline x86-64 codegen, which blocks cross-crate inlining of the hot framing/copy/MAC path.
Proposed Solution / Investigation Plan
Ranked by expected leverage:
Raise the SSH channel packet size. Set maximum_packet_size toward the 65535 cap (and a larger window_size) in build_russh_config (src/server/mod.rs:183-218) so a 256 KiB SFTP write is not fragmented into 8 packets. This is the single highest-leverage knob and is currently left at the library default. Both fields are plain russh::server::Config members, so this stays a bssh-side change after the fork removal. Verify negotiated sizes against clients.
Add server-side SFTP pipelining / write coalescing. Allow multiple in-flight WRITE requests (bounded) instead of the strict read→process→flush loop (crates/bssh-russh-sftp/src/server/mod.rs:79-96), and avoid the per-write seek for sequential offsets (src/server/sftp.rs:764-767).
Cut per-packet copies/allocs at the sites listed above (reuse buffers for the transport length prefix and SFTP packet buffer; avoid the Bytes::decode and serde Vec<u8> copies where feasible; consider zero-copy sendfile/splice for downloads to a regular file). Since chore(deps): drop bssh-russh fork, use upstream russh 0.62.1 #212 the russh-side copy sites live in upstream russh, so those fixes now require upstream PRs or carrying a patch; the bssh-russh-sftp and src/server/sftp.rs sites remain directly editable here.
Add a tuned release profile:[profile.release] lto = "fat", codegen-units = 1, and a .cargo/config.tomltarget-cpu (e.g. x86-64-v3) or a documented RUSTFLAGS=-C target-cpu=native for self-builds. Measure; consider PGO for the crypto+I/O hot path.
AES-CTR: either route through aws-lc-rs or drop CTR from the default cipher list; at minimum stop rebuilding the AES key schedule per call. The cipher list is negotiable from bssh via russh::Preferred (see the compression override already in build_russh_config), but the CTR implementation itself is upstream code (same caveat as item 3).
First diagnostic step: run cargo flamegraph on a 1 GiB SFTP upload to confirm the split between aws-lc-rs FFI, russh framing/copies, russh-sftp, and tokio, and to validate that (1) and (2) above are the dominant terms before investing in (3).
Implementation Notes
Concrete profiling / fix targets (verified against main on 2026-07-17; russh-0.62.1/... paths are upstream crate source):
russh-0.62.1/src/server/mod.rs:115-116 (defaults) and src/server/mod.rs:183-218 (build_russh_config, ..Default::default() at :217).
Crypto backend is already aws-lc-rs by default (upstream russh 0.62.1 Cargo.toml, default = ["flate2", "aws-lc-rs", "rsa"]).
Raising maximum_packet_size and enabling server pipelining both interact with buffer sizing and window management; validate interop (OpenSSH, FileZilla, WinSCP, sshj/Cyberduck) and re-measure the table in the report after each change.
Acceptance Criteria
A cargo flamegraph profile of a 1 GiB SFTP upload is captured and the dominant cost terms are documented (fragmentation / copies / FFI / disk).
maximum_packet_size (and window_size) are configurable and set to a larger value by bssh-server, verified to negotiate correctly with common clients.
Server-side SFTP write path no longer flushes-and-blocks per 32 KiB chunk (pipelining or write coalescing), with a benchmark showing improvement.
A tuned [profile.release] (LTO, codegen-units = 1) and a target-cpu mechanism are added and measured.
Single-connection SFTP upload throughput on the Xeon Silver 4214 baseline improves materially toward the OpenSSH/NIC-limited figure (target documented; e.g. ≥ 85 MiB/s), with a before/after table.
No interop regressions against OpenSSH sftp, FileZilla, WinSCP, and an sshj-based client.
Under identical hardware and workload, bssh-server sustains roughly half the SFTP throughput of OpenSSH's sftp-server. The gap does not appear to come from the cryptographic primitives themselves (bssh-russh uses aws-lc-rs for AEAD ciphers by default), but from per-SSH-packet framework and protocol overhead around the crypto path.
This matters because a realistic goal for bssh-server in Backend.AI is to replace dedicated SFTP agents (which currently rely on OpenSSH). On slower/enterprise-grade CPUs, the current overhead effectively halves single-connection SFTP throughput.
AEAD ciphers that route through aws-lc-rs (AES-GCM, ChaCha20-Poly1305) cap at the same ~57 MiB/s on Xeon Silver, even though the underlying primitives can do several GB/s. This strongly suggests the bottleneck is outside the AEAD primitive.
AES-CTR modes are additionally slowed by going through pure-Rust `aes`/`ctr` (see `crates/bssh-russh/src/cipher/block.rs`) plus a separate HMAC pass.
Adding a second core yields only ~5%; a single SSH connection is inherently mostly sequential, so the gain is expected to be small. The CPU architecture (Zen 2 + VAES vs. Cascade Lake) explains most of the host-to-host spread.
CPU profile during transfer: one tokio worker thread pegs ~90% CPU while the other tokio workers stay near 0%. The process is CPU-bound on this single hot thread.
Why the gap is interesting
`bssh-russh` already uses `aws-lc-rs` for AEAD ciphers (`crates/bssh-russh/src/cipher/gcm.rs`, `crates/bssh-russh/src/cipher/chacha20poly1305.rs`), which is the same family of assembly-optimised code that OpenSSL/OpenSSH use. So the crypto primitive cannot be twice as slow as OpenSSH — yet the end-to-end SFTP throughput is. Meaning the extra cycles are almost certainly spent in the code around the primitive.
Hypotheses for the per-packet overhead
Unverified; posted for discussion / profiling.
Per-packet AEAD invocation cost. SSH packets default to ~32 KiB, so a 1 GiB transfer issues ~32k encrypt/decrypt calls. Each call goes through `BoundKey` + `NonceSequence` construction and crosses the aws-lc-rs FFI boundary. OpenSSH amortises this with long-lived cipher contexts and direct function calls.
Extra buffer copies. Network buffer → decrypt buffer → russh channel → russh-sftp layer → file write: each stage potentially memcpys the full 32 KiB. OpenSSH's `sftp-server` uses `sendfile`/`splice` where possible and keeps data in fewer buffers.
Tokio channel/task overhead per SSH packet. Small-packet async pipelines are known to have non-trivial cost per message (channel send + await + wake), which becomes dominant when the crypto itself is fast.
russh-sftp protocol layer. Worth profiling to see whether SFTP request/response handling is request-blocked rather than pipelined.
Suggested investigations / directions
Profile with `cargo flamegraph` on a 1 GiB SFTP upload. This should immediately show whether time is spent in aws-lc-rs, in russh packet framing, in russh-sftp, or in tokio.
Increase SSH/SFTP packet size. Larger packets amortise per-call overhead; a quick experiment with the server's `max_packet_size` / SFTP buffer would confirm whether per-call overhead is the dominant factor.
Zero-copy file I/O on the server side. Teach russh-sftp to use `sendfile`/`splice` when the destination is a regular file.
Route AES-CTR through aws-lc-rs (or drop CTR ciphers from the default list). Currently `crates/bssh-russh/src/cipher/block.rs` uses pure-Rust `aes`/`ctr`, which is a secondary but real contributor for users who negotiate CTR.
Build-time optimisations. Confirm LTO / `codegen-units = 1` / `target-cpu` settings used for release binaries; PGO can be meaningful for crypto+I/O hot paths.
Reproduction
# Server (inside a container on the slower host)
/tmp/bssh-server gen-host-key --output /tmp/bssh_host_key -t ed25519
/tmp/bssh-server run -b 0.0.0.0 -p 2200 -k /tmp/bssh_host_key -D
# Client
dd if=/dev/urandom of=/tmp/testfile_1G bs=1M count=1024
echo'put /tmp/testfile_1G testfile_1G'| \
sftp -i id_container -P 2200 \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
work@<server>
Cross-check with a stock OpenSSH `sftp-server` on the same host/container/user for a baseline.
Checked off: none. No implementing PRs exist; [profile.release] and .cargo/config.toml are still absent, build_russh_config still uses ..Default::default(), and the server-side SFTP loop is still serial, so all six acceptance criteria remain open.
Renamed references: crates/bssh-russh/... paths now point at upstream russh-0.62.1/src/... (fork removed in chore(deps): drop bssh-russh fork, use upstream russh 0.62.1 #212); guard server/mod.rs:856 → :908; CHANNEL_DATA copy encrypted.rs:842 → :1019,1044; CTR key-schedule block.rs:80-92 → :75-93; bssh-side src/server/mod.rs:196-202 → :183-218 (..Default::default() at :217) and src/server/sftp.rs:757-780 → :744-783.
Scope note: plan items 3 and 5 (per-packet copies, CTR internals) now require upstream russh contributions or a carried patch; items 1, 2, and 4 remain directly actionable in this repo.
Summary
Under identical hardware and workload,
bssh-serversustains roughly half the single-connection SFTP throughput of OpenSSH'ssftp-server(e.g. 57 MiB/s vs. a NIC-limited 101 MiB/s on a Xeon Silver 4214). The reporter correctly isolated the cause to per-SSH-packet framework and protocol overhead around the crypto path, not the AEAD primitive itself. A code audit confirms and refines that diagnosis: the dominant costs are (1) 32 KiB SSH channel fragmentation of large SFTP writes, (2) a strictly request-blocked server-side SFTP loop, (3) several per-packet buffer copies and small heap allocations, and (4) an untuned release build. This is a blocker for usingbssh-serveras a drop-in replacement for OpenSSH in Backend.AI SFTP agent containers.Background
The reporter's measurements show AEAD ciphers (AES-GCM, ChaCha20-Poly1305), which route through
aws-lc-rs, capping at the same ~57 MiB/s on the Xeon even though the primitive can do multiple GB/s, with one tokio worker thread pegged at ~90% CPU. That points squarely at framing/protocol overhead rather than the cipher.A read-only code audit against the current tree (v2.2.3) confirms the bottlenecks and corrects one of the reporter's hypotheses:
aws-lc-rsbound context) is built once per key exchange inrussh-0.62.1/src/kex/mod.rs:450/:460and reused for every packet. The per-packet cost is one in-place FFIseal_in_place/open_in_placecall plus a nonceadvance()and two tiny (4-byte length, 16-byte tag)clone_from_slices (russh-0.62.1/src/cipher/gcm.rs:119,167,russh-0.62.1/src/cipher/chacha20poly1305.rs:81,130). The crypto itself is zero-copy. So the tax is the number of FFI calls, which the fragmentation below multiplies, not key setup.Root-cause findings (code audit)
1. 32 KiB SSH channel fragmentation is the highest-leverage factor. russh defaults
maximum_packet_size = 32768andwindow_size = 2097152(russh-0.62.1/src/server/mod.rs:115-116).bssh-servernever overrides them:build_russh_configuses..Default::default()(src/server/mod.rs:217). Meanwhile the SFTP layer accepts 256 KiB SFTP packets (max_client_packet_len = 262144,crates/bssh-russh-sftp/src/server/mod.rs:43). Net effect: every 256 KiB SFTP WRITE is chopped into 8 × 32 KiB CHANNEL_DATA packets, multiplying the per-packet AEAD FFI call, copies, and task hops by 8x.maximum_packet_sizeis capped at 65535 by a guard atrussh-0.62.1/src/server/mod.rs:908.2. The server-side SFTP loop is strictly request-blocked (no pipelining, no read-ahead).
process_handlerincrates/bssh-russh-sftp/src/server/mod.rs:79-96doesread_packet → process_request → write_all → flushserially, one request at a time, flushing before the next packet is even read. Each WRITE (src/server/sftp.rs:744-783) additionally takes a per-write async mutex on the handle map,seek(offset)even for sequential writes, thenwrite_all(&data).awaitbefore replying. Disk-write latency and the status round-trip are fully serialized. Note: the fork's "pipelined File I/O" (write_all_pipelined/read_to_writer_pipelinedincrates/bssh-russh-sftp/src/client/fs/file.rs) is client-side only and does not help the server upload path.3. Per-packet copies and small allocations around the (zero-copy) crypto. On upload, each 32 KiB packet incurs:
Bytes::decodealloc+copy plus adata.clone()per CHANNEL_DATA packet (russh-0.62.1/src/server/encrypted.rs:1019,1044),put_sliceinto the SFTP read buffer (russh-0.62.1/src/channels/io/rx.rs:51), avec![0; length]per-SFTP-packet alloc (crates/bssh-russh-sftp/src/utils.rs:21), a serdeVec<u8>copy of theWrite.databody, plus a per-packetvec![0; ..]for the transport length prefix (russh-0.62.1/src/cipher/mod.rs:301) and one tokio mpsc hop. On download,russh-0.62.1/src/channels/io/tx.rs:118doesBytes::copy_from_sliceper chunk and:86takes an owned async-mutex lock on the window perpoll_write.4. AES-CTR is a secondary contributor. CTR modes use pure-Rust
aes/ctr(russh-0.62.1/src/cipher/block.rs) plus a separate HMAC pass (two passes over each packet), andCtrWrapperre-creates the AES key schedule on every call (block.rs:75-93). CTR is not the primary bottleneck cipher (AEAD is), but it explains the lower CTR numbers in the report.5. No release tuning. There is no
[profile.release]anywhere in the workspace and no.cargo/config.tomlwithtarget-cpu. Release therefore builds withlto = false,codegen-units = 16, and baselinex86-64codegen, which blocks cross-crate inlining of the hot framing/copy/MAC path.Proposed Solution / Investigation Plan
Ranked by expected leverage:
maximum_packet_sizetoward the 65535 cap (and a largerwindow_size) inbuild_russh_config(src/server/mod.rs:183-218) so a 256 KiB SFTP write is not fragmented into 8 packets. This is the single highest-leverage knob and is currently left at the library default. Both fields are plainrussh::server::Configmembers, so this stays a bssh-side change after the fork removal. Verify negotiated sizes against clients.crates/bssh-russh-sftp/src/server/mod.rs:79-96), and avoid the per-writeseekfor sequential offsets (src/server/sftp.rs:764-767).Bytes::decodeand serdeVec<u8>copies where feasible; consider zero-copysendfile/splicefor downloads to a regular file). Since chore(deps): drop bssh-russh fork, use upstream russh 0.62.1 #212 the russh-side copy sites live in upstreamrussh, so those fixes now require upstream PRs or carrying a patch; thebssh-russh-sftpandsrc/server/sftp.rssites remain directly editable here.[profile.release] lto = "fat",codegen-units = 1, and a.cargo/config.tomltarget-cpu(e.g.x86-64-v3) or a documentedRUSTFLAGS=-C target-cpu=nativefor self-builds. Measure; consider PGO for the crypto+I/O hot path.aws-lc-rsor drop CTR from the default cipher list; at minimum stop rebuilding the AES key schedule per call. The cipher list is negotiable from bssh viarussh::Preferred(see the compression override already inbuild_russh_config), but the CTR implementation itself is upstream code (same caveat as item 3).First diagnostic step: run
cargo flamegraphon a 1 GiB SFTP upload to confirm the split between aws-lc-rs FFI, russh framing/copies, russh-sftp, and tokio, and to validate that (1) and (2) above are the dominant terms before investing in (3).Implementation Notes
mainon 2026-07-17;russh-0.62.1/...paths are upstream crate source):russh-0.62.1/src/server/mod.rs:115-116(defaults) andsrc/server/mod.rs:183-218(build_russh_config,..Default::default()at:217).crates/bssh-russh-sftp/src/server/mod.rs:79-96(serial loop) andsrc/server/sftp.rs:744-783(write: handle mutex:755,seek:764,write_all:767).russh-0.62.1/src/server/encrypted.rs:1019,1044,russh-0.62.1/src/channels/io/rx.rs:51,crates/bssh-russh-sftp/src/utils.rs:21,russh-0.62.1/src/cipher/mod.rs:301,russh-0.62.1/src/channels/io/tx.rs:86,118.russh-0.62.1/src/cipher/gcm.rs:119,167,russh-0.62.1/src/cipher/chacha20poly1305.rs:81,130; CTR:russh-0.62.1/src/cipher/block.rs:75-93,186,251.aws-lc-rsby default (upstreamrussh0.62.1Cargo.toml,default = ["flate2", "aws-lc-rs", "rsa"]).maximum_packet_sizeand enabling server pipelining both interact with buffer sizing and window management; validate interop (OpenSSH, FileZilla, WinSCP, sshj/Cyberduck) and re-measure the table in the report after each change.Acceptance Criteria
cargo flamegraphprofile of a 1 GiB SFTP upload is captured and the dominant cost terms are documented (fragmentation / copies / FFI / disk).maximum_packet_size(andwindow_size) are configurable and set to a larger value bybssh-server, verified to negotiate correctly with common clients.[profile.release](LTO,codegen-units = 1) and atarget-cpumechanism are added and measured.sftp, FileZilla, WinSCP, and an sshj-based client.Related
crates/bssh-russhfork with upstreamrussh0.62.1, which is why russh-side references in this issue point into the upstream crate.Original Suggestion
Title: bssh-server v2.1.1: SFTP throughput ~50% of OpenSSH despite aws-lc-rs; framework/protocol overhead suspected
Summary
Under identical hardware and workload,
bssh-serversustains roughly half the SFTP throughput of OpenSSH'ssftp-server. The gap does not appear to come from the cryptographic primitives themselves (bssh-russhusesaws-lc-rsfor AEAD ciphers by default), but from per-SSH-packet framework and protocol overhead around the crypto path.This matters because a realistic goal for
bssh-serverin Backend.AI is to replace dedicated SFTP agents (which currently rely on OpenSSH). On slower/enterprise-grade CPUs, the current overhead effectively halves single-connection SFTP throughput.Environment
linux-x86_64-muslbuild)sftp(Ubuntu 22.04)sftp put(SFTP subsystem)Measurements (1 GiB upload,
sftp put, average of 3 runs)Observations:
Why the gap is interesting
`bssh-russh` already uses `aws-lc-rs` for AEAD ciphers (`crates/bssh-russh/src/cipher/gcm.rs`, `crates/bssh-russh/src/cipher/chacha20poly1305.rs`), which is the same family of assembly-optimised code that OpenSSL/OpenSSH use. So the crypto primitive cannot be twice as slow as OpenSSH — yet the end-to-end SFTP throughput is. Meaning the extra cycles are almost certainly spent in the code around the primitive.
Hypotheses for the per-packet overhead
Unverified; posted for discussion / profiling.
Suggested investigations / directions
Reproduction
Cross-check with a stock OpenSSH `sftp-server` on the same host/container/user for a baseline.
Related
Refresh log
main(post chore(deps): drop bssh-russh fork, use upstream russh 0.62.1 #212/fix(sftp): sftp.root chroot rejects all non-root path resolution (cd/get/open/stat) because resolve_chroot does not re-root absolute virtual paths #214/fix(sftp): sshj/Cyberduck clients dropped on initial REALPATH/STAT via russh strict-parser SshEncoding::Length on incoming decode #215; workspace version still 2.2.3).[profile.release]and.cargo/config.tomlare still absent,build_russh_configstill uses..Default::default(), and the server-side SFTP loop is still serial, so all six acceptance criteria remain open.crates/bssh-russh/...paths now point at upstreamrussh-0.62.1/src/...(fork removed in chore(deps): drop bssh-russh fork, use upstream russh 0.62.1 #212); guardserver/mod.rs:856→:908; CHANNEL_DATA copyencrypted.rs:842→:1019,1044; CTR key-scheduleblock.rs:80-92→:75-93; bssh-sidesrc/server/mod.rs:196-202→:183-218(..Default::default()at:217) andsrc/server/sftp.rs:757-780→:744-783.russhcontributions or a carried patch; items 1, 2, and 4 remain directly actionable in this repo.