fix(sftp): server read path hangs under paramiko's unbounded READ prefetch; sequential reads incur ~37 ms per request

[inureyes]

Summary

paramiko's SFTPClient.get() with its default prefetch hangs indefinitely when downloading from bssh-server, while the same code completes in 0.1 s against OpenSSH sshd on the same host. This is NOT a regression from PR #224: it reproduces identically on pre-#224 (fbbf226) and post-#224 (e4f2810) builds. Additionally, even strictly sequential reads (prefetch=False) complete but take ~37 ms per 32 KiB request (8 MiB in 9.4 s, about 0.85 MiB/s), which points at a response-flush delay on the server read path and is likely the same root cause. Uploads via paramiko are unaffected (8 MiB in 0.2 s). paramiko is the SSH stack under Ansible and many Python tools, so this has real-world impact.

Environment

• bssh-server built from main at e4f2810 (also reproduced with fbbf226), russh 0.62.1
• Loopback 127.0.0.1 on Linux 6.17 (ARM), files on tmpfs
• paramiko 2.12.0 / Python 3, publickey auth (RSA 3072)
• Default server config (compression off)
• Control: OpenSSH 9.6p1 sshd with internal-sftp, same host, same key, same file

Reproduction

import paramiko
t = paramiko.Transport(("127.0.0.1", 22200))
t.connect(username=USER, pkey=paramiko.RSAKey.from_private_key_file(KEY))
sftp = paramiko.SFTPClient.from_transport(t)
sftp.put(SRC_8MIB, REMOTE)      # OK, 0.2 s
sftp.get(REMOTE, LOCAL)         # HANGS forever (default prefetch=True)
sftp.get(REMOTE, LOCAL, prefetch=False)  # completes, but 9.4 s for 8 MiB

Against OpenSSH sshd on the same host, the identical script completes put in 0.2 s and get in 0.1 s.

Evidence

• paramiko DEBUG log stops right after open(b'...', 'rb') -> <handle>; no READ responses ever arrive at the client (with default prefetch).
• Negotiation is healthy: server advertises max packet 65535 and window 8 MiB (post-perf(sftp): raise channel sizing, pipeline server writes, tune release build #224 values); channel open, auth, listdir, stat, upload all work.
• OpenSSH sftp CLI and sshj 0.38.0 both upload and download multi-GiB files against the same server without stalls and with byte-for-byte integrity (verified during test(sftp): third-party client interop validation for the raised channel sizing and pipelined write path (FileZilla, WinSCP, sshj/Cyberduck) #226 interop validation), so the hang is specific to the request pattern paramiko uses.
• Reproduces with the pre-perf(sftp): raise channel sizing, pipeline server writes, tune release build #224 serial SFTP loop and with the post-perf(sftp): raise channel sizing, pipeline server writes, tune release build #224 pipelined loop, so the defect predates the perf(sftp): raise channel sizing, pipeline server writes, tune release build #224 restructuring.
• Secondary cosmetic finding: when the paramiko session closes, the server logs ERROR bssh::server: Session error error=Connection reset by peer (os error 104); an ordinary client disconnect should not log at ERROR level.

Analysis

Hypothesis (not yet confirmed): paramiko's prefetch() fires SSH_FXP_READ requests for the entire file immediately without bounding the number of outstanding requests (unlike OpenSSH's sftp, which caps outstanding requests at 64, and sshj, which uses bounded read-ahead). For an 8 MiB file that is a burst of 256 READ requests with distinct offsets on one channel. The server's read-response path stalls under that burst.

The ~37 ms per request observed even in the strictly sequential case suggests READ responses are not flushed promptly (a deferred flush or missing flush on the response path); under the burst pattern, delayed responses plus SSH channel window backpressure can deadlock: the client stops growing the window because no data arrives, and the server never flushes what it has.

Investigation should start in the SFTP server request loop in crates/bssh-russh-sftp/src/server/mod.rs (both the current pipelined loop and the logic it inherited) and the interaction with russh channel window handling on the tx path.

Acceptance Criteria

• paramiko get() with default prefetch completes with byte-for-byte integrity for 8 MiB and for a multi-GiB file.
• Sequential reads (prefetch=False) no longer exhibit the ~37 ms-per-request stall (loopback per-request latency should be sub-millisecond).
• A regression test covers bursted/out-of-order READ requests (either a Rust-side test simulating the paramiko prefetch pattern, or the paramiko interop script from tools/bench once merged).
• No regressions for OpenSSH sftp and sshj clients (up/down, integrity).
• (Secondary) client-initiated disconnects no longer log at ERROR level.

Related

• Found during test(sftp): third-party client interop validation for the raised channel sizing and pipelined write path (FileZilla, WinSCP, sshj/Cyberduck) #226 (third-party client interop validation for perf(sftp): bssh-server throughput ~50% of OpenSSH from 32 KiB channel fragmentation, serial server-side writes, and per-packet copies #187 / PR perf(sftp): raise channel sizing, pipeline server writes, tune release build #224)
• Interop precedent: fix(sftp): sshj/Cyberduck clients dropped on initial REALPATH/STAT via russh strict-parser SshEncoding::Length on incoming decode #215
• Not caused by perf(sftp): raise channel sizing, pipeline server writes, tune release build #224 (bisected: reproduces on fbbf226)