feat(tools): add SFTP benchmark, flamegraph, and interop harness

.gitignore

@@ -28,3 +28,7 @@ Thumbs.db
 references/
 vendor/
 bssh-server.yaml
+
+# tools/bench artifacts
+tools/bench/interop/sshj/lib/
+*.class

tools/bench/README.md

@@ -0,0 +1,96 @@
+# bssh benchmark and interop harness
+
+Scripts for measuring `bssh-server` SFTP performance relative to OpenSSH on
+the same host, capturing CPU flamegraphs, and validating interoperability
+with third-party SFTP client stacks (sshj, paramiko).
+
+This harness produced the measurements recorded in issues
+[#225](https://github.com/lablup/bssh/issues/225) and
+[#226](https://github.com/lablup/bssh/issues/226), and found the paramiko
+read-path hang tracked in [#227](https://github.com/lablup/bssh/issues/227).
+Background: [#187](https://github.com/lablup/bssh/issues/187) and PR
+[#224](https://github.com/lablup/bssh/pull/224).
+
+## Layout
+
+| Script | Purpose |
+|---|---|
+| `bench.sh` | Relative SFTP throughput: OpenSSH sshd vs bssh-server (optionally plus a baseline bssh build) |
+| `profile.sh` | `perf` CPU profile and flamegraph of bssh-server during uploads |
+| `interop/paramiko_test.py` | Round-trip integrity check with paramiko (Python SSH stack, used by Ansible) |
+| `interop/sshj/run-sshj.sh` | Round-trip integrity and negotiation check with sshj (Java SSH stack, used by Cyberduck) |
+| `lib.sh` | Shared setup: keys, configs, test file, server lifecycle, timed transfers |
+
+All servers run on loopback only, with keys and configs generated fresh under
+`BENCH_DIR`. Nothing touches the user's real SSH configuration.
+
+## Prerequisites
+
+- A release build of the server: `cargo build --release --bin bssh-server`
+  (for `profile.sh`, prefer `CARGO_PROFILE_RELEASE_DEBUG=true cargo build --release --bin bssh-server` so stacks resolve)
+- OpenSSH client (`ssh`, `sftp`, `ssh-keygen`); OpenSSH server (`/usr/sbin/sshd`) for `bench.sh`
+- `profile.sh`: `perf`, `kernel.perf_event_paranoid <= 2`, and optionally `cargo install inferno` for SVG output
+- `interop/paramiko_test.py`: `pip install paramiko`
+- `interop/sshj/`: a JDK (`javac`/`java`, set `JAVA_HOME` if not on PATH), then `./fetch-deps.sh` once to download the pinned jars into `lib/` (not committed)
+
+## Configuration
+
+Everything is an environment variable with a default (see `lib.sh`):
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `BENCH_DIR` | `${TMPDIR:-/dev/shm}/bssh-bench-$USER` | Working directory (keys, configs, test file, results) |
+| `BSSH_SERVER_BIN` | `target/release/bssh-server` | Server binary under test |
+| `BSSH_BASELINE_BIN` | unset | Optional second binary for before/after comparison (`bench.sh`) |
+| `BSSH_PORT` / `SSHD_PORT` | `22200` / `22022` | Loopback ports |
+| `FILE_SIZE_MIB` | `2048` | Test file size |
+| `RUNS` | `3` | Timed runs per direction |
+| `SERVER_CORE` / `CLIENT_CORE` | unset (no pinning) | `taskset` core lists; pin the server to one core to emulate a single-core container |
+| `CIPHER` | `chacha20-poly1305@openssh.com` | Cipher forced on OpenSSH-client transfers |
+| `PROFILE_UPLOADS`, `PERF_FREQ`, `PERF_CALLGRAPH` | `3`, `499`, `fp` | `profile.sh` knobs |
+
+## Examples
+
+```bash
+# Single-core relative benchmark (the #187 scenario), cores 8 and 9:
+SERVER_CORE=8 CLIENT_CORE=9 tools/bench/bench.sh
+
+# Before/after against an older build:
+BSSH_BASELINE_BIN=/path/to/old/bssh-server tools/bench/bench.sh
+
+# Flamegraph of the upload path:
+CARGO_PROFILE_RELEASE_DEBUG=true cargo build --release --bin bssh-server
+SERVER_CORE=8 CLIENT_CORE=9 tools/bench/profile.sh
+
+# sshj interop (2 GiB round trip, negotiation log included):
+tools/bench/interop/sshj/fetch-deps.sh
+tools/bench/interop/sshj/run-sshj.sh
+
+# paramiko interop; --no-prefetch works today, the default reproduces #227:
+. tools/bench/lib.sh && bench_setup && start_bssh
+python3 tools/bench/interop/paramiko_test.py \
+  --user "$USER" --key "$BENCH_DIR/bench_key_rsa" \
+  --file "$BENCH_DIR/testfile_2048M" --remote-dir "$BENCH_DIR/up" --no-prefetch
+stop_servers
+```
+
+## Methodology notes
+
+- Loopback removes the NIC bottleneck, so absolute numbers exceed any real
+  network; the point is the ratio between servers measured under identical
+  conditions. Pin both servers to the same single core (`SERVER_CORE`) to
+  reproduce the CPU-bound single-core container scenario from #187.
+- Every transfer is verified byte-for-byte (`cmp`) before it counts.
+- Keep `BENCH_DIR` on tmpfs (the default) so disk I/O does not dominate.
+- Timing includes the SSH handshake; with multi-GiB files its share is a few
+  percent and identical across servers.
+
+## Known issues
+
+- `interop/paramiko_test.py` with default prefetch currently hangs against
+  bssh-server (pre-existing server bug, not a #224 regression); tracked in
+  [#227](https://github.com/lablup/bssh/issues/227). Once fixed, the default
+  invocation doubles as the regression test.
+- FileZilla and WinSCP are GUI-only and cannot run headless; both use
+  PuTTY-derived transports, so a `psftp` (putty-tools) check is the closest
+  scriptable proxy if needed.

tools/bench/bench.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+# Relative SFTP throughput: OpenSSH sshd vs bssh-server on the same host.
+#
+# Runs both servers on loopback (optionally pinned to the same CPU core to
+# emulate a single-core container) and measures timed uploads and downloads
+# with the OpenSSH sftp client. Set BSSH_BASELINE_BIN to also measure an
+# older bssh-server build for a before/after comparison.
+#
+# Example (single-core scenario, big cores 8 and 9):
+#   SERVER_CORE=8 CLIENT_CORE=9 tools/bench/bench.sh
+#
+# See README.md for all configuration variables and methodology notes.
+
+set -u
+cd "$(dirname "$0")"
+. ./lib.sh
+
+bench_setup
+
+measure() {
+  local label=$1 port=$2 dir vals i
+  for dir in up dl; do
+    vals=""
+    for i in $(seq 1 "$RUNS"); do
+      vals="$vals $(sftp_xfer "$port" "$dir")"
+    done
+    echo "RESULT $label $dir$vals"
+  done
+}
+
+echo "# cipher: $CIPHER, file: ${FILE_SIZE_MIB} MiB, runs: $RUNS, server core: ${SERVER_CORE:-unpinned}, client core: ${CLIENT_CORE:-unpinned}"
+echo "# results are MiB/s per run; every transfer is integrity-checked with cmp"
+
+if start_sshd; then
+  measure openssh "$SSHD_PORT"
+  stop_servers
+fi
+
+if start_bssh; then
+  measure bssh "$BSSH_PORT"
+  stop_servers
+fi
+
+if [ -n "${BSSH_BASELINE_BIN:-}" ]; then
+  if start_bssh "$BSSH_BASELINE_BIN"; then
+    measure bssh-baseline "$BSSH_PORT"
+    stop_servers
+  fi
+fi

tools/bench/interop/paramiko_test.py

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""SFTP interop check against bssh-server using paramiko.
+
+Uploads a file, downloads it back, and verifies byte-for-byte integrity.
+
+Known issue: with paramiko's default prefetch (a burst of READ requests for
+the whole file), the download currently hangs against bssh-server; see
+https://github.com/lablup/bssh/issues/227. Until that is fixed, the default
+invocation reproduces the bug (bounded by --timeout), and --no-prefetch
+exercises the sequential read path, which completes.
+
+Requires: pip install paramiko
+"""
+
+import argparse
+import filecmp
+import signal
+import sys
+import time
+
+import paramiko
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    parser.add_argument("--host", default="127.0.0.1")
+    parser.add_argument("--port", type=int, default=22200)
+    parser.add_argument("--user", required=True)
+    parser.add_argument("--key", required=True, help="RSA private key in PEM format")
+    parser.add_argument("--file", required=True, help="local test file to round-trip")
+    parser.add_argument("--remote-dir", required=True, help="writable directory on the server")
+    parser.add_argument(
+        "--no-prefetch",
+        action="store_true",
+        help="download sequentially instead of with paramiko's read-ahead burst",
+    )
+    parser.add_argument(
+        "--timeout",
+        type=int,
+        default=120,
+        help="hard limit in seconds for the whole round trip (default 120)",
+    )
+    args = parser.parse_args()
+
+    def on_timeout(signum, frame):
+        print(f"TIMEOUT after {args.timeout}s (see lablup/bssh#227)", flush=True)
+        sys.exit(2)
+
+    signal.signal(signal.SIGALRM, on_timeout)
+    signal.alarm(args.timeout)
+
+    transport = paramiko.Transport((args.host, args.port))
+    transport.banner_timeout = 15
+    transport.connect(
+        username=args.user, pkey=paramiko.RSAKey.from_private_key_file(args.key)
+    )
+    print("remote version:", transport.remote_version, flush=True)
+    sftp = paramiko.SFTPClient.from_transport(transport)
+    channel = sftp.get_channel()
+    print(
+        "server max packet:", channel.out_max_packet_size,
+        "window:", channel.out_window_size,
+        flush=True,
+    )
+
+    remote = f"{args.remote_dir}/paramiko_roundtrip"
+    local_back = args.file + ".paramiko_back"
+
+    t0 = time.time()
+    sftp.put(args.file, remote)
+    t1 = time.time()
+    sftp.get(remote, local_back, prefetch=not args.no_prefetch)
+    t2 = time.time()
+    print(
+        f"put {t1 - t0:.1f}s, get {t2 - t1:.1f}s (prefetch={not args.no_prefetch})",
+        flush=True,
+    )
+
+    sftp.remove(remote)
+    sftp.close()
+    transport.close()
+    signal.alarm(0)
+
+    if not filecmp.cmp(args.file, local_back, shallow=False):
+        print("INTEGRITY FAIL", flush=True)
+        return 1
+    print("OK", flush=True)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/bench/interop/sshj/SshjTest.java

@@ -0,0 +1,56 @@
+import net.schmizz.sshj.SSHClient;
+import net.schmizz.sshj.sftp.RemoteResourceInfo;
+import net.schmizz.sshj.sftp.SFTPClient;
+import net.schmizz.sshj.transport.verification.PromiscuousVerifier;
+
+import java.util.List;
+
+/**
+ * SFTP interop check against bssh-server using sshj (the SSH stack used by
+ * Cyberduck and other Java tools). Uploads a file, downloads it back, and
+ * prints timing; integrity is verified by the calling script with cmp.
+ *
+ * Args: port user keyPath localFile remoteUploadPath localDownloadPath listDir
+ */
+public class SshjTest {
+    public static void main(String[] args) throws Exception {
+        String host = "127.0.0.1";
+        int port = Integer.parseInt(args[0]);
+        String user = args[1];
+        String keyPath = args[2];
+        String localFile = args[3];
+        String remoteUp = args[4];
+        String localDown = args[5];
+        String listDir = args[6];
+
+        SSHClient ssh = new SSHClient();
+        ssh.addHostKeyVerifier(new PromiscuousVerifier());
+        ssh.connect(host, port);
+        try {
+            ssh.authPublickey(user, keyPath);
+            System.out.println("NEGOTIATED " + ssh.getTransport().getClientVersion()
+                    + " <-> " + ssh.getTransport().getServerVersion());
+            SFTPClient sftp = ssh.newSFTPClient();
+
+            List<RemoteResourceInfo> ls = sftp.ls(listDir);
+            System.out.println("LS_ENTRIES " + ls.size());
+
+            long t0 = System.nanoTime();
+            sftp.put(localFile, remoteUp);
+            long t1 = System.nanoTime();
+            sftp.get(remoteUp, localDown);
+            long t2 = System.nanoTime();
+
+            long size = sftp.stat(remoteUp).getSize();
+            System.out.println("REMOTE_SIZE " + size);
+            System.out.printf("UPLOAD_MIBS %.0f%n", size / 1048576.0 / ((t1 - t0) / 1e9));
+            System.out.printf("DOWNLOAD_MIBS %.0f%n", size / 1048576.0 / ((t2 - t1) / 1e9));
+
+            sftp.rm(remoteUp);
+            sftp.close();
+        } finally {
+            ssh.disconnect();
+        }
+        System.out.println("SSHJ_OK");
+    }
+}

tools/bench/interop/sshj/fetch-deps.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# Download the pinned sshj jars into lib/ (not committed to the repository).
+set -eu
+cd "$(dirname "$0")"
+mkdir -p lib
+M=https://repo1.maven.org/maven2
+DEPS="
+com/hierynomus/sshj/0.38.0/sshj-0.38.0.jar
+com/hierynomus/asn-one/0.6.0/asn-one-0.6.0.jar
+org/bouncycastle/bcprov-jdk18on/1.78.1/bcprov-jdk18on-1.78.1.jar
+org/bouncycastle/bcpkix-jdk18on/1.78.1/bcpkix-jdk18on-1.78.1.jar
+org/bouncycastle/bcutil-jdk18on/1.78.1/bcutil-jdk18on-1.78.1.jar
+org/slf4j/slf4j-api/2.0.13/slf4j-api-2.0.13.jar
+org/slf4j/slf4j-simple/2.0.13/slf4j-simple-2.0.13.jar
+net/i2p/crypto/eddsa/0.3.0/eddsa-0.3.0.jar
+"
+for dep in $DEPS; do
+  jar=$(basename "$dep")
+  [ -f "lib/$jar" ] && continue
+  echo "fetching $jar"
+  curl -sfL -o "lib/$jar" "$M/$dep"
+done
+echo "done: $(ls lib/*.jar | wc -l) jars in $(pwd)/lib"

tools/bench/interop/sshj/run-sshj.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+# Compile and run the sshj interop check against a locally started bssh-server.
+#
+# Requires a JDK (javac + java). Set JAVA_HOME to use a specific one, and run
+# ./fetch-deps.sh once to populate lib/. See ../../README.md for the
+# configuration variables shared with the rest of the harness.
+
+set -u
+cd "$(dirname "$0")"
+. ../../lib.sh
+
+JAVA=${JAVA_HOME:+$JAVA_HOME/bin/}java
+JAVAC=${JAVA_HOME:+$JAVA_HOME/bin/}javac
+command -v "$JAVAC" >/dev/null 2>&1 || { echo "FATAL: javac not found; install a JDK or set JAVA_HOME" >&2; exit 1; }
+[ -f lib/sshj-0.38.0.jar ] || { echo "FATAL: jars missing; run ./fetch-deps.sh first" >&2; exit 1; }
+
+CP=$(ls lib/*.jar | tr '\n' ':')
+"$JAVAC" -cp "$CP" SshjTest.java || exit 1
+
+bench_setup
+start_bssh || exit 1
+
+"$JAVA" -cp "$CP." \
+  -Dorg.slf4j.simpleLogger.log.net.schmizz.sshj.transport=debug \
+  -Dorg.slf4j.simpleLogger.log.net.schmizz.sshj.connection=debug \
+  SshjTest "$BSSH_PORT" "$USER_NAME" "$BENCH_DIR/bench_key_rsa" \
+  "$TEST_FILE" "$BENCH_DIR/up/sshj_up" "$BENCH_DIR/dl/sshj_down" "$BENCH_DIR" \
+  > "$BENCH_DIR/sshj-run.out" 2> "$BENCH_DIR/sshj-debug.log"
+RC=$?
+stop_servers
+
+echo "=== exit: $RC ==="
+cat "$BENCH_DIR/sshj-run.out"
+echo "=== integrity ==="
+if cmp -s "$TEST_FILE" "$BENCH_DIR/dl/sshj_down"; then
+  echo "ROUNDTRIP_INTEGRITY_OK"
+else
+  echo "ROUNDTRIP_INTEGRITY_FAIL"
+  RC=1
+fi
+rm -f "$BENCH_DIR/dl/sshj_down"
+echo "=== negotiation (client debug log) ==="
+grep -iE "Negotiated algorithms|Sending SSH_MSG_KEXINIT|Received SSH_MSG_KEXINIT" "$BENCH_DIR/sshj-debug.log" | head -5
+exit $RC