perf(python3): avoid quadratic copying on file writes

[josephbajor]

Repeated small writes from Python currently allocate a new Uint8Array and copy the entire previous file on every extending HOSTFS write. Sequential output therefore incurs quadratic copying, including when Python's own buffering coalesces writes from serializers such as json.dump.

Grow the buffer geometrically and track logical file length separately. For 8,192 consecutive 1 KiB extensions, the previous algorithm copies about 32 GiB of existing contents; doubling capacity reduces that to about 8 MiB. The benchmark below measures the actual Python command and verifies every file it writes.

Implementation

The runtime change is in packages/just-bash/src/commands/python3/worker.ts:

• Track hostLength separately from allocated hostContent.length.
• Double capacity when necessary, copying only logical contents and capping allocation at the existing maxFileSize guard.
• Use logical EOF for reads, append, and SEEK_END; send only logical contents across the bridge on close.
• Return immediately for a zero-byte write. This also fixes an existing edge case where seek(100); write(b'') enlarged a five-byte file to 100 bytes. Native Python leaves it at five bytes.

Buffer capacity can approach twice the logical file length, bounded by the existing file-size limit; the previous allocation is also temporarily live during growth. Existing bridge capacity and write-back-on-close behavior are retained. This changes TypeScript source and adds a changeset and tests. It applies independently on upstream 062ce005c0a7676163852fb6f0c8590cbdaa1d45 (3.4.2).

Reproduce and compare

Use Node 22 or 24 and the repository's pinned pnpm version. Save the following temporary benchmark as /tmp/just-bash-write-benchmark.mjs:

Benchmark script

import { resolve } from "node:path";
import { pathToFileURL } from "node:url";

const { Bash } = await import(
  pathToFileURL(resolve("packages/just-bash/dist/bundle/index.js")).href
);

const bash = new Bash({
  python: true,
  executionLimits: { maxPythonTimeoutMs: 120_000 },
});
const result = await bash.exec(`python3 - <<'PY'
import json
import statistics
import time

chunk = bytes(range(256)) * 4
rows = []
for buffering in [0, 8192]:
    for mib in [1, 2, 4, 8]:
        count = mib * 1024
        expected = chunk * count
        samples = []
        for trial in range(6):
            with open('/tmp/write-benchmark.bin', 'wb', buffering=buffering) as file:
                start = time.perf_counter()
                for _ in range(count):
                    file.write(chunk)
            elapsed = (time.perf_counter() - start) * 1000
            with open('/tmp/write-benchmark.bin', 'rb') as file:
                assert file.read() == expected
            if trial > 0:
                samples.append(elapsed)
        median_ms = statistics.median(samples)
        rows.append({'buffering': buffering, 'MiB': mib, 'writes': count, 'median_ms': round(median_ms, 2) if median_ms else '<1'})
print(json.dumps(rows))
PY`);

if (result.exitCode !== 0 || result.stderr !== "") {
  throw new Error(`Python exited with ${result.exitCode}: ${result.stderr}`);
}

console.log(`Node ${process.version}, ${process.platform}/${process.arch}`);
console.log(
  "1 KiB writes; median of 5 runs after one warmup per size/buffering pair.",
);
console.log(
  "Times include writes and close; exclude Python startup and verification.",
);
console.log(
  "The guest clock has millisecond granularity; '<1' means below its resolution.",
);
console.table(JSON.parse(result.stdout));

Then run from a fresh checkout:

git clone --branch codex/perf-python-file-writes https://github.com/josephbajor/just-bash.git just-bash-write-scaling
cd just-bash-write-scaling
pnpm install --frozen-lockfile

# Upstream baseline
git switch --detach 062ce005c0a7676163852fb6f0c8590cbdaa1d45
pnpm --filter just-bash build
node /tmp/just-bash-write-benchmark.mjs

# This change
git switch codex/perf-python-file-writes
pnpm --filter just-bash build
node /tmp/just-bash-write-benchmark.mjs

# File-behavior regressions
pnpm --filter just-bash test:wasm src/commands/python3/python3.write-scaling.test.ts

The benchmark makes repeated 1 KiB file.write() calls, with both unbuffered files and an 8 KiB Python buffer. It writes to the virtual filesystem, reads back every completed file, and asserts exact byte equality. Timing includes the writes and close, and excludes interpreter startup, opening the file, and verification. Each size/buffering pair gets one warmup followed by five measured runs.

Measured locally on Node 24.10.0, macOS 26.6.2, arm64:

Python buffering	File size	Upstream median	Patched median
Unbuffered	1 MiB	17 ms	1 ms
Unbuffered	2 MiB	59 ms	1 ms
Unbuffered	4 MiB	201 ms	2 ms
Unbuffered	8 MiB	1,927 ms	4 ms
8 KiB	1 MiB	2 ms	<1 ms
8 KiB	2 MiB	9 ms	1 ms
8 KiB	4 MiB	30 ms	2 ms
8 KiB	8 MiB	265 ms	3 ms

The worker deliberately limits its clock to millisecond precision, so small timings are coarse and machine/load dependent. These measurements describe the file-write phase; interpreter startup still contributes to total command latency. Performance measurements live in the standalone benchmark to avoid machine-dependent timing assertions in CI.

Validation

• Nine new regressions cover buffered/unbuffered binary writes, reads and readinto at EOF, overwrites, append after seeking and reopening, sparse zero-filled gaps, empty writes, truncating opens, an odd 1,025-byte file limit, and flushing an 8 MiB file whose allocated capacity exceeds the bridge limit. All nine pass on Node 22.21.0 and 24.10.0. The empty-write test fails on upstream with a reported EOF of 100 instead of 5.
• Full WASM suite: 720 passed, 2 skipped. Built-package tests: 17 passed. Executor package: 74 passed.
• Build, both packages' type checks, pnpm lint:fix, pnpm lint, pnpm knip, and pnpm check:worker-sync pass.
• The broader just-bash suite has 15,610 passed, 98 skipped, and six pre-existing host-filesystem failures in this macOS sandbox. All six were reproduced on unmodified upstream: two symlink-root cleanup failures in cross-fs-security.test.ts, plus four special-mode-bit assertions in read-write-fs.hard-links.test.ts, read-write-fs.mode-and-copy.test.ts, and read-write-fs.test.ts.

[vercel]

@josephbajor is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

[auto-maintain]

🤖 auto-maintain review

Automated, advisory triage for @josephbajor's PR. Facts below are read from the GitHub API.

Check	Result
Author's merged PRs (this repo)	0 — ⚠️ first-time contributor
Account established	✅ (age 3404d · 2 followers · 17 public repos)
Commits signed/verified	⚠️ 0/1
Changeset included	✅ (.changeset/quick-python-writes.md)

Review panel: 🟢 low highest severity

just-bash maintainer code review: 🟢 low

No actionable correctness, compatibility, or maintainability issues found in the complete diff.

General code review: 🟢 low

No actionable correctness or maintainability issues found in the complete diff.

Adversarial security: 🟢 low

No actionable adversarial security issues found in the complete diff.

Adversarial security (second opinion): 🟢 low

Geometric buffer growth in the Python HOSTFS worker is correctly bounded by the existing maxFileSize guard, never exposes spare capacity through reads, seeks, or the SharedArrayBuffer bridge, and introduces no new attack surface; no actionable findings.

Standard Bash and host portability: 🟢 low

No Bash or ordinary-host portability issues found in the complete diff.

_{Posted by auto-maintain. This automated code review is advisory; a human maintainer makes the call.}