Rust phase 3: native .iafbt bundle I/O

[MDUYN]

Goal

Native Rust reader/writer for the .iafbt bundle format so backtests can be loaded, mutated and rewritten without crossing the Python ↔ Rust boundary for every snapshot/trade row. This unlocks the streaming pipeline in the next phase.

Tasks

• Document the on-disk format (already partly defined in investing_algorithm_framework/domain/backtesting/bundle.py): msgpack manifest + parquet timeseries + JSON metadata. Treat bundle.py as the spec.
• Rust side:
  • rmp-serde for msgpack manifest read/write.
  • polars (or arrow2) for parquet portfolio snapshots / trades — zero-copy where possible.
  • Schema version check; fail closed on unknown versions.
• PyO3 functions:
  • read_bundle(path) -> NativeBacktestHandle (opaque handle holding the parsed bundle in Rust memory; Python never sees the full payload).
  • write_bundle(handle, dst_path, include_ohlcv: bool).
  • bundle_summary(path) -> dict for fast index rebuild without loading timeseries.
• Round-trip parity test: every fixture bundle under tests/resources/ must read in Rust → write to a tempdir → re-open in Python and produce a Backtest byte-identical to the original (manifests) and equity-curve-identical (timeseries).
• Benchmark vs the current Python bundle.py reader: target ≥ 5× faster cold read on a 5-year hourly bundle.

Acceptance criteria

• Round-trip parity tests green.
• The Python Backtest.open() path can optionally use the native reader behind the IAF_NATIVE flag, with byte-for-byte equivalent output.
• Benchmark report committed.

Dependencies

• Phase 1 (crate scaffold).

Notes

• Keep all schema knowledge in one place. If Rust and Python both encode/decode independently we will get drift. Prefer a shared spec doc and golden fixtures.

Part of the Epic: hybrid Python + Rust acceleration.

[MDUYN]

Status update

The storage pipeline has advanced substantially, but this specific native bundle-I/O phase remains partially addressed.

Delivered on the Python/framework side:

• Lazy serialized history records and batched metric Parquet encoding.
• Streamed MessagePack/compression and atomic bundle replacement.
• Disk-backed event histories and worker bundle-path handoff.
• Language-neutral/versioned record schemas, manifests, bounded chunk writing, and round-trip tests.

Still missing from this issue's acceptance criteria:

• A Rust .iafbt reader/writer owning the complete bundle payload.
• NativeBacktestHandle, native read_bundle, write_bundle, and bundle_summary APIs.
• Optional dispatch from Backtest.open() to native bundle I/O.
• The proposed native-vs-Python cold-read benchmark target.

Current bundle open/merge and some result transport paths remain eager. Details are tracked in docs/design/backtest-streaming-storage-todo.md.

Leaving this issue open because the original native-I/O boundary is not yet implemented.