Feat/writer crash recovery

.gitignore

@@ -1,5 +1,6 @@
 # Build output
 build/
+build-asan/
 build-bench/
 dist/
 out/

CHANGELOG.md

@@ -39,6 +39,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **reader/api**: **bucket names restored on read** -- bucket names never hit the wire (both formats serialize buckets positionally), so deserialized frames used to come back with an empty `bucket_map`.  The reader now stamps `bucket_map` from the embedded schema's `"buckets"` array when chunks are deserialized, so by-name lookups (`Frame::GetBucket(name)` const) and `bucket_map` iteration work on read frames.  Replays written without a `"buckets"` array keep the old positional-only behavior
 - **common/schema**: **array & map pre-sizing** -- extends the existing scalar pre-sizing (`type_max_indices`) so declared *array* and *map* fields are also materialized on load.  `SchemaStruct` / `StructSchemaCache` gain `array_max_indices` (max Array-container index per `FieldType`) and `map_max_index` (count of Struct-valued map fields); `Helpers::ResizeContainerToMaxIndices` now pre-creates one empty subarray per declared array field in the matching `FlatArray` (via new `FlatArray::EnsureSubArrayCount`) and pre-sizes `map_properties`.  A declared-but-unpopulated array/map field is now present-and-empty instead of absent, symmetric with scalars.  Applies wherever scalar pre-sizing already ran (the four frame loaders + `schema_dynamic_loader`); the loader CRTP hook `GetTypeMaxIndices` became `GetStructSizing` (returns the `StructSchemaCache`)
 - **reader/api**: **declared-empty arrays restored on read** -- an array with no data is not serialized (nothing to store), so deserialized entities used to come back missing those subarrays even for schema-declared array fields (maps already round-trip their slot count). The reader now re-creates the declared-but-empty subarrays from the embedded schema (`Helpers::EnsureDeclaredArrays`, grow-only, recursing into nested structs / struct-array elements / map values) so a read frame mirrors the array layout of an ingest-loaded frame. Arrays that carry data round-trip unchanged (their `offsets` already encode empty subarrays); scalars and maps are untouched
+- **writer/durability**: **crash recovery for the file sink** -- a recording that dies before `Stop()` (process crash or power loss, mid-chunk or mid-frame, before the body/footer are closed) can now be recovered up to the last recorded frame instead of losing the whole session.  Three cooperating pieces, all file-sink only (the network sink accepts the per-frame hook as a no-op):
+  - **per-chunk integrity + durability** -- `ChunkedFileSink::Config` gains `durable_writes` (default **on**: `fsync`/`_commit` each chunk and journal record to physical media via the new `DurableFile` FILE* wrapper; set **off** to only `fflush` to the OS -- survives a process crash but not power loss) and `enable_recovery_journal` (default **on**).  Every chunk carries an **xxHash64 `checksum`** of its on-disk payload, added to `ChunkIndexEntry` / `ChunkIndexData` and to both the FlatBuffers (`vtx_schema.fbs`) and Protobuf (`vtx_schema.proto`, field 6) footer schemas, so a torn or corrupt chunk is detectable on recovery.  The reader parses and exposes it; FlatBuffers header/footer parsing gained a `flatbuffers::Verifier` guard so a truncated footer is rejected cleanly instead of reading out of bounds
+  - **recovery journal** -- a single write-ahead sidecar **`<file>.recovery`** (`sdk/include/vtx/writer/policies/sinks/recovery_journal.h`) holds a typed, self-validating record stream (each record `[u8 type][u32 len][payload][u64 xxHash64]`, so a torn last record from a mid-write crash is detected and dropped): an `S` record (written once) carries the writer's timing parameters `{fps, is_increasing}`, `C` records commit a durable chunk's index entry, `T` records carry each committed frame's exact `{game_time, created_utc}`, and `F` records carry each in-flight (un-flushed) frame's times + serialized payload.  The log is strictly **append-only** in the crash-critical path -- a frame appends an `F` record; a flush appends the chunk's `C` + `T` records -- so at every instant the file is a valid prefix and there is no window in which a fsync'd chunk is described by neither its `F` records nor its `C` record.  Ordering is **data-before-journal**: a chunk is fsync'd into the `.vtx` first, then its `C`/`T` records are appended and fsync'd.  A committed chunk's now-redundant `F` records (repair dedups them by frame index) are reclaimed by periodic **compaction** -- the log is rewritten (all `C`/`T` + only the still-pending `F`) into a temp file that **atomically replaces** the sidecar, so a crash during compaction leaves either the old or the new complete journal.  The writer journals every frame as it is recorded (`ReplayWriter::TryRecordFrame` -> new `SinkPolicy::JournalFrame` hook; timing via `SinkPolicy::JournalTiming`), so a crash between flushes still recovers the pending batch.  If the journal cannot be opened cleanly, journaling is disabled and the torn sidecar is removed (a half-written journal would otherwise block repair).  On a clean `Close()` the footer is written and the `.recovery` is deleted -- its presence at open time signals an unclean shutdown
+  - **repair** -- **`VTX::RepairReplayFile(path)`** (`sdk/include/vtx/writer/core/vtx_replay_recovery.h`) reconstructs a valid, readable file from a footerless `.vtx` + its journal: it verifies each committed chunk's checksum (dropping a torn/corrupt tail), re-appends the in-flight frames as chunks, rebuilds the seek table and the **exact per-frame times**, and synthesizes the footer -- including the **derived time data** (`duration_seconds`, timeline **gaps**, game **segments**), reconstructed from the journaled timing (`S` record) with the same expressions `VTXGameTimes` uses, so a recovered footer matches a clean `Stop()` **exactly** (manual segment marks are the one thing not journaled).  Recovery is deliberately **not automatic on open** -- the user-driven flow is `ReplayNeedsRecovery(path)` (cheap sidecar check) / `RecoveryJournalPath(path)` (locate the sidecar) then `RepairReplayFile(path)`.  A file that already ends with a valid footer plus a leftover journal (a crash between the footer fsync and the journal delete) is detected and preserved untouched; a journal whose recorded format magic disagrees with the file, or whose header is unreadable, is refused rather than applied; a **recording still being written is refused** without touching it (on Windows the writer holds a deny-write handle, so repair's truncate fails cleanly).  Returns a `RepairResult` (`was_clean` / `repaired` / `recovered_chunks` / `recovered_frames` / `error`).  Both FlatBuffers and Protobuf files are supported
+- **tests**: **`tests/writer/test_crash_recovery.cpp`** -- 65 cases covering the full crash-window matrix: recovery between chunks (single- and multi-frame), between frames (in-flight batch, and nothing-flushed-yet), a torn tail chunk, a partial chunk written before its journal record, a checksum-detected corrupt chunk, a torn pending-frame record, a **torn chunk-commit record** (the batch falls back to its still-present `F` records -- the append-only guarantee), a crash mid-footer-write, a leftover-journal-over-valid-footer, a **stale compaction temp**, a mismatched journal format, a clean file (no-op), a header-only crash (0-frame result), the lingering-`F`-record dedup guard, compaction reclaim, the recovery helpers, exact per-frame time preservation, a **live-recording repair refusal** (Windows), and the FlatBuffers + Protobuf repair paths.  Crash states are fabricated byte-faithfully through the same `RecoveryJournal` API the sink uses, plus **end-to-end tests through the real writer** (a raw `ReplayWriter` dropped without `Stop()`) for both formats that require the recovered footer to match a cleanly-stopped control **exactly**: per-frame `game_time` + `created_utc`, `duration_seconds`, timeline gaps, game segments, and per-entity `content_hash` -- and require the recovered file to pass whole-replay `ValidateReplayFile` and cache-hostile out-of-order seeks across the committed-chunk/recovered-chunk boundary.  Also covered: the sink's **full config matrix** (`durable_writes` x `b_use_compression`) with frames large enough that **zstd compression genuinely engages** (committed chunks and journaled `F` payloads take the compressed path), a **repair interrupted mid-run and re-run** (idempotent), a **second repair on an already-repaired file** (clean no-op, byte-identical), a **journal-opted-out crash** (repair reports clean and leaves the footerless file byte-identical; the reader rejects it gracefully), a **session-start crash** (torn `S` record + header-only `.vtx` -> valid 0-frame file), **torn `T` records** (frame times fall back to the still-present `F` records -- exact, not zeroed), a **0-byte journal** (refused, main file untouched), **compaction through the real sink** under crash (new `Config::journal_compact_threshold_bytes`, 0 = default), **map-container frames** recovered intact from `F` records, a **non-ASCII path** through the full sidecar flow, **hostile checksummed journal records** (a `C` offset inside the header region, a `C` size at or below its own length prefix, an `F` with an empty payload -- each dropped safely, degrading to a valid file), a **torn main-file header** alongside a valid journal (refused, bytes untouched), and a **decreasing-time recording** (`is_increasing = false` -- the other branch of the segment reconstruction) matching its clean control.  Scale + pipeline coverage: a **stress run** (2030 multi-entity frames, 40 committed chunks + a 30-frame in-flight batch -- every timestamp exact, whole-replay validation clean), **rejected frames interleaved mid-recording** (timer rejections leave no trace and do not desync the journal's frame indexing), and a **post-processor recording** (journaled `F` records capture the frame as it would hit disk -- post-mutation -- proven by a pending frame that only ever existed in the journal).  Lifecycle + hygiene: a **double-crash lifecycle** over the same path (crash -> repair -> re-record -> crash -> repair, second recovery reflects only the second session), a **stale sidecar under journaling opt-out** (a session with `enable_recovery_journal = false` removes any leftover `.recovery` at start so it cannot masquerade as recovery state), a **foreign `.recovery` file** (no `VTXR` magic -- never deleted by repair, even on the clean-file path), a **hostile out-of-range `T` record** (ignored by the bounds check), the **map crash-recovery path on Protobuf** as well as FlatBuffers, a **stale journal from a different recording** over a replaced file (per-chunk checksums reject the foreign chunks -- graceful degradation to a valid empty file), a **read-only crashed file** (repair refuses without consuming the journal; succeeds untouched once the file is writable), a **no-time-registry recording** (`EGameTimeType::None`, fully FPS-synthesized timeline recovered exactly), and **byte-budget chunk splitting** (`max_bytes`, the other `ThresholdChunkPolicy` branch) through crash + recovery.  The maximal guarantee is pinned by **`BoundaryCrashRecoversByteIdenticalFile`** (FlatBuffers **and** Protobuf, plus a 120-frame large-footer variant that exercises the compressed-footer path): a crash exactly at a chunk boundary recovers to a file that is **bit-for-bit identical** to a clean `Stop()` with the same inputs -- repair passes the footer's time vectors exactly as `Stop()` does (present-but-empty rather than absent) and compresses the synthesized footer exactly as the sink's `Close()` would (the `S` record now also journals `{use_compression, compression_level}`; journal version 4).  A journal with an **incompatible version field** is refused outright, main file untouched.  On top of the hand-picked windows, a **brute-force sweep suite** (`CrashRecoverySweep`) proves the "any crash point" claim literally: the journal truncated at **every byte length**, the main `.vtx` truncated at **every byte length**, and **every journal byte flipped** one at a time -- thousands of repair runs, none may crash, and every claimed repair must open and agree with its own reported frame count and footer times.  `ReadValid` now bounds each record's payload allocation by the journal's **actual file size** (a corrupt length field can no longer trigger a giant transient allocation), pinned by `HostileRecordLengthIsBoundedByFileSize`; the truncation sweep also runs over the **post-compaction journal layout**.  Finally, `CrashRecoveryProcess` (Windows) performs a **genuine process kill**: vtx_tests spawns itself as a child in a hidden endless-writer mode and `TerminateProcess`es it mid-recording -- no destructors, no fclose, handles abandoned to the kernel -- then repairs and verifies frame content and exact footer times, in **both** durability modes (this is the honest validation of the flush-only claim that data handed to the OS survives a process death) -- plus a kill under an **aggressive compaction cadence** (a full close/rewrite/atomic-rename cycle per commit), so real process death lands amid compaction traffic and the atomic-rename guarantee holds, and a **varied-progress kill soak** (kills landing just after the first journaled frames, around the first flush, amid compaction cycles, and several chunks deep -- alternating durability modes).  The recovered output was also smoke-checked through the end-user toolchain: `vtx_cli` opens a repaired file and serves `info` / `footer` / `frame` normally.  `RecoveredFileTranscodesToCleanChunks` pins the **normalization workflow** for salvaged files (open -> drain frames with footer times -> re-record): the one-frame recovery chunks become proper chunks, `created_utc` round-trips exactly, and `game_time` drifts at most 1 tick (100 ns) per frame through the float-seconds register -- the documented transcode caveat.  `BM_ReaderRecoveredTail` (benchmarks) measures the salvage cost that motivates it: a 200-frame one-frame-chunk tail reads sequentially ~3x slower than clean chunking
+- **benchmarks**: new **`benchmarks/bench_crash_recovery.cpp`** (`BM_WriterDurabilityTier`) quantifies the writer hot-path cost of the recovery defaults across the three durability tiers (journal off / journal + flush-only / journal + fsync).  Reference numbers on an NVMe dev machine, 200 small frames per iteration: **~19 us/frame** with no journal, **~30 us/frame** flush-only (process-crash safe -- effectively free), **~289 us/frame** with the default per-operation fsync (power-loss safe; ~1.7% of a 60 fps frame budget on NVMe -- on spinning disks prefer `durable_writes = false`).  The byte-identity tests now compare from the end of the header on (the header embeds a second-granularity recording timestamp, so two separately recorded sessions may legitimately differ there; repair never rewrites the header)
+- **docs**: **`docs/SDK_API.md`** "Writing Replays" gains a **"Crash recovery"** section -- the `.recovery` sidecar model, the user-driven `ReplayNeedsRecovery` / `RepairReplayFile` / `RecoveryJournalPath` flow with `RepairResult` semantics, the exact-times guarantee, the refusal cases, and the sink durability knob table (`durable_writes` / `enable_recovery_journal` / `journal_compact_threshold_bytes`).  **`README.md`** feature list gains a "Crash-safe recording" bullet.  `DurableFile` hardening: `Seek`/`SeekEnd` failures now latch `Good()` false (a write after an unnoticed seek failure would land at the wrong offset), and the dead `Truncate` primitive left over from the pre-append-only journal design was removed
 
 ### Changed
 

README.md

@@ -14,6 +14,7 @@ VTX is an open binary format for real-time per-frame state data, plus a C++20 SD
 - **Random access by frame or timestamp.** Footer-indexed seek, not a linear scan. O(log n) lookup plus one chunk decompress.
 - **Both Protobuf and FlatBuffers.** SDK supports both backends out of the box. The file announces which in its magic bytes; readers auto-detect.
 - **Live streaming transports.** Ingest frames live from an OS pipe (Windows named pipe, POSIX FIFO, stdin) or a WebSocket connection, and emit the `.vtx` byte stream over a TCP socket instead of writing to disk. Same writer API behind each transport -- file or network, sink-agnostic.
+- **Crash-safe recording.** A write-ahead `.recovery` sidecar (checksummed, append-only, fsync'd per operation by default) makes a recording that dies before `Stop()` recoverable up to the last recorded frame -- exact per-frame timestamps included. `RepairReplayFile()` reconstructs the footer; a hours-long session that crashes is no longer lost.
 - **Validation & structured diagnostics.** Validate a schema, entity, frame, or whole replay independently; strict recording rejects frames observably (bad game-time, unresolved entity type). Every failure is a structured `VtxError` -- code, severity, location, expected-vs-provided type -- so automation acts on data, not parsed log lines.
 - **Engine-independent C++20.** No engine dependency. Language bindings wherever Protobuf or FlatBuffers exist (Python, Go, Rust, Java, JS).
 - **Open.** Apache-2.0. Spec, reference reader, and tooling all in the repo.

benchmarks/CMakeLists.txt

@@ -53,6 +53,7 @@ endif()
 # static init.  `benchmark::benchmark_main` provides the main() that calls
 # Initialize + RunSpecifiedBenchmarks + Shutdown.
 add_executable(vtx_benchmarks
+    bench_crash_recovery.cpp
     bench_reader.cpp
     bench_reader_ready.cpp
     bench_writer.cpp
@@ -79,6 +80,9 @@ target_include_directories(vtx_benchmarks PRIVATE
     # vtx_common keeps protobuf / flatbuffers headers PRIVATE; benchmarks
     # that touch PropertyContainer internals need the same thirdparty paths.
     "${PROJECT_SOURCE_DIR}/thirdparty/protobuf/include"
+    # bench_crash_recovery.cpp instantiates the raw ReplayWriter/ChunkedFileSink,
+    # which needs the generated schema headers (same as tests/).
+    "${VTX_COMMON_GENERATED_DIR}"
 )
 
 target_compile_definitions(vtx_benchmarks PRIVATE