Indexer: config validation, full ScVal coverage, parser fuzzing, dead-letter observability (#475, #478)

[Depo-dev]

Carries #475 and #478 (indexer config validation, ScVal coverage, proptest fuzzing, dead-letter hardening) into dev, with their defects fixed and the remaining acceptance criteria implemented.

Both PRs were merged onto integration/indexer-config-and-parser rather than straight into dev so the problems below could be repaired first.

Contributor work landing here

Config validation (#417) — every env var parsed and range-checked in one pass, with all errors collected and reported together rather than failing on the first. Good design: an operator fixes one misconfiguration per restart otherwise.

ScVal coverage (#415) — fixtures per variant asserted against real stellar_xdr types.

Fuzzing (#416) — a proptest harness over the decode entry points asserting the parser never panics.

Dead-letter hardening (#414) — bounded exponential backoff around the parse-error insert, so a transient DB hiccup no longer loses the audit record.

Defects fixed

The proptest generators panicked on their own inputs

Generators requested vec(any::<u8>(), 1..128) — as little as one byte — then indexed up to bytes[8] to fill a u64/i64. Five tests failed inside the generator, before the parser was ever called, so the suite reported failures that said nothing about the code under test. Minimum raised to 9 bytes.

256-bit ScVal decoding was silently wrong

Pre-existing on dev, not introduced by either PR — but the new fuzz coverage is what made it findable.

U256/I256 packed four 64-bit limbs into a u128 using 32-bit shifts, truncating the top half and mis-positioning the rest:

	value
hi_hi=1 decoded as	79228162514264337593543950341
correct	6277101735386680763835789423207666416102355444464034512896

A plausible-looking number, just the wrong one — which is exactly why it survived. Replaced with true 256-bit decimal conversion (long multiplication over decimal digits, no big-integer dependency), pinned by exact-value tests at both extremes and u256::MAX.

Cargo.lock and formatting

proptest was added to Cargo.toml but never locked, so --locked builds fail. Formatting fixed for cargo fmt --all -- --check.

Worth flagging: rust-toolchain.toml pins 1.94.0 and its comment claims "this file is therefore what actually pins CI", but the CI logs show the toolchain action resolving stable to rustc 1.98.0. Any Rust PR can pass fmt locally and fail it in CI for reasons the author cannot reproduce. Not fixed here — it deserves its own issue.

Remaining criteria implemented

#414 — dead-letter observability. Added trident_indexer_dead_lettered_total, incremented only once the row is durably written. Deliberately not on the failure path: counting there fires the alert for events never actually captured for replay. Kept distinct from parse_errors_total, which counts every parse failure including ones that later succeed — an alert wants the abandoned-event signal, not retry noise.

#415 — Timepoint, Duration, Error. All three fell through to the debug catch-all, rendering as Timepoint(1700000000) and — worse — incrementing unhandled_scvariant_total despite being fully understood types, which eroded the one signal meant to say "the decoder met something new". Now handled explicitly. Timepoint and Duration serialise as decimal strings, matching the 128/256-bit rule rather than the I64/U64 one: both are u64, and silent precision loss at the top of the range is worse than consistent strings.

Two documentation errors surfaced while checking the table against the encoder, both pre-existing: ScvBytes was documented as base64 (the code emits hex) and ScvError as a structured {"type", "code"} object (no such object is produced). Both corrected, with a test pinning the hex claim so they cannot drift again. The I64/U64-as-number inconsistency is now recorded explicitly rather than left to be rediscovered.

#416 — fuzzing in CI. The proptest suite ran only at its in-file case count. Added a dedicated step at PROPTEST_CASES=50000, so every push does a real fuzz pass. Longer-run invocation documented, along with where proptest persists failing inputs — a regression corpus only helps if the file gets committed.

Issue status

Closed #415 by hand (auto-close does not fire when the base is not the default branch). #417 was already closed.

Left open with specifics recorded on each:

• launch: dead-letter path so one poison event cannot wedge the cursor #414 — no admin endpoint or documented replay query; no integration test with a deliberately poisoned row asserting the cursor advances. That last one is the criterion that actually proves the fix, and it needs a live database. The alert on the new counter is also not yet configured.
• launch: fuzz the XDR parser to prove it cannot panic #416 — corpus not yet seeded from real testnet payloads. The generators synthesise structurally-valid ScVals and random bytes, which covers the shape of the input space but not its distribution; real payloads reach variant combinations random generation rarely hits.

Verification

cargo build, cargo clippy --all-targets --all-features -- -D warnings, cargo fmt --all -- --check, and cargo test all pass — 284 tests, up from 265 before this work.

Also carries the apk upgrade fix for CVE-2026-14456, cherry-picked from integration/launch-fixes, since this branch is off dev and would otherwise fail the same required trivy check.