Preserve NaN log probabilities in trajectory JSON

[bradhilton]

Saving captured Chat Completions NaN log probabilities writes JSON null, which fails validation on load. Encoding NaNs as strings also exposes equality checks that reject separately parsed NaNs when matching tokenized histories to their source exchanges and groups.

Use Pydantic's string encoding for non-finite numbers and NaN-aware source comparisons. JSON and compact round-trips preserve canonical exchange references for single-history and multi-history trajectories and groups, including tensorized forms. Actual source mismatches still raise. Token flags and training behavior stay unchanged.

Filter unrelated calls by capture timestamps before the NaN-aware fallback, avoiding repeated serialization of every nonmatching exchange. Full equality still verifies candidates with matching timestamps.

Validation: 300 focused tokenization, compact serialization, tokenized-model, and tensorized-model tests passed. Regressions cover finite and NaN captured log probabilities, source identity after restoration, rejection of mismatched groups, and bounded serialization work for multi-exchange histories. A 150-exchange local benchmark restored JSON in 36 ms with finite logprobs and 64 ms with NaNs. Ruff, formatting, and type checks for touched files passed.

Known pre-existing limitation: legacy Completions responses with NaN token_logprobs still need separate provider-model serialization support.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Consolidated review record (automated agent review traffic removed 2026-09-17)

Change. Captured Chat Completions NaN log probabilities were saved to trajectory JSON as null and failed validation on load. The shared model base now encodes non-finite floats as strings, and source-exchange matching is NaN-aware and timestamp-filtered so round-tripped histories still resolve.

Review. McCarthy and Minsky reviewed ff96504; both returned CLEAR for source correctness. The PR merged at b4ef151, which adds the source-matching fixes and has no recorded agent review. McCarthy verified the regressions fail on base and pass on head across eight NaN/Infinity/finite cases; Minsky ran test_tokenize.py (247 passed) with a negative control. CI at review: trainer-rank-gpu-validation passing, quality-checks pending; the description reports 300 focused tests and lint/type checks passing.

Findings addressed during review.

• Tokenized-wrapper JSON and tokenized-group compact round-trips still failed at ff96504 because source reconciliation rejected separately parsed NaNs; the merged head adds NaN-aware comparisons and capture-timestamp filtering.

Deferred / follow-ups.

• Legacy Completions responses with NaN token_logprobs still need provider-model serialization support (pre-existing; no issue recorded).