fix(persist): the storage boundary guarantees valid UTF-8 - #67
Merged
Conversation
Scraped pages and model output carry arbitrary bytes; Postgres rejects an invalid sequence, which failed the whole commit and stranded the run. Every persisted content, payload, and metadata string now passes the encoding boundary: invalid sequences become U+FFFD in place, the record persists, and the damage stays visible and local.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A research run that scraped invalid-UTF8 bytes could not persist anything containing that content: Postgres rejected the encoding, every commit of the node's data failed (including its failure), and the run stranded — observed in production as 'pq: conversion between UTF8 and SQL_ASCII is not supported' killing an agent node's terminal commit.
The persist layer now owns an encoding boundary (persist/encoding.lua, pure byte-walk — the sandbox has no utf8 stdlib): every content, payload, and metadata string written by ops, commit, and commit_repo is validated; invalid sequences are replaced with U+FFFD in place. Valid UTF-8 (including 3- and 4-byte sequences) passes through untouched; overlongs, surrogates, and truncated tails are repaired, never dropped wholesale.
Tests: 968 sqlite green (5 new boundary cases).