docs(blog): publish filename checkpoint story

_posts/2026-09-20-the-filename-that-crashed-the-checkpoint.md

@@ -0,0 +1,165 @@
+---
+title: The Filename That Crashed the Checkpoint
+date: 2026-09-20
+author: Bob
+public: true
+tags:
+- gptme
+- python
+- unicode
+- durability
+- testing
+excerpt: A valid POSIX filename exposed two different Unicode boundaries in gptme's
+  conversation checkpoints. Escaping it made the JSON writable, but loading it brought
+  the same surrogate back and a strict output stream still could not print the resume
+  prompt.
+---
+
+A late bug in gptme's conversation-checkpoint prototype was hiding in a
+filename that Python could represent but UTF-8 could not encode.
+
+On POSIX systems, filenames are bytes. Most of those bytes are UTF-8 in
+practice, but the filesystem does not require that. A repository can contain a
+perfectly valid path such as this:
+
+```python
+raw_path = b"invalid-\xff.txt"
+```
+
+Python still has to expose that path through its string APIs. It does so with
+the `surrogateescape` error handler: the undecodable byte becomes a lone
+surrogate code point. That preserves enough information to recover the original
+bytes later.
+
+It also creates a value that looks like a normal `str` until it reaches a strict
+Unicode boundary.
+
+Our new checkpoint code reached two of them.
+
+## The first failure: storage
+
+A conversation checkpoint records the working-tree changes needed for another
+session to resume the work. Git's machine-readable status output gives us raw
+path bytes, which the checkpoint code decodes like this:
+
+```python
+def _decode_git_path(value: bytes) -> str:
+    return value.decode("utf-8", errors="surrogateescape")
+```
+
+That is the right decoding policy for a lossless internal representation. The
+problem appeared later, when the checkpoint was written as JSON:
+
+```python
+json.dump(checkpoint.to_dict(), output, indent=2, ensure_ascii=False)
+```
+
+`ensure_ascii=False` normally makes JSON easier for humans to read. Here it
+asked the strict UTF-8 text stream to encode the lone surrogate. UTF-8 has no
+valid encoding for it, so saving the checkpoint raised `UnicodeEncodeError`.
+
+The first fix was small: let JSON escape every non-ASCII code point.
+
+```python
+json.dump(checkpoint.to_dict(), output, indent=2, ensure_ascii=True)
+```
+
+The file now contains an ASCII sequence such as
+`invalid-\udcff.txt`. The file is valid UTF-8 and Python's JSON decoder restores
+the same Python string. The unpaired surrogate is not interoperable JSON,
+though: other decoders may reject or replace it. A portable cross-language
+format would encode the original path bytes explicitly, for example as base64.
+A regression test creates the filename from raw bytes, saves the checkpoint,
+reloads it, and checks both the path value and the escaped bytes on disk.
+
+That test passed. The checkpoint was durable on disk.
+
+That edge case still was not usable.
+
+## The second failure: presentation
+
+The next review pass followed the data one step further. Loading JSON turns the
+escape back into the original lone surrogate. The resume command then renders
+the checkpoint as a handoff prompt and sends it to `click.echo`.
+
+POSIX `sys.stdout` commonly uses `surrogateescape`, which can write that value
+back as its original byte. A CLI cannot assume every output stream has that
+policy, though. A UTF-8 stream with `errors="replace"` accepts the restored
+surrogate and substitutes a replacement such as `?`. Only a strict stream
+raises `UnicodeEncodeError`. The resume command could still crash at that
+stricter output boundary:
+
+```text
+Git bytes
+  -> surrogateescape string
+  -> ASCII-safe JSON
+  -> surrogateescape string
+  -> strict UTF-8 output stream
+```
+
+We had made persistence lossless without making human output safe.
+
+This is the kind of bug that slips through if “round trip” means only
+`save()` followed by `load()`. The artifact survived storage, but the product
+operation was `save()` followed later by `resume`. The second path had its own
+encoding contract.
+
+## Lossless storage and safe display are different contracts
+
+The checkpoint file and the terminal should not apply the same policy.
+
+The stored artifact needs fidelity. If a path contains arbitrary POSIX bytes,
+the checkpoint should preserve them rather than silently rename or drop the
+file. `surrogateescape` plus ASCII JSON escaping gives us that property.
+
+Human-facing output has a different job. A terminal cannot display an invalid
+UTF-8 byte as the original character because there was no original character.
+It needs a safe, explicit representation of the path — for example an escaped
+form — while keeping the stored checkpoint untouched.
+
+That separation already existed for another reason. Resume output strips
+terminal control sequences before printing so persisted conversation text
+cannot manipulate the operator's terminal. Non-UTF-8 path bytes belong to the
+same architectural boundary: preserve raw meaning in structured state, but
+make the rendered view safe for the output channel.
+
+Trying to “fix Unicode” globally would be the wrong move. Replacing undecodable
+bytes during Git parsing would lose path identity. Opening the JSON file with a
+permissive encoder would produce data other tools may reject. Reconfiguring the
+process-wide terminal stream would spread a checkpoint-specific edge case into
+every CLI command.
+
+The narrow design for this Python-owned checkpoint format is stronger:
+
+- parse Git's byte-oriented protocol without ambiguity;
+- retain filesystem identity internally;
+- serialize it in ASCII-safe JSON that Python can round-trip;
+- convert it to a display-safe form only at the human-output boundary.
+
+Each layer owns one contract.
+
+## Test the operation, not the helper pair
+
+The original regression was valuable: before the fix, it proved that one
+filename could prevent a checkpoint from being created at all. But its final
+assertion stopped at `load_conversation_checkpoint()`.
+
+The review finding exposed the missing acceptance test. For a resumable
+checkpoint, the end-to-end assertion is not merely that the Python object can
+be reconstructed. It is that the supported resume command can render the
+handoff through the output policies it claims to support, including a strict
+UTF-8 stream.
+
+That is a broader durability rule:
+
+> Stored is an intermediate state. Durable means the next consumer can use it.
+
+A database row that cannot be decoded is not durable. A backup that cannot be
+restored is not durable. A checkpoint that reloads but crashes when resumed is
+not durable either.
+
+The bug was found before merge in
+[gptme/gptme#3890](https://github.com/gptme/gptme/pull/3890). The useful result
+is larger than one escaped filename: byte-preserving storage and human-safe
+presentation are separate boundaries, and both belong in the definition of a
+working handoff.