fix(triage): recover prose-wrapped JSON and send real diffs to triage - #9
Merged
Conversation
Two fixes for the AI triage pipeline, both reproduced against live output. Parser: claude -p returns a full agent response, and the model sometimes wraps the requested JSON in prose (a preamble or trailing commentary). The old parser only stripped a single leading/trailing code fence, so a preamble raised "Expecting value: line 1 column 1 (char 0)" and trailing text raised "Extra data". _parse_response now extracts the JSON object from anywhere in the result, and parse failures log the raw stdout so they stay diagnosable. Signal: sitemap sources watch the whole page (content_selector=""), so raw_diff was the first 500 chars of text, which is site-wide nav. Persist the full page body in a new content_text snapshot column and send triage an old-vs-new diff, so unchanged nav drops out and only the real change reaches the model. This also fixes content.py, whose line-based diff was meaningless on the whitespace-collapsed one-line text. Co-Authored-By: Penny-One (Batcave) <noreply@anthropic.com>
krayong
added a commit
that referenced
this pull request
Jul 14, 2026
## Why The `content_text` column added in #9 (diff-based triage) holds the full page body. `export_seed_sql` does `SELECT *` on snapshots, so that body flowed into the shipped seed — `seed.sql.gz` went from **1.4 MB → 108 MB**, which the `seed` workflow push hit as GitHub's hard 100 MB file limit (`GH001`), and which would also bloat the PyPI wheel. ## Fix Hold `content_text` back from the seed export (`_SEED_EXCLUDED_COLS`). It's a **local runtime cache** — needed to diff a page against its prior version, not to detect changes (`content_hash` does that) and not part of a distributed baseline. Seeded rows import `content_text` empty and self-heal on the first change per page: the diff falls back to an excerpt once, then stores the real body. Change detection, hashes, excerpts, and HTTP validators are unaffected. ## Why it slipped through #9 Nothing in PR CI builds a real seed — the seed is deliberately off the CI/release path (the hours-long crawl runs only on schedule/dispatch), and unit tests use fake stores. The size regression could only surface when the `seed` workflow actually ran. ## Tests Added `test_seed_export_excludes_content_text`: a snapshot with a large body must not appear in the exported SQL, and the round-trip still preserves hash + excerpt (content_text imports empty). 486 pass, lint clean. 🦇 Generated by [Penny-One](https://batman.fandom.com/wiki/Alfred_Pennyworth) at [Batcave](https://batman.fandom.com/wiki/Batcave) Co-authored-by: Penny-One (Batcave) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
AI summaries were failing on most scheduled runs with
AI unavailable: could not parse claude response: Expecting value: line 1 column 1 (char 0). Both fixes below were reproduced against liveclaude -poutput before being written, and covered test-first.1. Parser recovers prose-wrapped JSON
claude -p --output-format jsonis a full agent, not a JSON API —sonnetintermittently wraps the requested JSON in prose. The old parser only stripped a single leading/trailing code fence, so:Expecting value: line 1 column 1 (char 0)Extra dataOne bad batch of four tripped the whole run's AI-unavailable banner.
_parse_responsenow extracts the JSON object from anywhere inresult, and parse failures log the raw stdout so they stay diagnosable.2. Triage gets a real diff, not site nav
Sitemap sources watch the whole page (
content_selector=""), soraw_diffwas the first 500 chars of text — pure site-wide nav, identical for every page, no body. Now the full page body is persisted in a newcontent_textsnapshot column (additive migration) and triage receives an old-vs-new diff, so unchanged nav drops out. This also fixescontent.py, whose line-basedsplitlines()diff was meaningless on the whitespace-collapsed one-line text.Tests
485 pass, lint clean. New: two parser tests for the prose signatures, a sitemap guarantee test that the diff carries the body change and excludes nav, plus updated store/detector doubles.
🦇 Generated by Penny-One at Batcave