fix: escape interior double-quotes in FTS5 search to prevent 500s#462
Closed
Mubashirrrr wants to merge 1 commit into
Closed
fix: escape interior double-quotes in FTS5 search to prevent 500s#462Mubashirrrr wants to merge 1 commit into
Mubashirrrr wants to merge 1 commit into
Conversation
A search token containing a double-quote inside the word (e.g. foo"bar) was wrapped by sanitizeFTS into the unbalanced FTS5 phrase "foo"bar". SQLite rejects this with "unterminated string", so Store.Search returns an error and the GET /search and GET /prompts/search handlers respond with HTTP 500 on ordinary user input. Escape any interior double-quotes by doubling them, as required for an FTS5 string literal. Surrounding-quote stripping is preserved, so plain queries like `fix auth bug` and `"quoted phrase"` produce identical output to before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
|
This looks like a plausible FTS5 bug, but I am closing the PR because it skips the issue-first process and the PR body includes generated-tool attribution we do not carry in contributions. Open a clear bug issue with the reproduction, get it approved, then re-file a clean PR linked with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GET /search(andGET /prompts/search) return HTTP 500 for ordinary search input that contains a double-quote inside a word, e.g.foo"bar.Root cause
sanitizeFTS(internal/store/store.go) wraps each whitespace-separated token in double-quotes so FTS5 treats it as a phrase. It strips surrounding quotes withstrings.Trim(w, "\"")but leaves interior quotes untouched:So
foo"barbecomes the phrase"foo"bar". That is an unbalanced FTS5 string literal, and SQLite aborts the wholeMATCHquery withunterminated string.Store.Searchreturns that error and the HTTP handler maps it to 500. Whether a given input crashes is data-dependent (an even number of interior quotes happens to balance, an odd number does not), which makes it an easy-to-hit, hard-to-explain 500.Fix
Escape any interior double-quotes by doubling them, which is how a literal
"is represented inside an FTS5 string literal:foo"barnow sanitizes to the valid phrase"foo""bar", which both runs without error and correctly matches text containing the literal tokenfoo"bar. Plain queries are unaffected:fix auth bug→"fix" "auth" "bug"and"quoted phrase"→"quoted" "phrase", identical to before. One line of logic changed; surrounding-quote behavior preserved.Test (fail-before → pass-after)
Added two tests in
internal/store/store_test.go:TestSearchWithEmbeddedQuoteDoesNotCrash— stores an observation containingfoo"bar, then runsSearchwith several embedded-quote inputs (foo"bar,a"b"c,",he said "hi). Asserts none error, and that searchingfoo"barreturns the matching observation.TestSanitizeFTSPreservesPlainQueries— locks in the unchanged output for plain/edge-quoted queries (backward-compat guard).Full
go test ./internal/store/andgo test ./internal/server/pass;go vet ./internal/store/is clean.Repro steps
curl -s 'http://127.0.0.1:<port>/search?q=foo%22bar'500withsearch: SQL logic error: unterminated string. After:200with normal results.🤖 Generated with Claude Code