fix(matching): surface dense-embed failures + ground-source 0.5-fallback regression (dontguess-553) - #153
Open
baron-3dl wants to merge 1 commit into
Open
fix(matching): surface dense-embed failures + ground-source 0.5-fallback regression (dontguess-553)#153baron-3dl wants to merge 1 commit into
baron-3dl wants to merge 1 commit into
Conversation
…ack regression (dontguess-553) Every buy returned confidence=0.500/is_partial_match=false regardless of relevance. Root cause was operational: the live operator built its match index when onnxruntime was missing, so index-time Embed calls silently returned nil and buys degraded to the reputation-fallback path (computeConfidence = rep/100 = DefaultReputation/100 = 0.5). The matching code is correct — replayAll->rebuildMatchIndex repopulates the index and Search/Rank re-embeds at query time — so restarting the operator with the dep present fixes the live exchange (exact match now ranks #1 at sim 0.979). This change locks in correctness and prevents silent recurrence: - pkg/matching/dense_embedder.go: add OnError hook. The previous silent swallow of embed errors is why an index-time failure was invisible for weeks. Nil-default keeps test behavior unchanged. - cmd/dontguess/serve.go: wire OnError to the operator log ("embedder ERROR"). - pkg/exchange/dense_embed_match_regression_test.go: ground-source test driving a real put+buy through the engine with the REAL dense embedder (no mocks). Asserts exact sim>=0.80, unrelated<0.30, MatchIndexLen==2 so accepted entries cannot route to the 0.5 fallback. - .github/workflows/ci.yml: install python3 + numpy/onnxruntime/tokenizers so the dense regression test runs (not skipped) per the ground-source rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every buy returned
confidence=0.500/is_partial_match=falseregardless of semantic relevance — an exact-text query did not rank its exact match above unrelated entries.Root cause (operational, not a code defect)
The flat 0.5 is
computeConfidence = SellerReputation/100withDefaultReputation=50— the reputation-fallback path inengine_buy.go, taken only when an entry is absent from the match index (HasEmbedding=false). The matching code is correct:replayAll→rebuildMatchIndexrepopulates the index andSearch/Rankre-embeds at query time. The live operator had built its index ~25h earlier when onnxruntime was missing, so index-timeEmbedcalls silently returned nil and buys degraded to fallback.Live fix (verified)
Rebuilt + restarted the operator with the dep present → the CLI repro now ranks the exact entry #1 at similarity 0.979 / confidence 0.76, unrelated entry floor-excluded (older results showed
similarity=None).Durable changes
pkg/matching/dense_embedder.go— addOnErrorhook. The previous silent swallow of embed errors is why this was invisible for weeks. Nil-default → no test behavior change.cmd/dontguess/serve.go— wireOnErrorto the operator log (embedder ERROR).pkg/exchange/dense_embed_match_regression_test.go— ground-source test: real put+buy through the engine with the real dense embedder (no mocks). Asserts exactsim≥0.80, unrelated<0.30,MatchIndexLen==2so entries can't route to the 0.5 fallback..github/workflows/ci.yml— install python3 + numpy/onnxruntime/tokenizers so the dense test runs (not skipped) per the ground-source rule.Full suite green locally (
go test ./...),go vetclean.🤖 Generated with Claude Code