Score handover quality, and never store a blank handover - #2
Open
Laolex wants to merge 2 commits into
Open
Conversation
A cautious model is allowed to return empty fields, but a handover with an empty "What happened" is not a handover. The human edit paths already reject one; the model path did not, so 3 of the 8 drafts a live model has produced were stored blank. A blank draft now carries the family own words across from the source, and records `repaired` so the failure stays visible rather than being papered over. The error handler logged a full stack for every client-caused 404 — scanners probing for WordPress plugin paths — and returned error.message to the caller, which handed out the absolute server path. 4xx now logs one line, and errors marked `expose: false` return a generic message.
Memreda claim is that a family can trust what they read, which is testable. `npm run eval` opens the database read-only and scores every handover a live model drafted: whether it left "What happened" blank, whether it asserts words absent from the family own source, whether it dropped a follow-up the source asked for, and whether it reads as medical advice. Every check is deterministic string logic. No model grades another model, so a run is reproducible, costs nothing, and works while the providers are down — the same standard the product holds itself to. It exits non-zero on any failure so it can gate a build. Against the live database, across the 8 handovers a live model actually drafted: nothing invented and nothing that reads as advice, and 3 blank "What happened". Eight cases is too few to be stable and most are short test recordings, so the README says so rather than quoting the rates as fact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two week-one debt items, plus the eval harness that makes the first one measurable.
Never store a blank handover
A cautious model is allowed to return empty fields, and it does. A handover whose "What happened" is empty is not a handover. The server now carries the family's own words across instead of storing a blank one — and records that it did, as an
empty-field-fallbackprovider, so the eval still sees the model's failure rather than a repaired result masking it.Typed notes never reach the model (
deterministic-note), so the voice path is what needed covering.Stop 404s naming a server path
A missing static file returned an error mentioning a real filesystem path. It now 404s without disclosing one.
Handover quality eval
npm run evalscores generated handovers against the sources they came from. This exists because "the model got better" was previously an unfalsifiable claim about the core of the product.Verification
npm test— 20/20 pass.Not merged yet, by request.