feat(export): carry the alert's temporal model score - #382
Merged
Conversation
Ranking exported objects by the platform's temporal-model verdict is the entry point for hard data mining: high score + false_positive lane is a hard negative, low score + smoke lane a hard positive. The score sits at alert level rather than on each object because that is its true grain -- the platform scores a platform sequence, and our object-splitting is annotator-side. The verdict rides the primary lane while every split sibling stays NULL, so max() over the alert's lanes collapses the group losslessly. It deliberately spans ALL lanes rather than only exported ones, so an unsure primary lane cannot erase its alert's score. Objects still rank fine: each shares a manifest line with its alert. null means never scored, never 'scored low' -- 0.0 is a real verdict, so consumers must drop nulls rather than coalesce them. Scores only exist for alerts imported from 2026-08-10 on; the backfill is still deferred. The pull script needs no change: it copies the API payload verbatim, so the fields reach manifest.jsonl on their own.
Review of #382 surfaced two ways the documented semantics were wrong. null does not only mean 'never scored'. When the object split cannot tell which lane the platform scored, object_split.py clears the temporal keys on EVERY lane, primary included, so a genuinely scored alert exports null. A miner following the old wording would discard a real verdict as 'no verdict'. The incremental-sync watermark does not cover scores. PATCH /sequences/temporal-score writes only the Sequence columns and touches no annotation row, so last_annotated_at never moves. The deferred backfill would be invisible to any consumer syncing on annotation_updated_gte; it needs a full pull. Also refreshes the export spec, whose response example and alert-level field list enumerate the payload exhaustively and had gone stale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The alert export now carries the platform's temporal-model verdict:
temporal_model_score, plustemporal_model_version/temporal_api_versionfor provenance.Why
Ranking exported work by that score is the entry point for hard data mining: a high score on a
false_positivelane is a hard negative, a low score on asmokelane a hard positive. Scores from different model releases are not comparable, hence shipping the two version fields alongside.Why alert level, not per object
That is the score's true grain. The platform's temporal model scores a platform sequence — an alert — and our object-splitting is annotator-side. The verdict rides the primary lane while every split sibling stays NULL by import construction, so
max()over an alert's lanes collapses the group losslessly. Putting the raw value onObjectExportwould read as "object A scored 0.87, object B was never scored", which is false — the model never looked at objects.Objects still rank fine: each shares a manifest line with its alert.
The aggregate deliberately spans all lanes rather than only exported ones, mirroring
alert_recorded_at. If the scored primary lane is marked unsure (and so omitted fromobjects), the alert must still report the verdict. A test covers exactly this, and it is the one that dies under mutation.Semantics consumers must respect
nullmeans no verdict is attributed to this alert, which covers two distinct situations:MIN_FRAMESsequences, anything imported before the column existed;object_split.py,primary_identifiedfalse).It never means "scored low".
0.0is a real production verdict, so drop nulls when ranking rather than coalescing them to zero — and do not read a null as evidence of a low score.One blind spot worth knowing:
PATCH /sequences/temporal-scorewrites only theSequencecolumns and touches no annotation row, so a score refresh does not movelast_annotated_at. Scores backfilled onto already-annotated alerts are invisible toannotation_updated_gteand need a full pull. Documented on the parameter itself.Scope
No change to
export_alerts.py: it copies the API payload verbatim, so the fields reachmanifest.jsonlon their own (verified by runningto_manifest_itemagainst a payload carrying them). No score filter on the endpoint — ranking happens in the manifest.Testing
Four new tests in
test_export.py:0.0survives, not flattened to nullVerified: 816 backend tests pass. Each new test was watched failing with
KeyError: 'temporal_model_score'before the implementation existed. Mutating the aggregate to.filter(exported_lane)kills exactly the unsure-lane test, so it is not a vacuous assertion.ruff(v0.12.7, the pinned pre-commit version) clean;mypyreports the same 21 pre-existing errors asmain, none added.Review notes
A review raised that the three
MAX()aggregates are independent, so in principle the returned triple could mix lanes, andMAXover a version string compares lexicographically. Not fixed, deliberately: the failure needs two lanes of one alert to hold non-null values at once, and both writers prevent it —object_split.pyclears all three keys on every non-primary lane, and the refresh inshared.pyskips rather than writes when it cannot attribute the score. The three columns are always written as a unit from one record. With exactly one non-null lane,MAXperforms no comparison at all. The reliance on that invariant is stated in the code rather than defended with aROW()pick.Follow-up, not in this PR
Scores only exist for alerts imported from 2026-08-10 onward — the backfill is still deferred, so a pull today returns mostly
null. Worth a separate decision if mining should cover the historical corpus, keeping the watermark blind spot above in mind.