Skip to content

feat(export): carry the alert's temporal model score - #382

Merged
Chouffe merged 2 commits into
mainfrom
worktree-export-temporal-score
Aug 13, 2026
Merged

feat(export): carry the alert's temporal model score#382
Chouffe merged 2 commits into
mainfrom
worktree-export-temporal-score

Conversation

@Chouffe

@Chouffe Chouffe commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What

The alert export now carries the platform's temporal-model verdict: temporal_model_score, plus temporal_model_version / temporal_api_version for provenance.

Why

Ranking exported work by that score is the entry point for hard data mining: a high score on a false_positive lane is a hard negative, a low score on a smoke lane a hard positive. Scores from different model releases are not comparable, hence shipping the two version fields alongside.

Why alert level, not per object

That is the score's true grain. The platform's temporal model scores a platform sequence — an alert — and our object-splitting is annotator-side. The verdict rides the primary lane while every split sibling stays NULL by import construction, so max() over an alert's lanes collapses the group losslessly. Putting the raw value on ObjectExport would read as "object A scored 0.87, object B was never scored", which is false — the model never looked at objects.

Objects still rank fine: each shares a manifest line with its alert.

The aggregate deliberately spans all lanes rather than only exported ones, mirroring alert_recorded_at. If the scored primary lane is marked unsure (and so omitted from objects), the alert must still report the verdict. A test covers exactly this, and it is the one that dies under mutation.

Semantics consumers must respect

null means no verdict is attributed to this alert, which covers two distinct situations:

  1. the platform never scored it — pre-2026-06-11 alerts, fail-opens, risk-gated or sub-MIN_FRAMES sequences, anything imported before the column existed;
  2. it was scored, but the object split could not tell which lane the score belonged to and cleared it from all of them (object_split.py, primary_identified false).

It never means "scored low". 0.0 is a real production verdict, so drop nulls when ranking rather than coalescing them to zero — and do not read a null as evidence of a low score.

One blind spot worth knowing: PATCH /sequences/temporal-score writes only the Sequence columns and touches no annotation row, so a score refresh does not move last_annotated_at. Scores backfilled onto already-annotated alerts are invisible to annotation_updated_gte and need a full pull. Documented on the parameter itself.

Scope

No change to export_alerts.py: it copies the API payload verbatim, so the fields reach manifest.jsonl on their own (verified by running to_manifest_item against a payload carrying them). No score filter on the endpoint — ranking happens in the manifest.

Testing

Four new tests in test_export.py:

  • scored primary + unscored sibling → the alert reports the score once
  • never scored → all three fields null
  • 0.0 survives, not flattened to null
  • scored lane unsure → the alert still reports the score

Verified: 816 backend tests pass. Each new test was watched failing with KeyError: 'temporal_model_score' before the implementation existed. Mutating the aggregate to .filter(exported_lane) kills exactly the unsure-lane test, so it is not a vacuous assertion. ruff (v0.12.7, the pinned pre-commit version) clean; mypy reports the same 21 pre-existing errors as main, none added.

Review notes

A review raised that the three MAX() aggregates are independent, so in principle the returned triple could mix lanes, and MAX over a version string compares lexicographically. Not fixed, deliberately: the failure needs two lanes of one alert to hold non-null values at once, and both writers prevent it — object_split.py clears all three keys on every non-primary lane, and the refresh in shared.py skips rather than writes when it cannot attribute the score. The three columns are always written as a unit from one record. With exactly one non-null lane, MAX performs no comparison at all. The reliance on that invariant is stated in the code rather than defended with a ROW() pick.

Follow-up, not in this PR

Scores only exist for alerts imported from 2026-08-10 onward — the backfill is still deferred, so a pull today returns mostly null. Worth a separate decision if mining should cover the historical corpus, keeping the watermark blind spot above in mind.

Ranking exported objects by the platform's temporal-model verdict is the
entry point for hard data mining: high score + false_positive lane is a
hard negative, low score + smoke lane a hard positive.

The score sits at alert level rather than on each object because that is
its true grain -- the platform scores a platform sequence, and our
object-splitting is annotator-side. The verdict rides the primary lane
while every split sibling stays NULL, so max() over the alert's lanes
collapses the group losslessly. It deliberately spans ALL lanes rather
than only exported ones, so an unsure primary lane cannot erase its
alert's score.

Objects still rank fine: each shares a manifest line with its alert.

null means never scored, never 'scored low' -- 0.0 is a real verdict, so
consumers must drop nulls rather than coalesce them. Scores only exist
for alerts imported from 2026-08-10 on; the backfill is still deferred.

The pull script needs no change: it copies the API payload verbatim, so
the fields reach manifest.jsonl on their own.
Review of #382 surfaced two ways the documented semantics were wrong.

null does not only mean 'never scored'. When the object split cannot tell
which lane the platform scored, object_split.py clears the temporal keys on
EVERY lane, primary included, so a genuinely scored alert exports null. A
miner following the old wording would discard a real verdict as 'no verdict'.

The incremental-sync watermark does not cover scores. PATCH
/sequences/temporal-score writes only the Sequence columns and touches no
annotation row, so last_annotated_at never moves. The deferred backfill would
be invisible to any consumer syncing on annotation_updated_gte; it needs a
full pull.

Also refreshes the export spec, whose response example and alert-level field
list enumerate the payload exhaustively and had gone stale.
@Chouffe
Chouffe merged commit d65ee46 into main Aug 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant