Measure the effect of post-processing filters on final predictions (offline eval)

[mihow]

Summary

The post-processing framework (#1289) and its first filters — small-size filter, class masking, and rank roll-up (#999) — are deployed admin-only. Each filter rewrites detections or classifications and therefore the occurrence-level predictions a project shows, but we do not yet have a quantitative measure of whether a given filter improves the final predictions or merely changes them. This issue proposes an offline evaluation that runs each filter over a set with trusted labels and reports how the final predictions move relative to ground truth, so operators can decide which filters to enable per project on evidence rather than intuition.

This is a follow-up to the initial admin-only deployment, not a blocker for it.

What we want to learn

• For each filter, how the final predictions change against a verified / ground-truth set: accuracy and precision/recall by rank, the share of occurrences whose determination changed, and the direction of change (corrected vs regressed).
• Whether the effect depends on input confidence — class masking and rank roll-up are both expected to behave differently on diffuse, low-confidence distributions than on confident ones.
• Per-filter parameter sensitivity (size threshold; taxa list / region for class masking; rank thresholds for roll-up).

Sketch of an approach (to discuss)

1. Pick an evaluation set with trusted labels — human-verified occurrences, or a held-out labelled collection.
2. Snapshot the baseline final predictions before any filter runs.
3. Run each filter, and a few meaningful combinations, over the same set through the existing admin / Job path.
4. Compare the post-filter final predictions to the trusted labels and to the baseline, reporting the metrics above per filter and per parameter setting.

What we still need to verify

• A suitable ground-truth set, and how to freeze a comparable baseline. The filters mutate classifications in place, so the harness needs either a non-destructive snapshot or a copy of the data to compare against.
• Agreement on the metrics and on rank-aware scoring — a roll-up to a correct ancestor is partial credit, not a miss.
• Whether the existing model-agreement / verification tooling can be reused for the scoring step.

Notes

• Rank roll-up currently picks the global argmax per rank rather than restricting to the source taxon's ancestors, so on a diffuse distribution it can reparent a detection to a family outside its own lineage (documented TODO in ami/ml/post_processing/rank_rollup.py). The evaluation should treat that as one of the behaviours under measurement, and may inform the open lineage-constrained vs distribution-roll-up design choice.

[mihow]

Proposed experiment design (offline, non-destructive)

Fleshing out the eval with a concrete protocol: a paired before/after comparison of the classifier's top-1 against independent human ground truth.

Ground-truth test set — independent human identifications

Use occurrences that carry an Identification with agreed_with_prediction IS NULL AND agreed_with_identification IS NULL AND withdrawn = False — a human who chose the taxon independently, rather than agreeing with the ML prediction or another human's identification. This avoids the circularity of scoring the classifier against labels the classifier itself seeded. Ground-truth taxon = that identification's taxon. Restrict to occurrences the target classifier scored (with stored logits) in the region the mask targets.

Validation labels should be sourced from production verified data when we are ready; in practice that may mean a fresh production → dev data sync before a run, since the dev/staging data can lag.

Two predictions per occurrence (from stored logits — no re-inference)

• Baseline: argmax over all classes (raw classifier top-1).
• Masked: argmax over the classes whose taxon is in the regional list.

Both are recomputed from the same stored logit vector, so the whole experiment is an offline read — no re-inference, no writes.

Metrics (paired, same occurrences)

Bucket each occurrence by baseline-correct × masked-correct:

	masked correct	masked wrong
baseline correct	unchanged-right	broke (mask hurt)
baseline wrong	corrected (mask helped)	changed-still-wrong / unchanged-wrong

• Headline = net = corrected − broke.
• McNemar's test on the discordant pairs, for significance.
• Rank-aware partial credit (correct genus/family via parents_json).
• Confidence-stratified (e.g. masked score ≥ 0.5).

Separate the two failure causes

1. List-coverage failures — GT species not in the regional list, so masking cannot be right. Measures list quality, not reweighting.
2. Reweighting effect — restricted to GT-in-list and GT-in-label-set: the pure test of whether masking picks a better regional species than the raw top-1.

Report both the pure-reweighting number and the all-in number (which pays the coverage cost). Also require GT ∈ classifier label set (else the classifier could never predict it), and report macro-averaged (per-species) accuracy alongside micro.

On reweighting (renormalisation)

Reweighting does not change which species wins — argmax over the kept classes is identical with or without renormalisation; it only rescales confidence. So top-1 accuracy is identical either way. Reweighting matters for score-threshold-based downstream (default filters, determination thresholds). A separate calibration curve (predicted score vs empirical accuracy) can quantify whether reweighting overstates confidence when few classes survive. Recommendation: keep reweighting on by default, with an option to disable it — tracked on the class-masking PR.

Deliverable

An offline management command that reads stored logits and emits the tables above, non-destructively (no re-inference, no writes), parameterised by (project/region, classifier, regional list).