Import existing pipeline results

[mihow]

Summary

This PR lets an operator load machine-learning results that were produced outside Antenna back into a project, so the captures, detections, classifications, and occurrences show up in the platform as if Antenna had run the pipeline itself. It is aimed at people who process their images locally with the AMI Data Companion (ADC): ADC can crunch through local images far faster than triggering processing from Antenna today, and until now there was no clean way to get that work back into the web platform. The matching export side is ami-data-companion#82.

The core of the change is a set of new options on Pipeline.save_results() that let it create the supporting records an external result set needs (deployments, source images), plus a management command that reads a saved results file and feeds it through that path. Importantly, save_results() is the same function the live processing path uses — both the synchronous API flow and the async (PSv2/NATS) flow ingest a PipelineResultsResponse through it — so imported data is built exactly like processed data, and the importer is a thin wrapper rather than a parallel code path.

This branch has also been brought up to date with main (it had drifted behind a large batch of pipeline and event-grouping changes); the conflicts in save_results, group_images_into_events, and the algorithm-registration helpers were resolved so the importer sits on top of main's current behaviour.

How to use

Process images locally with ADC, export the results, then import a results file into an existing Antenna project:

python manage.py import_pipeline_results <results.json> --project <PROJECT_ID> \
    [--public-base-url https://host/path/]   # resolve relative image paths in the file
    [--create-new-algorithms]                # register algorithms from the file (see below)
    [--dry-run]                              # validate only, write nothing

• Input is a single JSON file containing one PipelineResultsResponse — the schema processing services already return, and the output of ADC's export api-occurrences command.
• The project must already exist. The command does not create it. Deployments and source images referenced by the results are created by name when missing, but their full configuration (location, device, research site) is not imported.
• Algorithms. By default the algorithms referenced by the results must already be registered (through a processing service's /info endpoint), otherwise the import raises PipelineNotConfigured. Pass --create-new-algorithms to register them from the results file instead — the common case when importing into an instance that has never seen that pipeline.
• Large exports are split by ADC into several *_batch_001.json, *_batch_002.json … files; import them one at a time (see follow-ups for multi-file/zip support).

List of Changes

#	Change (what it does for the user/operator)	How (implementation)
1	Operators can import an externally produced results file into a project from the command line.	New import_pipeline_results management command parses a PipelineResultsResponse JSON file and calls save_results. Supports --project, --public-base-url, --create-new-algorithms, and --dry-run.
2	Imports can create the deployments and source images the results refer to, even for a project that started empty.	save_results() gains create_missing_source_images, project_id, and public_base_url. New helpers get_or_create_deployments() and create_source_images() build the missing records and remap external IDs to internal ones.
3	Results can carry which deployment each capture belongs to, so imported captures land in the right station.	New DeploymentResponse schema; SourceImageResponse.deployment and PipelineResultsResponse.deployments added to the ML API schema.
4	Imports of a pipeline Antenna has never seen can register that pipeline's algorithms from the file.	--create-new-algorithms flag surfaces the existing create_new_algorithms path on save_results; default off so the strict, pre-registered flow stays the norm.
5	Imported detections that already have a hosted crop image reuse it; others get a crop generated locally.	get_or_create_detection() only reuses a crop URL when it is an http(s) URL; relative paths (or none) fall through to local crop generation.
6	A default deployment can be created with a caller-supplied name.	get_or_create_default_deployment() takes a name parameter (keeps main's "Default Station" default and latitude/longitude = 0).
7	The "register algorithms before processing" rule is now documented, and the API deprecation is surfaced correctly.	Added an "Algorithm and Category Map Registration" section to processing_services/README.md; fixed the misspelled depreciated=True on PipelineResultsResponse.algorithms to deprecated=True (now emits the standard OpenAPI marker) and explained why the field is ignored.
8	Branch brought up to date with main.	Merged origin/main; resolved conflicts in ami/main/models.py, ami/ml/models/pipeline.py, ami/ml/tests.py. The importer's create_new_algorithms path is preserved alongside main's pre-registered path and its null-detection marking (#1310).

Detailed Description

API contract additions. The processing-service schema now models deployments so a results file can say which station each capture came from. SourceImageResponse gained an optional deployment, and PipelineResultsResponse gained an optional deployments list, both typed as the new DeploymentResponse (id / name / key).

save_results import mode. When called with create_missing_source_images=True and a project_id, save_results first ensures the referenced deployments exist (get_or_create_deployments), then creates any missing source images (create_source_images, using public_base_url to resolve relative image paths), remaps the external IDs in the results to the new internal IDs, and only then runs the normal detection/classification/occurrence save path. The return value (PipelineSaveResults) now also carries the deployments that were touched, which the command reports and re-saves so sessions and stations are regrouped.

Shared ingestion. Both the live async path (ami/jobs/tasks.py::process_nats_pipeline_result, which parses a PipelineResultsResponse off NATS) and this importer call Pipeline.save_results. The importer is save_results plus the create_missing_source_images / create_new_algorithms flags; the live path needs neither because its source images already exist and its algorithms were registered via /info.

Follow-ups and Discussion

None of these block the core import path; they are decisions and enhancements worth tracking.

1. Import track associations (multi-detection occurrences). Antenna's model already supports an occurrence made of several detections across frames (a track) via Detection.occurrence, and the synthetic generators (create_demo_project, seed_synthetic_occurrences) build them. Antenna does not compute tracks itself, but ADC (or another service) can. The gap is the wire format: DetectionResponse has no field saying "these detections belong to the same occurrence", so this import gives each detection its own occurrence. Proposed: add an optional sequence_id / track_id to DetectionResponse (a pure grouping key, no Antenna concepts, so it respects the ML-backend schema boundary), and have create_and_update_occurrences_for_detections create one occurrence per (project, track id) when present, falling back to current behaviour when absent. Pairs with the export change in ami-data-companion#82, which currently flattens occurrences and drops the id.

2. Multi-file / archive import. ADC splits large exports into *_batch_NNN.json files plus an optional *_images folder of crops. The command imports one file at a time. Worth adding a directory/glob and/or zip-archive input so a whole export can be imported in one run.

3. Identify algorithms by key + version instead of name + version. Algorithms are unique on name + version (Algorithm.Meta.unique_together), and get_or_create_algorithm_and_category_map keys on the same pair. This is the source of near-duplicate algorithm records when the same algorithm comes back with a slightly different name (e.g. extra whitespace); the stable identifier is the key slug. Switching the constraint and the get-or-create to key + version is the right fix but touches registration, dedup, and existing data, so it needs its own testing pass and is tracked separately.

4. Two management commands named import_pipeline_results. One under ami/ml/management/commands/ (this PR's PipelineResultsResponse importer, the one Django actually resolves) and one under ami/exports/management/commands/ (a legacy occurrences.json parser that builds records directly and does preserve multi-detection occurrences, currently shadowed). They differ by input format and occurrence handling, not by "results vs whole project". Consolidate to one location and rename each to its real purpose (for example import_pipeline_results vs import_legacy_occurrences).

5. Category maps for imported classifiers. Registering a classification algorithm requires a category map; importing a classifier with --create-new-algorithms will raise PipelineNotConfigured if the results file does not include one. The ADC export does not yet emit category maps (include_category_maps raises NotImplementedError there). Decide whether the importer should require category maps for classifiers or synthesize a placeholder, and coordinate with the export side.

6. Full project package import. Neither command imports deployment configuration (location, device, research site) or creates the project. If "set up a project from an export" is a goal, that is a separate feature on top of this results importer.

Larger Directions (roadmap, beyond this PR)

These are bigger pieces of work this importer sets up rather than delivers. Recording them here so the design intent is visible.

1. Import from other offline processing tools and from standard datasets

The clean shape for all of these is a small per-source adapter that converts the foreign format into a canonical PipelineResultsResponse, after which the existing save_results path does the rest. That keeps a single ingestion path (the one this PR shares with live processing) and isolates the format-specific quirks in one converter per source. Candidates, roughly in order of fit:

• Mothbox offline software (Mothbot_Process, mothbot-classify). A DIY moth-monitoring stack that processes locally and emits its own outputs — Mothbot_Process exports CSV plus EXIF-embedded images; mothbot-classify keeps per-project JSON "botdetection" files and patch crops. Supporting these would let Mothbox users land their data in Antenna. Needs a CSV/JSON → PipelineResultsResponse adapter; their taxonomy and algorithm naming will have to be mapped, which makes the key-based algorithm identity question (follow-up 3 above) more pressing.

• Standardized / benchmark datasets. Loading published test sets into Antenna would make it a place to evaluate and compare models against known ground truth (this connects to the model-agreement work, antenna#1307).

  • AMI Dataset (Zenodo 12554005) — specifically AMI-Traps, ~2,893 expert-annotated camera-trap images covering ~52,948 labeled individuals, taxonomy-organized. These are expert/human labels, not ML predictions, so they should import as verified identifications/ground truth, not as model output — provenance needs to be modelled (a verified/MachineObservation-vs-human distinction), not just dumped through the prediction path.
  • flat-bug (Zenodo 14761447) — ~6,028 images with COCO instance-segmentation annotations aggregated from 23 sub-datasets; detection/segmentation only, no species. Importing this means detections without classifications, and segmentation masks that are richer than the bbox the current schema carries — worth deciding whether to keep masks or reduce to bounding boxes on import.

  Both ship as large COCO-style archives, which reinforces follow-ups 2 (multi-file/archive input) and 5 (category maps / ground-truth provenance).

2. A web UI for importing, mirroring the export workflow

Today import is command-line only. The platform already has a data export workflow in the UI; an import counterpart would let a user upload a results file (or point at a stored one), choose the target project, set options (create-missing, register-algorithms, public base URL), and run it as a tracked job with progress — symmetric with export. This is a frontend + endpoint + background-job piece on top of the importer this PR provides, and should reuse the same save_results entry point so the UI and CLI paths cannot diverge.

Testing

• ami.exports.tests.test_imports and ami.ml.tests — 67 passed, 2 skipped.
• ami.main.tests event-grouping tests — 11 passed.
• ami.jobs regroup tests — 4 passed.
• --create-new-algorithms flag confirmed in --help; import command exercised by test_imports.
• Pre-commit (black, isort, flake8) clean on the changed files.

Not yet exercised: an end-to-end import against a clean project with no pre-registered algorithms using --create-new-algorithms, and a real ADC export run through the command on a live stack.

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	20ba329
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6a3dc55e748aba00084e565b

[mihow]

@mihow confirm that sequence_ids are coming through as occurrences. Every occurrence has a single detection in the test data imported, but that could just be the data as well, since they are snapshots.

[netlify]

✅ Deploy Preview for antenna-ssec canceled.

Name	Link
🔨 Latest commit	ac37873
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/6a3db759257cb5000860a04c

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b21fb592-f2b6-4bc8-869b-daecd9d9a16a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/adc-importer

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[netlify]

✅ Deploy Preview for antenna-ssec canceled.

Name	Link
🔨 Latest commit	20ba329
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/6a3dc55e449bb20008ddaf86