Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
54a78ae
docs: add alert API connector design spec
Chouffe Aug 6, 2026
2c02ec6
docs: simplify connector schedule to a single daily sweep
Chouffe Aug 6, 2026
dc198a8
feat(api): add Fernet secret service for connector credentials
Chouffe Aug 6, 2026
c79af2f
feat(api): add alert API connector, organization, and coverage tables
Chouffe Aug 6, 2026
06e26ab
feat(api): add superuser connector CRUD with write-only credentials
Chouffe Aug 6, 2026
5338f48
feat(api): add connector verify with org discovery and cross-org probe
Chouffe Aug 6, 2026
fad7d00
fix(api): guard connector verify against malformed alert-API probe re…
Chouffe Aug 6, 2026
2793cd0
feat(api): add connector organization toggle and coverage endpoints
Chouffe Aug 6, 2026
c11e1c6
test(api): cover default 30-day coverage window and inclusive bounds
Chouffe Aug 6, 2026
26d6339
refactor(scripts): extract run_import library with org and skip filters
Chouffe Aug 6, 2026
b75adff
fix(scripts): thread the annotation API token through the import pipe…
Chouffe Aug 6, 2026
2a2e053
feat(worker): self-mint API token and ship importer into the image
Chouffe Aug 6, 2026
947c499
fix(worker): close password-reset hole and pin the real seed, not a f…
Chouffe Aug 6, 2026
880420e
feat(worker): add daily connector import sweep with coverage recording
Chouffe Aug 6, 2026
7bd83a7
fix(worker): guard the whole connector-import boundary, fix internal …
Chouffe Aug 6, 2026
822baa3
feat(frontend): add connector types, API methods, and query hooks
Chouffe Aug 6, 2026
e460ee5
feat(frontend): add connector coverage heatmap component
Chouffe Aug 6, 2026
681a2e8
fix(frontend): migrate CoverageHeatmap to design-system tokens
Chouffe Aug 6, 2026
d22563a
feat(frontend): add connectors list page, route, and nav entry
Chouffe Aug 6, 2026
fb3dd1c
fix(frontend): migrate the user-menu dropdown to design tokens
Chouffe Aug 6, 2026
2d7c47e
feat(frontend): add connector detail page with verify and coverage he…
Chouffe Aug 6, 2026
3b024a6
fix(frontend): render sample_date raw, scope heatmap test, add partia…
Chouffe Aug 6, 2026
698075f
style(frontend): prettier-format connector files from tasks 9-11
Chouffe Aug 6, 2026
f181679
fix(frontend): show a neutral Service pill for system users
Chouffe Aug 6, 2026
3858db5
docs: document alert API connectors
Chouffe Aug 6, 2026
a7dfbb7
fix(frontend): fix coverage heatmap state precedence and re-run seman…
Chouffe Aug 6, 2026
c9a848b
fix(frontend): match coverage window default to the server's 30 days
Chouffe Aug 6, 2026
c256777
fix(backend): bound the connector verify probe and alert-API client c…
Chouffe Aug 6, 2026
ad4fdaa
fix(backend): promote python-dotenv to a direct dependency
Chouffe Aug 6, 2026
5bf0f74
Merge remote-tracking branch 'origin/main' into worktree-alert-api-co…
Chouffe Aug 6, 2026
fa68b56
test(scripts): add same_frame_merges to the split-stats stub
Chouffe Aug 6, 2026
950a49a
fix(scripts): report same-frame merges in the runner's split summary
Chouffe Aug 6, 2026
a079394
Merge remote-tracking branch 'origin/main' into worktree-alert-api-co…
Chouffe Aug 6, 2026
1a0d8fa
fix(connectors): surface the alert API's error detail when verify get…
Chouffe Aug 7, 2026
4adf2e3
docs(specs): design for the connector test-connection button
Chouffe Aug 7, 2026
3bf3448
docs(specs): add full-width connectors-table rider to the test-connec…
Chouffe Aug 7, 2026
b0fe030
feat(connectors): stateless credential check service for the create form
Chouffe Aug 7, 2026
0a172e5
feat(connectors): POST /connectors/test pre-save credential check end…
Chouffe Aug 7, 2026
ec6d7c0
feat(connectors): testConnector client method and mutation hook
Chouffe Aug 7, 2026
da9dbe0
feat(connectors): pre-save Test connection button in the create form
Chouffe Aug 7, 2026
9c2918b
fix(connectors): drop the width cap so the table spans the layout
Chouffe Aug 7, 2026
f05cd0b
feat(connectors): styled hover tooltip and pale-pine quiet days on th…
Chouffe Aug 7, 2026
151bb3f
Merge remote-tracking branch 'origin/main' into worktree-alert-api-co…
Chouffe Aug 7, 2026
df24af0
Merge remote-tracking branch 'origin/main' into worktree-alert-api-co…
Chouffe Aug 11, 2026
28c425c
fix(migrations): commit the connector revision's new id, not just its…
Chouffe Aug 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ Writes `manifest.jsonl` (one line per alert) plus `images/{source_api}/{platform

**Backend data flow**: Alert API → ingestion scripts → annotation_api DB → frontend UI → human annotations

**Ingestion**: Daily ingestion is connector-driven (worker sweep at 03:00 UTC, configured at `/connectors`); the CLI import script is retained for backfilling arbitrary date ranges.

**Processing stages** (sequence): `IMPORTED` → `READY_TO_ANNOTATE` → `SEQ_ANNOTATION_DONE` → `ANNOTATED`. Two-lane exit: FP-only lanes jump straight to `ANNOTATED` at classify submit; smoke lanes park at `SEQ_ANNOTATION_DONE`, get auto-annotated per alert once every sibling (shared `platform_alert_id`) is classified, and reach `ANNOTATED` via the Smoke Localization submit (see `docs/specs/2026-07-28-smoke-localization-entry-point-design.md`).

**Backend patterns**: CRUD modules per entity, Pydantic schemas separate from SQLModel, dependency injection, fastapi-pagination, IoU-based annotation generation service.
Expand Down
6 changes: 6 additions & 0 deletions annotation_api/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,9 @@ ALERT_API_LOGIN=your_alert_api_username
ALERT_API_PASSWORD=your_alert_api_password
ALERT_API_ADMIN_LOGIN=your_admin_username
ALERT_API_ADMIN_PASSWORD=your_admin_password

# Fernet key encrypting alert-API connector passwords at rest.
# Generate: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
CONNECTOR_SECRET_KEY=
# Where the worker reaches the annotation API (compose service name).
ANNOTATION_API_INTERNAL_URL=http://annotation_api:5050
32 changes: 32 additions & 0 deletions annotation_api/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,38 @@ uv run python -m scripts.data_transfer.ingestion.alert_api.import \
- **Logging support** - Configurable log levels for debugging
- **Stage management** - Automatic transitions from alert API data to READY_TO_ANNOTATE stage

### Alert API Connectors

Ongoing daily ingestion runs through connectors instead of the CLI. A connector is
one alert API credential (base URL, login, password) plus the set of remote
organizations it should import from.

- **Configuration**: Connectors are managed in the frontend at `/connectors`,
superuser only. Creating one stores the credential, discovers the alert API's
organizations, and lets the superuser enable individual organizations for import.
- **Schedule**: The worker sweeps all enabled connectors daily at 03:00 UTC
(`schedule_connector_imports` in `src/app/worker.py`) and imports each one's
trailing window (`run_connector_import`). `trailing_days` (default 3, configurable
per connector) is both the re-check window and the catch-up mechanism — a worker
that missed a run recovers the lost date inside the next run's window, so no
"already ran today" bookkeeping is needed.
- **`CONNECTOR_SECRET_KEY`**: Required for connectors to work. Alert-API passwords
can't be hashed (the worker needs the plaintext to log in), so they're Fernet-
encrypted at rest with this key. Generate one with:
```bash
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
```
If unset, connector create/update returns `400` and the worker skips connector
imports — existing deployments that never set it keep working untouched. Losing
the key means re-entering credentials through the UI.
- **Backfill**: The UI has no backfill and enabling a new organization does not
retroactively import its history. `make import-alert-api DATE_FROM=… DATE_END=…`
remains the way to import an arbitrary date range by hand.
- **Coverage**: Each (connector, organization, day) import attempt is recorded as a
coverage row, rendered as a heatmap on the connector detail page. A day with zero
alerts is recorded as `ok` with zero counts — deliberately distinct from a day
that was never attempted, so a dashed cell always means "we never got there."

## Troubleshooting

### Common Issues
Expand Down
2 changes: 2 additions & 0 deletions annotation_api/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
COPY src/alembic.ini /app/alembic.ini
COPY src/migrations /app/migrations
COPY src/app /app/app
# The worker imports the alert-API importer as a library (see app/worker.py).
COPY scripts /app/scripts

# Install project
RUN --mount=type=cache,target=/root/.cache/uv \
Expand Down
4 changes: 3 additions & 1 deletion annotation_api/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ dependencies = [
"pillow>=10.0.0",
"procrastinate>=2.0.0,<4.0.0",
"psycopg[binary]>=3.1.0",
"cryptography>=42.0.0",
"pyyaml>=6.0",
"python-dotenv>=1.0.0",
]

[dependency-groups]
Expand All @@ -59,7 +62,6 @@ dev = [
"requests-mock>=1.11.0",
"httpx>=0.23.0",
"aiosqlite>=0.16.0,<1.0.0",
"pyyaml>=6.0",
]

[tool.coverage.run]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,13 @@ def api_get(route: str, access_token: str):
"""
headers = make_request_headers(access_token=access_token)
logging.debug(f"Making an HTTP request to route {route}")
response = requests.get(route, headers=headers)
# 30s: these are list endpoints (sequences/cameras/organizations) that can
# legitimately take longer than the 5s token exchange under real load, but
# this now also runs on the worker's connector-verify path behind a button
# a human is watching (via asyncio.to_thread), so it must never hang
# forever on a black-holed connection — no timeout previously bounded this
# call at all.
response = requests.get(route, headers=headers, timeout=30)
try:
return response.json()
except Exception:
Expand Down
Loading
Loading