CANCELLED jobs leak through /next filter, starve newer async_api jobs

[mihow]

Bug

/next (claim-next-job, currently list(ids_only=1) in ami/jobs/views.py:296 get_queryset()) hands out CANCELLED async_api jobs forever. The exclude filter does:

jobs.exclude(
    status=JobState.failed_states(),
    updated_at__lt=cutoff_datetime,
)

Two problems:

1. JobState.failed_states() does not include CANCELLED — only [FAILURE, REVOKED, UNKNOWN]. CANCELLED jobs are treated as still-active and offered to workers indefinitely.
2. status= is the wrong lookup for a list-valued return — should be status__in=. As written, Django compares the column to the list as a single value; the exclude effectively never matches.

The two combine: any user-cancelled async_api job becomes a permanent zombie that workers serially "claim" every poll, then immediately bail because the corresponding NATS stream is empty/exhausted. Newer async_api jobs created after the zombies are starved out — workers never see them.

Symptom

A real async_api job sits with collect stage = SUCCESS, process stage = CREATED 0% indefinitely. NATS stream for that job ID has num_pending = total_tasks, delivered = 0. Worker stdout cycles:

Processing job <old_cancelled_id> with pipeline ...
DataLoader subprocess 0/1 starting iteration for job <old_cancelled_id>
No jobs found, sleeping for 5 seconds

…on the same handful of stale CANCELLED job IDs every 5s, never picking up the new job.

Manual unblock

Bump CANCELLED zombies to REVOKED so they fall in failed_states(). .update() bypasses auto_now, so updated_at stays at the original old value (already past the 72h FAILED_JOBS_DISPLAY_MAX_HOURS cutoff):

from django.utils import timezone
Job.objects.filter(pk__in=[<zombie ids>]).update(
    status="REVOKED", finished_at=timezone.now()
)

Workers stop cycling within ~30s.

Suggested fix

Either:

• Add CANCELLED to JobState.failed_states(), and change status= to status__in= in the exclude so the comparison actually works.
• Or short-circuit in the /next handler with an explicit exclude(status__in=[CANCELLED, FAILURE, REVOKED, UNKNOWN]).

Links / urgency

Related to #1265 (dedicated /next action; the temporary list(ids_only=1) claim-semantics).

This bit a production deployment today — workers stuck cycling 3 zombies from 2+ weeks ago, blocking a freshly-started job. Would be good to merge alongside the next deploy today.

[mihow]

@copilot where did / does the CANCELLED state come from? This is not a known job state from Celery. was this perhaps a manual change injected while triaging jobs? or is it a genuine state that we missed?

[mihow]

Claude says: Closing as resolved. The starvation symptom — terminal jobs handed out by the worker claim path, blocking newer async jobs — is fixed: there is no CANCELLED state (cancel transitions CANCELING → REVOKED, which is in final_states()), the claim path applies IncompleteJobFilter (excludes terminal jobs), and #1338 / #1342 stop a finished job from being pulled back to a claimable state. The one remaining status= vs status__in= typo noted here lives in the UI display-cutoff branch (not the claim filter) and is removed when the ids_only branch is replaced by the dedicated /next action in #1265. Tracking that cleanup in #1265.