Prevent concurrency lock leaks that cause duplicate job execution

[rosa]

Supersedes #689 and ports the exact-once fix from starburstlabs/solid_queue#11. Both target the same failure: a job with limits_concurrency whose semaphore count gets permanently corrupted, letting it run well past its configured limit.

Background

Two independent reports converged on the same class of bug:

• Fix duplicate job execution despite limits_concurrency #689 (@mhenrixon) — duplicate execution despite limits_concurrency.
• Prevent false worker pruning from leaking concurrency permits starburstlabs/solid_queue#11 (@tblais1224) — a production incident where a limit: 1 job ran 11× concurrently after a transient heartbeat failure.

The failure: a worker stays alive on a long-running job but its DB heartbeat fails transiently. Another supervisor prunes it as dead — failing its claimed execution, releasing its concurrency lock, and unblocking a replacement — but the original execution keeps running. When it finally returns, its stale ClaimedExecution finalizes the job and releases the same lock a second time, pushing the semaphore value above its limit and admitting ever more concurrent work.

Changes

1. Guard signal_all against exceeding the limit (from #689, @mhenrixon)

Semaphore#signal already caps its increment with value < limit, but the batch signal_all path — used when releasing concurrency locks in bulk, e.g. discarding ready jobs — did not, letting a semaphore's value climb past its limit. It's now grouped by limit and capped, matching the single-signal path.

2. Prevent wrong release of blocked jobs and semaphore corruption (from starburstlabs#11, @tblais1224 & @briceburg-wb)

The problem: a worker stays alive on a long-running job but its database heartbeat fails transiently. Another supervisor prunes it as dead, fails its claimed execution, releases its concurrency lock, and unblocks a replacement — but the original execution keeps running. When it finally returns, its stale ClaimedExecution, which exists only in memory, because the DB record was deleted by the supervisor prune, also goes through the semaphore and blocked execution release, potentially violating the limit and allowing more concurrent jobs.

With this change, we take a lock on the claimed execution row so that it's required that it still exists and we're the only ones acting over it, before running the finalization code: deleting the claimed execution row, signaling the semaphore and unblocking blocked jobs. In the scenario described above, we'd see the claimed execution record as gone and would do nothing.

Note that the semaphore signal + blocked jobs release is still done after the transaction that marks the job as finished/failed, so that these two actions can't rollback a job that finished/failed (see 873760d). Because this can only happen for the ones who got to clear the claimed execution row, we still guarantee the lock is released just once.

Scope

• From Fix duplicate job execution despite limits_concurrency #689, only the signal_all guard is taken. Its other changes were dropped: not releasing blocked jobs while a same-key job runs (existing, expected behavior); the FOR UPDATE lock in Semaphore#wait (already on main in 8d90eab, Job with concurrency key blocked when it shouldn't have, eventually was run after duration expired #456); and the release re-dispatch / half-atomic unblock (subsumed by exact-once ownership).
• Prevent false worker pruning from leaking concurrency permits starburstlabs/solid_queue#11 also proposes not pruning a stale worker while its supervisor's heartbeat is fresh. That's deliberately not included here. Exact-once ownership already makes the lock leak impossible, and the supervisor's heartbeat runs on a separate thread from its reap loop (Process.waitpid2 + claim cleanup) — so a "fresh" supervisor can still have failed to reap a genuinely dead child, which would then shield that dead worker from heartbeat-based pruning and leave its jobs stuck. We prefer the airtight DB-level fix without that trade-off.

Credit to @mhenrixon (#689) and @tblais1224 & @briceburg-wb (starburstlabs#11). #689 can be closed in favor of this.

[tblais1224]

Thanks for prioritizing this change @rosa. I can confirm that starburstlabs#11 has been running stable in production and has helped concurrency issues. I am however experiencing the issue that #689 seems to resolve. So I think joining these together was a great call, and should make this much more stable. 👏

[rosa]

Thank you so much @tblais1224!

I'm going to run this one in production as well. It's not the whole change in starburstlabs#11, but a subset and with a small tweak. I described it in the commit 858fadb. I think this is enough to solve the issue you described in your PR.

[rosa]

We're running this in production now. I'll monitor for one day and will merge tomorrow; then I'll release a new version,

[rosa]

This has been working without issues, so I'm going ahead with it. Thanks again @mhenrixon, @tblais1224 and @briceburg-wb!