Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,9 @@ conductor.createTask(
pollInterval: 1000, // How often to poll for new executions in ms (default: 1000)
partition: false, // Enable partitioning (default: false)
window: ["09:00", "17:00"], // Time window for execution [start, end]
fifo: true, // Durable strict FIFO (exclusive with concurrency)
concurrency: 10, // Soft task-level limit
groupConcurrency: 2, // Soft task/group-level limit
},
handler,
);
Expand Down
4 changes: 3 additions & 1 deletion docs/content/api/conductor.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,9 @@ const task = conductor.createTask(
name: string; // Task name (required)
queue?: string; // Queue name (default: "default")
maxAttempts?: number; // Max retry attempts (default: 3)
concurrency?: number; // Max concurrent executions (default: unlimited)
fifo?: boolean; // Strict durable FIFO lane
concurrency?: number; // Soft max concurrent executions
groupConcurrency?: number; // Soft max per invocation group
window?: [string, string]; // Time window (e.g., ["09:00", "17:00"])
removeOnComplete?: { days: number } | false; // Retention policy
removeOnFail?: { days: number } | false; // Retention policy
Expand Down
16 changes: 9 additions & 7 deletions docs/content/task-execution/concurrency.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ Control the maximum number of concurrent executions for a specific task:
const processVideo = conductor.createTask(
{
name: "process-video",
concurrency: 3, // Max 3 videos processing at once
concurrency: 3, // Soft max of 3 videos at once
groupConcurrency: 1, // Soft max of 1 per invocation group
},
{ invocable: true },
async (event, ctx) => {
Expand All @@ -24,12 +25,7 @@ When the limit is reached, additional executions wait in the queue until a slot

## How It Works

Postgres Conductor uses a slot-based system to enforce concurrency limits:

1. **Slot allocation**: When a task has `concurrency: N`, Postgres creates N slots in the `_private_concurrency_slots` table
2. **Claiming slots**: Workers claim available slots using `FOR UPDATE SKIP LOCKED`
3. **Execution**: Task runs while holding the slot
4. **Release**: Slot is released when execution completes or fails
Postgres Conductor evaluates active executions when claiming work. Task and group limits are coordinated with `FOR UPDATE SKIP LOCKED`; limits are intentionally soft across concurrent workers. Grouped candidates whose group is full do not consume task-level capacity, so another available group can be claimed in the same batch.

This happens entirely in Postgres - no external coordination needed.

Expand All @@ -50,8 +46,14 @@ This happens entirely in Postgres - no external coordination needed.
- Set on worker/queue with `config: { concurrency }`
- Independent per worker instance

Child invocations inherit the group supplied to `ctx.invoke`. Dynamic cron schedules accept `group` alongside `cron`, and each next cron execution preserves the group.

## What's Next?

- [Worker Configuration](../api/worker-config.md) - Configure worker-level concurrency
- [Priority](priority.md) - Control execution order when waiting for slots
- [Batching](batching.md) - Process multiple executions together

## Group concurrency

`group` may be supplied when invoking a task. `groupConcurrency` limits active executions within each `(queue, task, group)` scope; ungrouped invocations bypass that limit. Task and group limits compose and are intentionally soft across concurrent workers.
31 changes: 31 additions & 0 deletions docs/content/task-execution/fifo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# FIFO execution

Set `fifo: true` on a task to serialize its executions in strict enqueue order:

```ts
const updateAccount = conductor.createTask(
{ name: "update-account", fifo: true },
{ invocable: true },
async (event, ctx) => { /* ... */ },
);
```

FIFO is scoped to `(queue, task)`. It uses a durable database lane owner, so only
one execution can be active at a time across all workers. The owner remains with
the execution during retries, sleeps, child waits, and `waitForEvent` waits, and
is released on completion, cancellation, or permanent failure. A worker crash
therefore resumes the owner before admitting its successor.

Priorities are ignored for FIFO tasks. A future, never-started scheduled or
delayed execution does not block currently runnable work; once selected, it
retains the lane even when it becomes delayed. Different FIFO tasks (and queue
identities) run independently.

FIFO cannot be combined with `concurrency` or `groupConcurrency`:

```ts
{ name: "invalid", fifo: true, concurrency: 2 }
```

Use soft `concurrency` or `groupConcurrency` instead when strict ordering is
not required.
1 change: 1 addition & 0 deletions docs/zensical.toml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ nav = [
{ "Cancellation" = "task-execution/cancellation.md" },
{ "Priority" = "task-execution/priority.md" },
{ "Concurrency" = "task-execution/concurrency.md" },
{ "FIFO" = "task-execution/fifo.md" },
{ "Deduplication" = "task-execution/deduplication.md" },
{ "Rate Limiting" = "task-execution/rate-limiting.md" },
{ "Batching" = "task-execution/batching.md" },
Expand Down
2 changes: 1 addition & 1 deletion justfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ alias r := ready
alias t := test

build-migrations:
sh ./scripts/build-migrations.sh
bash ./scripts/build-migrations.sh

lint:
bun run oxlint --type-aware --deny-warnings
Expand Down
Loading
Loading