diff --git a/designs/GROUP_CONCURRENCY.md b/designs/GROUP_CONCURRENCY.md deleted file mode 100644 index 67ed662..0000000 --- a/designs/GROUP_CONCURRENCY.md +++ /dev/null @@ -1,341 +0,0 @@ -# Group-Based Concurrency Design - -## Current System - -### Schema -``` -_private_tasks: - - concurrency_limit (integer, nullable) - -_private_concurrency_slots: - - task_key + slot_group_number (PK) - - capacity (always 1) - - used (0 or 1) -``` - -### Characteristics -- **Single dimension**: One global concurrency limit per task (e.g., `concurrency: 5`) -- **Pre-allocated slots**: Slots created at task registration (5 rows for limit=5) -- **Batch processing**: Query pairs up to 100 executions with available slots using ROW_NUMBER -- **Fast path**: When no tasks have concurrency, uses simpler query (no slot overhead) - -### Limitations -Cannot express multi-dimensional rate limits like: -- "Max 20 executions per tenant" -- "Max 1 execution per message" -- Both constraints applied simultaneously to same task - -## Problem Statement - -### Use Case -``` -Task: "process-message" -Requirements: - - Max 20 concurrent executions per tenant (tenant-level rate limit) - - Max 1 concurrent execution per message (message-level deduplication) - - Constraints apply independently and simultaneously - -Examples: - { tenant: "acme", message: "msg-1" } → needs slots from tenant=acme AND message=msg-1 - { tenant: "acme", message: "msg-2" } → needs slots from tenant=acme AND message=msg-2 - { tenant: "beta", message: "msg-1" } → needs slots from tenant=beta AND message=msg-1 -``` - -### Core Challenges -1. **Dynamic slot space**: Cannot pre-create slots for unknown tenant/message IDs -2. **Multi-dimensional claiming**: Each execution needs slots from multiple independent groups -3. **All-or-nothing atomicity**: Must claim ALL required group slots or none (avoid deadlocks) -4. **Optional groups**: Some executions may skip certain groups (no key provided) -5. **Query complexity**: Current batch approach breaks down - each execution has unique group keys - -## Design Options - -### Option 1: Fully Independent Groups (Recommended) - -#### Schema Changes -``` -_private_task_concurrency_groups: - - task_key + group_name (PK) - - concurrency_limit (integer) - -_private_concurrency_slots: - - task_key + group_name + group_key + slot_number (PK) - - used_by_execution_id (nullable UUID) - -_private_execution_group_keys: - - execution_id + group_name (PK) - - group_key (text) -``` - -#### API Surface -```typescript -// Task definition -defineTask({ - name: "process-message", - concurrency: { - tenant: 20, // max 20 per tenant - message: 1, // max 1 per message - // groups are optional per execution - } -}) - -// Invocation -invoke({ name: "process-message" }, { - payload: { ... }, - concurrencyGroups: { - tenant: "acme", - message: "msg-123" - } -}) -``` - -#### Semantics -- **Independent constraints**: Each group is an independent concurrency limit -- **All must pass**: Execution must satisfy ALL provided group constraints -- **Optional participation**: If execution doesn't provide key for a group, that group's constraint is skipped -- **Example**: `{ tenant: 20, message: 1 }` means "max 20 per tenant" AND "max 1 per message" - -#### Slot Lifecycle -1. **Creation**: On-demand when first execution with that (task, group, key) arrives -2. **Claiming**: Execution atomically claims one slot from EACH provided group -3. **Release**: All slots released when execution completes/fails/released -4. **Garbage collection**: Delete unused slots after TTL (e.g., 24 hours of inactivity) - -#### Query Strategy (Conceptual) -``` -for each execution candidate: - 1. extract concurrencyGroups from payload/metadata - 2. for each group defined on task: - - if execution provides key for group: - → check if slot available for (task, group, key) - → if no slot available → skip this execution - - if execution doesn't provide key: - → skip this group check (no constraint) - 3. if ALL checks pass: - → claim all required slots atomically (for update) - → return execution for processing -``` - -#### Pros -- ✅ Flexible: supports any number of groups -- ✅ Optional: groups can be skipped per execution -- ✅ No deadlocks: all-or-nothing claiming -- ✅ Intuitive semantics -- ✅ Matches stated use case exactly - -#### Cons -- ❌ Query complexity: O(executions × groups) checks needed -- ❌ Hard to batch: each execution has unique group keys -- ❌ Slot table growth: one row per unique (task, group, key, slot) combination -- ❌ Performance unknown: needs prototyping - -### Option 2: Single Primary Group + Global Limit - -#### Simplified Model -```typescript -defineTask({ - name: "process-message", - concurrency: 100, // global task limit (existing system) - concurrencyGroup: { // ONE group dimension only - name: "tenant", - limit: 20 - } -}) -``` - -#### Semantics -- Global limit (100) applied first using existing system -- Group limit (20 per tenant) applied as secondary filter -- Only one group dimension supported - -#### Pros -- ✅ Simpler to implement -- ✅ Covers common case (rate limiting per tenant) -- ✅ Can still use batch processing somewhat -- ✅ Lower query complexity - -#### Cons -- ❌ Only one group (not extensible to multi-group case) -- ❌ Doesn't solve stated use case (tenant + message simultaneously) -- ❌ Coupling between global and group limits may be confusing - -### Option 3: Composite Keys (Pre-computed) - -#### Idea -User declares all possible group key combinations upfront - -```typescript -defineTask({ - name: "process-message", - concurrency: { - tenant: ["acme", "beta", "gamma"], // fixed list of keys - message: 1 // per-key limit - } -}) -``` - -#### Pros -- ✅ Can pre-create slots (known space) -- ✅ Query similar to current system - -#### Cons -- ❌ Not dynamic (can't handle new tenants at runtime) -- ❌ Combinatorial explosion with multiple groups (tenant × message slots) -- ❌ Doesn't match stated use case (unknown keys upfront) - -## Open Design Questions - -### 1. Query Performance Strategy - -**Problem:** Current query efficiently batches 100 executions. With groups, need per-execution evaluation. - -**Options:** -- **A) PostgreSQL function with loop**: Procedural approach iterating through candidates -- **B) CTE-based with LATERAL joins**: Declarative query checking all groups per execution -- **C) Two-phase approach**: Batch-lock likely slots, then check per execution - -**Decision needed:** Which approach balances performance vs complexity? - -### 2. Slot Creation & Lifecycle - -**When to create slots?** -- Lazy: on first execution with that group key -- Eager: during registration if keys are known upfront -- Explicit: via separate admin API - -**Garbage collection:** -- Delete slots unused for X hours/days? -- Keep forever (infinite growth)? -- Manual cleanup API? - -**Decision needed:** What's the lifecycle management strategy? - -### 3. Atomicity & Locking Model - -**Scenario:** Execution needs [tenant=acme slot, message=msg-1 slot] - -**Option A - Optimistic:** -- Check both available -- Attempt to claim both -- If second claim fails → rollback first - -**Option B - Pessimistic:** -- Lock all required slots with for update -- Then assign to execution - -**Option C - Hierarchical:** -- Order groups (alphabetically?) to prevent deadlocks -- Always lock in same order - -**Decision needed:** Which locking strategy avoids deadlocks and contention best? - -### 4. Backwards Compatibility - -**Current API:** -```typescript -concurrency: 5 -``` - -**Proposed API:** -```typescript -concurrency: { tenant: 20 } -``` - -**Migration Options:** -- **A) Support both**: `number | Record` (number maps to global limit) -- **B) Explicit migration**: `concurrency: 5` becomes `concurrency: { default: 5 }` -- **C) Deprecate number**: Breaking change, require object format - -**Decision needed:** Migration path for existing tasks? - -### 5. Fast Path Optimization - -**Current:** When no tasks have concurrency, uses optimized query (no slot logic) - -**With groups:** Need to: -1. Check if task defines groups -2. Check if execution provides group keys -3. Mix grouped and ungrouped tasks in same queue - -**Options:** -- **A) Multiple fast paths**: ungrouped, single-group, multi-group -- **B) Unified query**: Always handle groups (slower but simpler) -- **C) Separate queues**: Group-enabled tasks use different queue - -**Decision needed:** Can we preserve fast path optimization? - -### 6. Payload vs Metadata - -**Where do group keys live?** - -**Option A - In payload:** -```typescript -invoke({ name: "task" }, { - tenant: "acme", // part of payload - message: "msg-1" -}) -``` -- ✅ Simple -- ❌ Mixes business data with infrastructure concerns - -**Option B - Separate metadata:** -```typescript -invoke({ name: "task" }, { - payload: { /* business data */ }, - concurrencyGroups: { - tenant: "acme", - message: "msg-1" - } -}) -``` -- ✅ Clean separation -- ❌ More verbose -- ❌ Need to persist metadata separately - -**Decision needed:** API ergonomics and data modeling? - -## Recommendation - -### Start with Option 1 (Fully Independent Groups) - -**Rationale:** -1. Matches stated use case exactly (tenant + message simultaneously) -2. Extensible to N groups without redesign -3. Optional groups handle partial constraints elegantly -4. Most flexible for future requirements - -### But Prototype Query Performance First - -Before full implementation, validate performance with prototype: - -**Test scenario:** -- 1000 pending executions -- 10 concurrency groups defined -- 100 unique keys per group -- Measure query time for group-based slot matching -- Target: maintain ~300 tasks/sec throughput (current optimized baseline) - -**Critical query sections to prototype:** -1. Extract concurrencyGroups from execution metadata (JSONB operations) -2. Check slot availability across N groups (LATERAL joins? function?) -3. Atomic all-or-nothing claiming (locking strategy) -4. Mixing grouped and ungrouped tasks (fast path preservation) - -**Success criteria:** -- Throughput degradation < 50% for grouped tasks -- Ungrouped tasks maintain fast path performance -- No deadlocks under concurrent worker load - -## Next Steps - -1. **Design review**: Validate Option 1 approach and semantics -2. **Decide on open questions**: Particularly #1 (query strategy) and #3 (locking) -3. **Query prototype**: Implement conceptual query logic in SQL -4. **Performance benchmark**: Compare against current system (diagnose-variance.ts) -5. **API finalization**: Lock down TypeScript interfaces -6. **Implementation**: Schema migration → query → tests → docs - -## Related Documents - -- `CONCURRENCY.md` - Current single-dimension concurrency implementation -- `perf/diagnose-variance.ts` - Benchmarking methodology for throughput testing diff --git a/docs/content/api/conductor.md b/docs/content/api/conductor.md index a8fbd7e..1956b1a 100644 --- a/docs/content/api/conductor.md +++ b/docs/content/api/conductor.md @@ -72,6 +72,7 @@ const task = conductor.createTask( removeOnComplete?: { days: number } | false; // Retention policy removeOnFail?: { days: number } | false; // Retention policy batch?: { size: number; timeoutMs: number }; // Batch processing config + deadLetter?: { queue: string; task?: Task }; // Final-failure destination } ``` diff --git a/docs/content/api/orchestrator.md b/docs/content/api/orchestrator.md index d7b043e..b9e86a2 100644 --- a/docs/content/api/orchestrator.md +++ b/docs/content/api/orchestrator.md @@ -24,6 +24,11 @@ const orchestrator = Orchestrator.create({ - `workers`: (optional) Array of custom workers from `conductor.createWorker()` - `defaultWorker`: (optional) Configuration for default worker +The Orchestrator is the registration authority for its workers. Configure at +most one worker for each queue (including the implicit `default` worker); put +all tasks for a queue on that worker. Multiple workers for one queue are +rejected rather than competing for registrations. + **Default worker config:** ```typescript @@ -71,7 +76,8 @@ await orchestrator.drain(); ``` - Starts the orchestrator -- Processes until queue is empty +- Processes until all queues are globally quiescent, including event fan-out + created while another queue is finishing - Automatically stops - Returns when all work is done diff --git a/docs/content/api/task-context.md b/docs/content/api/task-context.md index a88db44..8d81c01 100644 --- a/docs/content/api/task-context.md +++ b/docs/content/api/task-context.md @@ -70,6 +70,23 @@ const task = conductor.createTask( - Resumes after duration expires - Execution continues from where it left off +## ctx.waitForEvent() + +Wait durably for the next matching custom event. The subscription is persisted before the +worker is released, so a restart does not lose the wait. + +```typescript +const event = await ctx.waitForEvent("payment", { + event: paymentReceived, + filter: { orderId: [orderId] }, + timeout: 60_000, +}); +// event.name and event.payload are typed from paymentReceived +``` + +A matching event is delivered once and cached by the step key. If the timeout wins, +`WaitForEventTimeoutError` is thrown. + ## ctx.invoke() Invoke a child task and wait for result: diff --git a/docs/content/crafting-tasks/triggers.md b/docs/content/crafting-tasks/triggers.md index b4de1f2..7d51175 100644 --- a/docs/content/crafting-tasks/triggers.md +++ b/docs/content/crafting-tasks/triggers.md @@ -109,6 +109,8 @@ await conductor.emit("user.created", { }); ``` +Event fan-out and acknowledgement happen in one database transaction. If delivery fails, the event remains unprocessed and is retried automatically; successful delivery is acknowledged with `processed_at`. + ### Field Selection For large events, you can select only specific fields to reduce payload size: diff --git a/docs/content/task-execution/concurrency.md b/docs/content/task-execution/concurrency.md index eb93dd5..173537b 100644 --- a/docs/content/task-execution/concurrency.md +++ b/docs/content/task-execution/concurrency.md @@ -10,7 +10,8 @@ Control the maximum number of concurrent executions for a specific task: const processVideo = conductor.createTask( { name: "process-video", - concurrency: 3, // Max 3 videos processing at once + concurrency: 3, // Soft max of 3 videos at once + groupConcurrency: 1, // Soft max of 1 per invocation group }, { invocable: true }, async (event, ctx) => { @@ -24,12 +25,7 @@ When the limit is reached, additional executions wait in the queue until a slot ## How It Works -Postgres Conductor uses a slot-based system to enforce concurrency limits: - -1. **Slot allocation**: When a task has `concurrency: N`, Postgres creates N slots in the `_private_concurrency_slots` table -2. **Claiming slots**: Workers claim available slots using `FOR UPDATE SKIP LOCKED` -3. **Execution**: Task runs while holding the slot -4. **Release**: Slot is released when execution completes or fails +Postgres Conductor evaluates active executions when claiming work. Task and group limits are coordinated with `FOR UPDATE SKIP LOCKED`; limits are intentionally soft across concurrent workers. Grouped candidates whose group is full do not consume task-level capacity, so another available group can be claimed in the same batch. This happens entirely in Postgres - no external coordination needed. @@ -50,8 +46,14 @@ This happens entirely in Postgres - no external coordination needed. - Set on worker/queue with `config: { concurrency }` - Independent per worker instance +Child invocations inherit the group supplied to `ctx.invoke`. Dynamic cron schedules accept `group` alongside `cron`, and each next cron execution preserves the group. + ## What's Next? - [Worker Configuration](../api/worker-config.md) - Configure worker-level concurrency - [Priority](priority.md) - Control execution order when waiting for slots - [Batching](batching.md) - Process multiple executions together + +## Group concurrency + +`group` may be supplied when invoking a task. `groupConcurrency` limits active executions within each `(queue, task, group)` scope; ungrouped invocations bypass that limit. Task and group limits compose and are intentionally soft across concurrent workers. diff --git a/docs/content/task-execution/dead-letter-queue.md b/docs/content/task-execution/dead-letter-queue.md new file mode 100644 index 0000000..24151e1 --- /dev/null +++ b/docs/content/task-execution/dead-letter-queue.md @@ -0,0 +1,24 @@ +# Dead-letter queues + +Configure a destination for executions that fail on their final attempt: + +```ts +const failedPayment = conductor.createTask( + { name: "failed-payment", queue: "payments-dlq" }, + { invocable: true }, + async (event) => {}, +); + +const chargeCard = conductor.createTask( + { + name: "charge-card", + deadLetter: { queue: "payments-dlq", task: failedPayment }, + }, + { invocable: true }, + async (event) => {}, +); +``` + +The destination is a new execution with the original payload. Its execution row contains machine-readable source execution ID, source queue and task, final error, attempt count, and failure timestamp. The source remains a normal failed execution unless its retention policy removes it. + +Retries and cancellation do not deliver to a dead-letter queue. Delivery is transactional, claim-fenced, and idempotent. A destination may have its own retry, retention, and concurrency settings. Chains are supported, but a task cannot target itself directly. The destination task must accept the source payload. diff --git a/docs/zensical.toml b/docs/zensical.toml index 56b1789..7f9174b 100644 --- a/docs/zensical.toml +++ b/docs/zensical.toml @@ -73,6 +73,7 @@ nav = [ { "Cancellation" = "task-execution/cancellation.md" }, { "Priority" = "task-execution/priority.md" }, { "Concurrency" = "task-execution/concurrency.md" }, + { "Dead-letter queues" = "task-execution/dead-letter-queue.md" }, { "Deduplication" = "task-execution/deduplication.md" }, { "Rate Limiting" = "task-execution/rate-limiting.md" }, { "Batching" = "task-execution/batching.md" }, diff --git a/justfile b/justfile index f3c2532..e781d1d 100644 --- a/justfile +++ b/justfile @@ -8,7 +8,7 @@ alias r := ready alias t := test build-migrations: - sh ./scripts/build-migrations.sh + bash ./scripts/build-migrations.sh lint: bun run oxlint --type-aware --deny-warnings diff --git a/migrations/0000000001_setup.sql b/migrations/0000000001_setup.sql index abe054c..1ec5f99 100644 --- a/migrations/0000000001_setup.sql +++ b/migrations/0000000001_setup.sql @@ -95,6 +95,7 @@ create table pgconductor._private_executions ( run_at timestamptz default pgconductor._private_current_time() not null, locked_at timestamptz, locked_by uuid, + "group" text, is_available boolean generated always as (locked_at is null and failed_at is null and completed_at is null) stored not null, attempts integer default 0 not null, last_error text, @@ -104,6 +105,14 @@ create table pgconductor._private_executions ( waiting_step_key text, parent_execution_id uuid, singleton_on timestamptz, + + -- Dead-letter metadata is denormalized so retained source rows are optional. + dead_letter_source_execution_id uuid, + dead_letter_source_queue text, + dead_letter_source_task_key text, + dead_letter_error text, + dead_letter_attempts integer, + dead_letter_failed_at timestamptz, primary key (id, queue), unique (task_key, dedupe_key, queue) ) partition by list (queue); @@ -116,7 +125,7 @@ create unique index on pgconductor._private_executions (task_key, singleton_on, where singleton_on is not null and completed_at is null and failed_at is null and cancelled = false; create table pgconductor._private_tasks ( - key text primary key, + key text not null, -- queue that this task belongs to (used for queue-based worker assignment) queue text default 'default' not null, @@ -142,21 +151,26 @@ create table pgconductor._private_tasks ( ) ), - -- concurrency control: maximum number of concurrent executions across all workers + -- concurrency controls are intentionally soft and coordinated at claim time -- NULL means no limit (unlimited concurrency) - concurrency_limit integer -); + concurrency_limit integer, + group_concurrency_limit integer, -create table pgconductor._private_concurrency_slots ( - task_key text not null, - slot_group_number integer not null, - capacity integer not null, - used integer default 0 not null, - primary key (task_key, slot_group_number) -); + -- Destination copied onto each source task registration. + dead_letter_queue text, + dead_letter_task_key text, + + constraint positive_concurrency_limits check ( + (concurrency_limit is null or concurrency_limit > 0) and + (group_concurrency_limit is null or group_concurrency_limit > 0) + ), + constraint dead_letter_not_self check ( + dead_letter_queue is null or dead_letter_queue <> queue or + (dead_letter_task_key is not null and dead_letter_task_key <> key) + ), -create index idx_slots_claim - on pgconductor._private_concurrency_slots (task_key, capacity, used); + primary key (queue, key) +); create table pgconductor._private_steps ( id uuid default pgconductor._private_portable_uuidv7() primary key, @@ -171,6 +185,10 @@ create table pgconductor._private_steps ( create index idx_steps_execution_id on pgconductor._private_steps (execution_id); +create unique index idx_executions_dead_letter_delivery + on pgconductor._private_executions (dead_letter_source_execution_id, queue, task_key) + where dead_letter_source_execution_id is not null; + -- Trigger function to manage executions partitions per queue -- Automatically creates partition when queue is inserted create or replace function pgconductor._private_manage_queue_partition() @@ -196,7 +214,7 @@ begin -- main index for fetching available executions execute format( - 'create index if not exists %I on pgconductor.%I (priority, run_at) include (id, task_key) where is_available = true', + 'create index if not exists %I on pgconductor.%I (priority, run_at, created_at, id) include (task_key) where is_available = true', 'idx_' || v_partition_name || '_get_executions', v_partition_name ); @@ -250,6 +268,19 @@ begin v_partition_name ); + -- indexes used to count active executions for soft concurrency limits + execute format( + 'create index if not exists %I on pgconductor.%I (task_key) where locked_at is not null and failed_at is null and completed_at is null', + 'idx_' || v_partition_name || '_active_task', + v_partition_name + ); + + execute format( + 'create index if not exists %I on pgconductor.%I (task_key, "group") where "group" is not null and locked_at is not null and failed_at is null and completed_at is null', + 'idx_' || v_partition_name || '_active_task_group', + v_partition_name + ); + RETURN NEW; elsif tg_op = 'UPDATE' then @@ -312,7 +343,8 @@ create type pgconductor.execution_spec as ( dedupe_seconds integer, dedupe_next_slot boolean, cron_expression text, - priority integer + priority integer, + "group" text ); create type pgconductor.task_spec as ( @@ -323,7 +355,10 @@ create type pgconductor.task_spec as ( remove_on_fail_days integer, window_start timetz, window_end timetz, - concurrency_limit integer + concurrency_limit integer, + group_concurrency_limit integer, + dead_letter_queue text, + dead_letter_task_key text ); create type pgconductor._private_event_operation as enum ( @@ -341,7 +376,8 @@ create type pgconductor.event_subscription_spec as ( operation pgconductor._private_event_operation, when_clause text, payload_fields text[], - column_names text[] + column_names text[], + filter jsonb ); create or replace function pgconductor._private_register_worker( @@ -356,13 +392,37 @@ volatile set search_path to '' as $function$ begin + -- Filter arrays are equality allowlists; an empty list is almost always a + -- configuration mistake and must not silently match nothing. + if exists ( + select 1 + from unnest(p_event_subscriptions) as spec + cross join lateral jsonb_each(coalesce(spec.filter, '{}'::jsonb)) as f + where jsonb_typeof(f.value) <> 'array' or jsonb_array_length(f.value) = 0 + ) then + raise exception 'event subscription filters must contain non-empty arrays'; + end if; + if exists ( + select 1 from unnest(p_event_subscriptions) as spec + where spec.event_key is not null and spec.when_clause is not null + ) then + raise exception 'custom event subscriptions do not support when clauses'; + end if; + -- step 1: upsert queue (triggers partition creation) insert into pgconductor._private_queues (name) values (p_queue_name) on conflict (name) do nothing; + -- Dead-letter destinations may not have a worker yet; create their partitions. + insert into pgconductor._private_queues (name) + select distinct spec.dead_letter_queue + from unnest(p_task_specs) as spec + where spec.dead_letter_queue is not null + on conflict (name) do nothing; + -- step 2: register/update tasks - insert into pgconductor._private_tasks (key, queue, max_attempts, remove_on_complete_days, remove_on_fail_days, window_start, window_end, concurrency_limit) + insert into pgconductor._private_tasks (key, queue, max_attempts, remove_on_complete_days, remove_on_fail_days, window_start, window_end, concurrency_limit, group_concurrency_limit, dead_letter_queue, dead_letter_task_key) select spec.key, coalesce(spec.queue, 'default'), @@ -371,9 +431,12 @@ begin spec.remove_on_fail_days, spec.window_start, spec.window_end, - spec.concurrency_limit + spec.concurrency_limit, + spec.group_concurrency_limit, + spec.dead_letter_queue, + spec.dead_letter_task_key from unnest(p_task_specs) as spec - on conflict (key) + on conflict (queue, key) do update set queue = coalesce(excluded.queue, pgconductor._private_tasks.queue), max_attempts = coalesce(excluded.max_attempts, pgconductor._private_tasks.max_attempts), @@ -381,51 +444,28 @@ begin remove_on_fail_days = excluded.remove_on_fail_days, window_start = excluded.window_start, window_end = excluded.window_end, - concurrency_limit = excluded.concurrency_limit; + concurrency_limit = excluded.concurrency_limit, + group_concurrency_limit = excluded.group_concurrency_limit, + dead_letter_queue = excluded.dead_letter_queue, + dead_letter_task_key = excluded.dead_letter_task_key; - -- step 2a: manage concurrency slots - -- create one row per slot (capacity=1 each) - insert into pgconductor._private_concurrency_slots (task_key, slot_group_number, capacity, used) - select - spec.key, - slot_num, - 1 as capacity, - 0 as used - from unnest(p_task_specs) as spec - cross join lateral generate_series(1, spec.concurrency_limit) as slot_num - where spec.concurrency_limit is not null - on conflict (task_key, slot_group_number) - do update set - capacity = excluded.capacity, - used = least(pgconductor._private_concurrency_slots.used, excluded.capacity); - - -- clean up orphaned slots (tasks removed or concurrency_limit set to null) - delete from pgconductor._private_concurrency_slots - where task_key not in ( - select key from pgconductor._private_tasks - where concurrency_limit is not null - ); - - -- clean up excess slots when concurrency decreased - delete from pgconductor._private_concurrency_slots cs - where cs.slot_group_number > ( - select concurrency_limit - from pgconductor._private_tasks t - where t.key = cs.task_key - ); - - -- step 3: insert scheduled cron executions (on conflict do nothing) - insert into pgconductor._private_executions (task_key, queue, payload, run_at, dedupe_key, cron_expression) + -- step 3: insert scheduled cron executions + insert into pgconductor._private_executions (task_key, queue, payload, run_at, dedupe_key, cron_expression, "group") select spec.task_key, coalesce(spec.queue, 'default'), coalesce(spec.payload, '{}'::jsonb), coalesce(spec.run_at, pgconductor._private_current_time()), spec.dedupe_key, - spec.cron_expression + spec.cron_expression, + spec."group" from unnest(p_cron_schedules) as spec where spec.dedupe_key is not null - on conflict (task_key, dedupe_key, queue) do nothing; + on conflict (task_key, dedupe_key, queue) do update set + payload = excluded.payload, + run_at = excluded.run_at, + cron_expression = excluded.cron_expression, + "group" = excluded."group"; -- step 4: clean up stale schedules for this queue -- delete future executions for schedules that no longer exist @@ -468,10 +508,13 @@ begin s.operation, s.when_clause, s.payload_fields, - s.column_names + s.column_names, + s.filter, + 'task_trigger' as kind from unnest(p_event_subscriptions) as s ) as source on ( + target.kind = 'task_trigger' and target.queue = source.queue and target.task_key = source.task_key and coalesce(target.event_key, '') = coalesce(source.event_key, '') and @@ -482,20 +525,23 @@ begin coalesce(array_to_string(target.payload_fields, ','), '') = coalesce(array_to_string(source.payload_fields, ','), '') and coalesce(array_to_string(target.column_names, ','), '') = - coalesce(array_to_string(source.column_names, ','), '') + coalesce(array_to_string(source.column_names, ','), '') and + target.filter is not distinct from source.filter ) when not matched then insert ( task_key, queue, event_key, schema_name, table_name, operation, - when_clause, payload_fields, column_names + when_clause, payload_fields, column_names, filter, kind ) values ( source.task_key, source.queue, source.event_key, source.schema_name, source.table_name, source.operation, - source.when_clause, source.payload_fields, source.column_names + source.when_clause, source.payload_fields, source.column_names, source.filter, + source.kind ); -- step 6: delete old subscriptions for this queue not in new set delete from pgconductor._private_event_subscriptions target where target.queue = p_queue_name + and target.kind = 'task_trigger' and not exists ( select 1 from unnest(p_event_subscriptions) source where target.task_key = source.task_key @@ -508,6 +554,7 @@ begin coalesce(array_to_string(source.payload_fields, ','), '') and coalesce(array_to_string(target.column_names, ','), '') = coalesce(array_to_string(source.column_names, ','), '') + and target.filter is not distinct from source.filter ); end; $function$; @@ -526,19 +573,26 @@ begin v_now := pgconductor._private_current_time(); -- clear locked dedupe keys before batch insert - update pgconductor._private_executions as e + with superseded as ( + select e.id, e.queue, e.task_key + from pgconductor._private_executions as e + cross join unnest(specs) as spec + where e.dedupe_key = spec.dedupe_key + and e.task_key = spec.task_key + and e.queue = coalesce(spec.queue, 'default') + and e.locked_at is not null + and spec.dedupe_key is not null + for update of e + ) + update pgconductor._private_executions e set dedupe_key = null, locked_by = null, locked_at = null, failed_at = v_now, last_error = 'superseded by reinvoke' - from unnest(specs) as spec - where e.dedupe_key = spec.dedupe_key - and e.task_key = spec.task_key - and e.queue = coalesce(spec.queue, 'default') - and e.locked_at is not null - and spec.dedupe_key is not null; + from superseded s + where e.id = s.id; -- batch insert all executions -- note: duplicate dedupe_keys within same batch will cause error @@ -553,7 +607,8 @@ begin dedupe_key, singleton_on, cron_expression, - priority + priority, + "group" ) select pgconductor._private_portable_uuidv7(), @@ -572,14 +627,16 @@ begin else null end, spec.cron_expression, - coalesce(spec.priority, 0) + coalesce(spec.priority, 0), + spec."group" from unnest(specs) as spec on conflict (task_key, dedupe_key, queue) do update set payload = excluded.payload, run_at = excluded.run_at, priority = excluded.priority, cron_expression = excluded.cron_expression, - singleton_on = excluded.singleton_on + singleton_on = excluded.singleton_on, + "group" = excluded."group" returning pgconductor._private_executions.id; end; $function$ @@ -594,7 +651,8 @@ create or replace function pgconductor.invoke( p_dedupe_seconds integer default null, p_dedupe_next_slot boolean default false, p_cron_expression text default null, - p_priority integer default null + p_priority integer default null, + p_group text default null ) returns table(id uuid) language plpgsql @@ -613,17 +671,24 @@ begin -- clear locked dedupe key before insert (supersede pattern) if p_dedupe_key is not null then - update pgconductor._private_executions + with superseded as ( + select e.id, e.queue, e.task_key + from pgconductor._private_executions e + where e.dedupe_key = p_dedupe_key + and e.task_key = p_task_key + and e.queue = p_queue + and e.locked_at is not null + for update of e + ) + update pgconductor._private_executions e set dedupe_key = null, locked_by = null, locked_at = null, failed_at = v_now, last_error = 'superseded by reinvoke' - where dedupe_key = p_dedupe_key - and task_key = p_task_key - and queue = p_queue - and locked_at is not null; + from superseded s + where e.id = s.id; end if; -- singleton throttle/debounce logic @@ -647,7 +712,8 @@ begin dedupe_key, singleton_on, cron_expression, - priority + priority, + "group" ) values ( pgconductor._private_portable_uuidv7(), p_task_key, @@ -657,7 +723,8 @@ begin p_dedupe_key, v_singleton_on, p_cron_expression, - coalesce(p_priority, 0) + coalesce(p_priority, 0), + p_group ) on conflict (task_key, singleton_on, coalesce(dedupe_key, ''), queue) where singleton_on is not null and completed_at is null and failed_at is null and cancelled = false @@ -678,7 +745,8 @@ begin dedupe_key, singleton_on, cron_expression, - priority + priority, + "group" ) values ( pgconductor._private_portable_uuidv7(), p_task_key, @@ -688,35 +756,23 @@ begin p_dedupe_key, v_next_singleton_on, p_cron_expression, - coalesce(p_priority, 0) + coalesce(p_priority, 0), + p_group ) on conflict (task_key, singleton_on, coalesce(dedupe_key, ''), queue) where singleton_on is not null and completed_at is null and failed_at is null and cancelled = false do update set payload = excluded.payload, run_at = excluded.run_at, - priority = excluded.priority + priority = excluded.priority, + cron_expression = excluded.cron_expression, + "group" = excluded."group" returning _private_executions.id; return; end if; end if; -- regular invoke (no singleton) - if p_dedupe_key is not null then - -- clear keys that are currently locked so a subsequent insert can succeed. - update pgconductor._private_executions as e - set - dedupe_key = null, - locked_by = null, - locked_at = null, - failed_at = pgconductor._private_current_time(), - last_error = 'superseded by reinvoke' - where e.dedupe_key = p_dedupe_key - and e.task_key = p_task_key - and e.queue = p_queue - and e.locked_at is not null; - end if; - return query insert into pgconductor._private_executions as e ( id, task_key, @@ -725,7 +781,8 @@ begin run_at, dedupe_key, cron_expression, - priority + priority, + "group" ) values ( pgconductor._private_portable_uuidv7(), p_task_key, @@ -734,13 +791,15 @@ begin v_run_at, p_dedupe_key, p_cron_expression, - coalesce(p_priority, 0) + coalesce(p_priority, 0), + p_group ) on conflict (task_key, dedupe_key, queue) do update set payload = excluded.payload, run_at = excluded.run_at, priority = excluded.priority, - cron_expression = excluded.cron_expression + cron_expression = excluded.cron_expression, + "group" = excluded."group" returning e.id; end; $function$ @@ -759,6 +818,9 @@ as $function$ declare v_orchestrator_id uuid; v_queue text; + v_child_id uuid; + v_child_orchestrator_id uuid; + v_child_queue text; v_completed boolean; v_failed boolean; v_rows_affected integer; @@ -766,27 +828,79 @@ begin select locked_by, queue, + waiting_on_execution_id, completed_at is not null, failed_at is not null - into v_orchestrator_id, v_queue, v_completed, v_failed + into v_orchestrator_id, v_queue, v_child_id, v_completed, v_failed from pgconductor._private_executions - where id = p_execution_id; + where id = p_execution_id + for update; if not found or v_completed or v_failed then return false; end if; if v_orchestrator_id is null then - -- pending: fail immediately + -- pending: fail immediately. If this is a waiting parent, resolve its + -- child relationship in the same transaction so the child cannot become + -- orphaned or leave the workflow stranded. + if v_child_id is not null then + select locked_by, queue + into v_child_orchestrator_id, v_child_queue + from pgconductor._private_executions + where id = v_child_id + for update; + + if found and v_child_orchestrator_id is null then + update pgconductor._private_executions + set + failed_at = pgconductor._private_current_time(), + last_error = 'Cancelled: parent execution was cancelled', + locked_by = null, + locked_at = null, + waiting_on_execution_id = null, + waiting_step_key = null + where id = v_child_id + and completed_at is null + and failed_at is null; + elsif found then + update pgconductor._private_executions + set cancelled = true, last_error = p_reason + where id = v_child_id + and completed_at is null + and failed_at is null + and cancelled = false; + + get diagnostics v_rows_affected = row_count; + if v_rows_affected > 0 then + insert into pgconductor._private_orchestrator_signals + (orchestrator_id, type, execution_id, payload) + values ( + v_child_orchestrator_id, + 'cancel_execution', + v_child_id, + jsonb_build_object('queue', v_child_queue, 'reason', p_reason) + ) + on conflict (orchestrator_id, execution_id) + where type = 'cancel_execution' and execution_id is not null + do nothing; + end if; + end if; + end if; + update pgconductor._private_executions set failed_at = pgconductor._private_current_time(), last_error = p_reason, locked_by = null, - locked_at = null + locked_at = null, + waiting_on_execution_id = null, + waiting_step_key = null where id = p_execution_id and completed_at is null - and failed_at is null; + and failed_at is null + and locked_by is null + and locked_at is null; get diagnostics v_rows_affected = row_count; return v_rows_affected > 0; @@ -797,6 +911,8 @@ begin cancelled = true, last_error = p_reason where id = p_execution_id + and queue = v_queue + and locked_by = v_orchestrator_id and completed_at is null and cancelled = false; diff --git a/migrations/0000000002_events.sql b/migrations/0000000002_events.sql index fc29f37..abe7032 100644 --- a/migrations/0000000002_events.sql +++ b/migrations/0000000002_events.sql @@ -1,170 +1,413 @@ alter table pgconductor._private_executions - add column if not exists subscription_id uuid; + add column subscription_id uuid, + add column event_id uuid, + add column event_created_at timestamptz; + +create unique index idx_executions_event_subscription + on pgconductor._private_executions (event_created_at, event_id, subscription_id, queue) + where event_id is not null and event_created_at is not null and subscription_id is not null; + +-- Events are an append-only inbox. Fan-out and acknowledgement share one +-- transaction, so a failed processor leaves the event unprocessed. +create sequence pgconductor._private_event_position_seq as bigint; create table if not exists pgconductor._private_custom_events ( id uuid default pgconductor._private_portable_uuidv7() not null, event_key text not null, payload jsonb not null default '{}'::jsonb, + event_position bigint not null default nextval('pgconductor._private_event_position_seq'), created_at timestamptz default pgconductor._private_current_time() not null, + processed_at timestamptz, primary key (created_at, id) ) partition by range (created_at); -create index if not exists idx_custom_events_event_key - on pgconductor._private_custom_events (event_key, created_at desc); - --- Create initial partition for custom events (will cover many years) create table if not exists pgconductor._private_custom_events_default partition of pgconductor._private_custom_events for values from (minvalue) to (maxvalue); +create index if not exists idx_custom_events_wait_lookup + on pgconductor._private_custom_events (event_key, event_position, created_at, id); +create index if not exists idx_custom_events_pending + on pgconductor._private_custom_events (event_position, created_at, id) + where processed_at is null; + create table if not exists pgconductor._private_event_subscriptions ( id uuid primary key default pgconductor._private_portable_uuidv7(), - task_key text not null, queue text not null, - event_key text, schema_name text, table_name text, operation pgconductor._private_event_operation, - when_clause text, payload_fields text[], column_names text[], - + filter jsonb, + kind text not null default 'task_trigger', + execution_id uuid, + step_key text, + expires_at timestamptz, + wait_after_event_position bigint, created_at timestamptz not null default pgconductor._private_current_time(), - constraint chk_event_type check ( (event_key is not null and schema_name is null and table_name is null and operation is null) or (event_key is null and schema_name is not null and table_name is not null and operation is not null) - ) + ), + constraint chk_event_subscription_kind check (kind in ('task_trigger', 'execution_wait')), + constraint chk_event_subscription_shape check ( + (kind = 'task_trigger' and execution_id is null and step_key is null + and expires_at is null and wait_after_event_position is null) + or + (kind = 'execution_wait' and execution_id is not null and step_key is not null + and event_key is not null) + ), + constraint fk_event_subscription_execution foreign key (execution_id, queue) + references pgconductor._private_executions(id, queue) on delete cascade ); create index if not exists idx_event_subscriptions_custom on pgconductor._private_event_subscriptions (event_key) - where event_key is not null; - + where event_key is not null and kind = 'task_trigger'; +create index if not exists idx_event_subscriptions_wait_custom + on pgconductor._private_event_subscriptions (event_key, wait_after_event_position, expires_at) + where event_key is not null and kind = 'execution_wait'; create index if not exists idx_event_subscriptions_database on pgconductor._private_event_subscriptions (schema_name, table_name, operation) where schema_name is not null; +create index if not exists idx_event_subscriptions_wait_execution + on pgconductor._private_event_subscriptions (execution_id, queue, step_key) + where kind = 'execution_wait'; +create unique index if not exists idx_event_subscriptions_execution_step + on pgconductor._private_event_subscriptions (execution_id, step_key) + where execution_id is not null; + +-- Compiled equality constraints are deliberately separate from event payloads. +-- This keeps emission cheap while giving the set-wise matcher an indexed source. +create table if not exists pgconductor._private_event_subscription_filters ( + subscription_id uuid not null, + event_key text not null, + field_name text not null, + value jsonb not null, + primary key (subscription_id, field_name, value), + constraint fk_event_filter_subscription foreign key (subscription_id) + references pgconductor._private_event_subscriptions(id) on delete cascade +); +create index if not exists idx_event_subscription_filters_match + on pgconductor._private_event_subscription_filters (event_key, field_name, value); + +create table if not exists pgconductor._private_event_deliveries ( + event_created_at timestamptz not null, + event_id uuid not null, + subscription_id uuid not null, + delivered_at timestamptz not null default pgconductor._private_current_time(), + primary key (event_created_at, event_id, subscription_id), + constraint fk_event_delivery_event foreign key (event_created_at, event_id) + references pgconductor._private_custom_events(created_at, id) on delete cascade, + constraint fk_event_delivery_subscription foreign key (subscription_id) + references pgconductor._private_event_subscriptions(id) on delete cascade +); -create or replace function pgconductor._private_sync_custom_event_trigger() - returns trigger - language plpgsql - security definer - set search_path to '' -as $_$ -declare - v_invoke_blocks text; - v_has_subscriptions boolean; +create or replace function pgconductor._private_compile_event_filters() +returns trigger language plpgsql security definer set search_path to '' as $function$ begin - -- Only process custom event subscriptions (event_key is not null) - if coalesce(new.event_key, old.event_key) is null then - return coalesce(new, old); - end if; - - drop trigger if exists pgconductor_custom_event on pgconductor._private_custom_events; - drop function if exists pgconductor._private_trigger_custom_event; - - select exists( - select 1 - from pgconductor._private_event_subscriptions - where event_key is not null - ) into v_has_subscriptions; - - if v_has_subscriptions then - v_invoke_blocks := ( - select string_agg(format( - $sql$ - if new.event_key = %L and (%s) then - v_task_keys := array_append(v_task_keys, %L); - v_queues := array_append(v_queues, %L); - v_payloads := array_append(v_payloads, jsonb_build_object('event', new.event_key, 'payload', %s)); - v_subscription_ids := array_append(v_subscription_ids, %L); - end if; - $sql$, - sub.event_key, - coalesce(nullif(sub.when_clause, ''), 'true'), - sub.task_key, - t.queue, - pgconductor._private_build_payload_fields(sub.payload_fields, 'new.payload'), - sub.id - ), e'\n') - from pgconductor._private_event_subscriptions as sub - join pgconductor._private_tasks as t on t.key = sub.task_key - where sub.event_key is not null - ); - - execute format( - $sql$ - create or replace function pgconductor._private_trigger_custom_event() - returns trigger - language plpgsql - security definer - set search_path to '' - as $inner$ - declare - v_task_keys text[]; - v_queues text[]; - v_payloads jsonb[]; - v_subscription_ids uuid[]; - begin - %s - - if array_length(v_task_keys, 1) > 0 then - insert into pgconductor._private_executions (task_key, queue, payload, subscription_id) - select unnest(v_task_keys), unnest(v_queues), unnest(v_payloads), unnest(v_subscription_ids); - end if; - - return new; - end - $inner$ - $sql$, - v_invoke_blocks - ); - - execute $sql$ - create trigger pgconductor_custom_event - after insert on pgconductor._private_custom_events - for each row - execute function pgconductor._private_trigger_custom_event() - $sql$; - end if; - if tg_op = 'DELETE' then + delete from pgconductor._private_event_subscription_filters where subscription_id = old.id; return old; end if; - + delete from pgconductor._private_event_subscription_filters where subscription_id = new.id; + if new.event_key is not null and new.filter is not null then + insert into pgconductor._private_event_subscription_filters(subscription_id,event_key,field_name,value) + select new.id, new.event_key, f.key, values.value + from jsonb_each(new.filter) as f + cross join lateral jsonb_array_elements(f.value) as values(value) + on conflict do nothing; + end if; return new; end; -$_$; +$function$; -create trigger sync_custom_event_trigger - after insert or delete or update on pgconductor._private_event_subscriptions - for each row - execute function pgconductor._private_sync_custom_event_trigger(); +create trigger compile_event_filters + after insert or update or delete on pgconductor._private_event_subscriptions + for each row execute function pgconductor._private_compile_event_filters(); -create or replace function pgconductor._private_build_payload_fields( - p_payload_fields text[], - p_payload_expr text -) - returns text - language sql - immutable - set search_path to '' -as $_$ - select case - when p_payload_fields is null then p_payload_expr - else 'jsonb_build_object(' || array_to_string( - array( - select format('%L, %s->%L', field, p_payload_expr, field) - from unnest(p_payload_fields) as field - ), - ', ' - ) || ')' +create or replace function pgconductor._private_eval_event_when(p_expression text, p_payload jsonb) +returns boolean language plpgsql security definer set search_path to '' as $function$ +declare result boolean; +begin + -- `when` is retained as the legacy custom-event escape hatch. New filters + -- use compiled constraints; database-trigger `when` remains native below. + execute format('select (%s)', replace(p_expression, 'new.payload', '$1')) into result using p_payload; + return coalesce(result, false); +end; +$function$; + +create or replace function pgconductor._private_extract_event_payload(p_fields text[], p_payload jsonb) +returns jsonb language sql immutable set search_path to '' as $function$ + select case when p_fields is null then p_payload + else coalesce((select jsonb_object_agg(key, p_payload -> key) from unnest(p_fields) key), '{}'::jsonb) end; -$_$; +$function$; + +create or replace function pgconductor._private_resolve_event_waits( + p_batch_size integer default 100 +) returns integer language plpgsql volatile security definer set search_path to '' as $function$ +declare + v_now timestamptz; + v_wait record; + v_event record; + v_has_event boolean; + v_resolved integer := 0; +begin + perform pg_advisory_xact_lock(hashtext('pgconductor:event-waits')); + v_now := pgconductor._private_current_time(); + + -- Select resolvable subscriptions before applying the batch limit. This keeps + -- unmatched indefinite waits (and executions waiting on children, which have + -- no event subscription) from starving waits that can make progress. + -- Lock executions before subscriptions. Cancellation updates executions first + -- and its cleanup trigger then removes subscriptions in the same order. + for v_wait in + with candidate_subscriptions as materialized ( + select s.id subscription_id, s.execution_id, s.queue + from pgconductor._private_event_subscriptions s + join pgconductor._private_executions e + on e.id = s.execution_id and e.queue = s.queue + and e.waiting_step_key = s.step_key + where s.kind = 'execution_wait' + and e.completed_at is null and e.failed_at is null and e.cancelled = false + and ( + (s.expires_at is not null and s.expires_at <= v_now) + or exists ( + select 1 + from pgconductor._private_custom_events event + where event.event_key = s.event_key + and event.event_position > coalesce(s.wait_after_event_position, 0) + and (s.expires_at is null or event.created_at <= s.expires_at) + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = s.id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters a + where a.subscription_id = f.subscription_id + and a.field_name = f.field_name + and (event.payload -> a.field_name) = a.value + ) + ) + ) + ) + order by s.created_at, s.id + limit greatest(coalesce(p_batch_size, 0), 0) + ), locked_executions as materialized ( + select e.id, e.queue, e.waiting_step_key + from pgconductor._private_executions e + join candidate_subscriptions c on c.execution_id = e.id and c.queue = e.queue + where e.waiting_step_key is not null + and e.completed_at is null and e.failed_at is null and e.cancelled = false + order by e.id + for update skip locked + ), locked_waits as materialized ( + select e.id execution_id, e.queue, e.waiting_step_key step_key, + s.id subscription_id, s.event_key, s.payload_fields, + s.wait_after_event_position, s.expires_at + from locked_executions e + join pgconductor._private_event_subscriptions s + on s.execution_id = e.id and s.queue = e.queue + and s.step_key = e.waiting_step_key and s.kind = 'execution_wait' + for update of s + ) + select * from locked_waits + loop + -- Look at persisted events regardless of processed status. This prevents + -- concurrent task-event processors from changing wait delivery order. + select e.event_key, e.payload, e.event_position, e.created_at + into v_event + from pgconductor._private_custom_events e + where e.event_key = v_wait.event_key + and e.event_position > coalesce(v_wait.wait_after_event_position, 0) + and (v_wait.expires_at is null or e.created_at <= v_wait.expires_at) + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = v_wait.subscription_id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters a + where a.subscription_id = f.subscription_id + and a.field_name = f.field_name + and (e.payload -> a.field_name) = a.value + ) + ) + order by e.event_position, e.created_at, e.id + limit 1; + + v_has_event := found; + -- A matching event wins even if resolution runs after its deadline. + if v_has_event or (v_wait.expires_at is not null and v_wait.expires_at <= v_now) then + delete from pgconductor._private_event_subscriptions + where id = v_wait.subscription_id; + + if v_has_event then + insert into pgconductor._private_steps(execution_id, queue, key, result) + values ( + v_wait.execution_id, v_wait.queue, v_wait.step_key, + jsonb_build_object('result', jsonb_build_object( + 'name', v_event.event_key, + 'payload', pgconductor._private_extract_event_payload( + v_wait.payload_fields, v_event.payload + ) + )) + ) + on conflict (execution_id, key) do nothing; + else + insert into pgconductor._private_steps(execution_id, queue, key, result) + values ( + v_wait.execution_id, v_wait.queue, v_wait.step_key, + jsonb_build_object('__pgconductor_wait_for_event_timeout', true) + ) + on conflict (execution_id, key) do nothing; + end if; + + update pgconductor._private_executions + set run_at = v_now, waiting_on_execution_id = null, waiting_step_key = null + where id = v_wait.execution_id and queue = v_wait.queue; + v_resolved := v_resolved + 1; + end if; + end loop; + return v_resolved; +end; +$function$; + +create or replace function pgconductor._private_process_custom_events( + p_batch_size integer default 100 +) returns integer language plpgsql volatile security definer set search_path to '' as $function$ +declare + v_count integer := 0; + v_now timestamptz := pgconductor._private_current_time(); +begin + with candidates as materialized ( + select e.created_at, e.id, e.event_key, e.payload + from pgconductor._private_custom_events e + where e.processed_at is null + order by e.event_position, e.created_at, e.id + limit greatest(coalesce(p_batch_size, 0), 0) + for update skip locked + ), matches as materialized ( + select c.created_at as event_created_at, c.id as event_id, + s.id as subscription_id, s.task_key, s.queue, + pgconductor._private_extract_event_payload(s.payload_fields, c.payload) as selected_payload, + c.event_key + from candidates c + join pgconductor._private_event_subscriptions s on s.event_key = c.event_key + join pgconductor._private_tasks t on t.key = s.task_key and t.queue = s.queue + where s.kind = 'task_trigger' + and s.when_clause is null + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = s.id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters allowed + where allowed.subscription_id = f.subscription_id + and allowed.field_name = f.field_name + and (c.payload -> allowed.field_name) = allowed.value + ) + ) + ), inserted_deliveries as ( + insert into pgconductor._private_event_deliveries(event_created_at, event_id, subscription_id) + select event_created_at, event_id, subscription_id + from matches + on conflict do nothing + returning event_created_at, event_id, subscription_id + ), inserted_executions as ( + insert into pgconductor._private_executions( + task_key, queue, payload, event_created_at, event_id, subscription_id + ) + select m.task_key, m.queue, + jsonb_build_object('event', m.event_key, 'payload', m.selected_payload), + d.event_created_at, d.event_id, d.subscription_id + from inserted_deliveries d + join matches m using (event_created_at, event_id, subscription_id) + on conflict (event_created_at, event_id, subscription_id, queue) + where event_id is not null and event_created_at is not null and subscription_id is not null + do nothing + returning event_created_at, event_id + ), processed as ( + update pgconductor._private_custom_events e + set processed_at = v_now + from candidates c + where e.created_at = c.created_at + and e.id = c.id + returning e.id + ) + select count(*) into v_count from processed; + return v_count; +end; +$function$; + +-- Events are retained until acknowledged. This helper is safe to call from +-- maintenance: active/retryable work is never removed. +create or replace function pgconductor._private_remove_processed_events(p_before timestamptz, p_batch_size integer default 1000) +returns integer language sql volatile security definer set search_path to '' as $function$ + with event_wait_lock as materialized ( + select pg_advisory_xact_lock(hashtext('pgconductor:event-waits')) as locked + ), candidates as ( + select e.created_at, e.id + from pgconductor._private_custom_events e + cross join event_wait_lock + where e.processed_at is not null and e.processed_at < p_before + and not exists ( + select 1 + from pgconductor._private_event_subscriptions s + join pgconductor._private_executions x + on x.id = s.execution_id and x.queue = s.queue + and x.waiting_step_key = s.step_key + where s.kind = 'execution_wait' + and x.completed_at is null and x.failed_at is null and x.cancelled = false + and s.event_key = e.event_key + and e.event_position > coalesce(s.wait_after_event_position, 0) + and (s.expires_at is null or e.created_at <= s.expires_at) + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = s.id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters a + where a.subscription_id = f.subscription_id + and a.field_name = f.field_name + and (e.payload -> a.field_name) = a.value + ) + ) + ) + order by e.processed_at, e.created_at, e.id + limit greatest(coalesce(p_batch_size, 0), 0) + for update skip locked + ), deleted as ( + delete from pgconductor._private_custom_events e + using candidates c + where e.created_at = c.created_at and e.id = c.id + returning 1 + ) + select count(*)::integer from deleted; +$function$; + +create or replace function pgconductor.emit_event( + p_event_key text, + p_payload jsonb default '{}'::jsonb +) returns uuid language plpgsql volatile security definer set search_path to '' as $function$ +declare v_id uuid; +begin + perform pg_advisory_xact_lock(hashtext('pgconductor:event-waits')); + insert into pgconductor._private_custom_events (event_key, payload) + values (p_event_key, p_payload) + returning id into v_id; + return v_id; +end; +$function$; create or replace function pgconductor._private_build_column_list( p_column_names text[], @@ -222,7 +465,8 @@ begin select exists( select 1 from pgconductor._private_event_subscriptions - where table_name = v_table_name + where kind = 'task_trigger' + and table_name = v_table_name and schema_name = v_schema_name and operation = v_op ) into v_has_subscriptions; @@ -257,8 +501,9 @@ begin sub.id ), e'\n') from pgconductor._private_event_subscriptions as sub - join pgconductor._private_tasks as t on t.key = sub.task_key - where sub.table_name = v_table_name + join pgconductor._private_tasks as t on t.key = sub.task_key and t.queue = sub.queue + where sub.kind = 'task_trigger' + and sub.table_name = v_table_name and sub.schema_name = v_schema_name and sub.operation = v_op ); @@ -336,11 +581,31 @@ create or replace function pgconductor.emit_event( p_payload jsonb default '{}'::jsonb ) returns uuid - language sql + language plpgsql volatile + security definer set search_path to '' as $_$ +declare v_id uuid; +begin + perform pg_advisory_xact_lock(hashtext('pgconductor:event-waits')); insert into pgconductor._private_custom_events (event_key, payload) values (p_event_key, p_payload) - returning id; + returning id into v_id; + return v_id; +end; $_$; + + +create or replace function pgconductor._private_cleanup_execution_wait() +returns trigger language plpgsql security definer set search_path to '' as $function$ +begin + if new.cancelled or new.completed_at is not null or new.failed_at is not null then + delete from pgconductor._private_event_subscriptions + where execution_id = new.id and queue = new.queue and kind = 'execution_wait'; + end if; + return new; +end; +$function$; +create trigger cleanup_execution_wait after update of cancelled, completed_at, failed_at +on pgconductor._private_executions for each row execute function pgconductor._private_cleanup_execution_wait(); diff --git a/packages/pgconductor-js/src/conductor.ts b/packages/pgconductor-js/src/conductor.ts index ce75afc..8026a2a 100644 --- a/packages/pgconductor-js/src/conductor.ts +++ b/packages/pgconductor-js/src/conductor.ts @@ -8,6 +8,7 @@ import { type ValidateTasksQueue, type BatchConfig, type ExecuteFunction, + type ValidateDeadLetterConfiguration, } from "./task"; import type { TaskContext, BatchTaskContext } from "./task-context"; import { @@ -19,6 +20,7 @@ import { type TaskName, type Trigger, type ValidateTriggers, + type ValidateEventTriggers, } from "./task-definition"; import { Worker, type WorkerConfig } from "./worker"; import { DefaultLogger, type Logger } from "./lib/logger"; @@ -90,7 +92,8 @@ export class Conductor< // Inferred types from schemas Tasks extends readonly TaskDefinition[] = InferTasksFromSchema, - Events extends readonly EventDefinition[] = InferEventsFromSchema, + Events extends readonly EventDefinition[] = + InferEventsFromSchema, Database extends GenericDatabase = InferDatabaseFromSchema, > { /** @@ -162,8 +165,10 @@ export class Conductor< }, const TTriggers extends object | readonly object[], >( - definition: TDef, - triggers: TTriggers & ValidateTriggers>, + definition: TDef & ValidateDeadLetterConfiguration>, + triggers: TTriggers & + ValidateTriggers> & + ValidateEventTriggers, fn: TDef extends { readonly batch: BatchConfig } ? ResolvedReturns extends void ? ( @@ -190,6 +195,7 @@ export class Conductor< TaskContext & ExtraContext, TaskEventFromTriggers, Events, Database> > { + this.validateEventFilters(triggers); return Task.create< TDef["name"], ResolvedQueue, @@ -198,7 +204,11 @@ export class Conductor< TaskContext & ExtraContext, TaskEventFromTriggers, Events, Database> >( - definition as TaskConfiguration>, + definition as TaskConfiguration< + TDef["name"], + ResolvedQueue, + ResolvedPayload + >, triggers as NonEmptyArray | Trigger, fn as ExecuteFunction< TaskEventFromTriggers, Events, Database>, @@ -223,9 +233,42 @@ export class Conductor< this.logger, options.config, this.options.context, + this.options.events?.definitions ?? [], ); } + private validateEventFilters(triggers: object | readonly object[]): void { + const list = Array.isArray(triggers) ? triggers : [triggers]; + for (const trigger of list) { + if (!("event" in trigger)) continue; + const eventName = (trigger as { event: string }).event; + if ("when" in trigger) { + throw new Error(`Custom event "${eventName}" does not support a when clause`); + } + if (!("filter" in trigger) || !trigger.filter) continue; + const definition = this.options.events?.definitions.find( + (candidate: EventDefinition) => candidate.name === eventName, + ); + if (!definition) { + throw new Error(`Filtered event "${eventName}" has no runtime event definition`); + } + const allowed = new Set(definition.filterable ?? []); + for (const [field, values] of Object.entries(trigger.filter as Record)) { + if (!allowed.has(field)) { + throw new Error(`Filter for event "${eventName}" contains undeclared field "${field}"`); + } + if (!Array.isArray(values)) { + throw new Error( + `Filter value for event "${eventName}" field "${field}" must be an array`, + ); + } + if (values.length === 0) { + throw new Error(`Filter value for event "${eventName}" field "${field}" cannot be empty`); + } + } + } + } + async invoke( task: TTask, payload: InferPayload< @@ -270,6 +313,7 @@ export class Conductor< debounce: item.debounce, cron_expression: item.cron_expression, priority: item.priority, + group: item.group, })); return this.db.invokeBatch(specs); } @@ -292,6 +336,20 @@ export class Conductor< TName extends EventName, TDef extends FindEventByIdentifier = FindEventByIdentifier, >(event: TName, payload: InferEventPayload): Promise { + // Runtime schemas are deliberately validated before the database call. This + // keeps emit a persistence boundary: an invalid event can never be queued. + const definition = this.options.events?.definitions.find( + (candidate: EventDefinition) => candidate.name === event, + ); + const standard = (definition?.payload as any)?.["~standard"]; + if (standard?.validate) { + const result = await standard.validate(payload); + if (result && typeof result === "object" && "issues" in result && result.issues) { + throw new Error(`Invalid payload for event "${String(event)}"`); + } + payload = ((result as any)?.value ?? payload) as InferEventPayload; + } + return this.db.emitEvent({ eventKey: event, payload: payload as any, diff --git a/packages/pgconductor-js/src/database-client.ts b/packages/pgconductor-js/src/database-client.ts index c388fa2..df24d05 100644 --- a/packages/pgconductor-js/src/database-client.ts +++ b/packages/pgconductor-js/src/database-client.ts @@ -10,12 +10,14 @@ import { type CountActiveOrchestratorsBelowArgs, type GetExecutionsArgs, type RemoveExecutionsArgs, + type RemoveProcessedEventsArgs, type RegisterWorkerArgs, type ScheduleCronExecutionArgs, type UnscheduleCronExecutionArgs, type LoadStepArgs, type SaveStepArgs, type ClearWaitingStateArgs, + type RegisterEventWaitArgs, type EmitEventArgs, } from "./query-builder"; import { makeChildLogger, type Logger } from "./lib/logger"; @@ -33,6 +35,7 @@ export interface ExecutionSpec { debounce?: { seconds: number } | null; cron_expression?: string | null; priority?: number | null; + group?: string | null; parent_execution_id?: string | null; parent_step_key?: string | null; parent_timeout_ms?: number | null; @@ -46,8 +49,20 @@ export interface TaskSpec { removeOnFailDays?: number | null; window?: [string, string] | null; concurrency?: number | null; + groupConcurrency?: number | null; + deadLetterQueue?: string | null; + deadLetterTaskKey?: string | null; } +export type DeadLetterMetadata = { + sourceExecutionId: string; + sourceQueue: string; + sourceTaskKey: string; + error: string | null; + attempts: number; + failedAt: Date; +}; + export interface Execution { id: string; task_key: string; @@ -55,11 +70,18 @@ export interface Execution { payload: Payload; waiting_on_execution_id: string | null; waiting_step_key: string | null; + locked_by: string; cancelled: boolean; last_error: string | null; dedupe_key?: string | null; cron_expression?: string | null; - slot_group_number?: number | null; + group?: string | null; + dead_letter_source_execution_id?: string | null; + dead_letter_source_queue?: string | null; + dead_letter_source_task_key?: string | null; + dead_letter_error?: string | null; + dead_letter_attempts?: number | null; + dead_letter_failed_at?: Date | null; } // todo: move all of this to query-builder too or create new types.ts file @@ -73,6 +95,7 @@ export type ExecutionResult = export type GroupedExecutionResults = { count: number; + orchestratorId: string; completed: ExecutionCompleted[]; failed: (ExecutionFailed | ExecutionPermamentlyFailed)[]; released: ExecutionReleased[]; @@ -83,43 +106,45 @@ export type GroupedExecutionResults = { export interface ExecutionCompleted { execution_id: string; queue: string; + orchestrator_id: string; task_key: string; status: "completed"; result?: Payload; - slot_group_number?: number | null; } export interface ExecutionFailed { execution_id: string; queue: string; + orchestrator_id: string; task_key: string; status: "failed"; error: string; - slot_group_number?: number | null; } export interface ExecutionReleased { execution_id: string; queue: string; + orchestrator_id: string; task_key: string; status: "released"; reschedule_in_ms?: number | "infinity"; step_key?: string; - slot_group_number?: number | null; } export interface ExecutionPermamentlyFailed { execution_id: string; queue: string; + orchestrator_id: string; task_key: string; status: "permanently_failed"; error: string; - slot_group_number?: number | null; } export interface ExecutionInvokeChild { + group?: string | null; execution_id: string; queue: string; + orchestrator_id: string; task_key: string; status: "invoke_child"; timeout_ms: number | "infinity"; @@ -127,7 +152,6 @@ export interface ExecutionInvokeChild { child_task_name: string; child_task_queue: string; child_payload: Payload | null; - slot_group_number?: number | null; } export interface EventSubscriptionSpec { @@ -140,6 +164,7 @@ export interface EventSubscriptionSpec { when_clause: string | null; payload_fields: string[] | null; column_names: string[] | null; + filter: Record | null; } const RETRYABLE_SQLSTATE_CODES = new Set([ @@ -483,6 +508,17 @@ export class DatabaseClient { return deletedCount >= args.batchSize; } + async removeProcessedEvents( + args: RemoveProcessedEventsArgs, + opts?: QueryMethodOptions, + ): Promise { + const result = await this.query(() => this.builder.buildRemoveProcessedEvents(args), { + label: "removeProcessedEvents", + ...opts, + }); + return Number(result[0]?.deleted_count || 0) >= args.batchSize; + } + async registerWorker(args: RegisterWorkerArgs, opts?: QueryMethodOptions): Promise { await this.query(() => this.builder.buildRegisterWorker(args), { label: "registerWorker", @@ -549,6 +585,17 @@ export class DatabaseClient { }); } + async registerEventWait( + args: RegisterEventWaitArgs, + opts?: QueryMethodOptions, + ): Promise { + const result = await this.query(() => this.builder.buildRegisterEventWait(args), { + label: "registerEventWait", + ...opts, + }); + return result[0]?.registered === true; + } + async clearWaitingState(args: ClearWaitingStateArgs, opts?: QueryMethodOptions): Promise { await this.query(() => this.builder.buildClearWaitingState(args), { label: "clearWaitingState", @@ -556,6 +603,26 @@ export class DatabaseClient { }); } + async resolveEventWaits(args: { batchSize: number }, opts?: QueryMethodOptions): Promise { + const result = await this.query( + () => this.sql<{ resolved: number }[]>` + select pgconductor._private_resolve_event_waits(${args.batchSize}::integer) as resolved + `, + { label: "resolveEventWaits", ...opts }, + ); + return Number(result[0]?.resolved || 0); + } + + async processEvents(args: { batchSize: number }, opts?: QueryMethodOptions): Promise { + const result = await this.query( + () => this.sql<{ processed: number }[]>` + select pgconductor._private_process_custom_events(${args.batchSize}::integer) as processed + `, + { label: "processEvents", ...opts }, + ); + return Number(result[0]?.processed || 0); + } + async emitEvent(args: EmitEventArgs, opts?: QueryMethodOptions): Promise { const result = await this.query(() => this.builder.buildEmitEvent(args), { label: "emitEvent", diff --git a/packages/pgconductor-js/src/event-definition.ts b/packages/pgconductor-js/src/event-definition.ts index 6825092..8910249 100644 --- a/packages/pgconductor-js/src/event-definition.ts +++ b/packages/pgconductor-js/src/event-definition.ts @@ -2,10 +2,28 @@ import type { StandardSchemaV1 } from "@standard-schema/spec"; import type { ColumnSelectionError, SelectedRow, ValidateColumns } from "./select-columns"; type ObjectSchema = StandardSchemaV1; +type EnsureObject = T extends object ? T : {}; +type SchemaOutput = T extends StandardSchemaV1 ? O : T; +type EventPayload = T extends undefined ? {} : EnsureObject>; + +export type FilterableKeys = keyof EventPayload & string; +export type EventFilter = { + readonly [K in FilterableKeys]?: readonly EventPayload[K][]; +}; + +export type FilterForEvent = + T extends EventDefinition + ? { readonly [F in K & keyof EventPayload

]?: readonly EventPayload

[F][] } + : never; -export type EventDefinition = { +export type EventDefinition< + Name extends string, + Payload = undefined, + Filterable extends string = never, +> = { readonly name: Name; readonly payload: Payload; + readonly filterable?: readonly Filterable[]; }; /** @@ -21,35 +39,42 @@ export type DefineEvent< T extends { name: string; payload?: unknown; + filterable?: readonly (keyof EnsureObject & string)[]; }, -> = EventDefinition; +> = EventDefinition< + T["name"], + T extends { payload: infer P } ? P : undefined, + T extends { filterable: readonly (infer K extends string)[] } ? K : never +>; export function defineEvent(def: { name: Name; payload?: Payload; + filterable?: undefined; }): EventDefinition; +export function defineEvent< + Name extends string, + Payload extends ObjectSchema, + const Filterable extends readonly FilterableKeys[], +>(def: { + name: Name; + payload: Payload; + filterable: Filterable; +}): EventDefinition; export function defineEvent(def: any) { return def; } -export type EventName[]> = +export type EventName[]> = TEvents[number]["name"]; export type FindEventByIdentifier< - TEvents extends readonly EventDefinition[], + TEvents extends readonly EventDefinition[], TName extends string, > = Extract; -type EnsureObject = T extends object ? T : {}; - export type InferEventPayload = - T extends EventDefinition - ? P extends undefined - ? {} - : P extends StandardSchemaV1 - ? EnsureObject - : EnsureObject

// Plain type (type-only definition) - : never; + T extends EventDefinition ? EventPayload

: never; export type GenericDatabase = Record>; export type SchemaName = keyof TDatabase; diff --git a/packages/pgconductor-js/src/generated/sql.ts b/packages/pgconductor-js/src/generated/sql.ts index 94bedab..eeb078c 100644 --- a/packages/pgconductor-js/src/generated/sql.ts +++ b/packages/pgconductor-js/src/generated/sql.ts @@ -111,6 +111,7 @@ create table pgconductor._private_executions ( run_at timestamptz default pgconductor._private_current_time() not null, locked_at timestamptz, locked_by uuid, + "group" text, is_available boolean generated always as (locked_at is null and failed_at is null and completed_at is null) stored not null, attempts integer default 0 not null, last_error text, @@ -120,6 +121,14 @@ create table pgconductor._private_executions ( waiting_step_key text, parent_execution_id uuid, singleton_on timestamptz, + + -- Dead-letter metadata is denormalized so retained source rows are optional. + dead_letter_source_execution_id uuid, + dead_letter_source_queue text, + dead_letter_source_task_key text, + dead_letter_error text, + dead_letter_attempts integer, + dead_letter_failed_at timestamptz, primary key (id, queue), unique (task_key, dedupe_key, queue) ) partition by list (queue); @@ -132,7 +141,7 @@ create unique index on pgconductor._private_executions (task_key, singleton_on, where singleton_on is not null and completed_at is null and failed_at is null and cancelled = false; create table pgconductor._private_tasks ( - key text primary key, + key text not null, -- queue that this task belongs to (used for queue-based worker assignment) queue text default 'default' not null, @@ -158,21 +167,26 @@ create table pgconductor._private_tasks ( ) ), - -- concurrency control: maximum number of concurrent executions across all workers + -- concurrency controls are intentionally soft and coordinated at claim time -- NULL means no limit (unlimited concurrency) - concurrency_limit integer -); + concurrency_limit integer, + group_concurrency_limit integer, -create table pgconductor._private_concurrency_slots ( - task_key text not null, - slot_group_number integer not null, - capacity integer not null, - used integer default 0 not null, - primary key (task_key, slot_group_number) -); + -- Destination copied onto each source task registration. + dead_letter_queue text, + dead_letter_task_key text, -create index idx_slots_claim - on pgconductor._private_concurrency_slots (task_key, capacity, used); + constraint positive_concurrency_limits check ( + (concurrency_limit is null or concurrency_limit > 0) and + (group_concurrency_limit is null or group_concurrency_limit > 0) + ), + constraint dead_letter_not_self check ( + dead_letter_queue is null or dead_letter_queue <> queue or + (dead_letter_task_key is not null and dead_letter_task_key <> key) + ), + + primary key (queue, key) +); create table pgconductor._private_steps ( id uuid default pgconductor._private_portable_uuidv7() primary key, @@ -187,6 +201,10 @@ create table pgconductor._private_steps ( create index idx_steps_execution_id on pgconductor._private_steps (execution_id); +create unique index idx_executions_dead_letter_delivery + on pgconductor._private_executions (dead_letter_source_execution_id, queue, task_key) + where dead_letter_source_execution_id is not null; + -- Trigger function to manage executions partitions per queue -- Automatically creates partition when queue is inserted create or replace function pgconductor._private_manage_queue_partition() @@ -212,7 +230,7 @@ begin -- main index for fetching available executions execute format( - 'create index if not exists %I on pgconductor.%I (priority, run_at) include (id, task_key) where is_available = true', + 'create index if not exists %I on pgconductor.%I (priority, run_at, created_at, id) include (task_key) where is_available = true', 'idx_' || v_partition_name || '_get_executions', v_partition_name ); @@ -266,6 +284,19 @@ begin v_partition_name ); + -- indexes used to count active executions for soft concurrency limits + execute format( + 'create index if not exists %I on pgconductor.%I (task_key) where locked_at is not null and failed_at is null and completed_at is null', + 'idx_' || v_partition_name || '_active_task', + v_partition_name + ); + + execute format( + 'create index if not exists %I on pgconductor.%I (task_key, "group") where "group" is not null and locked_at is not null and failed_at is null and completed_at is null', + 'idx_' || v_partition_name || '_active_task_group', + v_partition_name + ); + RETURN NEW; elsif tg_op = 'UPDATE' then @@ -328,7 +359,8 @@ create type pgconductor.execution_spec as ( dedupe_seconds integer, dedupe_next_slot boolean, cron_expression text, - priority integer + priority integer, + "group" text ); create type pgconductor.task_spec as ( @@ -339,7 +371,10 @@ create type pgconductor.task_spec as ( remove_on_fail_days integer, window_start timetz, window_end timetz, - concurrency_limit integer + concurrency_limit integer, + group_concurrency_limit integer, + dead_letter_queue text, + dead_letter_task_key text ); create type pgconductor._private_event_operation as enum ( @@ -357,7 +392,8 @@ create type pgconductor.event_subscription_spec as ( operation pgconductor._private_event_operation, when_clause text, payload_fields text[], - column_names text[] + column_names text[], + filter jsonb ); create or replace function pgconductor._private_register_worker( @@ -372,13 +408,37 @@ volatile set search_path to '' as $function$ begin + -- Filter arrays are equality allowlists; an empty list is almost always a + -- configuration mistake and must not silently match nothing. + if exists ( + select 1 + from unnest(p_event_subscriptions) as spec + cross join lateral jsonb_each(coalesce(spec.filter, '{}'::jsonb)) as f + where jsonb_typeof(f.value) <> 'array' or jsonb_array_length(f.value) = 0 + ) then + raise exception 'event subscription filters must contain non-empty arrays'; + end if; + if exists ( + select 1 from unnest(p_event_subscriptions) as spec + where spec.event_key is not null and spec.when_clause is not null + ) then + raise exception 'custom event subscriptions do not support when clauses'; + end if; + -- step 1: upsert queue (triggers partition creation) insert into pgconductor._private_queues (name) values (p_queue_name) on conflict (name) do nothing; + -- Dead-letter destinations may not have a worker yet; create their partitions. + insert into pgconductor._private_queues (name) + select distinct spec.dead_letter_queue + from unnest(p_task_specs) as spec + where spec.dead_letter_queue is not null + on conflict (name) do nothing; + -- step 2: register/update tasks - insert into pgconductor._private_tasks (key, queue, max_attempts, remove_on_complete_days, remove_on_fail_days, window_start, window_end, concurrency_limit) + insert into pgconductor._private_tasks (key, queue, max_attempts, remove_on_complete_days, remove_on_fail_days, window_start, window_end, concurrency_limit, group_concurrency_limit, dead_letter_queue, dead_letter_task_key) select spec.key, coalesce(spec.queue, 'default'), @@ -387,9 +447,12 @@ begin spec.remove_on_fail_days, spec.window_start, spec.window_end, - spec.concurrency_limit + spec.concurrency_limit, + spec.group_concurrency_limit, + spec.dead_letter_queue, + spec.dead_letter_task_key from unnest(p_task_specs) as spec - on conflict (key) + on conflict (queue, key) do update set queue = coalesce(excluded.queue, pgconductor._private_tasks.queue), max_attempts = coalesce(excluded.max_attempts, pgconductor._private_tasks.max_attempts), @@ -397,51 +460,28 @@ begin remove_on_fail_days = excluded.remove_on_fail_days, window_start = excluded.window_start, window_end = excluded.window_end, - concurrency_limit = excluded.concurrency_limit; - - -- step 2a: manage concurrency slots - -- create one row per slot (capacity=1 each) - insert into pgconductor._private_concurrency_slots (task_key, slot_group_number, capacity, used) - select - spec.key, - slot_num, - 1 as capacity, - 0 as used - from unnest(p_task_specs) as spec - cross join lateral generate_series(1, spec.concurrency_limit) as slot_num - where spec.concurrency_limit is not null - on conflict (task_key, slot_group_number) - do update set - capacity = excluded.capacity, - used = least(pgconductor._private_concurrency_slots.used, excluded.capacity); - - -- clean up orphaned slots (tasks removed or concurrency_limit set to null) - delete from pgconductor._private_concurrency_slots - where task_key not in ( - select key from pgconductor._private_tasks - where concurrency_limit is not null - ); - - -- clean up excess slots when concurrency decreased - delete from pgconductor._private_concurrency_slots cs - where cs.slot_group_number > ( - select concurrency_limit - from pgconductor._private_tasks t - where t.key = cs.task_key - ); + concurrency_limit = excluded.concurrency_limit, + group_concurrency_limit = excluded.group_concurrency_limit, + dead_letter_queue = excluded.dead_letter_queue, + dead_letter_task_key = excluded.dead_letter_task_key; - -- step 3: insert scheduled cron executions (on conflict do nothing) - insert into pgconductor._private_executions (task_key, queue, payload, run_at, dedupe_key, cron_expression) + -- step 3: insert scheduled cron executions + insert into pgconductor._private_executions (task_key, queue, payload, run_at, dedupe_key, cron_expression, "group") select spec.task_key, coalesce(spec.queue, 'default'), coalesce(spec.payload, '{}'::jsonb), coalesce(spec.run_at, pgconductor._private_current_time()), spec.dedupe_key, - spec.cron_expression + spec.cron_expression, + spec."group" from unnest(p_cron_schedules) as spec where spec.dedupe_key is not null - on conflict (task_key, dedupe_key, queue) do nothing; + on conflict (task_key, dedupe_key, queue) do update set + payload = excluded.payload, + run_at = excluded.run_at, + cron_expression = excluded.cron_expression, + "group" = excluded."group"; -- step 4: clean up stale schedules for this queue -- delete future executions for schedules that no longer exist @@ -484,10 +524,13 @@ begin s.operation, s.when_clause, s.payload_fields, - s.column_names + s.column_names, + s.filter, + 'task_trigger' as kind from unnest(p_event_subscriptions) as s ) as source on ( + target.kind = 'task_trigger' and target.queue = source.queue and target.task_key = source.task_key and coalesce(target.event_key, '') = coalesce(source.event_key, '') and @@ -498,20 +541,23 @@ begin coalesce(array_to_string(target.payload_fields, ','), '') = coalesce(array_to_string(source.payload_fields, ','), '') and coalesce(array_to_string(target.column_names, ','), '') = - coalesce(array_to_string(source.column_names, ','), '') + coalesce(array_to_string(source.column_names, ','), '') and + target.filter is not distinct from source.filter ) when not matched then insert ( task_key, queue, event_key, schema_name, table_name, operation, - when_clause, payload_fields, column_names + when_clause, payload_fields, column_names, filter, kind ) values ( source.task_key, source.queue, source.event_key, source.schema_name, source.table_name, source.operation, - source.when_clause, source.payload_fields, source.column_names + source.when_clause, source.payload_fields, source.column_names, source.filter, + source.kind ); -- step 6: delete old subscriptions for this queue not in new set delete from pgconductor._private_event_subscriptions target where target.queue = p_queue_name + and target.kind = 'task_trigger' and not exists ( select 1 from unnest(p_event_subscriptions) source where target.task_key = source.task_key @@ -524,6 +570,7 @@ begin coalesce(array_to_string(source.payload_fields, ','), '') and coalesce(array_to_string(target.column_names, ','), '') = coalesce(array_to_string(source.column_names, ','), '') + and target.filter is not distinct from source.filter ); end; $function$; @@ -542,19 +589,26 @@ begin v_now := pgconductor._private_current_time(); -- clear locked dedupe keys before batch insert - update pgconductor._private_executions as e + with superseded as ( + select e.id, e.queue, e.task_key + from pgconductor._private_executions as e + cross join unnest(specs) as spec + where e.dedupe_key = spec.dedupe_key + and e.task_key = spec.task_key + and e.queue = coalesce(spec.queue, 'default') + and e.locked_at is not null + and spec.dedupe_key is not null + for update of e + ) + update pgconductor._private_executions e set dedupe_key = null, locked_by = null, locked_at = null, failed_at = v_now, last_error = 'superseded by reinvoke' - from unnest(specs) as spec - where e.dedupe_key = spec.dedupe_key - and e.task_key = spec.task_key - and e.queue = coalesce(spec.queue, 'default') - and e.locked_at is not null - and spec.dedupe_key is not null; + from superseded s + where e.id = s.id; -- batch insert all executions -- note: duplicate dedupe_keys within same batch will cause error @@ -569,7 +623,8 @@ begin dedupe_key, singleton_on, cron_expression, - priority + priority, + "group" ) select pgconductor._private_portable_uuidv7(), @@ -588,14 +643,16 @@ begin else null end, spec.cron_expression, - coalesce(spec.priority, 0) + coalesce(spec.priority, 0), + spec."group" from unnest(specs) as spec on conflict (task_key, dedupe_key, queue) do update set payload = excluded.payload, run_at = excluded.run_at, priority = excluded.priority, cron_expression = excluded.cron_expression, - singleton_on = excluded.singleton_on + singleton_on = excluded.singleton_on, + "group" = excluded."group" returning pgconductor._private_executions.id; end; $function$ @@ -610,7 +667,8 @@ create or replace function pgconductor.invoke( p_dedupe_seconds integer default null, p_dedupe_next_slot boolean default false, p_cron_expression text default null, - p_priority integer default null + p_priority integer default null, + p_group text default null ) returns table(id uuid) language plpgsql @@ -629,17 +687,24 @@ begin -- clear locked dedupe key before insert (supersede pattern) if p_dedupe_key is not null then - update pgconductor._private_executions + with superseded as ( + select e.id, e.queue, e.task_key + from pgconductor._private_executions e + where e.dedupe_key = p_dedupe_key + and e.task_key = p_task_key + and e.queue = p_queue + and e.locked_at is not null + for update of e + ) + update pgconductor._private_executions e set dedupe_key = null, locked_by = null, locked_at = null, failed_at = v_now, last_error = 'superseded by reinvoke' - where dedupe_key = p_dedupe_key - and task_key = p_task_key - and queue = p_queue - and locked_at is not null; + from superseded s + where e.id = s.id; end if; -- singleton throttle/debounce logic @@ -663,7 +728,8 @@ begin dedupe_key, singleton_on, cron_expression, - priority + priority, + "group" ) values ( pgconductor._private_portable_uuidv7(), p_task_key, @@ -673,7 +739,8 @@ begin p_dedupe_key, v_singleton_on, p_cron_expression, - coalesce(p_priority, 0) + coalesce(p_priority, 0), + p_group ) on conflict (task_key, singleton_on, coalesce(dedupe_key, ''), queue) where singleton_on is not null and completed_at is null and failed_at is null and cancelled = false @@ -694,7 +761,8 @@ begin dedupe_key, singleton_on, cron_expression, - priority + priority, + "group" ) values ( pgconductor._private_portable_uuidv7(), p_task_key, @@ -704,35 +772,23 @@ begin p_dedupe_key, v_next_singleton_on, p_cron_expression, - coalesce(p_priority, 0) + coalesce(p_priority, 0), + p_group ) on conflict (task_key, singleton_on, coalesce(dedupe_key, ''), queue) where singleton_on is not null and completed_at is null and failed_at is null and cancelled = false do update set payload = excluded.payload, run_at = excluded.run_at, - priority = excluded.priority + priority = excluded.priority, + cron_expression = excluded.cron_expression, + "group" = excluded."group" returning _private_executions.id; return; end if; end if; -- regular invoke (no singleton) - if p_dedupe_key is not null then - -- clear keys that are currently locked so a subsequent insert can succeed. - update pgconductor._private_executions as e - set - dedupe_key = null, - locked_by = null, - locked_at = null, - failed_at = pgconductor._private_current_time(), - last_error = 'superseded by reinvoke' - where e.dedupe_key = p_dedupe_key - and e.task_key = p_task_key - and e.queue = p_queue - and e.locked_at is not null; - end if; - return query insert into pgconductor._private_executions as e ( id, task_key, @@ -741,7 +797,8 @@ begin run_at, dedupe_key, cron_expression, - priority + priority, + "group" ) values ( pgconductor._private_portable_uuidv7(), p_task_key, @@ -750,13 +807,15 @@ begin v_run_at, p_dedupe_key, p_cron_expression, - coalesce(p_priority, 0) + coalesce(p_priority, 0), + p_group ) on conflict (task_key, dedupe_key, queue) do update set payload = excluded.payload, run_at = excluded.run_at, priority = excluded.priority, - cron_expression = excluded.cron_expression + cron_expression = excluded.cron_expression, + "group" = excluded."group" returning e.id; end; $function$ @@ -775,6 +834,9 @@ as $function$ declare v_orchestrator_id uuid; v_queue text; + v_child_id uuid; + v_child_orchestrator_id uuid; + v_child_queue text; v_completed boolean; v_failed boolean; v_rows_affected integer; @@ -782,27 +844,79 @@ begin select locked_by, queue, + waiting_on_execution_id, completed_at is not null, failed_at is not null - into v_orchestrator_id, v_queue, v_completed, v_failed + into v_orchestrator_id, v_queue, v_child_id, v_completed, v_failed from pgconductor._private_executions - where id = p_execution_id; + where id = p_execution_id + for update; if not found or v_completed or v_failed then return false; end if; if v_orchestrator_id is null then - -- pending: fail immediately + -- pending: fail immediately. If this is a waiting parent, resolve its + -- child relationship in the same transaction so the child cannot become + -- orphaned or leave the workflow stranded. + if v_child_id is not null then + select locked_by, queue + into v_child_orchestrator_id, v_child_queue + from pgconductor._private_executions + where id = v_child_id + for update; + + if found and v_child_orchestrator_id is null then + update pgconductor._private_executions + set + failed_at = pgconductor._private_current_time(), + last_error = 'Cancelled: parent execution was cancelled', + locked_by = null, + locked_at = null, + waiting_on_execution_id = null, + waiting_step_key = null + where id = v_child_id + and completed_at is null + and failed_at is null; + elsif found then + update pgconductor._private_executions + set cancelled = true, last_error = p_reason + where id = v_child_id + and completed_at is null + and failed_at is null + and cancelled = false; + + get diagnostics v_rows_affected = row_count; + if v_rows_affected > 0 then + insert into pgconductor._private_orchestrator_signals + (orchestrator_id, type, execution_id, payload) + values ( + v_child_orchestrator_id, + 'cancel_execution', + v_child_id, + jsonb_build_object('queue', v_child_queue, 'reason', p_reason) + ) + on conflict (orchestrator_id, execution_id) + where type = 'cancel_execution' and execution_id is not null + do nothing; + end if; + end if; + end if; + update pgconductor._private_executions set failed_at = pgconductor._private_current_time(), last_error = p_reason, locked_by = null, - locked_at = null + locked_at = null, + waiting_on_execution_id = null, + waiting_step_key = null where id = p_execution_id and completed_at is null - and failed_at is null; + and failed_at is null + and locked_by is null + and locked_at is null; get diagnostics v_rows_affected = row_count; return v_rows_affected > 0; @@ -813,6 +927,8 @@ begin cancelled = true, last_error = p_reason where id = p_execution_id + and queue = v_queue + and locked_by = v_orchestrator_id and completed_at is null and cancelled = false; @@ -842,172 +958,415 @@ $function$; `, "0000000002_events.sql": String.raw` alter table pgconductor._private_executions - add column if not exists subscription_id uuid; + add column subscription_id uuid, + add column event_id uuid, + add column event_created_at timestamptz; + +create unique index idx_executions_event_subscription + on pgconductor._private_executions (event_created_at, event_id, subscription_id, queue) + where event_id is not null and event_created_at is not null and subscription_id is not null; + +-- Events are an append-only inbox. Fan-out and acknowledgement share one +-- transaction, so a failed processor leaves the event unprocessed. +create sequence pgconductor._private_event_position_seq as bigint; create table if not exists pgconductor._private_custom_events ( id uuid default pgconductor._private_portable_uuidv7() not null, event_key text not null, payload jsonb not null default '{}'::jsonb, + event_position bigint not null default nextval('pgconductor._private_event_position_seq'), created_at timestamptz default pgconductor._private_current_time() not null, + processed_at timestamptz, primary key (created_at, id) ) partition by range (created_at); -create index if not exists idx_custom_events_event_key - on pgconductor._private_custom_events (event_key, created_at desc); - --- Create initial partition for custom events (will cover many years) create table if not exists pgconductor._private_custom_events_default partition of pgconductor._private_custom_events for values from (minvalue) to (maxvalue); +create index if not exists idx_custom_events_wait_lookup + on pgconductor._private_custom_events (event_key, event_position, created_at, id); +create index if not exists idx_custom_events_pending + on pgconductor._private_custom_events (event_position, created_at, id) + where processed_at is null; + create table if not exists pgconductor._private_event_subscriptions ( id uuid primary key default pgconductor._private_portable_uuidv7(), - task_key text not null, queue text not null, - event_key text, schema_name text, table_name text, operation pgconductor._private_event_operation, - when_clause text, payload_fields text[], column_names text[], - + filter jsonb, + kind text not null default 'task_trigger', + execution_id uuid, + step_key text, + expires_at timestamptz, + wait_after_event_position bigint, created_at timestamptz not null default pgconductor._private_current_time(), - constraint chk_event_type check ( (event_key is not null and schema_name is null and table_name is null and operation is null) or (event_key is null and schema_name is not null and table_name is not null and operation is not null) - ) + ), + constraint chk_event_subscription_kind check (kind in ('task_trigger', 'execution_wait')), + constraint chk_event_subscription_shape check ( + (kind = 'task_trigger' and execution_id is null and step_key is null + and expires_at is null and wait_after_event_position is null) + or + (kind = 'execution_wait' and execution_id is not null and step_key is not null + and event_key is not null) + ), + constraint fk_event_subscription_execution foreign key (execution_id, queue) + references pgconductor._private_executions(id, queue) on delete cascade ); create index if not exists idx_event_subscriptions_custom on pgconductor._private_event_subscriptions (event_key) - where event_key is not null; - + where event_key is not null and kind = 'task_trigger'; +create index if not exists idx_event_subscriptions_wait_custom + on pgconductor._private_event_subscriptions (event_key, wait_after_event_position, expires_at) + where event_key is not null and kind = 'execution_wait'; create index if not exists idx_event_subscriptions_database on pgconductor._private_event_subscriptions (schema_name, table_name, operation) where schema_name is not null; +create index if not exists idx_event_subscriptions_wait_execution + on pgconductor._private_event_subscriptions (execution_id, queue, step_key) + where kind = 'execution_wait'; +create unique index if not exists idx_event_subscriptions_execution_step + on pgconductor._private_event_subscriptions (execution_id, step_key) + where execution_id is not null; + +-- Compiled equality constraints are deliberately separate from event payloads. +-- This keeps emission cheap while giving the set-wise matcher an indexed source. +create table if not exists pgconductor._private_event_subscription_filters ( + subscription_id uuid not null, + event_key text not null, + field_name text not null, + value jsonb not null, + primary key (subscription_id, field_name, value), + constraint fk_event_filter_subscription foreign key (subscription_id) + references pgconductor._private_event_subscriptions(id) on delete cascade +); +create index if not exists idx_event_subscription_filters_match + on pgconductor._private_event_subscription_filters (event_key, field_name, value); + +create table if not exists pgconductor._private_event_deliveries ( + event_created_at timestamptz not null, + event_id uuid not null, + subscription_id uuid not null, + delivered_at timestamptz not null default pgconductor._private_current_time(), + primary key (event_created_at, event_id, subscription_id), + constraint fk_event_delivery_event foreign key (event_created_at, event_id) + references pgconductor._private_custom_events(created_at, id) on delete cascade, + constraint fk_event_delivery_subscription foreign key (subscription_id) + references pgconductor._private_event_subscriptions(id) on delete cascade +); -create or replace function pgconductor._private_sync_custom_event_trigger() - returns trigger - language plpgsql - security definer - set search_path to '' -as $_$ -declare - v_invoke_blocks text; - v_has_subscriptions boolean; +create or replace function pgconductor._private_compile_event_filters() +returns trigger language plpgsql security definer set search_path to '' as $function$ begin - -- Only process custom event subscriptions (event_key is not null) - if coalesce(new.event_key, old.event_key) is null then - return coalesce(new, old); + if tg_op = 'DELETE' then + delete from pgconductor._private_event_subscription_filters where subscription_id = old.id; + return old; + end if; + delete from pgconductor._private_event_subscription_filters where subscription_id = new.id; + if new.event_key is not null and new.filter is not null then + insert into pgconductor._private_event_subscription_filters(subscription_id,event_key,field_name,value) + select new.id, new.event_key, f.key, values.value + from jsonb_each(new.filter) as f + cross join lateral jsonb_array_elements(f.value) as values(value) + on conflict do nothing; end if; + return new; +end; +$function$; - drop trigger if exists pgconductor_custom_event on pgconductor._private_custom_events; - drop function if exists pgconductor._private_trigger_custom_event; +create trigger compile_event_filters + after insert or update or delete on pgconductor._private_event_subscriptions + for each row execute function pgconductor._private_compile_event_filters(); - select exists( - select 1 - from pgconductor._private_event_subscriptions - where event_key is not null - ) into v_has_subscriptions; +create or replace function pgconductor._private_eval_event_when(p_expression text, p_payload jsonb) +returns boolean language plpgsql security definer set search_path to '' as $function$ +declare result boolean; +begin + -- \`when\` is retained as the legacy custom-event escape hatch. New filters + -- use compiled constraints; database-trigger \`when\` remains native below. + execute format('select (%s)', replace(p_expression, 'new.payload', '$1')) into result using p_payload; + return coalesce(result, false); +end; +$function$; - if v_has_subscriptions then - v_invoke_blocks := ( - select string_agg(format( - $sql$ - if new.event_key = %L and (%s) then - v_task_keys := array_append(v_task_keys, %L); - v_queues := array_append(v_queues, %L); - v_payloads := array_append(v_payloads, jsonb_build_object('event', new.event_key, 'payload', %s)); - v_subscription_ids := array_append(v_subscription_ids, %L); - end if; - $sql$, - sub.event_key, - coalesce(nullif(sub.when_clause, ''), 'true'), - sub.task_key, - t.queue, - pgconductor._private_build_payload_fields(sub.payload_fields, 'new.payload'), - sub.id - ), e'\n') - from pgconductor._private_event_subscriptions as sub - join pgconductor._private_tasks as t on t.key = sub.task_key - where sub.event_key is not null - ); +create or replace function pgconductor._private_extract_event_payload(p_fields text[], p_payload jsonb) +returns jsonb language sql immutable set search_path to '' as $function$ + select case when p_fields is null then p_payload + else coalesce((select jsonb_object_agg(key, p_payload -> key) from unnest(p_fields) key), '{}'::jsonb) + end; +$function$; - execute format( - $sql$ - create or replace function pgconductor._private_trigger_custom_event() - returns trigger - language plpgsql - security definer - set search_path to '' - as $inner$ - declare - v_task_keys text[]; - v_queues text[]; - v_payloads jsonb[]; - v_subscription_ids uuid[]; - begin - %s - - if array_length(v_task_keys, 1) > 0 then - insert into pgconductor._private_executions (task_key, queue, payload, subscription_id) - select unnest(v_task_keys), unnest(v_queues), unnest(v_payloads), unnest(v_subscription_ids); - end if; - - return new; - end - $inner$ - $sql$, - v_invoke_blocks - ); +create or replace function pgconductor._private_resolve_event_waits( + p_batch_size integer default 100 +) returns integer language plpgsql volatile security definer set search_path to '' as $function$ +declare + v_now timestamptz; + v_wait record; + v_event record; + v_has_event boolean; + v_resolved integer := 0; +begin + perform pg_advisory_xact_lock(hashtext('pgconductor:event-waits')); + v_now := pgconductor._private_current_time(); - execute $sql$ - create trigger pgconductor_custom_event - after insert on pgconductor._private_custom_events - for each row - execute function pgconductor._private_trigger_custom_event() - $sql$; - end if; + -- Select resolvable subscriptions before applying the batch limit. This keeps + -- unmatched indefinite waits (and executions waiting on children, which have + -- no event subscription) from starving waits that can make progress. + -- Lock executions before subscriptions. Cancellation updates executions first + -- and its cleanup trigger then removes subscriptions in the same order. + for v_wait in + with candidate_subscriptions as materialized ( + select s.id subscription_id, s.execution_id, s.queue + from pgconductor._private_event_subscriptions s + join pgconductor._private_executions e + on e.id = s.execution_id and e.queue = s.queue + and e.waiting_step_key = s.step_key + where s.kind = 'execution_wait' + and e.completed_at is null and e.failed_at is null and e.cancelled = false + and ( + (s.expires_at is not null and s.expires_at <= v_now) + or exists ( + select 1 + from pgconductor._private_custom_events event + where event.event_key = s.event_key + and event.event_position > coalesce(s.wait_after_event_position, 0) + and (s.expires_at is null or event.created_at <= s.expires_at) + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = s.id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters a + where a.subscription_id = f.subscription_id + and a.field_name = f.field_name + and (event.payload -> a.field_name) = a.value + ) + ) + ) + ) + order by s.created_at, s.id + limit greatest(coalesce(p_batch_size, 0), 0) + ), locked_executions as materialized ( + select e.id, e.queue, e.waiting_step_key + from pgconductor._private_executions e + join candidate_subscriptions c on c.execution_id = e.id and c.queue = e.queue + where e.waiting_step_key is not null + and e.completed_at is null and e.failed_at is null and e.cancelled = false + order by e.id + for update skip locked + ), locked_waits as materialized ( + select e.id execution_id, e.queue, e.waiting_step_key step_key, + s.id subscription_id, s.event_key, s.payload_fields, + s.wait_after_event_position, s.expires_at + from locked_executions e + join pgconductor._private_event_subscriptions s + on s.execution_id = e.id and s.queue = e.queue + and s.step_key = e.waiting_step_key and s.kind = 'execution_wait' + for update of s + ) + select * from locked_waits + loop + -- Look at persisted events regardless of processed status. This prevents + -- concurrent task-event processors from changing wait delivery order. + select e.event_key, e.payload, e.event_position, e.created_at + into v_event + from pgconductor._private_custom_events e + where e.event_key = v_wait.event_key + and e.event_position > coalesce(v_wait.wait_after_event_position, 0) + and (v_wait.expires_at is null or e.created_at <= v_wait.expires_at) + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = v_wait.subscription_id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters a + where a.subscription_id = f.subscription_id + and a.field_name = f.field_name + and (e.payload -> a.field_name) = a.value + ) + ) + order by e.event_position, e.created_at, e.id + limit 1; + + v_has_event := found; + -- A matching event wins even if resolution runs after its deadline. + if v_has_event or (v_wait.expires_at is not null and v_wait.expires_at <= v_now) then + delete from pgconductor._private_event_subscriptions + where id = v_wait.subscription_id; + + if v_has_event then + insert into pgconductor._private_steps(execution_id, queue, key, result) + values ( + v_wait.execution_id, v_wait.queue, v_wait.step_key, + jsonb_build_object('result', jsonb_build_object( + 'name', v_event.event_key, + 'payload', pgconductor._private_extract_event_payload( + v_wait.payload_fields, v_event.payload + ) + )) + ) + on conflict (execution_id, key) do nothing; + else + insert into pgconductor._private_steps(execution_id, queue, key, result) + values ( + v_wait.execution_id, v_wait.queue, v_wait.step_key, + jsonb_build_object('__pgconductor_wait_for_event_timeout', true) + ) + on conflict (execution_id, key) do nothing; + end if; - if tg_op = 'DELETE' then - return old; - end if; + update pgconductor._private_executions + set run_at = v_now, waiting_on_execution_id = null, waiting_step_key = null + where id = v_wait.execution_id and queue = v_wait.queue; + v_resolved := v_resolved + 1; + end if; + end loop; + return v_resolved; +end; +$function$; - return new; +create or replace function pgconductor._private_process_custom_events( + p_batch_size integer default 100 +) returns integer language plpgsql volatile security definer set search_path to '' as $function$ +declare + v_count integer := 0; + v_now timestamptz := pgconductor._private_current_time(); +begin + with candidates as materialized ( + select e.created_at, e.id, e.event_key, e.payload + from pgconductor._private_custom_events e + where e.processed_at is null + order by e.event_position, e.created_at, e.id + limit greatest(coalesce(p_batch_size, 0), 0) + for update skip locked + ), matches as materialized ( + select c.created_at as event_created_at, c.id as event_id, + s.id as subscription_id, s.task_key, s.queue, + pgconductor._private_extract_event_payload(s.payload_fields, c.payload) as selected_payload, + c.event_key + from candidates c + join pgconductor._private_event_subscriptions s on s.event_key = c.event_key + join pgconductor._private_tasks t on t.key = s.task_key and t.queue = s.queue + where s.kind = 'task_trigger' + and s.when_clause is null + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = s.id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters allowed + where allowed.subscription_id = f.subscription_id + and allowed.field_name = f.field_name + and (c.payload -> allowed.field_name) = allowed.value + ) + ) + ), inserted_deliveries as ( + insert into pgconductor._private_event_deliveries(event_created_at, event_id, subscription_id) + select event_created_at, event_id, subscription_id + from matches + on conflict do nothing + returning event_created_at, event_id, subscription_id + ), inserted_executions as ( + insert into pgconductor._private_executions( + task_key, queue, payload, event_created_at, event_id, subscription_id + ) + select m.task_key, m.queue, + jsonb_build_object('event', m.event_key, 'payload', m.selected_payload), + d.event_created_at, d.event_id, d.subscription_id + from inserted_deliveries d + join matches m using (event_created_at, event_id, subscription_id) + on conflict (event_created_at, event_id, subscription_id, queue) + where event_id is not null and event_created_at is not null and subscription_id is not null + do nothing + returning event_created_at, event_id + ), processed as ( + update pgconductor._private_custom_events e + set processed_at = v_now + from candidates c + where e.created_at = c.created_at + and e.id = c.id + returning e.id + ) + select count(*) into v_count from processed; + return v_count; end; -$_$; +$function$; -create trigger sync_custom_event_trigger - after insert or delete or update on pgconductor._private_event_subscriptions - for each row - execute function pgconductor._private_sync_custom_event_trigger(); +-- Events are retained until acknowledged. This helper is safe to call from +-- maintenance: active/retryable work is never removed. +create or replace function pgconductor._private_remove_processed_events(p_before timestamptz, p_batch_size integer default 1000) +returns integer language sql volatile security definer set search_path to '' as $function$ + with event_wait_lock as materialized ( + select pg_advisory_xact_lock(hashtext('pgconductor:event-waits')) as locked + ), candidates as ( + select e.created_at, e.id + from pgconductor._private_custom_events e + cross join event_wait_lock + where e.processed_at is not null and e.processed_at < p_before + and not exists ( + select 1 + from pgconductor._private_event_subscriptions s + join pgconductor._private_executions x + on x.id = s.execution_id and x.queue = s.queue + and x.waiting_step_key = s.step_key + where s.kind = 'execution_wait' + and x.completed_at is null and x.failed_at is null and x.cancelled = false + and s.event_key = e.event_key + and e.event_position > coalesce(s.wait_after_event_position, 0) + and (s.expires_at is null or e.created_at <= s.expires_at) + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters f + where f.subscription_id = s.id + and not exists ( + select 1 + from pgconductor._private_event_subscription_filters a + where a.subscription_id = f.subscription_id + and a.field_name = f.field_name + and (e.payload -> a.field_name) = a.value + ) + ) + ) + order by e.processed_at, e.created_at, e.id + limit greatest(coalesce(p_batch_size, 0), 0) + for update skip locked + ), deleted as ( + delete from pgconductor._private_custom_events e + using candidates c + where e.created_at = c.created_at and e.id = c.id + returning 1 + ) + select count(*)::integer from deleted; +$function$; -create or replace function pgconductor._private_build_payload_fields( - p_payload_fields text[], - p_payload_expr text -) - returns text - language sql - immutable - set search_path to '' -as $_$ - select case - when p_payload_fields is null then p_payload_expr - else 'jsonb_build_object(' || array_to_string( - array( - select format('%L, %s->%L', field, p_payload_expr, field) - from unnest(p_payload_fields) as field - ), - ', ' - ) || ')' - end; -$_$; +create or replace function pgconductor.emit_event( + p_event_key text, + p_payload jsonb default '{}'::jsonb +) returns uuid language plpgsql volatile security definer set search_path to '' as $function$ +declare v_id uuid; +begin + perform pg_advisory_xact_lock(hashtext('pgconductor:event-waits')); + insert into pgconductor._private_custom_events (event_key, payload) + values (p_event_key, p_payload) + returning id into v_id; + return v_id; +end; +$function$; create or replace function pgconductor._private_build_column_list( p_column_names text[], @@ -1065,7 +1424,8 @@ begin select exists( select 1 from pgconductor._private_event_subscriptions - where table_name = v_table_name + where kind = 'task_trigger' + and table_name = v_table_name and schema_name = v_schema_name and operation = v_op ) into v_has_subscriptions; @@ -1100,8 +1460,9 @@ begin sub.id ), e'\n') from pgconductor._private_event_subscriptions as sub - join pgconductor._private_tasks as t on t.key = sub.task_key - where sub.table_name = v_table_name + join pgconductor._private_tasks as t on t.key = sub.task_key and t.queue = sub.queue + where sub.kind = 'task_trigger' + and sub.table_name = v_table_name and sub.schema_name = v_schema_name and sub.operation = v_op ); @@ -1179,13 +1540,33 @@ create or replace function pgconductor.emit_event( p_payload jsonb default '{}'::jsonb ) returns uuid - language sql + language plpgsql volatile + security definer set search_path to '' as $_$ +declare v_id uuid; +begin + perform pg_advisory_xact_lock(hashtext('pgconductor:event-waits')); insert into pgconductor._private_custom_events (event_key, payload) values (p_event_key, p_payload) - returning id; + returning id into v_id; + return v_id; +end; $_$; + + +create or replace function pgconductor._private_cleanup_execution_wait() +returns trigger language plpgsql security definer set search_path to '' as $function$ +begin + if new.cancelled or new.completed_at is not null or new.failed_at is not null then + delete from pgconductor._private_event_subscriptions + where execution_id = new.id and queue = new.queue and kind = 'execution_wait'; + end if; + return new; +end; +$function$; +create trigger cleanup_execution_wait after update of cancelled, completed_at, failed_at +on pgconductor._private_executions for each row execute function pgconductor._private_cleanup_execution_wait(); `, }); diff --git a/packages/pgconductor-js/src/index.ts b/packages/pgconductor-js/src/index.ts index 9689989..d131a52 100644 --- a/packages/pgconductor-js/src/index.ts +++ b/packages/pgconductor-js/src/index.ts @@ -4,3 +4,8 @@ export { Worker } from "./worker"; export { Task } from "./task"; export { SchemaManager } from "./schema-manager"; export { MigrationStore } from "./migration-store"; +export { WaitForEventTimeoutError } from "./task-context"; +export type { EventDefinition, DefineEvent, FilterForEvent } from "./event-definition"; +export { defineEvent } from "./event-definition"; +export { parseDuration } from "./lib/duration"; +export type { DurationInput, DurationUnit } from "./lib/duration"; diff --git a/packages/pgconductor-js/src/lib/duration.ts b/packages/pgconductor-js/src/lib/duration.ts new file mode 100644 index 0000000..a1e5051 --- /dev/null +++ b/packages/pgconductor-js/src/lib/duration.ts @@ -0,0 +1,43 @@ +export type DurationUnit = "ms" | "s" | "m" | "h" | "d"; +export type DurationInput = number | `${number}${DurationUnit}`; + +const DURATION_UNITS: Record = { + ms: 1, + s: 1_000, + m: 60_000, + h: 3_600_000, + d: 86_400_000, +}; + +function isDurationUnit(value: string): value is DurationUnit { + return value in DURATION_UNITS; +} + +/** Parse a non-negative duration into integer milliseconds. */ +export function parseDuration(value: DurationInput): number { + if (typeof value === "number") { + if (!Number.isFinite(value) || value < 0) { + throw new Error("duration must be a non-negative finite number"); + } + const milliseconds = Math.trunc(value); + if (!Number.isSafeInteger(milliseconds)) { + throw new Error("duration is too large"); + } + return milliseconds; + } + + const match = /^(\d+(?:\.\d+)?)(ms|s|m|h|d)$/.exec(value); + if (!match) { + throw new Error(`invalid duration: ${value}`); + } + + const unit = match[2]; + if (!unit || !isDurationUnit(unit)) { + throw new Error(`invalid duration: ${value}`); + } + const milliseconds = Number(match[1]) * DURATION_UNITS[unit]; + if (!Number.isSafeInteger(milliseconds)) { + throw new Error(`duration is too large: ${value}`); + } + return milliseconds; +} diff --git a/packages/pgconductor-js/src/lib/map-concurrent.ts b/packages/pgconductor-js/src/lib/map-concurrent.ts index 4eb0808..090589a 100644 --- a/packages/pgconductor-js/src/lib/map-concurrent.ts +++ b/packages/pgconductor-js/src/lib/map-concurrent.ts @@ -8,74 +8,55 @@ export async function* mapConcurrent( limit: number, mapper: (item: T) => Promise, ): AsyncGenerator { - const it = source[Symbol.asyncIterator](); + const iterator = source[Symbol.asyncIterator](); let sourceDone = false; + let pendingRead: Promise> | null = null; let nextId = 0; - // Track promises with unique IDs - type Task = { id: number; promise: Promise }; - const active = new Map(); + type ActiveTask = { id: number; promise: Promise }; + const active = new Map(); - const nextItem = async (): Promise => { - if (sourceDone) return null; - const { value, done } = await it.next(); - if (done) { - sourceDone = true; - return null; + const startRead = () => { + if (!sourceDone && !pendingRead && active.size < limit) { + pendingRead = iterator.next(); } - return value; }; - const fillSlots = async () => { - while (!sourceDone && active.size < limit) { - let item: T | null; - - if (active.size === 0) { - // No active tasks - MUST block to get at least one - item = await nextItem(); - } else { - // Try non-blocking poll - const polled = source.tryNext(); - if (polled === undefined) { - // Queue empty, stop filling - break; + try { + startRead(); + + const getPendingRead = (): Promise> | null => pendingRead; + + while (active.size > 0 || pendingRead) { + const read = getPendingRead(); + const reads = read ? [read.then((result) => ({ kind: "read" as const, result }))] : []; + const tasks = [...active.values()].map((task) => + task.promise.then((result) => ({ kind: "result" as const, id: task.id, result })), + ); + + const event = await Promise.race([...reads, ...tasks]); + if (event.kind === "read") { + pendingRead = null; + if (event.result.done) { + sourceDone = true; + } else { + const id = nextId++; + active.set(id, { id, promise: mapper(event.result.value) }); } - item = polled; + startRead(); + } else { + active.delete(event.id); + yield event.result; + startRead(); } - - if (item === null) break; - - const id = nextId++; - active.set(id, { id, promise: mapper(item) }); } - }; - - await fillSlots(); - - while (active.size > 0) { - // Wrap each promise to include its ID - const wrappedPromises = Array.from(active.values()).map(async (task) => ({ - id: task.id, - result: await task.promise, - })); - - // Race to get first completed task - const { id, result } = await Promise.race(wrappedPromises); - - // Remove the completed task - active.delete(id); - - yield result; - - // Refill slots - await fillSlots(); - } - - if (typeof it.return === "function") { - try { - await it.return(); - } catch { - // ignore + } finally { + if (typeof iterator.return === "function") { + try { + await iterator.return(); + } catch { + // Ignore cleanup errors. + } } } } diff --git a/packages/pgconductor-js/src/maintenance-task.ts b/packages/pgconductor-js/src/maintenance-task.ts index 0ab6a78..c41be59 100644 --- a/packages/pgconductor-js/src/maintenance-task.ts +++ b/packages/pgconductor-js/src/maintenance-task.ts @@ -13,6 +13,7 @@ function hashToJitter(str: string): number { // we could make this configurable later // should be lower for very high steps per task const BATCH_SIZE = 1000; +const EVENT_RETENTION_DAYS = 7; export const createMaintenanceTask = (queue: Queue) => { // Add consistent jitter based on queue name to spread load between midnight and 1am @@ -37,23 +38,31 @@ export const createMaintenanceTask = (queue: Q }, async (_, ctx) => { const { db, tasks, signal } = ctx; - // Skip if no tasks have retention settings (check in-memory config) + // Delete old completed/failed executions based on retention settings. + // Steps and event deliveries are removed by their foreign-key cascades. const hasRetention = Array.from(tasks.values()).some( (t) => t.removeOnComplete || t.removeOnFail, ); - if (!hasRetention) { - return; + if (hasRetention) { + let hasMore = true; + while (hasMore) { + hasMore = await db.removeExecutions( + { queueName: queue, batchSize: BATCH_SIZE }, + { signal }, + ); + } } - // Delete old completed/failed executions based on retention settings - // Steps are automatically deleted via CASCADE foreign key - let hasMore = true; - while (hasMore) { - hasMore = await db.removeExecutions( - { - queueName: queue, - batchSize: BATCH_SIZE, - }, + // Keep acknowledged inbox rows only for a bounded period. Process in + // batches so maintenance cannot monopolize the queue. + const before = new Date( + (await db.getCurrentTime({ signal })).getTime() - + EVENT_RETENTION_DAYS * 24 * 60 * 60 * 1000, + ); + let eventsRemain = true; + while (eventsRemain) { + eventsRemain = await db.removeProcessedEvents( + { before, batchSize: BATCH_SIZE }, { signal }, ); } diff --git a/packages/pgconductor-js/src/orchestrator.ts b/packages/pgconductor-js/src/orchestrator.ts index 8f68a64..2488dc9 100644 --- a/packages/pgconductor-js/src/orchestrator.ts +++ b/packages/pgconductor-js/src/orchestrator.ts @@ -48,6 +48,7 @@ export class Orchestrator { private _stopDeferred: Deferred | null = null; private _startDeferred: Deferred | null = null; private _abortController: AbortController | null = null; + private _eventProcessingGate: Deferred | null = null; private releaseSignalHandlers: (() => void) | null = null; @@ -69,17 +70,24 @@ export class Orchestrator { this.logger, options.defaultWorker, options.conductor.options.context, + options.conductor.options.events?.definitions ?? [], ); this.workers.push(worker); } for (const w of options.workers || []) { - if (this.workers.find((existing) => existing.queueName === w.queueName)) { - throw new Error(`Duplicate worker name: ${w.queueName}`); - } - this.workers.push(w); } + + const queues = new Set(); + for (const worker of this.workers) { + if (queues.has(worker.queueName)) { + throw new Error( + `Orchestrator cannot configure multiple workers for queue "${worker.queueName}"; configure one worker with all tasks for that queue`, + ); + } + queues.add(worker.queueName); + } } static create[]>( @@ -189,32 +197,55 @@ export class Orchestrator { // Start heartbeat loop this.startHeartbeatLoop(); - // Kick off all workers (don't await yet!) - if (runOnce) { - // Drain mode: workers will process and stop - this.workers.forEach((w) => void w.drain(this.orchestratorId)); - } else { - // Normal mode: workers will run continuously - this.workers.forEach((w) => void w.run(this.orchestratorId)); - } + const startWorkers = () => { + // Gate local event fan-out until every worker in this pass has + // registered. A pass is deliberately global: an event processor on + // one queue may create executions on every other queue. + this._eventProcessingGate = new Deferred(); + for (const worker of this.workers) + worker.setEventProcessingGate(this._eventProcessingGate.promise); + const workerLifecycles = this.workers.map((worker) => + runOnce ? worker.drain(this.orchestratorId) : worker.run(this.orchestratorId), + ); + return { + workerLifecycles, + allWorkers: Promise.all(workerLifecycles), + }; + }; - // Wait for ALL workers to finish starting (register() complete) + let { allWorkers } = startWorkers(); + + // Wait for ALL workers to finish registering before allowing event work. await Promise.all(this.workers.map((w) => w.started)); + this._eventProcessingGate?.resolve(); // NOW signal that orchestrator has started this.startDeferred.resolve(); - // Wait for shutdown signal or all workers to complete - await Promise.race([ - Promise.all(this.workers.map((w) => w.stopped)), - this.waitForShutdownSignal(), - ]); + if (runOnce) { + // A worker can observe its queue empty just before another queue + // finishes a task that fans an event out to it. Keep the workers as + // coordinated drain passes until an entire global pass observes no + // work. This is also why event processing is gated per pass rather + // than letting a queue permanently declare itself drained. + await allWorkers; + while (this.workers.some((worker) => worker.drainDidWork)) { + ({ allWorkers } = startWorkers()); + await Promise.all(this.workers.map((w) => w.started)); + this._eventProcessingGate?.resolve(); + await allWorkers; + } + } else { + // Wait for shutdown signal or all workers to complete + await Promise.race([allWorkers, this.waitForShutdownSignal()]); + } // Stop gracefully await this.stopWorkers(); } catch (err) { error = coerceError(err); this.logger.error(err); + await this.stopWorkers(); // Only reject startDeferred if startup hasn't completed yet if (!this.startDeferred.isSettled) { @@ -225,6 +256,7 @@ export class Orchestrator { const stopDeferred = this._stopDeferred; try { + this._eventProcessingGate?.resolve(); await this.cleanup(); } catch (cleanupErr) { this.logger.error("Cleanup failed:", cleanupErr); diff --git a/packages/pgconductor-js/src/query-builder.ts b/packages/pgconductor-js/src/query-builder.ts index a7f3a0f..dc6c2e6 100644 --- a/packages/pgconductor-js/src/query-builder.ts +++ b/packages/pgconductor-js/src/query-builder.ts @@ -37,7 +37,6 @@ export type GetExecutionsArgs = { queueName: string; batchSize: number; filterTaskKeys: string[]; - taskKeysWithConcurrency: string[]; }; export type RemoveExecutionsArgs = { @@ -45,6 +44,11 @@ export type RemoveExecutionsArgs = { batchSize: number; }; +export type RemoveProcessedEventsArgs = { + before: Date; + batchSize: number; +}; + export type RegisterWorkerArgs = { queueName: string; taskSpecs: TaskSpec[]; @@ -65,12 +69,15 @@ export type UnscheduleCronExecutionArgs = { export type LoadStepArgs = { executionId: string; + queue: string; + orchestratorId: string; key: string; }; export type SaveStepArgs = { executionId: string; queue: string; + orchestratorId: string; key: string; result: Payload | null; runAtMs?: number; @@ -78,6 +85,8 @@ export type SaveStepArgs = { export type ClearWaitingStateArgs = { executionId: string; + queue: string; + orchestratorId: string; }; export type EmitEventArgs = { @@ -85,6 +94,17 @@ export type EmitEventArgs = { payload?: JsonValue; }; +export type RegisterEventWaitArgs = { + executionId: string; + queue: string; + taskKey: string; + eventKey: string; + stepKey: string; + filter: Record | null; + timeoutMs: number | null; + orchestratorId: string; +}; + export class QueryBuilder { constructor(private readonly sql: Sql) {} @@ -178,6 +198,7 @@ export class QueryBuilder { and e.cancelled = true and e.failed_at is null and e.completed_at is null + returning e.id ) -- unlock remaining (non-cancelled) executions update pgconductor._private_executions e @@ -233,6 +254,7 @@ export class QueryBuilder { and e.cancelled = true and e.failed_at is null and e.completed_at is null + returning e.id ) -- unlock remaining (non-cancelled) executions update pgconductor._private_executions e @@ -250,567 +272,401 @@ export class QueryBuilder { queueName, batchSize, filterTaskKeys, - taskKeysWithConcurrency, }: GetExecutionsArgs): PendingQuery { - // fast path: no tasks have concurrency limits - if (!taskKeysWithConcurrency.length) { - return this.sql` - with e as ( - select - e.id, - e.task_key - from pgconductor._private_executions e - where e.queue = ${queueName}::text - ${filterTaskKeys?.length ? this.sql`and e.task_key != any(${this.sql.array(filterTaskKeys)}::text[])` : this.sql``} - and e.run_at <= pgconductor._private_current_time() - and e.is_available = true - order by e.priority asc, e.run_at asc - limit ${batchSize}::integer - for update skip locked - ) - - update pgconductor._private_executions + return this.sql` + with active_tasks as ( + select e.task_key, count(*)::integer as active_count + from pgconductor._private_executions e + where e.queue = ${queueName}::text + and e.locked_at is not null + and e.failed_at is null + and e.completed_at is null + group by e.task_key + ), active_groups as ( + select e.task_key, e."group", count(*)::integer as active_count + from pgconductor._private_executions e + where e.queue = ${queueName}::text + and e."group" is not null + and e.locked_at is not null + and e.failed_at is null + and e.completed_at is null + group by e.task_key, e."group" + ), ranked as ( + select + e.id, + e.task_key, + e.queue, + e.priority, + e.run_at, + e.created_at, + e."group", + t.concurrency_limit, + t.group_concurrency_limit, + coalesce(at.active_count, 0) as active_task_count, + coalesce(ag.active_count, 0) as active_group_count, + row_number() over ( + partition by e.task_key + order by e.priority asc, e.run_at asc, e.created_at asc, e.id asc + ) as task_rank, + row_number() over ( + partition by e.task_key, e."group" + order by e.priority asc, e.run_at asc, e.created_at asc, e.id asc + ) as group_rank + from pgconductor._private_executions e + left join pgconductor._private_tasks t + on t.key = e.task_key and t.queue = e.queue + left join active_tasks at on at.task_key = e.task_key + left join active_groups ag on ag.task_key = e.task_key and ag."group" = e."group" + where e.queue = ${queueName}::text + and e.run_at <= pgconductor._private_current_time() + and e.is_available = true + ${filterTaskKeys?.length ? this.sql`and not (e.task_key = any(${this.sql.array(filterTaskKeys)}::text[]))` : this.sql``} + ), group_eligible as ( + select r.*, + row_number() over ( + partition by r.task_key + order by r.priority asc, r.run_at asc, r.created_at asc, r.id asc + ) as available_task_rank + from ranked r + where r.group_concurrency_limit is null + or r."group" is null + or r.active_group_count + r.group_rank <= r.group_concurrency_limit + ), eligible as ( + select r.id, r.priority, r.run_at, r.created_at + from group_eligible r + where r.concurrency_limit is null + or r.active_task_count + r.available_task_rank <= r.concurrency_limit + order by r.priority asc, r.run_at asc, r.created_at asc, r.id asc + -- Keep a bounded candidate pool so SKIP LOCKED can backfill a batch. + limit greatest(${batchSize}::integer * 4, ${batchSize}::integer) + ), locked_candidates as ( + select e.id + from pgconductor._private_executions e + join eligible c on c.id = e.id + where e.queue = ${queueName}::text + order by c.priority asc, c.run_at asc, c.created_at asc, c.id asc + limit ${batchSize}::integer + for update of e skip locked + ), claimed as ( + update pgconductor._private_executions e set - attempts = _private_executions.attempts + 1, + attempts = e.attempts + 1, locked_by = ${orchestratorId}::uuid, locked_at = pgconductor._private_current_time() - from e - where _private_executions.id = e.id - and _private_executions.queue = ${queueName}::text - returning - _private_executions.id, - _private_executions.task_key, - _private_executions.queue, - _private_executions.payload, - _private_executions.waiting_on_execution_id, - _private_executions.waiting_step_key, - _private_executions.cancelled, - _private_executions.last_error, - _private_executions.dedupe_key, - _private_executions.cron_expression, - null as slot_group_number - `; - } - - // slow path: some tasks have concurrency limits (OPTIMIZED) - return this.sql` - with - -- lock up to batchSize slots per concurrency task - locked_slots_raw as ( - select t.task_key, ls.slot_group_number - from unnest(${this.sql.array(taskKeysWithConcurrency)}::text[]) as t(task_key) - cross join lateral ( - select s.slot_group_number - from pgconductor._private_concurrency_slots s - where s.task_key = t.task_key - and s.used = 0 - order by s.slot_group_number - limit ${batchSize}::integer - for update skip locked - ) as ls - ), - - -- count slots per task - slots_per_task as ( - select task_key, count(*) as slot_count - from locked_slots_raw - group by task_key - ), - - -- lock jobs for concurrency tasks (limit by slot count) - concurrency_execs as ( - select t.task_key, le.* - from slots_per_task t - cross join lateral ( - select - e.id, - e.task_key as exec_task_key, - e.queue, - e.payload, - e.waiting_on_execution_id, - e.waiting_step_key, - e.cancelled, - e.last_error, - e.dedupe_key, - e.cron_expression, - e.priority, - e.run_at - from pgconductor._private_executions e - where e.is_available = true - and e.run_at <= pgconductor._private_current_time() - and e.queue = ${queueName}::text - and e.task_key = t.task_key - ${filterTaskKeys.length ? this.sql`and not (e.task_key = any(${this.sql.array(filterTaskKeys)}::text[]))` : this.sql``} - order by e.priority asc, e.run_at asc, e.id asc - limit t.slot_count - for update skip locked - ) as le - ), - - -- lock jobs from non-concurrency tasks - unlimited_execs as ( - select - e.id, - e.task_key, - e.queue, - e.payload, - e.waiting_on_execution_id, - e.waiting_step_key, - e.cancelled, - e.last_error, - e.dedupe_key, - e.cron_expression, - e.priority, - e.run_at - from pgconductor._private_executions e - where e.is_available = true - and e.run_at <= pgconductor._private_current_time() - and e.queue = ${queueName}::text - and not (e.task_key = any(${this.sql.array(taskKeysWithConcurrency)}::text[])) - ${filterTaskKeys.length > 0 ? this.sql`and not (e.task_key = any(${this.sql.array(filterTaskKeys)}::text[]))` : this.sql``} - order by e.priority asc, e.run_at asc, e.id asc - limit ${batchSize}::integer - for update skip locked - ), - - -- row number concurrency executions by task - concurrency_execs_rn as ( - select - ce.*, - row_number() over (partition by ce.task_key order by ce.priority asc, ce.run_at asc, ce.id asc) as exec_rn - from concurrency_execs ce - ), - - -- row number slots by task - slots_rn as ( - select - ls.*, - row_number() over (partition by ls.task_key order by ls.slot_group_number) as slot_rn - from locked_slots_raw ls - ), - - -- pair concurrency executions with slots - concurrency_paired as ( - select - e.id, - e.exec_task_key as task_key, - e.queue, - e.payload, - e.waiting_on_execution_id, - e.waiting_step_key, - e.cancelled, - e.last_error, - e.dedupe_key, - e.cron_expression, - s.slot_group_number - from concurrency_execs_rn e - join slots_rn s - on e.task_key = s.task_key - and e.exec_rn = s.slot_rn - ), - - -- unlimited executions don't need slots - unlimited_paired as ( - select - id, - task_key, - queue, - payload, - waiting_on_execution_id, - waiting_step_key, - cancelled, - last_error, - dedupe_key, - cron_expression, - null::integer as slot_group_number - from unlimited_execs - ), - - -- merge all paired executions - paired as ( - select * from concurrency_paired - union all - select * from unlimited_paired - ), - - -- mark slots as used - mark_used as ( - update pgconductor._private_concurrency_slots cs - set used = 1 - from paired p - where cs.task_key = p.task_key - and cs.slot_group_number = p.slot_group_number - and p.slot_group_number is not null - ) - - -- update and return executions - update pgconductor._private_executions e - set - attempts = e.attempts + 1, - locked_by = ${orchestratorId}::uuid, - locked_at = pgconductor._private_current_time() - from paired p - where e.id = p.id - returning - e.id, - e.task_key, - e.queue, - e.payload, - e.waiting_on_execution_id, - e.waiting_step_key, - e.cancelled, - e.last_error, - e.dedupe_key, - e.cron_expression, - p.slot_group_number + from locked_candidates c + where e.id = c.id and e.queue = ${queueName}::text and e.is_available = true + returning e.id, e.task_key, e.queue, e.payload, e.waiting_on_execution_id, + e.waiting_step_key, e.cancelled, e.last_error, e.dedupe_key, e.cron_expression, + e.locked_by, e."group", e.priority, e.run_at, e.created_at, + e.dead_letter_source_execution_id, e.dead_letter_source_queue, + e.dead_letter_source_task_key, e.dead_letter_error, + e.dead_letter_attempts, e.dead_letter_failed_at + ) + select id, task_key, queue, payload, waiting_on_execution_id, waiting_step_key, + cancelled, last_error, dedupe_key, cron_expression, locked_by, "group", + dead_letter_source_execution_id, dead_letter_source_queue, + dead_letter_source_task_key, dead_letter_error, dead_letter_attempts, dead_letter_failed_at + from claimed + order by priority asc, run_at asc, created_at asc, id asc `; } buildReturnExecutions(grouped: GroupedExecutionResults): PendingQuery | null { - const completed = grouped.completed; - const failed = grouped.failed; - const released = grouped.released; - const invokeChild = grouped.invokeChild; + const allResults = [ + ...grouped.completed, + ...grouped.failed, + ...grouped.released, + ...grouped.invokeChild, + ]; - if (grouped.count === 0) { - return null; - } + if (allResults.length === 0) return null; const ctes: PendingQuery[] = []; - - // Precompute timestamp once ctes.push(this.sql`now_ts as (select pgconductor._private_current_time() as ts)`); - - // Load task configs once for all task_keys we're processing + ctes.push(this.sql`result_data as ( + select * from jsonb_to_recordset(${this.sql.json(JSON.parse(JSON.stringify(allResults)))}::jsonb) + as r( + execution_id uuid, queue text, task_key text, status text, + orchestrator_id uuid, result jsonb, error text, + reschedule_in_ms text, step_key text, timeout_ms text, + child_task_name text, child_task_queue text, child_payload jsonb, + "group" text + ) + )`); + // Lock the claimed rows for the whole statement. This prevents recovery or a + // new claim from racing the side effects below. + ctes.push(this.sql`valid_results as materialized ( + select r.*, + e.cancelled as execution_cancelled, e.last_error as execution_last_error + from result_data r + join pgconductor._private_executions e + on e.id = r.execution_id + and e.queue = r.queue + and e.task_key = r.task_key + and e.locked_by = r.orchestrator_id + for update of e + )`); ctes.push(this.sql`task_configs as ( - select key, max_attempts, remove_on_complete_days, remove_on_fail_days + select queue, key, max_attempts, remove_on_complete_days, remove_on_fail_days, + dead_letter_queue, dead_letter_task_key from pgconductor._private_tasks - where key = any(${this.sql.array(Array.from(grouped.taskKeys))}::text[]) + where queue = any(${this.sql.array(Array.from(new Set(allResults.map((r) => r.queue))))}::text[]) + )`); + ctes.push(this.sql`completed_results as ( + select * from valid_results where status = 'completed' and not execution_cancelled + )`); + ctes.push(this.sql`failed_results as ( + select * from valid_results + where status in ('failed', 'permanently_failed') + or (status = 'completed' and execution_cancelled) + )`); + ctes.push(this.sql`released_results as ( + select * from valid_results where status = 'released' + )`); + ctes.push(this.sql`invoke_child_data as ( + select * from valid_results where status = 'invoke_child' )`); - // Release concurrency slots for all results with slot_group_number - const allResults = [ - ...completed, - ...failed, - ...released, - ...invokeChild, - // ...waitForCustomEvent, - // ...waitForDbEvent, - ]; - const slotsToRelease = allResults - .filter((r) => r.slot_group_number != null) - .map((r) => ({ - task_key: r.task_key, - slot_group_number: r.slot_group_number, - })); - - if (slotsToRelease.length > 0) { - ctes.push(this.sql`released_slots as ( - update pgconductor._private_concurrency_slots cs - set used = 0 - from jsonb_to_recordset(${this.sql.json(slotsToRelease)}::jsonb) - as r(task_key text, slot_group_number integer) - where cs.task_key = r.task_key - and cs.slot_group_number = r.slot_group_number - )`); - } - - // Completed results - if (completed.length > 0) { - const completedData = completed.map((r) => ({ - execution_id: r.execution_id, - task_key: r.task_key, - result: r.result || null, - })); - - ctes.push(this.sql`completed_results as ( - select * from jsonb_to_recordset(${this.sql.json(completedData)}::jsonb) - as x(execution_id uuid, task_key text, result jsonb) - )`); - - // Insert parent steps for all completed - ctes.push(this.sql`parent_steps_all as ( - insert into pgconductor._private_steps (execution_id, queue, key, result) - select - parent_e.id, - parent_e.queue, - parent_e.waiting_step_key, - r.result - from completed_results r - join pgconductor._private_executions parent_e on parent_e.waiting_on_execution_id = r.execution_id - on conflict (execution_id, key) do nothing - returning execution_id - )`); + // A completed child may wake only a parent which is still waiting and is not + // currently claimed. The row lock makes this check race-safe. + ctes.push(this.sql`completed_parents as materialized ( + select parent.id as parent_id, parent.queue, parent.waiting_step_key, r.result + from completed_results r + join pgconductor._private_executions parent + on parent.waiting_on_execution_id = r.execution_id + where parent.completed_at is null + and parent.failed_at is null + and parent.locked_by is null + for update of parent + )`); + ctes.push(this.sql`parent_steps_all as ( + insert into pgconductor._private_steps (execution_id, queue, key, result) + select parent_id, queue, waiting_step_key, result + from completed_parents + where waiting_step_key is not null + on conflict (execution_id, key) do nothing + returning execution_id + )`); + ctes.push(this.sql`orphaned_children as ( + update pgconductor._private_executions e + set failed_at = nt.ts, completed_at = null, + last_error = 'Parent timed out before child completed', + locked_by = null, locked_at = null + from now_ts nt, completed_results r + where e.id = r.execution_id and e.queue = r.queue + and e.locked_by = r.orchestrator_id + and e.parent_execution_id is not null + and not exists ( + select 1 from pgconductor._private_executions parent + where parent.waiting_on_execution_id = r.execution_id + ) + returning e.id + )`); + ctes.push(this.sql`updated_parents_all as ( + update pgconductor._private_executions e + set run_at = nt.ts, waiting_on_execution_id = null, waiting_step_key = null, + locked_by = null, locked_at = null + from now_ts nt, completed_parents p + where e.id = p.parent_id and e.queue = p.queue + returning e.id + )`); + ctes.push(this.sql`deleted_completed as ( + delete from pgconductor._private_executions e + using completed_results r, task_configs tc + where e.id = r.execution_id and e.queue = r.queue + and e.locked_by = r.orchestrator_id + and tc.key = r.task_key and tc.queue = r.queue and tc.remove_on_complete_days = 0 + and not exists (select 1 from orphaned_children oc where oc.id = e.id) + returning e.id + )`); + ctes.push(this.sql`updated_completed as ( + update pgconductor._private_executions e + set completed_at = nt.ts, locked_by = null, locked_at = null + from now_ts nt, completed_results r, task_configs tc + where e.id = r.execution_id and e.queue = r.queue + and e.locked_by = r.orchestrator_id + and tc.key = r.task_key and tc.queue = r.queue + and (tc.remove_on_complete_days is null or tc.remove_on_complete_days != 0) + and not exists (select 1 from orphaned_children oc where oc.id = e.id) + returning e.id + )`); - // Mark orphaned children (completed but no parent waiting) as failed - ctes.push(this.sql`orphaned_children as ( - update pgconductor._private_executions e - set - failed_at = nt.ts, - completed_at = null, - last_error = 'Parent timed out before child completed', - locked_by = null, - locked_at = null - from now_ts nt, completed_results r - where e.id = r.execution_id - -- Only apply to child executions (has a parent) - and e.parent_execution_id is not null - -- No parent is waiting for this child - and not exists ( - select 1 from pgconductor._private_executions parent - where parent.waiting_on_execution_id = r.execution_id + ctes.push(this.sql`permanently_failed_children as materialized ( + select r.execution_id, r.queue, r.task_key, r.orchestrator_id, + r.execution_cancelled, + coalesce(r.error, r.execution_last_error, 'unknown error') as child_error, + e."group" as execution_group, + e.attempts as execution_attempts, + tc.remove_on_fail_days = 0 as should_remove + from failed_results r + join pgconductor._private_executions e on e.id = r.execution_id and e.queue = r.queue + join task_configs tc on tc.key = r.task_key and tc.queue = r.queue + where e.attempts >= tc.max_attempts + or r.status = 'permanently_failed' + or r.execution_cancelled + )`); + ctes.push(this.sql`failed_parent_targets as materialized ( + select p.execution_id as child_id, p.queue as child_queue, p.child_error, + p.execution_cancelled as child_cancelled, + parent.id as parent_id, parent.queue as parent_queue, + parent.task_key as parent_task_key, parent."group" as parent_group, + parent.payload as parent_payload, parent.attempts as parent_attempts, + pt.remove_on_fail_days = 0 as parent_should_remove + from permanently_failed_children p + join pgconductor._private_executions parent + on parent.waiting_on_execution_id = p.execution_id + join pgconductor._private_tasks pt + on pt.key = parent.task_key and pt.queue = parent.queue + where parent.completed_at is null and parent.failed_at is null and parent.locked_by is null + for update of parent + )`); + ctes.push(this.sql`terminal_failures as materialized ( + select p.execution_id, p.queue, p.task_key, p.execution_group, e.payload, + p.child_error as failure_error, p.execution_attempts as failure_attempts, + p.execution_cancelled, tc.dead_letter_queue, tc.dead_letter_task_key + from permanently_failed_children p + join pgconductor._private_executions e + on e.id = p.execution_id and e.queue = p.queue + join task_configs tc on tc.key = p.task_key and tc.queue = p.queue + union all + select p.parent_id, p.parent_queue, p.parent_task_key, p.parent_group, p.parent_payload, + 'Child execution failed: ' || p.child_error, p.parent_attempts, p.child_cancelled, + pt.dead_letter_queue, pt.dead_letter_task_key + from failed_parent_targets p + join pgconductor._private_tasks pt + on pt.key = p.parent_task_key and pt.queue = p.parent_queue + )`); + ctes.push(this.sql`dead_lettered as materialized ( + insert into pgconductor._private_executions ( + task_key, queue, payload, run_at, "group", + dead_letter_source_execution_id, dead_letter_source_queue, + dead_letter_source_task_key, dead_letter_error, + dead_letter_attempts, dead_letter_failed_at + ) + select + coalesce(p.dead_letter_task_key, p.task_key), + coalesce(p.dead_letter_queue, p.queue), + p.payload, nt.ts, p.execution_group, + p.execution_id, p.queue, p.task_key, p.failure_error, + p.failure_attempts, nt.ts + from terminal_failures p + cross join now_ts nt + where p.dead_letter_queue is not null + and not p.execution_cancelled + on conflict (dead_letter_source_execution_id, queue, task_key) + where dead_letter_source_execution_id is not null + do update set dead_letter_source_execution_id = excluded.dead_letter_source_execution_id + returning id, dead_letter_source_execution_id + )`); + ctes.push(this.sql`failed_updates as ( + select p.execution_id as target_id, p.queue, p.child_error, true as is_child + from permanently_failed_children p + where p.should_remove is not true + union all + select p.parent_id, p.parent_queue, p.child_error, false + from failed_parent_targets p + where p.parent_should_remove is not true + )`); + ctes.push(this.sql`deleted_failed as ( + delete from pgconductor._private_executions e + where exists ( + select 1 from permanently_failed_children p + where e.id = p.execution_id and e.queue = p.queue + and e.locked_by = p.orchestrator_id + and p.should_remove is true + and ( + not exists (select 1 from task_configs tc where tc.key = p.task_key and tc.queue = p.queue and tc.dead_letter_queue is not null) + or exists (select 1 from dead_lettered d where d.dead_letter_source_execution_id = p.execution_id) ) - returning e.id - )`); - - // Update parents for all completed - ctes.push(this.sql`updated_parents_all as ( - update pgconductor._private_executions e - set - run_at = nt.ts, - waiting_on_execution_id = null, - waiting_step_key = null, - locked_by = null, - locked_at = null - from now_ts nt, completed_results r - where e.waiting_on_execution_id = r.execution_id - )`); - - // Delete completed where remove_on_complete_days = 0 (excluding orphaned) - ctes.push(this.sql`deleted_completed as ( - delete from pgconductor._private_executions e - using completed_results r, task_configs tc - where e.id = r.execution_id - and tc.key = r.task_key - and tc.remove_on_complete_days = 0 - and not exists (select 1 from orphaned_children oc where oc.id = e.id) - )`); - - // Update completed where remove_on_complete_days != 0 (keep) - ctes.push(this.sql`updated_completed as ( - update pgconductor._private_executions e - set - completed_at = nt.ts, - locked_by = null, - locked_at = null - from now_ts nt, completed_results r, task_configs tc - where e.id = r.execution_id - and tc.key = r.task_key - and (tc.remove_on_complete_days is null or tc.remove_on_complete_days != 0) - and not exists (select 1 from orphaned_children oc where oc.id = e.id) - )`); - } - - // Failed results - if (failed.length > 0) { - ctes.push(this.sql`failed_results as ( - select * from jsonb_to_recordset(${this.sql.json(failed as unknown as JsonValue)}::jsonb) - as x(execution_id uuid, task_key text, status text, error text) - )`); - - // Permanently failed children (attempts >= max_attempts) - ctes.push(this.sql`permanently_failed_children as ( - select - r.execution_id, - r.task_key, - r.error as child_error, - tc.remove_on_fail_days = 0 as should_remove - from failed_results r, pgconductor._private_executions e, task_configs tc - where e.id = r.execution_id - and tc.key = r.task_key - and (e.attempts >= tc.max_attempts or r.status = 'permanently_failed') - )`); - - // Delete permanently failed children and their parents - ctes.push(this.sql`deleted_failed as ( - delete from pgconductor._private_executions e - using permanently_failed_children p - where - (e.id = p.execution_id and p.should_remove) - or ( - e.waiting_on_execution_id = p.execution_id - and exists ( - select 1 from pgconductor._private_tasks t - where t.key = e.task_key and t.remove_on_fail_days = 0 - ) + ) + or exists ( + select 1 from failed_parent_targets p + where e.id = p.parent_id and e.queue = p.parent_queue + and p.parent_should_remove is true + and ( + not exists (select 1 from pgconductor._private_tasks pt where pt.key = p.parent_task_key and pt.queue = p.parent_queue and pt.dead_letter_queue is not null) + or exists (select 1 from dead_lettered d where d.dead_letter_source_execution_id = p.parent_id) ) - )`); - - // Failed updates (permanently failed children + parents) - ctes.push(this.sql`failed_updates as ( - select - e.id as target_id, - p.child_error as error, - true as is_child - from permanently_failed_children p - join pgconductor._private_executions e on e.id = p.execution_id - where p.should_remove = false - union all - select - e.id as target_id, - p.child_error as error, - false as is_child - from permanently_failed_children p - join pgconductor._private_executions e on e.waiting_on_execution_id = p.execution_id - where not exists ( - select 1 from pgconductor._private_tasks t - where t.key = e.task_key and t.remove_on_fail_days = 0 - ) - )`); - - // Update all permanently failed - ctes.push(this.sql`updated_failed_all as ( - update pgconductor._private_executions e - set - failed_at = nt.ts, - last_error = case - when f.is_child then coalesce(f.error, 'unknown error') - else 'Child execution failed: ' || coalesce(f.error, 'unknown error') - end, - waiting_on_execution_id = null, - waiting_step_key = null, - locked_by = null, - locked_at = null - from now_ts nt, failed_updates f - where e.id = f.target_id - )`); - - // Retry failed (not permanently failed) - ctes.push(this.sql`retried as ( - update pgconductor._private_executions e - set - last_error = coalesce(r.error, 'unknown error'), - run_at = greatest(nt.ts, coalesce(e.run_at, nt.ts)) - + ((array[15, 30, 60, 120, 300, 600, 1200, 2400, 3600, 7200])[least(greatest(e.attempts, 1), 10)] * interval '1 second'), - locked_by = null, - locked_at = null - from now_ts nt, failed_results r, task_configs tc - where e.id = r.execution_id - and tc.key = r.task_key - and e.attempts < tc.max_attempts - )`); - } - - // Released - if (released.length > 0) { - // Save steps for released executions with step_key (e.g., sleep) - const releasedWithSteps = released.filter((r) => r.step_key !== undefined); - if (releasedWithSteps.length > 0) { - ctes.push(this.sql`released_steps as ( - insert into pgconductor._private_steps (execution_id, queue, key, result) - select - r.execution_id, - r.queue, - r.step_key, - null::jsonb - from jsonb_to_recordset(${this.sql.json(releasedWithSteps as unknown as JsonValue)}::jsonb) - as r(execution_id uuid, queue text, step_key text) - on conflict (execution_id, key) do nothing - returning id - )`); - } - - const shouldRescheduleSome = released.some((r) => r.reschedule_in_ms !== undefined); - - if (shouldRescheduleSome) { - const releasedData = released.map((r) => ({ - execution_id: r.execution_id, - reschedule_in_ms: r.reschedule_in_ms === "infinity" ? -1 : r.reschedule_in_ms, - })); - - ctes.push(this.sql`updated_released as ( - update pgconductor._private_executions e - set - attempts = greatest(attempts - 1, 0), - run_at = case - when r.reschedule_in_ms = -1 then - 'infinity'::timestamptz - when r.reschedule_in_ms is not null then - nt.ts + (r.reschedule_in_ms::integer || ' milliseconds')::interval - else - nt.ts - end, - locked_by = null, - locked_at = null - from now_ts nt, jsonb_to_recordset(${this.sql.json(releasedData)}::jsonb) - as r(execution_id uuid, reschedule_in_ms integer) - where e.id = r.execution_id - )`); - } else { - const releasedIds = released.map((r) => r.execution_id); - - ctes.push(this.sql`updated_released as ( - update pgconductor._private_executions - set - attempts = greatest(attempts - 1, 0), - locked_by = null, - locked_at = null - where id = any(${this.sql.array(releasedIds)}::uuid[]) - )`); - } - } - - if (invokeChild.length > 0) { - ctes.push(this.sql`invoke_child_data as ( - select * from jsonb_to_recordset(${this.sql.json(invokeChild as unknown as JsonValue)}::jsonb) - as x( - execution_id uuid, - task_key text, - queue text, - step_key text, - timeout_ms text, - child_task_name text, - child_task_queue text, - child_payload jsonb - ) - )`); - - // Insert child executions with parent reference - ctes.push(this.sql`inserted_children as ( - insert into pgconductor._private_executions ( - id, - task_key, - queue, - payload, - run_at, - parent_execution_id - ) - select - pgconductor._private_portable_uuidv7(), - icd.child_task_name, - icd.child_task_queue, - icd.child_payload, - nt.ts, - icd.execution_id - from invoke_child_data icd, now_ts nt - returning id, parent_execution_id - )`); + ) + returning e.id + )`); + ctes.push(this.sql`updated_failed as ( + update pgconductor._private_executions e + set failed_at = nt.ts, + last_error = case when f.is_child then coalesce(f.child_error, 'unknown error') + else 'Child execution failed: ' || coalesce(f.child_error, 'unknown error') end, + waiting_on_execution_id = null, waiting_step_key = null, + locked_by = null, locked_at = null + from now_ts nt, failed_updates f + where e.id = f.target_id and e.queue = f.queue + returning e.id + )`); + ctes.push(this.sql`retried as ( + update pgconductor._private_executions e + set last_error = coalesce(r.error, 'unknown error'), + run_at = greatest(nt.ts, coalesce(e.run_at, nt.ts)) + + ((array[15, 30, 60, 120, 300, 600, 1200, 2400, 3600, 7200])[least(greatest(e.attempts, 1), 10)] * interval '1 second'), + locked_by = null, locked_at = null + from now_ts nt, failed_results r, task_configs tc + where e.id = r.execution_id and e.queue = r.queue + and e.locked_by = r.orchestrator_id + and tc.key = r.task_key and tc.queue = r.queue + and r.status <> 'permanently_failed' + and not r.execution_cancelled + and e.attempts < tc.max_attempts + returning e.id + )`); - // Update parent executions - ctes.push(this.sql`updated_invoke_parents as ( - update pgconductor._private_executions e - set - waiting_on_execution_id = ic.id, - waiting_step_key = icd.step_key, - run_at = case - when icd.timeout_ms = 'infinity' then 'infinity'::timestamptz - else nt.ts + (icd.timeout_ms::bigint || ' milliseconds')::interval - end, - locked_by = null, - locked_at = null - from now_ts nt, inserted_children ic - join invoke_child_data icd on icd.execution_id = ic.parent_execution_id - where e.id = ic.parent_execution_id - )`); - } + ctes.push(this.sql`released_steps as ( + insert into pgconductor._private_steps (execution_id, queue, key, result) + select execution_id, queue, step_key, null::jsonb from released_results + where step_key is not null + on conflict (execution_id, key) do nothing + returning id + )`); + ctes.push(this.sql`updated_released as ( + update pgconductor._private_executions e + set attempts = greatest(e.attempts - 1, 0), + run_at = case when lower(nullif(trim(r.reschedule_in_ms), '')) = 'infinity' then 'infinity'::timestamptz + when nullif(trim(r.reschedule_in_ms), '') is not null then + nt.ts + (nullif(trim(r.reschedule_in_ms), '')::bigint || ' milliseconds')::interval + else nt.ts end, + locked_by = null, locked_at = null + from now_ts nt, released_results r + where e.id = r.execution_id and e.queue = r.queue + and e.locked_by = r.orchestrator_id + returning e.id + )`); - if (ctes.length <= 1) return null; // Only now_ts + ctes.push(this.sql`inserted_children as ( + insert into pgconductor._private_executions (id, task_key, queue, payload, run_at, parent_execution_id, "group") + select pgconductor._private_portable_uuidv7(), r.child_task_name, r.child_task_queue, r.child_payload, nt.ts, r.execution_id, r."group" + from invoke_child_data r, now_ts nt + where exists ( + select 1 from pgconductor._private_executions parent + where parent.id = r.execution_id and parent.queue = r.queue + and parent.locked_by = r.orchestrator_id + ) + returning id, parent_execution_id + )`); + ctes.push(this.sql`updated_invoke_parents as ( + update pgconductor._private_executions e + set waiting_on_execution_id = ic.id, waiting_step_key = r.step_key, + run_at = case when lower(nullif(trim(r.timeout_ms), '')) = 'infinity' then 'infinity'::timestamptz + when nullif(trim(r.timeout_ms), '') is not null then + nt.ts + (nullif(trim(r.timeout_ms), '')::bigint || ' milliseconds')::interval + else nt.ts end, + locked_by = null, locked_at = null + from now_ts nt, inserted_children ic + join invoke_child_data r on r.execution_id = ic.parent_execution_id + where e.id = r.execution_id and e.queue = r.queue + and e.locked_by = r.orchestrator_id + returning e.id + )`); const combined = ctes.reduce((acc, cte, i) => (i === 0 ? cte : this.sql`${acc}, ${cte}`)); - return this.sql<[{ result: number }]>`with ${combined} select 1 as result`; } - buildRemoveExecutions({ queueName, batchSize, @@ -819,7 +675,7 @@ export class QueryBuilder { with batch as ( select e.id from pgconductor._private_executions e - join pgconductor._private_tasks t on t.key = e.task_key + join pgconductor._private_tasks t on t.key = e.task_key and t.queue = e.queue where e.queue = ${queueName} and ( (e.completed_at is not null and t.remove_on_complete_days > 0 and e.completed_at < pgconductor._private_current_time() - t.remove_on_complete_days * interval '1 day') @@ -838,6 +694,17 @@ export class QueryBuilder { `; } + buildRemoveProcessedEvents({ + before, + batchSize, + }: RemoveProcessedEventsArgs): PendingQuery<[{ deleted_count: number }]> { + return this.sql<[{ deleted_count: number }]>` + select pgconductor._private_remove_processed_events( + ${before.toISOString()}::timestamptz, ${batchSize}::integer + ) as deleted_count + `; + } + buildRegisterWorker({ queueName, taskSpecs, @@ -853,6 +720,9 @@ export class QueryBuilder { window_start: spec.window?.[0] || null, window_end: spec.window?.[1] || null, concurrency_limit: spec.concurrency || null, + group_concurrency_limit: spec.groupConcurrency || null, + dead_letter_queue: spec.deadLetterQueue || null, + dead_letter_task_key: spec.deadLetterTaskKey || null, })); const cronScheduleRows = cronSchedules.map((spec) => { @@ -866,6 +736,7 @@ export class QueryBuilder { dedupe_key: spec.dedupe_key, cron_expression: spec.cron_expression, priority: spec.priority || null, + group: spec.group || null, }; }); @@ -879,6 +750,7 @@ export class QueryBuilder { when_clause: spec.when_clause, payload_fields: spec.payload_fields, column_names: spec.column_names, + filter: spec.filter, })); return this.sql` @@ -923,7 +795,8 @@ export class QueryBuilder { p_dedupe_seconds := ${dedupe_seconds}::integer, p_dedupe_next_slot := ${dedupe_next_slot}::boolean, p_cron_expression := ${spec.cron_expression || null}::text, - p_priority := ${spec.priority || null}::integer + p_priority := ${spec.priority || null}::integer, + p_group := ${spec.group || null}::text ) `; } @@ -959,7 +832,8 @@ export class QueryBuilder { p_dedupe_seconds := null::integer, p_dedupe_next_slot := false::boolean, p_cron_expression := ${cronExpression}::text, - p_priority := ${spec.priority || 0}::integer + p_priority := ${spec.priority || 0}::integer, + p_group := ${spec.group || null}::text ) `; } @@ -1029,6 +903,7 @@ export class QueryBuilder { dedupe_next_slot, cron_expression: spec.cron_expression || null, priority: spec.priority, + group: spec.group || null, }; }); @@ -1041,10 +916,22 @@ export class QueryBuilder { `; } - buildLoadStep({ executionId, key }: LoadStepArgs): PendingQuery<[{ result: Payload | null }]> { + buildLoadStep({ + executionId, + queue, + orchestratorId, + key, + }: LoadStepArgs): PendingQuery<[{ result: Payload | null }]> { return this.sql<[{ result: Payload | null }]>` select result from pgconductor._private_steps - where execution_id = ${executionId}::uuid and key = ${key}::text + where execution_id = ${executionId}::uuid + and queue = ${queue}::text + and exists ( + select 1 from pgconductor._private_executions e + where e.id = ${executionId}::uuid and e.queue = ${queue}::text + and e.locked_by = ${orchestratorId}::uuid + ) + and key = ${key}::text `; } @@ -1054,39 +941,118 @@ export class QueryBuilder { key, result, runAtMs, + orchestratorId, }: SaveStepArgs): PendingQuery> { if (runAtMs) { - return this.sql` - with inserted as ( + return this.sql>` + with claimed_execution as materialized ( + select e.id, e.queue + from pgconductor._private_executions e + where e.id = ${executionId}::uuid + and e.queue = ${queue}::text + and e.locked_by = ${orchestratorId}::uuid + for update + ), inserted as ( insert into pgconductor._private_steps (execution_id, queue, key, result) - values (${executionId}::uuid, ${queue}::text, ${key}::text, ${this.sql.json(result)}::jsonb) + select e.id, e.queue, ${key}::text, ${this.sql.json(result)}::jsonb + from claimed_execution e on conflict (execution_id, key) do nothing returning id ) - update pgconductor._private_executions + update pgconductor._private_executions e set run_at = pgconductor._private_current_time() + (${runAtMs}::integer || ' milliseconds')::interval - where id = ${executionId}::uuid - and queue = ${queue}::text + from claimed_execution c + where e.id = c.id and e.queue = c.queue and exists (select 1 from inserted) `; } - return this.sql` + return this.sql>` + with claimed_execution as materialized ( + select e.id, e.queue + from pgconductor._private_executions e + where e.id = ${executionId}::uuid + and e.queue = ${queue}::text + and e.locked_by = ${orchestratorId}::uuid + for update + ) insert into pgconductor._private_steps (execution_id, queue, key, result) - values (${executionId}::uuid, ${queue}::text, ${key}::text, ${this.sql.json(result)}::jsonb) + select e.id, e.queue, ${key}::text, ${this.sql.json(result)}::jsonb + from claimed_execution e on conflict (execution_id, key) do nothing `; } - buildClearWaitingState({ executionId }: ClearWaitingStateArgs): PendingQuery> { - return this.sql` - with child_info as ( - select - e.waiting_on_execution_id as child_id, - c.locked_by as child_locked_by + buildRegisterEventWait({ + executionId, + queue, + taskKey, + eventKey, + stepKey, + filter, + timeoutMs, + orchestratorId, + }: RegisterEventWaitArgs): PendingQuery> { + return this.sql>` + with event_wait_lock as materialized ( + select pg_advisory_xact_lock(hashtext('pgconductor:event-waits')) as locked + ), claimed as materialized ( + select e.id, e.queue + from pgconductor._private_executions e + cross join event_wait_lock + where e.id = ${executionId}::uuid and e.queue = ${queue}::text + and e.task_key = ${taskKey}::text + and e.locked_by = ${orchestratorId}::uuid + and e.completed_at is null and e.failed_at is null and e.cancelled = false + for update + ), event_position_state as materialized ( + select case when sequence_state.is_called then sequence_state.last_value + else sequence_state.last_value - 1 end as position + from pgconductor._private_event_position_seq sequence_state + cross join event_wait_lock + ), inserted as ( + insert into pgconductor._private_event_subscriptions + (task_key, queue, event_key, filter, kind, execution_id, step_key, expires_at, + wait_after_event_position) + select ${taskKey}::text, ${queue}::text, ${eventKey}::text, + ${filter ? this.sql.json(filter) : null}::jsonb, 'execution_wait', + ${executionId}::uuid, ${stepKey}::text, + case when ${timeoutMs}::bigint is null then null + else pgconductor._private_current_time() + (${timeoutMs}::bigint || ' milliseconds')::interval end, + event_position_state.position + from claimed cross join event_position_state + on conflict (execution_id, step_key) where execution_id is not null do nothing + returning id + ) + update pgconductor._private_executions e + set waiting_on_execution_id = null, waiting_step_key = ${stepKey}::text, + run_at = 'infinity'::timestamptz, + locked_by = null, locked_at = null + from claimed c + where e.id = c.id and e.queue = c.queue and exists (select 1 from inserted) + returning exists (select 1 from inserted) as registered + `; + } + + buildClearWaitingState({ + executionId, + queue, + orchestratorId, + }: ClearWaitingStateArgs): PendingQuery> { + return this.sql>` + with claimed_parent as materialized ( + select e.id, e.queue, e.waiting_on_execution_id from pgconductor._private_executions e - left join pgconductor._private_executions c on c.id = e.waiting_on_execution_id where e.id = ${executionId}::uuid + and e.queue = ${queue}::text + and e.locked_by = ${orchestratorId}::uuid + for update + ), child_info as ( + select + p.waiting_on_execution_id as child_id, + c.locked_by as child_locked_by + from claimed_parent p + left join pgconductor._private_executions c on c.id = p.waiting_on_execution_id ), -- Fail pending (not locked) children immediately failed_pending_child as ( @@ -1101,6 +1067,7 @@ export class QueryBuilder { and ci.child_locked_by is null -- not currently executing and e.completed_at is null and e.failed_at is null + returning e.id ), -- Signal executing (locked) children to cancel signaled_executing_child as ( @@ -1111,17 +1078,24 @@ export class QueryBuilder { and ci.child_locked_by is not null -- currently executing and e.completed_at is null and e.failed_at is null + returning e.id ), -- Always clear parent's waiting state cleared_parent as ( - update pgconductor._private_executions + update pgconductor._private_executions e set waiting_on_execution_id = null, waiting_step_key = null - where id = ${executionId}::uuid - returning id + from claimed_parent p + where e.id = p.id + and e.queue = p.queue + returning e.id ) select id from cleared_parent + union all + select id from failed_pending_child + union all + select id from signaled_executing_child `; } diff --git a/packages/pgconductor-js/src/schemas.ts b/packages/pgconductor-js/src/schemas.ts index 749a4a7..2dcd2a0 100644 --- a/packages/pgconductor-js/src/schemas.ts +++ b/packages/pgconductor-js/src/schemas.ts @@ -65,15 +65,15 @@ export class TaskSchemas< * .fromUnion() */ export class EventSchemas< - TSchemaTypes extends readonly EventDefinition[] = readonly [], - TUnionTypes extends EventDefinition = never, + TSchemaTypes extends readonly EventDefinition[] = readonly [], + TUnionTypes extends EventDefinition = never, > { private constructor(readonly definitions: TSchemaTypes) {} /** * Create EventSchemas from standard-schema based event definitions. */ - static fromSchema[]>( + static fromSchema[]>( events: T, ): EventSchemas { return new EventSchemas(events); @@ -82,7 +82,7 @@ export class EventSchemas< /** * Add type-only event definitions via union type. */ - static fromUnion>(): EventSchemas< + static fromUnion>(): EventSchemas< readonly [], TUnion > { @@ -92,7 +92,7 @@ export class EventSchemas< /** * Chain: Add more standard-schema based event definitions. */ - fromSchema[]>( + fromSchema[]>( events: T, ): EventSchemas { return new EventSchemas([...this.definitions, ...events]); @@ -101,7 +101,7 @@ export class EventSchemas< /** * Chain: Add type-only event definitions via union type. */ - fromUnion>(): EventSchemas< + fromUnion>(): EventSchemas< TSchemaTypes, TUnionTypes | TUnion > { diff --git a/packages/pgconductor-js/src/task-context.ts b/packages/pgconductor-js/src/task-context.ts index 509febe..035557e 100644 --- a/packages/pgconductor-js/src/task-context.ts +++ b/packages/pgconductor-js/src/task-context.ts @@ -1,5 +1,11 @@ import CronExpressionParser from "cron-parser"; -import type { DatabaseClient, JsonValue, Execution, Payload } from "./database-client"; +import type { + DatabaseClient, + JsonValue, + Execution, + Payload, + DeadLetterMetadata, +} from "./database-client"; import type { TaskDefinition, TaskName, @@ -11,11 +17,13 @@ import type { TaskIdentifier } from "./task"; import type { Logger } from "./lib/logger"; import { WindowChecker } from "./lib/window-checker"; import { TypedAbortController } from "./lib/typed-abort-controller"; +import { parseDuration, type DurationInput } from "./lib/duration"; import type { EventDefinition, EventName, FindEventByIdentifier, InferEventPayload, + FilterForEvent, } from "./event-definition"; export type TaskAbortReasons = @@ -30,6 +38,7 @@ export type TaskAbortReasons = } // the worker wants to shut down | { reason: "parent-aborted"; __pgconductorTaskAborted: true } + | { reason: "wait-for-event"; step_key: string; __pgconductorTaskAborted: true } // the task invoked a child | { reason: "child-invocation"; @@ -37,6 +46,7 @@ export type TaskAbortReasons = step_key: string; task: TaskIdentifier; payload: Payload | null; + group?: string | null; __pgconductorTaskAborted: true; }; @@ -73,6 +83,14 @@ export function createTaskSignal( return controller; } +export class WaitForEventTimeoutError extends Error { + readonly code = "PGCONDUCTOR_WAIT_FOR_EVENT_TIMEOUT"; + constructor(public readonly stepKey: string) { + super(`Timed out waiting for event at step "${stepKey}"`); + this.name = "WaitForEventTimeoutError"; + } +} + export type TaskContextOptions = { abortController: TypedAbortController; db: DatabaseClient; @@ -84,6 +102,7 @@ export type TaskContextOptions = { type ScheduleOptions = { cron: string; priority?: number; + group?: string; }; // second argument for tasks @@ -94,7 +113,11 @@ export class TaskContext< any, string >[], - Events extends readonly EventDefinition[] = readonly EventDefinition[], + Events extends readonly EventDefinition[] = readonly EventDefinition< + string, + any, + any + >[], > { private readonly windowChecker?: WindowChecker; @@ -106,7 +129,7 @@ export class TaskContext< static create< Tasks extends readonly TaskDefinition[], - Events extends readonly EventDefinition[], + Events extends readonly EventDefinition[], Extra extends object, >(opts: TaskContextOptions, extra?: Extra): TaskContext & Extra { const base = new TaskContext(opts); @@ -121,6 +144,28 @@ export class TaskContext< return this.opts.abortController.signal; } + /** Metadata describing the source execution when this is a DLQ delivery. */ + get deadLetter(): DeadLetterMetadata | null { + const execution = this.opts.execution; + if ( + !execution.dead_letter_source_execution_id || + !execution.dead_letter_source_queue || + !execution.dead_letter_source_task_key || + execution.dead_letter_attempts == null || + !execution.dead_letter_failed_at + ) { + return null; + } + return { + sourceExecutionId: execution.dead_letter_source_execution_id, + sourceQueue: execution.dead_letter_source_queue, + sourceTaskKey: execution.dead_letter_source_task_key, + error: execution.dead_letter_error ?? null, + attempts: execution.dead_letter_attempts, + failedAt: execution.dead_letter_failed_at, + }; + } + async step(name: string, fn: () => Promise | T): Promise { // Check abort signal if (this.signal.aborted) { @@ -147,6 +192,8 @@ export class TaskContext< const cached = await this.opts.db.loadStep( { executionId: this.opts.execution.id, + queue: this.opts.execution.queue, + orchestratorId: this.opts.execution.locked_by, key: name, }, { signal: this.signal }, @@ -163,6 +210,7 @@ export class TaskContext< { executionId: this.opts.execution.id, queue: this.opts.execution.queue, + orchestratorId: this.opts.execution.locked_by, key: name, result: { result: result as JsonValue }, runAtMs: undefined, @@ -201,6 +249,8 @@ export class TaskContext< const cached = await this.opts.db.loadStep( { executionId: this.opts.execution.id, + queue: this.opts.execution.queue, + orchestratorId: this.opts.execution.locked_by, key: id, }, { signal: this.signal }, @@ -217,6 +267,65 @@ export class TaskContext< }); } + async waitForEvent< + TName extends EventName, + TDef extends FindEventByIdentifier = FindEventByIdentifier, + >( + stepKey: string, + options: { + event: TDef; + filter?: FilterForEvent; + timeout?: DurationInput; + }, + ): Promise<{ name: TDef["name"]; payload: InferEventPayload }> { + const cached = await this.opts.db.loadStep( + { + executionId: this.opts.execution.id, + queue: this.opts.execution.queue, + orchestratorId: this.opts.execution.locked_by, + key: stepKey, + }, + { signal: this.signal }, + ); + if (cached !== undefined) { + if ((cached as Record).__pgconductor_wait_for_event_timeout === true) + throw new WaitForEventTimeoutError(stepKey); + return (cached as { result: { name: TDef["name"]; payload: InferEventPayload } }) + .result; + } + + if (!stepKey) throw new Error("waitForEvent stepKey is required"); + const timeoutMs = options.timeout === undefined ? null : parseDuration(options.timeout); + const allowed = new Set(options.event.filterable ?? []); + for (const [field, values] of Object.entries(options.filter ?? {})) { + if (!allowed.has(field)) + throw new Error( + `Filter for event "${options.event.name}" contains undeclared field "${field}"`, + ); + if (!Array.isArray(values) || values.length === 0) + throw new Error( + `Filter value for event "${options.event.name}" field "${field}" cannot be empty`, + ); + } + const registered = await this.opts.db.registerEventWait( + { + executionId: this.opts.execution.id, + queue: this.opts.execution.queue, + taskKey: this.opts.execution.task_key, + eventKey: options.event.name, + stepKey, + filter: (options.filter as Record | undefined) ?? null, + timeoutMs, + orchestratorId: this.opts.execution.locked_by, + }, + { signal: this.signal }, + ); + if (registered) return this.abortAndHangup({ reason: "wait-for-event", step_key: stepKey }); + // Lost ownership between loading the step and registration: release rather + // than hanging a task that can no longer be resumed by this claim. + return this.abortAndHangup({ reason: "released", reschedule_in_ms: 0, step_key: stepKey }); + } + async invoke< TName extends TaskName, TQueue extends string = "default", @@ -230,10 +339,13 @@ export class TaskContext< task: TaskIdentifier, payload: InferPayload = {} as InferPayload, timeout?: number, + group?: string, ): Promise> { const cached = await this.opts.db.loadStep( { executionId: this.opts.execution.id, + queue: this.opts.execution.queue, + orchestratorId: this.opts.execution.locked_by, key, }, { signal: this.signal }, @@ -251,6 +363,8 @@ export class TaskContext< await this.opts.db.clearWaitingState( { executionId: this.opts.execution.id, + queue: this.opts.execution.queue, + orchestratorId: this.opts.execution.locked_by, }, { signal: this.signal }, ); @@ -266,6 +380,7 @@ export class TaskContext< task, step_key: key, payload, + group, }); } @@ -308,6 +423,7 @@ export class TaskContext< run_at: nextTimestamp, cron_expression: options.cron, priority: options.priority || null, + group: options.group || null, }, scheduleName, }, diff --git a/packages/pgconductor-js/src/task-definition.ts b/packages/pgconductor-js/src/task-definition.ts index f847e1b..09171e1 100644 --- a/packages/pgconductor-js/src/task-definition.ts +++ b/packages/pgconductor-js/src/task-definition.ts @@ -1,4 +1,5 @@ import type { StandardSchemaV1 } from "@standard-schema/spec"; +import type { EventDefinition, FilterForEvent, FindEventByIdentifier } from "./event-definition"; type ObjectSchema = StandardSchemaV1; @@ -118,16 +119,17 @@ export type FindTaskByIdentifier< // Trigger types export type InvocableTrigger = { invocable: true }; -export type CronTrigger = { cron: string; name: string }; +export type CronTrigger = { cron: string; name: string; group?: string }; // Event trigger - triggers when a custom event is emitted export type CustomEventTrigger< TName extends string = string, TFields extends string | undefined = undefined, + TFilter extends Record | undefined = undefined, > = { event: TName; - when?: string; fields?: TFields; + filter?: TFilter; }; // Database event trigger - triggers on CDC events @@ -146,6 +148,32 @@ export type DatabaseEventTrigger< export type Trigger = InvocableTrigger | CronTrigger | CustomEventTrigger | DatabaseEventTrigger; +type ValidateCustomEventTrigger< + Events extends readonly EventDefinition[], + T, +> = T extends { event: infer Name extends string } + ? T extends { filter: infer Filter } + ? Events extends readonly [] + ? T + : FindEventByIdentifier extends infer Event + ? [Event] extends [never] + ? `Event "${Name}" is not defined in the conductor event catalog.` + : Filter extends FilterForEvent + ? Exclude> extends never + ? T + : `Filter for event "${Name}" contains undeclared fields.` + : `Filter for event "${Name}" contains undeclared or incorrectly typed fields.` + : T + : T + : T; + +export type ValidateEventTriggers< + Events extends readonly EventDefinition[], + TTriggers, +> = TTriggers extends readonly any[] + ? { [K in keyof TTriggers]: ValidateCustomEventTrigger } + : ValidateCustomEventTrigger; + // Check if triggers include invocable export type HasInvocable = TTriggers extends readonly any[] ? Extract extends never diff --git a/packages/pgconductor-js/src/task.ts b/packages/pgconductor-js/src/task.ts index 22973e5..a532895 100644 --- a/packages/pgconductor-js/src/task.ts +++ b/packages/pgconductor-js/src/task.ts @@ -29,20 +29,64 @@ export type BatchConfig = { timeoutMs: number; }; +export type DeadLetterConfiguration = { + readonly queue: string; + readonly task?: Task; +}; + export type TaskConfiguration< TName extends string = string, TQueue extends string = "default", + TPayload extends object = object, > = TaskIdentifier & { maxAttempts?: number; window?: [string, string]; removeOnComplete?: RetentionSettings; removeOnFail?: RetentionSettings; concurrency?: number; + groupConcurrency?: number; batch?: BatchConfig; + deadLetter?: DeadLetterConfiguration; }; +/** Type-level validation for a dead-letter destination. */ +type ExactType = [TLeft] extends [TRight] + ? [TRight] extends [TLeft] + ? true + : false + : false; + +type ValidateDeadLetterTarget = + TTarget extends Task + ? TPayload extends TTargetPayload + ? ExactType extends true + ? unknown + : "deadLetter.queue must match deadLetter.task.queue" + : "deadLetter.task must accept the source task payload" + : "deadLetter.task must be a Task"; + +export type ValidateDeadLetterConfiguration = T extends { + readonly deadLetter: infer TDeadLetter; +} + ? TDeadLetter extends { readonly queue: infer TQueue } + ? TQueue extends string + ? TDeadLetter extends { readonly task: infer TTarget } + ? ValidateDeadLetterTarget + : unknown + : "deadLetter.queue must be a string" + : "deadLetter must include a queue string" + : unknown; + export type RetentionSettings = boolean | { days: number }; +function validateConcurrency(value: number | undefined, name: string): number | undefined { + if (value === undefined) return undefined; + if (!Number.isInteger(value) || value <= 0) { + throw new Error(`${name} must be a positive integer`); + } + return value; +} + export type TaskEvent

= | { name: "pgconductor.cron" } | { name: "pgconductor.invoke"; payload: P }; @@ -89,11 +133,11 @@ type CronEventUnion = : never; // Build custom event union from triggers -type CustomEventUnion[]> = +type CustomEventUnion[]> = ExtractCustomEventTriggers extends infer T ? T extends { event: infer TName extends string } ? FindEventByIdentifier extends infer TEvent - ? TEvent extends EventDefinition + ? TEvent extends EventDefinition ? T extends { fields: infer TFields extends string } ? { name: TName; @@ -154,7 +198,7 @@ type DatabaseEventUnion = export type TaskEventFromTriggers< TTriggers, TPayload extends object, - Events extends readonly EventDefinition[] = [], + Events extends readonly EventDefinition[] = [], Database extends GenericDatabase = {}, > = | (HasInvocable extends true @@ -192,12 +236,14 @@ export class Task< public readonly removeOnComplete: RetentionSettings; public readonly removeOnFail: RetentionSettings; public readonly concurrency?: number; + public readonly groupConcurrency?: number; public readonly batch?: BatchConfig; + public readonly deadLetter?: DeadLetterConfiguration; public readonly triggers: NonEmptyArray; constructor( - definition: TaskConfiguration, + definition: TaskConfiguration, triggers: NonEmptyArray | Trigger, public readonly execute: ExecuteFunction, ) { @@ -209,8 +255,29 @@ export class Task< this.window = config.window; this.removeOnComplete = config.removeOnComplete ?? false; this.removeOnFail = config.removeOnFail ?? false; - this.concurrency = config.concurrency; + this.concurrency = validateConcurrency(config.concurrency, "concurrency"); + this.groupConcurrency = validateConcurrency(config.groupConcurrency, "groupConcurrency"); this.batch = config.batch; + this.deadLetter = config.deadLetter; + if (this.deadLetter) { + if (typeof this.deadLetter.queue !== "string") { + throw new Error("deadLetter.queue must be a string"); + } + if (this.deadLetter.task !== undefined) { + if (!(this.deadLetter.task instanceof Task)) { + throw new Error("deadLetter.task must be a Task"); + } + if (this.deadLetter.task.queue !== this.deadLetter.queue) { + throw new Error("deadLetter.queue must match deadLetter.task.queue"); + } + } + if ( + this.deadLetter.queue === this.queue && + (this.deadLetter.task?.name || this.name) === this.name + ) { + throw new Error("A task cannot dead-letter directly to itself"); + } + } this.triggers = Array.isArray(triggers) ? triggers : [triggers]; } @@ -223,7 +290,7 @@ export class Task< Context extends object, EventType, >( - definition: TaskConfiguration, + definition: TaskConfiguration, triggers: NonEmptyArray | Trigger, execute: ExecuteFunction, ): Task { diff --git a/packages/pgconductor-js/src/worker.ts b/packages/pgconductor-js/src/worker.ts index d18a18d..efd27e8 100644 --- a/packages/pgconductor-js/src/worker.ts +++ b/packages/pgconductor-js/src/worker.ts @@ -11,7 +11,7 @@ import type { ExecutionInvokeChild, } from "./database-client"; import type { AnyTask, BatchConfig } from "./task"; -import type { TaskDefinition } from "./task-definition"; +import type { TaskDefinition, CustomEventTrigger, DatabaseEventTrigger } from "./task-definition"; import { waitFor } from "./lib/wait-for"; import { mapConcurrent } from "./lib/map-concurrent"; import { Deferred } from "./lib/deferred"; @@ -46,7 +46,31 @@ export type WorkerConfig = { /** * The default configuration for the Worker. */ -export const DEFAULT_WORKER_CONFIG: WorkerConfig = { +export const EVENT_BATCH_SIZE = 100; + +const RETRYABLE_EVENT_ERROR_CODES = new Set([ + "40001", + "40P01", + "55P03", + "57P01", + "57P02", + "57P03", + "53300", + "08000", + "08003", + "08006", + "08001", + "ECONNRESET", + "ECONNREFUSED", + "ETIMEDOUT", +]); + +function isRetryableEventError(error: unknown): boolean { + const code = (error as { code?: string })?.code; + return code !== undefined && RETRYABLE_EVENT_ERROR_CODES.has(code); +} + +const DEFAULT_WORKER_CONFIG: WorkerConfig = { concurrency: 1, flushBatchSize: 2, fetchBatchSize: 2, @@ -58,6 +82,7 @@ export const DEFAULT_WORKER_CONFIG: WorkerConfig = { * Encapsulates buffered execution results with internal counting and task key tracking. */ class BufferState { + orchestratorId = ""; completed: ExecutionCompleted[] = []; failed: (ExecutionFailed | ExecutionPermamentlyFailed)[] = []; released: ExecutionReleased[] = []; @@ -66,6 +91,7 @@ class BufferState { count = 0; add(result: ExecutionResult): void { + this.orchestratorId = result.orchestrator_id || this.orchestratorId; this.taskKeys.add(result.task_key); this.count++; @@ -100,6 +126,7 @@ class BufferState { this.failed.push(...other.failed); this.released.push(...other.released); this.invokeChild.push(...other.invokeChild); + this.orchestratorId = this.orchestratorId || other.orchestratorId; this.count += other.count; for (const key of other.taskKeys) { this.taskKeys.add(key); @@ -120,7 +147,11 @@ export class Worker< any, string >[], - Events extends readonly EventDefinition[] = readonly EventDefinition[], + Events extends readonly EventDefinition[] = readonly EventDefinition< + string, + any, + any + >[], > { private orchestratorId: string | null = null; @@ -135,7 +166,14 @@ export class Worker< private _startDeferred: Deferred | null = null; private _stopDeferred: Deferred | null = null; private _abortController: AbortController | null = null; + private _drainDidWork = false; private _runningTasks = new Map>(); + private eventProcessingGate: Promise = Promise.resolve(); + + /** Used by Orchestrator to prevent local event fan-out before every worker registers. */ + setEventProcessingGate(gate: Promise): void { + this.eventProcessingGate = gate; + } constructor( public readonly queueName: string, @@ -144,6 +182,7 @@ export class Worker< private readonly logger: Logger, config: Partial = {}, private readonly extraContext: object = {}, + private readonly eventDefinitions: readonly EventDefinition[] = [], ) { const maintenanceTask = createMaintenanceTask(this.queueName); this.tasks = tasks.reduce( @@ -179,6 +218,11 @@ export class Worker< return this._stopDeferred?.promise || Promise.resolve(); } + /** @internal Whether the last run-once pass observed any work. */ + get drainDidWork(): boolean { + return this._drainDidWork; + } + /** * Start the worker. * Returns when startup is complete (registration done). @@ -217,12 +261,30 @@ export class Worker< } this.orchestratorId = orchestratorId; + this._drainDidWork = false; this._startDeferred = new Deferred(); this._stopDeferred = new Deferred(); this._abortController = new AbortController(); - // Synchronous registration - await this.register(); + // Synchronous registration. A failed registration rejects both lifecycle + // promises; callers must never observe a worker that started partially. + try { + await this.register(); + } catch (error) { + // Registration is part of startup, not a running pipeline. Resolve the + // stop promise so callers that only await `start()` do not get an + // unhandled rejection, then discard every piece of this failed attempt. + const stopDeferred = this._stopDeferred; + this._abortController.abort(); + // Reject `started` so an Orchestrator observes registration failure, but + // attach a noop handler because callers that only use start() do not + // necessarily observe this internal lifecycle promise. + this._startDeferred.promise.catch(() => {}); + this._startDeferred.reject(error); + if (!stopDeferred.isSettled) stopDeferred.resolve(); + this.resetLifecycle(); + throw error; + } // Worker is now started this._startDeferred.resolve(); @@ -237,24 +299,114 @@ export class Worker< } const queue = new BatchingAsyncQueue(this.fetchBatchSize * 2, batchConfigs); - void this.fetchExecutions(queue, { runOnce }); + if (runOnce) { + void this.runDrainPipeline(); + } else { + void this.fetchExecutions(queue, { runOnce }); + void (async () => { + try { + await Promise.all([ + this.flushResults(this.executeTasks(queue)), + this.runEventProcessor(), + ]); + } catch (error) { + this.logger.error("Worker pipeline error:", error); + this._stopDeferred?.reject(error); + } finally { + queue.close(); + if (this._stopDeferred && !this._stopDeferred.isSettled) this._stopDeferred.resolve(); + this.resetLifecycle(); + } + })(); + } - (async () => { - try { - // Consume from queue → execute → flush + return this._startDeferred.promise; + } + + private async runEventProcessor(): Promise { + await Promise.race([ + this.eventProcessingGate, + new Promise((resolve) => + this.signal.addEventListener("abort", () => resolve(), { once: true }), + ), + ]); + await this.processEventBatches({ runOnce: false }); + } + + private async runDrainPipeline(): Promise { + try { + while (!this.signal.aborted) { + await Promise.race([ + this.eventProcessingGate, + new Promise((resolve) => + this.signal.addEventListener("abort", () => resolve(), { + once: true, + }), + ), + ]); + const events = await this.processEventBatches({ runOnce: true }); + this._drainDidWork ||= events > 0; + const queue = new BatchingAsyncQueue( + this.fetchBatchSize * 2, + new Map( + Array.from(this.tasks.entries()).flatMap(([key, task]) => + task.batch ? [[key, task.batch] as const] : [], + ), + ), + ); + const fetched = this.fetchExecutions(queue, { runOnce: true }); await this.flushResults(this.executeTasks(queue)); - } catch (err) { - this.logger.error("Worker pipeline error:", err); - } finally { - queue.close(); - this._stopDeferred?.resolve(); - this._startDeferred = null; - this._stopDeferred = null; - this._abortController = null; + const executions = await fetched; + this._drainDidWork ||= executions > 0; + if (events === 0 && executions === 0) break; } - })(); + } catch (error) { + this.logger.error("Worker pipeline error:", error); + this._stopDeferred?.reject(error); + } finally { + if (this._stopDeferred && !this._stopDeferred.isSettled) this._stopDeferred.resolve(); + this.resetLifecycle(); + } + } - return this._startDeferred.promise; + private resetLifecycle(): void { + this._startDeferred = null; + this._stopDeferred = null; + this._abortController = null; + this.orchestratorId = null; + this.eventProcessingGate = Promise.resolve(); + } + + private async processEventBatches({ runOnce }: { runOnce: boolean }): Promise { + let total = 0; + while (!this.signal.aborted) { + try { + const resolvedBefore = await this.db.resolveEventWaits( + { batchSize: EVENT_BATCH_SIZE }, + { signal: this.signal }, + ); + const processed = await this.db.processEvents( + { batchSize: EVENT_BATCH_SIZE }, + { signal: this.signal }, + ); + const resolvedAfter = await this.db.resolveEventWaits( + { batchSize: EVENT_BATCH_SIZE }, + { signal: this.signal }, + ); + const didWork = resolvedBefore + processed + resolvedAfter; + total += didWork; + if (runOnce && didWork === 0) return total; + if (!runOnce && didWork === 0) { + await waitFor(this.pollIntervalMs, { signal: this.signal }); + } + } catch (error) { + // Event fan-out is transactional. Non-retryable errors indicate a + // broken processor/schema and must fail the worker, not spin forever. + if (!isRetryableEventError(error)) throw error; + await waitFor(this.pollIntervalMs, { signal: this.signal }); + } + } + return total; } /** @@ -294,6 +446,37 @@ export class Worker< } private async register(): Promise { + // Filters are an explicit event-definition allowlist. Validate again at + // registration so tasks assembled outside Conductor cannot bypass it. + for (const task of this.tasks.values()) { + for (const trigger of task.triggers) { + if (!("event" in trigger)) continue; + if ("when" in trigger) { + throw new Error(`Custom event "${trigger.event}" does not support a when clause`); + } + if (!("filter" in trigger) || !trigger.filter) continue; + const definition = this.eventDefinitions.find((event) => event.name === trigger.event); + if (!definition) { + throw new Error(`Filtered event "${trigger.event}" has no runtime event definition`); + } + const allowed = new Set(definition.filterable ?? []); + for (const [field, values] of Object.entries(trigger.filter as Record)) { + if (!allowed.has(field)) + throw new Error( + `Filter for event "${trigger.event}" contains undeclared field "${field}"`, + ); + if (!Array.isArray(values)) + throw new Error( + `Filter value for event "${trigger.event}" field "${field}" must be an array`, + ); + if (values.length === 0) + throw new Error( + `Filter value for event "${trigger.event}" field "${field}" cannot be empty`, + ); + } + } + } + // Convert RetentionSettings to integer: null=keep, 0=delete now, N=delete after N days const retentionToDays = (setting: boolean | { days: number } | undefined): number | null => { if (setting === undefined || setting === false) return null; @@ -309,13 +492,16 @@ export class Worker< removeOnFailDays: retentionToDays(task.removeOnFail), window: task.window, concurrency: task.concurrency, + groupConcurrency: task.groupConcurrency, + deadLetterQueue: task.deadLetter?.queue, + deadLetterTaskKey: task.deadLetter?.task?.name, })); const allTasks = Array.from(this.tasks.values()); const cronSchedules: ExecutionSpec[] = allTasks.flatMap((task) => task.triggers - .filter((t): t is { cron: string; name: string } => "cron" in t) + .filter((t): t is { cron: string; name: string; group?: string } => "cron" in t) .map((trigger) => { const interval = CronExpressionParser.parse(trigger.cron); const nextTimestamp = interval.next().toDate(); @@ -326,44 +512,55 @@ export class Worker< run_at: nextTimestamp, dedupe_key: `scheduled::${trigger.name}::${timestampSeconds}`, cron_expression: trigger.cron, + group: trigger.group || null, }; }), ); const eventSubscriptions: EventSubscriptionSpec[] = allTasks.flatMap((task) => { - const customEvents = task.triggers - .filter((t) => "event" in t && typeof t.event === "string") - .map((trigger): EventSubscriptionSpec => { - const customTrigger = trigger as any; - return { + const customEvents = task.triggers.flatMap((trigger): EventSubscriptionSpec[] => { + if (!("event" in trigger) || "schema" in trigger || typeof trigger.event !== "string") + return []; + const customTrigger = trigger as CustomEventTrigger< + string, + string | undefined, + Record | undefined + >; + return [ + { task_key: task.name, queue: this.queueName, event_key: customTrigger.event, schema_name: null, table_name: null, operation: null, - when_clause: customTrigger.when || null, - payload_fields: customTrigger.fields?.split(",").map((f: string) => f.trim()) || null, + when_clause: null, + payload_fields: customTrigger.fields?.split(",").map((field) => field.trim()) || null, column_names: null, - }; - }); + filter: customTrigger.filter || null, + }, + ]; + }); - const dbEvents = task.triggers - .filter((t) => "schema" in t && "table" in t && "operation" in t) - .map((trigger): EventSubscriptionSpec => { - const dbTrigger = trigger as any; - return { + const dbEvents = task.triggers.flatMap((trigger): EventSubscriptionSpec[] => { + if (!("schema" in trigger) || !("table" in trigger) || !("operation" in trigger)) return []; + const databaseTrigger = trigger as DatabaseEventTrigger; + return [ + { task_key: task.name, queue: this.queueName, event_key: null, - schema_name: dbTrigger.schema, - table_name: dbTrigger.table, - operation: dbTrigger.operation, - when_clause: dbTrigger.when || null, + schema_name: databaseTrigger.schema, + table_name: databaseTrigger.table, + operation: databaseTrigger.operation, + when_clause: databaseTrigger.when || null, payload_fields: null, - column_names: dbTrigger.columns?.split(",").map((c: string) => c.trim()) || null, - }; - }); + column_names: + databaseTrigger.columns?.split(",").map((column) => column.trim()) || null, + filter: null, + }, + ]; + }); return [...customEvents, ...dbEvents]; }); @@ -383,7 +580,8 @@ export class Worker< private async fetchExecutions( queue: BatchingAsyncQueue, { runOnce = false }: { runOnce?: boolean }, - ) { + ): Promise { + let fetched = 0; assert.ok(this.orchestratorId, "orchestratorId must be set when starting the pipeline"); // Pre-compute task metadata once @@ -392,10 +590,6 @@ export class Worker< for (const task of allTasks) { taskMaxAttempts[task.name] = task.maxAttempts || 3; } - const taskKeysWithConcurrency = allTasks - .filter((task) => task.concurrency != null) - .map((task) => task.name); - // Check if any tasks have windows - only then do we need time-based filtering const tasksWithWindows = allTasks.filter((task) => task.window); @@ -436,7 +630,6 @@ export class Worker< queueName: this.queueName, batchSize: this.fetchBatchSize, filterTaskKeys: disallowedTaskKeys, - taskKeysWithConcurrency, }, { signal: this.signal }, ); @@ -451,6 +644,7 @@ export class Worker< } for (const exec of executions) { + fetched++; await queue.push(exec); // waits if full if (this.signal.aborted) break; } @@ -460,6 +654,7 @@ export class Worker< } queue.close(); + return fetched; } // --- Stage 2: Execute tasks concurrently --- @@ -469,17 +664,20 @@ export class Worker< for await (const result of mapConcurrent( source, this.concurrency, - async ({ taskKey, items: executions }): Promise => { + async ({ + taskKey, + items: executions, + }): Promise => { // Dispatch to correct task based on task_key const task = this.tasks.get(taskKey); if (!task) { return executions.map((exec) => ({ queue: exec.queue, execution_id: exec.id, + orchestrator_id: exec.locked_by, task_key: taskKey, status: "failed", error: `Task not found: ${taskKey}`, - slot_group_number: exec.slot_group_number, })) as ExecutionResult[]; } @@ -489,11 +687,11 @@ export class Worker< // All cancelled - return failures for all return executions.map((exec) => ({ execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: taskKey, status: "permanently_failed", error: exec.last_error || "Execution was cancelled", - slot_group_number: exec.slot_group_number, })) as ExecutionResult[]; } @@ -523,7 +721,9 @@ export class Worker< return this.executeSingleTask(task, singleExec); }, )) { - // Yield results (may be single or array) + // Waiting executions are released atomically by registerEventWait and + // therefore have no result to settle here. + if (result === null) continue; if (Array.isArray(result)) { for (const r of result) yield r; } else { @@ -538,7 +738,7 @@ export class Worker< * @param task - The task to execute * @param exec - The execution details */ - private async executeSingleTask(task: AnyTask, exec: Execution): Promise { + private async executeSingleTask(task: AnyTask, exec: Execution): Promise { const taskAbortController = createTaskSignal(this.signal); this._runningTasks.set(exec.id, taskAbortController); @@ -589,6 +789,7 @@ export class Worker< execution: exec, logger: makeChildLogger(this.logger, { execution_id: exec.id, + orchestrator_id: exec.locked_by, task_key: exec.task_key, queue: exec.queue, }), @@ -602,9 +803,12 @@ export class Worker< if (isTaskAbortReason(output)) { switch (output.reason) { + case "wait-for-event": + return null; case "child-invocation": return { execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: exec.task_key, status: "invoke_child", @@ -613,27 +817,27 @@ export class Worker< child_task_name: output.task.name, child_task_queue: output.task.queue || "default", child_payload: output.payload, - slot_group_number: exec.slot_group_number, + group: output.group, } as const; case "cancelled": return { execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: exec.task_key, status: "permanently_failed", error: exec.last_error || "Task was cancelled", - slot_group_number: exec.slot_group_number, } as const; case "released": case "parent-aborted": return { execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, reschedule_in_ms: output.reason === "released" ? output.reschedule_in_ms : undefined, step_key: output.reason === "released" ? output.step_key : undefined, task_key: exec.task_key, status: "released", - slot_group_number: exec.slot_group_number, } as const; default: assert.never(output); @@ -642,20 +846,20 @@ export class Worker< return { execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: exec.task_key, status: "completed", result: output, - slot_group_number: exec.slot_group_number, } as const; } catch (err) { return { execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: exec.task_key, status: "failed", error: coerceError(err).message, - slot_group_number: exec.slot_group_number, } as const; } finally { // Clean up running task tracking @@ -725,23 +929,23 @@ export class Worker< // Batch sleep - reschedule all return executions.map((exec) => ({ execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: taskKey, status: "released" as const, reschedule_in_ms: result.reschedule_in_ms, step_key: result.step_key, - slot_group_number: exec.slot_group_number, })); } // Other abort reasons return executions.map((exec) => ({ execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: taskKey, status: "failed" as const, error: `Task aborted: ${result.reason}`, - slot_group_number: exec.slot_group_number, })); } @@ -749,11 +953,11 @@ export class Worker< if (result === undefined) { return executions.map((exec) => ({ execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: taskKey, status: "completed" as const, result: undefined, - slot_group_number: exec.slot_group_number, })); } @@ -771,22 +975,22 @@ export class Worker< // Individual results return executions.map((exec, i) => ({ execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: taskKey, status: "completed" as const, result: result[i], - slot_group_number: exec.slot_group_number, })); } catch (err) { // Handler threw: all fail together const errorMsg = coerceError(err).message; return executions.map((exec) => ({ execution_id: exec.id, + orchestrator_id: exec.locked_by, queue: exec.queue, task_key: taskKey, status: "failed" as const, error: errorMsg, - slot_group_number: exec.slot_group_number, })); } } @@ -819,6 +1023,7 @@ export class Worker< run_at: nextTimestamp, dedupe_key: nextDedupeKey, cron_expression: execution.cron_expression, + group: execution.group || null, }, { signal: this.signal }, ); @@ -841,6 +1046,7 @@ export class Worker< } try { + batch.orchestratorId = this.orchestratorId || batch.orchestratorId; await this.db.returnExecutions(batch, { signal: this.signal }); } catch (err) { this.logger.error("Error flushing results:", err); diff --git a/packages/pgconductor-js/tests/integration/concurrency.test.ts b/packages/pgconductor-js/tests/integration/concurrency.test.ts index 9446a83..1833b28 100644 --- a/packages/pgconductor-js/tests/integration/concurrency.test.ts +++ b/packages/pgconductor-js/tests/integration/concurrency.test.ts @@ -216,68 +216,6 @@ describe("Task-Level Concurrency", () => { await orchestrator.stopped; }, 30000); - test("high concurrency uses slot groups correctly", async () => { - const db = await pool.child(); - databases.push(db); - - const taskDef = defineTask({ name: "high-concurrency-task" }); - - const conductor = Conductor.create({ - sql: db.sql, - tasks: TaskSchemas.fromSchema([taskDef]), - context: {}, - }); - - let completed = 0; - - const highConcurrencyTask = conductor.createTask( - { name: "high-concurrency-task", concurrency: 500 }, - { invocable: true }, - async () => { - completed++; - }, - ); - - const orchestrator = Orchestrator.create({ - conductor, - tasks: [highConcurrencyTask], - defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50, concurrency: 50 }, - }); - - await orchestrator.start(); - - // queue 100 tasks - for (let i = 0; i < 100; i++) { - await conductor.invoke({ name: "high-concurrency-task" }, {}); - } - - // wait for completion - await new Promise((r) => setTimeout(r, 5000)); - - await orchestrator.stop(); - await orchestrator.stopped; - - // extra wait to ensure all flushes complete - await new Promise((r) => setTimeout(r, 1000)); - - // all tasks should complete - expect(completed).toBe(100); - - // verify slots were created (one row per slot, concurrency=500 → 500 rows) - const slots = await db.sql` - select * from pgconductor._private_concurrency_slots - where task_key = 'high-concurrency-task' - `; - - expect(slots.length).toBe(500); // one row per slot - expect(slots.every((s) => s.capacity === 1)).toBe(true); // each slot has capacity=1 - - // verify all slots are released (used = 0) - for (const slot of slots) { - expect(slot.used).toBe(0); - } - }, 30000); - test("mixed queue with concurrency and without", async () => { const db = await pool.child(); databases.push(db); diff --git a/packages/pgconductor-js/tests/integration/cron-scheduling.test.ts b/packages/pgconductor-js/tests/integration/cron-scheduling.test.ts index 6699181..409e7ab 100644 --- a/packages/pgconductor-js/tests/integration/cron-scheduling.test.ts +++ b/packages/pgconductor-js/tests/integration/cron-scheduling.test.ts @@ -7,6 +7,17 @@ import { TestDatabasePool } from "../fixtures/test-database"; import { waitFor } from "../../src/lib/wait-for"; import { TaskSchemas } from "../../src/schemas"; +async function waitForCondition( + condition: () => boolean | Promise, + timeoutMs = 5000, +): Promise { + const deadline = Date.now() + timeoutMs; + while (!(await condition())) { + if (Date.now() >= deadline) throw new Error("condition was not met before timeout"); + await waitFor(50); + } +} + describe("Cron Scheduling", () => { let pool: TestDatabasePool; @@ -101,26 +112,20 @@ describe("Cron Scheduling", () => { await orchestrator.start(); - // Wait for first execution - await waitFor(4000); - expect(executions.mock.calls.length).toBeGreaterThanOrEqual(1); - - // Check that next execution is scheduled - const schedules = await db.sql>` - SELECT dedupe_key, run_at - FROM pgconductor._private_executions - WHERE task_key = 'frequent-sync' - AND dedupe_key LIKE 'scheduled::%' - AND run_at > pgconductor._private_current_time() - ORDER BY run_at - LIMIT 1 - `; + await waitForCondition(() => executions.mock.calls.length >= 1, 7000); - expect(schedules.length).toBe(1); + // A completed cron execution durably creates a distinct next occurrence. + await waitForCondition(async () => { + const schedules = await db.sql>` + SELECT count(*)::integer AS count + FROM pgconductor._private_executions + WHERE task_key = 'frequent-sync' + AND dedupe_key LIKE 'scheduled::%' + `; + return (schedules[0]?.count ?? 0) >= 2; + }, 7000); - // Wait for second execution - await waitFor(4000); - expect(executions.mock.calls.length).toBeGreaterThanOrEqual(2); + await waitForCondition(() => executions.mock.calls.length >= 2, 7000); await orchestrator.stop(); await db.destroy(); @@ -361,21 +366,23 @@ describe("Cron Scheduling", () => { await orchestrator.start(); await conductor.invoke({ name: "dynamic-scheduler" }, {}); - await waitFor(4000); - expect(targetExecutions.mock.calls.length).toBeGreaterThanOrEqual(1); - - const nextSchedules = await db.sql>` - SELECT dedupe_key - FROM pgconductor._private_executions - WHERE task_key = 'dynamic-target' - AND cron_expression IS NOT NULL - AND run_at > pgconductor._private_current_time() - ORDER BY run_at - LIMIT 1 - `; - - expect(nextSchedules.length).toBe(1); - expect(nextSchedules[0]!.dedupe_key).toMatch(/^scheduled::reporting::\d+$/); + await waitForCondition(() => targetExecutions.mock.calls.length >= 1, 20_000); + + let nextSchedule: { dedupe_key: string } | undefined; + await waitForCondition(async () => { + [nextSchedule] = await db.sql>` + SELECT dedupe_key + FROM pgconductor._private_executions + WHERE task_key = 'dynamic-target' + AND cron_expression IS NOT NULL + AND run_at > pgconductor._private_current_time() + ORDER BY run_at + LIMIT 1 + `; + return nextSchedule !== undefined; + }, 20_000); + + expect(nextSchedule?.dedupe_key).toMatch(/^scheduled::reporting::\d+$/); await orchestrator.stop(); await db.destroy(); @@ -424,11 +431,13 @@ describe("Cron Scheduling", () => { }, ); + let unscheduled = false; const unschedulerTask = conductor.createTask( { name: "dynamic-unscheduler" }, { invocable: true }, async (_event, ctx) => { await ctx.unschedule({ name: "dynamic-target" }, "reporting"); + unscheduled = true; }, ); @@ -443,12 +452,22 @@ describe("Cron Scheduling", () => { await orchestrator.start(); await conductor.invoke({ name: "dynamic-scheduler" }, {}); - await waitFor(4000); - expect(targetExecutions.mock.calls.length).toBeGreaterThanOrEqual(1); + await waitForCondition(() => targetExecutions.mock.calls.length >= 1, 7000); + await waitForCondition(async () => { + const active = await db.sql>` + SELECT count(*)::integer AS count + FROM pgconductor._private_executions + WHERE task_key = 'dynamic-target' + AND locked_at IS NOT NULL + AND completed_at IS NULL + AND failed_at IS NULL + `; + return (active[0]?.count ?? 0) === 0; + }); const runsBeforeUnschedule = targetExecutions.mock.calls.length; await conductor.invoke({ name: "dynamic-unscheduler" }, {}); - await waitFor(2000); + await waitForCondition(() => unscheduled); const futureSchedules = await db.sql>` SELECT id @@ -558,18 +577,21 @@ describe("Cron Scheduling", () => { // Should have at least attempted twice (fail + success) expect(attemptCount).toBeGreaterThanOrEqual(2); - // Verify next cron execution is scheduled - const nextExecution = await db.sql>` - SELECT run_at, dedupe_key - FROM pgconductor._private_executions - WHERE task_key = 'flaky-cron' - AND dedupe_key LIKE 'scheduled::%' - AND run_at > pgconductor._private_current_time() - ORDER BY run_at - LIMIT 1 - `; + // Verify next cron execution is scheduled without sampling at a claim boundary. + let nextExecution: Array<{ run_at: Date; dedupe_key: string }> = []; + await waitForCondition(async () => { + nextExecution = await db.sql>` + SELECT run_at, dedupe_key + FROM pgconductor._private_executions + WHERE task_key = 'flaky-cron' + AND dedupe_key LIKE 'scheduled::%' + AND run_at > pgconductor._private_current_time() + ORDER BY run_at + LIMIT 1 + `; + return nextExecution.length === 1; + }); - expect(nextExecution.length).toBe(1); expect(nextExecution[0]?.dedupe_key).toMatch(/^scheduled::.*::\d+$/); await orchestrator.stop(); diff --git a/packages/pgconductor-js/tests/integration/dead-letter.test.ts b/packages/pgconductor-js/tests/integration/dead-letter.test.ts new file mode 100644 index 0000000..3f513cb --- /dev/null +++ b/packages/pgconductor-js/tests/integration/dead-letter.test.ts @@ -0,0 +1,411 @@ +import { afterAll, afterEach, beforeAll, describe, expect, test } from "bun:test"; +import { z } from "zod"; +import { Conductor } from "../../src/conductor"; +import { Orchestrator } from "../../src/orchestrator"; +import { TaskSchemas } from "../../src/schemas"; +import { defineTask } from "../../src/task-definition"; +import { TestDatabasePool, type TestDatabase } from "../fixtures/test-database"; + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +async function eventually(check: () => Promise, timeoutMs = 20_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (await check()) return; + await sleep(25); + } + throw new Error(`condition was not met within ${timeoutMs}ms`); +} + +describe("dead-letter queues (Postgres integration)", () => { + let pool: TestDatabasePool; + const databases: TestDatabase[] = []; + + beforeAll(async () => { + pool = await TestDatabasePool.create(); + }, 60000); + + afterEach(async () => { + await Promise.all(databases.map((db) => db.destroy())); + databases.length = 0; + }); + + afterAll(async () => { + await pool?.destroy(); + }); + + test("retries, then delivers the final failure with payload and metadata", async () => { + const db = await pool.child(); + databases.push(db); + const payloadSchema = z.object({ value: z.string() }); + const sourceDefinition = defineTask({ name: "charge", payload: payloadSchema }); + const destinationDefinition = defineTask({ + name: "alternate-failure", + queue: "dlq", + payload: payloadSchema, + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([sourceDefinition, destinationDefinition]), + context: {}, + }); + const seen: Array<{ + value: string; + sourceTask: string; + attempts: number; + error: string | null; + }> = []; + const destination = conductor.createTask( + { name: "alternate-failure", queue: "dlq" }, + { invocable: true }, + async (event, ctx) => { + if (event.name === "pgconductor.invoke") { + if (!ctx.deadLetter) throw new Error("missing dead-letter metadata"); + seen.push({ + value: event.payload.value, + sourceTask: ctx.deadLetter.sourceTaskKey, + attempts: ctx.deadLetter.attempts, + error: ctx.deadLetter.error, + }); + } + }, + ); + const source = conductor.createTask( + { + name: "charge", + maxAttempts: 2, + removeOnFail: true, + deadLetter: { queue: "dlq", task: destination }, + }, + { invocable: true }, + async () => { + throw new Error("card declined"); + }, + ); + const sourceOrchestrator = Orchestrator.create({ + conductor, + tasks: [source], + defaultWorker: { pollIntervalMs: 10, flushIntervalMs: 10 }, + }); + await sourceOrchestrator.start(); + await conductor.invoke({ name: "charge" }, { value: "order-42" }); + await eventually(async () => { + const rows = await db.sql<{ attempts: number; released: boolean }[]>` + select attempts, locked_by is null as released + from pgconductor._private_executions + where task_key = 'charge' + `; + return rows[0]?.attempts === 1 && rows[0].released; + }); + const [retry] = await db.sql<{ run_at: Date }[]>` + select run_at from pgconductor._private_executions + where task_key = 'charge' + `; + if (!retry) throw new Error("expected persisted retry"); + await db.client.setFakeTime({ date: new Date(retry.run_at.getTime() + 1) }); + await eventually(async () => { + const rows = await db.sql<{ count: string }[]>` + select count(*)::text as count from pgconductor._private_executions + where queue = 'dlq' + and dead_letter_source_task_key = 'charge' + `; + return rows[0]?.count === "1"; + }); + const sourceRows = await db.sql<{ id: string }[]>` + select id from pgconductor._private_executions where task_key = 'charge' + `; + expect(sourceRows).toHaveLength(0); + await sourceOrchestrator.stop(); + + // The source registration created the destination partition, but the destination + // worker was intentionally started later. + const destinationOrchestrator = Orchestrator.create({ + conductor, + workers: [ + conductor.createWorker({ + queue: "dlq", + tasks: [destination], + config: { pollIntervalMs: 10, flushIntervalMs: 10 }, + }), + ], + }); + await destinationOrchestrator.start(); + await eventually(async () => seen.length === 1); + await destinationOrchestrator.stop(); + expect(seen).toEqual([ + { value: "order-42", sourceTask: "charge", attempts: 2, error: "card declined" }, + ]); + }, 60_000); + + test("ignores duplicate settlements and wrong worker identity", async () => { + const db = await pool.child(); + databases.push(db); + const payloadSchema = z.object({ value: z.string() }); + const sourceDefinition = defineTask({ name: "settle-source", payload: payloadSchema }); + const destinationDefinition = defineTask({ + name: "settle-destination", + queue: "dlq", + payload: payloadSchema, + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([sourceDefinition, destinationDefinition]), + context: {}, + }); + await conductor.ensureInstalled(); + await db.client.registerWorker({ + queueName: "default", + taskSpecs: [ + { + key: "settle-source", + queue: "default", + maxAttempts: 1, + deadLetterQueue: "dlq", + deadLetterTaskKey: "settle-destination", + }, + ], + cronSchedules: [], + eventSubscriptions: [], + }); + const executionId = await db.client.invoke({ + task_key: "settle-source", + queue: "default", + payload: { value: "x" }, + }); + const orchestratorId = crypto.randomUUID(); + const claimed = await db.client.getExecutions({ + orchestratorId, + queueName: "default", + batchSize: 1, + filterTaskKeys: [], + }); + expect(claimed).toHaveLength(1); + const execution = claimed[0]!; + const result = { + execution_id: execution.id, + queue: execution.queue, + task_key: execution.task_key, + status: "permanently_failed" as const, + orchestrator_id: execution.locked_by, + error: "settlement failure", + }; + await db.client.returnExecutions({ + count: 1, + orchestratorId, + completed: [], + failed: [{ ...result }], + released: [], + invokeChild: [], + taskKeys: new Set([execution.task_key]), + }); + await db.client.returnExecutions({ + count: 1, + orchestratorId, + completed: [], + failed: [{ ...result }], + released: [], + invokeChild: [], + taskKeys: new Set([execution.task_key]), + }); + const rows = await db.sql<{ count: string; source: string | null }[]>` + select count(*)::text as count, min(dead_letter_source_execution_id::text) as source + from pgconductor._private_executions where queue = 'dlq' + `; + expect(rows[0]).toEqual({ count: "1", source: executionId }); + + // A result from a different worker is fenced out as well. + await db.client.returnExecutions({ + count: 1, + orchestratorId: crypto.randomUUID(), + completed: [], + failed: [{ ...result, orchestrator_id: crypto.randomUUID() }], + released: [], + invokeChild: [], + taskKeys: new Set([execution.task_key]), + }); + const count = await db.sql<{ count: string }[]>` + select count(*)::text as count from pgconductor._private_executions where queue = 'dlq' + `; + expect(count[0]?.count).toBe("1"); + }, 15000); + + test("cancellation never delivers to the DLQ", async () => { + const db = await pool.child(); + databases.push(db); + const sourceDefinition = defineTask({ name: "cancel-source", payload: z.object({}) }); + const destinationDefinition = defineTask({ + name: "cancel-destination", + queue: "dlq", + payload: z.object({}), + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([sourceDefinition, destinationDefinition]), + context: {}, + }); + await conductor.ensureInstalled(); + await db.client.registerWorker({ + queueName: "default", + taskSpecs: [ + { + key: "cancel-source", + queue: "default", + maxAttempts: 1, + deadLetterQueue: "dlq", + deadLetterTaskKey: "cancel-destination", + }, + ], + cronSchedules: [], + eventSubscriptions: [], + }); + const id = (await db.client.invoke({ + task_key: "cancel-source", + queue: "default", + payload: {}, + }))!; + expect(await db.client.cancelExecution(id)).toBe(true); + const rows = await db.sql<{ failed_at: Date | null; cancelled: boolean }[]>` + select failed_at, cancelled from pgconductor._private_executions where id = ${id}::uuid + `; + expect(rows[0]?.failed_at).not.toBeNull(); + expect(rows[0]?.cancelled).toBe(false); + const destinationRows = await db.sql<{ count: string }[]>` + select count(*)::text as count from pgconductor._private_executions where queue = 'dlq' + `; + expect(destinationRows[0]?.count).toBe("0"); + }, 15000); + + test("rejects direct self-targets in the database but permits cross-queue identity", async () => { + const db = await pool.child(); + databases.push(db); + const conductor = Conductor.create({ sql: db.sql, context: {} }); + await conductor.ensureInstalled(); + await expect( + db.client.registerWorker({ + queueName: "default", + taskSpecs: [ + { + key: "self", + queue: "default", + maxAttempts: 1, + deadLetterQueue: "default", + }, + ], + cronSchedules: [], + eventSubscriptions: [], + }), + ).rejects.toThrow(); + + await expect( + db.client.registerWorker({ + queueName: "default", + taskSpecs: [ + { + key: "same-name", + queue: "default", + maxAttempts: 1, + deadLetterQueue: "other-queue", + deadLetterTaskKey: "same-name", + }, + ], + cronSchedules: [], + eventSubscriptions: [], + }), + ).resolves.toBeUndefined(); + }, 15000); + + test("rolls back the source settlement when destination insertion fails", async () => { + const db = await pool.child(); + databases.push(db); + const payloadSchema = z.object({ value: z.string() }); + const sourceDefinition = defineTask({ name: "rollback-source", payload: payloadSchema }); + const destinationDefinition = defineTask({ + name: "rollback-destination", + queue: "dlq", + payload: payloadSchema, + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([sourceDefinition, destinationDefinition]), + context: {}, + }); + await conductor.ensureInstalled(); + await db.client.registerWorker({ + queueName: "default", + taskSpecs: [ + { + key: "rollback-source", + queue: "default", + maxAttempts: 1, + deadLetterQueue: "dlq", + deadLetterTaskKey: "rollback-destination", + }, + ], + cronSchedules: [], + eventSubscriptions: [], + }); + const id = await db.client.invoke({ + task_key: "rollback-source", + queue: "default", + payload: { value: "rollback" }, + }); + const claimed = await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "default", + batchSize: 1, + filterTaskKeys: [], + }); + const execution = claimed[0]!; + if (!execution.locked_by) throw new Error("execution was not claimed"); + const lockedBy = execution.locked_by; + await db.sql.unsafe(` + create function public.fail_dlq_insert() returns trigger language plpgsql as $$ + begin raise exception 'forced destination failure'; end; + $$; + create trigger fail_dlq_insert before insert on pgconductor.executions_dlq + for each row execute function public.fail_dlq_insert(); + `); + const settlement = { + execution_id: execution.id, + queue: execution.queue, + task_key: execution.task_key, + status: "permanently_failed" as const, + orchestrator_id: lockedBy, + error: "rollback failure", + }; + await expect( + db.client.returnExecutions({ + count: 1, + orchestratorId: lockedBy, + completed: [], + failed: [settlement], + released: [], + invokeChild: [], + taskKeys: new Set([execution.task_key]), + }), + ).rejects.toThrow("forced destination failure"); + const afterRollback = await db.sql< + { failed_at: Date | null; locked_by: string | null; attempts: number }[] + >`select failed_at, locked_by, attempts from pgconductor._private_executions where id = ${id}::uuid`; + expect(afterRollback[0]?.failed_at).toBeNull(); + expect(afterRollback[0]?.locked_by).toBe(lockedBy); + expect(afterRollback[0]?.attempts).toBe(1); + await db.sql.unsafe( + `drop trigger fail_dlq_insert on pgconductor.executions_dlq; drop function public.fail_dlq_insert();`, + ); + await db.client.returnExecutions({ + count: 1, + orchestratorId: lockedBy, + completed: [], + failed: [settlement], + released: [], + invokeChild: [], + taskKeys: new Set([execution.task_key]), + }); + const finalRows = await db.sql<{ source: string | null; source_failed: string }[]>` + select dead_letter_source_execution_id::text as source, (select failed_at is not null from pgconductor._private_executions where id = ${id}::uuid)::text as source_failed + from pgconductor._private_executions where queue = 'dlq' + `; + expect(Array.from(finalRows)).toEqual([{ source: id, source_failed: "true" }]); + }, 20000); +}); diff --git a/packages/pgconductor-js/tests/integration/event-pipeline.test.ts b/packages/pgconductor-js/tests/integration/event-pipeline.test.ts new file mode 100644 index 0000000..a50306f --- /dev/null +++ b/packages/pgconductor-js/tests/integration/event-pipeline.test.ts @@ -0,0 +1,287 @@ +import { afterAll, afterEach, beforeAll, describe, expect, test } from "bun:test"; +import { z } from "zod"; +import { Conductor } from "../../src/conductor"; +import { DatabaseClient } from "../../src/database-client"; +import { DefaultLogger } from "../../src/lib/logger"; +import { Orchestrator } from "../../src/orchestrator"; +import { defineEvent } from "../../src/event-definition"; +import { defineTask } from "../../src/task-definition"; +import { EventSchemas, TaskSchemas } from "../../src/schemas"; +import { TestDatabasePool } from "../fixtures/test-database"; +import type { TestDatabase } from "../fixtures/test-database"; +import postgres from "postgres"; + +describe("event pipeline", () => { + let pool: TestDatabasePool; + const databases: TestDatabase[] = []; + + beforeAll(async () => { + pool = await TestDatabasePool.create(); + }, 60000); + + afterEach(async () => { + await Promise.all(databases.map((database) => database.destroy())); + databases.length = 0; + }); + + afterAll(async () => { + await pool?.destroy(); + }); + + async function database(): Promise { + const db = await pool.child(); + databases.push(db); + await Conductor.create({ sql: db.sql, context: {} }).ensureInstalled(); + return db; + } + + async function waitUntil(check: () => Promise, timeout = 5000): Promise { + const deadline = Date.now() + timeout; + while (Date.now() < deadline) { + if (await check()) return; + await Bun.sleep(10); + } + throw new Error("condition was not met before timeout"); + } + + async function subscription( + db: TestDatabase, + taskKey: string, + eventKey: string, + filter?: Record, + ): Promise { + await db.sql` + insert into pgconductor._private_tasks (key, queue) + values (${taskKey}, 'default') + on conflict (queue, key) do nothing + `; + await db.sql` + insert into pgconductor._private_event_subscriptions + (task_key, queue, event_key, filter) + values (${taskKey}, 'default', ${eventKey}, ${filter ? db.sql.json(filter) : null}) + `; + } + + test("matches one and multiple allowed values across all filter fields", async () => { + const db = await database(); + const event = defineEvent({ + name: "pipeline.order", + payload: z.object({ status: z.enum(["paid", "trial", "cancelled"]), region: z.string() }), + filterable: ["status", "region"], + }); + const taskDefinition = defineTask({ name: "pipeline-order-task", payload: z.object({}) }); + const received: string[] = []; + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([taskDefinition]), + events: EventSchemas.fromSchema([event]), + context: {}, + }); + const task = conductor.createTask( + { name: "pipeline-order-task" }, + { event: "pipeline.order", filter: { status: ["paid", "trial"], region: ["us"] } }, + async (receivedEvent) => { + received.push(receivedEvent.payload.status); + }, + ); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [task], + defaultWorker: { pollIntervalMs: 10, flushIntervalMs: 10 }, + }); + await orchestrator.start(); + try { + await conductor.emit("pipeline.order", { status: "paid", region: "us" }); + await conductor.emit("pipeline.order", { status: "trial", region: "us" }); + await conductor.emit("pipeline.order", { status: "paid", region: "eu" }); + await conductor.emit("pipeline.order", { status: "cancelled", region: "us" }); + await waitUntil(async () => received.length === 2); + expect(received.sort()).toEqual(["paid", "trial"]); + } finally { + await orchestrator.stop(); + } + }); + + test("rejects undeclared filter fields at runtime", async () => { + const db = await database(); + const event = defineEvent({ + name: "pipeline.runtime-filter", + payload: z.object({ status: z.string(), secret: z.string() }), + filterable: ["status"], + }); + const taskDefinition = defineTask({ name: "pipeline-runtime-task", payload: z.object({}) }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([taskDefinition]), + events: EventSchemas.fromSchema([event]), + context: {}, + }); + expect(() => + conductor.createTask( + { name: "pipeline-runtime-task" }, + { event: "pipeline.runtime-filter", filter: { secret: ["nope"] } } as never, + async () => {}, + ), + ).toThrow(/undeclared field/); + }); + + test("fans one event out to multiple subscriptions and ignores nonmatches", async () => { + const db = await database(); + await subscription(db, "pipeline.email", "pipeline.fanout", { kind: ["email"] }); + await subscription(db, "pipeline.audit", "pipeline.fanout", { kind: ["audit"] }); + await db.client.emitEvent({ eventKey: "pipeline.fanout", payload: { kind: "email" } }); + await db.client.emitEvent({ eventKey: "pipeline.fanout", payload: { kind: "other" } }); + expect(await db.client.processEvents({ batchSize: 10 })).toBe(2); + const rows = await db.sql<{ task_key: string; count: string }[]>` + select task_key, count(*)::text as count + from pgconductor._private_executions group by task_key order by task_key + `; + expect([...rows]).toEqual([{ task_key: "pipeline.email", count: "1" }]); + }); + + test("processes multiple event keys in bounded batches", async () => { + const db = await database(); + await subscription(db, "pipeline.one", "pipeline.one"); + await subscription(db, "pipeline.two", "pipeline.two"); + await db.client.emitEvent({ eventKey: "pipeline.one", payload: { n: 1 } }); + await db.client.emitEvent({ eventKey: "pipeline.two", payload: { n: 2 } }); + await db.client.emitEvent({ eventKey: "pipeline.one", payload: { n: 3 } }); + expect(await db.client.processEvents({ batchSize: 2 })).toBe(2); + const [pending] = await db.sql<{ count: string }[]>` + select count(*)::text as count from pgconductor._private_custom_events where processed_at is null + `; + expect(pending?.count).toBe("1"); + expect(await db.client.processEvents({ batchSize: 2 })).toBe(1); + const [executions] = await db.sql<{ count: string }[]>` + select count(*)::text as count from pgconductor._private_executions + where event_id is not null + `; + expect(executions?.count).toBe("3"); + }); + + test("is idempotent under repeated and concurrent processing", async () => { + const db = await database(); + await subscription(db, "pipeline.once", "pipeline.once"); + await db.client.emitEvent({ eventKey: "pipeline.once", payload: {} }); + const sql = postgres(db.url, { max: 2 }); + const clients = [ + new DatabaseClient({ sql, logger: new DefaultLogger() }), + new DatabaseClient({ sql, logger: new DefaultLogger() }), + ]; + try { + const counts = await Promise.all( + clients.map((client) => client.processEvents({ batchSize: 1 })), + ); + expect(counts.reduce((sum, count) => sum + count, 0)).toBe(1); + expect(await db.client.processEvents({ batchSize: 1 })).toBe(0); + const [deliveries] = await db.sql<{ count: string }[]>` + select count(*)::text as count from pgconductor._private_event_deliveries + `; + expect(deliveries?.count).toBe("1"); + } finally { + await sql.end(); + } + }); + + test("drains a fanout emitted by one queue into two queues", async () => { + const db = await database(); + const event = defineEvent({ + name: "pipeline.drain-fanout", + payload: z.object({ value: z.string() }), + }); + const sourceDefinition = defineTask({ name: "pipeline.source", payload: z.object({}) }); + const leftDefinition = defineTask({ + name: "pipeline.left", + queue: "left", + payload: z.object({}), + }); + const rightDefinition = defineTask({ + name: "pipeline.right", + queue: "right", + payload: z.object({}), + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([sourceDefinition, leftDefinition, rightDefinition]), + events: EventSchemas.fromSchema([event]), + context: {}, + }); + const received: string[] = []; + const source = conductor.createTask( + { name: "pipeline.source" }, + { invocable: true }, + async () => { + await conductor.emit("pipeline.drain-fanout", { value: "done" }); + }, + ); + const left = conductor.createTask( + { name: "pipeline.left", queue: "left" }, + { event: "pipeline.drain-fanout" }, + async (receivedEvent) => { + received.push(`left:${receivedEvent.payload.value}`); + }, + ); + const right = conductor.createTask( + { name: "pipeline.right", queue: "right" }, + { event: "pipeline.drain-fanout" }, + async (receivedEvent) => { + received.push(`right:${receivedEvent.payload.value}`); + }, + ); + await conductor.invoke({ name: "pipeline.source" }, {}); + + const orchestrator = Orchestrator.create({ + conductor, + tasks: [source], + workers: [ + conductor.createWorker({ queue: "left", tasks: [left] }), + conductor.createWorker({ queue: "right", tasks: [right] }), + ], + defaultWorker: { pollIntervalMs: 10, flushIntervalMs: 10 }, + }); + await orchestrator.drain(); + expect(received.sort()).toEqual(["left:done", "right:done"]); + }); + + test("rejects duplicate workers for one queue", async () => { + const db = await database(); + const definition = defineTask({ name: "pipeline.duplicate" }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([definition]), + context: {}, + }); + const task = conductor.createTask( + { name: "pipeline.duplicate" }, + { invocable: true }, + async () => {}, + ); + const worker = conductor.createWorker({ queue: "default", tasks: [task] }); + expect(() => Orchestrator.create({ conductor, tasks: [task], workers: [worker] })).toThrow( + /multiple workers for queue/, + ); + }); + + test("rolls back a failed processing transaction and retries", async () => { + const db = await database(); + await db.client.emitEvent({ eventKey: "pipeline.retry", payload: {} }); + await db.sql` + create function public.fail_event_execution() returns trigger language plpgsql as $$ + begin raise exception 'event execution insert failed'; end; $$ + `; + await db.sql` + create trigger fail_event_execution after insert on pgconductor._private_executions + execute function public.fail_event_execution() + `; + await expect(db.client.processEvents({ batchSize: 1 })).rejects.toThrow( + "event execution insert failed", + ); + const [rolledBack] = await db.sql<{ processed_at: Date | null }[]>` + select processed_at from pgconductor._private_custom_events + `; + expect(rolledBack?.processed_at).toBeNull(); + await db.sql`drop trigger fail_event_execution on pgconductor._private_executions`; + await db.sql`drop function public.fail_event_execution()`; + expect(await db.client.processEvents({ batchSize: 1 })).toBe(1); + }); +}); diff --git a/packages/pgconductor-js/tests/integration/event-triggers.test.ts b/packages/pgconductor-js/tests/integration/event-triggers.test.ts index 2975e54..a31ebe3 100644 --- a/packages/pgconductor-js/tests/integration/event-triggers.test.ts +++ b/packages/pgconductor-js/tests/integration/event-triggers.test.ts @@ -146,6 +146,7 @@ describe("Event Triggers - Custom Events", () => { const orderPlaced = defineEvent({ name: "order.placed", payload: z.object({ orderId: z.string(), total: z.number() }), + filterable: ["total"], }); const taskDef = defineTask({ @@ -167,7 +168,7 @@ describe("Event Triggers - Custom Events", () => { const task = conductor.createTask( { name: "on-large-order" }, - { event: "order.placed", when: "(new.payload->>'total')::numeric > 1000" }, + { event: "order.placed", filter: { total: [1500] } }, taskFn, ); diff --git a/packages/pgconductor-js/tests/integration/execution-foundations.test.ts b/packages/pgconductor-js/tests/integration/execution-foundations.test.ts new file mode 100644 index 0000000..8668e9f --- /dev/null +++ b/packages/pgconductor-js/tests/integration/execution-foundations.test.ts @@ -0,0 +1,573 @@ +import { afterAll, afterEach, beforeAll, describe, expect, test } from "bun:test"; +import { Conductor } from "../../src/conductor"; +import { TestDatabasePool, type TestDatabase } from "../fixtures/test-database"; + +describe("execution foundations", () => { + let pool: TestDatabasePool; + const databases: TestDatabase[] = []; + + beforeAll(async () => { + pool = await TestDatabasePool.create(); + }, 60000); + + afterEach(async () => { + await Promise.all(databases.map((database) => database.destroy())); + databases.length = 0; + }); + + afterAll(async () => { + await pool?.destroy(); + }); + + async function database(): Promise { + const database = await pool.child(); + databases.push(database); + const conductor = Conductor.create({ sql: database.sql, context: {} }); + await conductor.ensureInstalled(); + return database; + } + + function grouped( + result: Parameters[0]["completed"][number], + ) { + return { + count: 1, + orchestratorId: result.orchestrator_id, + completed: [result], + failed: [], + released: [], + invokeChild: [], + taskKeys: new Set([result.task_key]), + }; + } + + test("registers and executes the same task key independently in two queues", async () => { + const db = await database(); + + await db.client.registerWorker({ + queueName: "queue-a", + taskSpecs: [ + { + key: "same-task", + queue: "queue-a", + maxAttempts: 2, + removeOnCompleteDays: 0, + }, + ], + cronSchedules: [], + eventSubscriptions: [], + }); + await db.client.registerWorker({ + queueName: "queue-b", + taskSpecs: [ + { + key: "same-task", + queue: "queue-b", + maxAttempts: 7, + removeOnCompleteDays: 1, + }, + ], + cronSchedules: [], + eventSubscriptions: [], + }); + + const tasks = await db.sql< + { queue: string; key: string; max_attempts: number; remove_on_complete_days: number | null }[] + >` + select queue, key, max_attempts, remove_on_complete_days + from pgconductor._private_tasks + where key = 'same-task' + order by queue + `; + expect([...tasks]).toEqual([ + { queue: "queue-a", key: "same-task", max_attempts: 2, remove_on_complete_days: 0 }, + { queue: "queue-b", key: "same-task", max_attempts: 7, remove_on_complete_days: 1 }, + ]); + + const firstId = await db.client.invoke({ task_key: "same-task", queue: "queue-a" }); + const secondId = await db.client.invoke({ task_key: "same-task", queue: "queue-b" }); + expect(firstId).not.toBeNull(); + expect(secondId).not.toBeNull(); + + const first = ( + await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "queue-a", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + const second = ( + await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "queue-b", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + expect(first?.queue).toBe("queue-a"); + expect(second?.queue).toBe("queue-b"); + if (!first || !second) throw new Error("expected both executions to be claimed"); + + await db.client.returnExecutions( + grouped({ + execution_id: first.id, + queue: first.queue, + task_key: first.task_key, + orchestrator_id: first.locked_by, + status: "completed", + }), + ); + await db.client.returnExecutions( + grouped({ + execution_id: second.id, + queue: second.queue, + task_key: second.task_key, + orchestrator_id: second.locked_by, + status: "completed", + }), + ); + + const remaining = await db.sql<{ queue: string; completed_at: Date | null }[]>` + select queue, completed_at + from pgconductor._private_executions + where task_key = 'same-task' + order by queue + `; + expect([...remaining]).toHaveLength(1); + expect(remaining[0]?.queue).toBe("queue-b"); + expect(remaining[0]?.completed_at).not.toBeNull(); + }); + + test("retains a parent when its permanently failed child is configured for removal", async () => { + const db = await database(); + await db.client.registerWorker({ + queueName: "parent-retention", + taskSpecs: [ + { key: "parent", queue: "parent-retention", removeOnFailDays: 1 }, + { key: "child", queue: "parent-retention", maxAttempts: 1, removeOnFailDays: 0 }, + ], + cronSchedules: [], + eventSubscriptions: [], + }); + const parentId = await db.client.invoke({ task_key: "parent", queue: "parent-retention" }); + if (!parentId) throw new Error("expected parent execution"); + const parent = ( + await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "parent-retention", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!parent) throw new Error("expected parent claim"); + + await db.client.returnExecutions({ + count: 1, + orchestratorId: parent.locked_by, + completed: [], + failed: [], + released: [], + invokeChild: [ + { + execution_id: parent.id, + queue: parent.queue, + orchestrator_id: parent.locked_by, + task_key: parent.task_key, + status: "invoke_child", + timeout_ms: 5000, + step_key: "child-step", + child_task_name: "child", + child_task_queue: "parent-retention", + child_payload: null, + }, + ], + taskKeys: new Set([parent.task_key]), + }); + + const childId = ( + await db.sql<{ id: string }[]>` + select id from pgconductor._private_executions + where parent_execution_id = ${parentId}::uuid + ` + )[0]?.id; + if (!childId) throw new Error("expected child execution"); + const child = ( + await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "parent-retention", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!child) throw new Error("expected child claim"); + + await db.client.returnExecutions({ + count: 1, + orchestratorId: child.locked_by, + completed: [], + failed: [ + { + execution_id: child.id, + queue: child.queue, + orchestrator_id: child.locked_by, + task_key: child.task_key, + status: "permanently_failed", + error: "child failed", + }, + ], + released: [], + invokeChild: [], + taskKeys: new Set([child.task_key]), + }); + + const retained = await db.sql< + { id: string; failed_at: Date | null; last_error: string | null }[] + >` + select id, failed_at, last_error from pgconductor._private_executions + where id = ${parentId}::uuid + `; + expect(retained[0]?.id).toBe(parentId); + expect(retained[0]?.failed_at).not.toBeNull(); + expect(retained[0]?.last_error).toContain("Child execution failed"); + expect( + await db.sql`select 1 from pgconductor._private_executions where id = ${childId}::uuid`, + ).toHaveLength(0); + }); + + test("propagates a permanently failed child across queues to its parent", async () => { + const db = await database(); + await db.client.registerWorker({ + queueName: "parent-queue", + taskSpecs: [{ key: "parent", queue: "parent-queue", removeOnFailDays: 1 }], + cronSchedules: [], + eventSubscriptions: [], + }); + await db.client.registerWorker({ + queueName: "child-queue", + taskSpecs: [{ key: "child", queue: "child-queue", maxAttempts: 1, removeOnFailDays: 1 }], + cronSchedules: [], + eventSubscriptions: [], + }); + + const parentId = await db.client.invoke({ task_key: "parent", queue: "parent-queue" }); + if (!parentId) throw new Error("expected parent execution"); + const parent = ( + await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "parent-queue", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!parent) throw new Error("expected parent claim"); + + await db.client.returnExecutions({ + count: 1, + orchestratorId: parent.locked_by, + completed: [], + failed: [], + released: [], + invokeChild: [ + { + execution_id: parent.id, + queue: parent.queue, + orchestrator_id: parent.locked_by, + task_key: parent.task_key, + status: "invoke_child", + timeout_ms: "infinity", + step_key: "child-step", + child_task_name: "child", + child_task_queue: "child-queue", + child_payload: null, + }, + ], + taskKeys: new Set([parent.task_key]), + }); + + const child = ( + await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "child-queue", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!child) throw new Error("expected child claim"); + await db.client.returnExecutions({ + count: 1, + orchestratorId: child.locked_by, + completed: [], + failed: [ + { + execution_id: child.id, + queue: child.queue, + orchestrator_id: child.locked_by, + task_key: child.task_key, + status: "permanently_failed", + error: "child failed", + }, + ], + released: [], + invokeChild: [], + taskKeys: new Set([child.task_key]), + }); + + const outcome = await db.sql<{ failed_at: Date | null; last_error: string | null }[]>` + select failed_at, last_error + from pgconductor._private_executions + where id = ${parentId}::uuid + `; + expect(outcome[0]?.failed_at).not.toBeNull(); + expect(outcome[0]?.last_error).toBe("Child execution failed: child failed"); + }); + + test("cancellation fences a buffered completion and does not retry it", async () => { + const db = await database(); + await db.client.registerWorker({ + queueName: "cancel-buffered", + taskSpecs: [{ key: "cancelled", queue: "cancel-buffered", maxAttempts: 5 }], + cronSchedules: [], + eventSubscriptions: [], + }); + const orchestratorId = crypto.randomUUID(); + await db.client.orchestratorHeartbeat({ orchestratorId, version: "test", migrationNumber: 1 }); + const executionId = await db.client.invoke({ task_key: "cancelled", queue: "cancel-buffered" }); + if (!executionId) throw new Error("expected execution"); + const claimed = ( + await db.client.getExecutions({ + orchestratorId, + queueName: "cancel-buffered", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!claimed) throw new Error("expected claim"); + await db.client.cancelExecution(executionId, { reason: "cancelled before flush" }); + await db.client.returnExecutions( + grouped({ + execution_id: claimed.id, + queue: claimed.queue, + task_key: claimed.task_key, + orchestrator_id: claimed.locked_by, + status: "completed", + }), + ); + + const outcome = await db.sql< + { + failed_at: Date | null; + completed_at: Date | null; + locked_by: string | null; + attempts: number; + last_error: string | null; + }[] + >` + select failed_at, completed_at, locked_by, attempts, last_error + from pgconductor._private_executions + where id = ${executionId}::uuid + `; + expect(outcome[0]?.failed_at).not.toBeNull(); + expect(outcome[0]?.completed_at).toBeNull(); + expect(outcome[0]?.locked_by).toBeNull(); + expect(outcome[0]?.attempts).toBe(1); + expect(outcome[0]?.last_error).toBe("cancelled before flush"); + }); + + test("cancelling a waiting parent permanently fails its pending child", async () => { + const db = await database(); + await db.client.registerWorker({ + queueName: "cascade-parent", + taskSpecs: [{ key: "parent", queue: "cascade-parent" }], + cronSchedules: [], + eventSubscriptions: [], + }); + await db.client.registerWorker({ + queueName: "cascade-child", + taskSpecs: [{ key: "child", queue: "cascade-child" }], + cronSchedules: [], + eventSubscriptions: [], + }); + const parentId = await db.client.invoke({ task_key: "parent", queue: "cascade-parent" }); + if (!parentId) throw new Error("expected parent execution"); + const parent = ( + await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "cascade-parent", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!parent) throw new Error("expected parent claim"); + await db.client.returnExecutions({ + count: 1, + orchestratorId: parent.locked_by, + completed: [], + failed: [], + released: [], + invokeChild: [ + { + execution_id: parent.id, + queue: parent.queue, + orchestrator_id: parent.locked_by, + task_key: parent.task_key, + status: "invoke_child", + timeout_ms: "infinity", + step_key: "child-step", + child_task_name: "child", + child_task_queue: "cascade-child", + child_payload: null, + }, + ], + taskKeys: new Set([parent.task_key]), + }); + const childId = ( + await db.sql<{ id: string }[]>` + select id from pgconductor._private_executions where parent_execution_id = ${parentId}::uuid + ` + )[0]?.id; + if (!childId) throw new Error("expected child execution"); + await db.client.cancelExecution(parentId); + const outcome = await db.sql< + { + id: string; + failed_at: Date | null; + waiting_on_execution_id: string | null; + }[] + >` + select id, failed_at, waiting_on_execution_id + from pgconductor._private_executions + where id in (${parentId}::uuid, ${childId}::uuid) + order by id + `; + expect(outcome).toHaveLength(2); + expect(outcome.every((execution) => execution.failed_at !== null)).toBe(true); + expect( + outcome.find((execution) => execution.id === parentId)?.waiting_on_execution_id, + ).toBeNull(); + }); + + test("orders equal-priority executions by created_at and id", async () => { + const db = await database(); + await db.client.registerWorker({ + queueName: "enqueue-order", + taskSpecs: [{ key: "ordered", queue: "enqueue-order" }], + cronSchedules: [], + eventSubscriptions: [], + }); + const ids = await db.client.invokeBatch([ + { task_key: "ordered", queue: "enqueue-order", priority: 0 }, + { task_key: "ordered", queue: "enqueue-order", priority: 0 }, + { task_key: "ordered", queue: "enqueue-order", priority: 0 }, + ]); + const expected = await db.sql<{ id: string }[]>` + select id + from pgconductor._private_executions + where id = any(${db.sql.array(ids)}::uuid[]) + order by priority, run_at, created_at, id + `; + const claimed = await db.client.getExecutions({ + orchestratorId: crypto.randomUUID(), + queueName: "enqueue-order", + batchSize: 3, + filterTaskKeys: [], + }); + expect(claimed.map((execution) => execution.id)).toEqual( + expected.map((execution) => execution.id), + ); + }); + + test("fences stale completion, failure, and release results after recovery and re-claim", async () => { + const db = await database(); + await db.client.registerWorker({ + queueName: "fenced", + taskSpecs: [{ key: "fenced-task", queue: "fenced", maxAttempts: 3 }], + cronSchedules: [], + eventSubscriptions: [], + }); + const executionId = await db.client.invoke({ task_key: "fenced-task", queue: "fenced" }); + if (!executionId) throw new Error("expected execution id"); + + const oldOrchestrator = crypto.randomUUID(); + await db.client.orchestratorHeartbeat({ + orchestratorId: oldOrchestrator, + version: "test", + migrationNumber: 1, + }); + const oldClaim = ( + await db.client.getExecutions({ + orchestratorId: oldOrchestrator, + queueName: "fenced", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!oldClaim) throw new Error("expected old claim"); + + await db.sql` + update pgconductor._private_orchestrators + set last_heartbeat_at = now() - interval '1 hour' + where id = ${oldOrchestrator}::uuid + `; + await db.client.recoverStaleOrchestrators({ maxAge: "1 second" }); + + const newOrchestrator = crypto.randomUUID(); + const currentClaim = ( + await db.client.getExecutions({ + orchestratorId: newOrchestrator, + queueName: "fenced", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]; + if (!currentClaim) throw new Error("expected recovered execution to be re-claimed"); + + const staleBase = { + execution_id: executionId, + queue: "fenced", + task_key: "fenced-task", + orchestrator_id: oldClaim.locked_by, + }; + await db.client.returnExecutions({ + count: 3, + orchestratorId: oldOrchestrator, + completed: [{ ...staleBase, status: "completed" }], + failed: [{ ...staleBase, status: "failed", error: "stale" }], + released: [{ ...staleBase, status: "released", reschedule_in_ms: 0 }], + invokeChild: [], + taskKeys: new Set(["fenced-task"]), + }); + + const untouched = await db.sql< + { + completed_at: Date | null; + failed_at: Date | null; + locked_by: string; + }[] + >` + select completed_at, failed_at, locked_by + from pgconductor._private_executions + where id = ${executionId}::uuid + `; + expect(untouched[0]?.completed_at).toBeNull(); + expect(untouched[0]?.failed_at).toBeNull(); + expect(untouched[0]?.locked_by).toBe(newOrchestrator); + + await db.client.returnExecutions( + grouped({ + execution_id: currentClaim.id, + queue: currentClaim.queue, + task_key: currentClaim.task_key, + orchestrator_id: currentClaim.locked_by, + status: "completed", + }), + ); + const settled = await db.sql<{ completed_at: Date | null; locked_by: string | null }[]>` + select completed_at, locked_by + from pgconductor._private_executions + where id = ${executionId}::uuid + `; + expect(settled[0]?.completed_at).not.toBeNull(); + expect(settled[0]?.locked_by).toBeNull(); + }); +}); diff --git a/packages/pgconductor-js/tests/integration/field-selection-types.test.ts b/packages/pgconductor-js/tests/integration/field-selection-types.test.ts index 717e994..709bd42 100644 --- a/packages/pgconductor-js/tests/integration/field-selection-types.test.ts +++ b/packages/pgconductor-js/tests/integration/field-selection-types.test.ts @@ -7,6 +7,7 @@ import { defineEvent } from "../../src/event-definition"; import { TaskSchemas, EventSchemas } from "../../src/schemas"; import { TestDatabasePool } from "../fixtures/test-database"; import type { TestDatabase } from "../fixtures/test-database"; +import { Deferred } from "../../src/lib/deferred"; describe("Field Selection - Type Safety & Runtime", () => { let pool: TestDatabasePool; @@ -48,6 +49,7 @@ describe("Field Selection - Type Safety & Runtime", () => { payload: z.object({}), }); + const handled = new Deferred(); const taskFn = mock(async (event) => { // Type-level: TypeScript should know these fields exist with correct types const str: string = event.payload.stringField; @@ -70,6 +72,7 @@ describe("Field Selection - Type Safety & Runtime", () => { // Non-selected fields should not exist at runtime expect(event.payload.extraString).toBeUndefined(); expect(event.payload.extraNumber).toBeUndefined(); + handled.resolve(); }); const conductor = Conductor.create({ @@ -107,9 +110,7 @@ describe("Field Selection - Type Safety & Runtime", () => { extraNumber: 999, }); - // Wait for task to execute - await new Promise((r) => setTimeout(r, 300)); - + await handled.promise; expect(taskFn).toHaveBeenCalledTimes(1); await orchestrator.stop(); @@ -135,6 +136,7 @@ describe("Field Selection - Type Safety & Runtime", () => { payload: z.object({}), }); + const handled = new Deferred(); const taskFn = mock(async (event) => { // Type-level: Selected fields with correct camelCase const userId: string = event.payload.userId; @@ -146,6 +148,7 @@ describe("Field Selection - Type Safety & Runtime", () => { expect(event.payload.firstName).toBeUndefined(); expect(event.payload.lastName).toBeUndefined(); expect(event.payload.accountType).toBeUndefined(); + handled.resolve(); }); const conductor = Conductor.create({ @@ -177,8 +180,7 @@ describe("Field Selection - Type Safety & Runtime", () => { accountType: "premium", }); - await new Promise((r) => setTimeout(r, 300)); - + await handled.promise; expect(taskFn).toHaveBeenCalledTimes(1); await orchestrator.stop(); @@ -202,6 +204,7 @@ describe("Field Selection - Type Safety & Runtime", () => { payload: z.object({}), }); + const handled = new Deferred(); const taskFn = mock(async (event) => { // Type-level: All fields should be available const f1: string = event.payload.field1; @@ -212,6 +215,7 @@ describe("Field Selection - Type Safety & Runtime", () => { expect(event.payload.field1).toBe("value1"); expect(event.payload.field2).toBe(123); expect(event.payload.field3).toBe(true); + handled.resolve(); }); const conductor = Conductor.create({ @@ -238,8 +242,7 @@ describe("Field Selection - Type Safety & Runtime", () => { field3: true, }); - await new Promise((r) => setTimeout(r, 300)); - + await handled.promise; expect(taskFn).toHaveBeenCalledTimes(1); await orchestrator.stop(); diff --git a/packages/pgconductor-js/tests/integration/group-concurrency.test.ts b/packages/pgconductor-js/tests/integration/group-concurrency.test.ts new file mode 100644 index 0000000..9dd3022 --- /dev/null +++ b/packages/pgconductor-js/tests/integration/group-concurrency.test.ts @@ -0,0 +1,380 @@ +import { test, expect, describe, beforeAll, afterAll, afterEach } from "bun:test"; +import { z } from "zod"; +import { Conductor } from "../../src/conductor"; +import { Orchestrator } from "../../src/orchestrator"; +import { defineTask } from "../../src/task-definition"; +import { TaskSchemas } from "../../src/schemas"; +import { Deferred } from "../../src/lib/deferred"; +import { TestDatabasePool } from "../fixtures/test-database"; +import type { TestDatabase } from "../fixtures/test-database"; + +const workerConfig = { + concurrency: 10, + fetchBatchSize: 10, + flushBatchSize: 10, + pollIntervalMs: 10, + flushIntervalMs: 10, +}; + +// Group limits are intentionally soft: concurrent claim transactions may race. +// These tests use blockers and avoid asserting exact global bounds across workers. + +async function waitUntil(predicate: () => boolean, timeoutMs = 20_000): Promise { + const deadline = Date.now() + timeoutMs; + while (!predicate()) { + if (Date.now() >= deadline) { + throw new Error("Timed out waiting for condition"); + } + await new Promise((resolve) => setTimeout(resolve, 10)); + } +} + +describe("Group concurrency", () => { + let pool: TestDatabasePool; + const databases: TestDatabase[] = []; + + beforeAll(async () => { + pool = await TestDatabasePool.create(); + }, 60000); + + afterEach(async () => { + await Promise.all(databases.map((db) => db.destroy())); + databases.length = 0; + }); + + afterAll(async () => { + await pool?.destroy(); + }); + + test("serializes executions in the same group", async () => { + const db = await pool.child(); + databases.push(db); + const definition = defineTask({ + name: "same-group", + payload: z.object({ id: z.number() }), + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([definition]), + context: {}, + }); + const started: number[] = []; + const blockers = new Map>([ + [1, new Deferred()], + [2, new Deferred()], + ]); + const task = conductor.createTask( + { name: "same-group", groupConcurrency: 1 }, + { invocable: true }, + async (event) => { + started.push(event.payload.id); + const blocker = blockers.get(event.payload.id); + if (blocker) await blocker.promise; + }, + ); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [task], + defaultWorker: workerConfig, + }); + + await orchestrator.start(); + try { + await conductor.invoke({ name: "same-group" }, { id: 1 }, { group: "tenant-a" }); + await conductor.invoke({ name: "same-group" }, { id: 2 }, { group: "tenant-a" }); + await waitUntil(() => started.length === 1); + await new Promise((resolve) => setTimeout(resolve, 100)); + const first = started[0]; + expect(first === 1 || first === 2).toBe(true); + + blockers.get(first!)?.resolve(); + await waitUntil(() => started.length === 2); + expect(new Set(started)).toEqual(new Set([1, 2])); + } finally { + blockers.forEach((blocker) => blocker.resolve()); + await orchestrator.stop(); + } + }, 30000); + + test("allows different groups to run in parallel", async () => { + const db = await pool.child(); + databases.push(db); + const definition = defineTask({ + name: "different-groups", + payload: z.object({ id: z.number() }), + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([definition]), + context: {}, + }); + const started: number[] = []; + const blocker = new Deferred(); + const task = conductor.createTask( + { name: "different-groups", groupConcurrency: 1 }, + { invocable: true }, + async (event) => { + started.push(event.payload.id); + await blocker.promise; + }, + ); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [task], + defaultWorker: workerConfig, + }); + + await orchestrator.start(); + try { + await conductor.invoke({ name: "different-groups" }, { id: 1 }, { group: "tenant-a" }); + await waitUntil(() => started.includes(1)); + await conductor.invoke({ name: "different-groups" }, { id: 2 }, { group: "tenant-b" }); + await waitUntil(() => started.length === 2); + expect(new Set(started)).toEqual(new Set([1, 2])); + } finally { + blocker.resolve(); + await orchestrator.stop(); + } + }, 30000); + + test("does not apply the group limit to ungrouped executions", async () => { + const db = await pool.child(); + databases.push(db); + const definition = defineTask({ + name: "ungrouped", + payload: z.object({ id: z.number() }), + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([definition]), + context: {}, + }); + const started: number[] = []; + const blocker = new Deferred(); + const task = conductor.createTask( + { name: "ungrouped", groupConcurrency: 1 }, + { invocable: true }, + async (event) => { + started.push(event.payload.id); + await blocker.promise; + }, + ); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [task], + defaultWorker: workerConfig, + }); + + await orchestrator.start(); + try { + await conductor.invoke({ name: "ungrouped" }, { id: 1 }, { group: "tenant-a" }); + await conductor.invoke({ name: "ungrouped" }, { id: 2 }); + await waitUntil(() => started.length === 2); + expect(new Set(started)).toEqual(new Set([1, 2])); + } finally { + blocker.resolve(); + await orchestrator.stop(); + } + }, 30000); + + test("composes task and group concurrency limits", async () => { + const db = await pool.child(); + databases.push(db); + const definition = defineTask({ + name: "composed-limits", + payload: z.object({ id: z.number() }), + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([definition]), + context: {}, + }); + const started: number[] = []; + const blockers = new Map>([ + [1, new Deferred()], + [2, new Deferred()], + [3, new Deferred()], + ]); + const task = conductor.createTask( + { name: "composed-limits", concurrency: 2, groupConcurrency: 1 }, + { invocable: true }, + async (event) => { + started.push(event.payload.id); + const blocker = blockers.get(event.payload.id); + if (blocker) await blocker.promise; + }, + ); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [task], + defaultWorker: workerConfig, + }); + + await orchestrator.start(); + try { + await conductor.invoke({ name: "composed-limits" }, { id: 1 }, { group: "tenant-a" }); + await conductor.invoke({ name: "composed-limits" }, { id: 3 }, { group: "tenant-b" }); + await conductor.invoke({ name: "composed-limits" }, { id: 2 }, { group: "tenant-a" }); + await waitUntil(() => started.includes(3) && started.some((id) => id === 1 || id === 2)); + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(started).toContain(3); + expect(started.filter((id) => id === 1 || id === 2)).toHaveLength(1); + + blockers.forEach((blocker) => blocker.resolve()); + await waitUntil(() => started.length === 3); + } finally { + blockers.forEach((blocker) => blocker.resolve()); + await orchestrator.stop(); + } + }, 30000); + + test("does not share a group across tasks or queues", async () => { + const db = await pool.child(); + databases.push(db); + const taskADefinition = defineTask({ name: "scope-a" }); + const taskBDefinition = defineTask({ name: "scope-b" }); + const queueTaskDefinition = defineTask({ name: "scope-queue", queue: "other" }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([taskADefinition, taskBDefinition, queueTaskDefinition]), + context: {}, + }); + const started = new Set(); + const blocker = new Deferred(); + const taskA = conductor.createTask( + { name: "scope-a", groupConcurrency: 1 }, + { invocable: true }, + async () => { + started.add("task-a"); + await blocker.promise; + }, + ); + const taskB = conductor.createTask( + { name: "scope-b", groupConcurrency: 1 }, + { invocable: true }, + async () => { + started.add("task-b"); + await blocker.promise; + }, + ); + const queueTask = conductor.createTask( + { name: "scope-queue", queue: "other", groupConcurrency: 1 }, + { invocable: true }, + async () => { + started.add("queue"); + await blocker.promise; + }, + ); + const otherWorker = conductor.createWorker({ + queue: "other", + tasks: [queueTask], + config: workerConfig, + }); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [taskA, taskB], + workers: [otherWorker], + defaultWorker: workerConfig, + }); + + await orchestrator.start(); + try { + await conductor.invoke({ name: "scope-a" }, {}, { group: "shared" }); + await conductor.invoke({ name: "scope-b" }, {}, { group: "shared" }); + await conductor.invoke({ name: "scope-queue", queue: "other" }, {}, { group: "shared" }); + await waitUntil(() => started.size === 3); + expect(started).toEqual(new Set(["task-a", "task-b", "queue"])); + } finally { + blocker.resolve(); + await orchestrator.stop(); + } + }, 30000); + + test("propagates group metadata through batch invocation", async () => { + const db = await pool.child(); + databases.push(db); + const definition = defineTask({ + name: "batch-group", + payload: z.object({ id: z.number() }), + }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([definition]), + context: {}, + }); + const started: number[] = []; + const blockers = new Map>([ + [1, new Deferred()], + [2, new Deferred()], + ]); + const task = conductor.createTask( + { name: "batch-group", groupConcurrency: 1 }, + { invocable: true }, + async (event) => { + started.push(event.payload.id); + const blocker = blockers.get(event.payload.id); + if (blocker) await blocker.promise; + }, + ); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [task], + defaultWorker: workerConfig, + }); + + await orchestrator.start(); + try { + await conductor.invoke({ name: "batch-group" }, [ + { payload: { id: 1 }, group: "batch-tenant" }, + { payload: { id: 2 }, group: "batch-tenant" }, + ]); + const rows = await db.sql<{ group: string | null }[]>` + select "group" + from pgconductor._private_executions + where task_key = 'batch-group' + order by created_at asc, id asc + `; + expect(rows.map((row) => row.group)).toEqual(["batch-tenant", "batch-tenant"]); + + await waitUntil(() => started.length === 1); + await new Promise((resolve) => setTimeout(resolve, 100)); + const first = started[0]; + expect(first === 1 || first === 2).toBe(true); + blockers.get(first!)?.resolve(); + await waitUntil(() => started.length === 2); + expect(new Set(started)).toEqual(new Set([1, 2])); + } finally { + blockers.forEach((blocker) => blocker.resolve()); + await orchestrator.stop(); + } + }, 30000); + + test("rejects invalid task and group concurrency values", async () => { + const db = await pool.child(); + databases.push(db); + const definition = defineTask({ name: "validation" }); + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([definition]), + context: {}, + }); + const invalidValues = [0, -1, 1.5]; + + for (const value of invalidValues) { + expect(() => + conductor.createTask( + { name: "validation", concurrency: value }, + { invocable: true }, + async () => {}, + ), + ).toThrow("concurrency must be a positive integer"); + expect(() => + conductor.createTask( + { name: "validation", groupConcurrency: value }, + { invocable: true }, + async () => {}, + ), + ).toThrow("groupConcurrency must be a positive integer"); + } + }, 30000); +}); diff --git a/packages/pgconductor-js/tests/integration/invoke-support.test.ts b/packages/pgconductor-js/tests/integration/invoke-support.test.ts index 47d9b3f..45add61 100644 --- a/packages/pgconductor-js/tests/integration/invoke-support.test.ts +++ b/packages/pgconductor-js/tests/integration/invoke-support.test.ts @@ -6,6 +6,16 @@ import { defineTask } from "../../src/task-definition"; import { TestDatabasePool } from "../fixtures/test-database"; import type { TestDatabase } from "../fixtures/test-database"; import { TaskSchemas } from "../../src/schemas"; +import { Deferred } from "../../src/lib/deferred"; + +async function waitForCondition(check: () => Promise, timeoutMs = 20_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (await check()) return; + await new Promise((resolve) => setTimeout(resolve, 25)); + } + throw new Error(`condition was not met within ${timeoutMs}ms`); +} describe("Invoke Support", () => { let pool: TestDatabasePool; @@ -41,6 +51,7 @@ describe("Invoke Support", () => { }); const childFn = mock((n: number) => n * 2); + const childCompleted = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -54,6 +65,7 @@ describe("Invoke Support", () => { async (event, _ctx) => { if (event.name === "pgconductor.invoke") { const result = childFn(event.payload.input); + childCompleted.resolve(); return { output: result }; } throw new Error("Unexpected event type"); @@ -85,9 +97,7 @@ describe("Invoke Support", () => { await orchestrator.start(); await conductor.invoke({ name: "parent-task" }, { value: 5 }); - - await new Promise((r) => setTimeout(r, 300)); - + await childCompleted.promise; await orchestrator.stop(); expect(childFn).toHaveBeenCalledTimes(1); @@ -156,15 +166,24 @@ describe("Invoke Support", () => { await orchestrator.start(); await conductor.invoke({ name: "timeout-parent" }, {}); - - await new Promise((r) => setTimeout(r, 300)); + await waitForCondition(async () => { + const [child] = await db.sql<{ released: boolean; sleeping: boolean }[]>` + select + locked_by is null as released, + exists ( + select 1 from pgconductor._private_steps s + where s.execution_id = e.id and s.key = 'long-sleep' + ) as sleeping + from pgconductor._private_executions e + where task_key = 'slow-child' + `; + return child?.released === true && child.sleeping; + }); // Advance time past timeout (1 second) but before sleep completes (5 seconds) const afterTimeout = new Date(startTime.getTime() + 1500); await db.client.setFakeTime({ date: afterTimeout }); - - await new Promise((r) => setTimeout(r, 300)); - + await waitForCondition(async () => parentError.mock.calls.length === 1); await orchestrator.stop(); await db.client.clearFakeTime(); @@ -172,7 +191,7 @@ describe("Invoke Support", () => { expect(parentError).toHaveBeenCalledTimes(1); const errorMsg = parentError.mock.results[0]?.value; expect(errorMsg).toContain("timed out after 1000ms"); - }, 5000); + }, 30000); test("invoke() caches child result on retry", async () => { const db = await pool.child(); @@ -191,6 +210,7 @@ describe("Invoke Support", () => { }); const childFn = mock((n: number) => n * 3); + const parentCompleted = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -228,6 +248,7 @@ describe("Invoke Support", () => { throw new Error("First attempt fails"); } + parentCompleted.resolve(); return { result: childResult.output }; } throw new Error("Unexpected event type"); @@ -243,9 +264,7 @@ describe("Invoke Support", () => { await orchestrator.start(); await conductor.invoke({ name: "retry-parent" }, { value: 7 }); - - await new Promise((r) => setTimeout(r, 500)); - + await parentCompleted.promise; await orchestrator.stop(); expect(childFn).toHaveBeenCalledTimes(1); @@ -269,6 +288,7 @@ describe("Invoke Support", () => { }); const childFn = mock(() => "completed"); + const childCompleted = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -282,6 +302,7 @@ describe("Invoke Support", () => { async (_event, ctx) => { await ctx.sleep("moderate-sleep", 2000); const result = childFn(); + childCompleted.resolve(); return { result }; }, ); @@ -304,9 +325,7 @@ describe("Invoke Support", () => { await orchestrator.start(); await conductor.invoke({ name: "patient-parent" }, {}); - - await new Promise((r) => setTimeout(r, 2500)); - + await childCompleted.promise; await orchestrator.stop(); expect(childFn).toHaveBeenCalledTimes(1); @@ -372,17 +391,31 @@ describe("Invoke Support", () => { await conductor.invoke({ name: "cascade-parent" }, {}); - // Wait for first execution cycle: parent runs → invokes child → child fails - await new Promise((r) => setTimeout(r, 500)); + await waitForCondition(async () => { + const [child] = await db.sql<{ attempts: number; released: boolean }[]>` + select attempts, locked_by is null as released + from pgconductor._private_executions + where task_key = 'failing-child' + `; + return child?.attempts === 1 && child.released; + }); expect(childFn).toHaveBeenCalledTimes(1); - // Advance fake time past first backoff (15 seconds) - const afterFirstBackoff = new Date(startTime.getTime() + 15000); - await db.client.setFakeTime({ date: afterFirstBackoff }); - - // Wait for second execution cycle: child retries and fails permanently - await new Promise((r) => setTimeout(r, 500)); + const [retry] = await db.sql<{ run_at: Date }[]>` + select run_at from pgconductor._private_executions + where task_key = 'failing-child' + `; + if (!retry) throw new Error("expected persisted child retry"); + await db.client.setFakeTime({ date: new Date(retry.run_at.getTime() + 1) }); + await waitForCondition(async () => { + const [parent] = await db.sql<{ failed: boolean }[]>` + select failed_at is not null as failed + from pgconductor._private_executions + where task_key = 'cascade-parent' + `; + return parent?.failed === true; + }); await orchestrator.stop(); await db.client.clearFakeTime(); @@ -418,6 +451,7 @@ describe("Invoke Support", () => { }); const childFn = mock((n: number) => n * 2); + const childCalled = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -452,6 +486,7 @@ describe("Invoke Support", () => { async (event, _ctx) => { if (event.name === "pgconductor.invoke") { const result = childFn(event.payload.input); + childCalled.resolve(); return { output: result }; } throw new Error("Unexpected event type"); @@ -472,9 +507,7 @@ describe("Invoke Support", () => { await orchestrator.start(); await conductor.invoke({ queue: "parent-queue", name: "parent-task" }, { value: 5 }); - - await new Promise((r) => setTimeout(r, 4000)); - + await childCalled.promise; await orchestrator.stop(); expect(childFn).toHaveBeenCalledTimes(1); @@ -493,6 +526,7 @@ describe("Invoke Support", () => { const childDefinition = defineTask({ name: "slow-child-2", + queue: "pending-child-queue", payload: z.object({}), returns: z.object({ completed: z.boolean() }), }); @@ -503,32 +537,34 @@ describe("Invoke Support", () => { context: {}, }); - // Child that takes 5 seconds - const slowChildTask = conductor.createTask( - { name: "slow-child-2" }, - { invocable: true }, - async (_event, ctx) => { - await ctx.sleep("long-sleep", 5000); - return { completed: true }; - }, - ); - // Parent with 1 second timeout - let error throw const timeoutParentTask = conductor.createTask( { name: "timeout-parent-2" }, { invocable: true }, async (_event, ctx) => { - await ctx.invoke("invoke-slow", { name: "slow-child-2" }, {}, 1000); + await ctx.invoke( + "invoke-slow", + { name: "slow-child-2", queue: "pending-child-queue" }, + {}, + 1000, + ); return { success: true }; }, ); + await conductor.ensureInstalled(); + await db.sql` + insert into pgconductor._private_queues (name) + values ('pending-child-queue') + on conflict (name) do nothing + `; + const startTime = new Date("2024-01-01T00:00:00Z"); await db.client.setFakeTime({ date: startTime }); const orchestrator = Orchestrator.create({ conductor, - tasks: [timeoutParentTask, slowChildTask], + tasks: [timeoutParentTask], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50, @@ -539,16 +575,29 @@ describe("Invoke Support", () => { await conductor.invoke({ name: "timeout-parent-2" }, {}); - // Wait for parent to invoke child - await new Promise((r) => setTimeout(r, 200)); + // Wait for the parent to persist the unclaimed child execution. + await waitForCondition(async () => { + const [child] = await db.sql<{ pending: boolean }[]>` + select locked_by is null as pending + from pgconductor._private_executions + where task_key = 'slow-child-2' + and queue = 'pending-child-queue' + `; + return child?.pending === true; + }); - // Advance time past timeout (child still pending due to 50ms intervals) + // Advance time past timeout while the child is still pending. const afterTimeout = new Date(startTime.getTime() + 1500); await db.client.setFakeTime({ date: afterTimeout }); - - // Wait for timeout to be processed - await new Promise((r) => setTimeout(r, 500)); - + await waitForCondition(async () => { + const [child] = await db.sql<{ failed: boolean }[]>` + select failed_at is not null as failed + from pgconductor._private_executions + where task_key = 'slow-child-2' + and queue = 'pending-child-queue' + `; + return child?.failed === true; + }); await orchestrator.stop(); await db.client.clearFakeTime(); @@ -563,6 +612,7 @@ describe("Invoke Support", () => { select cancelled, failed_at, last_error from pgconductor._private_executions where task_key = 'slow-child-2' + and queue = 'pending-child-queue' `; expect(children.length).toBe(1); @@ -668,8 +718,14 @@ describe("Invoke Support", () => { { dedupe_key: "locked-key" }, ); - // Wait for it to be locked - await new Promise((r) => setTimeout(r, 100)); + await waitForCondition(async () => { + const [execution] = await db.sql<{ locked: boolean }[]>` + select locked_by is not null as locked + from pgconductor._private_executions + where id = ${id1} + `; + return execution?.locked === true; + }); // Second invocation while first is locked - should create NEW execution const id2 = await conductor.invoke( @@ -681,9 +737,14 @@ describe("Invoke Support", () => { // Should be different IDs expect(id2).not.toBe(id1); - // Wait for both executions to complete and flush (100ms + 500ms + 500ms + 50ms + margin) - await new Promise((r) => setTimeout(r, 1200)); - + await waitForCondition(async () => { + const [execution] = await db.sql<{ completed: boolean }[]>` + select completed_at is not null as completed + from pgconductor._private_executions + where id = ${id2} + `; + return execution?.completed === true; + }); await orchestrator.stop(); // First execution should be marked as failed (superseded) @@ -755,6 +816,9 @@ describe("Invoke Support", () => { }, ); + const startTime = new Date("2024-01-01T00:00:00Z"); + await db.client.setFakeTime({ date: startTime }); + const orchestrator = Orchestrator.create({ defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, conductor, @@ -764,7 +828,7 @@ describe("Invoke Support", () => { await orchestrator.start(); // Rapid invocations with same dedupe_key and delayed run_at - const futureTime = new Date(Date.now() + 1000); + const futureTime = new Date(startTime.getTime() + 1000); const id1 = await conductor.invoke( { name: "debounce-task" }, @@ -772,16 +836,12 @@ describe("Invoke Support", () => { { dedupe_key: "debounce-1", run_at: futureTime }, ); - await new Promise((r) => setTimeout(r, 50)); - const id2 = await conductor.invoke( { name: "debounce-task" }, { value: 2 }, { dedupe_key: "debounce-1", run_at: futureTime }, ); - await new Promise((r) => setTimeout(r, 50)); - const id3 = await conductor.invoke( { name: "debounce-task" }, { value: 3 }, @@ -800,10 +860,10 @@ describe("Invoke Support", () => { `; expect(executions[0]?.count).toBe(1); - // Wait for execution - await new Promise((r) => setTimeout(r, 1500)); - + await db.client.setFakeTime({ date: futureTime }); + await waitForCondition(async () => executionCount === 1); await orchestrator.stop(); + await db.client.clearFakeTime(); // Should have executed only once with the last value expect(executionCount).toBe(1); diff --git a/packages/pgconductor-js/tests/integration/maintenance-task.test.ts b/packages/pgconductor-js/tests/integration/maintenance-task.test.ts index e85fe8c..b26e718 100644 --- a/packages/pgconductor-js/tests/integration/maintenance-task.test.ts +++ b/packages/pgconductor-js/tests/integration/maintenance-task.test.ts @@ -7,6 +7,7 @@ import { TestDatabasePool } from "../fixtures/test-database"; import type { TestDatabase } from "../fixtures/test-database"; import crypto from "crypto"; import { TaskSchemas } from "../../src/schemas"; +import { createMaintenanceTask } from "../../src/maintenance-task"; function hashToJitter(str: string): number { const hash = crypto.createHash("sha256").update(str).digest(); @@ -478,4 +479,87 @@ describe("Maintenance Task", () => { `; expect(exec.length).toBe(1); }, 30000); + + test("removes old processed events and cascades deliveries only", async () => { + const db = await pool.child(); + databases.push(db); + await Conductor.create({ sql: db.sql, context: {} }).ensureInstalled(); + const base = new Date("2025-01-20T00:00:00Z"); + + await db.sql` + insert into pgconductor._private_tasks (key, queue) + values ('maintenance-event-task', 'default') + `; + const [subscription] = await db.sql<{ id: string }[]>` + insert into pgconductor._private_event_subscriptions (task_key, queue, event_key) + values ('maintenance-event-task', 'default', 'maintenance.event') + returning id + `; + await db.client.setFakeTime({ date: new Date(base.getTime() - 8 * 24 * 60 * 60 * 1000) }); + const [oldEvent] = await db.sql<{ id: string }[]>` + insert into pgconductor._private_custom_events (event_key, payload) + values ('maintenance.event', '{}') + returning id + `; + await db.client.processEvents({ batchSize: 10 }); + const [waitingExecution] = await db.sql<{ id: string }[]>` + insert into pgconductor._private_executions + (task_key, queue, payload, run_at, waiting_step_key) + values ('maintenance-event-task', 'default', '{}', 'infinity'::timestamptz, 'retained-step') + returning id + `; + await db.sql` + insert into pgconductor._private_event_subscriptions + (task_key, queue, event_key, kind, execution_id, step_key, wait_after_event_position) + values ('maintenance-event-task', 'default', 'maintenance.event', 'execution_wait', + ${waitingExecution!.id}::uuid, 'retained-step', 0) + `; + + await db.client.setFakeTime({ date: base }); + const [currentEvent] = await db.sql<{ id: string }[]>` + insert into pgconductor._private_custom_events (event_key, payload) + values ('maintenance.event', '{}') + returning id + `; + await db.client.processEvents({ batchSize: 10 }); + const [pendingEvent] = await db.sql<{ id: string }[]>` + insert into pgconductor._private_custom_events (event_key, payload) + values ('maintenance.pending', '{}') + returning id + `; + const maintenance = createMaintenanceTask("default"); + await maintenance.execute({ name: "pgconductor.maintenance" }, { + db: db.client, + tasks: new Map(), + signal: new AbortController().signal, + } as never); + + const remaining = await db.sql<{ id: string; deliveries: string }[]>` + select e.id, count(d.subscription_id)::text as deliveries + from pgconductor._private_custom_events e + left join pgconductor._private_event_deliveries d + on d.event_created_at = e.created_at and d.event_id = e.id + where e.id = any(${[oldEvent!.id, currentEvent!.id, pendingEvent!.id]}::uuid[]) + group by e.id + `; + const remainingById = new Map(remaining.map((row) => [row.id, row.deliveries])); + expect(remainingById).toEqual( + new Map([ + [oldEvent!.id, "1"], + [currentEvent!.id, "1"], + [pendingEvent!.id, "0"], + ]), + ); + expect(subscription?.id).toBeString(); + await db.sql` + delete from pgconductor._private_event_subscriptions + where execution_id = ${waitingExecution!.id}::uuid + `; + expect(await db.client.removeProcessedEvents({ before: base, batchSize: 10 })).toBe(false); + const oldRemaining = await db.sql` + select id from pgconductor._private_custom_events where id = ${oldEvent!.id}::uuid + `; + expect(oldRemaining).toHaveLength(0); + await db.client.clearFakeTime(); + }, 30000); }); diff --git a/packages/pgconductor-js/tests/integration/step-support.test.ts b/packages/pgconductor-js/tests/integration/step-support.test.ts index 32e1638..b43fcb6 100644 --- a/packages/pgconductor-js/tests/integration/step-support.test.ts +++ b/packages/pgconductor-js/tests/integration/step-support.test.ts @@ -6,6 +6,7 @@ import { defineTask } from "../../src/task-definition"; import { TestDatabasePool } from "../fixtures/test-database"; import type { TestDatabase } from "../fixtures/test-database"; import { TaskSchemas } from "../../src/schemas"; +import { Deferred } from "../../src/lib/deferred"; describe("Step Support", () => { let pool: TestDatabasePool; @@ -35,6 +36,7 @@ describe("Step Support", () => { }); const expensiveFn = mock((n: number) => n * 2); + const taskCompleted = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -51,6 +53,7 @@ describe("Step Support", () => { return expensiveFn(event.payload.value); }); + taskCompleted.resolve(); return { result }; } throw new Error("Unexpected event type"); @@ -66,9 +69,7 @@ describe("Step Support", () => { await orchestrator.start(); await conductor.invoke({ name: "step-task" }, { value: 5 }); - - await new Promise((r) => setTimeout(r, 200)); - + await taskCompleted.promise; await orchestrator.stop(); // Expensive function should only be called once @@ -87,6 +88,8 @@ describe("Step Support", () => { }); const executionSteps = mock((step: string) => step); + const sleepStarted = new Deferred(); + const sleepFinished = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -100,8 +103,10 @@ describe("Step Support", () => { async (event, ctx) => { if (event.name === "pgconductor.invoke") { executionSteps("before-sleep"); + sleepStarted.resolve(); await ctx.sleep("wait", event.payload.delay); executionSteps("after-sleep"); + sleepFinished.resolve(); return { completed: true }; } throw new Error("Unexpected event type"); @@ -117,18 +122,12 @@ describe("Step Support", () => { await orchestrator.start(); await conductor.invoke({ name: "sleep-task" }, { delay: 2000 }); + await sleepStarted.promise; - // Wait for initial execution (should hit sleep and release) - await new Promise((r) => setTimeout(r, 200)); - - // At this point, should have seen "before-sleep" but not "after-sleep" expect(executionSteps).toHaveBeenCalledWith("before-sleep"); expect(executionSteps).not.toHaveBeenCalledWith("after-sleep"); - // Wait for sleep to complete and task to resume (2s sleep + buffer) - await new Promise((r) => setTimeout(r, 2200)); - - // Now should see "after-sleep" + await sleepFinished.promise; expect(executionSteps).toHaveBeenCalledWith("after-sleep"); await orchestrator.stop(); @@ -145,6 +144,7 @@ describe("Step Support", () => { }); const processedItems = mock((item: number) => item); + const checkpointCompleted = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -163,6 +163,7 @@ describe("Step Support", () => { // Simulate some work await new Promise((r) => setTimeout(r, 50)); } + checkpointCompleted.resolve(); return { processed: event.payload.items }; } throw new Error("Unexpected event type"); @@ -178,10 +179,7 @@ describe("Step Support", () => { await orchestrator.start(); await conductor.invoke({ name: "checkpoint-task" }, { items: 5 }); - - // Wait for task to complete - await new Promise((r) => setTimeout(r, 300)); - + await checkpointCompleted.promise; await orchestrator.stop(); // Should have processed all items @@ -201,6 +199,7 @@ describe("Step Support", () => { const step1Fn = mock((n: number) => n + 1); const step2Fn = mock((n: number) => n * 2); const step3Fn = mock((n: number) => n - 3); + const stepsCompleted = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -216,6 +215,7 @@ describe("Step Support", () => { const a = await ctx.step("step1", () => step1Fn(event.payload.x)); const b = await ctx.step("step2", () => step2Fn(a)); const c = await ctx.step("step3", () => step3Fn(b)); + stepsCompleted.resolve(); return { result: c }; } throw new Error("Unexpected event type"); @@ -231,9 +231,7 @@ describe("Step Support", () => { await orchestrator.start(); await conductor.invoke({ name: "multi-step-task" }, { x: 5 }); - - await new Promise((r) => setTimeout(r, 200)); - + await stepsCompleted.promise; await orchestrator.stop(); // All steps should execute once @@ -259,6 +257,7 @@ describe("Step Support", () => { let transformedValue: string[] | undefined; let countValue: number | undefined; + const completed = new Deferred(); const conductor = Conductor.create({ sql: db.sql, @@ -284,6 +283,7 @@ describe("Step Support", () => { return transformed.length; }); + completed.resolve(); return { count }; } throw new Error("Unexpected event type"); @@ -299,9 +299,7 @@ describe("Step Support", () => { await orchestrator.start(); await conductor.invoke({ name: "step-unwrap-task" }, { items: ["apple", "banana", "cherry"] }); - - await new Promise((r) => setTimeout(r, 300)); - + await completed.promise; await orchestrator.stop(); expect(transformedValue).toEqual(["APPLE", "BANANA", "CHERRY"]); diff --git a/packages/pgconductor-js/tests/integration/subscription-lifecycle.test.ts b/packages/pgconductor-js/tests/integration/subscription-lifecycle.test.ts index a535272..d5708d2 100644 --- a/packages/pgconductor-js/tests/integration/subscription-lifecycle.test.ts +++ b/packages/pgconductor-js/tests/integration/subscription-lifecycle.test.ts @@ -13,12 +13,15 @@ import type { Database } from "../database.types"; describe("Event Subscription Lifecycle", () => { let pool: TestDatabasePool; const databases: TestDatabase[] = []; + const orchestrators: Orchestrator[] = []; beforeAll(async () => { pool = await TestDatabasePool.create(); }, 60000); afterEach(async () => { + await Promise.all(orchestrators.map((orchestrator) => orchestrator.stop())); + orchestrators.length = 0; await Promise.all(databases.map((db) => db.destroy())); databases.length = 0; }); @@ -27,7 +30,16 @@ describe("Event Subscription Lifecycle", () => { await pool?.destroy(); }); - test("custom event triggers are created on orchestrator start", async () => { + async function waitUntil(check: () => Promise, timeout = 5000): Promise { + const deadline = Date.now() + timeout; + while (Date.now() < deadline) { + if (await check()) return; + await Bun.sleep(10); + } + throw new Error("condition was not met before timeout"); + } + + test("custom event subscriptions are persisted and processed asynchronously", async () => { const db = await pool.child(); databases.push(db); @@ -48,10 +60,11 @@ describe("Event Subscription Lifecycle", () => { context: {}, }); + const taskFn = mock(async () => {}); const task = conductor.createTask( { name: "on-user-created" }, { event: "user.created" }, - mock(async () => {}), + taskFn, ); const orchestrator = Orchestrator.create({ @@ -59,12 +72,13 @@ describe("Event Subscription Lifecycle", () => { tasks: [task], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator); await orchestrator.start(); - // Check subscription was created - const [sub] = await db.sql<[{ event_key: string; task_key: string }]>` - select event_key, task_key + // Check the persistent subscription was created + const [sub] = await db.sql<[{ id: string; event_key: string; task_key: string }]>` + select id, event_key, task_key from pgconductor._private_event_subscriptions where event_key = 'user.created' `; @@ -73,19 +87,24 @@ describe("Event Subscription Lifecycle", () => { expect(sub.event_key).toBe("user.created"); expect(sub.task_key).toBe("on-user-created"); - // Check trigger function was created - const [trigger] = await db.sql<[{ exists: boolean }]>` - select exists( - select 1 - from pg_trigger - where tgname = 'pgconductor_custom_event' - and tgrelid = 'pgconductor._private_custom_events'::regclass - ) + const [compiledFilters] = await db.sql<[{ count: string }]>` + select count(*)::text as count + from pgconductor._private_event_subscription_filters + where subscription_id = ${sub.id} `; + expect(compiledFilters.count).toBe("0"); - expect(trigger.exists).toBe(true); + await conductor.emit("user.created", { userId: "user-123" }); + await waitUntil(async () => taskFn.mock.calls.length === 1); - await orchestrator.stop(); + const [processedEvent] = await db.sql<[{ processed_at: Date | null }]>` + select processed_at + from pgconductor._private_custom_events + where event_key = 'user.created' + order by created_at desc + limit 1 + `; + expect(processedEvent.processed_at).not.toBeNull(); }, 30000); test("database triggers are created on orchestrator start", async () => { @@ -124,6 +143,7 @@ describe("Event Subscription Lifecycle", () => { tasks: [task], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator); await orchestrator.start(); @@ -154,13 +174,14 @@ describe("Event Subscription Lifecycle", () => { await orchestrator.stop(); }, 30000); - test("stopping orchestrator does not remove triggers", async () => { + test("custom event subscriptions and compiled filters persist after stop", async () => { const db = await pool.child(); databases.push(db); const userCreated = defineEvent({ name: "user.created.persistent", payload: z.object({ userId: z.string() }), + filterable: ["userId"], }); const taskDef = defineTask({ @@ -177,7 +198,7 @@ describe("Event Subscription Lifecycle", () => { const task = conductor.createTask( { name: "on-user-persistent" }, - { event: "user.created.persistent" }, + { event: "user.created.persistent", filter: { userId: ["user-123"] } }, mock(async () => {}), ); @@ -186,45 +207,40 @@ describe("Event Subscription Lifecycle", () => { tasks: [task], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator); await orchestrator.start(); - // Verify trigger exists - const [beforeStop] = await db.sql<[{ exists: boolean }]>` - select exists( + const [beforeStop] = await db.sql<[{ id: string; exists: boolean }]>` + select id, exists( select 1 - from pg_trigger - where tgname = 'pgconductor_custom_event' - and tgrelid = 'pgconductor._private_custom_events'::regclass - ) + from pgconductor._private_event_subscription_filters + where subscription_id = pgconductor._private_event_subscriptions.id + ) as exists + from pgconductor._private_event_subscriptions + where event_key = 'user.created.persistent' `; + expect(beforeStop).toBeTruthy(); expect(beforeStop.exists).toBe(true); await orchestrator.stop(); - // Verify trigger still exists after stop - const [afterStop] = await db.sql<[{ exists: boolean }]>` + const [afterStop] = await db.sql<[{ exists: boolean; filter_count: string }]>` select exists( - select 1 - from pg_trigger - where tgname = 'pgconductor_custom_event' - and tgrelid = 'pgconductor._private_custom_events'::regclass - ) + select 1 from pgconductor._private_event_subscriptions + where id = ${beforeStop.id} + ) as exists, + ( + select count(*)::text + from pgconductor._private_event_subscription_filters + where subscription_id = ${beforeStop.id} + ) as filter_count `; expect(afterStop.exists).toBe(true); - - // Verify subscription still exists - const [sub] = await db.sql<[{ exists: boolean }]>` - select exists( - select 1 - from pgconductor._private_event_subscriptions - where event_key = 'user.created.persistent' - ) - `; - expect(sub.exists).toBe(true); + expect(afterStop.filter_count).toBe("1"); }, 30000); - test("triggers are recreated when subscriptions change", async () => { + test("custom event subscriptions are replaced when their configuration changes", async () => { const db = await pool.child(); databases.push(db); @@ -262,6 +278,7 @@ describe("Event Subscription Lifecycle", () => { tasks: [task1], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator1); await orchestrator1.start(); @@ -294,6 +311,7 @@ describe("Event Subscription Lifecycle", () => { ], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator2); await orchestrator2.start(); @@ -352,6 +370,7 @@ describe("Event Subscription Lifecycle", () => { tasks: [task1, task2], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator); await orchestrator.start(); @@ -409,6 +428,7 @@ describe("Event Subscription Lifecycle", () => { tasks: [task1], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator1); await orchestrator1.start(); @@ -447,6 +467,7 @@ describe("Event Subscription Lifecycle", () => { tasks: [task2], defaultWorker: { pollIntervalMs: 50, flushIntervalMs: 50 }, }); + orchestrators.push(orchestrator2); await orchestrator2.start(); diff --git a/packages/pgconductor-js/tests/integration/wait-for-event.test.ts b/packages/pgconductor-js/tests/integration/wait-for-event.test.ts new file mode 100644 index 0000000..93734b4 --- /dev/null +++ b/packages/pgconductor-js/tests/integration/wait-for-event.test.ts @@ -0,0 +1,425 @@ +import { afterAll, afterEach, beforeAll, describe, expect, test } from "bun:test"; +import { z } from "zod"; +import { Conductor } from "../../src/conductor"; +import { Orchestrator } from "../../src/orchestrator"; +import { defineEvent } from "../../src/event-definition"; +import { defineTask } from "../../src/task-definition"; +import { EventSchemas, TaskSchemas } from "../../src/schemas"; +import { WaitForEventTimeoutError } from "../../src/index"; +import { TestDatabasePool, type TestDatabase } from "../fixtures/test-database"; + +const event = defineEvent({ + name: "wait.order", + payload: z.object({ id: z.string(), kind: z.enum(["match", "other"]) }), + filterable: ["kind"], +}); +const taskDefinition = defineTask({ name: "wait.task", payload: z.object({ id: z.string() }) }); + +type Handler = (id: string, ctx: any) => Promise; + +async function until(check: () => Promise, timeout = 20_000) { + const end = Date.now() + timeout; + while (Date.now() < end) { + if (await check()) return; + await Bun.sleep(10); + } + throw new Error("condition was not met"); +} + +async function setup(db: TestDatabase, fn: Handler, orchestrators: Orchestrator[] = []) { + const conductor = Conductor.create({ + sql: db.sql, + tasks: TaskSchemas.fromSchema([taskDefinition]), + events: EventSchemas.fromSchema([event]), + context: {}, + }); + const task = conductor.createTask( + { name: "wait.task" }, + { invocable: true }, + async (taskEvent, ctx) => fn(String(taskEvent.payload?.id), ctx), + ); + const orchestrator = Orchestrator.create({ + conductor, + tasks: [task], + defaultWorker: { pollIntervalMs: 10, flushIntervalMs: 10 }, + }); + orchestrators.push(orchestrator); + await orchestrator.start(); + return { conductor, orchestrator }; +} + +describe.serial("waitForEvent", () => { + let pool: TestDatabasePool; + const databases: TestDatabase[] = []; + const orchestrators: Orchestrator[] = []; + + beforeAll(async () => { + pool = await TestDatabasePool.create(); + }, 60000); + + afterEach(async () => { + // Stop workers before closing their database connections. In particular, a + // poll or event processor can otherwise be between two database queries. + await Promise.allSettled(orchestrators.map((orchestrator) => orchestrator.stop())); + orchestrators.length = 0; + await Promise.all(databases.map((database) => database.destroy())); + databases.length = 0; + }); + + afterAll(async () => { + await pool?.destroy(); + }); + + async function db() { + const database = await pool.child(); + databases.push(database); + await Conductor.create({ sql: database.sql, context: {} }).ensureInstalled(); + return database; + } + + async function waiting(database: TestDatabase, count = 1) { + await until( + async () => + Number( + ( + await database.sql<{ n: number }[]>` + select count(*)::int as n + from pgconductor._private_event_subscriptions + where kind = 'execution_wait' + ` + )[0]?.n ?? 0, + ) === count, + ); + } + + test.serial( + "matches filters, fans out, and replays a step without another subscription", + async () => { + const database = await db(); + const entries: string[] = []; + const { conductor } = await setup( + database, + async (id, ctx) => { + entries.push(`entered:${id}`); + const result = await ctx.waitForEvent(`order:${id}`, { + event, + filter: { kind: ["match"] }, + }); + entries.push(`result:${result.payload.id}`); + const replay = await ctx.waitForEvent(`order:${id}`, { event }); + entries.push(`replay:${replay.payload.id}`); + }, + orchestrators, + ); + + const one = await conductor.invoke({ name: "wait.task" }, { id: "one" }); + const two = await conductor.invoke({ name: "wait.task" }, { id: "two" }); + await waiting(database, 2); + await conductor.emit("wait.order", { id: "no", kind: "other" }); + await Bun.sleep(100); + expect(entries).toEqual(["entered:one", "entered:two"]); + await conductor.emit("wait.order", { id: "yes", kind: "match" }); + await until(async () => entries.filter((entry) => entry.startsWith("replay:")).length === 2); + expect(entries).toEqual([ + "entered:one", + "entered:two", + "entered:one", + "result:yes", + "replay:yes", + "entered:two", + "result:yes", + "replay:yes", + ]); + expect(one).not.toBe(two); + }, + ); + + test.serial("uses a stable payload-based key when a timeout resumes the function", async () => { + const database = await db(); + const entered: string[] = []; + const errors: unknown[] = []; + const { conductor } = await setup( + database, + async (id, ctx) => { + entered.push(id); + try { + await ctx.waitForEvent(`timeout:${id}`, { event, timeout: "20ms" }); + } catch (error) { + errors.push(error); + } + }, + orchestrators, + ); + + await conductor.invoke({ name: "wait.task" }, { id: "one" }); + await until(async () => errors.length === 1); + expect(entered).toEqual(["one", "one"]); + expect(errors).toHaveLength(1); + expect(errors[0]).toBeInstanceOf(WaitForEventTimeoutError); + }); + + test.serial("events emitted immediately after registration win the wait", async () => { + const database = await db(); + const result: string[] = []; + const { conductor } = await setup( + database, + async (id, ctx) => { + const value = await ctx.waitForEvent(`immediate:${id}`, { event }); + result.push(value.payload.id); + }, + orchestrators, + ); + + await conductor.invoke({ name: "wait.task" }, { id: "race" }); + await waiting(database); + await conductor.emit("wait.order", { id: "race", kind: "match" }); + await until(async () => result.length === 1); + expect(result).toEqual(["race"]); + }); + + test.serial("does not deliver events emitted before the subscription boundary", async () => { + const database = await db(); + const result: string[] = []; + await database.client.emitEvent({ + eventKey: "wait.order", + payload: { id: "old", kind: "match" }, + }); + const handler: Handler = async (id, ctx) => { + const value = await ctx.waitForEvent(`boundary:${id}`, { event }); + result.push(value.payload.id); + }; + const first = await setup(database, handler, orchestrators); + const executionId = await first.conductor.invoke({ name: "wait.task" }, { id: "run" }); + await waiting(database); + await Bun.sleep(100); + expect(result).toEqual([]); + await first.orchestrator.stop(); + const second = await setup(database, handler, orchestrators); + await second.conductor.emit("wait.order", { id: "new", kind: "match" }); + await until(async () => result.length === 1); + expect(result).toEqual(["new"]); + expect(executionId).toBeTruthy(); + }); + + test.serial("restarts the same handler with the same step key", async () => { + const database = await db(); + const entered: string[] = []; + const result: string[] = []; + const handler: Handler = async (id, ctx) => { + entered.push(id); + const value = await ctx.waitForEvent(`restart:${id}`, { event }); + result.push(value.payload.id); + }; + const first = await setup(database, handler, orchestrators); + await first.conductor.invoke({ name: "wait.task" }, { id: "restart" }); + await waiting(database); + await first.orchestrator.stop(); + const second = await setup(database, handler, orchestrators); + await second.conductor.emit("wait.order", { id: "restart", kind: "match" }); + await until(async () => result.length === 1); + expect(entered).toEqual(["restart", "restart"]); + expect(result).toEqual(["restart"]); + }); + + test.serial("delivers the oldest matching event", async () => { + const database = await db(); + const result: string[] = []; + const { conductor } = await setup( + database, + async (id, ctx) => { + const value = await ctx.waitForEvent(`oldest:${id}`, { + event, + filter: { kind: ["match"] }, + }); + result.push(value.payload.id); + }, + orchestrators, + ); + await conductor.invoke({ name: "wait.task" }, { id: "oldest" }); + await waiting(database); + await conductor.emit("wait.order", { id: "first", kind: "match" }); + await Bun.sleep(5); + await conductor.emit("wait.order", { id: "second", kind: "match" }); + await Bun.sleep(5); + await conductor.emit("wait.order", { id: "third", kind: "match" }); + await until(async () => result.length === 1); + expect(result).toEqual(["first"]); + }); + + test.serial("does not register a wait with a stale execution claim", async () => { + const database = await db(); + const { conductor, orchestrator } = await setup(database, async () => {}, orchestrators); + await orchestrator.stop(); + const executionId = await conductor.invoke({ name: "wait.task" }, { id: "stale" }); + if (!executionId) throw new Error("invoke did not return an execution id"); + const registered = await database.client.registerEventWait({ + executionId, + queue: "default", + taskKey: "wait.task", + eventKey: "wait.order", + stepKey: "stale:stale", + filter: null, + timeoutMs: null, + orchestratorId: crypto.randomUUID(), + }); + expect(registered).toBe(false); + const rows = await database.sql<{ n: number; waiting_step_key: string | null }[]>` + select count(s.*)::int as n, max(e.waiting_step_key) as waiting_step_key + from pgconductor._private_event_subscriptions s + right join pgconductor._private_executions e on e.id = ${executionId}::uuid + where s.execution_id = ${executionId}::uuid + `; + expect(Number(rows[0]?.n)).toBe(0); + expect(rows[0]?.waiting_step_key).toBeNull(); + }); + + test.serial( + "resolves an eligible wait after more than one event batch of child waits", + async () => { + const database = await db(); + await database.sql` + insert into pgconductor._private_tasks (key, queue) + values ('fairness.task', 'default') + on conflict (key, queue) do nothing + `; + await database.sql` + insert into pgconductor._private_executions + (task_key, queue, payload, run_at, waiting_on_execution_id, waiting_step_key) + select 'fairness.task', 'default', '{}', 'infinity'::timestamptz, + pgconductor._private_portable_uuidv7(), 'child-step' + from generate_series(1, 101) + `; + const eventId = await database.client.emitEvent({ + eventKey: "fairness.event", + payload: { id: "eligible" }, + }); + const [execution] = await database.sql<{ id: string }[]>` + insert into pgconductor._private_executions + (task_key, queue, payload, run_at, waiting_step_key) + values ('fairness.task', 'default', '{}', 'infinity'::timestamptz, 'event-step') + returning id + `; + await database.sql` + insert into pgconductor._private_event_subscriptions + (task_key, queue, event_key, kind, execution_id, step_key, wait_after_event_position) + values ('fairness.task', 'default', 'fairness.event', 'execution_wait', + ${execution!.id}::uuid, 'event-step', 0) + `; + + expect(await database.client.resolveEventWaits({ batchSize: 100 })).toBe(1); + const [step] = await database.sql<{ result: { result: { payload: { id: string } } } }[]>` + select result from pgconductor._private_steps + where execution_id = ${execution!.id}::uuid and key = 'event-step' + `; + expect(step?.result.result.payload.id).toBe("eligible"); + expect(eventId).toBeString(); + }, + ); + + test.serial("leaves attempts unchanged while an execution waits", async () => { + const database = await db(); + const { conductor } = await setup( + database, + async (id, ctx) => { + await ctx.waitForEvent(`attempts:${id}`, { event }); + }, + orchestrators, + ); + const executionId = await conductor.invoke({ name: "wait.task" }, { id: "attempts" }); + if (!executionId) throw new Error("invoke did not return an execution id"); + await waiting(database); + const rows = await database.sql<{ attempts: number }[]>` + select attempts from pgconductor._private_executions where id = ${executionId}::uuid + `; + expect(Number(rows[0]?.attempts)).toBe(1); + }); + + test.serial("match and timeout produce exactly one terminal outcome", async () => { + const database = await db(); + const outcomes: string[] = []; + const { conductor } = await setup( + database, + async (id, ctx) => { + try { + const value = await ctx.waitForEvent(`outcome:${id}`, { event, timeout: "100ms" }); + outcomes.push(`match:${value.payload.id}`); + } catch (error) { + if (error instanceof WaitForEventTimeoutError) outcomes.push("timeout"); + else throw error; + } + }, + orchestrators, + ); + await conductor.invoke({ name: "wait.task" }, { id: "outcome" }); + await waiting(database); + await conductor.emit("wait.order", { id: "outcome", kind: "match" }); + await until(async () => outcomes.length === 1); + await Bun.sleep(150); + expect(outcomes).toEqual(["match:outcome"]); + expect( + Number( + ( + await database.sql`select count(*)::int as n from pgconductor._private_event_subscriptions where kind = 'execution_wait'` + )[0]?.n, + ), + ).toBe(0); + }); + + test.serial("cancellation removes the wait and prevents a resume", async () => { + const database = await db(); + const entered: string[] = []; + const result: string[] = []; + const { conductor } = await setup( + database, + async (id, ctx) => { + entered.push(id); + const value = await ctx.waitForEvent(`cancel:${id}`, { event, timeout: "1h" }); + result.push(value.payload.id); + }, + orchestrators, + ); + const executionId = await conductor.invoke({ name: "wait.task" }, { id: "cancel" }); + if (!executionId) throw new Error("invoke did not return an execution id"); + await waiting(database); + await database.client.cancelExecution(executionId); + await waiting(database, 0); + await conductor.emit("wait.order", { id: "cancel", kind: "match" }); + await Bun.sleep(150); + expect(entered).toEqual(["cancel"]); + expect(result).toEqual([]); + }); + + test.serial( + "does not duplicate subscriptions or resume an execution twice", + async () => { + const database = await db(); + const entered: string[] = []; + const result: string[] = []; + const { conductor } = await setup( + database, + async (id, ctx) => { + entered.push(id); + const value = await ctx.waitForEvent(`once:${id}`, { event }); + result.push(value.payload.id); + }, + orchestrators, + ); + await conductor.invoke({ name: "wait.task" }, { id: "once" }); + await waiting(database); + await Bun.sleep(100); + expect( + Number( + ( + await database.sql`select count(*)::int as n from pgconductor._private_event_subscriptions where kind = 'execution_wait'` + )[0]?.n, + ), + ).toBe(1); + await conductor.emit("wait.order", { id: "once", kind: "match" }); + await until(async () => result.length === 1); + await Bun.sleep(100); + expect(entered).toEqual(["once", "once"]); + expect(result).toEqual(["once"]); + }, + 60_000, + ); +}); diff --git a/packages/pgconductor-js/tests/mocks/database-client.mock.ts b/packages/pgconductor-js/tests/mocks/database-client.mock.ts index db8b679..7b4e000 100644 --- a/packages/pgconductor-js/tests/mocks/database-client.mock.ts +++ b/packages/pgconductor-js/tests/mocks/database-client.mock.ts @@ -19,6 +19,7 @@ export class MockDatabaseClient implements IDatabaseClient { getExecutions = mock(async () => []); returnExecutions = mock(async (_results) => {}); removeExecutions = mock(async () => false); + removeProcessedEvents = mock(async () => false); registerWorker = mock(async () => {}); scheduleCronExecution = mock(async () => "mock-cron-id"); unscheduleCronExecution = mock(async () => {}); @@ -28,6 +29,7 @@ export class MockDatabaseClient implements IDatabaseClient { loadStep = mock(async () => null); saveStep = mock(async (_args) => {}); clearWaitingState = mock(async () => {}); + registerEventWait = mock(async () => true); cancelExecution = mock(async () => true); getCurrentTime = mock(async () => new Date()); setFakeTime = mock(async () => {}); @@ -35,6 +37,8 @@ export class MockDatabaseClient implements IDatabaseClient { subscribeEvent = mock(async () => "mock-subscription-id"); subscribeDbChange = mock(async () => "mock-subscription-id"); emitEvent = mock(async () => "mock-event-id"); + resolveEventWaits = mock(async () => 0); + processEvents = mock(async () => 0); constructor(overrides: Partial = {}) { Object.assign(this, overrides); diff --git a/packages/pgconductor-js/tests/mocks/in-memory-database-client.ts b/packages/pgconductor-js/tests/mocks/in-memory-database-client.ts index 7170e10..2a07142 100644 --- a/packages/pgconductor-js/tests/mocks/in-memory-database-client.ts +++ b/packages/pgconductor-js/tests/mocks/in-memory-database-client.ts @@ -5,6 +5,7 @@ import type { ExecutionSpec, TaskSpec, Payload, + JsonValue, SetFakeTimeArgs, // EventSubscriptionSpec, } from "../../src/database-client"; @@ -22,6 +23,7 @@ import type { LoadStepArgs, SaveStepArgs, ClearWaitingStateArgs, + RegisterEventWaitArgs, OrchestratorShutdownArgs, // EmitEventArgs, } from "../../src/query-builder"; @@ -39,6 +41,7 @@ interface StoredExecution { id: string; task_key: string; queue: string; + group: string | null; payload: Payload; state: "pending" | "running" | "completed" | "failed"; run_at: Date; @@ -57,9 +60,15 @@ interface StoredExecution { orchestrator_id: string | null; parent_execution_id: string | null; parent_step_key: string | null; - slot_group_number: number | null; created_at: Date; updated_at: Date; + failed_at: Date | null; + dead_letter_source_execution_id: string | null; + dead_letter_source_queue: string | null; + dead_letter_source_task_key: string | null; + dead_letter_error: string | null; + dead_letter_attempts: number | null; + dead_letter_failed_at: Date | null; } interface StoredStep { @@ -78,6 +87,9 @@ interface StoredTask { window_start: string | null; window_end: string | null; concurrency: number | null; + group_concurrency: number | null; + dead_letter_queue: string | null; + dead_letter_task_key: string | null; } interface StoredCronSchedule { @@ -97,15 +109,46 @@ interface StoredOrchestrator { interface StoredEventSubscription { id: string; - execution_id: string; - step_key: string; - source: "event" | "db"; - event_key?: string; - schema_name?: string; - table_name?: string; - operation?: string; - columns?: string[]; - timeout_at: Date | null; + task_key: string; + queue: string; + event_key: string | null; + filter: Record | null; + execution_id?: string; + step_key?: string; + source?: "event" | "db"; + timeout_at?: Date | null; + wait_after_event_position?: number; +} + +interface StoredCustomEvent { + id: string; + event_key: string; + payload: JsonValue; + created_at: Date; + processed_at: Date | null; + event_position: number; +} + +function isPayload(value: JsonValue): value is Payload { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function jsonEquals(left: JsonValue | undefined, right: JsonValue | undefined): boolean { + if (left === right) return true; + if (left === null || right === null || typeof left !== typeof right) return false; + if (Array.isArray(left) || Array.isArray(right)) { + if (!Array.isArray(left) || !Array.isArray(right) || left.length !== right.length) return false; + return left.every((value, index) => jsonEquals(value, right[index])); + } + if (typeof left === "object" && typeof right === "object") { + const leftKeys = Object.keys(left); + const rightKeys = Object.keys(right); + return ( + leftKeys.length === rightKeys.length && + leftKeys.every((key) => Object.hasOwn(right, key) && jsonEquals(left[key], right[key])) + ); + } + return false; } interface SignalData { @@ -123,10 +166,13 @@ export class InMemoryDatabaseClient implements IDatabaseClient { private cronSchedules = new Map(); private orchestrators = new Map(); private eventSubscriptions = new Map(); + private customEvents = new Map(); + private eventDeliveries = new Set(); private eventPartitions = new Set(); private currentTime: Date; private migrationNumber = -1; private idCounter = 0; + private lastEventPosition = 0; constructor(initialTime: Date = new Date()) { this.currentTime = new Date(initialTime); @@ -202,10 +248,14 @@ export class InMemoryDatabaseClient implements IDatabaseClient { // Release executions claimed by stale orchestrator for (const exec of this.executions.values()) { if (exec.orchestrator_id === orchestrator.id && exec.state === "running") { - exec.state = "pending"; + exec.state = exec.cancelled ? "failed" : "pending"; + exec.last_error = exec.cancelled + ? exec.last_error || "Task was cancelled" + : exec.last_error; exec.orchestrator_id = null; } } + this.orchestrators.delete(orchestrator.id); } } } @@ -241,6 +291,12 @@ export class InMemoryDatabaseClient implements IDatabaseClient { _opts?: { signal?: AbortSignal }, ): Promise { this.orchestrators.delete(args.orchestratorId); + for (const exec of this.executions.values()) { + if (exec.orchestrator_id !== args.orchestratorId || exec.state !== "running") continue; + exec.state = exec.cancelled ? "failed" : "pending"; + exec.last_error = exec.cancelled ? exec.last_error || "Task was cancelled" : exec.last_error; + exec.orchestrator_id = null; + } } // ============================================================================ @@ -278,8 +334,27 @@ export class InMemoryDatabaseClient implements IDatabaseClient { window_start: taskSpec.window?.[0] || null, window_end: taskSpec.window?.[1] || null, concurrency: taskSpec.concurrency || null, + group_concurrency: taskSpec.groupConcurrency || null, + dead_letter_queue: taskSpec.deadLetterQueue || null, + dead_letter_task_key: taskSpec.deadLetterTaskKey || null, }; - this.tasks.set(taskSpec.key, task); + this.tasks.set(this.taskId(taskSpec.key, task.queue), task); + } + + // Registration is authoritative for this queue, just like PostgreSQL. + for (const id of [...this.eventSubscriptions.keys()]) { + if (this.eventSubscriptions.get(id)?.queue === args.queueName) + this.eventSubscriptions.delete(id); + } + for (const spec of args.eventSubscriptions || []) { + const id = this.generateId(); + this.eventSubscriptions.set(id, { + id, + task_key: spec.task_key, + queue: spec.queue, + event_key: spec.event_key, + filter: spec.filter, + }); } // Register cron schedules (ExecutionSpec[]) @@ -307,19 +382,28 @@ export class InMemoryDatabaseClient implements IDatabaseClient { ): Promise { const results: Execution[] = []; const now = this.getInternalTime(); - const taskKeysWithConcurrency = new Set(args.taskKeysWithConcurrency || []); const filterTaskKeys = new Set(args.filterTaskKeys || []); const concurrencyCount = new Map(); - // Count running executions per task for concurrency limits + // Count running executions per task for concurrency limits. for (const exec of this.executions.values()) { - if (exec.state === "running" && taskKeysWithConcurrency.has(exec.task_key)) { - concurrencyCount.set(exec.task_key, (concurrencyCount.get(exec.task_key) || 0) + 1); + const task = this.tasks.get(this.taskId(exec.task_key, exec.queue)); + if (exec.state === "running" && task?.concurrency != null) { + concurrencyCount.set( + this.taskId(exec.task_key, exec.queue), + (concurrencyCount.get(this.taskId(exec.task_key, exec.queue)) || 0) + 1, + ); } } // Find eligible executions - for (const exec of this.executions.values()) { + for (const exec of Array.from(this.executions.values()).sort( + (a, b) => + a.priority - b.priority || + a.run_at.getTime() - b.run_at.getTime() || + a.created_at.getTime() - b.created_at.getTime() || + a.id.localeCompare(b.id), + )) { // Skip if wrong queue if (exec.queue !== args.queueName) continue; @@ -338,21 +422,34 @@ export class InMemoryDatabaseClient implements IDatabaseClient { if (parent && parent.state !== "completed") continue; } - // Check concurrency limit - if (taskKeysWithConcurrency.has(exec.task_key)) { - const task = this.tasks.get(exec.task_key); - const limit = task?.concurrency || 1; - const current = concurrencyCount.get(exec.task_key) || 0; - if (current >= limit) continue; + const task = this.tasks.get(this.taskId(exec.task_key, exec.queue)); + // Check concurrency limit. + if (task?.concurrency != null) { + const current = concurrencyCount.get(this.taskId(exec.task_key, exec.queue)) || 0; + if (current >= task.concurrency) continue; + } + if (task?.group_concurrency && exec.group) { + const activeGroup = Array.from(this.executions.values()).filter( + (other) => + other.queue === exec.queue && + other.task_key === exec.task_key && + other.group === exec.group && + other.state === "running", + ).length; + if (activeGroup >= task.group_concurrency) continue; } - // Claim execution + // Claim execution. exec.state = "running"; + exec.attempts += 1; exec.orchestrator_id = args.orchestratorId; // Update concurrency count - if (taskKeysWithConcurrency.has(exec.task_key)) { - concurrencyCount.set(exec.task_key, (concurrencyCount.get(exec.task_key) || 0) + 1); + if (task?.concurrency != null) { + concurrencyCount.set( + this.taskId(exec.task_key, exec.queue), + (concurrencyCount.get(this.taskId(exec.task_key, exec.queue)) || 0) + 1, + ); } results.push({ @@ -366,7 +463,14 @@ export class InMemoryDatabaseClient implements IDatabaseClient { last_error: exec.last_error, dedupe_key: exec.dedupe_key || undefined, cron_expression: exec.cron_expression || undefined, - slot_group_number: exec.slot_group_number || undefined, + group: exec.group, + dead_letter_source_execution_id: exec.dead_letter_source_execution_id, + dead_letter_source_queue: exec.dead_letter_source_queue, + dead_letter_source_task_key: exec.dead_letter_source_task_key, + dead_letter_error: exec.dead_letter_error, + dead_letter_attempts: exec.dead_letter_attempts, + dead_letter_failed_at: exec.dead_letter_failed_at, + locked_by: exec.orchestrator_id || "", }); if (results.length >= args.batchSize) break; @@ -396,7 +500,15 @@ export class InMemoryDatabaseClient implements IDatabaseClient { for (const result of results) { const exec = this.executions.get(result.execution_id); - if (!exec) continue; + if ( + !exec || + !this.ownsClaim({ + executionId: result.execution_id, + queue: result.queue, + orchestratorId: result.orchestrator_id, + }) + ) + continue; switch (result.status) { case "completed": { @@ -422,7 +534,7 @@ export class InMemoryDatabaseClient implements IDatabaseClient { } // Remove if cleanup is configured (only if task registered) - const task = this.tasks.get(exec.task_key); + const task = this.tasks.get(this.taskId(exec.task_key, exec.queue)); if (task && task.remove_on_complete_days != null) { this.executions.delete(exec.id); this.steps.delete(exec.id); @@ -431,18 +543,20 @@ export class InMemoryDatabaseClient implements IDatabaseClient { } case "failed": { - exec.attempts++; exec.last_error = result.error; exec.orchestrator_id = null; - const task = this.tasks.get(exec.task_key); + const task = this.tasks.get(this.taskId(exec.task_key, exec.queue)); const maxAttempts = task?.max_attempts || 3; if (exec.attempts >= maxAttempts) { // Permanently failed exec.state = "failed"; + exec.failed_at = now; + if (!exec.cancelled) this.deliverToDeadLetterQueue(exec, task, result.error, now); - // Fail parent if waiting + // Fail parent if waiting. A force-failed parent is itself terminal and + // follows its own DLQ and retention policy. if (exec.parent_execution_id) { const parent = this.executions.get(exec.parent_execution_id); if (parent && parent.waiting_on_execution_id === exec.id) { @@ -450,7 +564,14 @@ export class InMemoryDatabaseClient implements IDatabaseClient { parent.last_error = `Child execution failed: ${result.error}`; parent.waiting_on_execution_id = null; parent.waiting_step_key = null; - parent.waiting_timeout_at = null; + const parentTask = this.tasks.get(this.taskId(parent.task_key, parent.queue)); + if (!exec.cancelled) { + this.deliverToDeadLetterQueue(parent, parentTask, parent.last_error, now); + } + if (parentTask?.remove_on_fail_days != null) { + this.executions.delete(parent.id); + this.steps.delete(parent.id); + } } } @@ -489,10 +610,15 @@ export class InMemoryDatabaseClient implements IDatabaseClient { case "permanently_failed": { exec.state = "failed"; + exec.failed_at = now; exec.last_error = result.error; + + const task = this.tasks.get(this.taskId(exec.task_key, exec.queue)); + if (!exec.cancelled) this.deliverToDeadLetterQueue(exec, task, result.error, now); exec.orchestrator_id = null; - // Fail parent if waiting + // Fail parent if waiting. A force-failed parent is itself terminal and + // follows its own DLQ and retention policy. if (exec.parent_execution_id) { const parent = this.executions.get(exec.parent_execution_id); if (parent && parent.waiting_on_execution_id === exec.id) { @@ -501,10 +627,17 @@ export class InMemoryDatabaseClient implements IDatabaseClient { parent.waiting_on_execution_id = null; parent.waiting_step_key = null; parent.waiting_timeout_at = null; + const parentTask = this.tasks.get(this.taskId(parent.task_key, parent.queue)); + if (!exec.cancelled) { + this.deliverToDeadLetterQueue(parent, parentTask, parent.last_error, now); + } + if (parentTask?.remove_on_fail_days != null) { + this.executions.delete(parent.id); + this.steps.delete(parent.id); + } } } - const task = this.tasks.get(exec.task_key); if (task && task.remove_on_fail_days != null) { this.executions.delete(exec.id); this.steps.delete(exec.id); @@ -518,6 +651,7 @@ export class InMemoryDatabaseClient implements IDatabaseClient { task_key: result.child_task_name, queue: result.child_task_queue, payload: result.child_payload || {}, + group: result.group, parent_execution_id: exec.id, parent_step_key: result.step_key, }); @@ -625,13 +759,14 @@ export class InMemoryDatabaseClient implements IDatabaseClient { // Helper to create execution const createExecution = (singletonOnValue: Date | null): string => { - const task = this.tasks.get(spec.task_key); + const task = this.tasks.get(this.taskId(spec.task_key, spec.queue)); const id = this.generateId(); const execution: StoredExecution = { id, task_key: spec.task_key, queue: spec.queue, + group: spec.group || null, payload: spec.payload || {}, state: "pending", run_at: spec.run_at || now, @@ -650,9 +785,15 @@ export class InMemoryDatabaseClient implements IDatabaseClient { orchestrator_id: null, parent_execution_id: spec.parent_execution_id || null, parent_step_key: spec.parent_step_key || null, - slot_group_number: null, created_at: now, updated_at: now, + failed_at: null, + dead_letter_source_execution_id: null, + dead_letter_source_queue: null, + dead_letter_source_task_key: null, + dead_letter_error: null, + dead_letter_attempts: null, + dead_letter_failed_at: null, }; this.executions.set(id, execution); @@ -823,8 +964,7 @@ export class InMemoryDatabaseClient implements IDatabaseClient { exec.last_error = options.reason || "Execution was cancelled"; if (exec.state === "running") { - exec.state = "failed"; - exec.orchestrator_id = null; + exec.cancelled = true; } return true; @@ -915,6 +1055,7 @@ export class InMemoryDatabaseClient implements IDatabaseClient { run_at: nextRun, dedupe_key: dedupeKey, cron_expression: exec.cron_expression, + group: exec.group, }); } @@ -934,21 +1075,24 @@ export class InMemoryDatabaseClient implements IDatabaseClient { // Steps // ============================================================================ - async loadStep(args: LoadStepArgs, _opts?: { signal?: AbortSignal }): Promise { + async loadStep( + args: LoadStepArgs, + _opts?: { signal?: AbortSignal }, + ): Promise { + if (!this.ownsClaim(args)) return undefined; const execSteps = this.steps.get(args.executionId); - if (!execSteps) return null; - + if (!execSteps) return undefined; const step = execSteps.get(args.key); - return step ? step.result : null; + return step ? step.result : undefined; } async saveStep(args: SaveStepArgs, _opts?: { signal?: AbortSignal }): Promise { + if (!this.ownsClaim(args)) return; let execSteps = this.steps.get(args.executionId); if (!execSteps) { execSteps = new Map(); this.steps.set(args.executionId, execSteps); } - execSteps.set(args.key, { execution_id: args.executionId, step_key: args.key, @@ -957,12 +1101,56 @@ export class InMemoryDatabaseClient implements IDatabaseClient { }); } + async registerEventWait( + args: RegisterEventWaitArgs, + _opts?: { signal?: AbortSignal }, + ): Promise { + const exec = this.executions.get(args.executionId); + if ( + !exec || + exec.queue !== args.queue || + exec.task_key !== args.taskKey || + exec.orchestrator_id !== args.orchestratorId || + exec.state !== "running" || + exec.cancelled || + exec.failed_at + ) + return false; + const existing = [...this.eventSubscriptions.values()].find( + (subscription) => + subscription.execution_id === args.executionId && subscription.step_key === args.stepKey, + ); + if (existing) return false; + const id = this.generateId(); + this.eventSubscriptions.set(id, { + id, + task_key: args.taskKey, + queue: args.queue, + event_key: args.eventKey, + filter: args.filter, + execution_id: args.executionId, + step_key: args.stepKey, + source: "event", + wait_after_event_position: this.lastEventPosition, + timeout_at: + args.timeoutMs == null ? null : new Date(this.getInternalTime().getTime() + args.timeoutMs), + }); + exec.state = "pending"; + exec.run_at = new Date(8640000000000000); + exec.waiting_on_execution_id = null; + exec.waiting_step_key = args.stepKey; + exec.waiting_timeout_at = + args.timeoutMs == null ? null : new Date(this.getInternalTime().getTime() + args.timeoutMs); + exec.orchestrator_id = null; + return true; + } + async clearWaitingState( args: ClearWaitingStateArgs, _opts?: { signal?: AbortSignal }, ): Promise { const exec = this.executions.get(args.executionId); - if (!exec) return; + if (!exec || !this.ownsClaim(args)) return; exec.waiting_on_execution_id = null; exec.waiting_step_key = null; @@ -1012,14 +1200,204 @@ export class InMemoryDatabaseClient implements IDatabaseClient { return this.generateId(); } - async emitEvent(): Promise { - return this.generateId(); + async emitEvent(args: { eventKey: string; payload?: JsonValue }): Promise { + const id = this.generateId(); + this.customEvents.set(id, { + id, + event_key: args.eventKey, + payload: args.payload ?? {}, + created_at: this.getInternalTime(), + processed_at: null, + event_position: ++this.lastEventPosition, + }); + return id; + } + + async resolveEventWaits(args: { batchSize: number }): Promise { + let count = 0; + const now = this.getInternalTime(); + const candidates = [...this.eventSubscriptions.values()] + .filter((subscription) => subscription.execution_id) + .filter((subscription) => { + const exec = this.executions.get(subscription.execution_id!); + if (!exec || exec.waiting_step_key !== subscription.step_key) return false; + const hasEvent = [...this.customEvents.values()].some( + (candidate) => + candidate.event_key === subscription.event_key && + candidate.event_position > (subscription.wait_after_event_position ?? 0) && + (!subscription.timeout_at || candidate.created_at <= subscription.timeout_at) && + Object.entries(subscription.filter || {}).every(([field, values]) => + values.some((value) => + jsonEquals( + value, + isPayload(candidate.payload) ? candidate.payload[field] : undefined, + ), + ), + ), + ); + return hasEvent || Boolean(subscription.timeout_at && subscription.timeout_at <= now); + }) + .sort((left, right) => left.id.localeCompare(right.id)) + .slice(0, Math.max(args.batchSize || 0, 0)); + for (const subscription of candidates) { + const exec = this.executions.get(subscription.execution_id!); + if (!exec || exec.waiting_step_key !== subscription.step_key) continue; + const event = [...this.customEvents.values()] + .filter((candidate) => candidate.event_key === subscription.event_key) + .filter( + (candidate) => candidate.event_position > (subscription.wait_after_event_position ?? 0), + ) + .filter( + (candidate) => + !subscription.timeout_at || candidate.created_at <= subscription.timeout_at, + ) + .filter((candidate) => + Object.entries(subscription.filter || {}).every(([field, values]) => + values.some((value) => + jsonEquals( + value, + isPayload(candidate.payload) ? candidate.payload[field] : undefined, + ), + ), + ), + ) + .sort((a, b) => a.event_position - b.event_position)[0]; + if (!event && (!subscription.timeout_at || subscription.timeout_at > now)) continue; + this.saveWaitResult( + exec, + subscription.step_key!, + event + ? { + result: { name: event.event_key, payload: structuredClone(event.payload) as Payload }, + } + : { __pgconductor_wait_for_event_timeout: true }, + ); + exec.run_at = now; + exec.waiting_step_key = null; + exec.waiting_timeout_at = null; + this.eventSubscriptions.delete(subscription.id); + count++; + } + return count; + } + + async processEvents(_args: { batchSize: number }): Promise { + let count = 0; + const now = this.getInternalTime(); + const events = [...this.customEvents.values()] + .filter((event) => !event.processed_at) + .sort((left, right) => left.event_position - right.event_position); + for (const event of events) { + const payload = isPayload(event.payload) ? event.payload : {}; + for (const subscription of [...this.eventSubscriptions.values()]) { + if (subscription.execution_id || subscription.event_key !== event.event_key) continue; + const matches = Object.entries(subscription.filter || {}).every(([field, values]) => + values.some((value) => jsonEquals(value, payload[field])), + ); + if (!matches) continue; + const deliveryKey = `${event.id}:${subscription.id}`; + if (this.eventDeliveries.has(deliveryKey)) continue; + this.eventDeliveries.add(deliveryKey); + await this.invoke({ + task_key: subscription.task_key, + queue: subscription.queue, + payload: { event: event.event_key, payload: structuredClone(event.payload) as Payload }, + }); + } + event.processed_at = now; + count++; + } + return count; + } + + private saveWaitResult(exec: StoredExecution, stepKey: string, result: Payload): void { + let steps = this.steps.get(exec.id); + if (!steps) { + steps = new Map(); + this.steps.set(exec.id, steps); + } + steps.set(stepKey, { + execution_id: exec.id, + step_key: stepKey, + result, + created_at: this.getInternalTime(), + }); + } + + async removeProcessedEvents(args: { before: Date; batchSize: number }): Promise { + const old = [...this.customEvents.values()] + .filter((event) => event.processed_at && event.processed_at < args.before) + .slice(0, args.batchSize); + for (const event of old) this.customEvents.delete(event.id); + return old.length >= args.batchSize; } // ============================================================================ + private deliverToDeadLetterQueue( + exec: StoredExecution, + task: StoredTask | undefined, + error: string, + now: Date, + ): void { + if (!task?.dead_letter_queue || exec.cancelled) return; + const destinationTaskKey = task.dead_letter_task_key || exec.task_key; + const duplicate = Array.from(this.executions.values()).some( + (destination) => + destination.dead_letter_source_execution_id === exec.id && + destination.queue === task.dead_letter_queue && + destination.task_key === destinationTaskKey, + ); + if (duplicate) return; + const id = this.generateId(); + this.executions.set(id, { + id, + task_key: destinationTaskKey, + queue: task.dead_letter_queue, + group: exec.group, + payload: structuredClone(exec.payload), + state: "pending", + run_at: now, + attempts: 0, + max_attempts: 3, + last_error: null, + result: null, + cancelled: false, + waiting_on_execution_id: null, + waiting_step_key: null, + waiting_timeout_at: null, + dedupe_key: null, + singleton_on: null, + cron_expression: null, + priority: 0, + orchestrator_id: null, + parent_execution_id: null, + parent_step_key: null, + created_at: now, + updated_at: now, + failed_at: null, + dead_letter_source_execution_id: exec.id, + dead_letter_source_queue: exec.queue, + dead_letter_source_task_key: exec.task_key, + dead_letter_error: error, + dead_letter_attempts: exec.attempts, + dead_letter_failed_at: now, + }); + } + // Helpers // ============================================================================ + private taskId(key: string, queue: string): string { + return `${queue}\u0000${key}`; + } + + private ownsClaim(args: { executionId: string; queue: string; orchestratorId: string }): boolean { + const exec = this.executions.get(args.executionId); + return Boolean( + exec && exec.queue === args.queue && exec.orchestrator_id === args.orchestratorId, + ); + } + private generateId(): string { this.idCounter++; return `in-memory-${this.idCounter.toString().padStart(8, "0")}`; @@ -1075,6 +1453,10 @@ export class InMemoryDatabaseClient implements IDatabaseClient { return Array.from(this.eventSubscriptions.values()); } + getCustomEvents(): StoredCustomEvent[] { + return Array.from(this.customEvents.values()); + } + clear(): void { this.executions.clear(); this.steps.clear(); @@ -1082,8 +1464,11 @@ export class InMemoryDatabaseClient implements IDatabaseClient { this.cronSchedules.clear(); this.orchestrators.clear(); this.eventSubscriptions.clear(); + this.customEvents.clear(); + this.eventDeliveries.clear(); this.eventPartitions.clear(); this.idCounter = 0; + this.lastEventPosition = 0; } /** diff --git a/packages/pgconductor-js/tests/unit/dead-letter-queue-types.test.ts b/packages/pgconductor-js/tests/unit/dead-letter-queue-types.test.ts new file mode 100644 index 0000000..cc92147 --- /dev/null +++ b/packages/pgconductor-js/tests/unit/dead-letter-queue-types.test.ts @@ -0,0 +1,77 @@ +import { describe, expect, test } from "bun:test"; +import { z } from "zod"; +import { Conductor } from "../../src/conductor"; +import { TaskSchemas } from "../../src/schemas"; +import { defineTask } from "../../src/task-definition"; + +const sourceDefinition = defineTask({ + name: "source", + queue: "source-q", + payload: z.object({ value: z.string() }), +}); +const compatibleDefinition = defineTask({ + name: "compatible", + queue: "dlq", + payload: z.object({ value: z.string() }), +}); +const incompatibleDefinition = defineTask({ + name: "incompatible", + queue: "dlq", + payload: z.object({ other: z.number() }), +}); + +function conductor() { + return Conductor.create({ + sql: {} as any, + tasks: TaskSchemas.fromSchema([sourceDefinition, compatibleDefinition, incompatibleDefinition]), + context: {}, + }); +} + +describe("dead-letter task types and validation", () => { + test("accepts compatible targets and rejects incompatible payloads", () => { + const c = conductor(); + const compatible = c.createTask( + { name: "compatible", queue: "dlq" }, + { invocable: true }, + async () => {}, + ); + c.createTask( + { name: "source", queue: "source-q", deadLetter: { queue: "dlq", task: compatible } }, + { invocable: true }, + async () => {}, + ); + + const incompatible = c.createTask( + { name: "incompatible", queue: "dlq" }, + { invocable: true }, + async () => {}, + ); + if (false) + c.createTask( + // @ts-expect-error The DLQ handler must accept the source payload. + { name: "source", queue: "source-q", deadLetter: { queue: "dlq", task: incompatible } }, + { invocable: true }, + async () => {}, + ); + }); + + test("rejects a direct self-target but permits a cross-queue identity", () => { + const c = conductor(); + expect(() => + c.createTask( + { name: "source", queue: "source-q", deadLetter: { queue: "source-q" } }, + { invocable: true }, + async () => {}, + ), + ).toThrow("cannot dead-letter directly to itself"); + + expect(() => + c.createTask( + { name: "source", queue: "source-q", deadLetter: { queue: "other-q" } }, + { invocable: true }, + async () => {}, + ), + ).not.toThrow(); + }); +}); diff --git a/packages/pgconductor-js/tests/unit/duration.test.ts b/packages/pgconductor-js/tests/unit/duration.test.ts new file mode 100644 index 0000000..633e5f1 --- /dev/null +++ b/packages/pgconductor-js/tests/unit/duration.test.ts @@ -0,0 +1,16 @@ +import { describe, expect, test } from "bun:test"; +import { parseDuration } from "../../src/index"; + +describe("duration API", () => { + test("parses numeric and unit durations", () => { + expect(parseDuration(12.9)).toBe(12); + expect(parseDuration("1.5s")).toBe(1500); + expect(parseDuration("2m")).toBe(120000); + }); + + test("rejects invalid durations", () => { + for (const value of [-1, Number.NaN, Number.POSITIVE_INFINITY, "", "1", "-1s", "1x"] as const) { + expect(() => parseDuration(value as never)).toThrow(); + } + }); +}); diff --git a/packages/pgconductor-js/tests/unit/emit-types.test.ts b/packages/pgconductor-js/tests/unit/emit-types.test.ts index 09a6244..e92eb72 100644 --- a/packages/pgconductor-js/tests/unit/emit-types.test.ts +++ b/packages/pgconductor-js/tests/unit/emit-types.test.ts @@ -41,11 +41,13 @@ describe("emit method types", () => { Promise >(); - // @ts-expect-error - wrong payload type - conductor.emit("user.created", { orderId: 123 }); + if (false) { + // @ts-expect-error - wrong payload type + conductor.emit("user.created", { orderId: 123 }); - // @ts-expect-error - non-existent event - conductor.emit("non.existent", {}); + // @ts-expect-error - non-existent event + conductor.emit("non.existent", {}); + } }); test("ctx.emit accepts typed event payload", () => { diff --git a/packages/pgconductor-js/tests/unit/event-pipeline-types.test.ts b/packages/pgconductor-js/tests/unit/event-pipeline-types.test.ts new file mode 100644 index 0000000..e722ba2 --- /dev/null +++ b/packages/pgconductor-js/tests/unit/event-pipeline-types.test.ts @@ -0,0 +1,25 @@ +import { describe, expectTypeOf, test } from "bun:test"; +import { z } from "zod"; +import { defineEvent, type FilterForEvent } from "../../src/event-definition"; + +describe("event filter types", () => { + test("omitting filterable fields permits no filters", () => { + const event = defineEvent({ + name: "account.created", + payload: z.object({ accountId: z.string() }), + }); + expectTypeOf>().toEqualTypeOf<{}>(); + }); + + test("infer allowed fields and values from the payload", () => { + const event = defineEvent({ + name: "order.changed", + payload: z.object({ status: z.enum(["pending", "paid"]), accountId: z.string() }), + filterable: ["status"], + }); + + expectTypeOf>().toEqualTypeOf<{ + readonly status?: readonly ("pending" | "paid")[]; + }>(); + }); +}); diff --git a/packages/pgconductor-js/tests/unit/event-trigger-types.test.ts b/packages/pgconductor-js/tests/unit/event-trigger-types.test.ts index 1b5f5e6..5caca8d 100644 --- a/packages/pgconductor-js/tests/unit/event-trigger-types.test.ts +++ b/packages/pgconductor-js/tests/unit/event-trigger-types.test.ts @@ -193,7 +193,7 @@ describe("event triggers", () => { ); }); - test("custom event trigger with when clause", () => { + test("custom event trigger rejects a when clause", () => { const orderPlaced = defineEvent({ name: "order.placed", payload: z.object({ orderId: z.string(), total: z.number() }), @@ -211,17 +211,18 @@ describe("event triggers", () => { context: {}, }); - // Task with when clause - still receives full payload - conductor.createTask( - { name: "on-large-order" }, - { event: "order.placed", when: "new.payload->>'total'::numeric > 1000" }, - async (event) => { - expectTypeOf(event.payload).toEqualTypeOf<{ - orderId: string; - total: number; - }>(); - }, - ); + expect(() => + conductor.createTask( + { name: "on-large-order" }, + { event: "order.placed", when: "new.payload->>'total'::numeric > 1000" }, + async (event) => { + expectTypeOf(event.payload).toEqualTypeOf<{ + orderId: string; + total: number; + }>(); + }, + ), + ).toThrow("does not support a when clause"); }); test("database trigger with column selection", () => { diff --git a/packages/pgconductor-js/tests/unit/in-memory-database-client.test.ts b/packages/pgconductor-js/tests/unit/in-memory-database-client.test.ts index 366694c..8b57b09 100644 --- a/packages/pgconductor-js/tests/unit/in-memory-database-client.test.ts +++ b/packages/pgconductor-js/tests/unit/in-memory-database-client.test.ts @@ -3,6 +3,16 @@ import { test, expect, describe } from "bun:test"; import { InMemoryDatabaseClient } from "../mocks/in-memory-database-client"; describe("InMemoryDatabaseClient", () => { + function fencing(db: InMemoryDatabaseClient, executionId: string) { + const execution = db.getExecution(executionId); + if (!execution?.orchestrator_id) { + throw new Error(`execution ${executionId} is not claimed`); + } + return { + orchestrator_id: execution.orchestrator_id, + }; + } + describe("Basic Execution Lifecycle", () => { test("invoke creates pending execution", async () => { const db = new InMemoryDatabaseClient(); @@ -41,7 +51,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); @@ -67,13 +76,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "test-task", status: "completed", @@ -113,7 +122,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); @@ -122,6 +130,7 @@ describe("InMemoryDatabaseClient", () => { await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "test-task", status: "failed", @@ -165,13 +174,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "test-task", status: "failed", @@ -188,13 +197,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "test-task", status: "failed", @@ -243,18 +252,19 @@ describe("InMemoryDatabaseClient", () => { }, scheduleName: "test-schedule", }); + db.advanceTime(60000); - await db.getExecutions({ + const claimed = await db.getExecutions({ orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: id!, + orchestrator_id: claimed[0]!.locked_by, queue: "default", task_key: "cron-task", status: "completed", @@ -333,7 +343,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); @@ -350,6 +359,7 @@ describe("InMemoryDatabaseClient", () => { await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "cron-task", status: "failed", @@ -373,13 +383,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "cron-task", status: "completed", @@ -431,7 +441,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); @@ -453,6 +462,7 @@ describe("InMemoryDatabaseClient", () => { await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "slow-task", status: "failed", @@ -475,7 +485,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); @@ -486,6 +495,7 @@ describe("InMemoryDatabaseClient", () => { await db.returnExecutions([ { execution_id: id!, + ...fencing(db, id!), queue: "default", task_key: "slow-task", status: "failed", @@ -522,13 +532,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: parentId!, + ...fencing(db, parentId!), queue: "default", task_key: "parent", status: "invoke_child", @@ -565,13 +575,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: parentId!, + ...fencing(db, parentId!), queue: "default", task_key: "parent", status: "invoke_child", @@ -590,13 +600,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: childId, + ...fencing(db, childId), queue: "default", task_key: "child", status: "completed", @@ -636,13 +646,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: parentId!, + ...fencing(db, parentId!), queue: "default", task_key: "parent", status: "invoke_child", @@ -661,13 +671,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: childId, + ...fencing(db, childId), queue: "default", task_key: "child", status: "failed", @@ -691,16 +701,27 @@ describe("InMemoryDatabaseClient", () => { queue: "default", payload: {}, }); + const claimed = ( + await db.getExecutions({ + orchestratorId: "test-orch", + queueName: "default", + batchSize: 1, + filterTaskKeys: [], + }) + )[0]!; await db.saveStep({ executionId: execId!, - queue: "default", + queue: claimed.queue, + orchestratorId: claimed.locked_by, key: "step1", result: { data: "value" }, }); const result = await db.loadStep({ executionId: execId!, + queue: claimed.queue, + orchestratorId: claimed.locked_by, key: "step1", }); @@ -712,10 +733,12 @@ describe("InMemoryDatabaseClient", () => { const result = await db.loadStep({ executionId: "nonexistent", + queue: "default", + orchestratorId: "test-orch", key: "step1", }); - expect(result).toBeNull(); + expect(result).toBeUndefined(); }); }); @@ -745,7 +768,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); @@ -759,7 +781,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); @@ -797,7 +818,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: ["limited-task"], filterTaskKeys: [], }); @@ -808,7 +828,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: ["limited-task"], filterTaskKeys: [], }); @@ -818,6 +837,7 @@ describe("InMemoryDatabaseClient", () => { await db.returnExecutions([ { execution_id: batch1[0]!.id, + ...fencing(db, batch1[0]!.id), queue: "default", task_key: "limited-task", status: "completed", @@ -829,7 +849,6 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: ["limited-task"], filterTaskKeys: [], }); @@ -876,13 +895,13 @@ describe("InMemoryDatabaseClient", () => { orchestratorId: "test-orch", queueName: "default", batchSize: 10, - taskKeysWithConcurrency: [], filterTaskKeys: [], }); await db.returnExecutions([ { execution_id: id1!, + ...fencing(db, id1!), queue: "default", task_key: "test", status: "completed", diff --git a/packages/pgconductor-js/tests/unit/lib/map-concurrent.test.ts b/packages/pgconductor-js/tests/unit/lib/map-concurrent.test.ts index 4e8db55..e3da110 100644 --- a/packages/pgconductor-js/tests/unit/lib/map-concurrent.test.ts +++ b/packages/pgconductor-js/tests/unit/lib/map-concurrent.test.ts @@ -1,6 +1,7 @@ import { test, expect, describe } from "bun:test"; import { mapConcurrent } from "../../../src/lib/map-concurrent"; -import type { PollableAsyncIterable } from "../../../src/lib/async-queue"; +import { AsyncQueue, type PollableAsyncIterable } from "../../../src/lib/async-queue"; +import { Deferred } from "../../../src/lib/deferred"; class PollableGenerator implements PollableAsyncIterable { private buffer: T[] = []; @@ -57,6 +58,42 @@ describe("mapConcurrent", () => { expect(results).toEqual([0, 2, 4, 6, 8]); }); + test("starts items that arrive while another mapper is running", async () => { + const source = new AsyncQueue(2); + const firstStarted = new Deferred(); + const secondStarted = new Deferred(); + const releaseFirst = new Deferred(); + const results: number[] = []; + + const consume = async () => { + for await (const result of mapConcurrent(source, 2, async (value) => { + if (value === 1) { + firstStarted.resolve(); + await releaseFirst.promise; + } else { + secondStarted.resolve(); + } + return value; + })) { + results.push(result); + } + }; + + const consuming = consume(); + try { + await source.push(1); + await firstStarted.promise; + await source.push(2); + await secondStarted.promise; + } finally { + releaseFirst.resolve(); + source.close(); + } + await consuming; + + expect(new Set(results)).toEqual(new Set([1, 2])); + }); + test("respects concurrency limit", async () => { const concurrent: number[] = []; let maxConcurrent = 0; diff --git a/packages/pgconductor-js/tests/unit/throttle-debounce.test.ts b/packages/pgconductor-js/tests/unit/throttle-debounce.test.ts index 23816d5..6ae108d 100644 --- a/packages/pgconductor-js/tests/unit/throttle-debounce.test.ts +++ b/packages/pgconductor-js/tests/unit/throttle-debounce.test.ts @@ -134,7 +134,6 @@ describe("Throttle and Debounce", () => { queueName: "default", batchSize: 100, orchestratorId: "test-orch", - taskKeysWithConcurrency: [], filterTaskKeys: [], }); diff --git a/packages/pgconductor-js/tests/unit/wait-for-event-types.test.ts b/packages/pgconductor-js/tests/unit/wait-for-event-types.test.ts new file mode 100644 index 0000000..c8ee806 --- /dev/null +++ b/packages/pgconductor-js/tests/unit/wait-for-event-types.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, test } from "bun:test"; +import { expectTypeOf } from "expect-type"; +import { z } from "zod"; +import { Conductor } from "../../src/conductor"; +import { defineEvent } from "../../src/event-definition"; +import { defineTask } from "../../src/task-definition"; +import { EventSchemas, TaskSchemas } from "../../src/schemas"; +import { WaitForEventTimeoutError } from "../../src/index"; + +describe("waitForEvent API", () => { + test("returns the declared event and payload and constrains filters", () => { + const event = defineEvent({ + name: "api.order", + payload: z.object({ status: z.enum(["paid", "pending"]), id: z.string() }), + filterable: ["status"], + }); + const task = defineTask({ name: "api.wait", payload: z.object({}) }); + const conductor = Conductor.create({ + sql: {} as any, + tasks: TaskSchemas.fromSchema([task]), + events: EventSchemas.fromSchema([event]), + context: {}, + }); + conductor.createTask({ name: "api.wait" }, { invocable: true }, async (_event, ctx) => { + const result = await ctx.waitForEvent("order", { event, filter: { status: ["paid"] } }); + expectTypeOf(result).toEqualTypeOf<{ + name: "api.order"; + payload: { status: "paid" | "pending"; id: string }; + }>(); + // @ts-expect-error id is not declared filterable + ctx.waitForEvent("bad", { event, filter: { id: ["x"] } }); + }); + expect(WaitForEventTimeoutError).toBeDefined(); + }); +}); diff --git a/packages/pgconductor-js/tests/unit/worker-lifecycle.test.ts b/packages/pgconductor-js/tests/unit/worker-lifecycle.test.ts new file mode 100644 index 0000000..6da1797 --- /dev/null +++ b/packages/pgconductor-js/tests/unit/worker-lifecycle.test.ts @@ -0,0 +1,32 @@ +import { expect, test } from "bun:test"; +import { Worker } from "../../src/worker"; +import { DefaultLogger } from "../../src/lib/logger"; +import { MockDatabaseClient } from "../mocks/database-client.mock"; +import type { AnyTask } from "../../src/task"; + +const task = { + name: "lifecycle-task", + triggers: [{ invocable: true }], + maxAttempts: 3, + execute: async () => {}, +} as unknown as AnyTask; + +test("resets a worker after registration failure and permits retry", async () => { + let shouldFail = true; + const db = new MockDatabaseClient({ + registerWorker: async () => { + if (shouldFail) throw new Error("registration failed"); + }, + }); + const worker = new Worker("default", [task], db as never, new DefaultLogger()); + + await expect(worker.start("first-orchestrator")).rejects.toThrow("registration failed"); + // A failed startup is not a running worker: stopped must be already settled, + // rather than an unhandled rejected lifecycle promise. + await expect(worker.stopped).resolves.toBeUndefined(); + + shouldFail = false; + await worker.start("second-orchestrator"); + await worker.stop(); + await expect(worker.stopped).resolves.toBeUndefined(); +});