OpenTelemetry Instrumentation

[psteinroe]

Goal

Provide first-class OpenTelemetry tracing and metrics without monkey-patching pgconductor-js, wrapping exported prototypes, or taking ownership of an application's OpenTelemetry SDK/exporters.

OpenTelemetry is the observability API. Do not introduce a second generic tracing/metrics abstraction.

Direction

Instrumentation belongs directly at the durable runtime boundaries in pgconductor-js:

• enqueueing tasks and events;
• dispatching a task handler attempt;
• executing an uncached durable step;
• settling completion, retry, release, cancellation, child invocation, event-wait, or DLQ outcomes.

Delete/replace the implementation on feat/otel that uses InstrumentationBase to override Conductor, Worker, and TaskContext prototypes. That design is load-order-sensitive, requires manual patching under Bun, couples an instrumentation package to private implementation details, and can report an operation before its durable database transition actually commits.

Public API

The normal path should require no pgconductor-specific setup:

const sdk = new NodeSDK({
  // application-owned exporter, resource, processors, sampling, etc.
});

sdk.start();

const conductor = Conductor.create({
  connectionString,
  context: {},
});

pgconductor-js obtains its tracer, meter, active context, and propagator from @opentelemetry/api. If no SDK is registered, the global API is a no-op.

The only initial configuration is an opt-out:

const conductor = Conductor.create({
  connectionString,
  context: {},
  telemetry: false,
});

Do not initially expose independent tracer, meter, provider, context-manager, or propagator injection. Mixing those independently can create incoherent contexts. Applications needing custom providers should configure the standard global OpenTelemetry API before invoking work or starting workers.

Resolve global components at operation time rather than capturing a no-op propagator during Conductor.create().

Document that the application's SDK/context manager must start before the first invoke/emit operation and before workers begin processing.

Package boundaries

• Instrument directly inside pgconductor-js; do not publish a monkey-patching instrumentation package.
• Depend only on @opentelemetry/api; do not depend on an SDK, exporter, collector client, or vendor package.
• Explicitly externalize @opentelemetry/api from the Bun bundle so pgconductor uses the application's installed API/global registry.
• Use the pgconductor package name and version as the instrumentation scope.
• Keep semantic-convention constants internal and pin/document the emitted messaging convention version while those conventions remain in development.

Trace topology

Follow OpenTelemetry messaging semantic conventions.

Producer operations

Create a short PRODUCER send span around durable creation of:

• a direct or batch task invocation;
• a custom event;
• a child execution;
• the next cron execution;
• a dead-letter execution.

The span ends only after the database operation commits or fails definitively. A throttled invocation that inserts nothing is not reported as a sent message. Dedupe/debounce replacement stores the context associated with the newly accepted payload.

Use current messaging attributes such as:

• messaging.system = "postgres_conductor"
• messaging.operation.name
• messaging.operation.type = "send"
• messaging.destination.name
• messaging.message.id
• messaging.batch.message_count

Add bounded pgconductor attributes for task key, queue, execution attempt, trigger kind, and outcome where the standard conventions have no equivalent. Never capture task/event payloads automatically.

For task messages, the destination is the queue and the task key is a pgconductor attribute. For custom events, the destination is the event key.

Consumer operations

Create one CONSUMER process span for every real handler attempt.

• A retry creates a new attempt span.
• A span never remains open across retry backoff, sleep, a child wait, an event wait, a window release, or worker shutdown.
• A resumed execution creates a new attempt span.
• Run the handler under the active process context so user-created spans and instrumented logs correlate naturally.
• Explicitly select the parent/root context; never inherit an incidental worker polling context.

For a single execution, use its valid remote creation context as the process span's parent. This produces navigable end-to-end traces, with retries and fan-out consumers represented as siblings beneath their producer.

For a batch handler, start one process span with no message parent and add a link for every valid execution creation context. Include messaging.batch.message_count.

The process span covers delivery to and execution of the application handler. End and classify it when the handler produces an outcome. Settlement is represented separately, so a successful handler followed by failed acknowledgement correctly appears as successful processing plus failed settlement and may be redelivered.

Settlement operations

Create a CLIENT settle span around the database transaction that acknowledges completion or persists retry, release, cancellation, child invocation, event-wait, or DLQ state.

• Use messaging.operation.type = "settle" and a low-cardinality operation name such as ack, nack, or release.
• End and classify the span only after the transaction commits or fails.
• A settlement batch uses one span with messaging.batch.message_count and links to the corresponding process-span contexts rather than selecting an arbitrary parent.
• Producer spans for child, cron, event, or DLQ rows created by settlement end only when that same transaction commits.
• Keep process span contexts only in the worker's private in-memory result buffer for settlement links; never put span objects or live contexts into persisted/public execution DTOs.

A separate receive span is not required: pgconductor pushes claimed executions into application handlers, and the messaging conventions allow the process span to represent that delivery directly. PostgreSQL instrumentation already describes the underlying polling query.

Durable steps and control flow

Create an INTERNAL step span only when a ctx.step() callback actually runs:

1. perform abort/window checks;
2. load the durable step;
3. return immediately without a span on a cache hit;
4. start the span immediately before the callback;
5. keep it open through successful step persistence;
6. record exceptions and always end it.

Do not create standalone spans for:

• worker polling or maintenance;
• cached step replay;
• the elapsed duration of sleep or another durable wait;
• checkpoint, cancel, or wait registration.

Record relevant release/wait reasons as events or bounded attributes on the current attempt span. PostgreSQL client instrumentation remains responsible for SQL spans.

Context persistence

Add a dedicated nullable trace_context jsonb carrier to executions and custom events. Merge this into the existing setup/event migrations because the project is prerelease.

For the initial implementation:

• persist only valid W3C traceparent and optional tracestate string fields;
• do not persist baggage by default;
• enforce a small serialized size limit (maximum 1 KiB);
• treat extracted contexts as remote;
• ignore malformed or oversized metadata without affecting queue behavior;
• do not expose trace context through task payload APIs.

Keep the original creation context for every retry so attempt spans share the same producer parent.

Context must be handled by every execution-creation path: single/batch invoke, cron, dedupe/debounce replacement, event fan-out, child invocation, and DLQ delivery.

Custom events copy their creation context to triggered task executions. SQL-only database triggers and direct SQL invocation have no JavaScript active context and therefore normally produce root process spans.

When waitForEvent is implemented, a resumed execution keeps its original creation parent and links the process attempt to the matching event's creation context.

Metrics

Use the OpenTelemetry meter API directly and keep metric dimensions bounded.

Start with the current messaging metrics where their semantics apply:

• messaging.client.sent.messages
• messaging.client.consumed.messages
• messaging.process.duration
• messaging.client.operation.duration for producer and settlement operations

Metric durations must match their corresponding messaging span durations. Add pgconductor lifecycle metrics only when they can be derived from committed database outcomes, including retry, permanent failure, cancellation, and DLQ delivery. Settlement SQL must return the authoritative committed outcome before recording those values.

Never use execution ID, group/dedupe key, step name, payload, event payload, or error text as metric attributes. Do not add database-polled queue-depth gauges in this issue.

Logging

Keep the existing logger API separate. Do not implement an OpenTelemetry Logs exporter or replace application loggers.

Running task handlers under the active process context allows application logger instrumentation to add trace/span IDs. Pgconductor's child logger may also include current trace/span IDs when available, without making logging depend on an SDK.

Failure isolation

Telemetry is always fail-open:

• failures while creating spans, recording metrics, injecting/extracting context, or reading malformed metadata must not change queue behavior;
• telemetry: false disables pgconductor spans, metrics, and carrier persistence;
• it does not disable SQL spans created independently by PostgreSQL instrumentation.

Research rationale

• Trigger.dev and Inngest instrument directly inside their durable execution engines.
• Temporal composes explicit client/worker interceptors rather than overriding SDK methods.
• BullMQ instruments its producer and worker boundaries directly and persists context in job metadata.
• Hatchet's TypeScript integration uses prototype/module patching, closely matching the feat/otel approach we want to avoid.
• pg-boss, Graphile Worker, River, and Oban expose lifecycle hooks/telemetry events but do not provide first-party OpenTelemetry context propagation.
• OpenTelemetry messaging conventions recommend persisted message creation context and process-span links for batches; they permit the creation context to parent a single-message process span.

Dependencies

Implement this after the queue-scoped task identity and worker-owned settlement foundations. Final integration should follow the completed concurrency, DLQ, and event-wait creation/settlement paths so every durable transition propagates context exactly once.

Acceptance tests

• No prototype/module patching or manual registration is present.
• The library operates normally when no OpenTelemetry SDK is installed/configured.
• SDK registration after Conductor.create() but before operations is respected.
• telemetry: false emits no pgconductor telemetry and persists no carrier.
• Single invocation produces an end-to-end producer/consumer parent relationship.
• Batch processing produces one consumer span with links to each creation context.
• Retries and resumes create separate finite attempt spans.
• Sleep and durable waits do not leave spans open.
• Child, cron, event-triggered, and DLQ executions propagate context at their actual database insertion point.
• A cached step creates no step span; an executed step spans through durable persistence.
• User spans created inside handlers attach to the process span.
• Producer and settlement spans are not reported successful before their database transaction commits.
• Handler success followed by settlement failure appears as successful processing plus failed settlement.
• Throttled/no-op invocation does not increment sent-message metrics.
• Dedupe/debounce replacement updates the persisted creation context.
• Malformed and oversized carriers are ignored without failing work.
• No payload or high-cardinality metric dimensions are emitted.
• Telemetry failures cannot fail, duplicate, or lose task/event work.
• Span names, kinds, propagation, required messaging attributes, batch links, and messaging metric durations conform to the pinned OpenTelemetry messaging semantic-convention version.