Skip to content

Batch the telemetry transport - #423

Merged
TheGreatAxios merged 4 commits into
cl-5727-posthog-identity-session-linkingfrom
cl-5743-batch-the-telemetry-transport
Aug 9, 2026
Merged

Batch the telemetry transport#423
TheGreatAxios merged 4 commits into
cl-5727-posthog-identity-session-linkingfrom
cl-5743-batch-the-telemetry-transport

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Replaces the per-event telemetry transport with a queued, batched one.

Why this has to land first

The transport today opens one fetch POST to /capture/ per event, with no
queue, no cap, and no concurrency limit. That was tolerable while the event
catalog was O(turns). Two PRs about to land make it O(tool calls): one adds a
permission-prompt event per approval plus subagent start/end and compaction
events, the other adds a generation event per turn plus a span event per tool
call per turn.

A 100-turn session at five tool calls per turn is roughly 700 individual
HTTPS POSTs. A single 40-tool-call turn fires 41 concurrent fetches. Against a
hung endpoint - captive portal, corporate proxy, provider degradation - that
is hundreds of concurrent in-flight sockets and an unbounded set of pending
promises on a user's laptop, silently, for the length of the session. Either
of those PRs merged ahead of this one ships that behavior.

What changed

  • Events go to PostHog's /batch/ endpoint: { api_key, batch: [{ event, timestamp, properties }] }, with distinct_id inside each event's
    properties. Each event carries the timestamp of its capture, so batching
    does not smear event times.
  • An in-memory queue with both triggers: it is sent when it reaches the batch
    size, or when the batch interval elapses, whichever comes first.
  • A hard queue depth limit. On overflow the oldest events are dropped. When
    the endpoint is unreachable the oldest queued events are also the least
    worth reporting, and unbounded growth is not an acceptable alternative.
  • At most one request in flight. A drain loop owns the queue; events captured
    mid-flight are picked up by that loop's next iteration instead of opening a
    second socket.
  • flush() keeps its existing 500ms deadline and drains through the same
    single-request path, so the fatal-crash handler still cannot be held up.
  • Failures remain silent to the user, exactly as before, and the injected
    fetchFn seam is unchanged.

One behavioral consequence worth naming: toggling telemetry off discards the
outgoing instance along with anything still queued on it. That is the right
direction for an opt-out.

Tests

New coverage for the size trigger, the time trigger, drop-oldest on overflow,
single-in-flight serialization, flush() draining a partial queue inside its
deadline, and the /batch/ endpoint itself.

Separately, a test for the combined failure: the queue filling while a
request is hung. Drop-oldest and single-in-flight have to hold at the same
time - the queue caps and sheds oldest while one request is stuck and no
second request opens. That is the case that bites a user on a captive portal,
and a design can satisfy both rules independently and still fail it.

Two upcoming event catalogs turn telemetry volume from one event per
turn into one per tool call, which the previous transport would have
answered with an unbounded set of concurrent per-event POSTs. Queueing
behind one request bounds both sockets and memory when the endpoint is
unreachable.
@linear-code

linear-code Bot commented Aug 8, 2026

Copy link
Copy Markdown

CL-5743

Dropping the singleton on opt-out left the outgoing instance's batch
timer armed, so events captured before the toggle would still reach the
network afterwards. Opting out speaks to activity already generated, not
only to activity still to come.
The only existing deadline test resolved its fetch immediately, so it
would have kept passing even with the deadline race deleted. The new
test gates the fetch open forever and asserts flush() still returns,
proving the race is load-bearing. Also documents that a batch already
on the wire when a user opts out cannot be recalled, matching what the
code comment already says.
@TheGreatAxios
TheGreatAxios merged commit 100244b into cl-5727-posthog-identity-session-linking Aug 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant