Skip to content

Expand the anonymous product event catalog - #425

Merged
TheGreatAxios merged 5 commits into
mainfrom
cl-5722-expand-product-event-catalog
Aug 9, 2026
Merged

Expand the anonymous product event catalog#425
TheGreatAxios merged 5 commits into
mainfrom
cl-5722-expand-product-event-catalog

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Replaces #416, rebased onto cl-5743-batch-the-telemetry-transport (session id + batched transport).

The defect this fixes

#416 sent five user-authored identifiers to PostHog, each behind a comment asserting it was already a safe enum. The comments were false: permission_kind was the raw tool id, which for MCP tools is mcp__<server>__<tool> where <server> is the operator's own settings key; skill names, plugin ids, and agent profile ids are all project- or author-chosen. A sixth of the same class was found and fixed: slash_command's command_name came from a registry plugins also write into.

Every such value is now mapped to a fixed first-party enum at the emission site (src/telemetry/classify.ts) and reported as custom when it matches nothing, with mcp as its own bucket for permission_kind. There is no first-party list of skills or plugins to match against, so skill_used carries no name at all and plugin_loaded carries only origin — the discovery tier.

Also

  • Consumes getSessionId() from the base branch instead of re-declaring it.
  • parent_session_id dropped: sub-agents run in-process on the same session id, so it always restated session_id.
  • plugin_used dropped: it fired at startup over every enabled plugin and observed nothing about use, which plugin_loaded already measures honestly.
  • compaction no longer fires on no-ops, and its inverted trigger label is replaced by the compactor's actual mode.
  • crash reports only language-defined error types by name; an application or plugin error subclass buckets to custom.
  • The two byte-identical permission_prompt sites in the gate are one helper. They are mutually exclusive and buildRequests yields at most one request per call, so a prompt is counted once.
  • Emission takes Telemetry as an injected dependency, matching Emit PostHog AI observability events in privacy mode #417, rather than reading the process-wide handle.

Test plan

tests/unit/telemetry-product-events.test.ts drives each real emission site with a deliberately identifying input (mcp__acme-internal__deploy, a skill directory named after a company, a plugin manifest id containing a path, a project agent profile, a plugin-registered command) and asserts both the bucketed value and that the raw string appears nowhere in the serialized request body.

  • bun run typecheck — 0 errors (base: 0)
  • bun run build
  • bun run test — 4278 pass, 1 unrelated flake in src/tui-opentui/markdown-rows.test.ts (passes in isolation, no file it touches is in this diff)
  • bun test ./src ./tests ./evals --randomize — 4279 pass, 0 fail

Branch updated by rebase (authorized)

This PR branch was rebased onto main and force-updated. The pre-rebase head was 56ae4c98a. Those commits remain fetchable from that SHA.

This updates the branch behind this pull request; it does not bypass review. The PR still merges through review as normal. A plain push is refused after a rebase rewrites history, which is the only reason a force update was needed. Authorized by the operator.

Why the rebase was necessary

#412 was squash-merged to main, and #423 was squash-merged into its own base branch cl-5727-posthog-identity-session-linking after that branch had already merged. The #423 squash therefore never reached main: main carries three telemetry events, no /batch/ endpoint and no discard().

Rebasing this branch with cl-5743-batch-the-telemetry-transport as the upstream would have silently dropped the entire batching transport and the opt-out discard, without raising a conflict. The rebase instead used the top of the squashed #412 commits as the upstream, so only genuinely duplicated work was dropped. The batching commits replayed with zero conflicts, which is the evidence that this was the correct shape.

This PR is currently the only path by which the batching transport and the opt-out discard reach main. cl-5727-posthog-identity-session-linking must not be deleted until this lands.

@linear-code

linear-code Bot commented Aug 8, 2026

Copy link
Copy Markdown

CL-5722

@TheGreatAxios
TheGreatAxios changed the base branch from cl-5743-batch-the-telemetry-transport to main August 9, 2026 00:07
Two upcoming event catalogs turn telemetry volume from one event per
turn into one per tool call, which the previous transport would have
answered with an unbounded set of concurrent per-event POSTs. Queueing
behind one request bounds both sockets and memory when the endpoint is
unreachable.
Dropping the singleton on opt-out left the outgoing instance's batch
timer armed, so events captured before the toggle would still reach the
network afterwards. Opting out speaks to activity already generated, not
only to activity still to come.
Adds slash-command, skill, plugin, sub-agent, permission, compaction, and
crash events to the telemetry catalog.

Every identifier these events would naturally carry is named by someone
other than us — an MCP server key is a settings key, a skill is a directory
in the repo, a plugin id and an agent profile are author-chosen — so each is
mapped to a fixed first-party enum at the emission site and reported as
"custom" when it matches nothing. Emission takes Telemetry as an injected
dependency rather than reading the process-wide handle, so a module built
without one is silent by construction.
The auth_failure event reused error_class, which everywhere else means
the JS error constructor name. One column carrying two incompatible
meanings cannot be analysed, and it made the documented error_class
guarantee false: the value shipped was a local send-failure kind that
never passed through the classifier.
@TheGreatAxios
TheGreatAxios force-pushed the cl-5722-expand-product-event-catalog branch from 56ae4c9 to 33a7896 Compare August 9, 2026 00:34
@TheGreatAxios
TheGreatAxios merged commit b0ef2bd into main Aug 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant