feat(agents): add the error-tracking orchestrator flow - #393
Merged
Merged
Conversation
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a wizard branch:
Add Results will be posted here when complete. |
Collaborator
Agent DAG for the
|
gewenyu99
approved these changes
Sep 15, 2026
gewenyu99
left a comment
gewenyu99
left a comment
Collaborator
There was a problem hiding this comment.
Comments for your sanity, but it's your sanity so if you wanna ship as is I'm okay with it too
Collaborator
There was a problem hiding this comment.
@edwinyjlim As a part of your work, we will probably want variant mapping/guidance for agents in general, mostly in docs. This way our skills can probably become extremely simple
Contributor
Author
There was a problem hiding this comment.
Wdym? I don't get it
Contributor
Author
There was a problem hiding this comment.
Ah you tagged Edwin, not me. Sorry I'm going crazy
| effort_pi: low | ||
| model_sdk: claude-sonnet-5 | ||
| effort_sdk: medium | ||
| skills: [integration-v2-init, posthog-best-practices] |
Collaborator
There was a problem hiding this comment.
Let's fucking go @edwinyjlim @sarahxsanders @daniloc Our dream of orchestrated, reusable skills is coming alive!
| nothing populates `process.env`, `import.meta.env`, or the project's own | ||
| `src/environments/.env.ts` with your keys. | ||
|
|
||
| The trap is that each of those *looks* like a mechanism. `import.meta.env` and |
Collaborator
There was a problem hiding this comment.
This is great, but I'd move all the .env handling and init out to the shared skill. Also that if a .env is not reachable, don't fail the run report it clearly in the handoff
Contributor
Author
There was a problem hiding this comment.
I suggest a separate PR as it affects other flows? You mean to do it such that replay vision also uses that?
| Stay inside this project's directory and set up that one place; that is the | ||
| whole job. | ||
|
|
||
| ## Do not copy the SDK's own listeners |
Collaborator
There was a problem hiding this comment.
This will be very fragile. I think this info should be maintained in docs
Contributor
Author
There was a problem hiding this comment.
So our docs already explain how our auto capture works. But then agent during implementation really tried to to its best job and sometimes started hooking into things our auto capture already hooks into.
I don't think we should write in every technology doc "do not hook into x.y, it's already covered by auto capture". List here is not strict and doesn't really need to be updated. It's more of a general guidance to an agent so it can distinguish auto capture vs userful framework hooks
Nine prompts under context/agents/error-tracking/ backing the wizard's new `error-tracking` command. The seed establishes whether PostHog is integrated and which uploader variant applies, then queues install/init (integration-v2 step-skills, like replay-vision) on uninstrumented repos, exception capture via integration-v2-error-tracking-step, the source-map subgraph (adapted from the error-tracking-upload-source-maps flow) where the platform ships minified bundles or stripped binaries, and a report that merges it all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
…ness install/init move terra→sol low, capture-exceptions terra low→sol medium. Pairs with the wizard binding change routing the program to the orchestrator on pi; seed stays terra and report stays luna, matching the metrics flow's pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
…nager A configure run in a pnpm workspace copied the docs' npm install and npm hard-fails on workspace:* deps (EUNSUPPORTEDPROTOCOL); the agent retried npm three ways and never switched. New skill bullet: detect the manager from the lockfile, translate the docs' npm commands, and read EUNSUPPORTEDPROTOCOL as wrong-manager, never a flag to retry. The error-tracking flow's configure task now also mandates detect_package_manager before the first install. Verified: the pnpm monorepo fixture run that failed now completes 4/4, recovering to `pnpm add -D` on first contact with the error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
…anning A seed run saw "PostHog already integrated" and enqueued only capture+report, never applying the uploader precedence to a tsc-built Node service — while an identical sibling fixture planned the full graph. The two facts are now declared independent, a compiled/bundled JS project is explicitly never "none", and the success criteria demand the plan state the uploader decision (or the readable-stack reason) explicitly. Verified: the fixture that under-planned now queues the full node-uploader graph and completes 5/5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
The integration init skill documents its public keys in .env.example; the credentials task wrote its upload variables only to the real env file, so the names never reached the committed template. New step 4: append the names with placeholders (never values) — for the next developer, and so a later run's check_env_keys reports the expectation. Scoped to the error-tracking flow's task prompt; the shared uploader skill is untouched, so the standalone upload-source-maps program keeps its current behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
Astro inlines scripts below its asset limit into the HTML, so a build can emit a .map with no .js beside it. posthog-cli then finds zero pairs, exits non-zero, and takes the whole build down with it. Treat Astro as a no-variant platform: install and exception capture still run, the build script is left untouched, and the report says why upload was skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
…Node The shared integration-v2-init skill assumes the framework auto-loads .env, which is true for Next/Nuxt/Astro/SvelteKit/Vite but not for plain Node backends. On Express, Fastify, Hono, Koa and raw node:http the run wrote process.env reads and a .env file with nothing to bridge them, so every app died at module load on its own missing-variable guard. Teach the init task to wire the loading (dotenv, or --env-file) and to start the app once to confirm it boots. Also have configure check that some script actually runs a newly emitted bundle, since two apps built to dist/ while start still ran the source. Scoped to this flow's task prompts; the shared skill is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
…n Angular On the stock @angular/build builder nothing defines process.env or import.meta.env in the browser bundle, but the run wrote import.meta.env['NG_APP_POSTHOG_PROJECT_TOKEN'] into src/environments/*. Reading a property off undefined throws while the module evaluates, so <app-root> stayed empty and the app rendered a blank page. Name the mechanism's real source (@ngx-env/builder supplies import.meta.env and the NG_APP_ prefix) and require it to be an existing dependency before use. Otherwise fall back to the skill's existing "no valid environment to read from" case: embed the public project token in the committed environment files, which is where it ends up in the bundle anyway. Scoped to this flow's task prompt; the shared skill is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PVXmLWB31CBujD2FJ2ogvY
The earlier rule stopped the run reaching for import.meta.env, but it then read env['NG_APP_POSTHOG_PROJECT_TOKEN'] from the project's generated src/environments/.env.ts, which only defines npm_package_version. The key came back undefined, and because a production build takes the guard's quiet return path the app rendered normally while PostHog never initialised — no requests at all. Name that second empty container, require the lookup's source to be opened and the key seen before wiring it, and make the task prove initialisation by serving the build and confirming a request reaches the PostHog host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
Pointing `start` at dist/index.js is only safe if that bundle can execute. On koa-notes the esbuild command had no --format=esm while package.json sets "type": "module", so the emitted CommonJS died at boot with `ReferenceError: module is not defined in ES module scope` — a breakage that only appeared once `start` stopped running the source. Require the task to run the build and start the script once, and name the module-format mismatch as the usual cause. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
…uild rule The new "run the build and start that script once" instruction sat directly above "Do not run the build", leaving the task with contradictory orders. Carve the single verification out of the prohibition. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
The agent runs in a sandbox that cannot build or start the project, so rules telling init to boot the app and configure to run the built script were not executable. Keep the knowledge, drop the execution: the env loader and the module format must be correct by construction, checked by reading package.json's "type" and by opening the file a key lookup reads from before wiring it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
credentials and configure run in parallel: one writes the env file, the other writes the code that reads it, and neither could see the other's choice. On Nuxt 4 that split them — the skill's Nuxt reference documents variable names only for the posthog-cli path, so credentials wrote POSTHOG_CLI_* while configure wired the @posthog/nuxt module against POSTHOG_API_KEY. Nothing errored; the upload was simply skipped and every trace stayed minified. Give both tasks the same rule, in terms neither framework nor uploader specific: look for the other half of the contract before writing, adopt it verbatim if it exists, decide only when it does not, and always name the chosen set in full in the handoff. configure gets check_env_keys so it can read names without values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
…eads A repo can carry a PostHog init whose env variable was never defined — the call is there, the name it reads is nowhere, and the client is built from an empty string. The init task read that as "already initialised" and left it alone, so the app captured nothing while every other step looked clean. Init now confirms the name the code reads is present with check_env_keys before it leaves an existing init alone, and writes it when it is absent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
The seed read fact 1 as "the SDK is in the manifest", so a repo carrying an init whose key variable was defined nowhere skipped install and init entirely. The client was then built from an empty string: the build stayed clean, every capture call returned, and no event ever arrived. Fact 1 is now about the pair — the call and the key it reads. When the name is defined nowhere, or the seed cannot tell, it queues init, which re-checks the pair and leaves a complete init alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
A Next.js run copied the uploader example into next.config.ts without the non-null assertion the reference shows. process.env lookups are string | undefined, the plugin option is string, and the build failed type checking — after the upload step had already run, so the log showed a successful chunk upload and the build still exited 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
…dary wire-ci was written entirely around credentials, and pointed at the skill's CI step while the git-context guidance sits in a different step the task never named. In a container build neither reaches the uploader: .git is in .dockerignore, and the CI's own variables stop at the docker build command. Reproduced against the nested composite action fixture: Release fields are incomplete and git info is unavailable (exit 1) Docker build args are not inherited from the environment, so this fails on a GitHub Actions runner exactly as it does locally — verified by building with GITHUB_SHA exported but not forwarded. The ARG-plus-ENV shape is scoped to the git variables and non-secret settings; the API key keeps its build-secret mount, which a first pass had demoted to a plain build arg. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
…ndary wire-ci carried the upload credentials through the pipeline but nothing owned the variable the deploy step hands the running app. The step predates PostHog, so it passed the project's old name while init had settled on another, and the container booted with an undefined key. Reproduced by running the image exactly as the deploy step does: POSTHOG_PROJECT_TOKEN= undefined NODE_ENV= production exit=0 Nothing fails: in production the init guard returns rather than throwing, so the process serves traffic and reports nothing. The runtime boundary is now an explicit third place a name has to match, with the deploy step renamed to follow the code rather than the other way round. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
The release-identity rule was written as if a pipeline always exists, so a project whose Dockerfile says it is built and run by hand got eight GITHUB_* build args nothing ever fills. Empty variables resolve to no release, and a hand-run docker build failed: Oops! Release fields are incomplete and git info is unavailable (exit 1) Reading as wired while failing the build is the worst of both. With no provider the identity is supplied outright instead — the re-run took --release-version from the manifest, and the hand-run build resolves cicd-docker-node-raw@0.1.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
…s on The Go run wrote credentials to .env, rewrote main.go to read them from the process env, added a Makefile run target that loads nothing, and then pointed the user at that target to verify. It prints its own "variable missing" guard and exits: build green, symbols uploaded, no event. The run-script rule now covers reaching the values as well as running the right artifact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
error-tracking-step borrowed the integration group's variants, so the task that wires exception capture read the SDK library page. For Go and Rust those pages say nothing about panic hooks or log handlers, and the task invented the API: a capture_panics method on the wrong builder, and a slog handler with no distinct-ID resolver, which drops every capture without erroring. The matrix is unchanged - same 40 ids, frameworks and defaults, verified against the built menu before and after. Only docs_urls move: 39 variants now point at their error-tracking installation page, java being the one PostHog has none for. Where the page is a generic substitute (django to python, astro to web) the framework's own library page stays alongside. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
The Rust page shows a global initializer and an explicit client side by side. The task took the wiring from one and the calls from the other, binding init_global's Result<(), Error> as if it were a client, so cargo build failed with three E0599s. It also wrote the build as a !-prefixed cargo alias, which is a git convention cargo has no support for. Six lines: follow one setup the whole way through, and put a multi-step build where the tool actually runs it. The re-run compiles, uploads symbols, and a panic resolves to rust_app::kaboom_rust2 -> read_rust_retry_budget2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
The capture step's Nuxt matrix marked `nuxt-3-6` as the default, so every Nuxt project — including Nuxt 4 — was handed the v3.0-3.6 installation page and wired PostHog by hand through a client plugin. Nuxt 3.7 shipped in September 2023; the `@posthog/nuxt` module is the documented path for everything since, and it initialises both the Vue and Nitro sides itself. Moves `default: true` to the `nuxt-4` variant, which already pointed at installation/nuxt-3-7.md. Two lines change in the built menu and nothing else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5Kw3YahYz9RiFeJ8r2hxY
Exception autocapture already registers the global handlers — window.onerror and unhandledrejection in the browser, uncaughtException and unhandledRejection in Node, sys.excepthook in Python, the panic hook in Rust, the uncaught exception and signal handlers on iOS and Android. A second listener the task writes itself on the same event sends every error twice and drifts from the SDK's handling as the SDK changes. The capture task now turns autocapture on through the SDK's option and registers none of those handlers. Framework hooks stay in scope, because they see errors a framework swallows before any global handler fires: Express error middleware, Fastify setErrorHandler, Hono onError, Vue errorHandler, Angular ErrorHandler, SvelteKit handleError, a React error boundary, Next.js global-error. Go, which has no autocapture, keeps its entry-point boundary. The success check gains the same condition, so a reviewer can confirm it by reading the diff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SEsZQQrYjXPVEhQuoSwE6
error-tracking-step declares its variants literally, and the orchestrator resolves it per framework with no fallback. A variant added to integration and not copied here would abort the default orchestrator flow for that framework. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SEsZQQrYjXPVEhQuoSwE6
- model_sdk: claude-sonnet-5 and claude-haiku-4-5, as #389 did for the other flows. The old ids are no longer valid in the wizard. - install and init run terra at low effort, like integration-v2 and replay-vision. wire-ci runs sol at medium. The gateway mint pins sol to medium. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SEsZQQrYjXPVEhQuoSwE6
A React Native repo carries ios/Podfile, an .xcodeproj and Gradle files, so the seed's native-first precedence picked the ios uploader and wired no JavaScript source maps. react-native or expo in package.json now wins over every native marker. Verified with e2e runs on the react-native fixture (ios before, react-native after) and on react-native-expo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
In a type-checked project the init guard tested a separate variable and then passed the unchecked one to the SDK, so tsc stopped the build with TS2345 and no source maps uploaded. The init now passes the variable it checked, or keeps an existing ?? '' / ! as it is. Verified with e2e runs on node-webpack and node-raw: no type errors, and node-raw builds, uploads and starts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
With sourcemaps.enabled the webpack plugin adds its own SourceMapDevToolPlugin. A devtool that also emits maps makes webpack fail with "Conflict: Multiple assets emit different content to the same filename index.js.map", and nothing uploads. Verified with an e2e run on node-webpack: no devtool, the build passes, one chunk uploads, and the app starts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
The flow relies on SDK autocapture and does not change team settings. Remove the products-enable step and posthog_exec from the report task. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
A Gradle file alone matched `android`, so a Gradle JVM server got the Android upload tasks. Require an Android Gradle plugin or an AndroidManifest.xml. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
Review feedback: low effort is too little for the variety of real projects these two tasks meet. The gateway already admits terra at medium. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
Review feedback: test-setup ran before the report, so a user who stepped away waited on its question before the report was written. The report is now a plain task and test-setup is the sink after it. The question says setup is done, the report is written, and the test is optional. No answer counts as a decline, and a failed test is added to the report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
Review feedback: the wizard's pre-install warning is a status line that scrolls away. The configure task now checks that a machine-global posthog-cli is on the PATH when its uploader needs one, and names the install as a follow-up in its handoff. The report lists it under what the user still has to do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
ablaszkiewicz
force-pushed
the
error-tracking-agent-flow
branch
from
September 15, 2026 15:50
a3b8d62 to
0886e3b
Compare
The llm-gateway retired its legacy wizard product (PostHog/posthog#96861), which now refuses every caller with a 403. Every triage call failed, so each Warlock match defaulted to a true positive and the build blocked. The ci product serves CI runs with a personal API key and allows all models. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of a 3-PR set
Problem
wizard error-tracking(PostHog/wizard#1185) needs an orchestrator flow. The flow installs PostHog first when the repo has no PostHog.Changes
New flow
context/agents/error-tracking/, a seed and 8 tasks:setup-error-tracking(seed): checks the PostHog init and its key, and picks the source-map uploader. React Native and Expo usereact-native. Readable-stack platforms and Astro use none.install,init: add and initialize the SDK.initkeeps the env names that the code reads, loads.envwhere the platform does not, and writes init code that type-checks.capture-exceptions: turns on SDK autocapture and hooks the framework error handler. It adds no second global listener.credentials,configure,wire-ci,test-setup: wire the source-map upload into the build and CI. The API key stays a secret.report: writesposthog-error-tracking-report.md.Shared skill changes:
integration-v2/error-tracking-step: 40 variants with the error-tracking docs. The default integration-v2 flow uses them too. A new test keeps the variants in sync withintegration.error-tracking-upload-source-maps: install with the project's package manager. Do not set a webpackdevtoolbeside the PostHog plugin.Test plan
node scripts/build.jspasses.pnpm testpasses (174 tests).🤖 Generated with Claude Code
https://claude.ai/code/session_01668rN31F4aotxmenNksz4Z