From 7df613bda8d42e967468c2327355dd9f6fb10e11 Mon Sep 17 00:00:00 2001 From: sarahxsanders Date: Wed, 9 Sep 2026 10:14:41 -0400 Subject: [PATCH 1/6] feat: include AIO and Logs in default Wizard runs --- .../agents/integration-v2/ai-observability.md | 47 ++++++++++++++ .../integration-v2/integrate-posthog.md | 12 +++- context/agents/integration-v2/logs.md | 47 ++++++++++++++ context/agents/integration-v2/report.md | 3 +- context/agents/integration-v2/review.md | 2 +- .../integration-v2/report/description.md | 5 ++ .../skills/integration/references/2-edit.md | 31 +++++++++ .../skills/integration/references/3-revise.md | 5 ++ .../integration/references/4-conclude.md | 7 ++ .../integration-observability-flow.test.js | 65 +++++++++++++++++++ 10 files changed, 220 insertions(+), 4 deletions(-) create mode 100644 context/agents/integration-v2/ai-observability.md create mode 100644 context/agents/integration-v2/logs.md create mode 100644 scripts/lib/tests/integration-observability-flow.test.js diff --git a/context/agents/integration-v2/ai-observability.md b/context/agents/integration-v2/ai-observability.md new file mode 100644 index 00000000..436bdde0 --- /dev/null +++ b/context/agents/integration-v2/ai-observability.md @@ -0,0 +1,47 @@ +--- +type: ai-observability +flow: integration-v2 +label: Add AI Observability +model_pi: openai/gpt-5.6-terra +effort_pi: medium +model_sdk: claude-sonnet-5 +effort_sdk: high +skills: [] +allowedTools: [Read, Write, Edit, Glob, Grep, load_skill_menu, install_skill] +disallowedTools: [enqueue_task] +dependsOn: [capture, error-tracking] +--- + +## Goal + +Add AI Observability to the LLM calls this project already makes. Read the +upstream handoffs and inspect the call sites, not just the dependency names. +If there are no LLM calls, complete as `not needed` with the reason and make no +changes. Do not install a vendor SDK, add a demo call, or use manual capture to +invent an AI feature in an app that has none. + +For existing LLM calls, load the `ai-observability` skill menu and install the +variant matching the calling code's language and provider or agent framework. +Follow that skill's selection rules and instrumentation references. Manual +capture is appropriate only for existing calls without a supported wrapper. +Reuse the PostHog client and identity established earlier in this run. If a +variant cannot be chosen from the code, report the ambiguity as `not needed`; +this default run does not ask the user to choose a provider. + +This is the instrumentation part of a larger integration. Declare dependencies +in the manifest; the following review task installs them and checks imports, +builds, and lint. Defer the skill's package-manager and import-check commands +to that task. For Go, leave the required modules in the handoff for review to +add with `go get` and `go mod tidy`; do not hand-edit `go.mod` or `go.sum`. +Do not run the app or make paid LLM calls. Use `check_env_keys` and +`set_env_values` for environment files, using the project credentials supplied +by the wizard. + +## How you know you succeeded + +Existing LLM calls are wired using the selected skill, or the handoff clearly +explains why this task was not needed. List the variant, changed files, declared +dependencies, and a concrete call path the user can trigger to verify the +session/trace/generation tree. Pass any deferred dependency work to review. +Describe delivery as unverified unless this run actually observed it; do not +publish a separate setup report. diff --git a/context/agents/integration-v2/integrate-posthog.md b/context/agents/integration-v2/integrate-posthog.md index 9628e14c..c85202e9 100644 --- a/context/agents/integration-v2/integrate-posthog.md +++ b/context/agents/integration-v2/integrate-posthog.md @@ -23,11 +23,19 @@ Plan a PostHog integration and seed the task queue with this graph: the SDK installed and initialized, not the events. - `capture`, after `identify` — it decides the events and instruments them, and it reads how identity is already established before it instruments anything. -- `review`, after `install`, `init`, `identify`, `capture`, and `error-tracking` — +- `ai-observability`, after `capture` and `error-tracking` — it instruments existing + LLM calls, or reports that none apply. Queue it on every default run; the task + checks applicability rather than the planner guessing from a package name. +- `logs`, after `ai-observability` — it configures log capture on supported + platforms, or reports why it does not apply. Queue it even when AIO will be + skipped. These tasks run sequentially because they can edit the same dependency + manifests and initialization files as the earlier instrumentation tasks. +- `review`, after `install`, `init`, `identify`, `capture`, `error-tracking`, + `ai-observability`, and `logs` — it installs the dependencies, verifies the project builds/typechecks/lints, and reviews every change the run made, fixing what fails. There is no separate build step: verifying and reviewing are one pass over the same changeset. -- `dashboard`, after `capture`, parallel to `review` — it builds insights from the +- `dashboard`, after `capture`, independent of AIO, Logs, and `review` — it builds insights from the instrumented events, which `capture` has already defined; it needs no code review. - `report`, after `dashboard` **and** `review` — it writes the setup report last, so it describes the integration as reviewed rather than as first written. diff --git a/context/agents/integration-v2/logs.md b/context/agents/integration-v2/logs.md new file mode 100644 index 00000000..9e0863a1 --- /dev/null +++ b/context/agents/integration-v2/logs.md @@ -0,0 +1,47 @@ +--- +type: logs +flow: integration-v2 +label: Add log capture +model_pi: openai/gpt-5.6-terra +effort_pi: medium +model_sdk: claude-sonnet-5 +effort_sdk: high +skills: [] +allowedTools: [Read, Write, Edit, Glob, Grep, load_skill_menu, install_skill] +disallowedTools: [enqueue_task] +dependsOn: [ai-observability] +--- + +## Goal + +Configure PostHog log capture for this project's supported runtimes. Load the +`logs` skill menu, install its matching skill, and read the platform reference +before editing. Inspect the runtime and existing logging setup; use the +documented OTLP exporter or SDK-native log capture for that platform. A +browser-only app must not receive a server exporter. If no documented setup +applies, complete as `not needed` and explain why. + +Reuse any existing PostHog log export and leave its handlers and outputs +intact. If capture is already configured, report that without adding a second +exporter. Preserve any tracing provider AIO configured in the previous step. +Keep changes to log setup and existing logging paths; do not scatter new logs +through unrelated code or export secrets, request bodies, or user data. + +This is the instrumentation part of a larger integration. Instead of the +standalone skill's install command, declare required packages in the manifest +and leave installation, lockfile updates, builds, and lint to review. For +manifests managed by tooling (such as Go), pass the required packages to review +instead of editing them by hand. Use `detect_package_manager` if needed. Use +the wizard's supplied project and region and the env names from upstream +handoffs; inspect and write environment files only through `check_env_keys` +and `set_env_values`. Do not request credentials or select a different project. +If required configuration is unavailable, report what is missing instead of +guessing a region. + +## How you know you succeeded + +Log capture is configured once for the supported runtime, was already present, +or has a clear skip reason. The handoff names the changed files, packages, env +variable names (never values), and any dependency work for review. Give a +specific existing log path the user can trigger and find in PostHog Logs; +delivery remains unverified until observed. Do not publish a separate report. diff --git a/context/agents/integration-v2/report.md b/context/agents/integration-v2/report.md index 54e4e0e9..94b6f6b9 100644 --- a/context/agents/integration-v2/report.md +++ b/context/agents/integration-v2/report.md @@ -45,7 +45,8 @@ before those events mean anything. One `publish_handoff` call went through with the full report: what was installed and initialized, the events captured, whether identify was wired or -skipped, error tracking added, the dashboard link, any build conflict in full, +skipped, error tracking added, AI Observability and Logs configured or skipped +with reasons, the dashboard link, any build conflict in full, and the next steps for the user. Every claim in it traces to a handoff, and what the run could not confirm reads as unconfirmed. The same report is mirrored into a PostHog notebook whose URL is emitted with the diff --git a/context/agents/integration-v2/review.md b/context/agents/integration-v2/review.md index dde182e0..a329dc5a 100644 --- a/context/agents/integration-v2/review.md +++ b/context/agents/integration-v2/review.md @@ -9,7 +9,7 @@ effort_sdk: high skills: [posthog-best-practices, integration-v2-build] allowedTools: [Read, Edit, Glob, Grep, Bash] disallowedTools: [enqueue_task] -dependsOn: [install, init, identify, error-tracking, capture] +dependsOn: [install, init, identify, error-tracking, capture, ai-observability, logs] --- ## Goal diff --git a/context/skills/integration-v2/report/description.md b/context/skills/integration-v2/report/description.md index 0612d11d..06271d5b 100644 --- a/context/skills/integration-v2/report/description.md +++ b/context/skills/integration-v2/report/description.md @@ -26,6 +26,9 @@ Draw on two sources only: (from `.posthog-wizard-cache/.posthog-events.json`). - Whether user identification was wired or skipped, and why. - The error tracking added. +- AI Observability and Logs: what was configured or already present, or why each + was skipped. Use the task handoffs, including changed files and outstanding + verification; do not treat a skipped task as a successful setup. - The dashboard link. - Any build conflict, in full. - Clear next steps for the user. @@ -46,6 +49,8 @@ code changed this run and drop the ones that don't fit: for CSP violations — a blocked SDK queues events silently and never sends. - If LLM analytics was set up: trigger the instrumented call path and confirm `$ai_generation` events appear in PostHog. +- If Logs was set up: trigger the log path named in its handoff and confirm the + entry appears in PostHog Logs with the expected service and severity. - If auth exists and identify was wired: the returning-visitor path also calls identify, so returning sessions don't fragment onto anonymous distinct IDs. diff --git a/context/skills/integration/references/2-edit.md b/context/skills/integration/references/2-edit.md index e68d304f..2d9e3d60 100644 --- a/context/skills/integration/references/2-edit.md +++ b/context/skills/integration/references/2-edit.md @@ -20,6 +20,37 @@ It's essential to do this in both client code and server code, so that user beha You should also add PostHog exception capture error tracking to these files where relevant. +## AI Observability and Logs + +Include both in a default integration where applicable, after the SDK and event +instrumentation above and before the revise and conclusion steps. Run AIO first, +then Logs, so edits to manifests and initialization files do not conflict. + +1. **AI Observability:** inspect existing LLM call sites. If there are none, skip + AIO and record that reason; do not add a vendor SDK or artificial model calls. + Otherwise call `load_skill_menu` with `category: "ai-observability"`, install + the matching variant with `install_skill`, and follow its references. Choose + the provider/framework and language from the actual calling code. Use manual + capture only for existing calls without a supported wrapper. Reuse the client + and identity already established. If the provider remains ambiguous, skip with + that reason instead of asking a question during the default run. +2. **Logs:** call `load_skill_menu` with `category: "logs"`, install the matching + skill, and follow the documented setup for this runtime. Use SDK-native log + capture where documented, otherwise the platform's OTLP exporter. Do not add a + server exporter to browser-only code. If no documented setup applies, skip with + the reason. Preserve existing handlers, outputs, and any AIO tracing provider; + an existing PostHog exporter needs no duplicate. Keep edits to log setup and + existing logging paths, without adding logs to unrelated code. + +For both skills, use the project credentials and region supplied by the wizard. +Inspect and change environment files only through `check_env_keys` and +`set_env_values`, reusing existing variable names. Do not ask for credentials or +guess a region. Follow this runtime's tool restrictions; defer any dependency +installation or verification still needed to the revise step. Do not make paid +LLM calls or claim delivery based on code changes. Keep each result (configured, +already present, or skipped with a reason) for the final report, including a +concrete path the user can trigger to check delivery. + Remember: Do not alter the fundamental architecture of existing files. Make your additions minimal and targeted. Remember the documentation and example project resources you were provided at the beginning. Read them now. diff --git a/context/skills/integration/references/3-revise.md b/context/skills/integration/references/3-revise.md index 1d0f09ae..edcc2494 100644 --- a/context/skills/integration/references/3-revise.md +++ b/context/skills/integration/references/3-revise.md @@ -8,6 +8,11 @@ Check the project for errors. Read the package.json file for any type checking o Ensure that any components created were actually used. +Install any dependencies declared by the AI Observability and Logs steps before +verification, using the project's package manager. Include their changed files +in this review and check that Logs preserved any AIO tracing setup. Verification +of delivery belongs to the user's checklist; do not make paid LLM calls. + Once all other tasks are complete, run any linter or prettier-like scripts found in the package.json, but ONLY on the files you have edited or created during this session. Do not run formatting or linting across the entire project's codebase. ## Status diff --git a/context/skills/integration/references/4-conclude.md b/context/skills/integration/references/4-conclude.md index 6825e4f7..a28b7f96 100644 --- a/context/skills/integration/references/4-conclude.md +++ b/context/skills/integration/references/4-conclude.md @@ -78,6 +78,11 @@ Do not spawn subagents. Compose the setup report as markdown — do NOT write it to a file in the project. It should include a summary of the integration edits, a table with the event names, event descriptions, and files where events were added, a list of links for the dashboard and insights created, and a "Verify before merging" checklist (see below). Follow this format: +Include separate AI Observability and Logs results: what was configured, what was +already present, or why each was skipped. Name the changed files and distinguish +code verification from observed delivery. A skipped product is not a successful +setup. + # PostHog post-wizard report @@ -108,6 +113,8 @@ For the "Verify before merging" checklist, write GitHub-style checkboxes (`- [ ] - If you added environment variables: "Add the exact PostHog env var names you added to `.env.example` and any monorepo/bootstrap scripts so collaborators know what to set." - If this integration ships a minified production browser bundle (most SPA/SSR web frameworks — e.g. Next.js, Nuxt, SvelteKit, Astro, Vite-based apps): "Wire source-map upload (`posthog-cli sourcemap` or your bundler's upload step) into CI so production stack traces de-minify." - If LLM analytics was set up in this run: "Trigger the LLM call path(s) you instrumented and confirm `$ai_generation` events appear in PostHog AI Observability." +- If Logs was set up in this run: trigger the existing log path you configured and + confirm the entry appears in PostHog Logs with the expected service and severity. - If the app has user auth and an `identify` call was added: "Confirm the returning-visitor path also calls `identify` — a handler that only identifies on fresh login can leave returning sessions on anonymous distinct IDs." Do not invent items beyond what applies. If only the two "Always" items apply, the checklist is just those two. diff --git a/scripts/lib/tests/integration-observability-flow.test.js b/scripts/lib/tests/integration-observability-flow.test.js new file mode 100644 index 00000000..11424531 --- /dev/null +++ b/scripts/lib/tests/integration-observability-flow.test.js @@ -0,0 +1,65 @@ +import { afterEach, describe, expect, it } from 'vitest'; +import { mkdtempSync, readFileSync, readdirSync, rmSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import matter from 'gray-matter'; + +import { buildAgents } from '../agent-generator.js'; +import { expandSkillGroups, loadSkillsConfig } from '../skill-generator.js'; + +const configDir = join(process.cwd(), 'context'); +const flowDir = join(configDir, 'agents', 'integration-v2'); +const prompts = readdirSync(flowDir) + .filter((file) => file.endsWith('.md')) + .map((file) => matter(readFileSync(join(flowDir, file), 'utf8')).data); +const byType = new Map(prompts.map((prompt) => [prompt.type, prompt])); + +function ancestors(type, visiting = new Set()) { + if (visiting.has(type)) throw new Error(`Dependency cycle at ${type}`); + const prompt = byType.get(type); + if (!prompt) throw new Error(`Unknown dependency ${type}`); + const next = new Set([...visiting, type]); + return new Set((prompt.dependsOn ?? []).flatMap((dep) => [dep, ...ancestors(dep, next)])); +} + +describe('default integration observability flow', () => { + let distDir; + afterEach(() => { + if (distDir) rmSync(distDir, { recursive: true, force: true }); + }); + + it('publishes both tasks with tools that can discover their existing product skills', () => { + distDir = mkdtempSync(join(tmpdir(), 'integration-observability-')); + buildAgents({ configDir, distDir, baseUrl: 'https://example.test' }); + const menu = JSON.parse(readFileSync(join(distDir, 'agents', 'agent-menu.json'), 'utf8')); + const skills = expandSkillGroups(loadSkillsConfig(configDir), configDir); + + for (const type of ['ai-observability', 'logs']) { + const entry = menu.agents.find((agent) => agent.flow === 'integration-v2' && agent.id === type); + expect(entry).toBeDefined(); + const asset = entry.downloadUrl.split('/').pop(); + const { data } = matter(readFileSync(join(distDir, 'agents', asset), 'utf8')); + expect(data.allowedTools).toEqual(expect.arrayContaining(['load_skill_menu', 'install_skill'])); + expect(data.allowedTools).not.toContain('wizard_ask'); + expect(data.allowedTools).not.toContain('Bash'); + expect(data.seed).not.toBe(true); + expect(data.runnerSeeded).not.toBe(true); + expect(skills.some((skill) => skill._category === type)).toBe(true); + } + }); + + it('serializes the new writers after existing instrumentation and before review', () => { + for (const type of ['install', 'init', 'identify', 'capture', 'error-tracking']) { + expect(ancestors('ai-observability').has(type)).toBe(true); + } + expect(ancestors('logs').has('ai-observability')).toBe(true); + const reviewed = ancestors('review'); + for (const prompt of prompts.filter((p) => p.type !== 'review' && p.allowedTools?.some((t) => ['Write', 'Edit'].includes(t)))) { + expect(reviewed.has(prompt.type), `${prompt.type} must finish before review`).toBe(true); + } + expect(ancestors('report').has('logs')).toBe(true); + expect(ancestors('report').has('ai-observability')).toBe(true); + expect(byType.get('report').sink).toBe(true); + expect(ancestors('dashboard').has('logs')).toBe(false); + }); +}); From f89a46377eff54c4b0e60ff99cdda74bac49d990 Mon Sep 17 00:00:00 2001 From: Edwin Lim Date: Fri, 18 Sep 2026 15:48:53 -0700 Subject: [PATCH 2/6] move aio + logs parallel to error tracking --- context/agents/integration-v2/ai-observability.md | 2 +- context/agents/integration-v2/integrate-posthog.md | 6 +++--- scripts/lib/tests/integration-observability-flow.test.js | 4 +++- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/context/agents/integration-v2/ai-observability.md b/context/agents/integration-v2/ai-observability.md index 436bdde0..b7adc485 100644 --- a/context/agents/integration-v2/ai-observability.md +++ b/context/agents/integration-v2/ai-observability.md @@ -9,7 +9,7 @@ effort_sdk: high skills: [] allowedTools: [Read, Write, Edit, Glob, Grep, load_skill_menu, install_skill] disallowedTools: [enqueue_task] -dependsOn: [capture, error-tracking] +dependsOn: [capture] --- ## Goal diff --git a/context/agents/integration-v2/integrate-posthog.md b/context/agents/integration-v2/integrate-posthog.md index c85202e9..a2b24b09 100644 --- a/context/agents/integration-v2/integrate-posthog.md +++ b/context/agents/integration-v2/integrate-posthog.md @@ -23,9 +23,9 @@ Plan a PostHog integration and seed the task queue with this graph: the SDK installed and initialized, not the events. - `capture`, after `identify` — it decides the events and instruments them, and it reads how identity is already established before it instruments anything. -- `ai-observability`, after `capture` and `error-tracking` — it instruments existing - LLM calls, or reports that none apply. Queue it on every default run; the task - checks applicability rather than the planner guessing from a package name. +- `ai-observability`, after `capture`, parallel to `error-tracking` — it instruments + existing LLM calls, or reports that none apply. Queue it on every default run; the + task checks applicability rather than the planner guessing from a package name. - `logs`, after `ai-observability` — it configures log capture on supported platforms, or reports why it does not apply. Queue it even when AIO will be skipped. These tasks run sequentially because they can edit the same dependency diff --git a/scripts/lib/tests/integration-observability-flow.test.js b/scripts/lib/tests/integration-observability-flow.test.js index 11424531..fb33eb55 100644 --- a/scripts/lib/tests/integration-observability-flow.test.js +++ b/scripts/lib/tests/integration-observability-flow.test.js @@ -49,9 +49,11 @@ describe('default integration observability flow', () => { }); it('serializes the new writers after existing instrumentation and before review', () => { - for (const type of ['install', 'init', 'identify', 'capture', 'error-tracking']) { + for (const type of ['install', 'init', 'identify', 'capture']) { expect(ancestors('ai-observability').has(type)).toBe(true); } + // error-tracking runs parallel to AIO; review still waits on both. + expect(ancestors('ai-observability').has('error-tracking')).toBe(false); expect(ancestors('logs').has('ai-observability')).toBe(true); const reviewed = ancestors('review'); for (const prompt of prompts.filter((p) => p.type !== 'review' && p.allowedTools?.some((t) => ['Write', 'Edit'].includes(t)))) { From 53b14a88059bbd48d1d0987a9355fbe93be76152 Mon Sep 17 00:00:00 2001 From: Edwin Lim Date: Fri, 18 Sep 2026 17:27:35 -0700 Subject: [PATCH 3/6] optional field for agent tasks --- context/agents/integration-v2/ai-observability.md | 1 + context/agents/integration-v2/logs.md | 1 + context/agents/integration-v2/report.md | 4 ++-- context/agents/integration-v2/review.md | 5 ++++- context/skills/integration-v2/report/description.md | 5 +++-- scripts/lib/tests/integration-observability-flow.test.js | 2 ++ 6 files changed, 13 insertions(+), 5 deletions(-) diff --git a/context/agents/integration-v2/ai-observability.md b/context/agents/integration-v2/ai-observability.md index b7adc485..b793015c 100644 --- a/context/agents/integration-v2/ai-observability.md +++ b/context/agents/integration-v2/ai-observability.md @@ -10,6 +10,7 @@ skills: [] allowedTools: [Read, Write, Edit, Glob, Grep, load_skill_menu, install_skill] disallowedTools: [enqueue_task] dependsOn: [capture] +optional: true --- ## Goal diff --git a/context/agents/integration-v2/logs.md b/context/agents/integration-v2/logs.md index 9e0863a1..8d0c3e57 100644 --- a/context/agents/integration-v2/logs.md +++ b/context/agents/integration-v2/logs.md @@ -10,6 +10,7 @@ skills: [] allowedTools: [Read, Write, Edit, Glob, Grep, load_skill_menu, install_skill] disallowedTools: [enqueue_task] dependsOn: [ai-observability] +optional: true --- ## Goal diff --git a/context/agents/integration-v2/report.md b/context/agents/integration-v2/report.md index 94b6f6b9..98528dfb 100644 --- a/context/agents/integration-v2/report.md +++ b/context/agents/integration-v2/report.md @@ -45,8 +45,8 @@ before those events mean anything. One `publish_handoff` call went through with the full report: what was installed and initialized, the events captured, whether identify was wired or -skipped, error tracking added, AI Observability and Logs configured or skipped -with reasons, the dashboard link, any build conflict in full, +skipped, error tracking added, AI Observability and Logs configured, skipped, or +failed with reasons, the dashboard link, any build conflict in full, and the next steps for the user. Every claim in it traces to a handoff, and what the run could not confirm reads as unconfirmed. The same report is mirrored into a PostHog notebook whose URL is emitted with the diff --git a/context/agents/integration-v2/review.md b/context/agents/integration-v2/review.md index a329dc5a..9a5b8661 100644 --- a/context/agents/integration-v2/review.md +++ b/context/agents/integration-v2/review.md @@ -20,7 +20,10 @@ reading of the same changeset. **Install and verify.** The earlier steps edited code and declared the SDK in the manifest but did not install it — install now, then verify the project builds, -typechecks, and lints, whichever of those it defines. Do not run the test suite; the +typechecks, and lints, whichever of those it defines. An optional upstream task +(AI Observability, Logs) may have failed and left partial edits no handoff +describes — install whatever the manifest declares either way, and let the build +surface stray imports; fix or revert them like any other defect. Do not run the test suite; the runtime does not allow it, and a green build is the bar. A bad version only surfaces here: if the install fails because a declared version does not exist (npm `ETARGET`, "no matching version", a yanked release), fix the manifest to a real published diff --git a/context/skills/integration-v2/report/description.md b/context/skills/integration-v2/report/description.md index 06271d5b..7e362384 100644 --- a/context/skills/integration-v2/report/description.md +++ b/context/skills/integration-v2/report/description.md @@ -27,8 +27,9 @@ Draw on two sources only: - Whether user identification was wired or skipped, and why. - The error tracking added. - AI Observability and Logs: what was configured or already present, or why each - was skipped. Use the task handoffs, including changed files and outstanding - verification; do not treat a skipped task as a successful setup. + was skipped or failed. Use the task handoffs, including changed files and + outstanding verification; do not treat a skipped or failed task as a + successful setup. - The dashboard link. - Any build conflict, in full. - Clear next steps for the user. diff --git a/scripts/lib/tests/integration-observability-flow.test.js b/scripts/lib/tests/integration-observability-flow.test.js index fb33eb55..0050dcc5 100644 --- a/scripts/lib/tests/integration-observability-flow.test.js +++ b/scripts/lib/tests/integration-observability-flow.test.js @@ -44,6 +44,8 @@ describe('default integration observability flow', () => { expect(data.allowedTools).not.toContain('Bash'); expect(data.seed).not.toBe(true); expect(data.runnerSeeded).not.toBe(true); + // Optionality lives in the task definition; the planner has no say. + expect(data.optional).toBe(true); expect(skills.some((skill) => skill._category === type)).toBe(true); } }); From 89932e108ec6c677bd69e7acb346255868108815 Mon Sep 17 00:00:00 2001 From: Edwin Lim Date: Fri, 18 Sep 2026 19:36:33 -0700 Subject: [PATCH 4/6] comment --- context/skills/ai-observability/config.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/context/skills/ai-observability/config.yaml b/context/skills/ai-observability/config.yaml index 793695ec..c9c1ef0f 100644 --- a/context/skills/ai-observability/config.yaml +++ b/context/skills/ai-observability/config.yaml @@ -1,7 +1,8 @@ # PostHog AI Observability integration. # Single skill group, one variant per (LLM provider × language). -# Reachable via `wizard skill ai-observability`; also spawned as a single node -# by integration-v2 when an LLM SDK is detected in the project. +# Reachable via `wizard skill ai-observability`; also queued on every +# integration-v2 run — the task inspects the project's LLM calls and decides +# applicability itself. type: skill template: description.md description: PostHog AI Observability integration for {display_name} From 2cb9fd3a4de7da4eafd69b1167c9404be0daf4be Mon Sep 17 00:00:00 2001 From: Edwin Lim Date: Sat, 19 Sep 2026 00:04:11 -0700 Subject: [PATCH 5/6] prompt tweaks --- .../agents/integration-v2/ai-observability.md | 11 ++++++----- context/agents/integration-v2/logs.md | 9 +++++---- context/agents/integration-v2/review.md | 18 ++++++++++-------- .../skills/integration/references/2-edit.md | 4 +++- .../skills/integration/references/3-revise.md | 5 +++-- 5 files changed, 27 insertions(+), 20 deletions(-) diff --git a/context/agents/integration-v2/ai-observability.md b/context/agents/integration-v2/ai-observability.md index b793015c..33da456c 100644 --- a/context/agents/integration-v2/ai-observability.md +++ b/context/agents/integration-v2/ai-observability.md @@ -29,11 +29,12 @@ Reuse the PostHog client and identity established earlier in this run. If a variant cannot be chosen from the code, report the ambiguity as `not needed`; this default run does not ask the user to choose a provider. -This is the instrumentation part of a larger integration. Declare dependencies -in the manifest; the following review task installs them and checks imports, -builds, and lint. Defer the skill's package-manager and import-check commands -to that task. For Go, leave the required modules in the handoff for review to -add with `go get` and `go mod tidy`; do not hand-edit `go.mod` or `go.sum`. +This is the instrumentation part of a larger integration. Name the packages +this task needs in your handoff for review to install; do not edit dependency +manifests, and never write a version number you invented — the package manager +resolves real versions when review installs by name. Defer the skill's +package-manager and import-check commands to that task. For Go, review adds the +modules with `go get` and `go mod tidy`; do not hand-edit `go.mod` or `go.sum`. Do not run the app or make paid LLM calls. Use `check_env_keys` and `set_env_values` for environment files, using the project credentials supplied by the wizard. diff --git a/context/agents/integration-v2/logs.md b/context/agents/integration-v2/logs.md index 8d0c3e57..6bb7eab6 100644 --- a/context/agents/integration-v2/logs.md +++ b/context/agents/integration-v2/logs.md @@ -29,10 +29,11 @@ Keep changes to log setup and existing logging paths; do not scatter new logs through unrelated code or export secrets, request bodies, or user data. This is the instrumentation part of a larger integration. Instead of the -standalone skill's install command, declare required packages in the manifest -and leave installation, lockfile updates, builds, and lint to review. For -manifests managed by tooling (such as Go), pass the required packages to review -instead of editing them by hand. Use `detect_package_manager` if needed. Use +standalone skill's install command, name the required packages in your handoff +and leave installation, manifest and lockfile updates, builds, and lint to +review. Do not edit dependency manifests or write a version number you +invented — the package manager resolves real versions when review installs by +name. Use `detect_package_manager` if needed. Use the wizard's supplied project and region and the env names from upstream handoffs; inspect and write environment files only through `check_env_keys` and `set_env_values`. Do not request credentials or select a different project. diff --git a/context/agents/integration-v2/review.md b/context/agents/integration-v2/review.md index 9a5b8661..43f1382c 100644 --- a/context/agents/integration-v2/review.md +++ b/context/agents/integration-v2/review.md @@ -18,17 +18,19 @@ You are the last hands on the code. First make the integration run, then review then leave it building to the best of your ability. Do both in one pass, verifying and reviewing are the same reading of the same changeset. -**Install and verify.** The earlier steps edited code and declared the SDK in the -manifest but did not install it — install now, then verify the project builds, -typechecks, and lints, whichever of those it defines. An optional upstream task +**Install and verify.** The earlier steps edited code but did not install — +install now, then verify the project builds, typechecks, and lints, whichever +of those it defines. Install what the manifest declares plus every package the +upstream handoffs name, adding the named ones by bare name with the project's +package manager so it resolves real versions. An optional upstream task (AI Observability, Logs) may have failed and left partial edits no handoff -describes — install whatever the manifest declares either way, and let the build -surface stray imports; fix or revert them like any other defect. Do not run the test suite; the +describes — install and build either way, and let the build surface stray +imports; fix or revert them like any other defect. Do not run the test suite; the runtime does not allow it, and a green build is the bar. A bad version only surfaces here: if the install fails because a declared version does not exist (npm `ETARGET`, -"no matching version", a yanked release), fix the manifest to a real published -version — match the framework reference example's spec, or a caret range that -resolves — and install again. If the failure is the environment, not the +"no matching version", a yanked release), do not guess pins — reinstall the +package by bare name and let the package manager pick, matching the framework +reference example's spec only if that also fails. If the failure is the environment, not the integration's own change (a pre-existing broken dependency, a package manager that cannot provision), note it and move on; do not spiral. diff --git a/context/skills/integration/references/2-edit.md b/context/skills/integration/references/2-edit.md index 2d9e3d60..0bc92cab 100644 --- a/context/skills/integration/references/2-edit.md +++ b/context/skills/integration/references/2-edit.md @@ -46,7 +46,9 @@ For both skills, use the project credentials and region supplied by the wizard. Inspect and change environment files only through `check_env_keys` and `set_env_values`, reusing existing variable names. Do not ask for credentials or guess a region. Follow this runtime's tool restrictions; defer any dependency -installation or verification still needed to the revise step. Do not make paid +installation or verification still needed to the revise step, recording the +package names — never write a version number you invented into a manifest; the +package manager resolves real versions at install. Do not make paid LLM calls or claim delivery based on code changes. Keep each result (configured, already present, or skipped with a reason) for the final report, including a concrete path the user can trigger to check delivery. diff --git a/context/skills/integration/references/3-revise.md b/context/skills/integration/references/3-revise.md index edcc2494..83dfb05f 100644 --- a/context/skills/integration/references/3-revise.md +++ b/context/skills/integration/references/3-revise.md @@ -8,8 +8,9 @@ Check the project for errors. Read the package.json file for any type checking o Ensure that any components created were actually used. -Install any dependencies declared by the AI Observability and Logs steps before -verification, using the project's package manager. Include their changed files +Install any dependencies the AI Observability and Logs steps recorded before +verification, adding each by bare package name with the project's package +manager so it resolves real versions — do not guess version pins. Include their changed files in this review and check that Logs preserved any AIO tracing setup. Verification of delivery belongs to the user's checklist; do not make paid LLM calls. From 2151f434e1cbd7078ef0fce2cc0ba0ccd661ea44 Mon Sep 17 00:00:00 2001 From: Edwin Lim Date: Sat, 19 Sep 2026 01:45:57 -0700 Subject: [PATCH 6/6] tweaks to handoffs and less heavy handed instrumentation --- .../agents/integration-v2/ai-observability.md | 20 +++++++++++--- context/agents/integration-v2/logs.md | 27 +++++++++++++------ context/agents/integration-v2/review.md | 10 ++++++- .../integration-v2/report/description.md | 19 ++++++++++--- .../skills/integration/references/2-edit.md | 23 +++++++++++----- .../skills/integration/references/3-revise.md | 8 +++++- .../integration/references/4-conclude.md | 12 ++++++--- 7 files changed, 93 insertions(+), 26 deletions(-) diff --git a/context/agents/integration-v2/ai-observability.md b/context/agents/integration-v2/ai-observability.md index 33da456c..ebd97bd8 100644 --- a/context/agents/integration-v2/ai-observability.md +++ b/context/agents/integration-v2/ai-observability.md @@ -25,9 +25,19 @@ For existing LLM calls, load the `ai-observability` skill menu and install the variant matching the calling code's language and provider or agent framework. Follow that skill's selection rules and instrumentation references. Manual capture is appropriate only for existing calls without a supported wrapper. -Reuse the PostHog client and identity established earlier in this run. If a -variant cannot be chosen from the code, report the ambiguity as `not needed`; -this default run does not ask the user to choose a provider. +Instrument against the provider SDK version the app already uses; never +require, assume, or request an upgrade of the app's own dependencies — if the +wrapper cannot support the app's version, fall back to manual capture or +report why. Reuse the PostHog client and identity established earlier in this +run. Enable +privacy mode by default, so prompt and completion content stays out of PostHog +until the user opts in: set the privacy-mode option on the PostHog client the +instrumentation uses; where the variant configures capture elsewhere (a +callback handler, per-request parameters), set its documented privacy-mode +option there instead; for manual capture, omit `$ai_input` and +`$ai_output_choices`. Model, token counts, latency, and cost metadata still +flow. If a variant cannot be chosen from the code, report the ambiguity as +`not needed`; this default run does not ask the user to choose a provider. This is the instrumentation part of a larger integration. Name the packages this task needs in your handoff for review to install; do not edit dependency @@ -44,6 +54,8 @@ by the wizard. Existing LLM calls are wired using the selected skill, or the handoff clearly explains why this task was not needed. List the variant, changed files, declared dependencies, and a concrete call path the user can trigger to verify the -session/trace/generation tree. Pass any deferred dependency work to review. +session/trace/generation tree. Note for the report that privacy mode is on and +the one-line change that enables content capture if the user wants it. Pass any +deferred dependency work to review. Describe delivery as unverified unless this run actually observed it; do not publish a separate setup report. diff --git a/context/agents/integration-v2/logs.md b/context/agents/integration-v2/logs.md index 6bb7eab6..f5750d6c 100644 --- a/context/agents/integration-v2/logs.md +++ b/context/agents/integration-v2/logs.md @@ -18,15 +18,23 @@ optional: true Configure PostHog log capture for this project's supported runtimes. Load the `logs` skill menu, install its matching skill, and read the platform reference before editing. Inspect the runtime and existing logging setup; use the -documented OTLP exporter or SDK-native log capture for that platform. A +documented OTLP exporter or SDK-native log capture for that platform. If the +app already uses OpenTelemetry, write against its existing OTel version and +name exporter packages compatible with it — never require, or write code that +assumes, an upgrade of a package this integration did not introduce. A browser-only app must not receive a server exporter. If no documented setup applies, complete as `not needed` and explain why. -Reuse any existing PostHog log export and leave its handlers and outputs -intact. If capture is already configured, report that without adding a second -exporter. Preserve any tracing provider AIO configured in the previous step. -Keep changes to log setup and existing logging paths; do not scatter new logs -through unrelated code or export secrets, request bodies, or user data. +Export only log lines this integration adds. Create a dedicated logger for the +PostHog exporter and emit a few purpose-written lines at meaningful points the +app already passes through. Even where the platform reference attaches the +exporter to the application's root or existing loggers, do not — the app's +existing logs are data this run has no mandate to export, and every line that +leaves must be visible in this run's diff. Reuse any existing PostHog log +export and leave all existing handlers and outputs intact; if capture is +already configured, report that without adding a second exporter. Preserve any +tracing provider AIO configured in the previous step. Do not export secrets, +request bodies, or user data, and do not scatter logs through unrelated code. This is the instrumentation part of a larger integration. Instead of the standalone skill's install command, name the required packages in your handoff @@ -45,5 +53,8 @@ guessing a region. Log capture is configured once for the supported runtime, was already present, or has a clear skip reason. The handoff names the changed files, packages, env variable names (never values), and any dependency work for review. Give a -specific existing log path the user can trigger and find in PostHog Logs; -delivery remains unverified until observed. Do not publish a separate report. +specific code path the user can trigger to produce one of the added log lines +and find it in PostHog Logs; delivery remains unverified until observed. Note +for the report that only lines added by this run are exported, and the one-line +change that routes the app's existing loggers into the same exporter if the +user wants more. Do not publish a separate report. diff --git a/context/agents/integration-v2/review.md b/context/agents/integration-v2/review.md index 43f1382c..b36ec7c1 100644 --- a/context/agents/integration-v2/review.md +++ b/context/agents/integration-v2/review.md @@ -22,7 +22,15 @@ reading of the same changeset. install now, then verify the project builds, typechecks, and lints, whichever of those it defines. Install what the manifest declares plus every package the upstream handoffs name, adding the named ones by bare name with the project's -package manager so it resolves real versions. An optional upstream task +package manager so it resolves real versions. The app's existing dependencies +are not yours to touch: never upgrade, downgrade, or re-add a package this +integration did not introduce. If an instrumentation package's peer range +conflicts with a dependency the app already has, install an older +instrumentation version that fits the app's existing versions — find it with +the package manager, stepping down major versions (`pkg@^7`, then `@^6`, +…) for a few attempts rather than looking for documentation of one. If none +fits, revert that piece of instrumentation, leave the app's dependency alone, +and record the incompatibility so the report can say what was skipped and why. An optional upstream task (AI Observability, Logs) may have failed and left partial edits no handoff describes — install and build either way, and let the build surface stray imports; fix or revert them like any other defect. Do not run the test suite; the diff --git a/context/skills/integration-v2/report/description.md b/context/skills/integration-v2/report/description.md index 7e362384..ab2cf0fd 100644 --- a/context/skills/integration-v2/report/description.md +++ b/context/skills/integration-v2/report/description.md @@ -18,6 +18,12 @@ Draw on two sources only: table: grep the changed files for `capture(` calls and read the capture step's handoff in `queue.json`. +Handoffs are written mid-run, in task order, so a later handoff supersedes an +earlier one's forward-looking claims. Something an early task deferred — a +dependency to install, a check to run — was either done by a later task or +never done at all: report the final state, and let no "deferred" or "pending" +claim survive into the report. + ## What to include - A one-line summary of what was set up. @@ -29,7 +35,13 @@ Draw on two sources only: - AI Observability and Logs: what was configured or already present, or why each was skipped or failed. Use the task handoffs, including changed files and outstanding verification; do not treat a skipped or failed task as a - successful setup. + successful setup. When AI Observability was configured, say that privacy mode + is on — prompts and completions are not captured, while model, token, + latency, and cost metadata are — and include the handoff's one-line change to + enable content capture. When Logs was configured, say that only log lines + added by this run are exported, and include the handoff's one-line pointer + for routing the app's existing loggers into the same exporter if the user + wants more. - The dashboard link. - Any build conflict, in full. - Clear next steps for the user. @@ -50,8 +62,9 @@ code changed this run and drop the ones that don't fit: for CSP violations — a blocked SDK queues events silently and never sends. - If LLM analytics was set up: trigger the instrumented call path and confirm `$ai_generation` events appear in PostHog. -- If Logs was set up: trigger the log path named in its handoff and confirm the - entry appears in PostHog Logs with the expected service and severity. +- If Logs was set up: trigger the code path that emits one of the added log + lines and confirm the entry appears in PostHog Logs with the expected service + and severity. - If auth exists and identify was wired: the returning-visitor path also calls identify, so returning sessions don't fragment onto anonymous distinct IDs. diff --git a/context/skills/integration/references/2-edit.md b/context/skills/integration/references/2-edit.md index 0bc92cab..2402d50a 100644 --- a/context/skills/integration/references/2-edit.md +++ b/context/skills/integration/references/2-edit.md @@ -32,15 +32,24 @@ then Logs, so edits to manifests and initialization files do not conflict. the matching variant with `install_skill`, and follow its references. Choose the provider/framework and language from the actual calling code. Use manual capture only for existing calls without a supported wrapper. Reuse the client - and identity already established. If the provider remains ambiguous, skip with - that reason instead of asking a question during the default run. + and identity already established. Enable privacy mode by default so prompt and + completion content stays out of PostHog until the user opts in: set the + privacy-mode option on the client the instrumentation uses, or the variant's + documented handler/per-request option where capture is configured there; for + manual capture, omit `$ai_input` and `$ai_output_choices`. Metadata (model, + tokens, latency, cost) still flows. If the provider remains ambiguous, skip + with that reason instead of asking a question during the default run. 2. **Logs:** call `load_skill_menu` with `category: "logs"`, install the matching skill, and follow the documented setup for this runtime. Use SDK-native log capture where documented, otherwise the platform's OTLP exporter. Do not add a server exporter to browser-only code. If no documented setup applies, skip with - the reason. Preserve existing handlers, outputs, and any AIO tracing provider; - an existing PostHog exporter needs no duplicate. Keep edits to log setup and - existing logging paths, without adding logs to unrelated code. + the reason. Export only log lines this integration adds: wire the exporter to + a dedicated logger and emit a few purpose-written lines at meaningful points + the app already passes through — never attach it to the application's root or + existing loggers, even where the platform docs show that; the app's existing + logs are not this run's to export. Preserve existing handlers, outputs, and + any AIO tracing provider; an existing PostHog exporter needs no duplicate. Do + not add logs to unrelated code. For both skills, use the project credentials and region supplied by the wizard. Inspect and change environment files only through `check_env_keys` and @@ -48,7 +57,9 @@ Inspect and change environment files only through `check_env_keys` and guess a region. Follow this runtime's tool restrictions; defer any dependency installation or verification still needed to the revise step, recording the package names — never write a version number you invented into a manifest; the -package manager resolves real versions at install. Do not make paid +package manager resolves real versions at install. Write against the versions +of packages the app already has: never upgrade, or write code that assumes +upgrading, a dependency this integration did not introduce. Do not make paid LLM calls or claim delivery based on code changes. Keep each result (configured, already present, or skipped with a reason) for the final report, including a concrete path the user can trigger to check delivery. diff --git a/context/skills/integration/references/3-revise.md b/context/skills/integration/references/3-revise.md index 83dfb05f..ce9c530b 100644 --- a/context/skills/integration/references/3-revise.md +++ b/context/skills/integration/references/3-revise.md @@ -10,7 +10,13 @@ Ensure that any components created were actually used. Install any dependencies the AI Observability and Logs steps recorded before verification, adding each by bare package name with the project's package -manager so it resolves real versions — do not guess version pins. Include their changed files +manager so it resolves real versions — do not guess version pins. The app's +existing dependencies are not yours to touch: never upgrade, downgrade, or +re-add a package this integration did not introduce. On a peer conflict, +install an older instrumentation version that fits the app's versions — step +down major versions with the package manager (`pkg@^7`, then `@^6`, …) for a +few attempts; if none fits, revert that piece of instrumentation and note it +in the report. Include their changed files in this review and check that Logs preserved any AIO tracing setup. Verification of delivery belongs to the user's checklist; do not make paid LLM calls. diff --git a/context/skills/integration/references/4-conclude.md b/context/skills/integration/references/4-conclude.md index a28b7f96..e7223d89 100644 --- a/context/skills/integration/references/4-conclude.md +++ b/context/skills/integration/references/4-conclude.md @@ -81,7 +81,12 @@ Compose the setup report as markdown — do NOT write it to a file in the projec Include separate AI Observability and Logs results: what was configured, what was already present, or why each was skipped. Name the changed files and distinguish code verification from observed delivery. A skipped product is not a successful -setup. +setup. When AI Observability was configured, say that privacy mode is on — +prompts and completions are not captured, while model, token, latency, and cost +metadata are — and give the one-line change to enable content capture. When +Logs was configured, say that only log lines added by this run are exported, +and give the one-line change that routes the app's existing loggers into the +same exporter if the user wants more. # PostHog post-wizard report @@ -113,8 +118,9 @@ For the "Verify before merging" checklist, write GitHub-style checkboxes (`- [ ] - If you added environment variables: "Add the exact PostHog env var names you added to `.env.example` and any monorepo/bootstrap scripts so collaborators know what to set." - If this integration ships a minified production browser bundle (most SPA/SSR web frameworks — e.g. Next.js, Nuxt, SvelteKit, Astro, Vite-based apps): "Wire source-map upload (`posthog-cli sourcemap` or your bundler's upload step) into CI so production stack traces de-minify." - If LLM analytics was set up in this run: "Trigger the LLM call path(s) you instrumented and confirm `$ai_generation` events appear in PostHog AI Observability." -- If Logs was set up in this run: trigger the existing log path you configured and - confirm the entry appears in PostHog Logs with the expected service and severity. +- If Logs was set up in this run: trigger the code path that emits one of the + added log lines and confirm the entry appears in PostHog Logs with the + expected service and severity. - If the app has user auth and an `identify` call was added: "Confirm the returning-visitor path also calls `identify` — a handler that only identifies on fresh login can leave returning sessions on anonymous distinct IDs." Do not invent items beyond what applies. If only the two "Always" items apply, the checklist is just those two.