diff --git a/CHANGES.md b/CHANGES.md index 9290636..3f441c0 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -2,6 +2,12 @@ This port applies the Cursor → Claude Code substitutions in skill bodies. Earlier drafts left them flagged; this revision resolves them. A later pass added a Codex build that shares the same skills; see [Codex port](#codex-port) below. +## Unreleased: optional Devin workers + +Adds `devin` to the external runner for both Claude Code and Codex parents. SWE-2 supports medium/high/max through exact model UIDs; SWE-1.6 uses a fixed `default` effort token. Setup probes opted-in Devin families while preserving existing default panels. Arena uses the shared completion contract, which accepts matching Codex and Devin pinned-argv evidence. The upstream sync point is unchanged. + +The adapter extracts the final response from a private conversation export, records pinned-argv evidence without claiming a provider model report, and leaves unreported usage/cost/session fields null. Per-run configuration disables nested subagents and imported tool settings. Read-only workers deny shell and writes; writers use sandboxed shell commands for edits and tests in the assigned worktree, with direct unsandboxed file tools denied and explicit shell-only guidance prepended in a private prompt copy. Per-run shell onboarding is suppressed; a private conversation export must end with a final agent message without tool calls, preventing banner-only output, progress messages, and denied headless tools from counting as completed answers. Only final response text is retained; the export is removed on every outcome. Devin project settings and startup hooks remain a live-verification concern. Account restrictions fail without fallback. + ## 1.4.1 syncs to Cursor pstack 0.15.1 Open Pstack 1.4.1 tracks Cursor pstack 0.15.1 at `f8abeddd1862dc73704e3d719dd73df0d51b8c71`. Poteto-mode now requires each claim to include its evidence or a measured, inferred, or guess label in the same sentence. Agents also run any check they can run themselves instead of handing that check to the user. No playbook, model, runtime, or dependency changed. diff --git a/NOTICE.md b/NOTICE.md index 0ec8a23..2f52ba3 100644 --- a/NOTICE.md +++ b/NOTICE.md @@ -37,6 +37,8 @@ Summary of structural changes: - Claude-native Fable and Opus lanes are port-authored agent definitions. They select the rolling family alias plus requested effort for every selectable Claude-native pair in the provider-dispatch model matrix. - A Codex build shares the same `skills/` tree. It adds `plugins/pstack/.codex-plugin/plugin.json`, a root `.agents/plugins/marketplace.json`, and `plugins/pstack/skills/poteto-mode/references/codex-tools.md` (the Claude-to-Codex tool, model, and built-in map), plus a one-line Platform note in the skills that name a Claude primitive. The skill content itself is unchanged. See [CHANGES.md](CHANGES.md#codex-port). +The external runner also includes an independently authored Devin CLI adapter for optional SWE-2 and SWE-1.6 workers. It does not import Cognition source code. + ## Modifications Per the MIT license, modifications are permitted. Skill bodies have been edited to substitute Cursor-specific primitives with their Claude Code equivalents (the full substitution table is in [CHANGES.md](CHANGES.md)). All upstream copyright notices in source files (where present) are preserved. diff --git a/README.md b/README.md index 6490111..fb112b5 100644 --- a/README.md +++ b/README.md @@ -120,6 +120,12 @@ That is the main workflow. The other skills are there when poteto-mode needs the Plugin skills include `pstack:` in their name. In Claude Code, invoke a native skill such as `/pstack:architect`. In Codex, ask for the skill, such as `Use pstack:architect for this design.` See the [technical reference](docs/reference.md) for the full list. +## Optional Devin workers + +Codex or Claude Code can delegate selected roles to SWE-2 or SWE-1.6 through an authenticated [Devin CLI](https://docs.devin.ai/cli). Ask `setup-pstack` to use `devin:swe-2@high` (medium/high/max) or `devin:swe-1.6@default` for named roles. Devin remains an external worker; the default four-model panel stays unchanged. + +This adapter captures print output and pins the CLI model UID. It does not report provider-verified model identity, tokens, or cost. Read-only workers cannot execute shell commands. See the [Devin dispatch contract](plugins/pstack/skills/poteto-mode/references/provider-dispatch.md#optional-devin-models) for permissions and live verification requirements. + ## Models and token use Some pstack workflows use one model. Skills such as `architect`, `arena`, and `interrogate` can run several models in parallel. Each model run uses the subscription and token allowance of its own command-line tool. diff --git a/docs/reference.md b/docs/reference.md index 2301514..e99b5cb 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -193,7 +193,7 @@ The port is editorial, not mechanical. Anywhere upstream pstack assumed Cursor-s | Cursor's `/goal` (standing objective across turns) | The program objective written into the run's standing orders and restated in the todolist | | The Cursor agent store (path in the system prompt) | `~/.claude/orchestrate//`, which survives the session restarts a multi-day program expects | | Model rule `~/.cursor/rules/pstack-models.mdc` | Override sheet `~/.claude/pstack-models.md`, included from `CLAUDE.md` | -| Multi-model panels (arena, architect, interrogate) | Provider dispatch restores the upstream frontier quad: `claude:fable@max`, `codex:gpt-5.6-sol@max`, `grok:grok-4.6@xhigh`, `claude:opus@xhigh`. Same-provider lanes stay native; external lanes use the bundled runner. | +| Multi-model panels (arena, architect, interrogate) | Provider dispatch restores the upstream frontier quad: `claude:fable@max`, `codex:gpt-5.6-sol@max`, `grok:grok-4.6@xhigh`, `claude:opus@xhigh`. Same-provider lanes stay native; external lanes use the bundled runner. Optional Devin workers use `devin:swe-2@high` or `devin:swe-1.6@default`; see the [dispatch contract](../plugins/pstack/skills/poteto-mode/references/provider-dispatch.md#optional-devin-models). | ### Cross-vendor dispatch diff --git a/plugins/pstack/skills/arena/SKILL.md b/plugins/pstack/skills/arena/SKILL.md index 0e3b5f3..b6b4908 100644 --- a/plugins/pstack/skills/arena/SKILL.md +++ b/plugins/pstack/skills/arena/SKILL.md @@ -35,7 +35,7 @@ Start all N lanes in one fan-out phase through the provider-dispatch contract. N Each rationale names the alternatives the candidate considered and what it rejected. -An external lane counts only when its receipt says `complete` and carries either a matching `provider-report` or Codex's exact `pinned-argv` evidence; a native lane counts when its tool transcript returns the assigned model's result. If a candidate fails, proceed with N-1 and note the exact dropout in the synthesis record. Never replace it with another provider silently. +An external lane counts only when it satisfies [Completion and dropouts](../poteto-mode/references/provider-dispatch.md#completion-and-dropouts), including the assignment and model-argument checks for Codex or Devin `pinned-argv` evidence; a native lane counts when its tool transcript returns the assigned model's result. If a candidate fails, proceed with N-1 and note the exact dropout in the synthesis record. Never replace it with another provider silently. ## Phase C: Cross-judge diff --git a/plugins/pstack/skills/poteto-mode/references/codex-tools.md b/plugins/pstack/skills/poteto-mode/references/codex-tools.md index b967458..1d057b8 100644 --- a/plugins/pstack/skills/poteto-mode/references/codex-tools.md +++ b/plugins/pstack/skills/poteto-mode/references/codex-tools.md @@ -42,7 +42,7 @@ poteto-mode's Subagents section sets Claude-specific defaults (`subagent_type: " ## Models and providers -Do not replace every configured entry with a Codex model. `/setup-pstack` writes portable descriptors such as `claude:fable@max`, `codex:gpt-5.6-sol@max`, and `grok:grok-4.6@xhigh`. In a Codex parent, only `codex:*` is native. Route Claude and Grok descriptors through the external launcher exactly as `provider-dispatch.md` specifies. The current default panel intentionally keeps four-provider frontier diversity and contains no older GPT or Claude substitute. +Do not replace every configured entry with a Codex model. `/setup-pstack` writes portable descriptors such as `claude:fable@max`, `codex:gpt-5.6-sol@max`, and `grok:grok-4.6@xhigh`. In a Codex parent, only `codex:*` is native. Route Claude, Grok, and optional Devin descriptors through the external launcher exactly as `provider-dispatch.md` specifies. The current default panel intentionally keeps four-provider frontier diversity and contains no older GPT or Claude substitute. ## Claude built-in skills pstack references @@ -57,7 +57,7 @@ Some triggers name skills that ship with Claude Code, not pstack. They do not ex ## Vendored scripts -`skills/poteto-mode/scripts/` ships the `watch-pr` PR watcher, the `orch` store CLI, `worktree-audit.sh`, and `runner/pstack-runner`. They are plain bun and bash, so they run the same on Codex; invoke them through `shell`. The external runner additionally needs the assigned `claude`, `codex`, or `grok` executable already authenticated. It rejects a Codex provider when Codex is the parent because that lane belongs on native `spawn_agent`. The other scripts need `bun`, `gh`, (for stack work) `gt`, and (for `worktree-audit.sh`) `jq` and `rg`. `worktree-audit.sh` reads Claude Code transcripts under `~/.claude/projects/`; point it at your runtime's transcript directory instead when you run it elsewhere. +`skills/poteto-mode/scripts/` ships the `watch-pr` PR watcher, the `orch` store CLI, `worktree-audit.sh`, and `runner/pstack-runner`. They are plain bun and bash, so they run the same on Codex; invoke them through `shell`. The external runner additionally needs the assigned `claude`, `codex`, `grok`, or `devin` executable already authenticated. It rejects a Codex provider when Codex is the parent because that lane belongs on native `spawn_agent`. The other scripts need `bun`, `gh`, (for stack work) `gt`, and (for `worktree-audit.sh`) `jq` and `rg`. `worktree-audit.sh` reads Claude Code transcripts under `~/.claude/projects/`; point it at your runtime's transcript directory instead when you run it elsewhere. ## Instructions file diff --git a/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md b/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md index 74ab90d..43aff0c 100644 --- a/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md +++ b/plugins/pstack/skills/poteto-mode/references/provider-dispatch.md @@ -15,10 +15,27 @@ pstack model choices are provider-qualified descriptors: | grok | grok-4.6-fast-xhigh | grok | grok-4.6 | xhigh | low medium high xhigh max | - | | opus | opus | claude | opus | xhigh | low medium high xhigh max | opus | -The allowed effort universe is exactly `low`, `medium`, `high`, `xhigh`, `max`. First-run requested efforts are the Default effort cell of each row. A Claude-native agent stem of `-` means the family has no Claude-native agent. Otherwise the shipped agent name is `pstack--`. +For the default matrix, the allowed effort universe is exactly `low`, `medium`, `high`, `xhigh`, `max`. First-run requested efforts are the Default effort cell of each row. A Claude-native agent stem of `-` means the family has no Claude-native agent. Otherwise the shipped agent name is `pstack--`. `fable` and `opus` are Claude Code's rolling aliases. Claude resolves each alias to the latest available family revision. A runner receipt keeps the requested alias in `model` and the concrete provider-reported revision in `reportedModel`; verification accepts only a numeric `claude-fable-*` or `claude-opus-*` revision from the matching family. +## Optional Devin models + +Devin is an external provider from either parent, not a parent harness. These opt-in families do not change the four-model default panel. + +| Family | Provider | Model | Default effort | Selectable efforts | CLI model UID | +|---|---|---|---|---|---| +| swe-2 | devin | swe-2 | high | medium high max | swe-2- | +| swe-1.6 | devin | swe-1.6 | default | default | swe-1-6 | + +Use descriptors such as `devin:swe-2@high` or `devin:swe-1.6@default`. `default` records that SWE-1.6 has no selectable effort; it is rejected for other providers. Never clamp SWE-2's unsupported `low` or `xhigh` to another level. The runner pins the exact CLI UID instead of a rolling family alias or Fusion pairing. + +Install and sign in to [Devin CLI](https://docs.devin.ai/cli). Inspect `devin models list --format json` and probe each selected pair: listing a model does not prove the account can execute it. An upgrade-required response is an unavailable-model failure, never permission to substitute another model. + +The runner uses `--print` and a private temporary `--config` file, deleted after completion or failure. It disables recursive subagents and imports from other tools, and denies MCP and fetch calls. Read-only lanes deny edit, write, and shell execution; use them for file inspection, not test execution. Writer lanes use `--sandbox` in the dedicated worktree and perform edits and tests through sandboxed `exec`. Direct `edit`/`write` tools are denied in both modes: they run outside Devin's OS sandbox and can require interactive confirmation even with scoped write grants. The adapter prefixes each writer task with these execution constraints in a private prompt copy, preserving the assigned prompt and its receipt path. File-tool-only prompts are unsupported. Devin's own project configuration, rules, plugins, and hooks can still load; this is not a clean-room execution environment. Do not assign an untrusted checkout or rely on this adapter to isolate startup hooks. Verify effective permission behavior in the target CLI before release. + +The temporary config marks shell onboarding complete. The runner requests a private ATIF-v1.7 conversation export and accepts only a final agent step with a nonempty message and no tool calls. A progress message followed by denied or pending tools cannot count as completion, even if Devin exits zero without stderr. Only the final message is returned; system context, reasoning, and tool observations are never copied into output or receipts. The export directory is private and removed after every outcome. The adapter continues to use `pinned-argv` model evidence rather than treating the export's display model name as verified identity; model-report, session, usage, and cost fields remain null. + ## Read-time normalization Normalize configured descriptors before matching them to the matrix or choosing a route. If a provider-qualified Claude model starts with `claude-fable-` or `claude-opus-` and its remaining revision contains only digits and hyphens, replace that model component in memory with `fable` or `opus`. Preserve provider, effort, role, and lane order. Use only the normalized descriptor for native dispatch or runner argv. Never pass the versioned predecessor to Claude. @@ -31,10 +48,10 @@ This read-time rule makes an older installed sheet use the latest family revisio The top-level harness resolves the route once. A child receives an assigned provider, model, effort, access mode, prompt, working directory, and output path. A child never detects the harness, chooses a provider, or launches another model. Environment markers may corroborate the top-level harness before fan-out, but nested processes inherit parent markers and must not use them for routing. -| Parent | `claude:*` | `codex:*` | `grok:*` | -|---|---|---|---| -| Claude Code | native `Agent` | external runner | external runner | -| Codex | external runner | native `spawn_agent` | external runner | +| Parent | `claude:*` | `codex:*` | `grok:*` | `devin:*` | +|---|---|---|---|---| +| Claude Code | native `Agent` | external runner | external runner | external runner | +| Codex | external runner | native `spawn_agent` | external runner | external runner | `inherit-parent` and `auto` remain aliases. They use the parent's current model and effort through its native subagent primitive. In a panel they still consume one lane, but they reduce provider diversity; say so in the synthesis record. @@ -54,9 +71,9 @@ The launcher lives at `skills/poteto-mode/scripts/runner/pstack-runner` under th ```text pstack-runner \ --parent \ - --provider \ + --provider \ --model \ - --effort \ + --effort \ --mode \ --prompt \ --cwd \ @@ -90,7 +107,7 @@ Success requires all of these: 1. Exit status `0`. 2. Receipt status `complete`. -3. Either `modelVerified: true` with `modelEvidence: "provider-report"`, or a Codex receipt with `reportedModel: null`, `modelVerified: false`, and `modelEvidence: "pinned-argv"`. For Claude's `fable` and `opus` aliases, the concrete provider report must belong to the requested family. Codex 0.149.0 accepts the exact `--model` argument but does not report the served model in its JSONL stream. +3. Either `modelVerified: true` with `modelEvidence: "provider-report"`, or a Codex or Devin receipt with `reportedModel: null`, `modelVerified: false`, and `modelEvidence: "pinned-argv"`. For Claude's `fable` and `opus` aliases, the concrete provider report must belong to the requested family. For pinned-argv evidence, verify the receipt provider, model, and effort match the assignment and its argv pins the expected CLI model. Codex pins the assigned model directly; Devin uses the exact UID mapping in Optional Devin models (for example, `swe-2@high` pins `--model swe-2-high`). Neither Codex JSONL nor Devin print output supplies a provider model report; do not describe pinned argv as provider-verified identity. 4. A non-empty output file. The receipt also carries elapsed time, token usage when the CLI exposes it, and cost when available. Keep it with the arena or review artifacts so parent-harness comparisons are evidence-based. diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/cli.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/cli.ts index 4fcce24..a271cb7 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/cli.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/cli.ts @@ -13,7 +13,7 @@ import { UsageError, } from "./types.ts"; -const HELP = `Usage: pstack-runner --parent --provider \\ +const HELP = `Usage: pstack-runner --parent --provider \\ --model --effort --mode \\ --prompt --cwd --output --receipt [--timeout ] @@ -98,7 +98,7 @@ export function parseArgs(argv: readonly string[]): RunnerOptions | null { parent: oneOf("parent", stringValue(parsed.values.parent), PARENTS) as Parent, provider: oneOf("provider", stringValue(parsed.values.provider), PROVIDERS) as Provider, model: required("model", stringValue(parsed.values.model)), - effort: oneOf("effort", stringValue(parsed.values.effort), EFFORTS) as Effort, + effort: oneOf("effort", stringValue(parsed.values.effort), [...EFFORTS, "default"]) as Effort, mode, promptPath: required("prompt", stringValue(parsed.values.prompt)), cwd: required("cwd", stringValue(parsed.values.cwd)), diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/commands.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/commands.ts index 5f2b10c..65ae75e 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/commands.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/commands.ts @@ -5,6 +5,8 @@ import type { RunnerOptions, } from "./types.ts"; +import { devinConfigPath, devinExportPath, devinModel, devinPromptPath } from "./devin.ts"; + export interface CommandSpec { readonly command: string; readonly args: readonly string[]; @@ -13,6 +15,8 @@ export interface CommandSpec { export function preflightCommand(provider: Provider): CommandSpec { switch (provider) { + case "devin": + return { command: "devin", args: ["auth", "status"], stdin: "none" }; case "claude": return { command: "claude", @@ -65,6 +69,27 @@ function effortOverride(effort: Effort): string { export function invocationCommand(options: RunnerOptions): CommandSpec { switch (options.provider) { + case "devin": + return { + command: "devin", + args: [ + "--config", + devinConfigPath(options), + "--model", + devinModel(options.model, options.effort), + ...(options.mode === "isolated-write" + ? ["--sandbox"] + : ["--permission-mode", "auto"]), + "--respect-workspace-trust", + "false", + "--prompt-file", + devinPromptPath(options), + "--export", + devinExportPath(options), + "--print", + ], + stdin: "none", + }; case "claude": return { command: "claude", diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/devin.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/devin.test.ts new file mode 100644 index 0000000..ecb647e --- /dev/null +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/devin.test.ts @@ -0,0 +1,281 @@ +import { afterEach, beforeEach, describe, expect, it } from "bun:test"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { devinConfig, devinConfigPath, devinExportDirectory, devinExportPath, devinModel, devinPromptPath } from "./devin.ts"; +import { invocationCommand } from "./commands.ts"; +import { childEnvironment, runLane } from "./run.ts"; +import { parseArgs } from "./cli.ts"; +import type { RunnerOptions } from "./types.ts"; + +let scratch: string; +let oldPath: string | undefined; +let options: RunnerOptions; + +beforeEach(() => { + scratch = mkdtempSync(join(tmpdir(), "pstack-devin-test-")); + oldPath = process.env.PATH; + process.env.PATH = `${scratch}:${oldPath}`; + writeFileSync(join(scratch, "prompt.md"), "Inspect the assigned file."); + options = { + parent: "codex", provider: "devin", model: "swe-2", effort: "high", + mode: "read-only", cwd: scratch, promptPath: join(scratch, "prompt.md"), + outputPath: join(scratch, "result.md"), receiptPath: join(scratch, "receipt.json"), + timeoutMs: null, + }; +}); + +afterEach(() => { + if (oldPath === undefined) delete process.env.PATH; + else process.env.PATH = oldPath; + rmSync(scratch, { recursive: true, force: true }); +}); + +function transcript(message: string, toolCalls: unknown = []): { schema_version: string; steps: Record[] } { + return { + schema_version: "ATIF-v1.7", + steps: [ + { source: "system", message: "PRIVATE_SYSTEM_CONTEXT" }, + { source: "agent", message, tool_calls: toolCalls, reasoning_content: "PRIVATE_REASONING" }, + ], + }; +} + +function fakeDevin(response: string, exitCode = 0, auth = "Logged in (via Devin).", stderr = "", exported: unknown = transcript(response), omitExport = false) { + const path = join(scratch, "devin"); + writeFileSync(path, `#!/usr/bin/env bun +import { statSync, unlinkSync } from "node:fs"; +import { dirname } from "node:path"; +const args = process.argv.slice(2); +if (args[0] === "auth") { + console.log(${JSON.stringify(auth)}); + process.exit(0); +} +const config = await Bun.file(args[args.indexOf("--config") + 1]).json(); +if (config.subagents_enabled !== false) throw new Error("recursive agents enabled"); +const promptPath = args[args.indexOf("--prompt-file") + 1]; +await Bun.write(${JSON.stringify(join(scratch, "captured-prompt.txt"))}, await Bun.file(promptPath).text()); +if (args.includes("--sandbox") && (statSync(promptPath).mode & 0o777) !== 0o600) throw new Error("prompt not private"); +const exportPath = args[args.indexOf("--export") + 1]; +if ((statSync(exportPath).mode & 0o777) !== 0o600) throw new Error("export file not private"); +if ((statSync(dirname(exportPath)).mode & 0o777) !== 0o700) throw new Error("export directory not private"); +if (${omitExport}) unlinkSync(exportPath); +else await Bun.write(exportPath, ${JSON.stringify(typeof exported === "string" ? exported : JSON.stringify(exported))}); +console.log(${JSON.stringify(response)}); +console.error(${JSON.stringify(stderr)}); +process.exit(${exitCode}); +`); + chmodSync(path, 0o755); +} + +describe("Devin external provider", () => { + it("pins supported SWE variants and rejects unavailable effort combinations", () => { + for (const effort of ["medium", "high", "max"] as const) { + expect(devinModel("swe-2", effort)).toBe(`swe-2-${effort}`); + } + expect(devinModel("swe-1.6", "default")).toBe("swe-1-6"); + for (const effort of ["low", "xhigh", "default"] as const) { + expect(() => devinModel("swe-2", effort)).toThrow("Devin supports"); + } + expect(() => devinModel("swe-1.6", "high")).toThrow("Devin supports"); + expect(() => devinModel("swe", "high")).toThrow("Devin supports"); + }); + + it("accepts SWE-1.6's fixed effort through the public CLI", () => { + const parsed = parseArgs([ + "--parent", "codex", "--provider", "devin", "--model", "swe-1.6", + "--effort", "default", "--mode", "read-only", "--cwd", scratch, + "--prompt", options.promptPath, "--output", options.outputPath, + "--receipt", options.receiptPath, + ]); + expect(parsed?.effort).toBe("default"); + }); + + it("keeps metacharacters in prompt paths as argv data", () => { + const promptPath = join(scratch, "prompt $(touch BAD).md"); + const command = invocationCommand({ ...options, promptPath }); + expect(command.args).toContain(promptPath); + expect(command.args).toContain("swe-2-high"); + expect(command.args).not.toContain("--effort"); + expect(command.stdin).toBe("none"); + }); + + it("disables child delegation and imports, and denies writes and shell in read-only mode", () => { + const config = devinConfig(options); + expect(config.shell.setup_complete).toBe(true); + expect(config.subagents_enabled).toBe(false); + expect(Object.values(config.read_config_from).every(value => value === false)).toBe(true); + expect(config.permissions.deny).toEqual(expect.arrayContaining(["Write(**)", "exec", "mcp__*"])); + const writer = invocationCommand({ ...options, mode: "isolated-write" }); + expect(writer.args).toContain("--sandbox"); + expect(writer.args).not.toContain("--permission-mode"); + const writerConfig = devinConfig({ ...options, mode: "isolated-write" }); + expect(writerConfig.permissions.deny).toEqual(expect.arrayContaining(["edit", "write"])); + expect(writerConfig.permissions.deny).not.toContain("exec"); + expect(writer.args).not.toContain("dangerous"); + }); + + for (const parent of ["codex", "claude"] as const) { + for (const model of ["swe-2", "swe-1.6"]) { + it(`returns ${model} results to ${parent} with honest model evidence`, async () => { + fakeDevin("DEVIN_RESULT"); + const input = { ...options, parent, model, effort: model === "swe-2" ? "high" as const : "default" as const }; + const result = await runLane(input); + expect(result.exitCode).toBe(0); + expect(readFileSync(input.outputPath, "utf8")).toBe("DEVIN_RESULT"); + expect(result.receipt.modelEvidence).toBe("pinned-argv"); + expect(result.receipt.modelVerified).toBe(false); + expect(result.receipt.usage).toBeNull(); + expect(result.receipt.argv).toContain(devinModel(model, input.effort)); + expect(existsSync(devinConfigPath(input))).toBe(false); + expect(existsSync(devinExportDirectory(input))).toBe(false); + }); + } + } + + it("adds writer tool constraints without modifying the assigned prompt", async () => { + const original = "Create a file.\nThen run its test.\n"; + writeFileSync(options.promptPath, original); + fakeDevin("DONE"); + const input = { ...options, mode: "isolated-write" as const }; + const result = await runLane(input); + expect(result.exitCode).toBe(0); + const sent = readFileSync(join(scratch, "captured-prompt.txt"), "utf8"); + expect(sent).toContain("Use sandboxed exec for ALL file creation, modification, and testing"); + expect(sent.endsWith(original)).toBe(true); + expect(readFileSync(options.promptPath, "utf8")).toBe(original); + expect(result.receipt.promptPath).toBe(options.promptPath); + expect(result.receipt.argv).toContain(devinPromptPath(input)); + expect(existsSync(devinExportDirectory(input))).toBe(false); + }); + + it("reports account restrictions without substituting another model", async () => { + fakeDevin("Upgrade to Pro to access this model", 1); + const result = await runLane(options); + expect(result.receipt.status).toBe("unavailable-model"); + expect(existsSync(options.outputPath)).toBe(false); + expect(existsSync(devinConfigPath(options))).toBe(false); + expect(existsSync(devinExportDirectory(options))).toBe(false); + }); + + it("fails authentication even when auth status exits zero", async () => { + fakeDevin("SHOULD_NOT_RUN", 0, "Not logged in. Run devin auth login."); + const result = await runLane(options); + expect(result.receipt.status).toBe("unauthenticated"); + expect(result.receipt.preflight.status).toBe("failed"); + expect(existsSync(options.outputPath)).toBe(false); + }); + + it("does not report an empty successful process as a completed answer", async () => { + fakeDevin(" "); + const result = await runLane(options); + expect(result.receipt.status).toBe("malformed-output"); + expect(existsSync(options.outputPath)).toBe(false); + }); + + const welcome = "\x1b[1mWelcome to Devin CLI!\x1b[0m\n\n ✓ Logged in as test@example.com.\n\n\x1b[?2004lYou're all set. Run \x1b[1mdevin\x1b[0m to get started."; + + it("rejects the observed onboarding-only zero-exit output", async () => { + fakeDevin(welcome, 0, "Logged in (via Devin).", "", { schema_version: "ATIF-v1.7", steps: [] }); + const result = await runLane(options); + expect(result.exitCode).not.toBe(0); + expect(result.receipt.status).toBe("malformed-output"); + expect(existsSync(options.outputPath)).toBe(false); + }); + + it("keeps the answer after a recognized onboarding banner", async () => { + fakeDevin(`${welcome}\nProgress text`, 0, "Logged in (via Devin).", "", transcript("PSTACK_READ_OK")); + const result = await runLane(options); + expect(result.exitCode).toBe(0); + expect(readFileSync(options.outputPath, "utf8")).toBe("PSTACK_READ_OK"); + }); + + it("keeps response text that merely mentions the welcome message", async () => { + const response = "Welcome to Devin CLI! is the banner shown on first run."; + fakeDevin(response); + const result = await runLane(options); + expect(result.exitCode).toBe(0); + expect(readFileSync(options.outputPath, "utf8")).toBe(response); + }); + + it("rejects an incomplete turn when headless tool confirmation fails", async () => { + fakeDevin("I will create the file.", 0, "Logged in (via Devin).", + "warning: rejected a tool call that requires confirmation. Running in non-interactive mode. Use --permission-mode dangerous to auto-approve all tools."); + const result = await runLane({ ...options, mode: "isolated-write" }); + expect(result.receipt.status).toBe("malformed-output"); + expect(existsSync(options.outputPath)).toBe(false); + }); + + for (const [name, exported] of [ + ["progress before rejected tools", transcript("I'll attempt both writes now.", [{ function_name: "write" }])], + ["tool-only turn", transcript("", [{ function_name: "exec" }])], + ["malformed tool calls", transcript("Done", null)], + ["non-agent final step", { schema_version: "ATIF-v1.7", steps: [{ source: "user", message: "Done" }] }], + ["invalid JSON", "PRIVATE_INVALID_JSON"], + ["unknown schema", { schema_version: "ATIF-v9", steps: [] }], + ["malformed steps", { schema_version: "ATIF-v1.7", steps: {} }], + ]) { + it(`rejects ${name} without exposing private transcript content`, async () => { + fakeDevin("Public progress", 0, "Logged in (via Devin).", "", exported); + const result = await runLane(options); + expect(result.receipt.status).toBe("malformed-output"); + expect(existsSync(options.outputPath)).toBe(false); + expect(existsSync(devinExportDirectory(options))).toBe(false); + expect(JSON.stringify(result.receipt)).not.toContain("PRIVATE_"); + }); + } + + it("returns only the final writer message after a successful tool step", async () => { + const exported = transcript("PSTACK_TEST_PASSED"); + exported.steps.splice(1, 0, { source: "agent", message: "Editing", tool_calls: [{ function_name: "exec" }] }); + fakeDevin("Editing\nPSTACK_TEST_PASSED", 0, "Logged in (via Devin).", "", exported); + const result = await runLane({ ...options, mode: "isolated-write" }); + expect(result.exitCode).toBe(0); + expect(readFileSync(options.outputPath, "utf8")).toBe("PSTACK_TEST_PASSED"); + expect(existsSync(devinExportDirectory(options))).toBe(false); + }); + + it("fails closed when the CLI omits its export", async () => { + fakeDevin("Looks finished", 0, "Logged in (via Devin).", "", undefined, true); + const result = await runLane(options); + expect(result.receipt.status).toBe("malformed-output"); + expect(result.receipt.error?.message).toBe("devin did not produce a readable export"); + expect(existsSync(options.outputPath)).toBe(false); + expect(existsSync(devinExportDirectory(options))).toBe(false); + }); + + it("cleans private artifacts when an explicit deadline expires before dispatch", async () => { + fakeDevin("SHOULD_NOT_RUN"); + const result = await runLane({ ...options, timeoutMs: 1 }, Date.now() - 100); + expect(result.receipt.status).toBe("timed-out"); + expect(existsSync(devinConfigPath(options))).toBe(false); + expect(existsSync(devinExportDirectory(options))).toBe(false); + }); + + it("preserves an existing export directory on collision", async () => { + fakeDevin("SHOULD_NOT_RUN"); + mkdirSync(devinExportDirectory(options)); + writeFileSync(devinExportPath(options), "existing"); + const result = await runLane(options); + expect(result.receipt.status).toBe("child-failed"); + expect(readFileSync(devinExportPath(options), "utf8")).toBe("existing"); + expect(existsSync(devinConfigPath(options))).toBe(false); + }); + + it("preserves a pre-existing config path on collision", async () => { + fakeDevin("SHOULD_NOT_RUN"); + writeFileSync(devinConfigPath(options), "existing"); + const result = await runLane(options); + expect(result.receipt.status).toBe("child-failed"); + expect(readFileSync(devinConfigPath(options), "utf8")).toBe("existing"); + }); + + it("does not allow fixed effort for existing providers", async () => { + await expect(runLane({ ...options, provider: "claude", model: "fable", effort: "default" })).rejects.toThrow("only for Devin"); + }); + + it("removes both parent identity sets from Devin's environment", () => { + const env = childEnvironment("devin", { CODEX_THREAD_ID: "c", CLAUDECODE: "1", PATH: "/bin" }); + expect(env).toEqual({ PATH: "/bin" }); + }); +}); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/devin.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/devin.ts new file mode 100644 index 0000000..9d954fd --- /dev/null +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/devin.ts @@ -0,0 +1,71 @@ +import { readFileSync } from "node:fs"; +import { UsageError, type Effort, type RunnerOptions } from "./types.ts"; + +export function devinModel(model: string, effort: Effort): string { + if (model === "swe-2" && ["medium", "high", "max"].includes(effort)) { + return `swe-2-${effort}`; + } + if (model === "swe-1.6" && effort === "default") return "swe-1-6"; + throw new UsageError( + "Devin supports swe-2 at medium/high/max or swe-1.6 at default effort" + ); +} + +export function devinConfigPath(options: RunnerOptions): string { + return `${options.receiptPath}.devin-config.json`; +} + +export function devinExportDirectory(options: RunnerOptions): string { + return `${options.receiptPath}.devin-export`; +} + +export function devinExportPath(options: RunnerOptions): string { + return `${devinExportDirectory(options)}/turn.json`; +} + +export function devinPromptPath(options: RunnerOptions): string { + return options.mode === "isolated-write" + ? `${devinExportDirectory(options)}/prompt.md` + : options.promptPath; +} + +export function devinWriterPrompt(prompt: string): string { + return "Execution constraints for this Devin worker:\n" + + "You are running non-interactively in an isolated-write workspace. " + + "Direct write and edit tools are disabled and terminate the run if attempted. " + + "Use sandboxed exec for ALL file creation, modification, and testing, including the first file operation. " + + "Do not request permissions or use direct write/edit tools. " + + "Keep all changes inside the assigned working directory.\n\nAssigned task:\n" + prompt; +} + +export function readDevinExport(options: RunnerOptions): string { + try { + return readFileSync(devinExportPath(options), "utf8"); + } catch { + throw new Error("devin did not produce a readable export"); + } +} + +export function devinConfig(options: RunnerOptions) { + return { + shell: { setup_complete: true }, + subagents_enabled: false, + auto_update: false, + notify: "never", + read_config_from: { + agents_standard: false, + cursor: false, + windsurf: false, + claude: false, + copilot: false, + opencode: false, + zed: false, + }, + permissions: { + deny: [ + "mcp__*", "fetch", "edit", "write", + ...(options.mode === "read-only" ? ["exec", "Write(**)"] : []), + ], + }, + }; +} diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts index 81ed53d..2ffbef3 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/parse-output.ts @@ -164,6 +164,29 @@ export function parseProviderOutput( requestedModel: string ): ParsedOutput { switch (provider) { + case "devin": { + if (/^warning: rejected a tool call that requires confirmation\./im.test(stderr)) { + throw new Error("devin could not approve a tool in non-interactive mode"); + } + let exported: JsonObject | null; + try { + exported = object(JSON.parse(stdout)); + } catch { + throw new Error("devin export is not valid JSON"); + } + if (exported?.schema_version !== "ATIF-v1.7" || !Array.isArray(exported.steps)) { + throw new Error("devin export has an unsupported schema"); + } + const last = object(exported.steps.at(-1)); + const calls = last?.tool_calls; + if (last?.source !== "agent" || + (calls !== undefined && (!Array.isArray(calls) || calls.length !== 0)) || + typeof last.message !== "string" || last.message.trim().length === 0) { + throw new Error("devin export did not end with a final agent response"); + } + const text = last.message.trim(); + return { text, reportedModel: null, sessionId: null, usage: null, costUsd: null }; + } case "claude": return parseClaude(stdout, requestedModel); case "codex": diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts index 20743b5..5035664 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/run.test.ts @@ -552,26 +552,26 @@ describe("runLane", () => { }); it("spends one explicit deadline across preflight and model execution", async () => { - process.env.FAKE_PREFLIGHT_DELAY_MS = "1200"; - process.env.FAKE_MODEL_DELAY_MS = "1200"; - const input = { ...options("claude"), timeoutMs: 1_500 }; + process.env.FAKE_PREFLIGHT_DELAY_MS = "3000"; + process.env.FAKE_MODEL_DELAY_MS = "6000"; + const input = { ...options("claude"), timeoutMs: 8_000 }; const result = await runLane(input); const recorded = receipt(input.receiptPath); expect(result.exitCode).toBe(124); expect(recorded.status).toBe("timed-out"); expect(recorded.preflight.status).toBe("passed"); - expect(recorded.elapsedMs).toBeLessThan(2_100); - }); + expect(recorded.elapsedMs).toBeLessThan(10_000); + }, 12_000); it("bounds a descendant-held pipe by the explicit deadline without fabricating a signal", async () => { const descendantPidPath = join(scratch, "deadline-descendant.pid"); - const input = { ...options("claude", "deadline-drain"), timeoutMs: 700 }; + const input = { ...options("claude", "deadline-drain"), timeoutMs: 4_000 }; const runner = Bun.spawn([process.execPath, ...runnerArgs(input)], { cwd: scratch, env: { ...process.env, - FAKE_DESCENDANT_HOLDS_PIPES_MS: "5000", + FAKE_DESCENDANT_HOLDS_PIPES_MS: "15000", FAKE_DESCENDANT_PID_PATH: descendantPidPath, }, stdout: "pipe", @@ -580,29 +580,33 @@ describe("runLane", () => { const stdout = new Response(runner.stdout).text(); const stderr = new Response(runner.stderr).text(); - expect(await exitWithin(runner, 2_000)).toBe(124); - await Promise.all([stdout, stderr]); - const recorded = receipt(input.receiptPath); - expect(recorded).toMatchObject({ - status: "timed-out", - exitCode: 0, - signal: null, - preflight: { status: "passed" }, - }); - expect(recorded.elapsedMs).toBeLessThan(1_500); - - const descendantPid = Number(readFileSync(descendantPidPath, "utf8")); - if (processIsAlive(descendantPid)) process.kill(descendantPid, "SIGKILL"); - }); + try { + expect(await exitWithin(runner, 8_000)).toBe(124); + await Promise.all([stdout, stderr]); + const recorded = receipt(input.receiptPath); + expect(recorded).toMatchObject({ + status: "timed-out", + exitCode: 0, + signal: null, + preflight: { status: "passed" }, + }); + expect(recorded.elapsedMs).toBeLessThan(8_000); + } finally { + if (existsSync(descendantPidPath)) { + const descendantPid = Number(readFileSync(descendantPidPath, "utf8")); + if (processIsAlive(descendantPid)) process.kill(descendantPid, "SIGKILL"); + } + } + }, 12_000); it("does not claim a signal was sent to an already signal-reaped child", async () => { const descendantPidPath = join(scratch, "signalled-descendant.pid"); - const input = { ...options("claude", "signalled-drain"), timeoutMs: 700 }; + const input = { ...options("claude", "signalled-drain"), timeoutMs: 4_000 }; const runner = Bun.spawn([process.execPath, ...runnerArgs(input)], { cwd: scratch, env: { ...process.env, - FAKE_DESCENDANT_HOLDS_PIPES_MS: "5000", + FAKE_DESCENDANT_HOLDS_PIPES_MS: "15000", FAKE_DESCENDANT_PID_PATH: descendantPidPath, FAKE_SELF_SIGNAL: "SIGTERM", }, @@ -612,18 +616,22 @@ describe("runLane", () => { const stdout = new Response(runner.stdout).text(); const stderr = new Response(runner.stderr).text(); - expect(await exitWithin(runner, 2_000)).toBe(124); - await Promise.all([stdout, stderr]); - expect(receipt(input.receiptPath)).toMatchObject({ - status: "timed-out", - exitCode: 143, - signal: null, - preflight: { status: "passed" }, - }); - - const descendantPid = Number(readFileSync(descendantPidPath, "utf8")); - if (processIsAlive(descendantPid)) process.kill(descendantPid, "SIGKILL"); - }); + try { + expect(await exitWithin(runner, 8_000)).toBe(124); + await Promise.all([stdout, stderr]); + expect(receipt(input.receiptPath)).toMatchObject({ + status: "timed-out", + exitCode: 143, + signal: null, + preflight: { status: "passed" }, + }); + } finally { + if (existsSync(descendantPidPath)) { + const descendantPid = Number(readFileSync(descendantPidPath, "utf8")); + if (processIsAlive(descendantPid)) process.kill(descendantPid, "SIGKILL"); + } + } + }, 12_000); it("lets manual cancellation end a post-exit pipe drain without a default timeout", async () => { const descendantPidPath = join(scratch, "cancel-descendant.pid"); diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts index 054564a..c1fe340 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/run.ts @@ -4,6 +4,7 @@ import { mkdirSync, openSync, readFileSync, + rmSync, statSync, unlinkSync, writeFileSync, @@ -19,6 +20,7 @@ import type { RunnerReceipt, } from "./types.ts"; import { UsageError } from "./types.ts"; +import { devinConfig, devinConfigPath, devinExportDirectory, devinExportPath, devinModel, devinPromptPath, devinWriterPrompt, readDevinExport } from "./devin.ts"; const ERROR_EVIDENCE_LIMIT = 4_000; const GROK_PREFLIGHT_RETRY_DELAY_MS = 5_000; @@ -368,6 +370,8 @@ function preflightPassed(provider: Provider, model: string, result: ProcessResul } case "codex": return /logged in/i.test(combined); + case "devin": + return /^Logged in\b/im.test(combined); case "grok": return /logged in/i.test(combined) && combined.includes(model); } @@ -383,6 +387,7 @@ function unavailableStatus(value: string): ReceiptStatus { if (/not logged in|unauthenticated|authentication|sign in|login required/i.test(value)) { return "unauthenticated"; } + if (/upgrade to .{0,30}to access this model/i.test(value)) return "unavailable-model"; if (/model.{0,40}(not found|unknown|unavailable|unsupported|not supported|invalid)|invalid.{0,20}model/i.test(value)) { return "unavailable-model"; } @@ -450,7 +455,7 @@ function modelProof( modelEvidence: "provider-report", }; } - if (provider === "codex" && reported === null) { + if ((provider === "codex" || provider === "devin") && reported === null) { return { reportedModel: null, modelVerified: false, @@ -489,6 +494,11 @@ export function validateOptions(options: RunnerOptions): void { ); } if (options.model.trim().length === 0) throw new UsageError("model must not be empty"); + if (options.provider === "devin") { + devinModel(options.model, options.effort); + } else if (options.effort === "default") { + throw new UsageError("default effort is supported only for Devin SWE-1.6"); + } const staleAlias = options.provider === "claude" ? versionedClaudeAlias(options.model) : null; @@ -801,7 +811,7 @@ async function executeLane( try { const parsed = parseProviderOutput( options.provider, - result.stdout, + options.provider === "devin" ? readDevinExport(options) : result.stdout, result.stderr, options.model ); @@ -866,9 +876,25 @@ export async function runLane( argv: [invocation.command, ...invocation.args], }; const cancellation = installRunCancellation(); + let devinConfigCreated = false; + let devinExportCreated = false; try { reserveOutputs(options); try { + if (options.provider === "devin") { + writeFileSync(devinConfigPath(options), JSON.stringify(devinConfig(options)), { + encoding: "utf8", mode: 0o600, flag: "wx", + }); + devinConfigCreated = true; + mkdirSync(devinExportDirectory(options), { mode: 0o700 }); + devinExportCreated = true; + reserve(devinExportPath(options)); + if (options.mode === "isolated-write") { + writeFileSync(devinPromptPath(options), devinWriterPrompt(readFileSync(options.promptPath, "utf8")), { + encoding: "utf8", mode: 0o600, flag: "wx", + }); + } + } return await executeLane( options, cancellation, @@ -921,6 +947,8 @@ export async function runLane( } } finally { cancellation.dispose(); + if (devinConfigCreated) removeIfExists(devinConfigPath(options)); + if (devinExportCreated) rmSync(devinExportDirectory(options), { recursive: true, force: true }); } } diff --git a/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts b/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts index 11c6dfb..d2c9a96 100644 --- a/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts +++ b/plugins/pstack/skills/poteto-mode/scripts/runner/types.ts @@ -1,11 +1,11 @@ export const PARENTS = ["claude", "codex"] as const; -export const PROVIDERS = ["claude", "codex", "grok"] as const; +export const PROVIDERS = ["claude", "codex", "grok", "devin"] as const; export const EFFORTS = ["low", "medium", "high", "xhigh", "max"] as const; export const ACCESS_MODES = ["read-only", "isolated-write"] as const; export type Parent = (typeof PARENTS)[number]; export type Provider = (typeof PROVIDERS)[number]; -export type Effort = (typeof EFFORTS)[number]; +export type Effort = (typeof EFFORTS)[number] | "default"; export type AccessMode = (typeof ACCESS_MODES)[number]; export interface RunnerOptions { diff --git a/plugins/pstack/skills/setup-pstack/SKILL.md b/plugins/pstack/skills/setup-pstack/SKILL.md index 9a0e744..ddd7fe1 100644 --- a/plugins/pstack/skills/setup-pstack/SKILL.md +++ b/plugins/pstack/skills/setup-pstack/SKILL.md @@ -1,6 +1,6 @@ --- name: setup-pstack -description: Configure pstack's provider-qualified models, per-family requested effort, and parent-owned routes per role. Verifies native and external Claude, Codex, and Grok lanes before writing the override sheet. Use for /setup-pstack, "configure pstack models", or changing pstack's model choices. +description: Configure pstack's provider-qualified models, per-family requested effort, and parent-owned routes per role. Verifies native and external Claude, Codex, Grok, and optional Devin lanes before writing the override sheet. Use for /setup-pstack, "configure pstack models", or changing pstack's model choices. --- # Setup pstack @@ -35,7 +35,7 @@ Treat the normalized values as current role-to-family assignments. Overlay those ### 3. Parse per-family efforts -Read the model matrix. Every non-alias value must match `:@`. Map it to exactly one matrix family by `(provider, model)`, require its effort to appear in that row's Selectable efforts cell, and collect the effort. `inherit-parent` and `auto` rows carry no family effort. +Read the model matrix and Optional Devin models table. Every non-alias value must match `:@`. Map it to exactly one family in those tables by `(provider, model)`, require its effort to appear in that row's Selectable efforts cell, and collect the effort. `inherit-parent` and `auto` rows carry no family effort. An unmatched provider/model, out-of-domain effort, duplicate role, or unknown role is inconsistent state. Stop, show the conflicting rows verbatim, and ask for an explicit matrix family or alias replacement. If one or more families have mixed efforts, show every conflicting family and role row, then ask for one normalized effort per family from its Selectable efforts cell. Do not invent a precedence rule. Do not probe or write while any inconsistency is unresolved. @@ -43,11 +43,13 @@ One distinct effort per family is the current value. A family with no non-alias ### 4. Collect one requested effort per family -Ask exactly four effort questions, one each for Fable, Sol, Grok, and Opus. Name each model, its current or proposed value, and the Selectable efforts from its matrix row. Empty input keeps a current value or accepts the matrix proposal for an unassigned family. On a first run, state the four matrix defaults before asking. On a rerun, state the four parsed values without offering to reset customized role lanes. +Before probing, preserve any loaded Devin assignments and ask whether the operator wants to assign named roles to optional Devin workers. Only add or remove those assignments on explicit request. For each selected Devin family, use its Optional Devin models row: SWE-2 offers medium/high/max, while SWE-1.6 uses the fixed `default` token with no effort question. Do not probe or add an unused Devin family. Keep the existing first-run panel unchanged. + +Ask the four baseline effort questions, one each for Fable, Sol, Grok, and Opus. Name each model, its current or proposed value, and the Selectable efforts from its matrix row. Empty input keeps a current value or accepts the matrix proposal for an unassigned family. On a first run, state the four matrix defaults before asking. On a rerun, state the four parsed values without offering to reset customized role lanes. ### 5. Probe the four requested pairs -Probe only the four selected `provider:model@effort` pairs. Run one probe per family, even when two families share a provider. Do not enumerate or offer older models as substitutes. A failed probe writes nothing: report the failing pair and provider, stop, and keep the active sheet plus parent integration bytes unchanged. A failed first run creates neither artifact. +Probe the four baseline selected `provider:model@effort` pairs plus every opted-in Devin pair from step 4. For Devin, run `devin auth status`, inspect `devin models list --format json`, and use the external runner with its model and effort mapping. An account gate or missing CLI leaves both configuration files unchanged. Run one probe per family, even when two families share a provider. Do not enumerate or offer older models as substitutes. A failed probe writes nothing: report the failing pair and provider, stop, and keep the active sheet plus parent integration bytes unchanged. A failed first run creates neither artifact. | Family | Pair source | Claude parent route | Codex parent route | Availability proof | |---|---|---|---|---| @@ -67,11 +69,11 @@ Build the new sheet in memory. Do not write it yet. - First run: start from the complete role assignments in step 7. - Rerun: start from the normalized complete role map from step 2, preserving each loaded row's lane order and family (or alias) per lane. -After effort selection, ask whether to keep those role-to-family assignments or change named roles. Keeping them is the default. Apply only role changes the operator names; never offer a reset of a customized sheet to the first-run assignments. A changed role may use one of the four probed matrix families, `inherit-parent`, or `auto`. +After effort selection, ask whether to keep those role-to-family assignments or change named roles. Keeping them is the default. Apply only role changes the operator names; never offer a reset of a customized sheet to the first-run assignments. A changed role may use one of the four probed matrix families, a probed Devin family selected in step 4, `inherit-parent`, or `auto`. If a newly selected Devin pair was not probed, return to step 5 before rendering. -Require the final role map to contain at least one descriptor from each matrix family. The sheet stores effort only in role descriptors, so an unassigned family's selection cannot persist without adding a second source of truth. +Require the final role map to contain at least one descriptor from each of the four baseline matrix families. The sheet stores effort only in role descriptors, so an unassigned family's selection cannot persist without adding a second source of truth. -Rewrite every matrix-family descriptor to `provider:model@`. Leave `inherit-parent` and `auto` unchanged. An effort-only rerun cannot change a role's family. Changing Grok's effort updates every Grok occurrence and does not move a Sol role onto Grok. Refuse an unqualified slug, an unavailable route, a model other than the four matrix families, or a provider/model mismatch. +Rewrite every matrix-family descriptor to `provider:model@`. Leave `inherit-parent` and `auto` unchanged. An effort-only rerun cannot change a role's family. Changing Grok's effort updates every Grok occurrence and does not move a Sol role onto Grok. Refuse an unqualified slug, an unavailable route, a model outside the default matrix and Optional Devin models table, or a provider/model mismatch. Require each selected Devin family to occur in the final role map; unassigned optional families have no persisted setting. ### 7. Confirm and commit @@ -79,7 +81,7 @@ Show any rolling-alias migrations as original and normalized descriptors. Then s Why and Reflect require the parent's live MCP surface. Keep their investigator, reviewer, and synthesizer roles on `inherit-parent` or `auto`; the bounded external runner deliberately omits ambient MCPs. `inherit-parent` and `auto` always validate, but say when they reduce a panel's provider diversity. For panel roles, one lane runs per entry. The list length is the fan-out count. `arena cross-judge pool` is a list from which Arena chooses a provider different from the parent and base candidate when possible. `swarm workers` is the default for every worker unless a race explicitly assigns another descriptor. -Every non-alias value must match `:@` and must have passed step 5. +Every non-alias value must match `:@` and must have passed step 5, including optional Devin pairs. After the operator confirms, write the in-memory render from step 6. Never paste the example below as the result. It is only the complete first-run role map used to seed step 2; selected efforts and explicit role changes always replace its example values before writing. @@ -109,12 +111,12 @@ interrogate reviewers: claude:fable@max, codex:gpt-5.6-sol@max, grok:grok-4.6@xh Render the parent integration in memory before either write. On Claude, the integration is the single `@~/.claude/pstack-models.md` include in `~/.claude/CLAUDE.md`. On Codex, it is the exact sheet bytes between one `` and `` pair in `~/.codex/AGENTS.md`. Replace that whole bounded block on a rerun. Insert one block at the end on first run. If either marker is missing, duplicated, or reversed, stop and report inconsistent state instead of guessing a boundary. -Snapshot every target's current bytes. Write the sheet and parent integration only after all four probes pass and the operator confirms. Read both targets back and compare them with the in-memory render. If either write or readback fails, restore every snapshot and report the failure. An unchanged rerun must produce byte-identical sheet and integration content after normalization. +Snapshot every target's current bytes. Write the sheet and parent integration only after all baseline and selected Devin probes pass and the operator confirms. Read both targets back and compare them with the in-memory render. If either write or readback fails, restore every snapshot and report the failure. An unchanged rerun must produce byte-identical sheet and integration content after normalization. Do not copy the model sheet between harnesses without rerunning the parent-specific probes; route availability can differ even on the same host. ### 9. Behavioral smoke -Before declaring setup complete, run one small read-only mixed panel from this parent: all four chosen descriptors, distinct output/receipt paths, and an independent cross-judge. Launch Claude-native agents and every external process in the background with retained handles, then drain them. Verify the native transcript entries and every external receipt. A structural config check or unit test is not a substitute. +Before declaring setup complete, run one small read-only mixed panel from this parent: all four baseline descriptors and each selected Devin descriptor, distinct output/receipt paths, and an independent cross-judge. Launch Claude-native agents and every external process in the background with retained handles, then drain them. Verify the native transcript entries and every external receipt. A structural config check or unit test is not a substitute. Report the sheet path, parent route table, requested-effort probe results, smoke results, and external elapsed/token/cost receipts. Re-running this skill re-probes and updates the same sheet. Do not claim the provider exposed hidden applied-effort observability.