[Fix] Tasks stall when interrupted subtasks resume

docs/architecture/native-tool-call-parser-scoping-model.md

@@ -0,0 +1,46 @@
+# Native tool-call parser request-scope model check
+
+Zoo Code checks native tool-call parser request isolation with a bounded, exhaustive replay model. It is a child submodel in the umbrella task lifecycle verification suite, which runs in CI and locally with:
+
+```sh
+pnpm lifecycle:model-check
+```
+
+For focused debugging, run this submodel directly with:
+
+```sh
+pnpm parser-scope:model-check
+```
+
+The command is composed into the same verification suite, but this remains a separate protocol and state space from the persisted task lifecycle model and shared-store concurrency model. It owns its parser-scoping invariants and adds no parser state to `HistoryItem` or `taskLifecycle.ts`; instead, it replays the public production `NativeToolCallParser` APIs using two independent scope objects.
+
+## Bounds and replay
+
+The source of truth is `scripts/check-native-tool-call-parser-scoping.ts`. The model has two request scopes, A and B. Both receive provider raw tool index zero, but each has a distinct tool-call ID and two distinct JSON argument fragments. Each scope follows this local order:
+
+1. open the request scope;
+2. start raw call index zero and its streaming accumulator;
+3. add two distinct argument fragments through both production accumulation APIs;
+4. finalize the raw call and reject duplicate raw finalization;
+5. finalize the streaming call, reject duplicate streaming finalization, and clear both kinds of state; and
+6. deliver late raw and streaming fragments.
+
+The checker exhausts all 924 order-preserving interleavings of those two six-action sequences. Opening, raw start, fragment delivery, raw finalization, streaming finalization/cleanup, and late fragment delivery are independently schedulable protocol phases. Fragment delivery remains one bounded action per scope and replays both argument fragments through both production accumulation APIs; streaming cleanup remains attached to streaming finalization because late delivery is the only valid following local phase. This preserves each request's local order while keeping CI runtime bounded. The expected schedule count, maximum schedule budget, scope count, raw index, and actions per scope are explicit. It fails if schedule enumeration differs from the binomial bound or exceeds the budget, so truncated exploration cannot pass.
+
+Each schedule uses fresh production scope objects and calls `processRawChunk`, `startStreamingToolCall`, `processStreamingChunk`, `finalizeRawChunks`, `finalizeStreamingToolCall`, `clearRawChunkState`, `clearAllStreamingToolCalls`, and `hasActiveStreamingToolCalls`. It neither inspects private parser maps nor duplicates their transition logic.
+
+## Invariants and landmarks
+
+Every replay checks:
+
+1. emitted start, delta, and end events retain the owning scope's call ID;
+2. finalized arguments contain only the owning scope's fragments;
+3. cleanup in one scope cannot change the other scope's active streaming state;
+4. each scope emits exactly one raw end and one streaming final result;
+5. repeated finalization is empty/null rather than duplicate;
+6. late raw and streaming fragments are ignored after cleanup;
+7. every modeled action is reachable.
+
+Named landmarks require simultaneous active scopes, B opening while A has received its fragments, either scope raw-finalizing while the other remains active, either scope streaming-finalizing and cleaning up while the other remains active, and symmetric late-fragment schedules in which the other scope is still active.
+
+These are finite safety claims only. The model does not claim provider transport ordering, retry liveness, fairness, persistence, or arbitrary call counts. Provider suites separately test their public stream contracts with two overlapping streams, while focused parser and Task tests cover production integration.

package.json

@@ -13,7 +13,8 @@
 		"check-types": "turbo check-types --log-order grouped --output-logs new-only",
 		"test": "turbo test --log-order grouped --output-logs new-only",
 		"test:mutation-ci": "node --test scripts/stryker-diff.test.mjs",
-		"lifecycle:model-check": "tsx scripts/check-task-lifecycle.ts && tsx scripts/check-task-store-concurrency.ts",
+		"lifecycle:model-check": "tsx scripts/check-task-lifecycle.ts && tsx scripts/check-task-store-concurrency.ts && pnpm parser-scope:model-check",
+		"parser-scope:model-check": "node scripts/run-native-tool-call-parser-scoping.mjs",
 		"test:coverage": "turbo test:coverage --log-order grouped --output-logs new-only",
 		"format": "turbo format --log-order grouped --output-logs new-only",
 		"build": "turbo build --log-order grouped --output-logs new-only",

scripts/check-native-tool-call-parser-scoping.ts

@@ -0,0 +1,315 @@
+import assert from "node:assert/strict"
+
+import { NativeToolCallParser, type ToolCallStreamEvent } from "../src/core/assistant-message/NativeToolCallParser"
+
+const scopeIds = ["A", "B"] as const
+type ScopeId = (typeof scopeIds)[number]
+
+const localActions = [
+	"open",
+	"start-raw-call",
+	"add-fragments",
+	"finalize-raw-call",
+	"finalize-streaming-call-and-cleanup",
+	"late-fragments",
+] as const
+type LocalAction = (typeof localActions)[number]
+
+interface ScheduledAction {
+	scopeId: ScopeId
+	action: LocalAction
+}
+
+interface ScopeReplayState {
+	scope?: object
+	rawEndCount: number
+	streamFinalizationCount: number
+	lateFragmentsIgnored: boolean
+}
+
+interface ReplayState {
+	scopes: Record<ScopeId, ScopeReplayState>
+	events: Array<{ owner: ScopeId; event: ToolCallStreamEvent }>
+}
+
+const RAW_TOOL_INDEX = 0
+const MAX_ACTIONS_PER_SCOPE = localActions.length
+const MAX_TOTAL_ACTIONS = MAX_ACTIONS_PER_SCOPE * scopeIds.length
+const EXPECTED_SCHEDULES = binomial(MAX_TOTAL_ACTIONS, MAX_ACTIONS_PER_SCOPE)
+const MAX_SCHEDULES = EXPECTED_SCHEDULES
+
+const callIds = { A: "call_scope_a", B: "call_scope_b" } satisfies Record<ScopeId, string>
+const paths = { A: "scope-a.ts", B: "scope-b.ts" } satisfies Record<ScopeId, string>
+const fragments = {
+	A: ['{"path":"scope-', 'a.ts"}'],
+	B: ['{"path":"scope-', 'b.ts"}'],
+} satisfies Record<ScopeId, readonly [string, string]>
+
+const expectedActions = new Set<LocalAction>(localActions)
+const reachedActions = new Set<LocalAction>()
+const reachedLandmarks = new Set<string>()
+
+const landmarkNames = [
+	"simultaneous-active-scopes",
+	"B-opens-while-A-is-partial",
+	"A-raw-finalizes-while-B-is-active",
+	"B-raw-finalizes-while-A-is-active",
+	"A-stream-finalizes-while-B-is-active",
+	"B-stream-finalizes-while-A-is-active",
+	"A-late-fragment-while-B-is-active",
+	"B-late-fragment-while-A-is-active",
+] as const
+
+function binomial(n: number, k: number): number {
+	let result = 1
+	for (let index = 1; index <= k; index++) {
+		result = (result * (n - k + index)) / index
+	}
+	return result
+}
+
+function initialReplayState(): ReplayState {
+	return {
+		scopes: {
+			A: { rawEndCount: 0, streamFinalizationCount: 0, lateFragmentsIgnored: false },
+			B: { rawEndCount: 0, streamFinalizationCount: 0, lateFragmentsIgnored: false },
+		},
+		events: [],
+	}
+}
+
+function activeAtProgress(progress: number): boolean {
+	return (
+		progress >= localActions.indexOf("start-raw-call") + 1 &&
+		progress < localActions.indexOf("finalize-streaming-call-and-cleanup") + 1
+	)
+}
+
+function appendOwnedEvents(state: ReplayState, owner: ScopeId, events: ToolCallStreamEvent[]): void {
+	for (const event of events) {
+		state.events.push({ owner, event })
+		assert.equal(event.id, callIds[owner], `${owner} emitted an event owned by the other request scope`)
+	}
+}
+
+function requireScope(state: ReplayState, scopeId: ScopeId): object {
+	const scope = state.scopes[scopeId].scope
+	assert.ok(scope, `${scopeId} must be opened before ${scopeId}'s parser APIs are replayed`)
+	return scope
+}
+
+function replayAction(state: ReplayState, scheduled: ScheduledAction): void {
+	const { scopeId, action } = scheduled
+	const scopeState = state.scopes[scopeId]
+	reachedActions.add(action)
+
+	switch (action) {
+		case "open":
+			scopeState.scope = NativeToolCallParser.createScope()
+			break
+		case "start-raw-call": {
+			const scope = requireScope(state, scopeId)
+			const events = NativeToolCallParser.processRawChunk(
+				{ index: RAW_TOOL_INDEX, id: callIds[scopeId], name: "read_file" },
+				scope,
+			)
+			assert.deepEqual(events, [{ type: "tool_call_start", id: callIds[scopeId], name: "read_file" }])
+			appendOwnedEvents(state, scopeId, events)
+			NativeToolCallParser.startStreamingToolCall(callIds[scopeId], "read_file", scope)
+			break
+		}
+		case "add-fragments": {
+			const scope = requireScope(state, scopeId)
+			for (const fragment of fragments[scopeId]) {
+				const events = NativeToolCallParser.processRawChunk(
+					{ index: RAW_TOOL_INDEX, arguments: fragment },
+					scope,
+				)
+				assert.deepEqual(events, [{ type: "tool_call_delta", id: callIds[scopeId], delta: fragment }])
+				appendOwnedEvents(state, scopeId, events)
+				assert.notEqual(
+					NativeToolCallParser.processStreamingChunk(callIds[scopeId], fragment, scope),
+					null,
+					`${scopeId}'s fragment was not accepted by its streaming accumulator`,
+				)
+			}
+			break
+		}
+		case "finalize-raw-call": {
+			const scope = requireScope(state, scopeId)
+			const events = NativeToolCallParser.finalizeRawChunks(scope)
+			assert.deepEqual(events, [{ type: "tool_call_end", id: callIds[scopeId] }])
+			appendOwnedEvents(state, scopeId, events)
+			scopeState.rawEndCount += events.length
+			assert.deepEqual(
+				NativeToolCallParser.finalizeRawChunks(scope),
+				[],
+				`${scopeId} emitted a duplicate raw end`,
+			)
+			break
+		}
+		case "finalize-streaming-call-and-cleanup": {
+			const scope = requireScope(state, scopeId)
+			const result = NativeToolCallParser.finalizeStreamingToolCall(callIds[scopeId], scope)
+			assert.equal(result?.type, "tool_use")
+			if (result?.type !== "tool_use" || result.name !== "read_file") {
+				throw new Error(`${scopeId}'s streaming result was not a read_file tool use`)
+			}
+			if (!result.nativeArgs || !("path" in result.nativeArgs)) {
+				throw new Error(`${scopeId}'s streaming result did not use current read_file arguments`)
+			}
+			assert.equal(result.nativeArgs?.path, paths[scopeId], `${scopeId}'s arguments crossed request scopes`)
+			scopeState.streamFinalizationCount += 1
+			assert.equal(
+				NativeToolCallParser.finalizeStreamingToolCall(callIds[scopeId], scope),
+				null,
+				`${scopeId} finalized its streaming call twice`,
+			)
+			NativeToolCallParser.clearRawChunkState(scope)
+			NativeToolCallParser.clearAllStreamingToolCalls(scope)
+			break
+		}
+		case "late-fragments": {
+			const scope = requireScope(state, scopeId)
+			const rawEvents = NativeToolCallParser.processRawChunk(
+				{ index: RAW_TOOL_INDEX, arguments: `late-${scopeId}` },
+				scope,
+			)
+			const streamingResult = NativeToolCallParser.processStreamingChunk(
+				callIds[scopeId],
+				`late-${scopeId}`,
+				scope,
+			)
+			assert.deepEqual(rawEvents, [], `${scopeId} accepted a late raw fragment`)
+			assert.equal(streamingResult, null, `${scopeId} accepted a late streaming fragment`)
+			scopeState.lateFragmentsIgnored = true
+			break
+		}
+	}
+}
+
+function checkInvariants(state: ReplayState, progress: Record<ScopeId, number>, trace: ScheduledAction[]): void {
+	for (const scopeId of scopeIds) {
+		const scopeState = state.scopes[scopeId]
+		const scope = scopeState.scope
+		const expectedActive = activeAtProgress(progress[scopeId])
+		assert.equal(
+			scope ? NativeToolCallParser.hasActiveStreamingToolCalls(scope) : false,
+			expectedActive,
+			`${scopeId}'s active streaming state was changed by the other request scope`,
+		)
+		assert.ok(scopeState.rawEndCount <= 1, `${scopeId} emitted duplicate raw finalization events`)
+		assert.ok(scopeState.streamFinalizationCount <= 1, `${scopeId} finalized its streaming call more than once`)
+	}
+
+	for (const { owner, event } of state.events) {
+		assert.equal(event.id, callIds[owner], `${owner}'s event log contains another scope's call ID`)
+	}
+
+	const last = trace.at(-1)
+	if (!last) return
+	if (activeAtProgress(progress.A) && activeAtProgress(progress.B)) reachedLandmarks.add("simultaneous-active-scopes")
+	if (last.scopeId === "B" && last.action === "open" && progress.A === 3) {
+		reachedLandmarks.add("B-opens-while-A-is-partial")
+	}
+	if (last.action === "finalize-raw-call" && activeAtProgress(progress[last.scopeId === "A" ? "B" : "A"])) {
+		reachedLandmarks.add(`${last.scopeId}-raw-finalizes-while-${last.scopeId === "A" ? "B" : "A"}-is-active`)
+	}
+	if (
+		last.action === "finalize-streaming-call-and-cleanup" &&
+		activeAtProgress(progress[last.scopeId === "A" ? "B" : "A"])
+	) {
+		reachedLandmarks.add(`${last.scopeId}-stream-finalizes-while-${last.scopeId === "A" ? "B" : "A"}-is-active`)
+	}
+	if (last.action === "late-fragments" && activeAtProgress(progress[last.scopeId === "A" ? "B" : "A"])) {
+		reachedLandmarks.add(`${last.scopeId}-late-fragment-while-${last.scopeId === "A" ? "B" : "A"}-is-active`)
+	}
+}
+
+function cleanupReplay(state: ReplayState): void {
+	for (const scopeId of scopeIds) {
+		const scope = state.scopes[scopeId].scope
+		if (!scope) continue
+		NativeToolCallParser.clearRawChunkState(scope)
+		NativeToolCallParser.clearAllStreamingToolCalls(scope)
+	}
+}
+
+function replaySchedule(trace: ScheduledAction[]): void {
+	const state = initialReplayState()
+	const progress: Record<ScopeId, number> = { A: 0, B: 0 }
+	try {
+		for (const scheduled of trace) {
+			replayAction(state, scheduled)
+			progress[scheduled.scopeId] += 1
+			checkInvariants(state, progress, trace.slice(0, progress.A + progress.B))
+		}
+		for (const scopeId of scopeIds) {
+			assert.equal(state.scopes[scopeId].rawEndCount, 1, `${scopeId} did not emit exactly one raw end`)
+			assert.equal(state.scopes[scopeId].streamFinalizationCount, 1, `${scopeId} did not finalize exactly once`)
+			assert.equal(
+				state.scopes[scopeId].lateFragmentsIgnored,
+				true,
+				`${scopeId}'s late fragments were not checked`,
+			)
+		}
+	} catch (error) {
+		const formattedTrace = trace
+			.map(({ scopeId, action }, index) => `${index + 1}. ${scopeId}.${action}`)
+			.join("\n")
+		throw new Error(
+			`Native tool-call parser scope invariant failed within bounds scopes=${scopeIds.length}, actions-per-scope=${MAX_ACTIONS_PER_SCOPE}, schedules=${MAX_SCHEDULES}\n${formattedTrace}`,
+			{ cause: error },
+		)
+	} finally {
+		cleanupReplay(state)
+	}
+}
+
+function enumerateSchedules(): number {
+	const trace: ScheduledAction[] = []
+	const progress: Record<ScopeId, number> = { A: 0, B: 0 }
+	let exploredSchedules = 0
+
+	function visit(): void {
+		if (trace.length === MAX_TOTAL_ACTIONS) {
+			exploredSchedules += 1
+			if (exploredSchedules > MAX_SCHEDULES) {
+				throw new Error(`Parser-scope exploration exceeded its ${MAX_SCHEDULES}-schedule budget`)
+			}
+			replaySchedule(trace)
+			return
+		}
+
+		for (const scopeId of scopeIds) {
+			const localProgress = progress[scopeId]
+			if (localProgress === MAX_ACTIONS_PER_SCOPE) continue
+			const action = localActions[localProgress]
+			if (!action) throw new Error(`${scopeId} has no modeled action at local progress ${localProgress}`)
+			trace.push({ scopeId, action })
+			progress[scopeId] += 1
+			visit()
+			progress[scopeId] -= 1
+			trace.pop()
+		}
+	}
+
+	visit()
+	return exploredSchedules
+}
+
+const exploredSchedules = enumerateSchedules()
+assert.equal(
+	exploredSchedules,
+	EXPECTED_SCHEDULES,
+	`Parser-scope exploration truncated: expected ${EXPECTED_SCHEDULES} schedules, explored ${exploredSchedules}`,
+)
+
+const unreachableActions = [...expectedActions].filter((action) => !reachedActions.has(action))
+assert.deepEqual(unreachableActions, [], `Parser-scope model has unreachable actions: ${unreachableActions.join(", ")}`)
+const missingLandmarks = landmarkNames.filter((name) => !reachedLandmarks.has(name))
+assert.deepEqual(missingLandmarks, [], `Parser-scope model has unreachable landmarks: ${missingLandmarks.join(", ")}`)
+
+console.log(
+	`Native tool-call parser scope model check passed: ${exploredSchedules}/${EXPECTED_SCHEDULES} valid local-order interleavings, ${localActions.length}/${localActions.length} actions reachable, ${landmarkNames.length}/${landmarkNames.length} landmarks reached, scopes=${scopeIds.length}, raw-index=${RAW_TOOL_INDEX}, actions-per-scope=${MAX_ACTIONS_PER_SCOPE}`,
+)

scripts/run-native-tool-call-parser-scoping.mjs

@@ -0,0 +1,24 @@
+import { rm } from "node:fs/promises"
+import { tmpdir } from "node:os"
+import { join } from "node:path"
+import { fileURLToPath, pathToFileURL } from "node:url"
+
+import { build } from "esbuild"
+
+const entryPoint = fileURLToPath(new URL("./check-native-tool-call-parser-scoping.ts", import.meta.url))
+const outfile = join(tmpdir(), `zoo-parser-scope-model-${process.pid}.cjs`)
+
+try {
+	await build({
+		entryPoints: [entryPoint],
+		bundle: true,
+		platform: "node",
+		format: "cjs",
+		external: ["vscode"],
+		outfile,
+		logLevel: "info",
+	})
+	await import(pathToFileURL(outfile).href)
+} finally {
+	await rm(outfile, { force: true })
+}