Skip to content

feat(benchmarks): Support local Claude UI benchmark suites #794

feat(benchmarks): Support local Claude UI benchmark suites

feat(benchmarks): Support local Claude UI benchmark suites #794

Triggered via pull request May 26, 2026 02:10
Status Success
Total duration 20s
Artifacts

secret-scan.yml Required

on: pull_request
Secret Scan
15s
Secret Scan
Fit to window
Zoom out
Zoom in

Annotations

5 warnings
Unit test spawns real python3 subprocess to invoke parser script: src/benchmarks/claude-ui/__tests__/claude-ui-benchmark.test.ts#L1
`claude-ui-tool-config.test.ts` defines `runParserScript` (lines 45–62) which calls `spawn('python3', args, ...)` and invokes it at line 665 against the on-disk `benchmarks/claude-ui/parse_claude_conversation.py`, asserting on real filesystem output (`readdir`, `readFile`). This violates the unit-test guardrail that external processes/filesystem dependencies must be injected/mocked rather than executed for real. The test depends on a `python3` interpreter being installed on the host, unlike the sibling new tests (e.g. `first-run-preflight.test.ts`, `simulator-lifecycle.test.ts`) which correctly inject a `LifecycleCommandExecutor` mock.
`stringifyContent(block.content)` may return `undefined`, causing `message.length` to throw: src/benchmarks/claude-ui/transcript.ts#L399
If a `tool_result` block has no `content` field, `JSON.stringify(undefined)` returns `undefined` at runtime (not a string), making `message.length` on the next line throw a `TypeError`. Consider defaulting: `stringifyContent(block.content ?? '')`
`timedOut` flag overwrites exit code even when task already signalled completion: src/benchmarks/claude-ui/harness.ts#L333
If the overall `timeoutMs` fires while the `terminalJsonResultGraceMs` grace timer is still running (i.e., Claude emitted a `result` record before the deadline but cleanup hadn't finished), `timedOut` is set to `true` and the resolved exit code is hard-coded to 143 — discarding `terminalResultExitCode` that was already recorded. A task that completed before the timeout is then reported as timed out.
Single Bash tool_result error produces duplicate failure entries per matched command prefix: src/benchmarks/claude-ui/transcript.ts#L405
When a Bash command contains a tracked prefix multiple times (e.g. `xcodebuild build && xcodebuild test`), `classifyBashCommandTool` returns one `ToolClassification` per occurrence; if the tool result errors, the loop `for (const trackedTool of trackedTools)` pushes a separate failure record for each classification, inflating `audit.failures.length` used by `countCompletionIssues`.
defaultLifecycleLogWriter and tryAppendLifecycleLog are verbatim duplicates of unexported helpers in simulator-lifecycle.ts: src/benchmarks/claude-ui/simulator-deletion.ts#L9
Both `defaultLifecycleLogWriter` (lines 9–11) and `tryAppendLifecycleLog` (lines 23–32) are identical to their counterparts in `simulator-lifecycle.ts`; export both from that module and import them here to eliminate the redundancy.