diff --git a/.claude/superpowers/plans/2026-06-03-m5stick-voice-remote-control.md b/.claude/superpowers/plans/2026-06-03-m5stick-voice-remote-control.md deleted file mode 100644 index 11fbddf..0000000 --- a/.claude/superpowers/plans/2026-06-03-m5stick-voice-remote-control.md +++ /dev/null @@ -1,595 +0,0 @@ -# m5stick-voice Remote Control Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Say "ok push app" in the m5stick-voice viewer and have the server push the simulator's current Lua app to the physical device, which suspends any running app while in push-to-talk "Listening" mode and resumes it on release. - -**Architecture:** Add a small app-suspend primitive to `Resident::Sandbox` (pause/resume the Lua tick without unloading), wire the device's push-to-talk handler to it, add a `repaint()` to the shared DisplayDriver so the app's last frame is restored on resume, and add a `push_app` realtime function tool to the server's `VoiceAgent` that sends `{type:"app", code}` to the device connection. Bump the library to `0.5.1-dev`. - -**Tech Stack:** C++17 / Arduino / PlatformIO (device + library), Esp32Lua, TanStack Start + Cloudflare Agents SDK + Vitest (server). - ---- - -## Testing posture (read first) - -- **Firmware/library (Tasks 1–3, 5):** the `test/unit` native harness cannot construct a `Resident::Sandbox` (no native stubs for Courier::Client / ezTime / ArduinoJson; no test instantiates `Sandbox`). Per the project's firmware-verify rule, these changes are verified by the **build gate** + **on-hardware test** (Task 7), not new unit tests. This is a deliberate, approved decision (see the spec). -- **Server (Task 4):** verified by `typecheck` + `build` + the existing Vitest suite staying green. `push_app` is I/O glue over the Agents SDK connection API, which has no DO test harness here, so no contrived unit test is added. -- **Commits:** the skill's "frequent commits" is overridden by the user's firmware-verify rule (user instructions win). The **server** change commits after its checks pass (Task 4). All **firmware/library** changes are staged but **held uncommitted** until the on-hardware checkpoint (Task 7), then committed in Task 8. - -## Pre-flight (existing uncommitted state) - -`git status` currently shows, in addition to this feature's future edits: -- `examples/m5stick-voice/device/src/main.cpp` — the deviceId-under-prompt edit (already hardware-verified: "firmware looks fine"). It will ride along with this feature's `main.cpp` changes in the Task 8 firmware commit. -- `CLAUDE.md`, `.gitignore` — the Superpowers-working-docs policy change. These are unrelated to the feature; recommend committing them on their own: - ```bash - git add CLAUDE.md .gitignore - git commit -m "docs: route Superpowers working docs to .claude/superpowers/" - ``` - Do this whenever convenient — not gated by hardware. - -## File structure - -- `src/ResidentSandbox.h` — declare `suspendApp()`/`resumeApp()`/`isAppSuspended()` + `_appSuspended` member. -- `src/ResidentSandbox.cpp` — implement them; gate the `loop()` tick; clear the flag in `loadApp()`. -- `examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.h` / `.cpp` — add `repaint()` (shared by m5stick-demo + m5stick-voice). -- `examples/m5stick-voice/device/src/main.cpp` — suspend on talk, resume+repaint on release. -- `examples/m5stick-voice/server/src/agents/voice-agent.ts` — `push_app` tool, handler, system-prompt line, `DEFAULT_APP` import. -- `library.json`, `idf_component.yml`, `docs/changelog.md` — version bump + changelog. - ---- - -### Task 1: Library — app-suspend primitive - -**Files:** -- Modify: `src/ResidentSandbox.h` -- Modify: `src/ResidentSandbox.cpp` - -- [ ] **Step 1: Declare the public methods** - -In `src/ResidentSandbox.h`, replace: - -```cpp - // State queries - bool isAppRunning() const; - - // Timezone — no-op on nullptr/empty. Success means ezTime resolved the -``` - -with: - -```cpp - // State queries - bool isAppRunning() const; - - // App suspend/resume. Pauses the Lua tick (on_tick + event dispatch) - // without unloading the app — Courier and extension update() keep running. - // While suspended the status display is freed (notifyAppRunning(false)) so - // displayText() can show e.g. a "Listening" overlay. Both are no-ops when - // no app is loaded. isAppRunning() stays true while suspended; suspension - // is a separate axis queried via isAppSuspended(). - void suspendApp(); - void resumeApp(); - bool isAppSuspended() const; - - // Timezone — no-op on nullptr/empty. Success means ezTime resolved the -``` - -- [ ] **Step 2: Declare the private member** - -In `src/ResidentSandbox.h`, replace: - -```cpp - struct lua_State* _lua = nullptr; - bool _appRunning = false; -``` - -with: - -```cpp - struct lua_State* _lua = nullptr; - bool _appRunning = false; - bool _appSuspended = false; -``` - -- [ ] **Step 3: Gate the tick in loop()** - -In `src/ResidentSandbox.cpp`, replace: - -```cpp - // Lua tick + event dispatch only when an app is running. - if (!_appRunning) return; -``` - -with: - -```cpp - // Lua tick + event dispatch only when an app is running and not suspended. - if (!_appRunning || _appSuspended) return; -``` - -- [ ] **Step 4: Clear the flag in loadApp()** - -In `src/ResidentSandbox.cpp`, replace: - -```cpp -void Sandbox::loadApp(const char* luaCode) -{ - // Stop current app before loading new one - if (_appRunning) { -``` - -with: - -```cpp -void Sandbox::loadApp(const char* luaCode) -{ - // A freshly loaded app starts running, never suspended. - _appSuspended = false; - - // Stop current app before loading new one - if (_appRunning) { -``` - -- [ ] **Step 5: Implement the three methods** - -In `src/ResidentSandbox.cpp`, replace: - -```cpp -bool Sandbox::isAppRunning() const -{ - return _appRunning; -} - -// --- Lua compilation --- -``` - -with: - -```cpp -bool Sandbox::isAppRunning() const -{ - return _appRunning; -} - -void Sandbox::suspendApp() -{ - if (!_appRunning || _appSuspended) return; - _appSuspended = true; - notifyAppRunning(false); // free the status display for overlay text -} - -void Sandbox::resumeApp() -{ - if (!_appRunning || !_appSuspended) return; - _appSuspended = false; - notifyAppRunning(true); // re-suppress status display; app owns the screen -} - -bool Sandbox::isAppSuspended() const -{ - return _appSuspended; -} - -// --- Lua compilation --- -``` - -- [ ] **Step 6: Verify it compiles into firmware** - -Run: `cd examples/m5stick-voice/device && pio run -e m5stick` -Expected: `[SUCCESS]` (links `firmware.elf`). ~10s. - -- [ ] **Step 7: Verify the native suite is unaffected** - -Run: `cd /Users/matt/code/resident && ./tools/run-tests.py unit` -Expected: all tests pass (no Sandbox tests exist; this confirms the header/impl change didn't break the config/lua tests). - -> Do NOT commit yet — firmware/library changes are held until the Task 7 hardware checkpoint. - ---- - -### Task 2: Shared DisplayDriver — repaint() - -**Files:** -- Modify: `examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.h` -- Modify: `examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.cpp` - -- [ ] **Step 1: Declare repaint()** - -In `examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.h`, replace: - -```cpp - // Call once after M5.begin() to create the sprite framebuffer - void begin() override; -``` - -with: - -```cpp - // Call once after M5.begin() to create the sprite framebuffer - void begin() override; - - // Re-push the current off-screen sprite to the display without redrawing it. - // Restores the last app frame after a direct displayText() overlay (e.g. the - // m5stick-voice "Listening" prompt) so static apps don't stay blank on resume. - void repaint(); -``` - -- [ ] **Step 2: Implement repaint()** - -In `examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.cpp`, replace: - -```cpp -void DisplayDriver::onAppReset() { -``` - -with: - -```cpp -void DisplayDriver::repaint() { - if (_initialized) _canvas.pushSprite(0, 0); -} - -void DisplayDriver::onAppReset() { -``` - -- [ ] **Step 3: Verify it compiles** - -Run: `cd examples/m5stick-voice/device && pio run -e m5stick` -Expected: `[SUCCESS]`. - -> Do NOT commit yet. - ---- - -### Task 3: Device example — wire push-to-talk to suspend/resume - -**Files:** -- Modify: `examples/m5stick-voice/device/src/main.cpp` - -- [ ] **Step 1: Suspend the app when a talk hold starts** - -In `examples/m5stick-voice/device/src/main.cpp`, replace: - -```cpp - dbgHoldStarts++; - streaming = true; - displayDriver.displayText("Listening"); -``` - -with: - -```cpp - dbgHoldStarts++; - streaming = true; - if (sandbox.isAppRunning()) sandbox.suspendApp(); - displayDriver.displayText("Listening"); -``` - -- [ ] **Step 2: Resume the app (or show the idle prompt) on release** - -In `examples/m5stick-voice/device/src/main.cpp`, replace: - -```cpp - } else { - streaming = false; - showIdlePrompt(); - Serial.printf("[voice] %lu HOLD end -> stopped " -``` - -with: - -```cpp - } else { - streaming = false; - if (sandbox.isAppRunning()) { - sandbox.resumeApp(); - displayDriver.repaint(); // restore the app's last frame immediately - } else { - showIdlePrompt(); - } - Serial.printf("[voice] %lu HOLD end -> stopped " -``` - -- [ ] **Step 3: Verify it compiles** - -Run: `cd examples/m5stick-voice/device && pio run -e m5stick` -Expected: `[SUCCESS]`. - -> Do NOT commit yet — firmware changes wait for Task 7. - ---- - -### Task 4: Server — push_app realtime tool - -**Files:** -- Modify: `examples/m5stick-voice/server/src/agents/voice-agent.ts` - -- [ ] **Step 1: Import the default app** - -In `voice-agent.ts`, replace: - -```ts -import { validateLuaCode } from "../lib/lua-validator" -``` - -with: - -```ts -import { validateLuaCode } from "../lib/lua-validator" -import { DEFAULT_APP } from "../lib/default-app" -``` - -- [ ] **Step 2: Describe push_app in the system prompt** - -In `voice-agent.ts`, replace: - -```ts -2. **create_app** — generate and run a Lua app on the SIMULATED M5StickC DEVICE shown on the page (a small 240×135 screen with two buttons). Use when the user asks for something to happen "on the device", "on the m5stick", "on the screen", asks for a clock, a counter, a game, a bouncing ball, anything interactive. Returns asynchronously — the coding agent writes Lua and pushes it; the user sees status in the UI. - -For ambiguous requests like "show stripes", default to apply_css (the page) unless the user mentioned the device. Prefer acting through a tool over talking; keep spoken replies brief.` -``` - -with: - -```ts -2. **create_app** — generate and run a Lua app on the SIMULATED M5StickC DEVICE shown on the page (a small 240×135 screen with two buttons). Use when the user asks for something to happen "on the device", "on the m5stick", "on the screen", asks for a clock, a counter, a game, a bouncing ball, anything interactive. Returns asynchronously — the coding agent writes Lua and pushes it; the user sees status in the UI. - -3. **push_app** — push the app CURRENTLY SHOWN IN THE SIMULATOR to the user's PHYSICAL device. Use when the user says to push / send / deploy / load the app onto the device, the stick, or the hardware (e.g. "ok push app", "send it to my stick"). No arguments — it sends whatever the simulator is showing. - -For ambiguous requests like "show stripes", default to apply_css (the page) unless the user mentioned the device. Prefer acting through a tool over talking; keep spoken replies brief.` -``` - -- [ ] **Step 3: Add the push_app tool to the session** - -In `voice-agent.ts`, replace: - -```ts - required: ["description"], - }, - }, - ], - tool_choice: "auto", -``` - -with: - -```ts - required: ["description"], - }, - }, - { - type: "function", - name: "push_app", - description: - "Push the app currently shown in the simulator to the PHYSICAL device over its WebSocket. Use when the user says to push / send / deploy / load the app onto the device / stick / hardware. No arguments.", - parameters: { type: "object", properties: {} }, - }, - ], - tool_choice: "auto", -``` - -- [ ] **Step 4: Route the tool call** - -In `voice-agent.ts`, replace: - -```ts - if (name === "apply_css") { - this.handleApplyCss(callId, argsJson) - return - } - if (name !== "create_app") { -``` - -with: - -```ts - if (name === "apply_css") { - this.handleApplyCss(callId, argsJson) - return - } - if (name === "push_app") { - this.handlePushApp(callId) - return - } - if (name !== "create_app") { -``` - -- [ ] **Step 5: Implement handlePushApp** - -In `voice-agent.ts`, replace: - -```ts - private sendToolResult(callId: string, payload: unknown): void { -``` - -with: - -```ts - private handlePushApp(callId: string): void { - // "The app in the sim" is currentApp (set by create_app), or the default - // bouncing ball the viewer renders when nothing has been generated yet. - const code = this.currentApp?.code ?? DEFAULT_APP - const frame = JSON.stringify({ type: "app", code }) - const devices = Array.from(this.getConnections("device")) - for (const d of devices) d.send(frame) - console.log("[voice] push_app ->", devices.length, "device(s),", code.length, "chars") - this.sendToolResult( - callId, - devices.length > 0 - ? { ok: true, devices: devices.length } - : { ok: false, error: "no device connected" }, - ) - if (this.openai && this.openaiReady) { - this.openai.send(JSON.stringify({ type: "response.create" })) - } - } - - private sendToolResult(callId: string, payload: unknown): void { -``` - -- [ ] **Step 6: Typecheck, build, test** - -Run: `cd examples/m5stick-voice/server && npm run typecheck && npm run build && npm test` -Expected: `tsc` clean; vite build succeeds (client + ssr); Vitest `5 passed`. - -- [ ] **Step 7: Commit the server change** - -```bash -cd /Users/matt/code/resident -git add examples/m5stick-voice/server/src/agents/voice-agent.ts -git commit -m "feat(m5stick-voice): push_app realtime tool sends the sim's app to the device - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - ---- - -### Task 5: Version bump + changelog - -**Files:** -- Modify: `library.json` -- Modify: `idf_component.yml` -- Modify: `docs/changelog.md` - -- [ ] **Step 1: Bump library.json** - -In `library.json`, replace: - -```json - "version": "0.5.0", -``` - -with: - -```json - "version": "0.5.1-dev", -``` - -- [ ] **Step 2: Bump idf_component.yml** - -In `idf_component.yml`, replace: - -```yaml -version: "0.5.0" -``` - -with: - -```yaml -version: "0.5.1-dev" -``` - -- [ ] **Step 3: Add the changelog section** - -In `docs/changelog.md`, replace: - -```markdown -# Changelog - -## v0.5.0 - -First public alpha. -``` - -with: - -```markdown -# Changelog - -## v0.5.1-dev () - -### New features - -- `Resident::Sandbox::suspendApp()` / `resumeApp()` / `isAppSuspended()` — pause - and resume a running app's tick without unloading it. While suspended, - `loop()` skips the Lua `on_tick`/event dispatch (Courier and extension updates - keep running) and the status display is freed for direct text via - `StatusDisplay::displayText()`. The m5stick-voice example uses it to show - "Listening" over a running app during push-to-talk. - -## v0.5.0 - -First public alpha. -``` - -> `` is filled in Task 8 (after the firmware commit exists). Do NOT commit yet. - ---- - -### Task 6: Full pre-hardware test sweep - -- [ ] **Step 1: Run the firmware test suite** - -Run: `cd /Users/matt/code/resident && ./tools/run-tests.py unit static-analysis build` -Expected: unit tests pass; cppcheck clean; all example PlatformIO envs build `[SUCCESS]`. - -If cppcheck flags the new methods, address it before proceeding (the additions are simple enough that none is expected). - ---- - -### Task 7: Hardware verification checkpoint (Matt) - -> This is a manual gate. Do not proceed to commit (Task 8) until Matt confirms. - -- [ ] **Step 1: Flash the device** - -Run: `cd examples/m5stick-voice/device && pio run -e m5stick -t upload -t monitor` - -- [ ] **Step 2: End-to-end check** — confirm each: - - Open `https:///devices//`, ask the agent to make an app (e.g. "make a bouncing ball on the device"); it appears in the sim. - - Hold the device button, say "ok push app", release. The app appears **on the stick**. - - With the app running on the stick, **hold to talk**: the app freezes and "Listening" shows. - - **Release**: the app reappears (restored frame) and resumes animating. - - Repeat with a *static* app (e.g. a clock or text) to confirm `repaint()` restores it on release. - - With **no** app generated yet, "ok push app" pushes the default bouncing ball. - -- [ ] **Step 2: Get Matt's explicit go-ahead** before committing. - ---- - -### Task 8: Commit the firmware/library changes - -> Only after Task 7 passes. - -- [ ] **Step 1: Stage and commit the firmware bundle** - -```bash -cd /Users/matt/code/resident -git add src/ResidentSandbox.h src/ResidentSandbox.cpp \ - examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.h \ - examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.cpp \ - examples/m5stick-voice/device/src/main.cpp \ - library.json idf_component.yml docs/changelog.md -git commit -m "feat(resident): Sandbox suspendApp/resumeApp; m5stick-voice suspends app while listening - -Adds Resident::Sandbox::suspendApp()/resumeApp()/isAppSuspended() to pause a -running app's tick without unloading it, and DisplayDriver::repaint() to restore -the last frame. m5stick-voice suspends the running app and shows \"Listening\" -during push-to-talk, resuming on release. Bumps library to 0.5.1-dev. - -Co-Authored-By: Claude Opus 4.8 (1M context) " -``` - -- [ ] **Step 2: Fill the changelog git-hash and amend** - -```bash -HASH=$(git rev-parse --short HEAD) -# Replace the literal placeholder in the new changelog heading: -sed -i '' "s/## v0.5.1-dev ()/## v0.5.1-dev ($HASH)/" docs/changelog.md -git add docs/changelog.md -git commit --amend --no-edit -``` - -- [ ] **Step 3: Confirm clean tree** - -Run: `git status --short` -Expected: no tracked changes remain from this feature (the `.claude/superpowers/` spec + plan are git-ignored and won't appear). - ---- - -## Self-review - -- **Spec coverage:** Component 1 → Task 1; Component 2 → Task 2; Component 3 → Task 3; Component 4 → Task 4; Component 5 → Task 5. Verification → Tasks 6–7. ✅ -- **No-app-yet** (push `DEFAULT_APP`) → Task 4 Step 5. **Resume-in-place** (no `init()` rerun) → suspend/resume only toggle a flag (Task 1). **repaint()** → Tasks 2/3. ✅ -- **Type/name consistency:** `suspendApp`/`resumeApp`/`isAppSuspended`/`_appSuspended` used identically across Tasks 1 and 3; `repaint()` declared (Task 2) and called (Task 3); `handlePushApp`/`DEFAULT_APP`/`getConnections("device")` consistent within Task 4. ✅ -- **No placeholders:** every code step shows exact text; the only literal placeholder is the changelog ``, intentionally resolved in Task 8 Step 2. ✅ diff --git a/.claude/superpowers/plans/2026-06-10-coding-status-updates.md b/.claude/superpowers/plans/2026-06-10-coding-status-updates.md deleted file mode 100644 index dd4cd81..0000000 --- a/.claude/superpowers/plans/2026-06-10-coding-status-updates.md +++ /dev/null @@ -1,1062 +0,0 @@ -# Coding Status Updates Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Stream high-resolution coding-job status (`working` with live line count → `validating` → `done`) from the `VoiceAgent` Durable Object to both browser monitors and connected physical devices. - -**Architecture:** Switch the codegen call from a buffered fetch to a streamed SSE read, counting lines as Lua arrives and emitting throttled `working` updates. Merge the old `agent_status` enum into one model (`idle | working | validating | done`), broadcast every status to monitor **and** device connections, and surface `done` as a transient toast on the client. - -**Tech Stack:** TypeScript, Cloudflare Workers / Durable Objects (`agents` SDK), React 19, TanStack Start/Router, Vitest. All work is under `examples/m5stick-voice/server/`. - ---- - -## Working directory - -All paths below are relative to `examples/m5stick-voice/server/`. Run all commands from there: - -```bash -cd examples/m5stick-voice/server -``` - -## File structure - -- **Create** `src/lib/codegen-stream.ts` — pure helpers for codegen streaming: SSE line parsing, line counting, and the time-throttled progress emitter. One responsibility: turn a stream of SSE text into throttled line-count callbacks. Unit-tested. -- **Create** `src/lib/codegen-stream.test.ts` — Vitest tests for the above. -- **Create** `src/components/DoneToast.tsx` — dismissable toast showing job success/error. -- **Modify** `src/agents/voice-agent.ts` — streaming `callCodegenChat`, broadcast `setAgentStatus`, reworked `runCodingJob` + `finishJob`, new `AgentStatus` type, snapshot shape. -- **Modify** `src/hooks/useVoiceMonitor.ts` — consume the merged `agent_status`, expose `workingLines` / `retryCount` / `lastDone` / `dismissDone`. -- **Modify** `src/components/StatusPill.tsx` — new enum, render line count + retry. -- **Modify** `src/components/Header.tsx` — pass line count / retry to the pill, drop the inline done/error message block (replaced by the toast). -- **Modify** `src/routes/devices.$deviceId.tsx` — wire the new hook fields and render `DoneToast`. - -The firmware that consumes `{type:"agent_status",...}` on the device side is owned separately and is **not** part of this plan — only the wire contract is. - ---- - -## Task 1: SSE line parser - -**Files:** -- Create: `src/lib/codegen-stream.ts` -- Test: `src/lib/codegen-stream.test.ts` - -- [ ] **Step 1: Write the failing test** - -Create `src/lib/codegen-stream.test.ts`: - -```ts -import { describe, expect, it } from "vitest" -import { parseSSELine } from "./codegen-stream" - -describe("parseSSELine", () => { - it("extracts delta content from a data line", () => { - const line = `data: ${JSON.stringify({ choices: [{ delta: { content: "hi" } }] })}` - expect(parseSSELine(line)).toEqual({ content: "hi" }) - }) - - it("flags the [DONE] sentinel", () => { - expect(parseSSELine("data: [DONE]")).toEqual({ done: true }) - }) - - it("ignores blank lines and non-data lines", () => { - expect(parseSSELine("")).toEqual({}) - expect(parseSSELine(": keep-alive")).toEqual({}) - }) - - it("ignores data lines with no content delta (e.g. role-only chunk)", () => { - const line = `data: ${JSON.stringify({ choices: [{ delta: { role: "assistant" } }] })}` - expect(parseSSELine(line)).toEqual({}) - }) - - it("returns empty on malformed JSON rather than throwing", () => { - expect(parseSSELine("data: {not json")).toEqual({}) - }) -}) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `npm test -- codegen-stream` -Expected: FAIL — `parseSSELine` is not exported / module not found. - -- [ ] **Step 3: Write minimal implementation** - -Create `src/lib/codegen-stream.ts`: - -```ts -/** Parse one line of an OpenAI chat-completions SSE stream. */ -export function parseSSELine(line: string): { content?: string; done?: boolean } { - const trimmed = line.trim() - if (!trimmed.startsWith("data:")) return {} - const payload = trimmed.slice(5).trim() - if (payload === "[DONE]") return { done: true } - try { - const json = JSON.parse(payload) as { - choices?: Array<{ delta?: { content?: unknown } }> - } - const content = json.choices?.[0]?.delta?.content - return typeof content === "string" ? { content } : {} - } catch { - return {} - } -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `npm test -- codegen-stream` -Expected: PASS (5 tests). - -- [ ] **Step 5: Commit** - -```bash -git add src/lib/codegen-stream.ts src/lib/codegen-stream.test.ts -git commit -m "feat(m5stick-voice): SSE line parser for streamed codegen" -``` - ---- - -## Task 2: Line-counting throttled progress emitter - -**Files:** -- Modify: `src/lib/codegen-stream.ts` -- Test: `src/lib/codegen-stream.test.ts` - -- [ ] **Step 1: Write the failing test** - -Append to `src/lib/codegen-stream.test.ts`: - -```ts -import { countLines, createLineProgress } from "./codegen-stream" - -describe("countLines", () => { - it("counts completed (newline-terminated) lines", () => { - expect(countLines("")).toBe(0) - expect(countLines("one line, no newline")).toBe(0) - expect(countLines("a\nb\n")).toBe(2) - expect(countLines("a\nb")).toBe(1) - }) -}) - -describe("createLineProgress", () => { - it("emits only when the line count changes and the interval has elapsed", () => { - let clock = 0 - const emitted: number[] = [] - const p = createLineProgress((n) => emitted.push(n), { intervalMs: 250, now: () => clock }) - - p.update("a\n") // t=0: first change, 0>=250? no -> suppressed - clock = 100 - p.update("a\nb\n") // t=100: still < 250 -> suppressed - clock = 300 - p.update("a\nb\nc\n") // t=300: >=250 and changed -> emit 3 - clock = 350 - p.update("a\nb\nc\n") // no change -> nothing - expect(emitted).toEqual([3]) - }) - - it("flush emits the final count if it was never emitted", () => { - let clock = 0 - const emitted: number[] = [] - const p = createLineProgress((n) => emitted.push(n), { intervalMs: 250, now: () => clock }) - p.update("a\nb\n") // suppressed (t=0) - p.flush() // emits 2 - expect(emitted).toEqual([2]) - }) - - it("flush is a no-op when the latest count was already emitted", () => { - let clock = 1000 - const emitted: number[] = [] - const p = createLineProgress((n) => emitted.push(n), { intervalMs: 250, now: () => clock }) - p.update("a\nb\n") // t=1000, elapsed since 0 -> emit 2 - p.flush() // already emitted 2 -> no-op - expect(emitted).toEqual([2]) - }) -}) -``` - -> Note: the first `update` at `t=0` is suppressed because `0 - 0 >= 250` is false; the job separately emits `working({lines:0})` at start (Task 5), so the initial state is still shown. - -- [ ] **Step 2: Run test to verify it fails** - -Run: `npm test -- codegen-stream` -Expected: FAIL — `countLines` / `createLineProgress` not exported. - -- [ ] **Step 3: Write minimal implementation** - -Append to `src/lib/codegen-stream.ts`: - -```ts -/** Number of completed (newline-terminated) lines in `text`. */ -export function countLines(text: string): number { - let n = 0 - for (let i = 0; i < text.length; i++) if (text[i] === "\n") n++ - return n -} - -export interface LineProgress { - /** Feed the full accumulated text so far. */ - update(accumulated: string): void - /** Emit the final count if it has not been emitted yet. */ - flush(): void -} - -/** - * Emits the completed-line count of streamed text, throttled to at most one - * emission per `intervalMs`, only when the count changes. `now` is injectable - * for tests; defaults to `Date.now`. - */ -export function createLineProgress( - emit: (lines: number) => void, - opts: { intervalMs?: number; now?: () => number } = {}, -): LineProgress { - const intervalMs = opts.intervalMs ?? 250 - const now = opts.now ?? Date.now - let lastEmitAt = 0 - let emittedLines = -1 - let latestLines = 0 - return { - update(accumulated) { - latestLines = countLines(accumulated) - const t = now() - if (latestLines !== emittedLines && t - lastEmitAt >= intervalMs) { - emittedLines = latestLines - lastEmitAt = t - emit(latestLines) - } - }, - flush() { - if (latestLines !== emittedLines) { - emittedLines = latestLines - emit(latestLines) - } - }, - } -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `npm test -- codegen-stream` -Expected: PASS (all `codegen-stream` tests, including Task 1's). - -- [ ] **Step 5: Commit** - -```bash -git add src/lib/codegen-stream.ts src/lib/codegen-stream.test.ts -git commit -m "feat(m5stick-voice): throttled line-count progress emitter" -``` - ---- - -## Task 3: Stream the codegen call - -**Files:** -- Modify: `src/agents/voice-agent.ts` (imports; `callCodegenChat` at `voice-agent.ts:429-461`) - -This task has no unit test (it drives a network fetch inside the DO); it is covered by `npm run typecheck` here and the hardware verification in Task 10. - -- [ ] **Step 1: Add the import** - -At the top of `src/agents/voice-agent.ts`, after the existing `validateLuaCode` import, add: - -```ts -import { parseSSELine, createLineProgress } from "../lib/codegen-stream" -``` - -- [ ] **Step 2: Replace `callCodegenChat` with a streaming version** - -Replace the entire `callCodegenChat` method with: - -```ts - private async callCodegenChat( - description: string, - followups: { role: "assistant" | "user"; content: string }[], - signal: AbortSignal, - onProgress: (lines: number) => void, - ): Promise { - const key = this.env.OPENAI_API_KEY - if (!key) throw new Error("OPENAI_API_KEY not set") - - const messages = [ - { role: "system", content: CODEGEN_SYSTEM }, - { role: "user", content: description }, - ...followups, - ] - - const resp = await fetch("https://api.openai.com/v1/chat/completions", { - method: "POST", - headers: { - Authorization: `Bearer ${key}`, - "Content-Type": "application/json", - }, - body: JSON.stringify({ model: CODEGEN_MODEL, messages, stream: true }), - signal, - }) - if (!resp.ok) { - const body = await resp.text().catch(() => "") - throw new Error(`OpenAI ${resp.status}: ${body.slice(0, 400)}`) - } - if (!resp.body) throw new Error("OpenAI returned no response stream") - - const progress = createLineProgress(onProgress) - const reader = resp.body.getReader() - const decoder = new TextDecoder() - let buffer = "" - let content = "" - - try { - for (;;) { - const { done, value } = await reader.read() - if (done) break - buffer += decoder.decode(value, { stream: true }) - let nl: number - while ((nl = buffer.indexOf("\n")) !== -1) { - const line = buffer.slice(0, nl) - buffer = buffer.slice(nl + 1) - const parsed = parseSSELine(line) - if (parsed.done) { buffer = ""; break } - if (parsed.content) { - content += parsed.content - progress.update(content) - } - } - } - } finally { - reader.releaseLock() - } - progress.flush() - - // The line count above includes any markdown fence lines; the displayed app - // strips them here. Progress is an indicator, so the small discrepancy is fine. - return content.replace(/^```(?:lua)?\s*/i, "").replace(/```\s*$/i, "").trim() - } -``` - -- [ ] **Step 3: Verify it still type-checks (callers updated in Task 5)** - -Run: `npm run typecheck` -Expected: ONE error only — `runCodingJob` calls `callCodegenChat` without the new `onProgress` argument. That is fixed in Task 5. Do not commit yet; continue to Task 4. - -> If you prefer a green checkpoint, you may do Tasks 4 and 5 before running `typecheck`/committing. Tasks 3–5 form one logically atomic server change. - ---- - -## Task 4: Broadcast status to monitors and devices - -**Files:** -- Modify: `src/agents/voice-agent.ts` (`AgentStatus` type `:51`; instance fields `:69-71`; `onConnect` snapshot `:82-90`; `setAgentStatus` `:483-487`) - -- [ ] **Step 1: Widen the `AgentStatus` type** - -Replace: - -```ts -type AgentStatus = "idle" | "working" | "done" | "error" -``` - -with: - -```ts -type AgentStatus = "idle" | "working" | "validating" | "done" -``` - -- [ ] **Step 2: Add a persisted line-count field** - -In the class, find: - -```ts - // M2 codegen state. - private agentStatus: AgentStatus = "idle" - private agentMessage?: string -``` - -and replace with: - -```ts - // M2 codegen state. - private agentStatus: AgentStatus = "idle" - private agentLines = 0 -``` - -(`agentMessage` is removed — error text now travels only with the transient -`done` event, which is never persisted.) - -- [ ] **Step 3: Update the snapshot payload** - -In `onConnect`, replace the snapshot send: - -```ts - connection.send(JSON.stringify({ - type: "snapshot", - agent_status: this.agentStatus, - message: this.agentMessage, - app: this.currentApp, - css: this.currentCss, - })) -``` - -with: - -```ts - connection.send(JSON.stringify({ - type: "snapshot", - agent_status: this.agentStatus, // idle | working | validating (never done) - lines: this.agentLines, - app: this.currentApp, - css: this.currentCss, - })) -``` - -- [ ] **Step 4: Replace `setAgentStatus` with the broadcaster** - -Replace the entire `setAgentStatus` method: - -```ts - private setAgentStatus(state: AgentStatus, message: string | undefined): void { - this.agentStatus = state - this.agentMessage = message - this.toMonitors({ type: "agent_status", state, message }) - } -``` - -with: - -```ts - private setAgentStatus( - state: AgentStatus, - extra: { lines?: number; success?: boolean; message?: string } = {}, - ): void { - // `done` is a transient event (the toast); the caller follows it with `idle`. - // Only persist resting states so a refreshed tab's snapshot is accurate. - if (state !== "done") { - this.agentStatus = state - this.agentLines = extra.lines ?? 0 - } - this.broadcastAgentStatus({ type: "agent_status", state, ...extra }) - } - - /** Send a JSON status frame to every monitor AND every device connection. - * Named to avoid colliding with the base class's `broadcastStatus()`. */ - private broadcastAgentStatus(obj: unknown): void { - const s = JSON.stringify(obj) - for (const m of this.getConnections("monitor")) m.send(s) - for (const d of this.getConnections("device")) d.send(s) - } -``` - -(Type-check is deferred to Task 5, where the remaining callers are updated.) - ---- - -## Task 5: Rework the job lifecycle emissions - -**Files:** -- Modify: `src/agents/voice-agent.ts` (`handleFunctionCall` `:317`; `runCodingJob` `:379-427`; `finishJob` `:463-481`) - -- [ ] **Step 1: Update the "started" emission in `handleFunctionCall`** - -Find: - -```ts - // Tell the viewer. - this.setAgentStatus("working", undefined) -``` - -Replace with: - -```ts - // Tell the viewer + device: started (zero lines so far). - this.setAgentStatus("working", { lines: 0 }) -``` - -- [ ] **Step 2: Rewrite `runCodingJob` as an attempt loop** - -Replace the entire `runCodingJob` method with: - -```ts - private async runCodingJob( - jobId: string, - description: string, - signal: AbortSignal, - ): Promise { - try { - const followups: { role: "assistant" | "user"; content: string }[] = [] - let code = "" - let validation: ValidationResult = { ok: false } - - for (let attempt = 1; attempt <= 2; attempt++) { - code = await this.callCodegenChat( - description, - followups, - signal, - (lines) => this.setAgentStatus("working", { lines }), - ) - if (signal.aborted) return - - this.setAgentStatus("validating", {}) - validation = await validateLuaCode(code) - if (signal.aborted) return - if (validation.ok) break - - console.warn(`[voice] codegen v${attempt} validation failed:`, validation.error) - followups.push( - { role: "assistant", content: code }, - { - role: "user", - content: `That code failed validation with: ${validation.error}. Fix it. Return only Lua, no commentary.`, - }, - ) - } - - if (!validation.ok) { - this.finishJob(jobId, false, validation.error ?? "validation failed") - return - } - - this.appVersion += 1 - this.currentApp = { code, version: this.appVersion } - - // With a monitor present, push into the simulator (the user pushes to - // hardware separately via push_app). With no monitor, send straight to - // any connected physical device. - const monitors = Array.from(this.getConnections("monitor")).length - if (monitors > 0) { - this.toMonitors({ type: "app", code, version: this.appVersion }) - } else { - const devices = this.pushAppToDevices(code) - console.log("[voice] no monitor — pushed app ->", devices, "device(s)") - } - this.finishJob(jobId, true, undefined) - } catch (err) { - if (signal.aborted) return - const msg = err instanceof Error ? err.message : String(err) - this.finishJob(jobId, false, msg) - } - } -``` - -- [ ] **Step 3: Import the `ValidationResult` type** - -Update the validator import so the type used above is in scope. Change: - -```ts -import { validateLuaCode } from "../lib/lua-validator" -``` - -to: - -```ts -import { validateLuaCode, type ValidationResult } from "../lib/lua-validator" -``` - -- [ ] **Step 4: Rewrite `finishJob` to emit `done` then `idle`** - -Replace the entire `finishJob` method: - -```ts - private finishJob(jobId: string, state: "done" | "error", message: string | undefined): void { - this.setAgentStatus(state, message) - if (this.openai && this.openaiReady) { - const content = state === "done" - ? `[create_app jobId=${jobId}] completed successfully` - : `[create_app jobId=${jobId}] failed: ${message ?? "unknown error"}` - this.openai.send(JSON.stringify({ - type: "conversation.item.create", - item: { - type: "message", - role: "user", - content: [{ type: "input_text", text: `[system] ${content}` }], - }, - })) - } - } -``` - -with: - -```ts - private finishJob(jobId: string, success: boolean, message: string | undefined): void { - // Transient terminal event (drives the client toast), then back to idle. - this.setAgentStatus("done", { success, message }) - this.setAgentStatus("idle", {}) - - if (this.openai && this.openaiReady) { - const content = success - ? `[create_app jobId=${jobId}] completed successfully` - : `[create_app jobId=${jobId}] failed: ${message ?? "unknown error"}` - this.openai.send(JSON.stringify({ - type: "conversation.item.create", - item: { - type: "message", - role: "user", - content: [{ type: "input_text", text: `[system] ${content}` }], - }, - })) - } - } -``` - -- [ ] **Step 5: Type-check the whole server change (Tasks 3–5)** - -Run: `npm run typecheck` -Expected: PASS (no errors). - -- [ ] **Step 6: Run the unit tests** - -Run: `npm test` -Expected: PASS — `codegen-stream` and `lua-validator` suites all green. - -- [ ] **Step 7: Commit** - -```bash -git add src/agents/voice-agent.ts -git commit -m "feat(m5stick-voice): stream codegen status to monitors and devices" -``` - ---- - -## Task 6: StatusPill — new enum, line count, retry - -**Files:** -- Modify: `src/components/StatusPill.tsx` - -- [ ] **Step 1: Replace the component** - -Replace the entire contents of `src/components/StatusPill.tsx` with: - -```tsx -export type AgentStatus = "idle" | "working" | "validating" | "done" - -interface Props { - status: AgentStatus - lines?: number - retryCount?: number -} - -/** Coarse status pill. `done` is shown as a toast, not here, but kept in the - * style map for type completeness. */ -export function StatusPill({ status, lines = 0, retryCount = 0 }: Props) { - const styles: Record = { - idle: { dot: "#666", label: "idle" }, - working: { dot: "#e0c542", label: lines > 0 ? `working · ${lines} lines` : "working" }, - validating: { dot: "#54a0e0", label: "validating" }, - done: { dot: "#3fd07d", label: "done" }, - } - const s = styles[status] - const active = status === "working" || status === "validating" - const label = retryCount > 0 ? `${s.label} · retry ${retryCount}` : s.label - return ( - - - {label} - - - ) -} -``` - -- [ ] **Step 2: Commit** - -```bash -git add src/components/StatusPill.tsx -git commit -m "feat(m5stick-voice): status pill shows line count, validating, retry" -``` - -> The repo has no React component test harness; correctness here is covered by `npm run typecheck` (Task 9) and the hardware/browser check (Task 10). - ---- - -## Task 7: DoneToast component - -**Files:** -- Create: `src/components/DoneToast.tsx` - -- [ ] **Step 1: Create the component** - -Create `src/components/DoneToast.tsx`: - -```tsx -interface Props { - success: boolean - message?: string - onDismiss: () => void -} - -/** Dismissable bottom-centre toast shown when a coding job finishes. */ -export function DoneToast({ success, message, onDismiss }: Props) { - return ( -
- - {success ? "App ready" : `error: ${message ?? "unknown error"}`} - - -
- ) -} -``` - -- [ ] **Step 2: Commit** - -```bash -git add src/components/DoneToast.tsx -git commit -m "feat(m5stick-voice): dismissable done/error toast component" -``` - ---- - -## Task 8: useVoiceMonitor — consume merged status - -**Files:** -- Modify: `src/hooks/useVoiceMonitor.ts` - -- [ ] **Step 1: Update the exported interface and imports** - -At the top, the existing import already pulls `AgentStatus` from `../components/StatusPill` — keep it. Replace the `VoiceMonitor` interface with: - -```ts -export interface DoneEvent { - success: boolean - message?: string -} - -export interface VoiceMonitor { - status: string - transcript: TranscriptItem[] - agentStatus: AgentStatus // idle | working | validating - workingLines: number - retryCount: number - lastDone: DoneEvent | null - dismissDone: () => void - currentApp?: CurrentApp - css: string - setFrameHandler: (cb: ((buf: ArrayBuffer) => void) | null) => void -} -``` - -- [ ] **Step 2: Replace the status state hooks** - -Find: - -```ts - const [agentStatus, setAgentStatus] = useState("idle") - const [agentMessage, setAgentMessage] = useState(undefined) -``` - -Replace with: - -```ts - const [agentStatus, setAgentStatus] = useState("idle") - const [workingLines, setWorkingLines] = useState(0) - const [retryCount, setRetryCount] = useState(0) - const [lastDone, setLastDone] = useState(null) - const prevStatusRef = useRef("idle") -``` - -- [ ] **Step 3: Add the `dismissDone` callback** - -Just below `setFrameHandler` definition, add: - -```ts - const dismissDone: VoiceMonitor["dismissDone"] = () => setLastDone(null) -``` - -- [ ] **Step 4: Replace the `agent_status` case** - -Replace: - -```ts - case "agent_status": - if (isAgentStatus(m.state)) setAgentStatus(m.state) - setAgentMessage(typeof m.message === "string" ? m.message : undefined) - break -``` - -with: - -```ts - case "agent_status": { - if (!isAgentStatus(m.state)) break - const next = m.state - const prev = prevStatusRef.current - if (next === "working") { - // validating -> working means a retry; a fresh job starts from - // idle/done and resets the counter. - if (prev === "validating") setRetryCount((n) => n + 1) - else if (prev === "idle" || prev === "done") setRetryCount(0) - setWorkingLines(typeof m.lines === "number" ? m.lines : 0) - setAgentStatus("working") - } else if (next === "validating") { - setAgentStatus("validating") - } else if (next === "idle") { - setWorkingLines(0) - setAgentStatus("idle") - } else if (next === "done") { - setLastDone({ - success: m.success === true, - message: typeof m.message === "string" ? m.message : undefined, - }) - } - prevStatusRef.current = next - break - } -``` - -- [ ] **Step 5: Replace the `snapshot` case** - -Replace: - -```ts - case "snapshot": - if (isAgentStatus(m.agent_status)) setAgentStatus(m.agent_status) - if (typeof m.message === "string") setAgentMessage(m.message) - if (m.app && typeof m.app === "object") { - const a = m.app as { code?: unknown; version?: unknown } - if (typeof a.code === "string" && typeof a.version === "number") { - setCurrentApp({ code: a.code, version: a.version }) - } - } - if (typeof m.css === "string") setCss(m.css) - break -``` - -with: - -```ts - case "snapshot": - if (isAgentStatus(m.agent_status) && m.agent_status !== "done") { - setAgentStatus(m.agent_status) - prevStatusRef.current = m.agent_status - } - if (typeof m.lines === "number") setWorkingLines(m.lines) - if (m.app && typeof m.app === "object") { - const a = m.app as { code?: unknown; version?: unknown } - if (typeof a.code === "string" && typeof a.version === "number") { - setCurrentApp({ code: a.code, version: a.version }) - } - } - if (typeof m.css === "string") setCss(m.css) - break -``` - -- [ ] **Step 6: Update the `isAgentStatus` guard** - -Replace: - -```ts -function isAgentStatus(s: unknown): s is AgentStatus { - return s === "idle" || s === "working" || s === "done" || s === "error" -} -``` - -with: - -```ts -function isAgentStatus(s: unknown): s is AgentStatus { - return s === "idle" || s === "working" || s === "validating" || s === "done" -} -``` - -- [ ] **Step 7: Update the hook's return value** - -Replace: - -```ts - return { status, transcript, agentStatus, agentMessage, currentApp, css, setFrameHandler } -``` - -with: - -```ts - return { - status, transcript, agentStatus, workingLines, retryCount, - lastDone, dismissDone, currentApp, css, setFrameHandler, - } -``` - -- [ ] **Step 8: Commit** - -```bash -git add src/hooks/useVoiceMonitor.ts -git commit -m "feat(m5stick-voice): monitor hook tracks line count, retries, done toast" -``` - ---- - -## Task 9: Wire the route and header - -**Files:** -- Modify: `src/components/Header.tsx` -- Modify: `src/routes/devices.$deviceId.tsx` - -- [ ] **Step 1: Replace `Header.tsx`** - -Replace the entire contents of `src/components/Header.tsx` with: - -```tsx -import { StatusPill, type AgentStatus } from "./StatusPill" - -interface Props { - deviceId: string - status: string - agentStatus: AgentStatus - workingLines: number - retryCount: number -} - -export function Header({ deviceId, status, agentStatus, workingLines, retryCount }: Props) { - return ( -
-
-

- m5stick-voice · {deviceId} -

- -
-
{status}
-
- ) -} -``` - -- [ ] **Step 2: Update the route** - -In `src/routes/devices.$deviceId.tsx`, add the `DoneToast` import after the other component imports: - -```tsx -import { DoneToast } from "../components/DoneToast" -``` - -Replace the hook destructuring: - -```tsx - const { - status, transcript, agentStatus, agentMessage, currentApp, css, setFrameHandler, - } = useVoiceMonitor(deviceId) -``` - -with: - -```tsx - const { - status, transcript, agentStatus, workingLines, retryCount, - lastDone, dismissDone, currentApp, css, setFrameHandler, - } = useVoiceMonitor(deviceId) -``` - -Replace the `
` element: - -```tsx -
-``` - -with: - -```tsx -
-``` - -Then add the toast just before the closing ``, after ``: - -```tsx - {lastDone && ( - - )} -``` - -- [ ] **Step 3: Catch any stragglers referencing the removed fields** - -Run: `grep -rn "agentMessage\|\"error\"\|'error'" src/` -Expected: no remaining references in `components/`, `hooks/`, or `routes/` tied to the old status model. (`onOpenAIEvent` / `console.error` strings in `voice-agent.ts` are unrelated — leave them.) - -- [ ] **Step 4: Type-check and build** - -Run: `npm run typecheck && npm run build` -Expected: both PASS. - -- [ ] **Step 5: Commit** - -```bash -git add src/components/Header.tsx src/routes/devices.\$deviceId.tsx -git commit -m "feat(m5stick-voice): render line count in header and done toast in viewer" -``` - ---- - -## Task 10: Full verification - -**Files:** none (verification only) - -- [ ] **Step 1: Run the whole server check suite** - -Run: `npm run typecheck && npm test && npm run build` -Expected: all PASS. - -- [ ] **Step 2: Browser smoke test (local dev)** - -Run: `npm run dev`, open the viewer for a device id, and trigger a `create_app` -(via the voice path or by however the project drives it locally). Confirm: -- the pill goes `working` → shows a climbing `working · N lines` → `validating` -- on success a green "App ready" toast appears and is dismissable; the pill returns to `idle` -- forcing a validation failure shows `working · retry 1` after the first `validating`, and a red error toast on final failure -- refreshing the tab mid-job restores `working`/`validating` (never a stale toast) - -- [ ] **Step 3: Hardware verification (REQUIRED before any device-facing claim)** - -Per repo policy, firmware/device behaviour is verified on real hardware before -the work is called done. With a physical M5Stick connected, run a `create_app` -and confirm the device receives the `{type:"agent_status",...}` frames alongside -the running app, and that the `WORKING_APP` placeholder still appears. Pause here -for Matt's on-board test before treating the device path as complete. - -- [ ] **Step 4: Final confirmation** - -Report the exact command output from Step 1 and the observations from Steps 2–3. -Do not claim completion without this evidence. - ---- - -## Self-review notes - -- **Spec coverage:** unified `idle|working|validating|done` model (Tasks 4, 6, 8); per-line time-throttled progress (Tasks 2, 3); broadcast to monitors + devices (Task 4); retry inferred client-side from `validating→working` (Task 8); `done` as transient toast then `idle` (Tasks 5, 7, 8); `WORKING_APP` push unchanged (untouched in Task 5); firmware contract documented, not implemented (file-structure note + Task 10). `lastLine` intentionally omitted. -- **Type consistency:** `AgentStatus = "idle"|"working"|"validating"|"done"` is defined identically in `voice-agent.ts` (Task 4) and `StatusPill.tsx` (Task 6) and imported by the hook/header. `setAgentStatus(state, { lines?, success?, message? })`, `finishJob(jobId, success: boolean, message?)`, and `callCodegenChat(..., onProgress)` signatures match every call site. `createLineProgress`, `parseSSELine`, `countLines`, `ValidationResult` all referenced as defined. -- **Placeholders:** none — every code step is complete. -``` diff --git a/.claude/superpowers/plans/2026-06-11-device-agent-status-display.md b/.claude/superpowers/plans/2026-06-11-device-agent-status-display.md deleted file mode 100644 index 7e715e4..0000000 --- a/.claude/superpowers/plans/2026-06-11-device-agent-status-display.md +++ /dev/null @@ -1,305 +0,0 @@ -# Device Agent-Status Display Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Have the physical M5Stick render coding-job status directly from the `agent_status` WebSocket frames, replacing the server-pushed `WORKING_APP` Lua placeholder. - -**Architecture:** Delete the server's `WORKING_APP` device push (the `agent_status` stream already reaches devices). On the device, register `sandbox.onMessage` to render generic status text via `displayText` — suspending a running app on the first `working` frame so the text isn't gated off — and add a tap-to-recover on error using `getTotalPressCount()` (only taps bump it, not push-to-talk holds). - -**Tech Stack:** TypeScript / Cloudflare Worker (server); C++ / PlatformIO / Arduino-ESP32, ArduinoJson, M5Unified (device firmware). Spec: `.claude/superpowers/specs/2026-06-10-device-agent-status-display-design.md`. - ---- - -## File structure - -- **Modify** `examples/m5stick-voice/server/src/agents/voice-agent.ts` — remove the `WORKING_APP` push in `handleFunctionCall` and the `WORKING_APP` import. -- **Modify** `examples/m5stick-voice/server/src/lib/default-app.ts` — delete the now-unused `WORKING_APP` export (keep `DEFAULT_APP`). -- **Modify** `examples/m5stick-voice/device/src/main.cpp` — add `agent_status` handling (state + `onMessage` + tap-recover in `loop()`). - -The firmware has no unit-test harness (Arduino/ESP32 C++); the device task is verified by compiling both PlatformIO envs and by an on-hardware check. Per repo policy, **the device commit is gated on a real-hardware test** — it is not committed on a compile pass alone. - ---- - -## Task 1: Server — drop the `WORKING_APP` placeholder push - -**Files:** -- Modify: `examples/m5stick-voice/server/src/agents/voice-agent.ts` (import line 6; `handleFunctionCall` lines 320-323) -- Modify: `examples/m5stick-voice/server/src/lib/default-app.ts:31-44` - -This is a deletion; no new test. It is verified by `typecheck` / existing `test` / `build` staying green. Run all commands from `examples/m5stick-voice/server`. - -- [ ] **Step 1: Remove the device placeholder push** - -In `src/agents/voice-agent.ts`, delete the push block in `handleFunctionCall`. Change: - -```ts - // Tell the viewer + device: started (zero lines so far). - this.setAgentStatus("working", { lines: 0 }) - - // Immediately show a "Working..." placeholder on the physical device while - // the coding agent generates the real app — regardless of any monitor. - const working = this.pushAppToDevices(WORKING_APP) - if (working > 0) console.log("[voice] pushed Working... ->", working, "device(s)") - - // Fire-and-forget. - this.ctx.waitUntil(this.runCodingJob(jobId, parsed.description, this.codingAbort.signal)) -``` - -to: - -```ts - // Tell the viewer + device: started (zero lines so far). The device renders - // its own status from this agent_status stream — no placeholder app push. - this.setAgentStatus("working", { lines: 0 }) - - // Fire-and-forget. - this.ctx.waitUntil(this.runCodingJob(jobId, parsed.description, this.codingAbort.signal)) -``` - -- [ ] **Step 2: Drop the now-unused `WORKING_APP` import** - -In `src/agents/voice-agent.ts`, change the import (line 6): - -```ts -import { DEFAULT_APP, WORKING_APP } from "../lib/default-app" -``` - -to: - -```ts -import { DEFAULT_APP } from "../lib/default-app" -``` - -- [ ] **Step 3: Delete the `WORKING_APP` export** - -In `src/lib/default-app.ts`, delete the entire `WORKING_APP` block (the trailing export), i.e. remove: - -```ts - -/** - * Placeholder shown on the device while the coding agent is generating an app. - * Screen is small (240×135) so keep it to one short line. - */ -export const WORKING_APP = ` -function init(ctx) - screen.clear() - screen.text(58, 58, "Working...", 2, 255, 255, 255) - screen.flip() -end - -function on_tick(ctx, dt_ms) -end -` -``` - -Leave `DEFAULT_APP` (and its doc comment) intact as the end of the file. - -- [ ] **Step 4: Verify no stray references remain** - -Run: `grep -rn "WORKING_APP" src/` -Expected: no matches. - -- [ ] **Step 5: Typecheck, test, build** - -Run: `npm run typecheck && npm test && npm run build` -Expected: typecheck clean; `Tests 15 passed (15)`; build succeeds. - -- [ ] **Step 6: Commit** - -```bash -git add src/agents/voice-agent.ts src/lib/default-app.ts -git commit -m "feat(m5stick-voice): device renders status from agent_status, drop WORKING_APP push" -``` - -End the commit body with: -``` -Co-Authored-By: Claude Opus 4.8 (1M context) -``` - -Stay on branch `spike-voice`. Do not branch or merge. - ---- - -## Task 2: Device — render `agent_status` + tap-to-recover - -**Files:** -- Modify: `examples/m5stick-voice/device/src/main.cpp` (includes near top; file-scope state after line 49; `setup()` ~line 171; `loop()` after line 192) - -Run device commands from `examples/m5stick-voice/device`. - -> **HARDWARE GATE:** Do all edits and the compile checks (Steps 1–5), then **STOP at Step 6 and hand back to the human (Matt) for an on-board test**. Do **not** run the commit step (Step 7) until Matt confirms it works on a real M5Stick. This is repo policy for `device/` changes. - -- [ ] **Step 1: Add the includes for JSON + string helpers** - -In `src/main.cpp`, the existing includes are: - -```cpp -#include -#include -#include "DisplayDriver.h" -#include "IMUDriver.h" -#include "BuzzerDriver.h" -#include "PushButtonsDriver.h" -``` - -Add `` and `` immediately after `#include `: - -```cpp -#include -#include -#include -#include -#include "DisplayDriver.h" -#include "IMUDriver.h" -#include "BuzzerDriver.h" -#include "PushButtonsDriver.h" -``` - -(`JsonDocument` and the `doc["x"] | default` operator come from ArduinoJson; `strcmp` from ``. `snprintf` is already available via the Arduino core.) - -- [ ] **Step 2: Add file-scope status state** - -In `src/main.cpp`, find: - -```cpp -static volatile bool streaming = false; -``` - -Add directly below it: - -```cpp - -// ---- Coding-agent status (agent_status frames from the server) ------------- -// errorActive gates "tap to dismiss the error and restore the suspended app". -// lastPressCount detects taps: only a short press bumps getTotalPressCount(); -// a push-to-talk hold fires onHold() instead, so the two never collide. -static bool errorActive = false; -static uint16_t lastPressCount = 0; -``` - -- [ ] **Step 3: Register the `onMessage` handler in `setup()`** - -In `setup()`, find the push-to-talk registration: - -```cpp - // Push-to-talk on button 0 (the front button). Uses the 200ms default - // threshold so a tap is rejected but talk starts promptly. - buttonDriver.setLongPress(0, onHold); -``` - -Insert this block immediately **before** it: - -```cpp - // Coding-agent status from the server. Render generic status text through - // the status display, suspending a running app on the first "working" frame - // so displayText() isn't gated off — the same takeover push-to-talk uses. - sandbox.onMessage([](const char* /*transport*/, const char* type, JsonDocument& doc) { - if (strcmp(type, "agent_status") != 0) return; - const char* state = doc["state"] | ""; - - if (strcmp(state, "working") == 0) { - if (sandbox.isAppRunning() && !sandbox.isAppSuspended()) sandbox.suspendApp(); - errorActive = false; - int lines = doc["lines"] | 0; - if (lines > 0) { - char buf[40]; - snprintf(buf, sizeof(buf), "Working...\n%d lines", lines); - displayDriver.displayText(buf); - } else { - displayDriver.displayText("Working..."); - } - } else if (strcmp(state, "validating") == 0) { - displayDriver.displayText("Validating..."); - } else if (strcmp(state, "done") == 0) { - bool success = doc["success"] | false; - if (!success) { - const char* msg = doc["message"] | "Error"; - displayDriver.displayText(msg && msg[0] ? msg : "Error"); - errorActive = true; - } - // success: do nothing — the finished app's {type:"app"} frame loads - // next and loadApp() takes over the screen. - } - // "idle": ignored — avoids flashing the idle prompt before the app loads. - }); - -``` - -- [ ] **Step 4: Add tap-to-recover in `loop()`** - -In `loop()`, find: - -```cpp - M5.update(); - sandbox.loop(); // drives buttonDriver.update(), which fires onHold -``` - -Insert directly **after** the `sandbox.loop();` line: - -```cpp - - // Tap (short press) dismisses a coding-agent error and restores the app - // that was suspended to show status. Only taps bump pressCount (holds fire - // onHold instead), so this never interferes with push-to-talk. ">" tolerates - // the driver resetting pressCount to 0 on app load/unload (onAppReset). - uint16_t pc = buttonDriver.getTotalPressCount(); - if (pc > lastPressCount && errorActive) { - errorActive = false; - if (sandbox.isAppRunning()) { - sandbox.resumeApp(); - displayDriver.repaint(); - } else { - showIdlePrompt(); - } - } - lastPressCount = pc; -``` - -- [ ] **Step 5: Compile both board environments** - -Run: `pio run` (M5StickC Plus2) -Expected: `SUCCESS`. - -Run: `pio run -e m5sticks3` (M5StickS3) -Expected: `SUCCESS`. - -If either fails on `JsonDocument`/`strcmp`/`snprintf` being undeclared, confirm the Step 1 includes are present (``, ``); `snprintf` may additionally need `` — add it alongside `` if so. - -- [ ] **Step 6: HARDWARE verification (required — pause here)** - -Flash a device and have Matt confirm on the real M5Stick: - -```bash -pio run -t upload # M5StickC Plus2 -# or: pio run -e m5sticks3 -t upload -``` - -Confirm, with the deployed server (redeploy the Task-1 server change first): -- Speaking "make a bouncing ball on the device" shows **"Working…"**, then **"Working… N lines"** with the count climbing, then **"Validating…"**, then the finished app runs. -- Inducing a failure (or simulating a `done` with `success:false`) shows the error text, and a **tap** on the front button restores the previously running app (or the idle prompt if none). -- Push-to-talk still works (hold → "Listening" → transcribes), and a tap during normal operation does nothing. - -**Do not proceed to Step 7 until Matt confirms.** If something is off on hardware, return to the relevant step. - -- [ ] **Step 7: Commit (only after Matt's hardware confirmation)** - -```bash -git add examples/m5stick-voice/device/src/main.cpp -git commit -m "feat(m5stick-voice): device displays coding status from agent_status frames" -``` - -End the commit body with: -``` -Co-Authored-By: Claude Opus 4.8 (1M context) -``` - -Stay on branch `spike-voice`. Do not branch or merge. - ---- - -## Self-review notes - -- **Spec coverage:** server `WORKING_APP` push + constant removal (Task 1, Steps 1–3); device `onMessage` mapping `working`/`validating`/error→`displayText` with suspend-on-first-`working` (Task 2, Step 3); ignore `idle`/`done`-success (Step 3 comments); tap-to-recover via `getTotalPressCount()` (Step 4); malformed-field defaults via ArduinoJson `| default` (Step 3); hardware-verify-before-commit (Task 2 gate). Out-of-scope items (reasoning phase, overlay, animations) are not added. -- **Type/name consistency:** `errorActive` / `lastPressCount` declared (Task 2 Step 2) and used (Steps 3–4); `sandbox.onMessage`, `isAppRunning`, `isAppSuspended`, `suspendApp`, `resumeApp`, `getTotalPressCount`, `displayDriver.displayText`, `displayDriver.repaint`, `showIdlePrompt` all match the existing API surface in `main.cpp`/`ResidentSandbox.h`/`PushButtonsDriver.h`. `DEFAULT_APP` retained; `WORKING_APP` fully removed. -- **Placeholders:** none — every code step is complete. -``` diff --git a/.claude/superpowers/plans/2026-06-12-grove-vision-ai-example.md b/.claude/superpowers/plans/2026-06-12-grove-vision-ai-example.md deleted file mode 100644 index 07a1a4e..0000000 --- a/.claude/superpowers/plans/2026-06-12-grove-vision-ai-example.md +++ /dev/null @@ -1,1335 +0,0 @@ -# Grove Vision AI V2 Example Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** New example `examples/m5stick-grove-vision-ai` — M5StickS3 + Grove Vision AI V2 (I2C), with a model-agnostic `GroveVisionDriver` that feeds inference results into the Resident sandbox as condensed events plus a pollable `vision.*` Lua module, with verbose serial logging of everything the module emits. - -**Architecture:** A header-only pure helper (`vision_frame.h`) holds the frame structs, kind classification, and best-detection selection — natively unit-testable with no Arduino deps. `GroveVisionDriver` (a `Resident::Driver`) polls the module via the Seeed SSCMA library at ~5 Hz from `update()`, copies results into a `vision::Frame`, emits one condensed `"vision"` event per frame-with-detections (plus empty-transition and link events), exposes the cached frame to Lua, and logs full frame detail to serial. The example project copies m5stick-demo's scaffold, symlinks its shared drivers, and targets the S3 only. - -**Tech Stack:** PlatformIO (espressif32@6.12.0 / Arduino), `seeed-studio/Seeed_Arduino_SSCMA@^1.0`, M5Unified, Resident (in-tree symlink), Unity native tests. - -**Spec:** `.claude/superpowers/specs/2026-06-12-grove-vision-ai-example-design.md` - -**⚠️ Commit policy for this plan (house rule overrides "commit every task"):** firmware under `examples/*/src|lib` and `platformio.ini` is NOT committed until Matt has bench-verified it on the device. Tasks below build up the working tree; commits happen at the two marked CHECKPOINT gates. Native-test-only steps still run tests as they go. - -**Branch:** `spike/grove` (already created, current HEAD = main's 087dd07). - -**Reference numbers used throughout:** Grove I2C on M5StickS3: SDA=GPIO9, SCL=GPIO10. Module I2C addr 0x62 (SSCMA lib default). SSCMA result types: `boxes_t{uint16 x,y,w,h; uint8 score,target}`, `classes_t{uint8 target,score}`, `point_t{uint16 x,y,z; uint8 score,target}`, `keypoints_t{boxes_t box; vector points}` (pose = 17 COCO points). Scores 0–100. Coordinates = model-frame pixels (typically 192×192). - ---- - -### Task 1: Scaffold the example project (copy m5stick-demo, S3-only) - -**Files:** -- Create: `examples/m5stick-grove-vision-ai/device/platformio.ini` -- Create: `examples/m5stick-grove-vision-ai/device/src/main.cpp` (vision driver added in Task 4) -- Create: `examples/m5stick-grove-vision-ai/send-app.sh` (byte-copy) -- Create: `examples/m5stick-grove-vision-ai/.gitignore` - -- [ ] **Step 1: Copy the unchanged pieces** - -```bash -cd /Users/matt/code/resident -mkdir -p examples/m5stick-grove-vision-ai/device/src examples/m5stick-grove-vision-ai/device-apps -cp examples/m5stick-demo/send-app.sh examples/m5stick-grove-vision-ai/send-app.sh -printf '.pio/\n.resident-device-id\n' > examples/m5stick-grove-vision-ai/.gitignore -``` - -(No `partitions.csv`: m5stick-demo's S3 env doesn't use one — only the C Plus2 env does.) - -- [ ] **Step 2: Write `device/platformio.ini`** - -S3-only firmware env plus a standalone native env for the helper tests. The native env deliberately does NOT inherit a base `[env]` section — that's why there is none. - -```ini -[platformio] -default_envs = m5sticks3 - -[env:m5sticks3] -platform = espressif32@6.12.0 -framework = arduino -board = esp32-s3-devkitc-1 -board_build.flash_size = 8MB -board_build.arduino.memory_type = qio_opi -monitor_speed = 115200 -lib_deps = - ; resident: symlink to in-tree source (repo root, three levels up). - symlink://../../.. - ; PIO quirk: symlinking a parent that contains this project suppresses PIO's - ; auto-scan of our own lib/, so re-add local libs explicitly. - symlink://lib/vision - ; shared M5Stick drivers (display/imu/buzzer/buttons) live in m5stick-demo. - symlink://../../m5stick-demo/device/lib/drivers - git+https://github.com/inanimate-tech/courier.git - tzapu/WiFiManager@^2.0.17 - bblanchon/ArduinoJson@^7.4.2 - ropg/ezTime@^0.8.3 - fischer-simon/Esp32Lua@^5.4.7 - m5stack/M5Unified - M5PM1=https://github.com/m5stack/M5PM1 - seeed-studio/Seeed_Arduino_SSCMA@^1.0 -; Resident headers use std::optional (C++17). arduino-espressif32 6.12 -; defaults to gnu++11, so override. -build_unflags = -std=gnu++11 -build_flags = - -std=gnu++17 - -DBOARD_HAS_PSRAM - -DARDUINO_USB_MODE=1 - -DARDUINO_USB_CDC_ON_BOOT=1 - -DBOARD_M5STICKS3 - -; Native unit tests for the pure vision_frame helper (Task 2). The test -; includes vision_frame.h by relative path, so no lib_deps are needed and -; the SSCMA-dependent driver code is never compiled natively. -[env:native] -platform = native -test_framework = unity -build_flags = -std=gnu++17 -``` - -- [ ] **Step 3: Write `device/src/main.cpp`** - -m5stick-demo's main, S3-only (drop the C Plus2 pin branch), renamed deviceType. The vision driver is added in Task 4 — this compiles without it first. - -```cpp -#include -#include -#include "DisplayDriver.h" -#include "IMUDriver.h" -#include "BuzzerDriver.h" -#include "PushButtonsDriver.h" - -// Default endpoint: the canonical Resident relay. Devs can self-host by -// changing RESIDENT_HOST below (or extending Courier with a config portal). -// The relay speaks the Resident canonical protocol: -// wss:///devices/ ← device WS (here) -// POST https:///devices//send ← skill/curl pushes JSON -static constexpr const char* RESIDENT_HOST = "resident.inanimate.tech"; -static constexpr uint16_t RESIDENT_PORT = 443; - -// M5StickS3 buttons (ESP32-S3 with OPI PSRAM): GPIO 11 + 12. GPIO 37 is part -// of the OPI PSRAM interface — reading it via digitalRead() triggers a -// watchdog reset, which is why the C Plus2 pin map doesn't apply here. -static constexpr uint8_t BUTTON_PINS[] = {11, 12}; -static constexpr PushButtonsConfig buttonConfig = {.numButtons = 2, .pins = BUTTON_PINS}; - -DisplayDriver displayDriver; -IMUDriver imuDriver; -BuzzerDriver buzzerDriver{255}; -PushButtonsDriver buttonDriver{buttonConfig}; - -Resident::SandboxConfig makeConfig() { - Resident::SandboxConfig cfg; - cfg.deviceType = "m5stick-vision"; - cfg.extensions = {&displayDriver, &imuDriver, &buzzerDriver, &buttonDriver}; - cfg.statusDisplay = &displayDriver; - - // Courier::Config has a constructor with default args, so designated - // initializers (.host = ...) don't compile under strict ESP-IDF builds. - // Use direct field assignment. - Courier::Config courier; - courier.host = RESIDENT_HOST; - courier.port = RESIDENT_PORT; - cfg.network = courier; - - return cfg; -} - -Resident::Sandbox sandbox{makeConfig()}; - -void setup() { - Serial.begin(115200); - delay(2000); // Wait for USB CDC on M5StickS3 - auto cfg = M5.config(); - M5.begin(cfg); - M5.Display.setRotation(1); - - // Override the default /agents/-agent/ path with the - // canonical /devices/ path used by resident.inanimate.tech. - sandbox.onTransportsWillConnect([]() { - String wsPath = String("/devices/") + sandbox.getDeviceId(); - sandbox.ws().setEndpoint(RESIDENT_HOST, RESIDENT_PORT, wsPath.c_str()); - }); - - // On first successful connection, replace the StatusDisplay's "Connected" - // text with a sandbox app that shows the device ID prominently (so the - // user knows what to push to). A real app sent via push-app or - // send-app.sh will replace this. - sandbox.onConnected([]() { - static bool loaded = false; - if (loaded) return; - loaded = true; - String app = "function init(ctx)\n" - " screen.clear()\n" - " screen.text(10, 15, 'Resident', 3)\n" - " screen.text(10, 60, 'Device ID:', 2)\n" - " screen.text(10, 90, '"; - app += sandbox.getDeviceId(); - app += "', 3, 0, 255, 0)\n" - " screen.flip()\n" - "end\n"; - sandbox.loadApp(app.c_str()); - }); - - sandbox.setup(); -} - -void loop() { - M5.update(); - sandbox.loop(); -} -``` - -- [ ] **Step 4: Create the lib placeholder so the symlink resolves** - -```bash -mkdir -p examples/m5stick-grove-vision-ai/device/lib/vision/src -cat > examples/m5stick-grove-vision-ai/device/lib/vision/library.json <<'EOF' -{ - "name": "GroveVisionDriver", - "version": "0.1.0" -} -EOF -``` - -- [ ] **Step 5: Compile-check the scaffold** - -Run: `cd examples/m5stick-grove-vision-ai/device && pio run -e m5sticks3` -Expected: `SUCCESS` (lib/vision is an empty lib at this point — fine). - -**No commit yet — firmware gate at Task 5's checkpoint.** - ---- - -### Task 2: `vision_frame.h` pure helper (TDD, native) - -**Files:** -- Create: `examples/m5stick-grove-vision-ai/device/lib/vision/src/vision_frame.h` -- Test: `examples/m5stick-grove-vision-ai/device/test/test_vision_frame/test_vision_frame.cpp` - -Header-only and Arduino-free on purpose: the native test includes it by -relative path, so PlatformIO's LDF never tries to compile the -SSCMA-dependent driver under `platform = native`. - -- [ ] **Step 1: Write the failing test** - -```cpp -// examples/m5stick-grove-vision-ai/device/test/test_vision_frame/test_vision_frame.cpp -#include -#include "../../lib/vision/src/vision_frame.h" - -void setUp(void) {} -void tearDown(void) {} - -void test_classify_precedence(void) { - using vision::Kind; - // keypoints beat boxes beat points beat classes - TEST_ASSERT_EQUAL(Kind::Pose, vision::classify(1, 2, 3, 4)); - TEST_ASSERT_EQUAL(Kind::Boxes, vision::classify(0, 2, 3, 4)); - TEST_ASSERT_EQUAL(Kind::Points, vision::classify(0, 0, 3, 4)); - TEST_ASSERT_EQUAL(Kind::Classes, vision::classify(0, 0, 0, 4)); - TEST_ASSERT_EQUAL(Kind::None, vision::classify(0, 0, 0, 0)); -} - -void test_kind_name(void) { - TEST_ASSERT_EQUAL_STRING("pose", vision::kindName(vision::Kind::Pose)); - TEST_ASSERT_EQUAL_STRING("boxes", vision::kindName(vision::Kind::Boxes)); - TEST_ASSERT_EQUAL_STRING("points", vision::kindName(vision::Kind::Points)); - TEST_ASSERT_EQUAL_STRING("classes", vision::kindName(vision::Kind::Classes)); - TEST_ASSERT_EQUAL_STRING("none", vision::kindName(vision::Kind::None)); -} - -void test_frame_count_follows_kind(void) { - vision::Frame f; - f.boxes = {{10, 20, 30, 40, 90, 1}, {50, 60, 70, 80, 70, 0}}; - f.classes = {{0, 99}}; - f.kind = vision::Kind::Boxes; - TEST_ASSERT_EQUAL_INT(2, f.count()); - f.kind = vision::Kind::Classes; - TEST_ASSERT_EQUAL_INT(1, f.count()); - f.kind = vision::Kind::None; - TEST_ASSERT_EQUAL_INT(0, f.count()); -} - -void test_best_index_picks_highest_score(void) { - vision::Frame f; - f.kind = vision::Kind::Boxes; - f.boxes = {{0, 0, 1, 1, 50, 0}, {0, 0, 1, 1, 95, 2}, {0, 0, 1, 1, 70, 1}}; - TEST_ASSERT_EQUAL_INT(1, vision::bestIndex(f)); -} - -void test_best_index_pose_uses_box_score(void) { - vision::Frame f; - f.kind = vision::Kind::Pose; - vision::Person a; a.box = {0, 0, 1, 1, 40, 0}; - vision::Person b; b.box = {0, 0, 1, 1, 88, 0}; - f.people = {a, b}; - TEST_ASSERT_EQUAL_INT(1, vision::bestIndex(f)); -} - -void test_best_index_empty_is_minus_one(void) { - vision::Frame f; - TEST_ASSERT_EQUAL_INT(-1, vision::bestIndex(f)); - f.kind = vision::Kind::Boxes; // kind set but vector empty - TEST_ASSERT_EQUAL_INT(-1, vision::bestIndex(f)); -} - -int main(int, char**) { - UNITY_BEGIN(); - RUN_TEST(test_classify_precedence); - RUN_TEST(test_kind_name); - RUN_TEST(test_frame_count_follows_kind); - RUN_TEST(test_best_index_picks_highest_score); - RUN_TEST(test_best_index_pose_uses_box_score); - RUN_TEST(test_best_index_empty_is_minus_one); - UNITY_END(); - return 0; -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd examples/m5stick-grove-vision-ai/device && pio test -e native` -Expected: FAIL — `vision_frame.h: No such file or directory`. - -- [ ] **Step 3: Write `lib/vision/src/vision_frame.h`** - -```cpp -// vision_frame.h — pure data model for Grove Vision AI V2 results. -// -// Header-only and Arduino-free so it runs under native unit tests. The -// driver copies SSCMA's result vectors into these structs once per invoke; -// everything downstream (events, Lua module, logging) reads this frame. -// -// Units: coordinates are integer pixels in the model's input frame -// (typically 192×192) exactly as SSCMA reports them; scores are 0–100. -#pragma once -#include -#include - -namespace vision { - -struct Box { - uint16_t x = 0, y = 0, w = 0, h = 0; - uint8_t score = 0, target = 0; -}; - -struct Point { - uint16_t x = 0, y = 0, z = 0; - uint8_t score = 0, target = 0; -}; - -struct Classification { - uint8_t target = 0, score = 0; -}; - -struct Person { - Box box; - std::vector points; // COCO order; pose models emit 17 -}; - -enum class Kind { None, Pose, Boxes, Points, Classes }; - -// One model emits one result type per invoke; the precedence only matters -// defensively (richer kinds win if a model ever emits several). -inline Kind classify(size_t nPeople, size_t nBoxes, size_t nPoints, size_t nClasses) { - if (nPeople > 0) return Kind::Pose; - if (nBoxes > 0) return Kind::Boxes; - if (nPoints > 0) return Kind::Points; - if (nClasses > 0) return Kind::Classes; - return Kind::None; -} - -inline const char* kindName(Kind k) { - switch (k) { - case Kind::Pose: return "pose"; - case Kind::Boxes: return "boxes"; - case Kind::Points: return "points"; - case Kind::Classes: return "classes"; - default: return "none"; - } -} - -struct Frame { - Kind kind = Kind::None; - std::vector boxes; - std::vector points; - std::vector classes; - std::vector people; - - int count() const { - switch (kind) { - case Kind::Pose: return (int)people.size(); - case Kind::Boxes: return (int)boxes.size(); - case Kind::Points: return (int)points.size(); - case Kind::Classes: return (int)classes.size(); - default: return 0; - } - } -}; - -// Index of the highest-score detection for the frame's kind; -1 if none. -// Pose people rank by their box score. -inline int bestIndex(const Frame& f) { - int best = -1; - int bestScore = -1; - int n = f.count(); - for (int i = 0; i < n; i++) { - int s = -1; - switch (f.kind) { - case Kind::Pose: s = f.people[i].box.score; break; - case Kind::Boxes: s = f.boxes[i].score; break; - case Kind::Points: s = f.points[i].score; break; - case Kind::Classes: s = f.classes[i].score; break; - default: break; - } - if (s > bestScore) { bestScore = s; best = i; } - } - return best; -} - -} // namespace vision -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd examples/m5stick-grove-vision-ai/device && pio test -e native` -Expected: `6 test cases: 6 succeeded`. - -**No commit yet — gate at Task 5's checkpoint.** - ---- - -### Task 3: GroveVisionDriver (poll, events, verbose logging) - -**Files:** -- Create: `examples/m5stick-grove-vision-ai/device/lib/vision/src/GroveVisionDriver.h` -- Create: `examples/m5stick-grove-vision-ai/device/lib/vision/src/GroveVisionDriver.cpp` - -- [ ] **Step 1: Write `GroveVisionDriver.h`** - -```cpp -#ifndef GROVE_VISION_DRIVER_H -#define GROVE_VISION_DRIVER_H - -#include -#include -#include -#include "vision_frame.h" - -struct GroveVisionConfig { - uint8_t sdaPin = 9; // M5StickS3 Grove port - uint8_t sclPin = 10; - uint32_t pollMs = 200; // ~5 Hz invoke rate - bool verboseLog = true; // per-frame serial dump of everything received -}; - -// Resident driver for the Grove Vision AI V2 module (I2C addr 0x62, SSCMA -// protocol, stock SenseCraft firmware). Model-agnostic: whatever model is -// flashed via SenseCraft, results arrive as one of four SSCMA types and map -// onto vision::Frame. -// -// Lua API (module "vision"): -// vision.kind() -> "pose"|"boxes"|"classes"|"points"|"none" -// vision.count() -> detections in the last frame -// vision.detection(i) -> table (1-based; fields per kind) or nil -// vision.keypoint(i, k) -> {x,y,score} for pose person i, COCO point k (1..17), or nil -// vision.age_ms() -> ms since the last successful invoke -// vision.ok() -> true when the module link is up -// -// Events (sendEvent name "vision"): -// frame with detections: kind, n, target, score, x, y, w, h -// (classes: no box fields; points: x,y,z) -// transition to empty: kind, n=0 (once, not per empty frame) -// link up/down: kind="link", ok=1|0 -class GroveVisionDriver : public Resident::Driver { -public: - explicit GroveVisionDriver(const GroveVisionConfig& config = {}) - : _config(config) {} - - const char* name() const override { return "vision"; } - void begin() override; - void update() override; - void registerModule(Resident::LuaModule& m) override { - m.method("kind") - .method("count") - .method("detection") - .method("keypoint") - .method("age_ms") - .method("ok"); - } - - int luaKind(lua_State* L); - int luaCount(lua_State* L); - int luaDetection(lua_State* L); - int luaKeypoint(lua_State* L); - int luaAgeMs(lua_State* L); - int luaOk(lua_State* L); - -private: - void pollModule(); - void copyResults(); - void emitFrameEvent(); - void setLink(bool up); - void logModelInfo(); - void logFrame(); - - GroveVisionConfig _config; - SSCMA _ai; - vision::Frame _frame; - - bool _linkUp = false; - bool _beginOk = false; - uint8_t _failStreak = 0; - unsigned long _lastPollMs = 0; - unsigned long _lastFrameMs = 0; // last successful invoke - int _lastCount = 0; // for the transition-to-empty event - - static constexpr uint8_t FAILS_TO_LINK_DOWN = 3; - static constexpr uint32_t RETRY_WHEN_DOWN_MS = 2000; -}; - -#endif // GROVE_VISION_DRIVER_H -``` - -- [ ] **Step 2: Write `GroveVisionDriver.cpp`** - -```cpp -#include "GroveVisionDriver.h" - -#include -#include - -extern "C" { - #include "lua/lua.h" - #include "lua/lualib.h" - #include "lua/lauxlib.h" -} - -namespace { -// EventField helpers — the union member init is fiddly inline. -void setInt(Resident::EventField& f, const char* key, int v) { - f.key = key; - f.type = Resident::EventField::INT; - f.i = v; -} -void setStr(Resident::EventField& f, const char* key, const char* s) { - f.key = key; - f.type = Resident::EventField::STRING; - f.s = s; -} -} // namespace - -void GroveVisionDriver::begin() { - Wire.begin(_config.sdaPin, _config.sclPin); - _beginOk = _ai.begin(&Wire); - Serial.printf("[vision] begin: SDA=%d SCL=%d -> %s\n", - _config.sdaPin, _config.sclPin, - _beginOk ? "module found" : "MODULE OFFLINE (will retry)"); - if (_beginOk) { - setLink(true); - logModelInfo(); - } -} - -void GroveVisionDriver::update() { - unsigned long now = millis(); - uint32_t interval = _linkUp ? _config.pollMs : RETRY_WHEN_DOWN_MS; - if (now - _lastPollMs < interval) return; - _lastPollMs = now; - - if (!_beginOk) { - // begin() failed entirely (module absent at boot) — retry the handshake. - _beginOk = _ai.begin(&Wire); - if (!_beginOk) return; - } - - pollModule(); -} - -void GroveVisionDriver::pollModule() { - if (_ai.invoke(1, false, false) != CMD_OK) { - if (_linkUp && ++_failStreak >= FAILS_TO_LINK_DOWN) { - setLink(false); - } else if (_config.verboseLog) { - Serial.printf("[vision] invoke failed (%d/%d)\n", - _failStreak, FAILS_TO_LINK_DOWN); - } - return; - } - - _failStreak = 0; - _lastFrameMs = millis(); - if (!_linkUp) { - setLink(true); - logModelInfo(); // model may have been re-flashed while away - } - - copyResults(); - if (_config.verboseLog) logFrame(); - emitFrameEvent(); - _lastCount = _frame.count(); -} - -void GroveVisionDriver::copyResults() { - _frame = vision::Frame{}; - - for (const auto& b : _ai.boxes()) { - _frame.boxes.push_back({b.x, b.y, b.w, b.h, b.score, b.target}); - } - for (const auto& c : _ai.classes()) { - _frame.classes.push_back({c.target, c.score}); - } - for (const auto& p : _ai.points()) { - _frame.points.push_back({p.x, p.y, p.z, p.score, p.target}); - } - for (const auto& k : _ai.keypoints()) { - vision::Person person; - person.box = {k.box.x, k.box.y, k.box.w, k.box.h, k.box.score, k.box.target}; - for (const auto& p : k.points) { - person.points.push_back({p.x, p.y, p.z, p.score, p.target}); - } - _frame.people.push_back(std::move(person)); - } - - _frame.kind = vision::classify(_frame.people.size(), _frame.boxes.size(), - _frame.points.size(), _frame.classes.size()); -} - -void GroveVisionDriver::emitFrameEvent() { - int n = _frame.count(); - if (n == 0) { - if (_lastCount > 0) { // transition to empty, once - Resident::EventField f[2]; - setStr(f[0], "kind", vision::kindName(_frame.kind)); - setInt(f[1], "n", 0); - sendEvent("vision", f, 2); - } - return; - } - - int best = vision::bestIndex(_frame); - Resident::EventField f[8]; - setStr(f[0], "kind", vision::kindName(_frame.kind)); - setInt(f[1], "n", n); - - switch (_frame.kind) { - case vision::Kind::Classes: { - const auto& c = _frame.classes[best]; - setInt(f[2], "target", c.target); - setInt(f[3], "score", c.score); - sendEvent("vision", f, 4); - break; - } - case vision::Kind::Points: { - const auto& p = _frame.points[best]; - setInt(f[2], "target", p.target); - setInt(f[3], "score", p.score); - setInt(f[4], "x", p.x); - setInt(f[5], "y", p.y); - setInt(f[6], "z", p.z); - sendEvent("vision", f, 7); - break; - } - default: { // Boxes and Pose both lead with a box - const vision::Box& b = (_frame.kind == vision::Kind::Pose) - ? _frame.people[best].box - : _frame.boxes[best]; - setInt(f[2], "target", b.target); - setInt(f[3], "score", b.score); - setInt(f[4], "x", b.x); - setInt(f[5], "y", b.y); - setInt(f[6], "w", b.w); - setInt(f[7], "h", b.h); - sendEvent("vision", f, 8); - break; - } - } -} - -void GroveVisionDriver::setLink(bool up) { - if (_linkUp == up) return; - _linkUp = up; - _failStreak = 0; - Serial.printf("[vision] link %s\n", up ? "UP" : "DOWN"); - Resident::EventField f[2]; - setStr(f[0], "kind", "link"); - setInt(f[1], "ok", up ? 1 : 0); - sendEvent("vision", f, 2); -} - -void GroveVisionDriver::logModelInfo() { - // name() and info() come from the module's SenseCraft metadata. info() is - // raw JSON including the model's class-label table — exactly what you need - // when eyeballing an unfamiliar model's targets. - char* n = _ai.name(false); - Serial.printf("[vision] model name: %s\n", n ? n : "(null)"); - String info = _ai.info(false); - Serial.printf("[vision] model info: %s\n", info.c_str()); -} - -void GroveVisionDriver::logFrame() { - int n = _frame.count(); - if (n == 0) { - if (_lastCount > 0) Serial.println("[vision] frame empty"); - return; // stay quiet while the scene remains empty - } - - auto& perf = _ai.perf(); - Serial.printf("[vision] kind=%s n=%d perf=%u/%u/%ums\n", - vision::kindName(_frame.kind), n, - perf.prepocess, perf.inference, perf.postprocess); - - switch (_frame.kind) { - case vision::Kind::Boxes: - for (int i = 0; i < n; i++) { - const auto& b = _frame.boxes[i]; - Serial.printf(" box[%d] target=%u score=%u x=%u y=%u w=%u h=%u\n", - i, b.target, b.score, b.x, b.y, b.w, b.h); - } - break; - case vision::Kind::Classes: - for (int i = 0; i < n; i++) { - const auto& c = _frame.classes[i]; - Serial.printf(" class[%d] target=%u score=%u\n", i, c.target, c.score); - } - break; - case vision::Kind::Points: - for (int i = 0; i < n; i++) { - const auto& p = _frame.points[i]; - Serial.printf(" point[%d] target=%u score=%u x=%u y=%u z=%u\n", - i, p.target, p.score, p.x, p.y, p.z); - } - break; - case vision::Kind::Pose: - for (int i = 0; i < n; i++) { - const auto& person = _frame.people[i]; - const auto& b = person.box; - Serial.printf(" person[%d] score=%u box=(%u,%u %ux%u) kp:", - i, b.score, b.x, b.y, b.w, b.h); - for (size_t k = 0; k < person.points.size(); k++) { - const auto& p = person.points[k]; - Serial.printf(" %zu:(%u,%u,%u)", k, p.x, p.y, p.score); - } - Serial.println(); - } - break; - default: - break; - } -} - -// --- Lua bindings --- - -int GroveVisionDriver::luaKind(lua_State* L) { - lua_pushstring(L, vision::kindName(_frame.kind)); - return 1; -} - -int GroveVisionDriver::luaCount(lua_State* L) { - lua_pushinteger(L, _frame.count()); - return 1; -} - -int GroveVisionDriver::luaDetection(lua_State* L) { - int i = (int)luaL_optinteger(L, 1, 1) - 1; // 1-based from Lua - if (i < 0 || i >= _frame.count()) { - lua_pushnil(L); - return 1; - } - - lua_newtable(L); - auto setField = [L](const char* k, int v) { - lua_pushinteger(L, v); - lua_setfield(L, -2, k); - }; - - switch (_frame.kind) { - case vision::Kind::Classes: { - const auto& c = _frame.classes[i]; - setField("target", c.target); - setField("score", c.score); - break; - } - case vision::Kind::Points: { - const auto& p = _frame.points[i]; - setField("target", p.target); - setField("score", p.score); - setField("x", p.x); - setField("y", p.y); - setField("z", p.z); - break; - } - case vision::Kind::Pose: - case vision::Kind::Boxes: { - const vision::Box& b = (_frame.kind == vision::Kind::Pose) - ? _frame.people[i].box - : _frame.boxes[i]; - setField("target", b.target); - setField("score", b.score); - setField("x", b.x); - setField("y", b.y); - setField("w", b.w); - setField("h", b.h); - break; - } - default: - break; - } - return 1; -} - -int GroveVisionDriver::luaKeypoint(lua_State* L) { - int i = (int)luaL_optinteger(L, 1, 1) - 1; - int k = (int)luaL_optinteger(L, 2, 1) - 1; - if (_frame.kind != vision::Kind::Pose || - i < 0 || i >= (int)_frame.people.size() || - k < 0 || k >= (int)_frame.people[i].points.size()) { - lua_pushnil(L); - return 1; - } - const auto& p = _frame.people[i].points[k]; - lua_newtable(L); - lua_pushinteger(L, p.x); - lua_setfield(L, -2, "x"); - lua_pushinteger(L, p.y); - lua_setfield(L, -2, "y"); - lua_pushinteger(L, p.score); - lua_setfield(L, -2, "score"); - return 1; -} - -int GroveVisionDriver::luaAgeMs(lua_State* L) { - lua_pushinteger(L, (lua_Integer)(millis() - _lastFrameMs)); - return 1; -} - -int GroveVisionDriver::luaOk(lua_State* L) { - lua_pushboolean(L, _linkUp); - return 1; -} -``` - -- [ ] **Step 3: Compile-check** - -Run: `cd examples/m5stick-grove-vision-ai/device && pio run -e m5sticks3` -Expected: `SUCCESS`. If `EventField` member access fails to compile (the union is anonymous in `ResidentDriver.h`), adjust the `setInt`/`setStr` helpers to match the actual struct — check `src/ResidentDriver.h:8-16`. - -- [ ] **Step 4: Re-run native tests (must still pass)** - -Run: `cd examples/m5stick-grove-vision-ai/device && pio test -e native` -Expected: `6 test cases: 6 succeeded` (proves the driver didn't leak Arduino deps into vision_frame.h). - -**No commit yet — gate at Task 5's checkpoint.** - ---- - -### Task 4: Wire the driver into main.cpp - -**Files:** -- Modify: `examples/m5stick-grove-vision-ai/device/src/main.cpp` - -- [ ] **Step 1: Add the driver** - -Three edits to the Task 1 main.cpp: - -After `#include "PushButtonsDriver.h"`: -```cpp -#include "GroveVisionDriver.h" -``` - -After `PushButtonsDriver buttonDriver{buttonConfig};`: -```cpp -// Grove Vision AI V2 on the Grove port (I2C). Defaults: SDA=9, SCL=10, -// 5 Hz poll, verbose serial logging of every frame (the point of this spike -// is eyeballing what each SenseCraft model emits). -GroveVisionDriver visionDriver; -``` - -In `makeConfig()`, replace the extensions line with: -```cpp - cfg.extensions = {&displayDriver, &imuDriver, &buzzerDriver, - &buttonDriver, &visionDriver}; -``` - -- [ ] **Step 2: Build** - -Run: `cd examples/m5stick-grove-vision-ai/device && pio run -e m5sticks3` -Expected: `SUCCESS`. - ---- - -### Task 5: ⚙️ CHECKPOINT — bench verification, then first commit - -- [ ] **Step 1: Hand to Matt for bench test** - -Flash and watch serial: -```bash -cd examples/m5stick-grove-vision-ai/device -pio run -e m5sticks3 -t upload && pio device monitor -``` - -With the Vision AI V2 on the Grove port and (at least) a detection model -flashed via SenseCraft, expect: -- `[vision] begin: SDA=9 SCL=10 -> module found` -- `[vision] model name: …` and `[vision] model info: {…}` (label table visible) -- `[vision] link UP` -- Per-frame lines when something is in view, e.g. - `[vision] kind=boxes n=1 perf=…` + ` box[0] target=0 score=87 x=… y=… w=… h=…` -- `[vision] frame empty` once when the scene clears -- Unplug the module → `[vision] link DOWN` within ~3 polls; replug → `link UP` - + model info again. - -**STOP. Do not commit until Matt confirms the above on hardware.** Iterate on -the driver if reality disagrees with the plan (likely candidates: SSCMA -`name()`/`info()` content, perf availability, invoke latency). - -- [ ] **Step 2: Commit firmware (after Matt's confirmation)** - -```bash -cd /Users/matt/code/resident -git add examples/m5stick-grove-vision-ai/ -git commit -m "feat(examples): m5stick-grove-vision-ai — GroveVisionDriver for Grove Vision AI V2 - -New S3-only example pairing the M5StickS3 with Seeed's Grove Vision AI V2 -module over Grove I2C (SSCMA protocol, stock SenseCraft firmware). -GroveVisionDriver polls at 5 Hz, normalizes the four SSCMA result types -(boxes/classes/points/keypoints) into a cached frame, emits condensed -\"vision\" events (best detection + count, empty/link transitions), exposes -full detail to Lua via vision.* (kind/count/detection/keypoint/age_ms/ok), -and verbose-logs every frame to serial for model eyeballing. Pure frame -helper is header-only with native unit tests. - -Verified on hardware: . - -Co-Authored-By: Claude Fable 5 " -``` - ---- - -### Task 6: DEVICE-SKILL.md (Lua app authors only) - -**Files:** -- Create: `examples/m5stick-grove-vision-ai/DEVICE-SKILL.md` - -- [ ] **Step 1: Copy the m5stick-demo skill and adapt** - -```bash -cp examples/m5stick-demo/DEVICE-SKILL.md examples/m5stick-grove-vision-ai/DEVICE-SKILL.md -``` - -Then: in the intro paragraph and `## Hardware` section, state this is the -**M5StickS3** (not C Plus2) with a **Grove Vision AI V2 camera module** on the -Grove port; keep the screen/imu/buzzer/button sections verbatim (same shared -drivers); delete any C Plus2-specific notes. - -- [ ] **Step 2: Add the `vision.*` module section after `### button.*`** - -Insert exactly: - -````markdown -### vision.* - -A Grove Vision AI V2 camera module runs a SenseCraft AI model on-device and -this module exposes the latest inference results. Which *kind* of result you -get depends on the model currently flashed on the camera module — write apps -against `vision.kind()`: - -| kind | models | detection fields | -|-------------|---------------------------------------------------|-------------------------| -| `"boxes"` | person/face/gesture(rock-paper-scissors)/object detection | `x,y,w,h,score,target` | -| `"classes"` | classification (no location) | `score,target` | -| `"pose"` | human pose (YOLOv8) | `x,y,w,h,score,target` + 17 keypoints | -| `"points"` | point models | `x,y,z,score,target` | -| `"none"` | nothing detected yet / no frame | — | - -Coordinates are pixels in the **model's frame** (typically 192×192), NOT -screen pixels — scale before drawing. Scores are 0–100. `target` is an -integer class index; the mapping to labels (e.g. gesture model: 0=paper, -1=rock, 2=scissors) depends on the flashed model. - -```lua -vision.kind() -- current result kind (string, see table) -vision.count() -- number of detections in the latest frame -vision.detection(i) -- 1-based; table of fields per kind, or nil -vision.keypoint(i, k) -- pose only: person i, keypoint k (1..17) -> {x,y,score} or nil -vision.age_ms() -- ms since the camera last answered (big = stale) -vision.ok() -- false when the camera module is unreachable -``` - -COCO keypoints (k is 1-based): 1 nose, 2 l-eye, 3 r-eye, 4 l-ear, 5 r-ear, -6 l-shoulder, 7 r-shoulder, 8 l-elbow, 9 r-elbow, 10 l-wrist, 11 r-wrist, -12 l-hip, 13 r-hip, 14 l-knee, 15 r-knee, 16 l-ankle, 17 r-ankle. - -New inference arrives ~5×/sec; `on_tick` (10/sec) sees each frame about -twice. Apps also receive **events** (`on_event`) named `"vision"`: - -- detections present: `e.kind`, `e.n`, plus the best detection's fields - (same per-kind fields as the table above) -- scene became empty: `e.kind`, `e.n == 0` — sent once per transition -- camera link: `e.kind == "link"`, `e.ok` (1 up / 0 down) -```` - -- [ ] **Step 3: Add vision stubs to the `## Validation stubs` section** - -If the copied file has a `## Validation stubs` section, append the block -below to it; otherwise add the section at the end of the file: - -````markdown -## Validation stubs - -```lua --- Vision: pretend a person-detection model sees one person mid-frame. -vision = { - kind = function() return "boxes" end, - count = function() return 1 end, - detection = function(i) - if i ~= nil and i > 1 then return nil end - return {x = 80, y = 60, w = 40, h = 80, score = 85, target = 0} - end, - keypoint = function(i, k) - return {x = 96, y = 60, score = 80} - end, - age_ms = function() return 120 end, - ok = function() return true end, -} -``` -```` - -- [ ] **Step 4: Sanity-check the doc** - -Per the house rule, DEVICE-SKILL.md is for Lua app authors targeting the -sandbox — re-read and strip anything that talks about C++, drivers, I2C -internals, or firmware builds. - ---- - -### Task 7: Demo apps - -**Files:** -- Create: `examples/m5stick-grove-vision-ai/device-apps/presence.lua` -- Create: `examples/m5stick-grove-vision-ai/device-apps/tracker.lua` -- Create: `examples/m5stick-grove-vision-ai/device-apps/skeleton.lua` - -Also copy the inherited generic apps so the example is self-contained: - -```bash -cp examples/m5stick-demo/device-apps/hello.lua \ - examples/m5stick-demo/device-apps/bounce.lua \ - examples/m5stick-demo/device-apps/buttons-buzzer.lua \ - examples/m5stick-grove-vision-ai/device-apps/ -``` - -- [ ] **Step 1: Write `presence.lua` (event-driven; any detection model)** - -```lua --- presence.lua — beep and flash when the camera sees something. --- Works with any detection/classification model. Event-driven: no polling. - -local state = "waiting" -- "waiting" | "seen" -local last_target = -1 -local last_score = 0 - -local function draw() - screen.clear() - if state == "seen" then - screen.fill_rect(0, 0, screen.width(), screen.height(), 0, 80, 0) - screen.text(10, 15, "SEEN!", 4, 255, 255, 255) - screen.text(10, 70, "target " .. last_target .. " score " .. last_score, 2) - else - screen.text(10, 15, "Watching...", 3, 0, 200, 200) - if not vision.ok() then - screen.text(10, 70, "camera offline", 2, 255, 80, 80) - end - end - screen.flip() -end - -function init(ctx) - draw() -end - -function on_event(ctx, e) - if e.name ~= "vision" then return end - if e.kind == "link" then - draw() - return - end - if e.n and e.n > 0 then - if state ~= "seen" then buzzer.beep(880, 80) end - state = "seen" - last_target = e.target or -1 - last_score = e.score or 0 - else - state = "waiting" - end - draw() -end -``` - -- [ ] **Step 2: Write `tracker.lua` (poll-driven; box-emitting models)** - -```lua --- tracker.lua — draw the best detection's box live on the LCD. --- Works with boxes and pose kinds. Polls vision.* each tick. - --- Model-frame size. SenseCraft detection models typically infer on 192x192; --- eyeball the serial log (x/y/w/h ranges) and adjust if your model differs. -local FRAME = 192 - -local function sx(v) return math.floor(v * screen.width() / FRAME) end -local function sy(v) return math.floor(v * screen.height() / FRAME) end - -function init(ctx) - screen.clear() - screen.text(10, 10, "Tracker", 3) - screen.flip() -end - -function on_tick(ctx, dt) - screen.clear() - local kind = vision.kind() - local d = vision.detection(1) - if d ~= nil and (kind == "boxes" or kind == "pose") then - -- box is centered on (x, y) in model frame - local x = sx(d.x - d.w / 2) - local y = sy(d.y - d.h / 2) - screen.rect(x, y, sx(d.w), sy(d.h), 0, 255, 0) - screen.text(5, 5, "t=" .. d.target .. " s=" .. d.score, 2, 0, 255, 0) - screen.text(5, screen.height() - 25, vision.count() .. " in frame", 2) - elseif not vision.ok() then - screen.text(10, 50, "camera offline", 2, 255, 80, 80) - else - screen.text(10, 50, "nothing in frame", 2, 120, 120, 120) - end - screen.flip() -end -``` - -Note for the engineer: whether SSCMA box x/y is the centre or the top-left -corner must be confirmed on the bench (serial log makes it obvious — stand -still, compare). Adjust the `- d.w / 2` terms if it's top-left. - -- [ ] **Step 3: Write `skeleton.lua` (pose model)** - -```lua --- skeleton.lua — stick figure from the pose model's 17 COCO keypoints. - -local FRAME = 192 -local MIN_SCORE = 30 - --- COCO skeleton edges (1-based keypoint indices, see DEVICE-SKILL.md) -local EDGES = { - {6, 7}, -- shoulders - {6, 8}, {8, 10}, -- left arm - {7, 9}, {9, 11}, -- right arm - {6, 12}, {7, 13}, -- torso sides - {12, 13}, -- hips - {12, 14}, {14, 16}, -- left leg - {13, 15}, {15, 17}, -- right leg -} - -local function sx(v) return math.floor(v * screen.width() / FRAME) end -local function sy(v) return math.floor(v * screen.height() / FRAME) end - -function init(ctx) - screen.clear() - screen.text(10, 10, "Skeleton", 3) - screen.text(10, 50, "needs the pose model", 2, 120, 120, 120) - screen.flip() -end - -function on_tick(ctx, dt) - screen.clear() - if vision.kind() ~= "pose" or vision.count() == 0 then - screen.text(10, 50, vision.ok() and "no one in frame" or "camera offline", - 2, 120, 120, 120) - screen.flip() - return - end - - for e = 1, #EDGES do - local a = vision.keypoint(1, EDGES[e][1]) - local b = vision.keypoint(1, EDGES[e][2]) - if a and b and a.score >= MIN_SCORE and b.score >= MIN_SCORE then - screen.line(sx(a.x), sy(a.y), sx(b.x), sy(b.y), 0, 255, 255) - end - end - -- head: a dot at the nose - local nose = vision.keypoint(1, 1) - if nose and nose.score >= MIN_SCORE then - screen.fill_rect(sx(nose.x) - 3, sy(nose.y) - 3, 6, 6, 255, 255, 0) - end - screen.flip() -end -``` - -- [ ] **Step 4: Validate all three apps locally** - -Use the resident plugin's validator with the new DEVICE-SKILL (its stubs make -`vision.*` resolvable): - -Invoke the `resident:validate-app` skill for each of -`device-apps/presence.lua`, `device-apps/tracker.lua`, -`device-apps/skeleton.lua`, passing -`examples/m5stick-grove-vision-ai/DEVICE-SKILL.md` as the reference doc. -Expected: all three pass (compile + init + a few ticks). `skeleton.lua` will -exercise only its "not pose" path under the default boxes stub — that's fine; -its pose path runs on the bench. - ---- - -### Task 8: README - -**Files:** -- Create: `examples/m5stick-grove-vision-ai/README.md` - -- [ ] **Step 1: Write the README** - -```markdown -# m5stick-grove-vision-ai - -[Resident](../..) example: an M5StickS3 with a Seeed **Grove Vision AI V2** -camera module on the Grove port. The Vision AI module runs a SenseCraft model -entirely on-device (Himax WiseEye2: Cortex-M55 + Ethos-U55 NPU) and the -M5Stick polls results over I2C, feeding them into the Lua sandbox as events -and a pollable `vision.*` module — so hot-reloadable Lua apps can react to -what the camera sees. - -The driver is **model-agnostic**: flash a different SenseCraft model onto the -camera module and the same firmware keeps working; only the result kind -(`boxes` / `classes` / `pose` / `points`) changes. The driver also logs every -frame verbosely to serial so you can eyeball what an unfamiliar model emits. - -## Hardware - -| Component | Details | -|---|---| -| M5StickS3 | ESP32-S3, 1.14" 135×240 LCD, Grove port (I2C: SDA=GPIO9, SCL=GPIO10) | -| Grove Vision AI Module V2 | Himax WiseEye2 HX6538; SSCMA/I2C host interface (addr 0x62) | -| OV5647 camera | CSI ribbon to the Vision AI module | - -Wiring: camera ribbon → Vision AI module, Grove cable → M5StickS3 Grove -port. The Grove port powers the module; USB on the module is only needed -while flashing models. - -## Flash a model onto the camera module - -1. Open [SenseCraft AI](https://sensecraft.seeed.cc/ai/home) in Chrome/Edge. -2. Connect the Vision AI module via USB-C, pick **Grove Vision AI V2**. -3. Choose a model (person detection, face detection, gesture - rock-paper-scissors, human pose, …) and deploy. -4. Unplug USB; the module now runs that model standalone. - -## Build and flash the M5Stick - -```sh -cd device -pio run -e m5sticks3 -t upload -pio device monitor # watch [vision] logs (115200 baud) -pio test -e native # vision_frame helper unit tests (no hardware) -``` - -On boot the device joins WiFi (Courier config portal on first run), connects -to the Resident relay, and shows its device ID. Push an app: - -```sh -./send-app.sh --device-id device-apps/presence.lua -``` - -## Demo apps - -- `device-apps/presence.lua` — beep + flash when anything is detected - (any detection model). -- `device-apps/tracker.lua` — draw the best detection's box live (detection - or pose models). -- `device-apps/skeleton.lua` — stick figure from the 17 pose keypoints - (human pose model). -- plus the generic m5stick apps (`hello.lua`, `bounce.lua`, - `buttons-buzzer.lua`). - -The Lua surface (screen/imu/buzzer/button/vision) is documented in -[DEVICE-SKILL.md](DEVICE-SKILL.md). -``` - ---- - -### Task 9: CI build list + full test sweep - -**Files:** -- Modify: `tools/run-tests.py:69-76` (the `PLATFORMIO_EXAMPLES` list) - -- [ ] **Step 1: Add the example to the hardcoded build list** - -In `tools/run-tests.py`, change: - -```python -PLATFORMIO_EXAMPLES = [ - ROOT / "examples" / "m5stick-demo" / "device", - ROOT / "examples" / "adafruit-esp32-s2-feather" / "device", -``` - -to: - -```python -PLATFORMIO_EXAMPLES = [ - ROOT / "examples" / "m5stick-demo" / "device", - ROOT / "examples" / "m5stick-grove-vision-ai" / "device", - ROOT / "examples" / "adafruit-esp32-s2-feather" / "device", -``` - -Note: `pio run` inside the example builds ALL its envs including `native`; -if the runner builds with `pio run` (not `-e`), the native env builds too — -that's harmless (it compiles nothing without test sources). Check the -runner's invocation; if it errors on the native env, scope it with -`pio run -e m5sticks3` semantics per the runner's existing pattern. - -- [ ] **Step 2: Run everything** - -```bash -cd /Users/matt/code/resident -./tools/run-tests.py all -cd examples/m5stick-grove-vision-ai/device && pio test -e native -``` - -Expected: unit tests pass (root 20 + example 6 run separately), cppcheck -clean, all example builds succeed. - ---- - -### Task 10: ⚙️ CHECKPOINT — bench-run the apps, then final commit - -- [ ] **Step 1: Hand to Matt** - -Push each demo app to the device (`send-app.sh`) with a matching model -flashed and confirm: presence beeps on appearance; tracker's box follows you -(verify centre-vs-corner; fix `tracker.lua` if boxes land offset); skeleton -draws a plausible figure on the pose model. Eyeball serial output per model — -this is the spike's payoff; capture anything surprising in the napkin -(e.g. actual frame size per model, info() label table format). - -- [ ] **Step 2: Commit docs + apps + CI (after Matt's confirmation)** - -```bash -cd /Users/matt/code/resident -git add examples/m5stick-grove-vision-ai/ tools/run-tests.py -git commit -m "docs(examples): m5stick-grove-vision-ai apps, DEVICE-SKILL, README; CI build - -Three demo apps (presence/tracker/skeleton) exercising events, the vision.* -poll API, and all SSCMA result kinds; DEVICE-SKILL.md for Lua app authors -with vision validation stubs; README covering wiring and SenseCraft model -flashing. Example added to run-tests.py's build list. - -Verified on hardware with . - -Co-Authored-By: Claude Fable 5 " -``` - ---- - -## Self-review notes (already applied) - -- Spec coverage: §1 layout → T1; §2 driver → T3; §3 events → T3 (emitFrameEvent); §4 Lua module → T3 (bindings); §5 apps → T7; §6 verbose logging → T3 (logFrame/logModelInfo); §7 testing → T2/T9; target_name() stretch goal → deliberately not planned (spec marks it opportunistic; bench `info()` output decides). -- Bench-unknowns called out where they bite: box centre-vs-corner (T7 tracker), `name()`/`info()` content and perf availability (T5), per-model frame size (T7 FRAME constant). -- Type consistency: `vision::Frame/Box/Point/Classification/Person/Kind` defined once in T2, used in T3; Lua method names in `registerModule` match the implementations; event field names match DEVICE-SKILL's event docs. -``` diff --git a/.claude/superpowers/specs/2026-06-03-m5stick-voice-remote-control-design.md b/.claude/superpowers/specs/2026-06-03-m5stick-voice-remote-control-design.md deleted file mode 100644 index 2a6bfa5..0000000 --- a/.claude/superpowers/specs/2026-06-03-m5stick-voice-remote-control-design.md +++ /dev/null @@ -1,180 +0,0 @@ -# m5stick-voice remote control — push the sim's app to the device - -**Date:** 2026-06-03 -**Status:** Approved (design); implementation pending -**Scope:** `examples/m5stick-voice` (device + server) + a new `Resident::Sandbox` app-suspend primitive in the library, version bump to `0.5.1-dev`. - -## Goal - -From the m5stick-voice web viewer, the user holds the device button and says -**"ok push app"**. The server then pushes the Lua app currently shown in the -**simulator** to the **physical device** over its existing WebSocket. The -physical device keeps its push-to-talk behaviour and **suspends** any running -app while in "Listening" mode, resuming it on button release. - -## Background / current state - -- **Device already receives apps.** `Resident::Sandbox` routes an incoming - `{type:"app", code}` frame to `loadApp()` automatically - (`src/ResidentSandbox.cpp:294`). The m5stick-voice server simply never - forwards apps to the *device* connection today — only to browser monitors. -- **"The app in the sim" is server-authoritative.** `VoiceAgent.currentApp` - (set by the `create_app` tool) is broadcast to monitors; the viewer renders - `currentApp?.code ?? DEFAULT_APP` (`server/src/routes/devices.$deviceId.tsx:21`). - There is no independent browser-side editor — the sim shows exactly that code. -- **Server tool pattern exists.** The realtime model already drives the app via - function tools (`apply_css`, `create_app`) declared in `session.update` - (`server/src/agents/voice-agent.ts`). `push_app` is a sibling tool. -- **The gap is app suspend.** `Sandbox::loop()` ticks the Lua app whenever - `_appRunning` and connected (`src/ResidentSandbox.cpp:409`), and - `DisplayDriver::displayText()` is suppressed while an app runs. There is no - primitive to pause the tick and free the screen for a "Listening" indicator. - Neither Resident nor HawthornRoomDevice has this: Hawthorn's - `suspendTransports()` pauses the *network* (Courier) for OTA, and its - heavy-message *deferral* stashes inbound app pushes during recording — but - nothing pauses a *running* app's tick. - -## Design - -### Component 1 — Resident library: app-suspend primitive - -Files: `src/ResidentSandbox.h`, `src/ResidentSandbox.cpp`. Additive, -non-breaking. Naming follows the existing app-scoped convention (`loadApp`, -`sendAppEvent`, `isAppRunning`); bare `suspend()`/`resume()` is avoided because -it collides with `Courier::Client::suspend()` reachable via `sandbox.courier()`. - -New public API: - -```cpp -void suspendApp(); // if running & not suspended: _appSuspended=true; notifyAppRunning(false) -void resumeApp(); // if running & suspended: _appSuspended=false; notifyAppRunning(true) -bool isAppSuspended() const; // pairs with isAppRunning() -``` - -Behaviour: - -- New private member `bool _appSuspended = false;`. -- `loop()` tick gate (`ResidentSandbox.cpp:409`) changes from - `if (!_appRunning) return;` to `if (!_appRunning || _appSuspended) return;`. - Courier `loop()` and extension `update()` continue to run while suspended — - audio streaming, button polling, and app *reception* are unaffected; only the - Lua `on_tick`/event dispatch stops. -- `notifyAppRunning(false)` while suspended flips `DisplayDriver._appRunning` - to false, so `displayText("Listening")` is no longer suppressed. -- `isAppRunning()` keeps returning `_appRunning` (an app stays *loaded* while - suspended). Suspension is a separate axis queried via `isAppSuspended()`. -- `loadApp()` clears `_appSuspended` (a freshly pushed app starts running, never - stuck suspended). -- Both methods are no-ops when no app is loaded (`!_appRunning`). - -**No native unit test.** The `test/unit` native harness cannot construct a -`Resident::Sandbox` — it links Esp32Lua and exposes `src/` headers, but compiles -no `ResidentSandbox.cpp` and has no native stubs for Courier::Client / ezTime -`Timezone` / ArduinoJson. No existing test instantiates `Sandbox` (by design — -`test_smoke` notes real Sandbox tests don't exist yet). Building that harness is -a disproportionate yak-shave for a 3-method, one-flag change. Coverage for this -primitive is therefore the **build gate** (it compiles into every example via -`run-tests.py build`) plus **on-hardware verification** (the firmware-verify -rule already mandates a board test). A `Sandbox` native test harness is a -worthwhile separate effort, out of scope here. - -### Component 2 — Shared DisplayDriver: repaint-on-resume - -File: `examples/m5stick-demo/device/lib/drivers/src/DisplayDriver.{h,cpp}` -(shared by m5stick-demo and m5stick-voice via `symlink://`). - -"Listening" clears the *physical* display, but the off-screen sprite retains the -app's last frame. Animating apps redraw within ~100 ms on resume; a *static* app -(clock, QR — draws only in `init()`) would stay blank. Add: - -```cpp -void repaint(); // if (_initialized) _canvas.pushSprite(0, 0); -``` - -The device calls it on resume to restore the last frame instantly, regardless of -whether the app redraws in `on_tick`. (Optional: if we choose not to touch the -shared driver, static apps stay blank until their next self-redraw — accepted -limitation. Design includes `repaint()`.) - -### Component 3 — Device example: wire push-to-talk to suspend/resume - -File: `examples/m5stick-voice/device/src/main.cpp`, in `onHold()`: - -- `started == true`: `if (sandbox.isAppRunning()) sandbox.suspendApp();` then - `displayDriver.displayText("Listening")` (existing line). -- `started == false`: - `if (sandbox.isAppRunning()) { sandbox.resumeApp(); displayDriver.repaint(); }` - `else { showIdlePrompt(); }` - -When no app is loaded the existing idle-prompt behaviour is unchanged; the -deviceId line added earlier still shows. - -### Component 4 — Server: `push_app` realtime tool - -File: `server/src/agents/voice-agent.ts`. - -- Add a no-arg `push_app` function tool to the `session.update` `tools` array, - alongside `apply_css`/`create_app`. Description: "Push the app currently shown - in the simulator to the physical device. Use when the user says to push / send - / deploy / load the app onto the device / stick / hardware." -- Extend `SYSTEM_PROMPT` with a third numbered item describing `push_app`. -- `handleFunctionCall`: route `name === "push_app"` to `handlePushApp(callId)`. -- `handlePushApp(callId)`: - - `const code = this.currentApp?.code ?? DEFAULT_APP` (import `DEFAULT_APP` - from `../lib/default-app`) — **pushes exactly what the sim shows**, the - default bouncing ball when no `create_app` has run. - - `const frame = JSON.stringify({ type: "app", code })`. - - `const devices = Array.from(this.getConnections("device"))`; `d.send(frame)` - for each. The device's Courier WS parses it via `onCourierMessage` → - `loadApp` (same shape the relay `/send` forwards). - - Tool result: `{ ok: true, devices: devices.length }`, or - `{ ok: false, error: "no device connected" }` when none. - - Then `response.create` so the model speaks a brief confirmation. - -### Component 5 — Version bump + changelog - -- `library.json`: `"version": "0.5.0"` → `"0.5.1-dev"`. -- `idf_component.yml`: `version: "0.5.0"` → `"0.5.1-dev"`. -- `docs/changelog.md`: new section above `## v0.5.0`: - - ``` - ## v0.5.1-dev () - - ### New features - - - `Resident::Sandbox::suspendApp()` / `resumeApp()` / `isAppSuspended()` — - pause and resume a running app's tick without unloading it. While suspended, - `loop()` skips the Lua `on_tick`/event dispatch (Courier and extension - updates keep running) and the status display is freed for direct text. Used - by the m5stick-voice example to show "Listening" over a running app during - push-to-talk. - ``` - - `` filled with the short hash at commit time. - -## Decisions (resolved) - -- **No-app-yet:** push `DEFAULT_APP` (what the sim shows), not a refusal. -- **Resume:** in place — keep Lua state; do not re-run `init()`. -- **No deferral:** the realtime model calls `push_app` ~0.7 s after button - release (post-commit), so the device is idle when the app lands — normal - `loadApp`. Hawthorn-style inbound deferral is out of scope; if a push ever - arrives mid-suspend, `loadApp` clears `_appSuspended` and the new app runs - (may briefly draw over "Listening"). Accepted edge, documented not built. -- **repaint():** included, to keep static apps correct after a talk turn. - -## Out of scope - -- Authentication beyond the deviceId (same caveat as the rest of the relay). -- Browser-side app editing / choosing an app independent of the agent. -- Lifting Hawthorn's transport-suspend or message-deferral into Resident. - -## Verification - -- `./tools/run-tests.py unit` (new suspend/resume test) and `build`. -- Server: `npm test` + `npm run build` in `examples/m5stick-voice/server`. -- **On-hardware check before any commit** to `examples/*/src/`, - `platformio.ini`, or the library (per the firmware-verify rule): flash the - device, generate an app in the sim, say "ok push app", confirm it appears on - the stick, then hold-to-talk and confirm the app suspends to "Listening" and - resumes on release. diff --git a/.claude/superpowers/specs/2026-06-10-coding-status-updates-design.md b/.claude/superpowers/specs/2026-06-10-coding-status-updates-design.md deleted file mode 100644 index e3c385b..0000000 --- a/.claude/superpowers/specs/2026-06-10-coding-status-updates-design.md +++ /dev/null @@ -1,155 +0,0 @@ -# Coding status updates — design - -**Date:** 2026-06-10 -**Component:** `examples/m5stick-voice` (server `VoiceAgent` DO + browser monitor; firmware contract only) - -## Goal - -Give the coding job (`runCodingJob` in `server/src/agents/voice-agent.ts`) a -higher-resolution, live status stream, and broadcast it to **both** the browser -monitor connections and connected physical devices. Today the job reports only -`working` → (silence) → `done`, monitor-only, because codegen is a single -buffered fetch. - -## Unified status model - -Merge the old `agent_status` enum (`idle | working | done | error`) and the -proposed finer stream into **one** concept. One message type, broadcast to -monitors + devices: - -```json -{ "type": "agent_status", - "state": "idle | working | validating | done", - "lines": 0, - "success": true, - "message": "" } -``` - -| state | carries | meaning | -|--------------|-------------------------------------------|---------| -| `idle` | — | resting; the state `snapshot` reports | -| `working` | `lines: N` | generating Lua. "started" = `working` with `lines: 0`, then climbs | -| `validating` | — | running `validateLuaCode` | -| `done` | `success: bool`, `message` (err text when `success:false`) | terminal **event** — state then immediately returns to `idle` | - -`done` is fire-once and never persisted. A tab refreshing after a job finishes -gets `idle` from `snapshot` — no stale toast. The old `error` state is folded -into `done` with `success:false`. - -### Sequence (one retry) - -``` -working(0) → working(1..N) → validating → working(1..M) → validating → done(success) → idle -``` - -The client infers retry count from `validating → working` transitions; the -server does not send a distinct "retrying" signal. - -## Server-side implementation (`voice-agent.ts`) - -### Streaming codegen - -`callCodegenChat` switches to `stream: true`: - -- Read `resp.body` as an SSE stream: `getReader()` → `TextDecoder` → split on - `\n\n` → parse `data:` lines, stop on `[DONE]`, accumulate - `choices[0].delta.content`. -- New param `onProgress(lines: number)`. Count newlines in the accumulated text - and call `onProgress` **time-throttled** (≥250 ms between emissions; always one - final flush at stream end). -- Fence-stripping (`replace(/^```.../)`) applies to the fully-accumulated text, - as today. -- Signature: `callCodegenChat(description, followups, signal, onProgress)`. - -The throttle + newline-counting is extracted into a small standalone helper so -it is unit-testable independently of the DO. - -### Broadcast - -`setAgentStatus` becomes the broadcaster. New signature -`setAgentStatus(state, { lines?, success?, message? })`. It builds the -`agent_status` frame and sends it to **both** `getConnections("monitor")` and -`getConnections("device")`. `toMonitors` stays unchanged for monitor-only -traffic (transcript, css, app code, binary FFT frames). - -### Emission points in `runCodingJob` - -Refactor the attempt-1-then-conditional-attempt-2 block into a small -`for (attempt of [1, 2])` loop so per-attempt emissions aren't duplicated: - -``` -working({ lines: 0 }) // at handleFunctionCall, as today -for attempt in [1, 2]: - callCodegenChat(..., lines => working({ lines })) // streamed progress - validating({}) - validateLuaCode() - if ok: break -done({ success, message? }) // success=false carries the error -idle({}) // immediately after -``` - -`done` + `idle` fire from a reworked `finishJob`, which keeps its existing -`[system] completed/failed` injection back to the realtime model (a separate -channel, unaffected by this work). - -### WORKING_APP placeholder is unchanged - -`handleFunctionCall` still pushes the `WORKING_APP` Lua placeholder to physical -devices at job start (`voice-agent.ts:321`). Physical devices display the -running app and the status stream as **separate layers** — the placeholder -gives them an app to show; `agent_status` rides alongside it. - -## Client (`server/src/`) - -### `useVoiceMonitor` / types - -- `AgentStatus` type → `"idle" | "working" | "validating" | "done"`. -- The `agent_status` handler reads `state`, `lines`, `success`, `message`. -- New exposed state: - - `workingLines: number` - - `lastDone: { success: boolean; message?: string } | null` — rendered as a - dismissable toast. - - `retryCount: number` — incremented on each `validating → working` transition - within a job; reset when a fresh job starts (`working`, lines 0). -- `snapshot` only ever carries `idle | working | validating` (never `done`). - -### `StatusPill` - -- Renders `working` as "Working… N lines", "Validating…" for `validating`, - nothing for `idle`. Shows a retry hint when `retryCount > 0`. -- `done` is **not** the pill — a separate dismissable toast component shows - success ("App ready") or the error `message`. - -## Firmware contract (device side — owned separately) - -The device receives, on its existing WebSocket, alongside `{type:"app",code}`: - -```json -{ "type": "agent_status", "state": "working|validating|done|idle", - "lines": 0, "success": true, "message": "" } -``` - -It renders this however it likes, independent of the running app. No firmware -changes are specified here beyond the contract. - -## Error handling - -- Streaming fetch failures (`!resp.ok`, network) throw inside `callCodegenChat`; - `runCodingJob`'s catch path maps to `done({ success:false, message })` → `idle`. -- A second failed validation → `done({ success:false, message: validation.error })`. -- Aborted job (`signal.aborted`, superseded by a newer request) returns silently - without emitting `done`, as today. - -## Testing - -- Unit-test the extracted throttle / newline-count helper (pure function). -- Update `lua-validator` tests only if its surface is touched (not expected). -- DO streaming, broadcast fan-out, and device rendering are verified on - hardware before committing (firmware changes are never committed on a - compile-pass alone). - -## Out of scope - -- `lastLine` field (dropped — `lines: N` is enough; YAGNI). -- Any firmware rendering implementation. -- Changing the realtime-model tool surface or the `apply_css` / `push_app` paths. diff --git a/.claude/superpowers/specs/2026-06-10-device-agent-status-display-design.md b/.claude/superpowers/specs/2026-06-10-device-agent-status-display-design.md deleted file mode 100644 index ccbaefc..0000000 --- a/.claude/superpowers/specs/2026-06-10-device-agent-status-display-design.md +++ /dev/null @@ -1,184 +0,0 @@ -# Device agent-status display — design - -**Date:** 2026-06-10 -**Component:** `examples/m5stick-voice` — server `VoiceAgent` (one deletion) + device firmware `device/src/main.cpp` - -## Goal - -Make the physical M5Stick render coding-job status directly from the -`agent_status` WebSocket frames, replacing the current `WORKING_APP` Lua -placeholder that the server pushes. Keep the display generic and -straightforward — plain status text via the device's existing `displayText` -path, carrying the live line count. - -## Background - -The server already broadcasts `agent_status` frames to device connections -(`broadcastAgentStatus` in `voice-agent.ts`): - -```json -{ "type": "agent_status", "state": "idle|working|validating|done", - "lines": 0, "success": true, "message": "" } -``` - -Today the device shows "Working…" only because the server *also* pushes a -`WORKING_APP` Lua app (`pushAppToDevices(WORKING_APP)` in `handleFunctionCall`), -which the device runs like any app. That push is the trigger we are replacing. - -The Resident library routes unknown frame `type`s to a user callback -(`sandbox.onMessage(...)`, `src/ResidentSandbox.cpp:323`); `agent_status` is -unknown to the library, so the firmware can handle it with no library change. - -## Server change (`server/src/agents/voice-agent.ts`) - -In `handleFunctionCall`, the job already emits -`setAgentStatus("working", { lines: 0 })` (broadcast to devices). Remove the -placeholder push that follows it: - -```ts -// delete: -const working = this.pushAppToDevices(WORKING_APP) -if (working > 0) console.log("[voice] pushed Working... ->", working, "device(s)") -``` - -`WORKING_APP` is then unused (verified: its only reference is this push; -`DEFAULT_APP` is still used by `handlePushApp` and the route). So: - -- remove `WORKING_APP` from the import in `voice-agent.ts`, and -- delete the `WORKING_APP` export from `server/src/lib/default-app.ts`. - -No other server change — devices already receive the full `agent_status` stream. - -## Device change (`device/src/main.cpp`) - -### Display surface - -Status renders through the existing `displayDriver.displayText(const char*)` — -the same path used for "Listening", "Connecting…", and the idle prompt. -`displayText` is gated off while a Lua app is running (returns early when -`_appRunning`), so to draw over a running app we suspend it first, exactly as -the existing `onHold` push-to-talk does -(`if (sandbox.isAppRunning()) sandbox.suspendApp();`). - -### State - -File-scope additions: - -```cpp -static bool errorActive = false; // showing an error, awaiting a tap -static uint16_t lastPressCount = 0; // for tap detection (see below) -``` - -### `onMessage` handler (registered in `setup()`) - -```cpp -sandbox.onMessage([](const char* /*transport*/, const char* type, JsonDocument& doc) { - if (strcmp(type, "agent_status") != 0) return; - const char* state = doc["state"] | ""; - - if (strcmp(state, "working") == 0) { - if (sandbox.isAppRunning() && !sandbox.isAppSuspended()) sandbox.suspendApp(); - errorActive = false; - int lines = doc["lines"] | 0; - if (lines > 0) { - char buf[40]; - snprintf(buf, sizeof buf, "Working...\n%d lines", lines); - displayDriver.displayText(buf); - } else { - displayDriver.displayText("Working..."); - } - } else if (strcmp(state, "validating") == 0) { - displayDriver.displayText("Validating..."); - } else if (strcmp(state, "done") == 0) { - bool success = doc["success"] | false; - if (!success) { - const char* msg = doc["message"] | "Error"; - displayDriver.displayText(msg && msg[0] ? msg : "Error"); - errorActive = true; - } - // success: do nothing — the finished app's {type:"app"} frame loads next - // and loadApp() takes over the screen. - } - // "idle": ignored — avoids flashing the idle prompt before the app loads. -}); -``` - -Notes: -- On the first `working` frame an actively-running app is suspended; subsequent - `working` frames (line count climbing) find it already suspended and just - redraw the text. `displayText` clears and reprints, so the number updates in - place. -- The handler does not unload anything; the previous app stays loaded-but- - suspended, so it can be restored on error (below) or replaced by `loadApp` - when the finished app arrives. - -### Tap-to-recover on error - -The button driver increments `getTotalPressCount()` **only on a tap** (a short -press); a long-press fires `onHold(false)` on release and never bumps the count -(`PushButtonsDriver::update`). So taps are cleanly distinguishable from -push-to-talk holds without modifying the shared driver. - -In `loop()` (after `sandbox.loop()`): - -```cpp -uint16_t pc = buttonDriver.getTotalPressCount(); -if (pc > lastPressCount && errorActive) { - errorActive = false; - if (sandbox.isAppRunning()) { - sandbox.resumeApp(); - displayDriver.repaint(); // restore the suspended app's last frame - } else { - showIdlePrompt(); - } -} -lastPressCount = pc; -``` - -`pc > lastPressCount` (rather than `!=`) tolerates the driver resetting -`pressCount` to 0 on `onAppReset` (app load/unload). A tap while no error is -showing does nothing — identical to current behavior. A hold still works as -today (talk again); on release `onHold(false)` already resumes the suspended app -or shows the idle prompt. - -## End-to-end behavior - -Device-only flow (no browser monitor), one job: - -1. User holds button, talks → `onHold(true)` suspends any app, shows "Listening". -2. Release → `onHold(false)` resumes the app (or idle prompt). -3. Server transcribes → realtime model calls `create_app` → server broadcasts - `agent_status working{lines:0}` → device suspends the app, shows "Working...". -4. Codegen streams → `working{lines:N}` frames → device shows "Working… N lines". -5. `validating` → "Validating…". -6a. **Success:** `done{success:true}` (device ignores) → `idle` (ignored) → - server pushes `{type:"app", code}` → `loadApp` replaces the screen, app runs. -6b. **Error:** `done{success:false, message}` → device shows the message; a tap - resumes the previously-suspended app (or idle prompt). - -## Error handling - -- Malformed / missing `agent_status` fields: `doc["..."] | default` (ArduinoJson) - yields safe defaults, so a missing `lines`/`success`/`message` degrades to - "Working...", non-success false, "Error" respectively. -- Frames arriving while push-to-talk is held are not expected (the job runs - after release); no special guard is added. (If observed on hardware to clobber - "Listening", add a `streaming` guard — deferred unless seen.) - -## Testing - -- Server: `npm run typecheck` / `npm test` / `npm run build` stay green after - removing the push and the constant. -- Device: builds for both envs (`pio run` and `pio run -e m5sticks3`). -- **Hardware verification is required before committing the `device/` change** - (repo policy): confirm on a real M5Stick that a spoken "make a … on the - device" shows "Working… N lines" climbing, then "Validating…", then the - finished app runs; and that an induced error shows the message and a tap - restores the prior app. - -## Out of scope - -- Any "thinking/reasoning" phase or `reasoning_effort` change (separate, later). -- Animations, status-bar overlay, or compositing over a running app (the chosen - approach is a generic full-screen takeover via `displayText`). -- Browser/monitor behavior (unchanged). diff --git a/.claude/superpowers/specs/2026-06-12-grove-vision-ai-example-design.md b/.claude/superpowers/specs/2026-06-12-grove-vision-ai-example-design.md deleted file mode 100644 index e091490..0000000 --- a/.claude/superpowers/specs/2026-06-12-grove-vision-ai-example-design.md +++ /dev/null @@ -1,158 +0,0 @@ -# Design: `examples/m5stick-grove-vision-ai` — Grove Vision AI V2 driver + example - -Date: 2026-06-12 · Branch: `spike/grove` · Status: approved by Matt (pending spec review) - -## Goal - -A new Resident example pairing the M5StickS3 with the Seeed Grove Vision AI V2 -module (Himax WiseEye2, I2C addr 0x62 over the Grove connector). A new driver -polls the module and feeds inference results into the sandbox as events plus a -pollable Lua module, so Lua apps can react to vision — model-agnostically, -because Matt will flash different SenseCraft models onto the module while -experimenting. - -Reference implementation: `~/code/wave-demo-spike` (verified working -M5StickS3 + Vision AI V2 + YOLOv8-pose pipeline using -`Seeed_Arduino_SSCMA@^1.0`; Grove I2C SDA=9, SCL=10, ~6 Hz invoke rate, stock -SenseCraft firmware on the module). - -## Key facts driving the design - -- **SSCMA fixes the result surface regardless of model.** Every model returns - one or more of four result types: `boxes` (x,y,w,h,score,target), `classes` - (target,score), `points` (x,y,z,score,target), `keypoints` (box + N points; - pose = 17 COCO keypoints/person). A model-agnostic driver needs no changes - when models are swapped. -- **Models are flashed via the SenseCraft web studio** (USB to the module) — - not switchable from the host at runtime. -- **Resident driver events serialize into 256-byte JSON** (8-slot ring, one - event dispatched per `loop()` pass). A 17-keypoint pose frame cannot fit in - one event → condensed events + pollable detail API. -- House precedent: examples symlink m5stick-demo's shared drivers - (`symlink://../../m5stick-demo/device/lib/drivers`), new drivers live in the - new example's own `lib/`. - -## 1. Project layout (copied from m5stick-demo, S3-only) - -``` -examples/m5stick-grove-vision-ai/ -├── README.md # hardware wiring, model flashing via SenseCraft, build -├── DEVICE-SKILL.md # Lua surface for app authors (incl. validation stubs) -├── send-app.sh -├── device-apps/ # demo Lua apps (§5) -└── device/ - ├── platformio.ini # [env:m5sticks3] only - ├── partitions.csv - ├── lib/vision/ # GroveVisionDriver (new code) - └── src/main.cpp # m5stick-demo main + vision driver registered -``` - -- `platformio.ini` lib_deps: `symlink://../../..` (resident), - `symlink://../../m5stick-demo/device/lib/drivers` (shared M5Stick drivers), - `symlink://lib/vision` (new driver), `seeed-studio/Seeed_Arduino_SSCMA@^1.0`, - M5Unified. -- `deviceType = "m5stick-vision"` → WS path `/agents/m5stick-vision-agent/`. -- `SERVER_HOST` stays a `YOUR-CF-ACCOUNT` placeholder (public repo rule). - -## 2. GroveVisionDriver (`Resident::Driver`, in `lib/vision/`) - -- **Config struct**: SDA pin (default 9), SCL pin (default 10), poll interval - ms (default 200 ≈ 5 Hz), I2C clock, verbose logging flag (default ON). -- **`begin()`**: `Wire.begin(sda, scl)`; `ai.begin(&wire)`. On success, query - and log `ai.name()` / `ai.info()` so serial identifies the flashed model. - Module absent ≠ crash: mark link down, retry with backoff from `update()` - (wave-demo "MODULE OFFLINE" lesson). -- **`update()`**: rate-limited `ai.invoke(1, false, false)`; copy SSCMA result - vectors into a cached frame; classify kind with precedence - `keypoints → boxes → points → classes` (→ `pose`/`boxes`/`points`/`classes`); - emit condensed event (§3); verbose-log the frame (§6). -- **Link state machine**: consecutive invoke failures → link down (emit link - event, log); successful invoke after down → link up (emit, log, re-query - model name). - -## 3. Events (driver `sendEvent`, name `"vision"`) - -All fit the 256-byte budget for every model type: - -- Frame with detections: `kind` (string), `n` (int), best detection flattened: - `target`, `score`, `x`, `y`, `w`, `h` (box-ish kinds; `classes` omits box - fields; `points` uses x,y,z). "Best" = highest score. -- Transition to empty: single event with `kind`, `n=0` (no per-frame spam - while the scene stays empty). -- Link transitions: `kind="link"`, `ok=0|1`. - -## 4. Lua module `vision` (registerModule on the driver) - -Serves full detail from the cached last frame; apps poll during `on_tick`: - -```lua -vision.kind() -- "pose"|"boxes"|"classes"|"points"|"none" -vision.count() -- detections in last frame -vision.detection(i) -- 1-based; table with fields per kind: - -- boxes/pose: x,y,w,h,score,target - -- classes: score,target - -- points: x,y,z,score,target -vision.keypoint(i, k) -- pose person i, COCO keypoint k (1..17): table x,y,score -vision.age_ms() -- ms since last successful invoke (staleness check) -vision.ok() -- link up? -``` - -Out-of-range indices return nil. `keypoint()` returns nil for non-pose kinds. - -Units, everywhere (events, module, logs): coordinates are integer pixels in -the model's frame as SSCMA reports them (no normalization); scores are -integers 0–100. Values pass through untransformed so what Lua sees matches -what serial logging shows. - -`target` is an integer class index in all result types; the label table -("rock"/"paper"/"scissors", "face", …) lives in model metadata returned by -`ai.info()`. The driver logs that at startup/link-up. Opportunistic extra, -not a commitment: if the metadata parses reliably on real firmware, expose -`vision.target_name(t)` → string|nil. Zoo ground truth for commonality: -gesture (RPS), face, person/apple/strawberry/pet/intrusion are all Swift-YOLO -detection models → `boxes`; person-classification/gender → `classes`; -YOLOv8 pose → `keypoints`. - -## 5. Demo apps (`device-apps/`) - -Keep the inherited m5stick-demo apps; add three vision apps that between them -exercise events, the poll API, and all result kinds: - -- `presence.lua` — event-driven: beep + screen flash when a detection appears - (works with person/face/any detection model). -- `tracker.lua` — poll-driven: draw the best box live on the LCD. -- `skeleton.lua` — stick figure from the 17 pose keypoints (pose model). - -## 6. Verbose serial logging (Matt: "eyeball what is coming out of these models") - -Driver logs at 115200, default ON via config flag: - -- At `begin()`/link-up: model name + info string from SSCMA. -- Per frame: one header line (`[vision] kind=pose n=2 perf=12/85/3ms`) and one - line per detection — boxes/classes/points with all fields; pose adds one - compact keypoints line per person (`kp: 0:(112,40,89) 1:(118,36,91) …`). - Full detail, not just the condensed event — the point is seeing what each - model emits. -- Empty frames log a single quiet line only when the scene transitions to - empty (mirrors event behavior) so the console stays readable. -- Link transitions and invoke errors always log. - -## 7. Testing - -- The kind-classification + event-condensing logic lives in a small pure - helper (no SSCMA/Wire includes) so it *can* get native tests cheaply; add - them if the helper ends up non-trivial, otherwise compile-check suffices - for the spike. -- `pio run -e m5sticks3` must pass; add the env to `tools/run-tests.py`'s - hardcoded `PLATFORMIO_EXAMPLES` list so CI builds it. -- Hardware behavior (I2C link, real models, log output) verified on Matt's - bench **before committing** (house rule: compile-passing isn't enough for - `examples/*/src/` changes). - -## 8. Out of scope - -- No server component. -- No runtime model switching (SenseCraft reflashes the module over USB). -- No semantic gesture events in firmware (wave detection etc. is Lua's job - now; promote to firmware later if a pattern sticks). -- No support for the C Plus2 board in this example. diff --git a/.gitignore b/.gitignore index 47fea20..f5d13f9 100644 --- a/.gitignore +++ b/.gitignore @@ -18,7 +18,7 @@ test/unit/.pio/ # Python bytecode (from tools/run-tests.py and friends) __pycache__/ -# Git worktrees (created by superpowers' using-git-worktrees skill) +# Git worktrees .worktrees/ # clangd LSP index (per-project) @@ -33,6 +33,7 @@ __pycache__/ # personal context, in-progress thinking, and absolute paths. docs/superpowers/ examples/*/docs/superpowers/ +.claude/superpowers/ # Local pack output (pio pkg pack / compote component pack) dist/ diff --git a/CLAUDE.md b/CLAUDE.md index acac685..ab3b9d4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,8 +72,7 @@ This is a public open-source repo. A few things to keep out of tracked files: `` placeholders. - Per-developer Claude Code artefacts (`.claude/settings.local.json`, `.claude/projects/`) — all `.gitignore`-d. -- Working files from plan/spec skills (Superpowers `writing-plans`, - `writing-skills`, etc.). Save them outside the repo, not under - `docs/superpowers/`. That path is `.gitignore`-d as a safety net. +- Agent plan/spec working files. Save them outside the repo; common output + paths are `.gitignore`-d as a safety net. - Personal account identifiers in URLs and example configs. For Cloudflare Worker hostnames, use a `YOUR-CF-ACCOUNT` placeholder.