diff --git a/.claude/skills/kaizen-research/SKILL.md b/.claude/skills/kaizen-research/SKILL.md new file mode 100644 index 00000000..aa1933fd --- /dev/null +++ b/.claude/skills/kaizen-research/SKILL.md @@ -0,0 +1,384 @@ +--- +name: kaizen-research +description: Weekly Friday early-morning external + internal scan for emerging functionality, agentic trends, tools, and feature/UX improvements in the AgentCore Public Stack repo. Tracks AWS Bedrock + AgentCore announcements, Strands Agents releases, FastMCP (used by externally hosted MCP servers), the aws-samples/sample-strands-agent-with-agentcore reference repo, the MCP ecosystem (including MCP Apps + extensions), frontier model announcements, agent-harness patterns, and agentic UI/UX patterns (MCP Apps, Vercel AI SDK, assistant-ui, NN/g AI research, Linear/Cursor/Anthropic product blogs). Audits internal signals (recent commits, open PRs, CI failures, version-pin lag, dormant skills). Outputs a dated research doc + queues ideas in `docs/kaizen/review-queue.md` for that same morning's `kaizen-review-prep` (runs ~2 hours later) to rank into decisions. Opens a PR into `develop`. **Out of scope**: security advisories / Dependabot / CodeQL — those have dedicated tooling and don't need a weekly kaizen lens. Triggers: "kaizen research", "weekly research scan", "external scan", "what should we look at this week". +--- + +# Kaizen Research + +Friday early morning. The "what's the rest of the world learning that we should consider, and what's our own week telling us?" scan. Pairs with `kaizen-review-prep` which runs ~2 hours later the same morning and ranks this skill's output into a decision agenda — both docs ready before Phil sits down to review Friday morning. + +## Philosophy + +- **Subtraction first.** Every research run should propose at least as many things to *remove or simplify* as to add. A smaller stack you trust beats a bigger one you route around. **Subtraction explicitly includes replacing custom code with library-native equivalents** — when an upstream release (Strands, AgentCore SDK, FastMCP, MCP, etc.) ships a capability we'd already built or filed an issue for, the win is closing our version and adopting upstream. Example: the 2026-05-10 bootstrap run found that Strands v1.37/v1.38 silently closed our open issues #266 and #267 — the codebase surface area shrinks even though we "added" a dep bump. +- **Dual lens — impact + capability-unlock.** Evaluate every upstream feature through *two* lenses, not one: (a) **impact on existing code** (does it change, simplify, or obsolete something we already have?) and (b) **capability unlock** (what *new* product capability, UX pattern, or enhancement does this make possible that we couldn't easily do before?). Subtraction-first still applies to the first lens. But capability-unlock items — features that enable net-new product surface — must be evaluated on their strategic merit, *not* hedged into "replaces future glue we haven't written." Example: the 2026-05-10 AgentCore Runtime BYO filesystem was first framed only as "could replace future filesystem-staging glue" — under-weighting the real story (code-interpreter sandboxes, cross-session uploads, shared skill hot-swap, persistent vector indexes). A dep-bump's win is usually subtraction; a *new* platform primitive's win is usually capability unlock. Don't mis-classify. +- **Subagent fan-out.** External sources are independent — fan them out to parallel subagents and synthesize. Keeps the main context clean and runs faster. +- **Web budget soft cap.** Target ≤50 web requests. If a source is exhausted, unreachable, or rate-limited, list it as "not scanned this week" — don't skip silently. Going modestly over the cap (say, to 60) is fine if the extra requests are surfacing real signal; document the overage in the Web Budget block. Don't pad — if 30 requests covered every source meaningfully, stop at 30. +- **Cite everything.** Every external claim gets a URL + access date in the Sources Scanned appendix. Web findings rot fast and you'll re-read them next week. +- **No edits outside `docs/kaizen/`.** This skill writes a dated research doc and updates `review-queue.md`. It never touches `backend/`, `frontend/`, `infrastructure/`, `CLAUDE.md`, or skill files. + +## When to run + +Friday early morning (~6am MT). `kaizen-review-prep` runs ~2 hours later (~8am MT) so both docs are waiting when Phil sits down Friday morning. Phil reviews, picks 1–3 to ship over the coming week, and POCs additional items over the weekend. Last weekend's POC findings surface in *this* run's review-prep as Carried Over items (lifted from comments on the previous week's research PR). + +## Sources + +### External (web — last 7 days unless noted) + +1. **AWS Bedrock + AgentCore "What's New"** + - https://aws.amazon.com/about-aws/whats-new/recent/feed/ (canonical AWS What's New RSS — filter entries for Bedrock/AgentCore) + - https://aws.amazon.com/blogs/machine-learning/ (filter: bedrock, agentcore) + - Filter to: Bedrock, AgentCore, Bedrock Agents, Knowledge Bases, Guardrails, model availability/region/quota changes. + +2. **Strands Agents SDK** + - https://github.com/strands-agents/sdk-python/releases + - https://github.com/strands-agents/sdk-python/blob/main/CHANGELOG.md + - https://github.com/strands-agents/sdk-python/issues?q=is%3Aissue+sort%3Aupdated-desc + - For each new release, identify: breaking changes, new hooks/features, fixes that map to current usage in `backend/src/agents/main_agent/`. + +3. **Reference repo — `aws-samples/sample-strands-agent-with-agentcore`** + - https://github.com/aws-samples/sample-strands-agent-with-agentcore/commits/main + - Diff the last 7 days (or "since last research run" — whichever is longer). Identify new patterns, removed approaches, or fixes that map to constructs in this repo: agent setup, tool registration, AgentCore Identity flows, Memory configuration, Gateway/MCP wiring. + - This repo has historically informed our architecture; week-over-week deltas are first-class signal. + +4. **MCP ecosystem** + - https://modelcontextprotocol.io (blog, spec changes) + - https://github.com/modelcontextprotocol/servers (new servers, retired servers) + - MCP registry / awesome-mcp lists for new servers relevant to the stack (Bedrock, AWS, GitHub, Slack, observability). + +4a. **FastMCP** — used by our externally hosted MCP servers (Lambda-backed, behind AgentCore Gateway). FastMCP is **not** pinned in this repo's `pyproject.toml`; it lives in the MCP server repos this stack consumes via Gateway. Track upstream releases because changes affect server behavior we depend on. + - https://github.com/jlowin/fastmcp/releases + - https://github.com/jlowin/fastmcp/blob/main/CHANGELOG.md + - https://github.com/jlowin/fastmcp/issues?q=is%3Aissue+sort%3Aupdated-desc + - https://pypi.org/project/fastmcp/ (for latest version + release date) + - Identify: breaking changes, new server-side primitives (resources/prompts/tool decorators, lifespan, auth helpers), transport changes (especially relevant if MCP SEP-2567 sessionless transport lands), and Lambda/runtime adapter changes. + +4b. **Agentic UI/UX patterns** — emerging UI and UX conventions for AI/agentic apps. We're Angular + Tailwind, so React-specific libraries are **pattern-only** references (extract the idea, implement in signals). Focus on functionality + interaction + visual conventions, not generic "good chat UX". + - **MCP Apps + extensions** (priority): https://modelcontextprotocol.io/extensions/apps/overview, https://github.com/modelcontextprotocol/ext-apps, https://blog.modelcontextprotocol.io. The "MCP server returns an interactive UI inline with the chat" standard. Track host adoption (Claude Desktop, ChatGPT, VS Code Copilot, Goose, Postman) and new MCP extension SEPs. + - **AI SDK / Generative UI** (Vercel): https://ai-sdk.dev/docs/ai-sdk-ui, https://ai-sdk.dev/cookbook. Canonical reference for tool-call rendering, multi-step UI, generative UI, streaming state patterns. React, but the patterns port. + - **assistant-ui**: https://www.assistant-ui.com/docs, https://github.com/Yonom/assistant-ui/releases. React component library purpose-built for AI chat UI. Tracks attachment UX, threading, tool-call rendering primitives. + - **Vendor product-blog UX writeups**: https://linear.app/blog (Linear Agent), https://www.cursor.com/blog (canvas, agent harness), https://www.anthropic.com/news filtered for `artifact`/`ui`/`design`. Where in-app agentic patterns get documented by the teams shipping them. + - **OpenAI Canvas + ChatGPT UI**: https://openai.com/blog filtered for `canvas`, `chatgpt`, agent UI updates. + - **Nielsen Norman Group AI articles**: https://www.nngroup.com/topic/artificial-intelligence/. UX-research perspective; evidence-based; slow cadence — surfaces in ~1 of 4 weekly runs but high signal when it does. + - Identify: new agentic UI standards (especially MCP Apps + adjacent SEPs), tool-result rendering patterns, attachment/preview UX, multi-agent attribution patterns, consent/elicitation UX, evidence-based usability findings. + +5. **Frontier model announcements** + - https://www.anthropic.com/news + - https://openai.com/blog (filter: API, agents, tools) + - https://blog.google/technology/google-deepmind/ (Gemini) + - https://ai.meta.com/blog/ (Llama) + - Focus on capability deltas affecting agent harness design: longer context, native tool use changes, prompt caching APIs, computer use, structured output, latency/cost shifts. + +6. **Agent harness patterns** + - https://www.anthropic.com/engineering (Claude Code, agent design posts) + - https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md + - LangChain / LlamaIndex / Pydantic-AI release notes — for ideas, not adoption. + +7. **AWS Bedrock pricing + quota** + - https://aws.amazon.com/bedrock/pricing/ + - Note any model price/quota changes that could shift architecture choices in this repo (e.g., model selection in `inference_api`). + +8. **AgentCore SDK / starter-toolkit issues** + - https://github.com/aws/bedrock-agentcore-sdk-python/issues + - https://github.com/aws/bedrock-agentcore-starter-toolkit/issues + - Early-signal bugs/limits other users hit before we do. + +9. **Community signal (filtered)** + - HN search: `site:news.ycombinator.com bedrock OR agentcore OR strands OR "claude code"` (last 7 days) + - r/LocalLLaMA, r/MachineLearning — agent-harness critiques and patterns surface here before vendor blogs. + +10. **Anthropic cookbook** + - https://github.com/anthropics/anthropic-cookbook + - Worked examples often outpace docs — especially for caching, tool use, and agent loops. + +11. **Seasonal sources** (only when in window) + - AWS re:Invent (typically late Nov / early Dec) — Bedrock/AgentCore announcements. + - NeurIPS / ICLR / EMNLP agent tracks (when proceedings drop). + - If today's date is not in a known window, skip with "no seasonal sources this week". + +### Internal (this repo) + +13. **Recent commits.** `git log develop --since="7 days ago" --oneline --no-merges`. Cluster by area (`backend/`, `frontend/`, `infrastructure/`). Reverts and high-churn files signal pain points. + +14. **Open PRs + review comments.** `gh pr list --base develop --state open --limit 20`, then `gh pr view --comments` on the top 3 by comment count. Repeated review feedback is a CLAUDE.md or skill-update signal. + +15. **GitHub issues opened in last 7 days.** `gh issue list --state open --search "created:>$(date -v-7d +%Y-%m-%d)"`. Bug clustering = refactor signal. + +16. **CI failures.** `gh run list --status=failure --limit 30`. Group by workflow + job. Flaky tests and recurring infra failures. + +17. **Recent CHANGELOG.md / RELEASE_NOTES.md entries** (last 14 days). Used as the "don't re-propose what we just shipped" filter. + +18. **Skill inventory.** `find .claude/skills -name SKILL.md -exec stat -f "%Sm %N" {} \;`. Skills not modified in 60+ days and not visibly referenced in recent PRs are retirement candidates. + +19. **Version-pin lag.** For each tracked dep, fetch latest release version and compute lag: + - Backend: `strands-agents`, `boto3`, `botocore`, `fastapi`, `pydantic`, `bedrock-agentcore`, `mcp` + - Frontend: `@angular/core`, `@analogjs/platform`, `vitest` + - Infrastructure: `aws-cdk-lib`, `constructs` + - Source files: `backend/pyproject.toml`, `frontend/ai.client/package.json`, `infrastructure/package.json`. + +20. **Decisions log** — `docs/kaizen/decisions.md` (if it exists). Items previously declined; don't re-propose without materially new context. + +21. **Recent reviews** — `docs/kaizen/reviews/*.md` (last 1–2). Used to avoid duplicate proposals. + +## Output + +### 1. Primary doc — `docs/kaizen/research/YYYY-MM-DD.md` + +```markdown +# Kaizen Research — [Day, Month D, YYYY] +> Scan window: [Month D – Month D, YYYY] (7 days) +> Web budget: N/50 used (target). + +## TL;DR + +[2-3 sentences. The single most important external move and the single most pressing internal signal. Name the recommended #1 idea here.] + +## External Scan + +### What's moving this week + +[1-2 paragraphs — gestalt. What's the shape of the week? Are vendors converging on a pattern? Anything surprise you?] + +### Notable items by source + +> **Annotation conventions:** +> - `*relevance*:` — impact-on-existing-code lens. What construct/file does this affect? What does it replace, simplify, or obsolete? +> - `*unlocks*:` — capability-unlock lens (use when applicable, especially for *new* platform primitives, SDK hooks, or UX patterns). What net-new product capability or enhancement does this make possible? What could we now build that we couldn't before? +> +> Bug-fixes and incremental dep-bumps usually only need `*relevance*`. New platform features, new SDK primitives, new spec capabilities, and new UX patterns usually deserve both. + +#### AWS Bedrock / AgentCore +- **[Item]** — [1-2 sentence summary] — [URL] — *relevance*: [specific construct/file] — *unlocks* (if applicable): [net-new capability or enhancement this enables] + +#### Strands Agents +- **[Item]** — … + +#### Reference repo (aws-samples/sample-strands-agent-with-agentcore) +- **[Commit / change]** — [diff summary] — [URL] — *applicability*: [does our equivalent code do this differently? worth porting?] + +#### MCP ecosystem +- … + +#### FastMCP +- **[Release / change]** — [URL] — *implications for our MCP servers*: [breaking change? new primitive worth adopting?] + +#### Agentic UI/UX patterns +- **[Pattern / release]** — [URL] — *what it is*: [1-2 sentences] — *fit for our stack*: [direct port / pattern-only (Angular equivalent: …) / not applicable] — *where it'd land*: [SSE event / component / route] + +#### Frontier model announcements +- … + +#### Agent harness patterns +- … + +#### Pricing / quota +- … + +#### Community + GitHub issues +- … + +#### Cookbook / courses +- … + +#### Seasonal +- [content, or "Out of window — none scanned this week"] + +### Patterns worth considering + +- **[Pattern]** — [3 sentences: what it is, where it's appearing, fit for this repo] + - **Where**: [examples] + - **Fit**: [would this help? what does it replace? cost to adopt?] + - **Verdict**: [Worth trying / Not a fit / Monitor] + +## Internal Audit + +### Activity (last 7 days) +- **Commits on develop**: N (across N PRs) +- **PRs opened**: N — **merged**: N — **reverted**: N +- **Issues opened**: N — **closed**: N +- **CI failures (workflow → count)**: … + +### Repeated friction signals +- **[Pattern]** (N occurrences) — [evidence: commit SHAs, PR numbers, issue links] + - **Hypothesis**: [root cause] + - **Fix candidate**: [specific change — file + behavior] + +### Version-pin lag +| Dep | Pinned | Latest | Lag | Notes | +|---|---|---|---|---| +| strands-agents | x.y.z | a.b.c | N releases / N days | [breaking? new feature relevant to us?] | + +### Retirement candidates +- **[Skill / file / config]** — [evidence: not modified in N days, replaced by X, never referenced] + +### Risks introduced this week + +- **[Risk]** — [source URL or PR] — *what breaks if we ignore this* + +## Ideas — Top 5 (ranked) + +| # | Idea | Surface | Effort | Impact | Subtracts? | Unlocks? | +|---|---|---|---|---|---|---| +| 1 | [Title] | backend / frontend / infra / cross-cutting | L/M/H | L/M/H | [what it retires, or "addition only — justified because…"] | [net-new capability, or "—" if not applicable] | +| 2 | … | | | | | | + +### 1. [Idea title] +- **Source**: [external item / internal signal — URL or commit SHA] +- **Surface area**: [paths affected] +- **Change**: [what specifically would change] +- **Subtracts**: [what this retires/simplifies, or explicitly: "addition only — justified because…"] +- **Unlocks** (if applicable): [net-new product capability, UX pattern, or enhancement this enables — bulleted if multiple. Omit field when not a capability-unlock item.] +- **Effort × Impact**: [Low/Med/High] × [Low/Med/High] +- **Verdict**: [Worth trying / Not a fit / Monitor] + +### 2. … + +## Take + +[2-4 sentences. Net read of the week. Is the system trending toward the ecosystem or away from it? One change that would matter most. What Phil would notice first if shipped.] + +--- + +## Sources Scanned + +| # | Source | URL | Accessed | Items | +|---|---|---|---|---| +| 1 | AWS Bedrock What's New | https://… | 2026-05-10 | 3 | + +## Web Budget + +Used: N / 50 requests (target). +Skipped (unreachable / rate-limited): [list] +Skipped (other): [list with reason] +Notes: [if the cap was exceeded, name the source category that justified it] +``` + +### 2. Handoff — `docs/kaizen/review-queue.md` (rolling, not dated) + +The explicit contract with `kaizen-review-prep`. This skill **appends** new entries under `## Open`. It never edits `## Resolved` (review-prep does the move). + +```markdown +# Kaizen Review Queue + +Items added by `kaizen-research`, consumed by `kaizen-review-prep`. + +## Open + + +### [YYYY-MM-DD] [Idea title] +- **Source**: research/YYYY-MM-DD.md +- **Surface**: backend | frontend | infrastructure | cross-cutting +- **Effort × Impact**: L/M/H × L/M/H +- **Subtracts**: [yes — what / no — justification] +- **Unlocks** (if applicable): [net-new capability, UX pattern, or enhancement this enables; bulleted if multiple. Omit when not a capability-unlock item.] +- **Status**: open + +## Resolved + + +### [YYYY-MM-DD] [Idea title] +- **Source**: research/YYYY-MM-DD.md +- **Decision**: Ship | Decline | Defer until [date] +- **Reasoning**: [Phil's reason, one sentence] +- **Reviewed in**: reviews/YYYY-MM-DD.md +``` + +## How to run + +1. **Bootstrap.** If `docs/kaizen/`, `docs/kaizen/research/`, `docs/kaizen/reviews/`, or `docs/kaizen/review-queue.md` don't exist, create them. The queue starts with the headers above and empty sections. + +2. **Read recent context** (sequential — small reads): + - Last 1-2 files in `docs/kaizen/research/` + - Last 1-2 files in `docs/kaizen/reviews/` + - `docs/kaizen/decisions.md` if present + - `docs/kaizen/review-queue.md` + - Last 14 days of `CHANGELOG.md` and `RELEASE_NOTES.md` + +3. **Inventory internal signals** (parallel Bash calls): + - `git log develop --since="7 days ago" --oneline --no-merges` + - `gh pr list --base develop --state open --limit 20` + - `gh issue list --state open --search "created:>$(date -v-7d +%Y-%m-%d)"` + - `gh run list --status=failure --limit 30` + - `find .claude/skills -name SKILL.md -exec stat -f "%Sm %N" {} \;` + - Read pinned versions from the three manifest files. + +4. **Fan out external scan** — spawn parallel `general-purpose` subagents (or `Explore` for sources requiring multiple targeted lookups). One subagent per source category 1–11 above (13 categories total including 4a FastMCP and 4b Agentic UI/UX). Each subagent receives: + - The exact URLs to scan + - Scope: last 7 days + - Web budget for that subagent (3–5 requests soft target) + - Required output: 3-5 bullet items max — title, 1-2 sentence summary, URL, "relevance to this repo" line. + - **Required**: cite URLs; never fabricate. If empty, return "no notable items this week". + + Total budget across subagents targets ≤50. Track centrally; modest overage (~60) is acceptable when surfacing real signal — beyond that, stop and document the skip. + +5. **Version-pin diff.** For each tracked dep, fetch latest release version (WebFetch on the release page or registry equivalent — counts toward budget). Compute lag in releases and days. If a budget hit prevents a check, list the dep under "Skipped". + +6. **Synthesize.** Write the research doc per the shape above. Pull subagent reports verbatim into source sections; write the gestalt narrative (TL;DR, "What's moving", Take) yourself. **Top 5 weighting**: + - **Library-native subtraction** opportunities (where upstream closed a custom-code need) get a subtraction boost. + - **Capability-unlock** items — new platform primitives, SDK hooks, spec capabilities, or UX patterns that enable net-new product surface we couldn't easily build before — rank on their strategic merit, *not* deprioritized just because they don't intersect existing code. Apply the dual lens from Philosophy: if a feature genuinely unlocks new capability (code-interpreter, persistent agent state, multi-agent UI attribution, etc.), rank it like a fit item, not like a "monitor" item. Resist the temptation to hedge unlock items into "replaces future glue we haven't written" — that under-weights the real story. + - **Concrete fit** UI/UX patterns that match an existing surface (tool-call rendering, attachments, A2A attribution, consent flows) get a fit boost over generic "interesting trend" items. + +7. **Update review queue.** For each Top 5 idea, prepend a new entry under `## Open` in `docs/kaizen/review-queue.md`. Never touch `## Resolved`. + +8. **Open a PR** — see "PR creation". + +## PR creation + +```bash +DATE=$(TZ=America/Denver date +'%Y-%m-%d') +BRANCH="kaizen/research-${DATE}" + +git checkout -b "$BRANCH" develop +git add docs/kaizen/ +git commit -m "$(cat <2 weeks. +- **Concrete, not aspirational.** "Consider Strands hooks" is too vague. "Add a Strands `BeforeToolCall` hook in `backend/src/agents/main_agent/hooks/` to attribute tokens by tool" is actionable. +- **No edits to source code.** This skill only writes under `docs/kaizen/`. +- **Honest about dry weeks.** A quiet week produces a short doc, not a padded one. +- **Don't re-propose declined ideas** without materially new context. Check `docs/kaizen/decisions.md` and recent reviews. +- **Cite everything.** Every external claim has a URL + access date in the Sources Scanned appendix. +- **Don't auto-merge the PR.** Phil reviews and merges Friday morning. Review-prep runs against the unmerged PR's docs — it reads the file from the working tree, not from `develop`. + +## Confirmation + +After the PR is opened, tell Phil: +1. PR URL. +2. Top 1-2 ideas (title + Effort×Impact). +3. One-sentence Take. +4. Web budget used (N/50 target) and any skipped sources. + +Brief. The full doc is on the PR. diff --git a/.claude/skills/kaizen-review-prep/SKILL.md b/.claude/skills/kaizen-review-prep/SKILL.md new file mode 100644 index 00000000..a986b46f --- /dev/null +++ b/.claude/skills/kaizen-review-prep/SKILL.md @@ -0,0 +1,255 @@ +--- +name: kaizen-review-prep +description: Friday late-morning synthesis. Runs ~2 hours after `kaizen-research` the same morning. Consumes this week's research doc, open items in `docs/kaizen/review-queue.md`, last weekend's POC findings (from comments on the previous week's research PR), and recent merges/reverts/CI signal — produces a ranked, decision-oriented agenda. Every item has a Ship / Decline / Defer recommendation. Opens a PR into `develop`. Triggers: "kaizen review prep", "weekly review prep", "friday review", "rank kaizen ideas". +--- + +# Kaizen Review Prep + +Friday late morning, after `kaizen-research` ran earlier the same morning. This skill consolidates this week's research + open queue items + last weekend's POC findings (lifted from PR comments on the previous week's research PR) + recent repo state into a ranked decision agenda. Phil reviews Friday morning, marks ✅/❌/⏸ on each item, ships 1–3 the following week, and POCs the next batch over the weekend. + +## Philosophy + +- **Review is a decision forum, not a status update.** Everything that lands in the output should be either: (a) actionable this week, (b) explicitly deferred with a reason and revisit date, or (c) declined. Nothing is "noted." Noted-and-forgotten is how systems accumulate friction. +- **Subtraction first.** Every proposal ranks against "do nothing" and "retire something instead." If a proposal adds anything, it must explain what existing thing it either replaces or simplifies. +- **Dual lens — impact + capability-unlock.** Rank proposals through *two* lenses, not one: (a) **impact on existing code** (does this change, simplify, or obsolete something we already have?) and (b) **capability unlock** (what *new* product capability or UX enhancement does this enable that we couldn't easily build before?). Subtraction-first applies to lens (a). But proposals that genuinely unlock new product surface — code-interpreter sandboxes, persistent agent state, multi-agent UI attribution, new SSE event types that enable inline UI, etc. — must be evaluated on their strategic merit, *not* auto-deferred because they don't intersect existing code. A proposal with no `Subtracts` value but a substantive `Unlocks` value can rank above a low-impact dep-bump. Don't penalize net-new capability for not being a cleanup. +- **Multiple cycles.** Kaizen is small changes, weekly, compounding. If this week's review touches 3 things, next week's will touch 3 different things. Phil doesn't need a grand plan — he needs a reliable weekly cadence. +- **One-week feedback lag is intentional.** Phil reviews Friday → POCs over the weekend → those POC findings surface in the *next* Friday's review-prep as Carried Over items. Don't try to fold same-day POC findings in — they don't exist yet. +- **No edits outside `docs/kaizen/`.** This skill writes one Markdown file under `docs/kaizen/reviews/` and updates `docs/kaizen/review-queue.md` (moves Open → Resolved post-review). It never touches source code, `CLAUDE.md`, or skill files. Those changes happen in separate PRs after the review. + +## When to run + +Friday late morning (~8am MT), ~2 hours after `kaizen-research` runs. Phil reviews both docs Friday morning, picks 1–3 to ship over the coming week, and POCs additional items over the weekend. POC findings from last weekend's POC session surface here as Carried Over items (lifted from PR comments on the *previous* week's research PR — not this week's, which Phil hasn't seen yet). + +## Inputs + +1. **Most recent `docs/kaizen/research/YYYY-MM-DD.md`** — Friday's scan. Its Top 5 ideas are the primary candidate list. +2. **`docs/kaizen/review-queue.md`** — `## Open` entries. Includes both this week's ideas (just appended by `kaizen-research`) and any prior-week items that weren't resolved. +3. **Last 1–2 `docs/kaizen/reviews/*.md`** — what was proposed before, what was decided, anything deferred to "revisit by [date]". +4. **PR comments on the *previous* week's kaizen-research PR.** `gh pr view --comments` — Phil's reactions and weekend POC findings are first-class signal. The PR opened *this* morning by `kaizen-research` is too fresh; comments accumulate over the week as Phil POCs ideas. Pick the research PR from one week ago (or the most recent merged/closed kaizen-research PR), not today's. +5. **`docs/kaizen/decisions.md`** (if it exists) — declined items with reasons. Don't re-propose without materially new context. +6. **Recent activity since last review:** + - `git log develop --since="" --oneline --no-merges` — what shipped. + - `gh pr list --base develop --state merged --search "merged:>$(date -v-7d +%Y-%m-%d)"` — what landed. + - `gh run list --status=failure --limit 30` — fresh CI failures. +7. **`CLAUDE.md` + skill inventory** — surface concerns only; never propose unilateral edits to these. +8. **`CHANGELOG.md` / `RELEASE_NOTES.md`** — most recent ~14 days, for the "what shipped this week" celebration block + the don't-re-propose filter. + +## Output + +### 1. Review doc — `docs/kaizen/reviews/YYYY-MM-DD.md` + +```markdown +# Kaizen Review — [Day, Month D, YYYY] +> Prepared HH:MMam MT. Review window: [Month D – D] (7 days). +> Source: research/YYYY-MM-DD.md + review-queue.md (N open items). + +## Week in Review + +[2-4 sentences. What did the week reveal about the system? Use concrete language — +"The aws-samples reference repo introduced a new agent-loop pattern and we're 2 +Strands releases behind" beats "some external changes". This is Phil's pulse +check before decisions.] + +## Friction — the week's signal + +### Repeated patterns (≥2 occurrences) +- **[Pattern]** (N times) — [concrete description; quote PR review comments or commit messages where helpful] + - *Hypothesis*: [root cause] + - *Candidate fix*: [specific change — file + behavior] + +### One-offs worth watching +- **[Pattern]** (1 occurrence) — [context] + +### Silence that matters + +- **[Silence]** — [what wasn't used + what that might mean] + +## Proposals — ranked + + + +### 1. [Proposal title] +- **Source**: research/YYYY-MM-DD.md ▸ Top 5 #N | review-queue.md (open since YYYY-MM-DD) | PR comment | direct observation +- **Surface area**: backend / frontend / infrastructure / cross-cutting / docs / skills +- **Change**: [concrete description — what files change, what the new behavior is] +- **Subtracts**: [required field — what this retires, simplifies, or replaces. Or explicitly "addition only — justified because…"] +- **Unlocks** (if applicable): [net-new product capability, UX pattern, or enhancement this enables — bulleted if multiple. Required for proposals where `Subtracts: no — addition only`; the unlock is the justification. Omit when purely a cleanup/dep-bump and not applicable.] +- **Effort**: Low / Med / High +- **Impact**: Low / Med / High +- **POC findings (if Phil tried it)**: [summary or "not POCed"] +- **Ship means**: [specific action — "open PR updating X to do Y" or "retire skill Z"] +- **Decline means**: [what happens instead — usually "keep current behavior, revisit in N weeks"] +- **Recommendation**: Ship / Decline / Defer N weeks — [one-sentence why] + +### 2. [Next proposal] +… + +## Carried Over From Prior Reviews + + +- **[Deferred item]** (deferred YYYY-MM-DD until YYYY-MM-DD) — [original context]. Now due. + +## Retirement Candidates + + + +- **[Candidate]** — [evidence: not modified in N days, not referenced, replaced by X] + +## Risks Acknowledged But Not Acted On + + +- **[Risk]** — [source URL] — *what breaks if ignored* — recommendation: [Address now / Watch until [date] / Accept] + +## What Shipped This Week + + + +- [shipped item] — *why it mattered* + +## Take + +[2-4 sentences. Is the system trending toward trust or toward friction? Is the kaizen +loop catching real signal or generating noise? What's the one change that would +matter most this week if shipped? Don't sugarcoat — if a skill or pattern isn't +pulling its weight, say so.] + +--- + +## Review Protocol (for Phil) + +1. Read Friction (2 min). +2. Scan Proposals — mark ✅ Ship / ❌ Decline / ⏸ Defer on each (3-5 min). +3. Scan Retirement Candidates — same marks (1-2 min). +4. Resolve Carried Over items (1-2 min). +5. Resolve Risks block. +6. Pick 1-3 to ship this week. Decline or defer the rest with a reason. + +Target: 10-15 minutes. + +## Post-review (for Phil — separate PRs) + +- ✅ Ship items → individual feature PRs over the week. The decision is logged in this doc; the implementation lives elsewhere. +- ❌ Decline items → appended to `docs/kaizen/decisions.md` with Phil's reason so future research doesn't re-propose. +- ⏸ Defer items → kept open in `review-queue.md` with a "revisit by [date]"; surface again in the next review when due. + +This skill produces the agenda. Implementation never happens here. +``` + +### 2. Queue update — `docs/kaizen/review-queue.md` + +After Phil reviews and the decisions are logged in the review doc, this skill (or Phil himself, manually) **moves resolved items** from `## Open` to `## Resolved` with a Decision and Reasoning. On a fresh run before Phil has reviewed, the skill leaves Open as-is — only the *prior* review's outcomes get processed for queue movement. + +## How to run + +1. **Bootstrap.** Confirm `docs/kaizen/reviews/` exists; create it if not. + +2. **Read inputs** (sequential — small reads): + - Latest file in `docs/kaizen/research/` + - `docs/kaizen/review-queue.md` (full) + - Last 1–2 files in `docs/kaizen/reviews/` + - `docs/kaizen/decisions.md` if present + - Last ~14 days of `CHANGELOG.md` and `RELEASE_NOTES.md` + - `CLAUDE.md` (read-only — for context, not edits) + +3. **Pull PR comments on the latest research PR** (parallel with step 4): + ``` + gh pr list --base develop --state all --search "kaizen/research" --limit 1 --json number,url + gh pr view --comments + ``` + Capture Phil's reactions. POC findings he mentions get folded into proposal entries. + +4. **Pull recent activity** (parallel Bash): + - `git log develop --since="" --oneline --no-merges` + - `gh pr list --base develop --state merged --search "merged:>$(date -v-7d +%Y-%m-%d)" --limit 30` + - `gh run list --status=failure --limit 30` + - `gh issue list --state open --search "created:>$(date -v-7d +%Y-%m-%d)"` + +5. **Process prior-review queue movement.** For each entry in `## Open` that was resolved in the most recent review doc, move it to `## Resolved` with the Decision + Reasoning + Reviewed-in fields. Items with no decision in the prior review stay open. + +6. **Identify Carried Over items.** Scan prior review docs for `Defer N weeks` recommendations whose revisit date has hit. Add those to the new review's Carried Over section. + +7. **Synthesize the review doc** per the shape above. The Proposals list is built from: + - All `## Open` entries in `review-queue.md` (the primary source) + - Any new friction patterns surfaced from PR comments / merged PRs / CI that weren't already in the queue + - Carried Over items + Rank: + - Low-effort × High-impact first. + - **Retirement candidates** get a +1 boost (subtraction bias). + - **Capability-unlock items** (proposals with a substantive `Unlocks` field — new product capability, UX surface, or platform primitive adoption) rank on their strategic merit. Do not auto-defer just because `Subtracts: no`. A High-impact unlock can rank above a Low-impact subtraction. + - Items with **POC findings** rank above untested items at the same effort/impact. + +8. **Cap the proposal count at 10.** If more than 10 candidates, defer the lowest-ranked to next week with a note. The review is supposed to take 10-15 minutes, not be exhaustive. + +9. **Open a PR** — see "PR creation". + +## PR creation + +```bash +DATE=$(TZ=America/Denver date +'%Y-%m-%d') +BRANCH="kaizen/review-${DATE}" + +git checkout -b "$BRANCH" develop +git add docs/kaizen/ +git commit -m "$(cat <2 weeks, *that's* the finding — flag it in the Take. +- **Don't re-propose declined items** without materially new context. Cross-check `docs/kaizen/decisions.md` and the last 1–2 reviews. +- **Carried Over is not a graveyard.** Deferred items resurface on their revisit date. No silent deferrals. +- **No fabrication.** If a week was quiet, the review is short. Length tracks signal, not target word count. +- **Never edit `CLAUDE.md` or skill files unilaterally.** A proposal can recommend a change to them, but the change itself is always Phil-approved in review and shipped in a separate PR. +- **Cap at 10 proposals.** A 15-item list defeats the 10-15 min target. + +## Confirmation + +After the PR is opened, tell Phil: +1. PR URL. +2. Top 1–2 proposals (title, Effort×Impact, recommendation). +3. Top 1 retirement candidate if any. +4. One-sentence Take. +5. Estimated review time. + +Brief. Phil reads the full doc on the PR and marks decisions there or in a follow-up commit. diff --git a/.github/ACTIONS-REFERENCE.md b/.github/ACTIONS-REFERENCE.md index 229ff65d..e9a63489 100644 --- a/.github/ACTIONS-REFERENCE.md +++ b/.github/ACTIONS-REFERENCE.md @@ -29,6 +29,10 @@ GitHub provides two mechanisms for storing configuration values: | CDK_APP_API_ENABLED | Variable | No | `true` | App API | Enable/disable App API stack deployment | | CDK_APP_API_MAX_CAPACITY | Variable | No | `10` | Infrastructure, App API | Maximum App API tasks for auto-scaling | | CDK_APP_API_MEMORY | Variable | No | `1024` | Infrastructure, App API | Memory (MB) for App API ECS task (512, 1024, 2048, 4096, 8192) | +| CDK_ARTIFACTS_CERTIFICATE_ARN | Variable | No | None | Artifacts | ACM certificate ARN that covers `artifacts.{CDK_DOMAIN_NAME}`. **Must be in `us-east-1`** regardless of deployment region. Required when `CDK_ARTIFACTS_ENABLED=true`. Reuse `CDK_FRONTEND_CERTIFICATE_ARN` **only when `CDK_DOMAIN_NAME` is the apex** — TLS wildcards are one label deep, so `*.example.com` covers `artifacts.example.com` but not `artifacts.alpha.example.com`. When `CDK_DOMAIN_NAME` is a subdomain, issue a dedicated `us-east-1` cert for `*.{CDK_DOMAIN_NAME}`. | +| CDK_ARTIFACTS_ENABLED | Variable | No | `false` | Artifacts, Infrastructure, App API, Inference API, Frontend | Enable iframe-isolated artifact rendering. Toggling on provisions a DDB metadata table, S3 content bucket, CloudFront + Lambda render service, and the supporting IAM grants / env vars on the consumer stacks. | +| CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS | Variable | No | None | Artifacts | Comma-separated extra origins (beyond `https://{CDK_DOMAIN_NAME}`) permitted to embed artifact iframes via CSP `frame-ancestors` — e.g. `http://localhost:4200` for a local SPA pointed at this deployment. Applied to both the CloudFront response-headers policy and the render Lambda's CSP. **Leave unset in production**: every listed origin can frame users' artifacts (still render-token gated, but a real loosening on a shared environment). | +| CDK_ARTIFACTS_RETENTION_DAYS | Variable | No | `90` | Artifacts | Days after which soft-deleted artifacts (objects tagged `lifecycle-class=deleted`) are reaped by the S3 lifecycle rule. | | CDK_ASSISTANTS_CORS_ORIGINS | Variable | No | None | Infrastructure | Additional CORS origins for the assistants module only (appended to global CORS origins) | | CDK_AWS_ACCOUNT | Variable | Yes | None | All | 12-digit AWS account ID for CDK deployment | | CDK_CERTIFICATE_ARN | Variable | No | None | Infrastructure | ACM certificate ARN for HTTPS on ALB | diff --git a/.github/README-ACTIONS.md b/.github/README-ACTIONS.md index 0aec9de5..a259a02a 100644 --- a/.github/README-ACTIONS.md +++ b/.github/README-ACTIONS.md @@ -12,6 +12,7 @@ Deploy a production-ready multi-agent AI platform to your AWS account in about 4 |-----------|-------------| | **VPC + ALB + ECS** | Networking, load balancer, and container orchestration | | **Fine-Tuning** *(optional)* | SageMaker training/inference infrastructure, S3 artifact storage, DynamoDB job tracking | +| **Artifacts** *(optional)* | Iframe-isolated artifact rendering (DDB metadata, S3 content, CloudFront at `artifacts.{domain}`, Lambda render service) | | **RAG Ingestion** | Document ingestion pipeline for retrieval-augmented generation | | **Inference API** | Strands Agent runtime powered by AWS Bedrock AgentCore | | **App API** | Backend REST API for chat, sessions, admin, and auth | diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 00000000..ec532f7b --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,84 @@ +# Copilot Instructions — AgentCore Public Stack + +Production multi-agent conversational AI platform built on AWS Bedrock AgentCore + Strands Agents. Monorepo with four top-level packages: `backend/` (Python 3.13, FastAPI), `frontend/ai.client/` (Angular 21 + Analog.js), `infrastructure/` (AWS CDK, TypeScript), and `scripts/`. + +Authoritative deeper docs: `CLAUDE.MD` (architecture), `CONTRIBUTING.md` (setup), `.kiro/steering/` and `.claude/skills/` (topic-specific patterns — CDK, Tailwind, Angular signals, CORS, release notes, versioning). + +## Build, Test, Lint + +### Backend (`cd backend`) +```bash +uv sync --extra agentcore --extra dev +uv run python -m pytest tests/ -v +uv run python -m pytest tests/path/to/test_file.py::test_name -v # single test +uv run black src/ && uv run ruff check src/ && uv run mypy src/ +# Run services locally: +cd src/apis/app_api && uv run python main.py # port 8000 +cd src/apis/inference_api && uv run python main.py # port 8001 +``` + +### Frontend (`cd frontend/ai.client`) +```bash +npm ci +npm run start # dev server on 4200 +npm test # Vitest via Analog.js +npx vitest run path/to/file.spec.ts # single test file +npx eslint src/ && npx prettier --check src/ +``` + +### Infrastructure (`cd infrastructure`) +```bash +npm ci && npm run build +npx cdk synth # validates stacks +npx cdk deploy --all +npm test -- test/stack-dependencies.test.ts # verifies new stacks are registered +``` + +## Architecture — the big picture + +- **Three independent backend consumers** of `apis.shared`: `app_api`, `inference_api`, and `agents/`. They must **never import from each other** — only from `apis.shared`. Enforced by `backend/tests/architecture/test_import_boundaries.py`. +- **Inference API runs inside an AgentCore Runtime container.** The runtime data plane only proxies `POST /invocations` and `GET /ping` — any other route returns 404 in cloud (works locally because `localhost:8001` bypasses the gateway). User-facing CRUD endpoints **belong in app-api**, not inference-api. To get workload context on app-api, use the `AGENTCORE_RUNTIME_WORKLOAD_NAME` mint fallback in `apis/shared/oauth/agentcore_identity.py`. +- **Deploy order** (cross-stack SSM references): Infrastructure → (Gateway, RAG Ingestion, SageMaker Fine-Tuning, Artifacts, MCP Sandbox in parallel) → Inference API → App API → Frontend. App API reads `runtime-workload-identity-name` from SSM, published by Inference API. +- **Errors stream as assistant messages over SSE**, not HTTP error codes. See SSE event table in `CLAUDE.MD` (`message_start`, `content_block_*`, `tool_use`/`tool_result`, `ui_resource`, `stream_error`, `oauth_required`, `compaction`, `done`). +- **Multi-protocol tools:** direct/AWS-SDK tools live in `agents/main_agent/tools/`; remote tools come via MCP+SigV4 (Gateway Lambda) or A2A (Runtime). A2A is currently **client-only**; if exposing an A2A server, `capabilities` must include `streaming=True` or clients hang. +- **Frontend is signal-based** throughout (`signal()`, `computed()`). API shapes are defined by backend routes; matching TS interfaces must be updated in the same PR as breaking backend changes. + +## Conventions specific to this repo + +- **Auth on `apis/app_api/` routes** uses `Depends(get_current_user_from_session)` (cookie-based) or `Depends(require_admin)`. The SPA sends an httpOnly session cookie, **not** `Authorization: Bearer`. Bearer-only deps on user-facing routes cause a 401 → redirect loop. Exceptions: `auth/api_keys/` (X-API-Key) and `voice/` (voice-ticket cookie) — do not template off these. +- **Admin endpoints** go under `/admin//`, user-facing under `//`. +- **Exact dependency pins only** — no `^`, `~`, or `>=` anywhere (Python, npm, CDK). +- **Never install new packages without explicit user approval.** +- **Branch from `develop`**, never `main`. PRs target `develop`; `main` advances only via squash-merge releases. Branch naming: `feature/`. Sign commits with `git commit -s` (DCO). +- **Conventional commits** (`feat:`, `fix:`, `chore:`, ...), one logical change per commit. +- **No `print()` in backend** — use `logging`. Python: `snake_case` / `PascalCase`, type hints required. TS: strict mode, no `any` unless unavoidable. + +## File placement + +| Change | Location | +|---|---| +| New API route | `backend/src/apis/app_api//` | +| Admin endpoint | `backend/src/apis/app_api/admin//` | +| New agent tool | `backend/src/agents/main_agent/tools/` + register in `__init__.py` | +| Shared backend code | `backend/src/apis/shared//` | +| Lambda for an infra stack | `backend/src/lambdas//` (not part of `apis/` boundary) | +| Angular page | `frontend/ai.client/src/app//` | +| New CDK stack | `infrastructure/lib/-stack.ts` — also register in `test/stack-dependencies.test.ts` with a tier, add `scripts/stack-/`, add a workflow under `.github/workflows/`, update `.github/docs/deploy/step-04-deploy.md` | + +## Debugging cheatsheet + +- **Tool not appearing:** check `__init__.py` export, RBAC permissions, `enabled_tools`, ToolRegistry. +- **Session not persisting:** check AgentCore Memory config, `session_id`, `TurnBasedSessionManager` flush. +- **SSE stream disconnecting:** check the 600s timeout, client connection, quota-exceeded events. +- **Local inference-api route works, cloud returns 404:** the route isn't `/invocations` or `/ping` — move it to app-api (see Architecture). + +## Topic deep-dives + +Before non-trivial work in these areas, consult the matching skill/steering doc: + +- CDK stacks/constructs → `.claude/skills/cdk-infrastructure/` and `.kiro/steering/cdk-*.md` +- Angular components/signals → `.claude/skills/angualar-best-practices/` and `.kiro/steering/angular-*.md` +- Tailwind v4 / a11y → `.claude/skills/tailwind-ui/` and `.kiro/steering/tailwind-*.md` +- CORS across stacks → `.claude/skills/cors-deployment/SKILL.md` +- Release notes / CHANGELOG → `.claude/skills/release-notes/SKILL.md` +- Version bumps → `.claude/skills/versioning/SKILL.md` diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 70103ca9..59d6848d 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,4 +1,8 @@ version: 2 +# Version-update PRs are disabled across all ecosystems +# (open-pull-requests-limit: 0). We handle dependency upgrades manually on a +# weekly cadence. Security updates are unaffected and will still be raised by +# Dependabot when a CVE is published against a dependency. updates: # ── Python backend ── - package-ecosystem: "pip" @@ -9,7 +13,7 @@ updates: day: "monday" time: "09:00" timezone: "America/Boise" - open-pull-requests-limit: 10 + open-pull-requests-limit: 0 versioning-strategy: "increase-if-necessary" commit-message: prefix: "chore(deps)" @@ -33,7 +37,7 @@ updates: day: "monday" time: "09:00" timezone: "America/Boise" - open-pull-requests-limit: 10 + open-pull-requests-limit: 0 versioning-strategy: "increase-if-necessary" commit-message: prefix: "chore(deps)" @@ -58,7 +62,7 @@ updates: day: "monday" time: "09:00" timezone: "America/Boise" - open-pull-requests-limit: 5 + open-pull-requests-limit: 0 versioning-strategy: "increase-if-necessary" commit-message: prefix: "chore(deps)" @@ -83,7 +87,7 @@ updates: day: "monday" time: "09:00" timezone: "America/Boise" - open-pull-requests-limit: 5 + open-pull-requests-limit: 0 commit-message: prefix: "chore(deps)" include: "scope" diff --git a/.github/docs/deploy/step-02-aws-setup.md b/.github/docs/deploy/step-02-aws-setup.md index 698dc046..940a74b7 100644 --- a/.github/docs/deploy/step-02-aws-setup.md +++ b/.github/docs/deploy/step-02-aws-setup.md @@ -133,6 +133,13 @@ This allows the certificate to cover subdomains like `api.example.com` and `app. - `ALB Certificate ARN` (e.g. `arn:aws:acm:us-west-2:123456789012:certificate/abc-123`) - `CloudFront Certificate ARN` (e.g. `arn:aws:acm:us-east-1:123456789012:certificate/def-456`) +> [!IMPORTANT] +> **If you plan to enable the optional Artifacts stack, mind the wildcard depth.** A TLS wildcard covers **exactly one** label — `*.example.com` matches `artifacts.example.com` but **not** `artifacts.alpha.example.com`. +> - If `CDK_DOMAIN_NAME` is your **apex** (e.g. `example.com`), the artifact origin is `artifacts.example.com` and the existing `*.example.com` CloudFront cert covers it — reuse that cert ARN, no third certificate needed. +> - If `CDK_DOMAIN_NAME` is **already a subdomain** (e.g. `alpha.example.com`), the artifact origin is `artifacts.alpha.example.com`, which `*.example.com` does **not** cover. Issue a dedicated `us-east-1` cert for `*.alpha.example.com` (or exactly `artifacts.alpha.example.com`) and use that ARN for `CDK_ARTIFACTS_CERTIFICATE_ARN`. +> +> Verify before deploying: `aws acm describe-certificate --region us-east-1 --certificate-arn --query 'Certificate.SubjectAlternativeNames'` should list a SAN that matches `artifacts.{CDK_DOMAIN_NAME}`. +
My certificate is stuck in "Pending validation" diff --git a/.github/docs/deploy/step-03-github-config.md b/.github/docs/deploy/step-03-github-config.md index 165308eb..97b1fa40 100644 --- a/.github/docs/deploy/step-03-github-config.md +++ b/.github/docs/deploy/step-03-github-config.md @@ -100,6 +100,9 @@ This prefix is prepended to all AWS resource names to avoid conflicts. Use somet | Variable Name | Default | Description | |---------------|---------|-------------| | `CDK_FINE_TUNING_ENABLED` | `false` | Set to `true` to enable the SageMaker Fine-Tuning stack. Must be set before running the fine-tuning deployment workflow in Step 4. | +| `CDK_ARTIFACTS_ENABLED` | `false` | Set to `true` to enable iframe-isolated artifact rendering. Provisions the artifacts CloudFront origin, DDB table, S3 bucket, and Lambda. Requires `CDK_ARTIFACTS_CERTIFICATE_ARN`. | +| `CDK_ARTIFACTS_CERTIFICATE_ARN` | — | ACM certificate ARN that covers `artifacts.{CDK_DOMAIN_NAME}`. **Must be in `us-east-1`** (CloudFront requirement). Reuse `CDK_FRONTEND_CERTIFICATE_ARN` **only if `CDK_DOMAIN_NAME` is your apex** (a `*.example.com` cert covers `artifacts.example.com`). If `CDK_DOMAIN_NAME` is itself a subdomain (e.g. `alpha.example.com`), wildcards are one label deep so `*.example.com` does **not** cover `artifacts.alpha.example.com` — issue a dedicated `us-east-1` cert for `*.alpha.example.com`. See [Step 2c](./step-02-aws-setup.md#2c-create-acm-certificates). | +| `CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS` | — | Comma-separated extra origins (beyond `https://{CDK_DOMAIN_NAME}`) allowed to embed artifact iframes via CSP `frame-ancestors` — applied to both the CloudFront response-headers policy and the render Lambda. Set to `http://localhost:4200` to point a local SPA at this deployment. **Leave unset in production**: every listed origin can frame your users' artifacts (still render-token gated, but a real loosening on a shared environment). Prefer a one-off `cdk deploy '*ArtifactsStack*'` with this exported over committing it as a CI variable. | --- diff --git a/.github/docs/deploy/step-04-deploy.md b/.github/docs/deploy/step-04-deploy.md index 0b678a89..d1861656 100644 --- a/.github/docs/deploy/step-04-deploy.md +++ b/.github/docs/deploy/step-04-deploy.md @@ -10,7 +10,7 @@ --- -Now for the fun part. You'll trigger up to 8 GitHub Actions workflows in order. Each one deploys a different layer of the stack. Workflows that share the same step number can be run in parallel — just wait for the previous step to finish first. +Now for the fun part. You'll trigger up to 9 GitHub Actions workflows in order. Each one deploys a different layer of the stack. Workflows that share the same step number can be run in parallel — just wait for the previous step to finish first. ## What you'll need for this step @@ -41,6 +41,8 @@ Now for the fun part. You'll trigger up to 8 GitHub Actions workflows in order. | 1 | **Deploy Infrastructure** | VPC, subnets, ALB, ECS cluster, DynamoDB tables, S3 buckets | | 2 | **Deploy RAG Ingestion** | Document ingestion pipeline for retrieval-augmented generation | | 2 | **Deploy SageMaker Fine-Tuning** *(optional)* | SageMaker training/inference resources, S3 bucket, DynamoDB tables. Requires `CDK_FINE_TUNING_ENABLED=true` ([Step 3](./step-03-github-config.md#optional-features)). | +| 2 | **Deploy Artifacts** *(optional)* | DynamoDB metadata + S3 content + CloudFront at `artifacts.{domain}` + Lambda render service. Requires `CDK_ARTIFACTS_ENABLED=true` and `CDK_ARTIFACTS_CERTIFICATE_ARN` (cert MUST be in `us-east-1`). | +| 2 | **Deploy MCP Sandbox** *(optional)* | S3 + CloudFront at `mcp-sandbox.{domain}` + Route53 serving the MCP Apps sandbox-proxy shell. Requires `CDK_MCP_SANDBOX_ENABLED=true` and `CDK_MCP_SANDBOX_CERTIFICATE_ARN` (cert MUST be in `us-east-1`). Inert until later MCP Apps host-renderer PRs wire the SPA to it. | | 3 | **Deploy Inference API** | Strands Agent runtime container on ECS (Bedrock AgentCore) | | 4 | **Deploy App API** | Backend REST API container on ECS | | 5 | **Deploy Frontend** | Angular app to S3 + CloudFront distribution | @@ -59,6 +61,8 @@ You can monitor the current state of each workflow: | Deploy Infrastructure | [![1.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/infrastructure.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/infrastructure.yml) | | Deploy RAG Ingestion | [![2.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/rag-ingestion.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/rag-ingestion.yml) | | Deploy SageMaker Fine-Tuning *(optional)* | [![2.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/sagemaker-fine-tuning.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/sagemaker-fine-tuning.yml) | +| Deploy Artifacts *(optional)* | [![2.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/artifacts.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/artifacts.yml) | +| Deploy MCP Sandbox *(optional)* | [![2.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/mcp-sandbox.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/mcp-sandbox.yml) | | Deploy Inference API | [![3.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/inference-api.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/inference-api.yml) | | Deploy App API | [![4.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/app-api.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/app-api.yml) | | Deploy Frontend | [![5.](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/frontend.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/frontend.yml) | @@ -70,6 +74,22 @@ You can monitor the current state of each workflow: --- +## First-time deploy of a new optional stack + +> [!IMPORTANT] +> When you flip a previously-disabled stack on for the first time (e.g. setting `CDK_FINE_TUNING_ENABLED=true` or `CDK_ARTIFACTS_ENABLED=true`), the PR that enables it will modify multiple stacks at once — the new stack itself, plus any consumer stacks that read its SSM exports. If you simply merge to `develop`, every affected workflow fires in parallel and the consumers will fail with `Parameter not found` because the new stack hasn't written its SSM keys yet. Subsequent normal pushes don't hit this — the race only happens on the very first deploy of new cross-stack SSM dependencies. + +**Recommended sequence — deploy in tier order via `workflow_dispatch` against the feature branch, then merge:** + +1. Push your feature branch. Do **not** merge yet. +2. Open the **Actions** tab, pick the **Infrastructure** workflow, click **Run workflow**, and select your feature branch. Wait for it to go green. This publishes any new foundation SSM keys (e.g. the JWT signing secret for artifacts). +3. Run the new stack's workflow (e.g. **Artifacts** or **SageMaker Fine-Tuning**) the same way. Wait for green. This publishes the new stack's SSM exports. +4. Merge the PR. The consumer workflows (Inference API, App API, Frontend) re-deploy automatically on push to `develop` and find every SSM key they need on the first try. + +If you skip steps 2–3 and merge directly, the failing consumer workflows aren't broken — just click **Re-run** in the Actions tab after the foundation + new stack have completed, and they'll succeed. The pre-merge sequence above just spares you the retry dance and keeps your Actions history clean. + +--- + ## What Each Workflow Does
@@ -109,6 +129,50 @@ After deployment, grant fine-tuning access to users via the admin dashboard.
+
+2. Deploy Artifacts (Optional) + +Provisions iframe-isolated artifact rendering — versioned, sandboxed HTML/code artifacts the agent can generate and the user can render inline. Skip if you don't need this capability; the rest of the stack works without it. + +Creates: +- DynamoDB table for artifact metadata + version log +- S3 bucket for artifact content blobs (private, no CORS) +- CloudFront distribution serving `artifacts.{CDK_DOMAIN_NAME}` +- Route 53 A record for the artifacts subdomain +- Python render Lambda that wraps content in a strict CSP shell + +To enable, set these GitHub environment variables before running: + +- `CDK_ARTIFACTS_ENABLED=true` +- `CDK_ARTIFACTS_CERTIFICATE_ARN` — ACM cert ARN that covers `artifacts.{domain}`. **Must be in `us-east-1`** (CloudFront requirement). Reuse `CDK_FRONTEND_CERTIFICATE_ARN` **only if `CDK_DOMAIN_NAME` is your apex** — a `*.example.com` cert covers `artifacts.example.com` but, because TLS wildcards are one label deep, does **not** cover `artifacts.alpha.example.com`. If `CDK_DOMAIN_NAME` is a subdomain, issue a dedicated `us-east-1` cert for `*.{CDK_DOMAIN_NAME}`. See [Step 2c](./step-02-aws-setup.md#2c-create-acm-certificates). +- `CDK_ARTIFACTS_RETENTION_DAYS` *(optional, default 90)* — how long soft-deleted artifacts linger before lifecycle expiry. +- `CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS` *(optional, default none)* — comma-separated extra origins allowed to embed artifact iframes via CSP `frame-ancestors`, on top of `https://{domain}`. Set to `http://localhost:4200` to point a local SPA at this environment. **Leave unset in production** — every listed origin can frame users' artifacts. Prefer a one-off targeted `cdk deploy '*ArtifactsStack*'` with this var exported over committing it as a CI variable, so localhost never lands in automated shared-env deploys. + +The artifact origin is intentionally a sibling subdomain (not the SPA origin) so artifact JS runs cross-origin and cannot access the `__Host-` session cookies, `localStorage`, or the app API. Defense in depth via strict CSP (`connect-src 'none'`, pinned `frame-ancestors`) is enforced both at the Lambda response and at the CloudFront response-headers policy. + +
+ +
+2. Deploy MCP Sandbox (Optional) + +Provisions the MCP Apps **sandbox-proxy origin** — a dedicated cross-origin shell that the SPA's MCP App iframe is pointed at, so interactive MCP App UIs run isolated from the `ai.client` origin. This is PR #1 of the MCP Apps host-renderer initiative (`docs/kaizen/scoping/mcp-apps-host-renderer.md`). Skip if you don't need MCP Apps; the rest of the stack works without it. + +Creates: +- S3 bucket (private, OAC-only) holding the static `proxy.html` + `proxy.js` shell +- CloudFront distribution serving `mcp-sandbox.{CDK_DOMAIN_NAME}` +- Route 53 A record for the sandbox subdomain +- A CloudFront response-headers policy that stamps `Content-Security-Policy: frame-ancestors ` + +To enable, set these GitHub environment variables before running: + +- `CDK_MCP_SANDBOX_ENABLED=true` +- `CDK_MCP_SANDBOX_CERTIFICATE_ARN` — ACM cert ARN that covers `mcp-sandbox.{domain}`. **Must be in `us-east-1`** (CloudFront requirement). The same wildcard-depth caveat as Artifacts applies: a `*.example.com` cert covers `mcp-sandbox.example.com` but **not** `mcp-sandbox.alpha.example.com`. If `CDK_DOMAIN_NAME` is a subdomain, issue a dedicated `us-east-1` cert for `*.{CDK_DOMAIN_NAME}`. See [Step 2c](./step-02-aws-setup.md#2c-create-acm-certificates). +- `CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS` *(optional, default none)* — comma-separated extra origins allowed to embed the proxy iframe via CSP `frame-ancestors`, on top of `https://{domain}`. Set to `http://localhost:4200` to point a local SPA at this environment. **Leave unset in production** — every listed origin can frame the proxy. Prefer a one-off targeted `cdk deploy '*McpSandboxStack*'` with this var exported over committing it as a CI variable, so localhost never lands in automated shared-env deploys. + +The sandbox origin is intentionally a sibling subdomain (not the SPA origin), matching the Artifacts pattern: it is the **outer** half of the spec's Sandbox Proxy pattern, giving a stable cross-origin boundary so the inner MCP App content frame's `allow-same-origin` never reaches the SPA's cookies/`localStorage`/app API. When this stack is enabled, the Inference API conditionally consumes its `/mcp-sandbox/origin` SSM export into `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN` and surfaces it on the `ui_resource` SSE event so the SPA knows where to frame an App. The MCP Apps host surface is now on by default (`AGENTCORE_MCP_APPS_HOST_ENABLED=true` since PR #7), but **stays dormant until an MCP-Apps-capable server is registered** in the tool catalog — see [Register an MCP-Apps-capable MCP server](#register-an-mcp-apps-capable-mcp-server) below. If you skip this stack the surface stays dormant regardless, because the SPA has no proxy origin to frame an App in. + +
+
3. Deploy Inference API (AgentCore Runtime) @@ -161,6 +225,77 @@ You can re-run this workflow later to update seed data. --- +## Register an MCP-Apps-capable MCP server + +The MCP Apps host renderer (`docs/kaizen/scoping/mcp-apps-host-renderer.md`) is **on by default** (`AGENTCORE_MCP_APPS_HOST_ENABLED=true` since PR #7). But it stays completely dormant until you register an MCP-Apps-capable MCP server in the tool catalog — there is no built-in MCP App. Registration is a normal **tool-catalog** operation (DynamoDB, via the admin API); it is *not* a code constant or part of bootstrap seeding, so each environment opts in explicitly. + +### What "MCP-Apps-capable" means + +A server is MCP-Apps-capable when, on top of being a normal external MCP server, it: + +- advertises `_meta.ui` on its `tools/list` entries (a `resourceUri` of the form `ui://…` plus a `visibility` that includes `app`), and +- serves that `ui://` resource via `resources/read` as `text/html;profile=mcp-app`. + +Our host advertises the `io.modelcontextprotocol/ui` extension on every outbound MCP `initialize` automatically (Gateway + external clients) — the server side needs no per-server opt-in here. Servers that don't speak the extension simply ignore it and behave as plain MCP tools. + +### Prerequisites + +1. **MCP Sandbox stack deployed** (`CDK_MCP_SANDBOX_ENABLED=true`, see *Deploy MCP Sandbox (Optional)* under [What Each Workflow Does](#what-each-workflow-does)). Without it the Inference API has no `/{prefix}/mcp-sandbox/origin` SSM value to consume, `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN` stays empty, and the SPA has no cross-origin shell to frame the App in — every other piece can be wired and the App still won't render. +2. **An MCP-Apps server reachable over Streamable HTTP.** External MCP tools connect via the Streamable-HTTP client (not stdio). Run the example server in HTTP mode (below). +3. **Admin access** to the running App API (admin session cookie; the tool admin endpoints chain through `require_admin`). + +### Step 1 — Run the example server (dogfood) + +We dogfood with [`modelcontextprotocol/ext-apps` → `budget-allocator-server`](https://github.com/modelcontextprotocol/ext-apps/tree/main/examples/budget-allocator-server) — a form-style App (sliders, presets, benchmarks) that exercises `ui/update-model-context` and app-initiated `tools/call` without any 3D/charting backend infra. `scenario-modeler-server` works the same way if you prefer it. + +```bash +git clone https://github.com/modelcontextprotocol/ext-apps +cd ext-apps/examples/budget-allocator-server +npm install +npm run start:http # stateless Streamable HTTP on http://localhost:3001/mcp +``` + +(Override the port with `PORT=…`. The README's default config is stdio — use `start:http`, since external MCP tools here are Streamable-HTTP only.) + +### Step 2 — Register it in the tool catalog + +Either use the **Admin UI** (Settings → Tools → Add tool → protocol *MCP (external)*) or the admin API directly. A ready-to-POST body is committed at [`docs/kaizen/scoping/mcp-apps-budget-allocator.tool.json`](../../../docs/kaizen/scoping/mcp-apps-budget-allocator.tool.json). + +Optionally pre-flight discovery (lists the server's tool names so you can confirm reachability/auth before creating the catalog entry): + +```bash +curl -X POST "$APP_API/admin/tools/discover" \ + -H 'Content-Type: application/json' --cookie "$ADMIN_COOKIE" \ + -d '{"serverUrl":"http://localhost:3001/mcp","transport":"streamable-http","authType":"none"}' +``` + +Then create the catalog entry: + +```bash +curl -X POST "$APP_API/admin/tools/" \ + -H 'Content-Type: application/json' --cookie "$ADMIN_COOKIE" \ + -d @docs/kaizen/scoping/mcp-apps-budget-allocator.tool.json +``` + +`authType: none` is only appropriate for an unauthenticated local/dev server. A real deployed MCP-Apps server uses `aws-iam` (Lambda Function URL / API Gateway behind SigV4), `api-key` (+ `secretArn`), or `oauth2` — exactly like any other external MCP tool. After creating the tool, grant it to the relevant App roles (Settings → Tools → Roles, or the `/admin/tools/{id}/roles` endpoints); a freshly created tool is in the catalog but not yet visible to any role. + +### Step 3 — Local-dev environment + +When running the backend locally (App API + Inference API), set in `backend/src/.env` (template: `backend/src/.env.example`): + +| Var | Value | Why | +|-----|-------|-----| +| `AGENTCORE_MCP_APPS_HOST_ENABLED` | `true` | Default; set `false` to opt this env out entirely. | +| `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN` | the deployed `mcp-sandbox.{domain}` origin (SSM `/{prefix}/mcp-sandbox/origin`) | The agent puts this on the `ui_resource` SSE event as `sandboxOrigin`; the SPA frames the App in it. Empty ⇒ no render. There is no local sandbox shell — point at a deployed one. | + +For the local SPA (`http://localhost:4200`) to be allowed to embed that deployed sandbox origin, the sandbox stack must list it in CSP `frame-ancestors` — deploy McpSandbox once with `CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS=http://localhost:4200` (one-off targeted deploy; never commit localhost as a shared CI variable — see *Deploy MCP Sandbox (Optional)* under [What Each Workflow Does](#what-each-workflow-does)). + +### Step 4 — Verify end-to-end + +In a chat, prompt the agent so it invokes the registered tool. You should see, in order: a `tool_use`/`tool_result` card, then the App's iframe rendering inside it (`ui_resource` SSE event carrying a non-empty `sandboxOrigin`). Driving the form should push `ui/notifications/tool-input`, app-initiated buttons should round-trip through `tools/call`, and an `ui/update-model-context` write should be visible to the model on the **next** turn. If the iframe stays blank, the `sandboxOrigin` is almost always empty (prerequisite 1) or the SPA origin isn't in the sandbox `frame-ancestors` (Step 3). + +--- + ## If a Workflow Fails 1. Click into the failed workflow run to see the error logs diff --git a/.github/docs/deploy/step-05-verify.md b/.github/docs/deploy/step-05-verify.md index 5f4ea981..9e3450d0 100644 --- a/.github/docs/deploy/step-05-verify.md +++ b/.github/docs/deploy/step-05-verify.md @@ -72,6 +72,34 @@ The user who completed the first-boot setup is automatically the system admin. > [!TIP] > To add federated identity providers (Entra ID, Okta, Google, etc.), use the admin dashboard's authentication settings. No redeployment is needed. +### 5. (Optional) MCP Apps dogfood — end-to-end + +Run this only if you've enabled the MCP Apps host renderer (MCP Sandbox stack deployed and an MCP-Apps server registered — see [Register an MCP-Apps-capable MCP server](./step-04-deploy.md#register-an-mcp-apps-capable-mcp-server)). It is the manual e2e scenario for the host-renderer initiative and walks every host↔App interaction. Using the `budget-allocator-server` example from the runbook: + +**Setup** +- [ ] `budget-allocator-server` running over Streamable HTTP and registered as an `mcp_external` tool, granted to your role +- [ ] `AGENTCORE_MCP_APPS_HOST_ENABLED=true` (default) and `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN` resolves to the deployed `mcp-sandbox.{domain}` origin +- [ ] Your SPA origin is in the sandbox's CSP `frame-ancestors` + +**Scenario** — in a fresh chat, ask the agent to "help me allocate a budget" (or anything that invokes the tool): + +- [ ] **Resource fetch** — a `tool_use` then `tool_result` card appears; backend logs show a server-side `resources/read` for the tool's `ui://…` resource (no client fetch) +- [ ] **Iframe render** — the App renders *inside* the tool card: a `ui_resource` SSE event arrives with a **non-empty** `sandboxOrigin`, and the iframe is sourced from that origin (not `srcdoc` against the SPA origin) +- [ ] **Tool-input push** — the App shows the arguments the model called it with (host pushed `ui/notifications/tool-input` from the active stream) +- [ ] **App-initiated `tools/call`** — drive the form (move a slider / pick a preset) so the App calls a server tool; the call shows up as its own tool card in the thread *and* the App updates from the `ui/notifications/tool-result` it gets back +- [ ] **`ui/update-model-context` mutates the next turn** — after changing the allocation, send a new chat message that asks about it (e.g. "is my current split reasonable?"); the model's reply reflects the App's latest state — i.e. context written via `ui/update-model-context` was merged into the **next** turn (not the one that opened the App) +- [ ] **`ui/open-link` consent prompt** — trigger a link-open from the App (e.g. an "industry benchmarks" link); an inline consent prompt appears in the message list (modeled on the OAuth-consent prompt) and the link only opens after you approve. (Consent is **frontend-only** — there is no `ui_consent_required` SSE event; don't look for one.) + +
+The App card appears but the iframe is blank + +In order of likelihood: +1. `sandboxOrigin` is empty on the `ui_resource` event → the MCP Sandbox stack isn't deployed, or `CDK_MCP_SANDBOX_ENABLED` wasn't `true` when the Inference API deployed (it consumes `/{prefix}/mcp-sandbox/origin` conditionally). +2. The SPA origin isn't in the sandbox CSP `frame-ancestors` → the browser blocks the frame (console shows a `frame-ancestors` violation). Redeploy McpSandbox with the right origin (`CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS` for localhost). +3. The server didn't return `_meta.ui` on `tools/list`, or its `ui://` resource isn't `text/html;profile=mcp-app` → it isn't actually MCP-Apps-capable; re-check with the discover endpoint and the server's own logs. + +
+ --- ## You're Done! diff --git a/.github/workflows/app-api.yml b/.github/workflows/app-api.yml index c518c286..ab6a216a 100644 --- a/.github/workflows/app-api.yml +++ b/.github/workflows/app-api.yml @@ -273,7 +273,6 @@ jobs: CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} - CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_APP_API_ENABLED: ${{ vars.CDK_APP_API_ENABLED }} CDK_APP_API_CPU: ${{ vars.CDK_APP_API_CPU }} CDK_APP_API_MEMORY: ${{ vars.CDK_APP_API_MEMORY }} @@ -284,6 +283,9 @@ jobs: CDK_FILE_UPLOAD_MAX_SIZE_MB: ${{ vars.CDK_FILE_UPLOAD_MAX_SIZE_MB }} CDK_FINE_TUNING_ENABLED: ${{ vars.CDK_FINE_TUNING_ENABLED }} CDK_FINE_TUNING_DEFAULT_QUOTA_HOURS: ${{ vars.CDK_FINE_TUNING_DEFAULT_QUOTA_HOURS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} @@ -348,6 +350,9 @@ jobs: CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} @@ -412,6 +417,9 @@ jobs: IMAGE_TAG: ${{ needs.build-docker.outputs.image-tag }} CDK_AWS_REGION: ${{ vars.AWS_REGION }} CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} @@ -473,7 +481,6 @@ jobs: CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} - CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_APP_API_ENABLED: ${{ vars.CDK_APP_API_ENABLED }} CDK_APP_API_CPU: ${{ vars.CDK_APP_API_CPU }} CDK_APP_API_MEMORY: ${{ vars.CDK_APP_API_MEMORY }} @@ -484,6 +491,9 @@ jobs: CDK_FILE_UPLOAD_MAX_SIZE_MB: ${{ vars.CDK_FILE_UPLOAD_MAX_SIZE_MB }} CDK_FINE_TUNING_ENABLED: ${{ vars.CDK_FINE_TUNING_ENABLED }} CDK_FINE_TUNING_DEFAULT_QUOTA_HOURS: ${{ vars.CDK_FINE_TUNING_DEFAULT_QUOTA_HOURS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} diff --git a/.github/workflows/artifacts.yml b/.github/workflows/artifacts.yml new file mode 100644 index 00000000..fd280255 --- /dev/null +++ b/.github/workflows/artifacts.yml @@ -0,0 +1,384 @@ +name: "2. Deploy Artifacts (DDB, S3, CloudFront, Lambda)" + +# Provisions the artifact iframe rendering pipeline (DynamoDB metadata table, +# S3 content bucket, CloudFront distribution at artifacts.{domain}, and the +# render Lambda). Gated on CDK_ARTIFACTS_ENABLED — the stack is skipped end- +# to-end when disabled, identical to the SageMaker Fine-Tuning pattern. +# +# Deploy tier 1: depends only on InfrastructureStack (via SSM read of the +# render-token signing-secret ARN). Parallel-safe with RAG Ingestion, +# Gateway, and Fine-Tuning. Must complete before Inference API, App API, +# and Frontend, which read this stack's SSM exports. + +on: + push: + branches: + - main + - develop + paths: + - 'infrastructure/lib/artifacts-stack.ts' + - 'infrastructure/lib/config.ts' + - 'infrastructure/bin/infrastructure.ts' + - 'infrastructure/cdk.json' + - 'infrastructure/cdk.context.json' + - 'infrastructure/package.json' + - 'backend/src/lambdas/artifact_render/**' + - 'scripts/stack-artifacts/**' + - '.github/workflows/artifacts.yml' + pull_request: + branches: + - main + - develop + paths: + - 'infrastructure/lib/artifacts-stack.ts' + - 'infrastructure/lib/config.ts' + - 'infrastructure/bin/infrastructure.ts' + - 'infrastructure/cdk.json' + - 'infrastructure/cdk.context.json' + - 'infrastructure/package.json' + - 'backend/src/lambdas/artifact_render/**' + - 'scripts/stack-artifacts/**' + - '.github/workflows/artifacts.yml' + workflow_dispatch: + inputs: + environment: + description: 'Deployment environment' + required: true + default: 'production' + type: choice + options: + - production + - staging + - development + skip_tests: + description: 'Skip tests' + required: false + default: 'false' + skip_deploy: + description: 'Skip deployment' + required: false + default: 'false' + +permissions: + contents: read + +env: + CDK_REQUIRE_APPROVAL: never + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true + LOAD_ENV_QUIET: true + +concurrency: + group: artifacts-${{ github.ref }} + cancel-in-progress: false + +jobs: + check-stack-dependencies: + name: Check Stack Dependencies + runs-on: ubuntu-24.04 + if: github.event_name != 'pull_request' + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Run stack dependency tests + run: | + bash scripts/common/test-stack-dependencies.sh + + install: + name: Install Dependencies + runs-on: ubuntu-24.04 + if: github.event_name != 'pull_request' + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Install system dependencies + run: | + bash scripts/common/install-deps.sh + + - name: Install CDK dependencies + run: | + bash scripts/stack-artifacts/install.sh + + - name: Save node_modules cache + uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + build: + name: Build CDK Code + runs-on: ubuntu-24.04 + needs: [install, check-stack-dependencies] + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Build CDK + run: | + bash scripts/stack-artifacts/build-cdk.sh + + - name: Save build artifacts cache + uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + synth: + name: Synthesize CloudFormation Template + runs-on: ubuntu-24.04 + needs: build + if: github.event_name != 'pull_request' + outputs: + enabled: ${{ steps.check.outputs.enabled }} + + environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production') || (github.ref == 'refs/heads/develop' && 'development') || 'development' }} + + permissions: + id-token: write + contents: read + + env: + CDK_AWS_REGION: ${{ vars.AWS_REGION }} + CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} + CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} + CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_ARTIFACTS_RETENTION_DAYS: ${{ vars.CDK_ARTIFACTS_RETENTION_DAYS }} + CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS: ${{ vars.CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS }} + CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} + AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + + steps: + - name: Check if artifacts are enabled + id: check + run: | + if [ "${{ vars.CDK_ARTIFACTS_ENABLED }}" = "true" ] || [ "${{ vars.CDK_ARTIFACTS_ENABLED }}" = "1" ]; then + echo "enabled=true" >> "$GITHUB_OUTPUT" + else + echo "enabled=false" >> "$GITHUB_OUTPUT" + echo "Artifacts stack is disabled — skipping synth" + fi + + - name: Checkout code + if: steps.check.outputs.enabled == 'true' + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + if: steps.check.outputs.enabled == 'true' + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Restore build artifacts + if: steps.check.outputs.enabled == 'true' + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + - name: Configure AWS credentials + if: steps.check.outputs.enabled == 'true' + uses: ./.github/actions/configure-aws-credentials + with: + aws-region: ${{ env.CDK_AWS_REGION }} + role-session-name: GitHubActions-Artifacts-Synth + aws-role-arn: ${{ env.AWS_ROLE_ARN }} + aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }} + + - name: Install system dependencies + if: steps.check.outputs.enabled == 'true' + run: | + bash scripts/common/install-deps.sh + + - name: Synthesize CloudFormation template + if: steps.check.outputs.enabled == 'true' + run: | + bash scripts/stack-artifacts/synth.sh + + - name: Upload synthesized templates + if: steps.check.outputs.enabled == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: artifacts-cdk-synth + path: infrastructure/cdk.out/ + retention-days: 7 + + test: + name: Validate CloudFormation Template + runs-on: ubuntu-24.04 + needs: synth + if: ${{ needs.synth.outputs.enabled == 'true' && github.event.inputs.skip_tests != 'true' }} + + environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production') || (github.ref == 'refs/heads/develop' && 'development') || 'development' }} + + permissions: + id-token: write + contents: read + + env: + CDK_AWS_REGION: ${{ vars.AWS_REGION }} + CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} + CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} + CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_ARTIFACTS_RETENTION_DAYS: ${{ vars.CDK_ARTIFACTS_RETENTION_DAYS }} + CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS: ${{ vars.CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS }} + CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} + AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Restore build artifacts + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + - name: Download synthesized templates + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: artifacts-cdk-synth + path: infrastructure/cdk.out/ + + - name: Configure AWS credentials + uses: ./.github/actions/configure-aws-credentials + with: + aws-region: ${{ env.CDK_AWS_REGION }} + role-session-name: GitHubActions-Artifacts-Test + aws-role-arn: ${{ env.AWS_ROLE_ARN }} + aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }} + + - name: Install system dependencies + run: | + bash scripts/common/install-deps.sh + + - name: Run CDK diff to validate template + run: | + bash scripts/stack-artifacts/test-cdk.sh + + deploy: + name: Deploy Artifacts Stack + runs-on: ubuntu-24.04 + needs: [synth, test] + if: | + always() && !cancelled() && + needs.synth.outputs.enabled == 'true' && + !contains(needs.*.result, 'failure') && + (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && + (github.event_name != 'workflow_dispatch' || github.event.inputs.skip_deploy != 'true') + + environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production') || (github.ref == 'refs/heads/develop' && 'development') || 'development' }} + + permissions: + id-token: write + contents: read + + env: + CDK_AWS_REGION: ${{ vars.AWS_REGION }} + CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} + CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} + CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_ARTIFACTS_RETENTION_DAYS: ${{ vars.CDK_ARTIFACTS_RETENTION_DAYS }} + CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS: ${{ vars.CDK_ARTIFACTS_EXTRA_FRAME_ANCESTORS }} + CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} + AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Restore build artifacts + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + - name: Download synthesized templates + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: artifacts-cdk-synth + path: infrastructure/cdk.out/ + + - name: Configure AWS credentials + uses: ./.github/actions/configure-aws-credentials + with: + aws-region: ${{ env.CDK_AWS_REGION }} + role-session-name: GitHubActions-Artifacts-Deploy + aws-role-arn: ${{ env.AWS_ROLE_ARN }} + aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }} + + - name: Install system dependencies + run: | + bash scripts/common/install-deps.sh + + - name: Deploy Artifacts Stack + run: | + bash scripts/stack-artifacts/deploy.sh + + - name: Upload stack outputs + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + if: always() + with: + name: artifacts-outputs + path: infrastructure/artifacts-outputs.json + retention-days: 30 diff --git a/.github/workflows/backup-data.yml b/.github/workflows/backup-data.yml new file mode 100644 index 00000000..bc2c7735 --- /dev/null +++ b/.github/workflows/backup-data.yml @@ -0,0 +1,102 @@ +name: "Backup Data (Pre-Migration)" + +# Manual-dispatch only — this is a one-shot tool intended to be run before +# a destructive infrastructure migration. See scripts/backup-data/README.md. + +on: + workflow_dispatch: + inputs: + project_prefix: + description: "CDK_PROJECT_PREFIX of the environment to back up (e.g. 'boisestate-prod')" + required: true + type: string + aws_region: + description: "AWS region" + required: true + default: "us-west-2" + type: string + aws_environment: + description: "GitHub Environment (selects AWS_ROLE_ARN secret + approvals)" + required: true + default: "production" + type: choice + options: + - production + - staging + - development + include_ephemeral: + description: "Also back up TTL-driven session/state tables" + required: false + default: false + type: boolean + dry_run: + description: "Discover and list sources without performing any writes" + required: false + default: false + type: boolean + allow_partial: + description: "Succeed even if some components fail (manifest still reflects state)" + required: false + default: false + type: boolean + +permissions: + id-token: write # OIDC role assumption + contents: read + +concurrency: + # One backup at a time per environment to avoid bucket-name collisions + # and double-export of the same tables. + group: backup-data-${{ inputs.aws_environment }} + cancel-in-progress: false + +jobs: + backup: + name: Backup ${{ inputs.project_prefix }} + runs-on: ubuntu-24.04 + environment: ${{ inputs.aws_environment }} + timeout-minutes: 360 # large S3 syncs can take a while + + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Configure AWS credentials + uses: ./.github/actions/configure-aws-credentials + with: + aws-region: ${{ inputs.aws_region }} + aws-role-arn: ${{ secrets.AWS_ROLE_ARN }} + role-session-name: "backup-data-${{ inputs.project_prefix }}" + + - name: Install uv + uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.8.0 + with: + version: "0.5.x" + + - name: Set up Python 3.13 + run: uv python install 3.13 + + - name: Install backup tool + working-directory: scripts/backup-data + run: uv sync + + - name: Run backup + working-directory: scripts/backup-data + env: + AWS_REGION: ${{ inputs.aws_region }} + run: | + set -euo pipefail + ARGS=( + --project-prefix "${{ inputs.project_prefix }}" + --region "${{ inputs.aws_region }}" + ) + if [[ "${{ inputs.include_ephemeral }}" == "true" ]]; then + ARGS+=(--include-ephemeral) + fi + if [[ "${{ inputs.dry_run }}" == "true" ]]; then + ARGS+=(--dry-run) + fi + if [[ "${{ inputs.allow_partial }}" == "true" ]]; then + ARGS+=(--allow-partial) + fi + uv run python backup.py "${ARGS[@]}" diff --git a/.github/workflows/frontend.yml b/.github/workflows/frontend.yml index 9b46c12c..e50e65db 100644 --- a/.github/workflows/frontend.yml +++ b/.github/workflows/frontend.yml @@ -235,6 +235,9 @@ jobs: CDK_FRONTEND_ENABLED: ${{ vars.CDK_FRONTEND_ENABLED }} CDK_FRONTEND_CLOUDFRONT_PRICE_CLASS: ${{ vars.CDK_FRONTEND_CLOUDFRONT_PRICE_CLASS }} CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} CDK_CERTIFICATE_ARN: ${{ vars.CDK_CERTIFICATE_ARN }} CDK_FRONTEND_CERTIFICATE_ARN: ${{ vars.CDK_FRONTEND_CERTIFICATE_ARN }} @@ -297,6 +300,9 @@ jobs: # Environment-scoped configuration CDK_AWS_REGION: ${{ vars.AWS_REGION }} CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} @@ -365,6 +371,9 @@ jobs: CDK_FRONTEND_ENABLED: ${{ vars.CDK_FRONTEND_ENABLED }} CDK_FRONTEND_CLOUDFRONT_PRICE_CLASS: ${{ vars.CDK_FRONTEND_CLOUDFRONT_PRICE_CLASS }} CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} CDK_CERTIFICATE_ARN: ${{ vars.CDK_CERTIFICATE_ARN }} CDK_FRONTEND_CERTIFICATE_ARN: ${{ vars.CDK_FRONTEND_CERTIFICATE_ARN }} @@ -438,6 +447,9 @@ jobs: CDK_AWS_REGION: ${{ vars.AWS_REGION }} CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} CDK_PRODUCTION: ${{ vars.CDK_PRODUCTION }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} CDK_FRONTEND_BUCKET_NAME: ${{ vars.CDK_FRONTEND_BUCKET_NAME }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} diff --git a/.github/workflows/inference-api.yml b/.github/workflows/inference-api.yml index 88bdddc8..a210b133 100644 --- a/.github/workflows/inference-api.yml +++ b/.github/workflows/inference-api.yml @@ -295,6 +295,9 @@ jobs: CDK_INFERENCE_API_MAX_CAPACITY: ${{ vars.CDK_INFERENCE_API_MAX_CAPACITY }} ENV_INFERENCE_API_LOG_LEVEL: ${{ vars.ENV_INFERENCE_API_LOG_LEVEL }} CDK_INFERENCE_API_CORS_ORIGINS: ${{ vars.CDK_INFERENCE_API_CORS_ORIGINS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} CDK_APP_VERSION: ${{ needs.build-docker.outputs.app-version }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} @@ -356,6 +359,9 @@ jobs: CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} @@ -422,6 +428,9 @@ jobs: APP_VERSION: ${{ needs.build-docker.outputs.app-version }} CDK_AWS_REGION: ${{ vars.AWS_REGION }} CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} @@ -491,6 +500,9 @@ jobs: CDK_INFERENCE_API_MAX_CAPACITY: ${{ vars.CDK_INFERENCE_API_MAX_CAPACITY }} ENV_INFERENCE_API_LOG_LEVEL: ${{ vars.ENV_INFERENCE_API_LOG_LEVEL }} CDK_INFERENCE_API_CORS_ORIGINS: ${{ vars.CDK_INFERENCE_API_CORS_ORIGINS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} diff --git a/.github/workflows/infrastructure.yml b/.github/workflows/infrastructure.yml index 6ed82f5d..d3a60487 100644 --- a/.github/workflows/infrastructure.yml +++ b/.github/workflows/infrastructure.yml @@ -157,7 +157,6 @@ jobs: CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} - CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_ALB_SUBDOMAIN: ${{ vars.CDK_ALB_SUBDOMAIN }} CDK_CERTIFICATE_ARN: ${{ vars.CDK_CERTIFICATE_ARN }} CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} @@ -167,6 +166,9 @@ jobs: CDK_COGNITO_CALLBACK_URLS: ${{ vars.CDK_COGNITO_CALLBACK_URLS }} CDK_COGNITO_LOGOUT_URLS: ${{ vars.CDK_COGNITO_LOGOUT_URLS }} CDK_COGNITO_SUPPORTED_IDPS: ${{ vars.CDK_COGNITO_SUPPORTED_IDPS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} @@ -235,7 +237,6 @@ jobs: CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} - CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_ALB_SUBDOMAIN: ${{ vars.CDK_ALB_SUBDOMAIN }} CDK_CERTIFICATE_ARN: ${{ vars.CDK_CERTIFICATE_ARN }} CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} @@ -245,6 +246,9 @@ jobs: CDK_COGNITO_CALLBACK_URLS: ${{ vars.CDK_COGNITO_CALLBACK_URLS }} CDK_COGNITO_LOGOUT_URLS: ${{ vars.CDK_COGNITO_LOGOUT_URLS }} CDK_COGNITO_SUPPORTED_IDPS: ${{ vars.CDK_COGNITO_SUPPORTED_IDPS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} @@ -317,7 +321,6 @@ jobs: CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} - CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_ALB_SUBDOMAIN: ${{ vars.CDK_ALB_SUBDOMAIN }} CDK_CERTIFICATE_ARN: ${{ vars.CDK_CERTIFICATE_ARN }} CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} @@ -335,6 +338,9 @@ jobs: CDK_COGNITO_CALLBACK_URLS: ${{ vars.CDK_COGNITO_CALLBACK_URLS }} CDK_COGNITO_LOGOUT_URLS: ${{ vars.CDK_COGNITO_LOGOUT_URLS }} CDK_COGNITO_SUPPORTED_IDPS: ${{ vars.CDK_COGNITO_SUPPORTED_IDPS }} + CDK_ARTIFACTS_ENABLED: ${{ vars.CDK_ARTIFACTS_ENABLED }} + CDK_ARTIFACTS_CERTIFICATE_ARN: ${{ vars.CDK_ARTIFACTS_CERTIFICATE_ARN }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} diff --git a/.github/workflows/mcp-sandbox.yml b/.github/workflows/mcp-sandbox.yml new file mode 100644 index 00000000..af80d449 --- /dev/null +++ b/.github/workflows/mcp-sandbox.yml @@ -0,0 +1,383 @@ +name: "2. Deploy MCP Sandbox (S3, CloudFront, Route53)" + +# Provisions the MCP Apps sandbox-proxy origin: an S3 bucket holding the +# static proxy shell, a CloudFront distribution at mcp-sandbox.{domain} that +# stamps the CSP (frame-ancestors = SPA origin only), and a Route53 record. +# Gated on CDK_MCP_SANDBOX_ENABLED — the stack is skipped end-to-end when +# disabled, identical to the Artifacts / SageMaker Fine-Tuning pattern. +# +# PR #1 of docs/kaizen/scoping/mcp-apps-host-renderer.md. Deploy tier 1: +# reads no cross-stack SSM. Parallel-safe with Artifacts, RAG Ingestion, +# Gateway, and Fine-Tuning. Inert until the SPA wiring (PR #4) and +# MCP_APPS_HOST_ENABLED (PR #7) land — nothing consumes its SSM origin +# export before then. + +on: + push: + branches: + - main + - develop + paths: + - 'infrastructure/lib/mcp-sandbox-stack.ts' + - 'infrastructure/lib/config.ts' + - 'infrastructure/bin/infrastructure.ts' + - 'infrastructure/cdk.json' + - 'infrastructure/cdk.context.json' + - 'infrastructure/package.json' + - 'infrastructure/assets/mcp-sandbox/**' + - 'scripts/stack-mcp-sandbox/**' + - '.github/workflows/mcp-sandbox.yml' + pull_request: + branches: + - main + - develop + paths: + - 'infrastructure/lib/mcp-sandbox-stack.ts' + - 'infrastructure/lib/config.ts' + - 'infrastructure/bin/infrastructure.ts' + - 'infrastructure/cdk.json' + - 'infrastructure/cdk.context.json' + - 'infrastructure/package.json' + - 'infrastructure/assets/mcp-sandbox/**' + - 'scripts/stack-mcp-sandbox/**' + - '.github/workflows/mcp-sandbox.yml' + workflow_dispatch: + inputs: + environment: + description: 'Deployment environment' + required: true + default: 'production' + type: choice + options: + - production + - staging + - development + skip_tests: + description: 'Skip tests' + required: false + default: 'false' + skip_deploy: + description: 'Skip deployment' + required: false + default: 'false' + +permissions: + contents: read + +env: + CDK_REQUIRE_APPROVAL: never + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true + LOAD_ENV_QUIET: true + +concurrency: + group: mcp-sandbox-${{ github.ref }} + cancel-in-progress: false + +jobs: + check-stack-dependencies: + name: Check Stack Dependencies + runs-on: ubuntu-24.04 + if: github.event_name != 'pull_request' + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Run stack dependency tests + run: | + bash scripts/common/test-stack-dependencies.sh + + install: + name: Install Dependencies + runs-on: ubuntu-24.04 + if: github.event_name != 'pull_request' + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Install system dependencies + run: | + bash scripts/common/install-deps.sh + + - name: Install CDK dependencies + run: | + bash scripts/stack-mcp-sandbox/install.sh + + - name: Save node_modules cache + uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + build: + name: Build CDK Code + runs-on: ubuntu-24.04 + needs: [install, check-stack-dependencies] + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Build CDK + run: | + bash scripts/stack-mcp-sandbox/build-cdk.sh + + - name: Save build artifacts cache + uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + synth: + name: Synthesize CloudFormation Template + runs-on: ubuntu-24.04 + needs: build + if: github.event_name != 'pull_request' + outputs: + enabled: ${{ steps.check.outputs.enabled }} + + environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production') || (github.ref == 'refs/heads/develop' && 'development') || 'development' }} + + permissions: + id-token: write + contents: read + + env: + CDK_AWS_REGION: ${{ vars.AWS_REGION }} + CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} + CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} + CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_MCP_SANDBOX_ENABLED: ${{ vars.CDK_MCP_SANDBOX_ENABLED }} + CDK_MCP_SANDBOX_CERTIFICATE_ARN: ${{ vars.CDK_MCP_SANDBOX_CERTIFICATE_ARN }} + CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS: ${{ vars.CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS }} + CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} + AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + + steps: + - name: Check if MCP sandbox is enabled + id: check + run: | + if [ "${{ vars.CDK_MCP_SANDBOX_ENABLED }}" = "true" ] || [ "${{ vars.CDK_MCP_SANDBOX_ENABLED }}" = "1" ]; then + echo "enabled=true" >> "$GITHUB_OUTPUT" + else + echo "enabled=false" >> "$GITHUB_OUTPUT" + echo "MCP Sandbox stack is disabled — skipping synth" + fi + + - name: Checkout code + if: steps.check.outputs.enabled == 'true' + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + if: steps.check.outputs.enabled == 'true' + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Restore build artifacts + if: steps.check.outputs.enabled == 'true' + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + - name: Configure AWS credentials + if: steps.check.outputs.enabled == 'true' + uses: ./.github/actions/configure-aws-credentials + with: + aws-region: ${{ env.CDK_AWS_REGION }} + role-session-name: GitHubActions-McpSandbox-Synth + aws-role-arn: ${{ env.AWS_ROLE_ARN }} + aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }} + + - name: Install system dependencies + if: steps.check.outputs.enabled == 'true' + run: | + bash scripts/common/install-deps.sh + + - name: Synthesize CloudFormation template + if: steps.check.outputs.enabled == 'true' + run: | + bash scripts/stack-mcp-sandbox/synth.sh + + - name: Upload synthesized templates + if: steps.check.outputs.enabled == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: mcp-sandbox-cdk-synth + path: infrastructure/cdk.out/ + retention-days: 7 + + test: + name: Validate CloudFormation Template + runs-on: ubuntu-24.04 + needs: synth + if: ${{ needs.synth.outputs.enabled == 'true' && github.event.inputs.skip_tests != 'true' }} + + environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production') || (github.ref == 'refs/heads/develop' && 'development') || 'development' }} + + permissions: + id-token: write + contents: read + + env: + CDK_AWS_REGION: ${{ vars.AWS_REGION }} + CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} + CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} + CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_MCP_SANDBOX_ENABLED: ${{ vars.CDK_MCP_SANDBOX_ENABLED }} + CDK_MCP_SANDBOX_CERTIFICATE_ARN: ${{ vars.CDK_MCP_SANDBOX_CERTIFICATE_ARN }} + CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS: ${{ vars.CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS }} + CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} + AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Restore build artifacts + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + - name: Download synthesized templates + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: mcp-sandbox-cdk-synth + path: infrastructure/cdk.out/ + + - name: Configure AWS credentials + uses: ./.github/actions/configure-aws-credentials + with: + aws-region: ${{ env.CDK_AWS_REGION }} + role-session-name: GitHubActions-McpSandbox-Test + aws-role-arn: ${{ env.AWS_ROLE_ARN }} + aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }} + + - name: Install system dependencies + run: | + bash scripts/common/install-deps.sh + + - name: Run CDK diff to validate template + run: | + bash scripts/stack-mcp-sandbox/test-cdk.sh + + deploy: + name: Deploy MCP Sandbox Stack + runs-on: ubuntu-24.04 + needs: [synth, test] + if: | + always() && !cancelled() && + needs.synth.outputs.enabled == 'true' && + !contains(needs.*.result, 'failure') && + (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && + (github.event_name != 'workflow_dispatch' || github.event.inputs.skip_deploy != 'true') + + environment: ${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production') || (github.ref == 'refs/heads/develop' && 'development') || 'development' }} + + permissions: + id-token: write + contents: read + + env: + CDK_AWS_REGION: ${{ vars.AWS_REGION }} + CDK_PROJECT_PREFIX: ${{ vars.CDK_PROJECT_PREFIX }} + CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} + CDK_DOMAIN_NAME: ${{ vars.CDK_DOMAIN_NAME }} + CDK_HOSTED_ZONE_DOMAIN: ${{ vars.CDK_HOSTED_ZONE_DOMAIN }} + CDK_CORS_ORIGINS: ${{ vars.CDK_CORS_ORIGINS }} + CDK_RETAIN_DATA_ON_DELETE: ${{ vars.CDK_RETAIN_DATA_ON_DELETE }} + CDK_MCP_SANDBOX_ENABLED: ${{ vars.CDK_MCP_SANDBOX_ENABLED }} + CDK_MCP_SANDBOX_CERTIFICATE_ARN: ${{ vars.CDK_MCP_SANDBOX_CERTIFICATE_ARN }} + CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS: ${{ vars.CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS }} + CDK_AWS_ACCOUNT: ${{ vars.CDK_AWS_ACCOUNT }} + AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Restore node_modules cache + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: infrastructure/node_modules + key: infrastructure-node-modules-${{ hashFiles('infrastructure/package-lock.json') }} + + - name: Restore build artifacts + uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + infrastructure/lib/**/*.js + infrastructure/lib/**/*.d.ts + infrastructure/bin/**/*.js + infrastructure/bin/**/*.d.ts + key: infrastructure-build-${{ github.sha }} + + - name: Download synthesized templates + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: mcp-sandbox-cdk-synth + path: infrastructure/cdk.out/ + + - name: Configure AWS credentials + uses: ./.github/actions/configure-aws-credentials + with: + aws-region: ${{ env.CDK_AWS_REGION }} + role-session-name: GitHubActions-McpSandbox-Deploy + aws-role-arn: ${{ env.AWS_ROLE_ARN }} + aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }} + + - name: Install system dependencies + run: | + bash scripts/common/install-deps.sh + + - name: Deploy MCP Sandbox Stack + run: | + bash scripts/stack-mcp-sandbox/deploy.sh + + - name: Upload stack outputs + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + if: always() + with: + name: mcp-sandbox-outputs + path: infrastructure/mcp-sandbox-outputs.json + retention-days: 30 diff --git a/.github/workflows/nightly-deploy-pipeline.yml b/.github/workflows/nightly-deploy-pipeline.yml index 422dbb05..fbaa1139 100644 --- a/.github/workflows/nightly-deploy-pipeline.yml +++ b/.github/workflows/nightly-deploy-pipeline.yml @@ -431,6 +431,7 @@ jobs: CDK_VPC_CIDR: ${{ vars.CDK_VPC_CIDR }} CDK_DOMAIN_NAME: "" CDK_HOSTED_ZONE_DOMAIN: "" + CDK_CERTIFICATE_ARN: ${{ vars.CDK_CERTIFICATE_ARN }} CDK_FRONTEND_CERTIFICATE_ARN: "" CDK_FRONTEND_BUCKET_NAME: "" CDK_FRONTEND_CLOUDFRONT_PRICE_CLASS: "" diff --git a/.github/workflows/skip-auth-guard.yml b/.github/workflows/skip-auth-guard.yml new file mode 100644 index 00000000..d3fd2b5b --- /dev/null +++ b/.github/workflows/skip-auth-guard.yml @@ -0,0 +1,49 @@ +name: "Guard: SKIP_AUTH must not appear in deployed config" + +# Refuses any PR or push that lets the local-dev SKIP_AUTH=true bypass +# leak into deployed configuration. The bypass itself is implemented in +# backend/src/apis/shared/auth/dependencies.py and gated at boot in +# backend/src/apis/app_api/main.py — both of those legitimately reference +# the string and are excluded from the scan. Anywhere else is a leak. + +permissions: + contents: read + +on: + push: + branches: [main, develop] + pull_request: + branches: [main, develop] + +jobs: + scan: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Scan for SKIP_AUTH=true in deployable config + run: | + set -euo pipefail + # Look in CDK, GitHub Actions, Dockerfiles, and any task definitions. + # Exclude the two files that legitimately reference the variable + # (the bypass implementation + its startup guard) and this workflow. + # Catches `SKIP_AUTH=true`, `SKIP_AUTH: true`, `SKIP_AUTH: "true"`, etc. + # Covers shell, Dockerfile, YAML, and CDK TypeScript object-literal styles. + PATTERN='SKIP_AUTH[[:space:]]*[:=][[:space:]]*["'\'']*true' + MATCHES=$(grep -RInE "$PATTERN" \ + infrastructure/lib \ + infrastructure/bin \ + scripts \ + .github/workflows \ + backend/Dockerfile.app-api \ + backend/Dockerfile.inference-api \ + 2>/dev/null \ + | grep -v '^.github/workflows/skip-auth-guard.yml:' \ + || true) + if [ -n "$MATCHES" ]; then + echo "::error::SKIP_AUTH=true is a local-dev bypass and must never appear in deployed config." + echo "Found in:" + echo "$MATCHES" + exit 1 + fi + echo "OK — no SKIP_AUTH=true in deployable config." diff --git a/.gitignore b/.gitignore index 58ed0a5a..a20a3e3e 100644 --- a/.gitignore +++ b/.gitignore @@ -126,3 +126,11 @@ coverage/ # Local dev scripts start.sh +refresh-aws-sso.ps1 +start.ps1 +test-api-key.py +.kiro/steering/github.md +backend/start_app_api.ps1 +backend/start_inference_api.ps1 +scripts/stack-bootstrap/install.sh +test-accounts diff --git a/.kiro/specs/bff-middleware-event-loop-blocking/.config.kiro b/.kiro/specs/bff-middleware-event-loop-blocking/.config.kiro new file mode 100644 index 00000000..d4b7813d --- /dev/null +++ b/.kiro/specs/bff-middleware-event-loop-blocking/.config.kiro @@ -0,0 +1 @@ +{"specId": "075212d4-ee53-4e7a-bc6d-9d99dacb7cef", "workflowType": "requirements-first", "specType": "bugfix"} diff --git a/.kiro/specs/bff-middleware-event-loop-blocking/bugfix.md b/.kiro/specs/bff-middleware-event-loop-blocking/bugfix.md new file mode 100644 index 00000000..4e345403 --- /dev/null +++ b/.kiro/specs/bff-middleware-event-loop-blocking/bugfix.md @@ -0,0 +1,78 @@ +# Bugfix Requirements Document + +## Introduction + +Since the `v1.0.0-beta.24` BFF Token Handler release (commit `258193d`, deployed 2026-05-06), the app-api service has exhibited severe tail-latency and ingress stalls on page loads. Angular's refresh fan-out (~8 concurrent endpoints — `/auth/session`, `/models`, `/tools/`, `/files/quota`, `/users/me/permissions`, `/sessions`, `/assistants`, `/connectors/`) sees requests time out or exceed the ALB 60s idle cap. Observed signals over the last 24h on the affected fleet: + +- Two `HTTPCode_ELB_504_Count` events (13:37 and 14:40 UTC) — driven by ALB idle timeout, not target 5xx. +- `TargetResponseTime` p-max of 15.6s at 15:25 UTC; `/files/quota` outliers reaching ~80s. +- Endpoint p95s under load: `/models` ~141ms, `/tools/` ~289ms, `/users/me/permissions` ~239ms, `/sessions` ~188ms. +- ECS task at 0.7% CPU / 23% memory. No DDB throttling (0 `ReadThrottleEvents` / `WriteThrottleEvents` across all 23 tables). Zero target 5xx. + +The new `SessionRefreshMiddleware` runs on every request carrying a `__Host-bff_session` cookie. Its hot-path calls block the single-worker uvicorn event loop on synchronous boto3 I/O (DynamoDB + Cognito), its cache TTL and its sliding-renewal throttle are aligned on the same 60s boundary, and the per-session lock that coalesces refreshes does not coalesce the broader session-resolve path. The result is ~16 serialized blocking AWS calls at the front of every page load per active user, once per minute — with no concurrency slack because the service runs one uvicorn worker in one ECS task. + +Impact: degraded UX for every logged-in user (spinners, stale data, failed tab refreshes), 504s visible to users, and a fragile service posture where any single slow AWS call stalls every in-flight request on the same task. + +## Bug Analysis + +### Current Behavior (Defect) + +What currently happens under the new middleware on cookie-bearing requests: + +1.1 WHEN `SessionRepository.get`, `touch_last_seen`, `update_tokens`, `put`, or `delete` is awaited inside a request handler THEN the uvicorn event loop blocks for the full DynamoDB round-trip because the methods are declared `async def` but call boto3 synchronously with no `asyncio.to_thread` offload and no aioboto3. + +1.2 WHEN `SessionRefreshMiddleware._resolve_session` invokes `CognitoRefreshClient.refresh` THEN the uvicorn event loop blocks for the full `cognito-idp:initiate_auth` round-trip because `CognitoRefreshClient.refresh` is a plain `def` called without threadpool offload, and it runs while the per-session `asyncio.Lock` from `get_session_lock()` is held. + +1.3 WHEN N concurrent requests for the same `session_id` arrive with no valid cached `SessionRecord` THEN the middleware issues N independent DynamoDB `get_item` calls because the existing per-session lock only coalesces the refresh exchange, not the upstream unseal → `SessionCache.get` → `SessionRepository.get` sequence. + +1.4 WHEN the `SessionCache` TTL elapses at the same moment the sliding-renewal throttle window elapses THEN a single request triggers both a DynamoDB `get_item` and a DynamoDB `update_item` on its critical path because `_DEFAULT_REFRESH_LEEWAY_SECONDS` and `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` are both `60` in `sessions_bff/config.py`, so cache expiry and throttle expiry are aligned. + +1.5 WHEN a request passes `SessionRefreshMiddleware` and a slide is warranted THEN the caller's response waits for `touch_last_seen` to complete against DynamoDB because `_maybe_slide` `await`s the write inline on the request path, even though the code is documented to swallow failures ("Don't fail the request if the slide-write fails"). + +1.6 WHEN the app-api container starts THEN the service has no concurrency slack because the `CMD` in `backend/Dockerfile.app-api` launches a single uvicorn worker with no `--workers` flag and the service runs as a single ECS task — one blocked event loop stalls all ingress, consistent with low CPU/memory while latency climbs. + +1.7 WHEN Angular fires its ~8-endpoint page-load fan-out with a session cookie and the per-session cache window has just elapsed THEN ~16 serialized blocking DynamoDB operations (per-coroutine `get_item` plus per-coroutine `update_item`) accumulate at the front of the page load because each coroutine independently observes cache-miss and past-throttle on its local `SessionRecord` copy and each runs its own blocking AWS I/O on the event loop thread. + +### Expected Behavior (Correct) + +What should happen instead, keeping the same middleware surface and contracts: + +2.1 WHEN `SessionRepository.get`, `touch_last_seen`, `update_tokens`, `put`, or `delete` is awaited inside a request handler THEN the method SHALL execute its underlying boto3 call in a threadpool (via `asyncio.to_thread` or an equivalent offload) so the uvicorn event loop continues scheduling other coroutines for the full DynamoDB round-trip. + +2.2 WHEN `SessionRefreshMiddleware._resolve_session` invokes `CognitoRefreshClient.refresh` THEN the Cognito `initiate_auth` call SHALL execute in a threadpool so the uvicorn event loop continues scheduling other coroutines — including coroutines for different `session_id`s — while the per-session `asyncio.Lock` is held. + +2.3 WHEN N concurrent requests for the same `session_id` arrive with no valid cached `SessionRecord` THEN the middleware SHALL coalesce them so at most one DynamoDB `get_item` is issued per `session_id` per cache window; the remaining N−1 requests SHALL await a shared in-process future keyed by `session_id` and consume its result. + +2.4 WHEN the `SessionCache` TTL elapses THEN a cache miss SHALL NOT imply a sliding-renewal DynamoDB write on the same request because `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` SHALL be a strict multiple of `_DEFAULT_REFRESH_LEEWAY_SECONDS` (e.g. 300s versus 60s) so cache-expiry and throttle-expiry are de-aligned. + +2.5 WHEN a request passes `SessionRefreshMiddleware` and a slide is warranted THEN the caller's response SHALL NOT wait for `touch_last_seen` because `_maybe_slide` SHALL dispatch the DynamoDB write as a detached `asyncio.Task` (fire-and-forget) and SHALL return the computed `Max-Age` to the response path immediately. + +2.6 WHEN the app-api container starts THEN the service SHALL have concurrency slack such that a single blocked event loop does not stall all ingress — uvicorn SHALL run with `--workers >= 2` (at current `cpu=512`) and/or the ECS service SHALL run `>= 2` tasks; the chosen configuration SHALL be deployed. + +2.7 WHEN Angular fires its ~8-endpoint page-load fan-out with a session cookie and the per-session cache window has just elapsed THEN the middleware SHALL issue at most 1 DynamoDB `get_item` and at most 1 DynamoDB `update_item` per `session_id` per cache window across the fan-out (not ~16 blocking calls), and all 8 responses SHALL return without any one of them serially waiting on another's AWS I/O to complete. + +### Unchanged Behavior (Regression Prevention) + +Existing guarantees that MUST continue to hold after the fix: + +3.1 WHEN `BFFConfig.is_enabled()` returns `False` (env vars unset) THEN `SessionRefreshMiddleware` SHALL CONTINUE TO short-circuit as a pass-through with no AWS calls (dormant pass-through invariant preserved). + +3.2 WHEN a request arrives without a `__Host-bff_session` cookie THEN `SessionRefreshMiddleware` SHALL CONTINUE TO pass the request through without unsealing, cache lookup, or DynamoDB access. + +3.3 WHEN an unrecoverable cookie is detected (bad seal, missing DynamoDB row, expired TTL, or terminal `CognitoRefreshError`) THEN the middleware SHALL CONTINUE TO clear both `__Host-bff_session` and `__Host-bff_csrf` on the response. + +3.4 WHEN `_maybe_slide` returns a non-`None` `Max-Age` THEN the response's `Set-Cookie` headers for `__Host-bff_session` and `__Host-bff_csrf` SHALL CONTINUE TO use that exact value (the cookie re-emit contract between `_maybe_slide` and `_reemit_cookies` is preserved under fire-and-forget dispatch of the DynamoDB write). + +3.5 WHEN N concurrent requests for the same `session_id` cross the refresh-leeway window at the same moment THEN exactly one `cognito-idp:initiate_auth` exchange SHALL CONTINUE TO be issued per `session_id` per leeway window (the existing refresh-storm coalescing via `get_session_lock(session_id)` is preserved end-to-end). + +3.6 WHEN `CookieCodec._ensure_cipher` is called on a hot request THEN the AES-GCM cipher SHALL CONTINUE TO be served from the process-wide `get_default_codec()` singleton with no per-request `kms:GenerateDataKey` call. + +3.7 WHEN `resolve_bff_client_secret` is called on a hot request THEN the BFF Cognito app-client secret SHALL CONTINUE TO be served from the module-scope cache with no per-request `secretsmanager:GetSecretValue` call. + +3.8 WHEN `CSRFMiddleware` validates an unsafe-method request with `request.state.bff_session` set THEN it SHALL CONTINUE TO accept/reject using the existing in-memory HMAC double-submit check with no new I/O introduced on its path. + +3.9 WHEN the absolute-lifetime cap (`created_at + absolute_lifetime_seconds`) has passed THEN `_maybe_slide` SHALL CONTINUE TO return `None` so no further cookie re-emission or DynamoDB slide is issued. + +3.10 WHEN a refresh rotates the Cognito refresh token and the `update_tokens` persist fails THEN the middleware SHALL CONTINUE TO invalidate the cache entry and clear the cookie so the user is forced to re-authenticate (fail-closed rotation behavior preserved). + +3.11 WHEN the BFF cookie seal fails to decode THEN the middleware SHALL CONTINUE TO treat every decode failure identically (no timing or response-shape oracle introduced by the new offload or single-flight paths). diff --git a/.kiro/specs/bff-middleware-event-loop-blocking/code-review-report.md b/.kiro/specs/bff-middleware-event-loop-blocking/code-review-report.md new file mode 100644 index 00000000..f30360f5 --- /dev/null +++ b/.kiro/specs/bff-middleware-event-loop-blocking/code-review-report.md @@ -0,0 +1,245 @@ +# Code Review Report: BFF Middleware Event-Loop Blocking Bugfix + +**Branch**: `fix/bff-middleware-event-loop-blocking` +**PR**: [#264](https://github.com/Boise-State-Development/agentcore-public-stack/pull/264) +**Commits reviewed**: +- `db3d2e06` — Initial fix (tasks 3.1–3.7) +- `dd91d6fd` — Test polling adjustment +- `78891e2e` — Strong-reference fix for fire-and-forget tasks + +This report reviews each technical decision in the bugfix against authoritative external sources (Python docs, AWS docs, canonical patterns from the Python ecosystem) to demonstrate the approach was sound. Where my initial implementation missed a production nuance, I flag it and cite the source that caught it. + +--- + +## 1. Offloading sync boto3 to threads via `asyncio.to_thread` + +**Change**: `SessionRepository.{get,put,update_tokens,touch_last_seen,delete}` and `CognitoRefreshClient.refresh` now wrap their boto3 calls in `await asyncio.to_thread(...)`. + +**Why this is correct**: + +The official Python documentation for [`asyncio.to_thread()`](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread) describes it as: + +> This coroutine function is primarily intended to be used for executing IO-bound functions/methods that would otherwise block the event loop if they were run in the main thread. + +The docs state explicitly that `asyncio.to_thread` is the idiomatic solution for IO-bound blocking work — which is exactly what boto3's synchronous HTTP calls to DynamoDB and Cognito are. They also note: + +> Due to the GIL, asyncio.to_thread() can typically only be used to make IO-bound functions non-blocking. + +boto3 is a well-known offender in this exact scenario. [Stack Overflow](https://stackoverflow.com/questions/72092993/i-want-to-use-boto3-in-async-function-python) recommends two options for using boto3 in async code: (a) use `aioboto3`/`aiobotocore`, or (b) wrap boto3 in `asyncio.to_thread`/`loop.run_in_executor`. Both are valid; `to_thread` is the lower-friction choice because it doesn't introduce a new async SDK with a different API surface. + +The existing codebase had a documented awareness of this gap — the `SessionRepository` docstring before the fix acknowledged that boto3 runs on the event loop thread. The fix simply closes that gap without reshaping the API. + +**Alternative considered (not taken)**: Replacing boto3 with [`aioboto3`](https://pypi.org/project/aioboto3/). Rejected because: (a) adds a new dependency, (b) changes method signatures across the repository (e.g. `async with table.get_item(...)` vs `table.get_item(...)`), (c) the per-method offload is a surgical change with no ripple effect on callers. The spec explicitly called for "targeted, minimal-surface intervention that keeps the middleware's public contracts intact." + +**Verdict**: ✅ Correct approach, supported by official Python docs. + +--- + +## 2. Per-session single-flight via `asyncio.Future` + +**Change**: New `backend/src/apis/shared/sessions_bff/single_flight.py` exports `async def resolve_once(session_id, loader_coro_factory)`. The first caller per `session_id` creates a Future, runs the loader, sets the result; concurrent callers await the same Future. + +**Why this is correct**: + +This is the canonical **request coalescing** / **single-flight** pattern. The Python ecosystem recognizes it as the standard solution for N-concurrent-callers-one-backend-hit. From [OneUptime's "How to Reduce DB Load with Request Coalescing in Python"](https://oneuptime.com/blog/post/2026-01-23-request-coalescing-python/view): + +> Request coalescing, also known as request deduplication or single-flighting, is a technique where concurrent requests for the same resource are merged into a single backend call. +> +> _(paraphrased for licensing compliance)_ + +And from [SystemDesignSandbox](https://www.systemdesignsandbox.com/learn/hot-key-cache-stampede), "request coalescing" is listed as a textbook solution to fan-out amplification on hot keys / concurrent cache misses. + +The name comes from Go's `golang.org/x/sync/singleflight` package, which is the reference implementation of this pattern. Python's `asyncio.Future` is the natural primitive for it: multiple coroutines can `await` the same Future, and setting the result/exception wakes all of them. + +**Why a Future and not an `asyncio.Lock`**: The existing `get_session_lock(session_id)` in `lock.py` already serializes the Cognito refresh exchange. A lock would serialize the fan-out (N callers run sequentially through one DDB call), but we want to **coalesce** it (N callers share one result). A Future is the right primitive for coalescing. The design doc called this out: + +> The fix needs a different primitive — an `asyncio.Future` stored in a per-session slot that N waiters can await — because a lock would serialize N requests through one DDB call instead of consolidating them to one call. + +**Implementation notes**: +- The registry is a plain `dict` guarded by a `threading.Lock` with double-checked locking — mirrors the pattern in `lock.py` which is already approved by the team. +- Leader always removes the entry in a `try/except/finally` pattern so a failed loader doesn't sticky-cache. +- Exceptions propagate to all waiters via `future.set_exception(exc)`; the leader additionally calls `future.exception()` to silence the "Future exception was never retrieved" warning if no follower attached. + +**Verdict**: ✅ Canonical pattern, implemented against Python's standard asyncio primitives. + +--- + +## 3. De-aligning cache TTL and slide-throttle windows + +**Change**: `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` raised from 60s to 300s while `_DEFAULT_REFRESH_LEEWAY_SECONDS` stays at 60s. + +**Why this is correct**: + +Aligned TTL boundaries are the textbook cause of **cache stampede / thundering herd**. Multiple sources document this: + +- [Redis (antirez) on cache stampedes](https://redis.antirez.com/fundamental/cache-stampede-prevention.html): a popular cache key expiring causes many concurrent requests to regenerate it, overwhelming the backend. +- [Aman Maharshi, "Cache Stampede: Solving the Thundering Herd Problem"](https://www.amanmaharshi.com/blog/cache-stampede): "Synchronized Expiration" — caching N items at once with one TTL causes them all to expire at the same second, creating a spike. +- [softwarepatternslexicon.com "Thundering Herds and Backend Pressure"](https://softwarepatternslexicon.com/caching-patterns-and-invalidation/consistency-and-stampede-control/thundering-herds-backend-pressure/): "A synchronized TTL boundary... can create a wave of misses that ripples into databases." + +Our case was a miniature version of this: whenever `SessionCache` TTL (60s) elapsed at the same moment as the slide-throttle window (60s), a single request paid **both** a `get_item` AND an `update_item` on its critical path. Making the throttle a strict multiple (300s, 5× the leeway) guarantees that a cache miss at boundary T will never coincide with a slide-throttle expiry at the same T — by construction, the slide throttle expiry is at T + offset where `offset != 0 mod 60`. + +**Why 300s and not some other value**: The design doc explicitly says "strict multiple of refresh leeway (e.g. 300s vs 60s)". 300s is 5× 60s. The key property is that `throttle % leeway == 0` AND `throttle > leeway` — the multiplier could be 2, 5, 10, etc. 5× was chosen because it matches industry practice of caching session metadata for minutes, not seconds. + +**Related patterns we didn't need but recognized**: TTL jitter (randomizing per-key expiry) is another standard mitigation. We don't need it because we only have one key class (sessions) and the single-flight already coalesces; jitter would add complexity without bounded benefit. + +**Verdict**: ✅ Direct application of a well-documented cache-stampede prevention technique. + +--- + +## 4. Fire-and-forget slide-write via `asyncio.create_task` + +**Change**: `_maybe_slide` now dispatches `touch_last_seen` as a detached task rather than awaiting it inline. + +**Why the approach is correct**: + +The inline `await` was causing the response path to wait on a DDB round-trip for a write that was already documented to swallow failures — i.e. the response didn't actually need the write to complete. That's the textbook scenario for fire-and-forget. + +**What I got wrong initially**: I wrote `asyncio.create_task(self._slide_write_task(...))` without holding a reference to the returned Task. This is a **known dangerous anti-pattern**. The [Python docs for `asyncio.create_task`](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task) contain this explicit warning: + +> **Important** +> +> Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn't referenced elsewhere may get garbage collected at any time, even before it's done. +> +> For reliable "fire-and-forget" background tasks, gather them in a collection: +> +> ```python +> background_tasks = set() +> for i in range(10): +> task = asyncio.create_task(some_coro(param=i)) +> background_tasks.add(task) +> task.add_done_callback(background_tasks.discard) +> ``` + +The fix in commit `78891e2e` applies this exact pattern: `self._slide_tasks: set[asyncio.Task]` on the middleware instance, with `task.add_done_callback(self._slide_tasks.discard)` to prevent the set from leaking. + +**Multiple external sources reinforce this**: +- [SuperFastPython, "Asyncio Disappearing Task Bug"](http://superfastpython.com/asyncio-disappearing-task-bug/): "Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks." +- [Michael Kennedy, "Fire and forget (or never) with Python's asyncio"](https://mkennedy.codes/posts/fire-and-forget-or-never-with-python-s-asyncio/): "create_task() can silently garbage collect your fire-and-forget tasks starting in Python 3.12 — they may never run. The fix: store task references in a set and register a done_callback to clean them up." +- [Ruff's `RUF006` lint rule ("asyncio-dangling-task")](https://docs.astral.sh/ruff/rules/asyncio-dangling-task/) flags exactly this anti-pattern automatically. +- [Runebook, "Replacing Low-Level Task Registration"](http://runebook.dev/en/docs/python/library/asyncio-extending/asyncio._register_task): describes the weak-reference behavior and the risk of collection mid-execution. + +**Why the bug surfaced only on CI**: Python 3.12 made garbage collection more aggressive. On my local Python 3.13 (different GC tuning, different scheduler timing), the task usually completed before GC ran. On CI's Python 3.12 runners, the GC occasionally collected the task first, causing a missing `update_item`. Hypothesis caught it as `FlakyFailure` — failed once, passed on retry — which is the signature of exactly this kind of race. + +**Verdict**: ✅ Fire-and-forget is the right approach; ❌ my initial implementation had a canonical asyncio bug; ✅ the fix matches the Python docs' recommended pattern verbatim. + +--- + +## 5. ECS `desiredCount` raised from 1 to 2 + +**Change**: `infrastructure/cdk.context.json` `appApi.desiredCount: 1 → 2` in the production context. + +**Why this is correct**: + +The issue was a single point of failure at the deployment layer: one ECS task running one uvicorn worker means any slow AWS call on that task's event loop halts every in-flight request. AWS's own [ECS availability best practices](https://aws.amazon.com/blogs/containers/amazon-ecs-availability-best-practices/) document explicitly recommends multi-task deployments for availability. + +Independently from the event-loop issue, single-task services fail basic availability requirements: if the one task crashes, restarts, or becomes unreachable, the service has zero capacity until a replacement boots — which for Fargate is tens of seconds to minutes. Two tasks means rolling restarts always keep one healthy instance serving traffic. + +This change is belt-and-suspenders: even if the event-loop-blocking fix is 100% correct, running `desiredCount: 1` would still be a latent availability liability. Raising to 2 gives us: +1. Concurrency slack so a single stuck loop can't halt all ingress (primary rationale). +2. Rolling deploy safety (automatic secondary benefit). +3. Resilience to a single task's AZ failure (automatic tertiary benefit). + +`maxCapacity` stays at 10 so auto-scaling can still burst upward under load. + +**Verdict**: ✅ Standard AWS multi-task posture, with a specific and documented trigger in the bug analysis. + +--- + +## 6. Lock scope preservation (existing `get_session_lock`) + +**Change**: None — the `async with get_session_lock(session_id)` scope around the Cognito refresh exchange is deliberately preserved exactly as it was. + +**Why this is correct**: + +The existing lock exists for a specific purpose: the Cognito refresh-token rotation flow invalidates the previous refresh token as soon as a new one is issued. If N concurrent requests all call `initiate_auth` with the same refresh token, only the first succeeds; the rest receive the token-rotated-out error and have to be failed or retried. Serializing the exchange with a per-session lock prevents this race. + +The new single-flight primitive sits **upstream** of this lock — it coalesces the resolve path (cache, repo.get, needs_refresh decision) so typically only the leader ever reaches the Cognito refresh at all. But in the edge case where the leader decides refresh is NOT needed but a follower does (race with TTL expiry), the existing lock is still needed as a defense-in-depth. The design doc was explicit about not moving or widening the lock. + +The preservation test `test_3_5_refresh_storm_coalesces_to_single_initiate_auth` verifies that exactly one `cognito-idp:initiate_auth` fires per 10 concurrent same-session requests — which is the original contract, preserved end-to-end. + +**Verdict**: ✅ Correctly preserved. The contract the existing lock was enforcing continues to hold. + +--- + +## 7. Testing approach + +**Property-Based Tests over scenario-based tests**: Used `hypothesis` for: +- Sub-conditions that generalize over a domain (fan-out size, request shapes across the non-buggy input domain). +- Preservation properties that must hold "for all" inputs meeting certain criteria. + +This is the approach the project's Kiro spec workflow calls for (Property-Based Testing Integration section). Property-based testing for preservation invariants is particularly strong because it catches edge cases in the fix (single-flight exception paths, background task races, Set-Cookie attribute sets) that scenario tests would miss. + +**Bug Condition exploration test FAILS on unfixed code, PASSES on fixed code**: This is the core methodology of the bugfix workflow — the test serves as the executable specification. 10 of 12 sub-conditions failed on unfixed code (proving the bug); all 12 pass after the fix. + +**What the tests caught that scenario tests would have missed**: +- Hypothesis's `FlakyFailure` detection caught the `asyncio.create_task` GC race on CI — a scenario test at a fixed seed likely wouldn't have reproduced it at all. + +**Verdict**: ✅ Correct methodology; the tests caught a real bug I introduced. + +--- + +## 8. What I did well + +1. **Read before writing**: traced the full middleware path, repository, lock, and config before proposing changes. +2. **Preservation-first**: wrote the preservation test suite on unfixed code before implementing any fix, so regressions surface immediately. +3. **Separate primitive for separate concern**: new `single_flight.py` module instead of overloading `lock.py` — keeps each primitive's contract clear. +4. **Minimal-surface interventions**: no new async SDK, no public API changes, no lock-scope shift. + +## 9. What I got wrong (and corrected) + +1. **Missed the `asyncio.create_task` strong-reference requirement** on the first pass. The Python docs warn about this in bold, Ruff has a lint rule for it, and multiple blog posts cover it. This is directly traceable to me not running the full CI script locally before pushing — my local Python 3.13 GC didn't hit the race. +2. **Initial CI fix was a band-aid** (polling on the test side) rather than a root-cause fix (strong reference in the middleware). The polling remains as defensive depth but the real fix is the set-based reference in commit `78891e2e`. + +## 10. Root cause summary + +The fix addresses four independent but correlated defects in `SessionRefreshMiddleware`, each with a canonical industry solution: + +| Defect | Canonical fix | Authority | +|---|---|---| +| Sync boto3 blocks event loop | `asyncio.to_thread` | [Python docs](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread) | +| N concurrent same-session → N DDB calls | Single-flight / request coalescing via `asyncio.Future` | [OneUptime](https://oneuptime.com/blog/post/2026-01-23-request-coalescing-python/view), Go's `singleflight` | +| Aligned TTL = cache stampede | De-align boundaries (strict multiple) | [Redis on cache stampedes](https://redis.antirez.com/fundamental/cache-stampede-prevention.html), [softwarepatternslexicon.com](https://softwarepatternslexicon.com/caching-patterns-and-invalidation/consistency-and-stampede-control/thundering-herds-backend-pressure/) | +| Response waits on non-critical DDB write | Fire-and-forget task with strong reference | [Python docs on `asyncio.create_task`](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task) | +| Single ECS task = no concurrency slack | `desiredCount >= 2` | [AWS ECS availability best practices](https://aws.amazon.com/blogs/containers/amazon-ecs-availability-best-practices/) | + +Each fix is directly traceable to a published authority. The overall shape — coalesce upstream, offload sync I/O to threads, dispatch non-critical writes asynchronously, stagger TTLs, add replica slack — is the standard stack of techniques for keeping an ASGI service's event loop free under concurrent load. + +## 11. Verification status + +- **Local**: `scripts/stack-app-api/test.sh` and `scripts/stack-inference-api/test.sh` both pass with 2459 tests inside the `agentcore-dev` container. +- **Bug condition exploration suite**: 12/12 pass on fixed code (0/12 passed before fix). +- **Preservation suite**: 19/19 pass on both unfixed and fixed code (baseline intact). +- **Single-flight primitive unit tests**: 6/6 pass. +- **CDK unit tests**: 25/25 pass for `app-api-stack` including new production-context `DesiredCount: 2` assertion. +- **CI PR #264**: pushed commit `78891e2e` with the strong-reference fix; awaiting CI verification. + +--- + +## Sources consulted + +Primary: +- [Python 3 docs: `asyncio.to_thread`](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread) +- [Python 3 docs: `asyncio.create_task` (Important: Save a reference...)](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task) + +Supporting (asyncio task lifecycle): +- [SuperFastPython: Asyncio Disappearing Task Bug](http://superfastpython.com/asyncio-disappearing-task-bug/) +- [Michael Kennedy: Fire and forget (or never) with Python's asyncio](https://mkennedy.codes/posts/fire-and-forget-or-never-with-python-s-asyncio/) +- [Ruff RUF006: asyncio-dangling-task](https://docs.astral.sh/ruff/rules/asyncio-dangling-task/) + +Supporting (boto3 + async): +- [Stack Overflow: I want to use boto3 in async function, python](https://stackoverflow.com/questions/72092993/i-want-to-use-boto3-in-async-function-python) +- [aioboto3 on PyPI](https://pypi.org/project/aioboto3/) — considered and rejected as too invasive + +Supporting (cache stampede / thundering herd): +- [Redis on cache stampede prevention](https://redis.antirez.com/fundamental/cache-stampede-prevention.html) +- [softwarepatternslexicon.com: Thundering Herds and Backend Pressure](https://softwarepatternslexicon.com/caching-patterns-and-invalidation/consistency-and-stampede-control/thundering-herds-backend-pressure/) +- [Aman Maharshi: Cache Stampede: Solving the Thundering Herd Problem](https://www.amanmaharshi.com/blog/cache-stampede) + +Supporting (request coalescing): +- [OneUptime: How to Reduce DB Load with Request Coalescing in Python](https://oneuptime.com/blog/post/2026-01-23-request-coalescing-python/view) +- [SystemDesignSandbox: Hot Keys and Cache Stampedes](https://www.systemdesignsandbox.com/learn/hot-key-cache-stampede) + +Supporting (ECS availability): +- [AWS ECS availability best practices](https://aws.amazon.com/blogs/containers/amazon-ecs-availability-best-practices/) + +Content was paraphrased for compliance with licensing restrictions; verbatim quotes are limited to short excerpts attributed inline. diff --git a/.kiro/specs/bff-middleware-event-loop-blocking/design.md b/.kiro/specs/bff-middleware-event-loop-blocking/design.md new file mode 100644 index 00000000..b5e99a44 --- /dev/null +++ b/.kiro/specs/bff-middleware-event-loop-blocking/design.md @@ -0,0 +1,380 @@ +# BFF Middleware Event Loop Blocking Bugfix Design + +## Overview + +The `SessionRefreshMiddleware` runs on every cookie-bearing request and, as of `v1.0.0-beta.24`, executes four independent classes of blocking/serialized work on the uvicorn event loop: + +1. **Sync boto3 I/O on the event loop thread** — `SessionRepository.*` and `CognitoRefreshClient.refresh` are declared `async def` but call boto3 synchronously. Every DynamoDB `get_item`/`update_item` and every Cognito `initiate_auth` freezes the whole event loop for its round-trip duration. +2. **Missing fan-out coalescing** — the per-session `asyncio.Lock` wraps only the refresh exchange. The upstream `unseal → cache → get_item → maybe_slide` path is not coalesced, so Angular's ~8-endpoint page-load fan-out produces ~16 serialized blocking DDB calls per cache window. +3. **Aligned cache TTL / throttle window** — `_DEFAULT_REFRESH_LEEWAY_SECONDS` and `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` both default to 60s. Cache expiry and slide-throttle expiry land on the same boundary, so a single request crossing that boundary incurs both a `get_item` and an `update_item` on its critical path. +4. **Inline awaited slide-write** — `_maybe_slide` awaits `touch_last_seen` on the request path even though the call is already written defensively (failures are swallowed). The caller's response waits on DDB. + +All of this runs inside a **single uvicorn worker on a single ECS task** (no `--workers` flag in `backend/Dockerfile.app-api`, `desiredCount: 1` in CDK), so any one blocked round-trip stalls every other in-flight request. + +The fix is a targeted, minimal-surface intervention that keeps the middleware's public contracts intact: + +- Offload every synchronous boto3 call in `SessionRepository` and `CognitoRefreshClient.refresh` via `asyncio.to_thread`. +- Introduce a per-session `asyncio.Future`-based single-flight in front of the `get_item → needs_refresh → maybe-refresh` path so N concurrent requests for the same `session_id` share one lookup result. +- De-align `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` from the cache/leeway window (raise to 300s) so cache-miss does not imply slide-write. +- Dispatch `_maybe_slide`'s `touch_last_seen` as a detached `asyncio.Task` and return the `Max-Age` synchronously. +- Add concurrency slack at the deployment layer (raise `CDK_APP_API_DESIRED_COUNT` to ≥ 2 for production config, keeping 1 valid for dev) so a single stuck event loop can no longer halt all ingress. + +## Glossary + +- **Bug_Condition (C)**: The condition that triggers the bug — a cookie-bearing request reaches `SessionRefreshMiddleware` while the middleware is active (`BFFConfig.is_enabled()` is True), under any of the sub-conditions 1.1–1.7 in `bugfix.md#Current Behavior`. +- **Property (P)**: The desired behavior when the bug condition holds — AWS I/O never freezes the uvicorn event loop, fan-outs share a single coalesced lookup, and slide-writes never block the response path. +- **Preservation**: Existing contracts that must remain unchanged — dormant pass-through (`is_enabled() == False`), no-cookie pass-through, unrecoverable-cookie clearing, refresh-storm coalescing, Max-Age re-emit contract, CSRF unchanged, absolute-lifetime cap, fail-closed rotation, uniform cookie decode failure. +- **SessionRefreshMiddleware**: The middleware in `backend/src/apis/shared/middleware/session_refresh.py` that unseals the BFF cookie, resolves the `SessionRecord`, optionally refreshes Cognito tokens, and slides the session's DDB TTL. +- **SessionRepository**: The repository in `backend/src/apis/shared/sessions_bff/repository.py` that wraps boto3 DynamoDB calls with `async def` signatures. Today the methods call boto3 synchronously on the event loop thread. +- **CognitoRefreshClient**: The class in `backend/src/apis/shared/sessions_bff/refresh.py` whose `refresh()` method is plain `def` and calls `cognito-idp:initiate_auth` synchronously. +- **SessionCache**: The process-wide `TTLCache` in `backend/src/apis/shared/sessions_bff/cache.py` whose TTL defaults to `refresh_leeway_seconds` (60s). +- **`_DEFAULT_REFRESH_LEEWAY_SECONDS`**: 60s constant in `config.py` — both the refresh pre-expiry window and the SessionCache TTL. +- **`_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS`**: 60s constant in `config.py` — the minimum interval between DDB `touch_last_seen` writes for a single session. Currently aligned with leeway, will be de-aligned to 300s. +- **per-session `asyncio.Lock`**: The lock from `get_session_lock(session_id)` in `sessions_bff/lock.py`. Today it wraps only the Cognito refresh exchange; the fix does NOT move its scope — a separate single-flight `Future` is added upstream. +- **Single-flight Future**: New per-session `asyncio.Future` added for this fix that coalesces the upstream `get_item → needs_refresh → refresh?` resolution across concurrent callers within one task. + +## Bug Details + +### Bug Condition + +The bug manifests when a request reaches `SessionRefreshMiddleware.dispatch` with `BFFConfig.is_enabled() == True` AND a `__Host-bff_session` cookie present. Under this condition the middleware's resolve/slide path performs at least one event-loop-blocking AWS call, and — under fan-out — performs 2×N blocking calls for N concurrent same-session requests. The observable symptoms (504s, 80s `/files/quota` tails, 15.6s p-max at 0.7% CPU) follow directly. + +**Formal Specification:** + +``` +FUNCTION isBugCondition(input) + INPUT: input of type HTTPRequest + OUTPUT: boolean + + # Middleware-level precondition — everything else is scoped inside this. + IF NOT BFFConfig.from_env().is_enabled() THEN + RETURN false + END IF + IF input.cookies["__Host-bff_session"] IS NULL THEN + RETURN false + END IF + + # Sub-condition 1.1: sync boto3 in SessionRepository blocks the loop. + blocks_on_repo := ( + awaitedIn(request, SessionRepository.get) + OR awaitedIn(request, SessionRepository.touch_last_seen) + OR awaitedIn(request, SessionRepository.update_tokens) + OR awaitedIn(request, SessionRepository.put) + OR awaitedIn(request, SessionRepository.delete) + ) + AND NOT executesInThreadpool(boto3_call_of_that_method) + + # Sub-condition 1.2: sync boto3 in CognitoRefreshClient blocks the loop, + # AND it runs while get_session_lock(session_id) is held. + blocks_on_cognito := ( + invokedIn(request, CognitoRefreshClient.refresh) + AND NOT executesInThreadpool(initiate_auth_call) + AND sessionLockHeldDuring(initiate_auth_call) + ) + + # Sub-condition 1.3: N concurrent same-session requests are not coalesced + # across the session-resolve path. + missing_resolve_coalescing := ( + concurrentRequestsForSameSession(input.session_id) > 1 + AND countOf(SessionRepository.get calls for input.session_id in this window) + = concurrentRequestsForSameSession(input.session_id) + ) + + # Sub-condition 1.4: cache-miss boundary aligns with throttle boundary. + aligned_windows := ( + BFFConfig._DEFAULT_REFRESH_LEEWAY_SECONDS + == BFFConfig._DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS + ) + + # Sub-condition 1.5: response waits on inline-awaited touch_last_seen. + inline_slide := ( + slideWarrantedFor(request) + AND responseWaitsFor(touch_last_seen_call_of_this_request) + ) + + # Sub-condition 1.6: no concurrency slack at the deployment boundary. + no_slack := ( + uvicornWorkerCount() == 1 + AND ecsDesiredCount() == 1 + ) + + # Sub-condition 1.7: page-load fan-out amplifies 1.1 + 1.3 + 1.4. + amplified_fanout := ( + concurrentRequestsForSameSession(input.session_id) >= 8 + AND cacheWindowJustElapsedFor(input.session_id) + AND countOf(DDB calls on critical path during this window) + >= 2 * concurrentRequestsForSameSession(input.session_id) + ) + + RETURN blocks_on_repo + OR blocks_on_cognito + OR missing_resolve_coalescing + OR aligned_windows + OR inline_slide + OR no_slack + OR amplified_fanout +END FUNCTION +``` + +### Examples + +- **1.1 blocking repo call**: Any request that hits `request.state.bff_session = record` → `_maybe_slide` → `touch_last_seen`. Expected: the DDB round-trip runs off the event loop thread; other coroutines continue to be scheduled. Actual: the event loop is frozen for the full round-trip. +- **1.2 blocking Cognito call**: Two tabs refresh concurrently at minute 59 of the access token's lifetime. Expected: the Cognito `initiate_auth` for session A runs off the loop thread; unrelated requests (different cookies, Bearer-token requests, health checks) proceed. Actual: the loop is frozen for the full Cognito round-trip AND the per-session lock is held during that freeze. +- **1.3 missing resolve coalescing**: Angular fan-out of 8 same-session requests with no cached `SessionRecord`. Expected: 1 DDB `get_item`. Actual: 8 DDB `get_item` calls, each blocking. +- **1.4 aligned windows**: A request at T when `T - last_seen_at == 60s` AND `SessionCache` entry for this session has just TTL-evicted at T. Expected: at most 1 of `{get_item, update_item}`. Actual: both, serialized. +- **1.5 inline slide**: Request with `_maybe_slide` returning non-None. Expected: the response Set-Cookie lands immediately; the DDB write happens in the background. Actual: the response waits for DDB. +- **1.7 page-load fan-out**: Angular page load fires 8 endpoints at once right after a cache window elapses. Expected: ≤1 `get_item` + ≤1 `update_item` across the 8 requests. Actual: up to 16 serialized blocking calls at the front of the page load. +- **Edge case — `is_enabled() == False`**: The middleware must short-circuit before any of the above sub-conditions can manifest. No AWS calls, no locks, no futures. + +## Expected Behavior + +### Preservation Requirements + +**Unchanged Behaviors:** + +- **3.1 Dormant pass-through**: `BFFConfig.is_enabled() == False` → `dispatch` short-circuits to `call_next(request)` with no AWS calls, no cache lookup, no single-flight registration. +- **3.2 No-cookie pass-through**: No `__Host-bff_session` cookie → same short-circuit as 3.1. +- **3.3 Unrecoverable cookie → clear both cookies**: Bad seal, missing DDB row, expired TTL, or terminal `CognitoRefreshError` → `_clear_cookies(response)` clears both `__Host-bff_session` AND `__Host-bff_csrf` with the same attribute set as today. +- **3.4 Max-Age re-emit contract**: When `_maybe_slide` returns a non-None `Max-Age`, the `Set-Cookie` headers for both BFF cookies use that exact value and the exact attribute set in `_reemit_cookies` today. Fire-and-forget dispatch of the DDB write does not change this contract. +- **3.5 Refresh-storm coalescing (existing)**: For N concurrent same-session requests crossing the refresh-leeway boundary, exactly one `cognito-idp:initiate_auth` is issued per `session_id` per leeway window. The existing `get_session_lock(session_id)` scope around the Cognito exchange is preserved end-to-end. +- **3.6 Codec singleton**: `get_default_codec()` is the same process-wide instance used by the auth/callback seal path and the middleware unseal path. No per-request `kms:GenerateDataKey` is introduced. +- **3.7 Client-secret cache**: `resolve_bff_client_secret` continues to serve from the module-scope cache. No per-request `secretsmanager:GetSecretValue`. +- **3.8 CSRF middleware path**: `CSRFMiddleware` continues to validate unsafe-method requests using the existing in-memory HMAC double-submit check against `request.state.bff_csrf_token`. No new I/O is introduced on that path. +- **3.9 Absolute-lifetime cap**: `_maybe_slide` returns `None` once `created_at + absolute_lifetime_seconds` has passed. No further cookie re-emit or DDB slide. +- **3.10 Fail-closed rotation**: When Cognito rotates the refresh token and `_persist_refresh` exhausts its retries, the middleware invalidates the cache and clears the cookie. +- **3.11 Uniform cookie decode failure**: Every `CookieDecodeError` branch produces the same response shape and timing signature. No new oracle is introduced by the offload or single-flight paths. + +**Scope:** + +All inputs that do NOT involve the BFF middleware path should be completely unaffected by this fix. This includes: + +- Bearer-token requests (no `__Host-bff_session` cookie) — untouched. +- Anonymous endpoints (health, static assets) — untouched. +- WebSocket voice routes — they replicate the cookie unseal + DDB lookup outside the middleware (see `voice/routes.py`); this fix does not change their path. +- The auth/callback token-exchange route — it uses the same `CookieCodec` singleton to seal cookies; the singleton is not disturbed. +- The logout route — its cache `invalidate(session_id)` call is preserved. + +## Hypothesized Root Cause + +Based on the bug description and code inspection, the root causes are concurrent and independent — each sub-condition has its own root cause, and the fix addresses all of them: + +1. **Sync boto3 in `async def` methods (1.1, 1.2)**: The `SessionRepository` docstring explicitly acknowledges this ("The methods are declared `async` to match the rest of `apis.shared`, but boto3 is sync — calls run on the event loop thread"). The original reasoning was that refresh-storm coalescing via `get_session_lock()` would hold fan-out low enough to make thread-pool offload unnecessary. That reasoning is wrong for two reasons: (a) the lock only covers the Cognito exchange, not the DDB path — so fan-out is not coalesced at all for cache misses; and (b) even a single blocking call is enough to freeze the event loop for the round-trip duration, which is directly observable in `TargetResponseTime` p-max. + +2. **Wrong lock scope (1.3, 1.7)**: `get_session_lock(session_id)` is acquired inside `_resolve_session` only after the `_cache.get → _repository.get → needs_refresh` decision has been made. An `asyncio.Lock` held this narrowly cannot coalesce anything upstream of itself. The fix needs a different primitive — an `asyncio.Future` stored in a per-session slot that N waiters can await — because a lock would serialize N requests through one DDB call instead of consolidating them to one call. + +3. **Aligned windows by default (1.4)**: Both constants default to 60s in `config.py`. A strict-multiple relationship (e.g. throttle = 5 × leeway) de-aligns the boundaries. This is a config fix with no code change needed in the middleware. + +4. **`await` on `touch_last_seen` by pattern (1.5)**: `_maybe_slide` awaits the write because that matches the rest of the codebase's DB access shape. The surrounding `try/except` already swallows failures (documented as "Don't fail the request if the slide-write fails"), which is exactly the pre-condition that makes fire-and-forget safe. + +5. **Single-worker container (1.6)**: The Dockerfile CMD ships one uvicorn worker and `desiredCount: 1` in CDK ships one task. This was fine for the Bearer-token era; under the BFF middleware, it means any one blocked round-trip halts every other in-flight request. Concurrency slack is a separate lever from event-loop non-blocking — both are required, neither is sufficient alone. + +## Correctness Properties + +Property 1: Bug Condition — Event-Loop Non-Blocking, Coalesced, Window-Staggered, Fire-and-Forget BFF Middleware + +_For any_ request where the bug condition holds (`isBugCondition` returns true), the fixed middleware and its collaborators SHALL (a) execute every boto3 DynamoDB and Cognito call off the event loop thread (via `asyncio.to_thread` or equivalent), (b) coalesce N concurrent same-`session_id` requests crossing a cold cache window to at most one DynamoDB `get_item` via a per-session `asyncio.Future`, (c) hold the `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` default to a strict multiple of `_DEFAULT_REFRESH_LEEWAY_SECONDS` (300s vs 60s) so cache-expiry and throttle-expiry do not align, (d) dispatch `_maybe_slide`'s `touch_last_seen` as a detached `asyncio.Task` and return the `Max-Age` to the response path synchronously, and (e) run with concurrency slack such that `desiredCount >= 2` in production configuration. The observable result SHALL be that Angular's ~8-endpoint page-load fan-out issues at most 1 `get_item` and at most 1 `update_item` per `session_id` per cache window (not ~16), and no single AWS call serializes unrelated requests. + +**Validates: Requirements 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7** + +Property 2: Preservation — BFF Middleware Contracts Unchanged for Non-Buggy Inputs + +_For any_ request where the bug condition does NOT hold (`isBugCondition` returns false), the fixed middleware SHALL produce the same externally observable result as the original middleware, preserving: dormant pass-through (`is_enabled() == False`), no-cookie pass-through, unrecoverable-cookie clearing of both `__Host-bff_session` and `__Host-bff_csrf` with the same attribute set, the `Max-Age` re-emit contract between `_maybe_slide` and `_reemit_cookies`, exactly-one Cognito `initiate_auth` per `session_id` per leeway window, the `CookieCodec` and client-secret process-wide singletons, the `CSRFMiddleware` in-memory HMAC double-submit check, the absolute-lifetime cap behavior, fail-closed refresh-token rotation, and uniform `CookieDecodeError` handling. + +**Validates: Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, 3.11** + +## Fix Implementation + +### Changes Required + +Assuming the root cause analysis above is correct, the fix spans four code locations and one infrastructure config. + +**File**: `backend/src/apis/shared/sessions_bff/repository.py` + +**Function**: `SessionRepository.get`, `touch_last_seen`, `update_tokens`, `put`, `delete` + +**Specific Changes**: + +1. **Threadpool offload for every boto3 call**: Extract each method's boto3 invocation into a nested sync helper and invoke it via `await asyncio.to_thread(helper, ...)`. Example for `get`: + ``` + async def get(self, session_id): + if not self._enabled: + return None + def _call(): + return self._table.get_item(Key=self._key(session_id)) + try: + response = await asyncio.to_thread(_call) + except ClientError as exc: + ... + ``` + The method signatures, return types, and exception handling stay identical. The post-decode TTL defense-in-depth check and `_item_to_record` translation stay on the calling coroutine. + +2. **No change to public API**: Every callsite in the middleware (`self._repository.get`, `self._repository.touch_last_seen`, `self._repository.update_tokens`) remains an `await`. The offload is purely internal. + +**File**: `backend/src/apis/shared/sessions_bff/refresh.py` + +**Function**: `CognitoRefreshClient.refresh` + +**Specific Changes**: + +3. **Add async wrapper that offloads to a threadpool**: Either rename `refresh` to `_refresh_sync` and add a new `async def refresh(...)` that calls `await asyncio.to_thread(self._refresh_sync, username=..., refresh_token=...)`, or convert `refresh` to `async def` in-place with the same offload. The middleware callsite (`self._refresh_client.refresh(...)`) becomes `await self._refresh_client.refresh(...)`. The Cognito SDK call and the `CognitoRefreshError` contract are unchanged. + +**File**: `backend/src/apis/shared/middleware/session_refresh.py` + +**Function**: `SessionRefreshMiddleware._resolve_session`, `_maybe_slide`, `dispatch` + +**Specific Changes**: + +4. **Add per-session single-flight for the session-resolve path**: Introduce a new module-level `dict[str, asyncio.Future[tuple[Optional[SessionRecord], bool]]]` guarded by a thread lock in a new small module `backend/src/apis/shared/sessions_bff/single_flight.py` (mirroring `lock.py`'s shape), with an API: + ``` + async def resolve_once(session_id, loader_coro_factory) -> tuple[Optional[SessionRecord], bool] + ``` + The leader creates an `asyncio.Future`, registers it, runs the loader, sets the result/exception, and removes the entry. Followers `await` the existing Future. In `_resolve_session`, wrap the `_cache.get → _repository.get → needs_refresh → (maybe refresh)` block (from cache lookup through return) inside this single-flight, keyed by `session_id`. The existing `get_session_lock(session_id)` scope around the Cognito refresh exchange is **not** moved or widened — it stays exactly where it is today. + +5. **Fire-and-forget slide-write in `_maybe_slide`**: Replace `await self._repository.touch_last_seen(...)` with a detached task. The function still computes `new_max_age` and returns it synchronously. The DDB write happens in the background; the existing `try/except` that was already documented to swallow failures moves into a `_slide_write_task(...)` helper that logs on failure. Update the local cache (`record.last_seen_at = now`, `record.ttl = new_ttl`, `self._cache.set(record)`) before scheduling the task, so subsequent same-request reads see the slid state. + +6. **No change to `dispatch` structure or the cookie-clear / cookie-reemit branches**: Keep `clear_cookie` and `renewal_max_age` handling identical. + +**File**: `backend/src/apis/shared/sessions_bff/config.py` + +**Constant**: `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` + +**Specific Changes**: + +7. **Raise default from 60s to 300s**: Change the constant from `60` to `60 * 5` (or explicitly `300`) so the cache TTL (tied to `_DEFAULT_REFRESH_LEEWAY_SECONDS = 60`) and the slide-throttle window are strict multiples. The env var `BFF_SESSION_SLIDING_RENEWAL_THROTTLE_SECONDS` continues to override. + +**File**: `infrastructure/cdk.context.json` (and test fixtures under `infrastructure/test/`) + +**Key**: `appApi.desiredCount` + +**Specific Changes**: + +8. **Raise production `desiredCount` to 2**: Keep `maxCapacity` as-is (4). Update only the production/non-test context — test fixtures can stay at 1 if needed to keep CDK unit tests fast, but the top-level production context value must flip to 2. This is a **deployment-time** behavior change and the last item in the fix plan; it does not become necessary until the other changes ship. + +**No changes required** in: `backend/src/apis/shared/sessions_bff/cache.py`, `backend/src/apis/shared/sessions_bff/cookie.py`, `backend/src/apis/shared/sessions_bff/lock.py`, `backend/src/apis/shared/sessions_bff/csrf.py`, `backend/src/apis/shared/middleware/csrf.py`, `backend/src/apis/app_api/auth/bff/*`, or the uvicorn `CMD` in `backend/Dockerfile.app-api` (the ECS `desiredCount` bump is the chosen vector for concurrency slack in 2.6 — a `--workers N` flag would require reworking the in-process singletons in `cache.py` and `refresh.py`, which is out of scope). + +## Testing Strategy + +### Validation Approach + +The testing strategy follows a two-phase approach: first, surface counterexamples that demonstrate the bug on unfixed code, then verify the fix works correctly and preserves existing behavior. Because four of the sub-conditions are independent, we run the exploratory phase against each one. + +### Exploratory Bug Condition Checking + +**Goal**: Surface counterexamples that demonstrate the bug BEFORE implementing the fix. Confirm or refute the root-cause analysis for each sub-condition. If any is refuted, we re-hypothesize. + +**Test Plan**: Write tests that inject a slow/instrumented boto3 stub (for DDB and Cognito) and drive the middleware directly under `pytest-asyncio`. For each sub-condition, assert the blocking/serialization behavior is present on unfixed code. Run on UNFIXED code first; the assertions SHALL fail against fixed code later. + +**Test Cases**: + +1. **Event loop blocked by `SessionRepository.get`** (validates 1.1): Stub the boto3 `table.get_item` with a 500ms `time.sleep`. Submit a `SessionRepository.get` call and a concurrent `asyncio.sleep(0.05)` marker coroutine on the same loop. Assert the marker resolves strictly after the `get` (will hold on unfixed code, will fail on fixed code where the marker completes long before `get` returns). + +2. **Event loop blocked by `CognitoRefreshClient.refresh`** (validates 1.2): Same shape as (1) but against a stubbed `cognito-idp:initiate_auth`. Additionally assert that `get_session_lock(other_session_id)` can be acquired concurrently (will fail on unfixed code because the sync Cognito call has frozen the whole loop thread). + +3. **N fan-out → N `get_item` calls** (validates 1.3): Spin up 8 concurrent `dispatch` calls with the same cookie and a cold `SessionCache`. Count `table.get_item` invocations on the stub. Assert count == 8 on unfixed code; the fix target is 1. + +4. **Aligned windows → both writes on one request** (validates 1.4): Set clock to a moment where the cache TTL just elapsed AND `now - last_seen_at == 60s`. Drive a single request. Assert both `get_item` AND `update_item` are called on unfixed code; on fixed code with the new 300s throttle default, only `get_item` is called. + +5. **Response waits on `touch_last_seen`** (validates 1.5): Stub `table.update_item` with a 500ms delay. Measure time from `dispatch` entry to `call_next(request)` return. On unfixed code, response time ≥ 500ms; on fixed code, response time is independent of the DDB write latency. + +6. **Single-worker container / `desiredCount: 1`** (validates 1.6): This is a deployment-level property, not a middleware-level one. Verified by reading `infrastructure/cdk.context.json` and the Dockerfile `CMD`. No runtime test; CDK unit test asserts `DesiredCount: 2` on the production context. + +7. **Page-load fan-out amplification** (validates 1.7): Combine (3) + (4) — 8 concurrent requests at a boundary moment. Count blocking DDB calls. Assert ≥ 16 on unfixed code, ≤ 2 on fixed code. + +**Expected Counterexamples**: + +- Blocked-loop markers do not complete until the stubbed AWS call returns. +- `table.get_item` call count on the stub matches the fan-out, not 1. +- `Set-Cookie` response latency tracks `table.update_item` latency. +- Possible causes confirmed: sync boto3 on event loop, narrow lock scope, aligned constants, inline-awaited slide write. + +### Fix Checking + +**Goal**: Verify that for all inputs where the bug condition holds, the fixed middleware produces the expected behavior defined by Property 1. + +**Pseudocode:** + +``` +FOR ALL input WHERE isBugCondition(input) DO + # (a) event loop non-blocking + marker_latency := measureConcurrentMarker(dispatch(input)) + ASSERT marker_latency << AWS_call_latency + + # (b) fan-out coalescing + ddb_get_calls := countGetItemCalls(during_dispatch(input_fanout_n=8)) + ASSERT ddb_get_calls <= 1 + + # (c) window staggering + ASSERT config.slidingRenewalThrottleSeconds + % config.refreshLeewaySeconds == 0 + ASSERT config.slidingRenewalThrottleSeconds + > config.refreshLeewaySeconds + + # (d) fire-and-forget slide + response_latency := measureDispatchTime(input_with_slide) + ASSERT response_latency independent_of touch_last_seen_latency + + # (e) concurrency slack (deployment assertion) + ASSERT cdkContextAppApiDesiredCount >= 2 +END FOR +``` + +### Preservation Checking + +**Goal**: Verify that for all inputs where the bug condition does NOT hold, the fixed middleware produces the same externally observable result as the original middleware. + +**Pseudocode:** + +``` +FOR ALL input WHERE NOT isBugCondition(input) DO + ASSERT dispatch_original(input).response == dispatch_fixed(input).response + ASSERT dispatch_original(input).set_cookie_headers + == dispatch_fixed(input).set_cookie_headers + ASSERT dispatch_original(input).request_state_bff_session + == dispatch_fixed(input).request_state_bff_session + ASSERT dispatch_original(input).cleared_cookies + == dispatch_fixed(input).cleared_cookies + ASSERT countOf(cognito.initiate_auth across N same-session concurrent requests) + == 1 per leeway window +END FOR +``` + +**Testing Approach**: Property-based testing is recommended for preservation checking because: + +- It generates many request shapes across the input domain (cookie present/absent, cookie seal valid/invalid, cache hit/miss, needs_refresh yes/no, rotation yes/no, slide warranted yes/no, absolute cap passed yes/no, `is_enabled()` true/false) and asserts equivalence against a mocked `SessionRepository` + `CognitoRefreshClient`. +- It catches edge cases in the single-flight and fire-and-forget paths that manual unit tests might miss (e.g. an exception inside the single-flight leader; a background slide task racing with the next request). +- It provides strong guarantees that the observable middleware contract is unchanged for the entire `¬C` input domain. + +**Test Plan**: First, exercise the unfixed middleware with an expressive `Hypothesis` strategy over request shapes and record observable outputs (response status, `Set-Cookie` headers, `request.state.bff_session`, DDB/Cognito call counts). Then, swap in the fixed middleware and assert equivalence on the same inputs. The strategy must skip any input that satisfies `isBugCondition` — only `¬C` inputs enter the preservation assertion. + +**Test Cases**: + +1. **Dormant pass-through unchanged** (3.1): With `is_enabled() == False`, every request shape produces identical responses under fixed and unfixed middleware with zero AWS calls. +2. **No-cookie pass-through unchanged** (3.2): Request with no `__Host-bff_session` header, for any method/path, produces identical responses with zero AWS calls. +3. **Unrecoverable cookie clears both cookies** (3.3): Bad-seal, missing-row, expired-row, and terminal-refresh-error inputs produce the same `Set-Cookie` headers with `Max-Age=0` for both `__Host-bff_session` and `__Host-bff_csrf`, same attribute set. +4. **Max-Age re-emit contract** (3.4): For inputs where `_maybe_slide` returns a non-None value, the resulting `Set-Cookie` headers match the original exactly (including attribute set). Fire-and-forget dispatch does not delay or drop the re-emit. +5. **Refresh-storm coalescing preserved** (3.5): For 10 concurrent same-session requests crossing the refresh-leeway window, exactly one `initiate_auth` call is observed on the Cognito stub. +6. **Codec / secret singletons preserved** (3.6, 3.7): Across many requests, `get_default_codec()` returns the same instance, and `resolve_bff_client_secret()` hits Secrets Manager exactly once per process. +7. **CSRF path unchanged** (3.8): Requests that trigger `CSRFMiddleware` produce identical accept/reject decisions with no new I/O. +8. **Absolute lifetime cap preserved** (3.9): Inputs with `created_at + absolute_lifetime_seconds < now` produce `_maybe_slide → None`, no slide write scheduled. +9. **Fail-closed rotation preserved** (3.10): With rotation triggered and `_persist_refresh` forced to exhaust retries, the cache is invalidated and both cookies are cleared. +10. **Cookie decode uniformity** (3.11): All `CookieDecodeError` branches produce identical response shapes and timing profiles on the fixed middleware (no new oracle via single-flight or fire-and-forget). + +### Unit Tests + +- **Repository offload**: Assert each `SessionRepository.*` method calls `asyncio.to_thread` (monkeypatched) exactly once per call and that the wrapped boto3 call receives the expected arguments. Assert `ClientError` propagation still matches today's behavior. +- **Cognito offload**: Assert `CognitoRefreshClient.refresh` is awaitable, offloads to a threadpool, preserves `CognitoRefreshError`, and returns the same `RefreshResult` shape. +- **Single-flight**: Two concurrent `resolve_once(session_id, factory)` calls share one loader invocation; the entry is removed after completion; an exception in the loader propagates to all waiters; distinct `session_id`s do not share. +- **Fire-and-forget slide**: `_maybe_slide` returns Max-Age before `touch_last_seen` completes; the background task writes to DDB; failure inside the task logs and does not bubble to `dispatch`; the local cache is updated synchronously before the task is scheduled. +- **Config constant**: `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS == 300`; strict multiple of `_DEFAULT_REFRESH_LEEWAY_SECONDS`. + +### Property-Based Tests + +- **Preservation over `¬C` input domain**: As described in Preservation Checking — generate request shapes, assert fixed ≡ original on response, cookies, `request.state`, and AWS call counts. +- **Fan-out coalescing invariant**: For any N ∈ [2, 32] and any cookie-bearing same-session fan-out, the number of DDB `get_item` calls observed on the stub is ≤ 1 per cache window. Randomize cache warm/cold state, `needs_refresh` outcomes, and concurrent-request arrival ordering. +- **Window-staggering invariant**: For any request timing `t` within one leeway window of a cache TTL boundary, the fixed middleware issues at most one of `{get_item, update_item}` on the critical path — never both. + +### Integration Tests + +- **End-to-end page-load fan-out**: Drive the app-api container (under `moto` for DDB, a stubbed Cognito client) with a simulated 8-endpoint Angular page load. Measure total wall-clock time and count of DDB/Cognito calls. Assert ≤ 1 `get_item` and ≤ 1 `update_item` across the fan-out, and total latency bounded by the slowest individual handler (not by serialized AWS I/O). +- **Concurrency slack at the deployment boundary**: CDK unit test asserts `DesiredCount: 2` for the production `app-api` service. Integration smoke test asserts that a deliberately slow endpoint (e.g., a route that sleeps 5s) does not stall a concurrent fast endpoint on a parallel request. +- **Refresh-storm under fan-out**: 8 concurrent requests across the refresh-leeway boundary on the same session. Assert exactly 1 Cognito `initiate_auth`, all 8 responses succeed, and `request.state.bff_session` carries the freshly rotated tokens. diff --git a/.kiro/specs/bff-middleware-event-loop-blocking/tasks.md b/.kiro/specs/bff-middleware-event-loop-blocking/tasks.md new file mode 100644 index 00000000..b6e58c64 --- /dev/null +++ b/.kiro/specs/bff-middleware-event-loop-blocking/tasks.md @@ -0,0 +1,169 @@ +# Implementation Plan + +- [x] 1. Write bug condition exploration test + - **Property 1: Bug Condition** - Event-Loop Blocking, Missing Coalescing, Aligned Windows, Inline Slide-Write + - **CRITICAL**: This test MUST FAIL on unfixed code - failure confirms the bug exists + - **DO NOT attempt to fix the test or the code when it fails** + - **NOTE**: This test encodes the expected behavior - it will validate the fix when it passes after implementation + - **GOAL**: Surface counterexamples that demonstrate each sub-condition of the bug in `SessionRefreshMiddleware` + - **Scoped PBT Approach**: Scope the property to concrete failing cases that deterministically reproduce each sub-condition under `pytest-asyncio` + - Test location: `backend/tests/apis/shared/middleware/test_session_refresh_bug_condition.py` + - Use `hypothesis` + `pytest-asyncio`; inject slow/instrumented boto3 stubs for DynamoDB (`table.get_item`, `table.update_item`) and Cognito (`initiate_auth`) via monkeypatching on `SessionRepository._table` and `CognitoRefreshClient` + - Bug Condition (from design `isBugCondition`): `BFFConfig.is_enabled() == True` AND `__Host-bff_session` cookie present AND any of sub-conditions 1.1 through 1.7 hold + - Expected Behavior assertions (from design Property 1 / Expected Behavior 2.1–2.7) that must hold for all inputs satisfying the bug condition: + - **(1.1) Repository offload**: Stub `table.get_item`/`update_item`/`put_item`/`delete_item` with a 500ms `time.sleep`. Run `SessionRepository.get(session_id)` concurrently with an `asyncio.sleep(0.05)` marker coroutine. ASSERT the marker completes strictly BEFORE the repository call returns (loop is not blocked). Repeat for `touch_last_seen`, `update_tokens`, `put`, `delete`. + - **(1.2) Cognito offload**: Stub Cognito `initiate_auth` with a 500ms `time.sleep`. Run `CognitoRefreshClient.refresh(...)` concurrently with a marker coroutine AND a concurrent `get_session_lock(other_session_id)` acquisition. ASSERT the marker and unrelated lock acquisition complete while `refresh` is in flight. + - **(1.3) Resolve-path coalescing**: Drive 8 concurrent `SessionRefreshMiddleware.dispatch` calls for the same `session_id` with cold `SessionCache` and a valid sealed cookie. Count `table.get_item` invocations on the stub. ASSERT count == 1 (bug: count == 8). + - **(1.4) Window de-alignment**: ASSERT `BFFConfig._DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS % BFFConfig._DEFAULT_REFRESH_LEEWAY_SECONDS == 0` AND `BFFConfig._DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS > BFFConfig._DEFAULT_REFRESH_LEEWAY_SECONDS`. Drive a single request with `SessionCache` TTL just elapsed AND `now - last_seen_at == 60s`. ASSERT at most one of `{get_item, update_item}` is observed on the critical path. + - **(1.5) Fire-and-forget slide**: Stub `table.update_item` with a 500ms delay. Drive a `dispatch` call where a slide is warranted. Measure elapsed time from `dispatch` entry to `call_next(request)` returning. ASSERT elapsed time < 250ms (bug: elapsed time ≥ 500ms because the response waits on the DDB write). + - **(1.6) Concurrency slack at deployment**: Read `infrastructure/cdk.context.json` and assert `appApi.desiredCount >= 2` for the production context. + - **(1.7) Fan-out amplification**: Drive 8 concurrent `dispatch` calls on the same session at a cache-boundary moment. Count blocking DDB calls across the fan-out. ASSERT count ≤ 2 (bug: count ≥ 16). + - Run all property cases on UNFIXED code + - **EXPECTED OUTCOME**: Test FAILS (this is correct - it proves the bug exists). Document the counterexamples in the test output: marker coroutines starved, 8 `get_item` calls per fan-out, both `get_item` and `update_item` on single request, response latency tracking `update_item` latency + - Mark task complete when test is written, run, and failures are documented + - _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7_ + +- [x] 2. Write preservation property tests (BEFORE implementing fix) + - **Property 2: Preservation** - BFF Middleware Contracts Unchanged for Non-Buggy Inputs + - **IMPORTANT**: Follow observation-first methodology + - Test location: `backend/tests/apis/shared/middleware/test_session_refresh_preservation.py` + - Use `hypothesis` to generate request shapes across the `¬C` input domain; skip any input for which `isBugCondition` returns true + - Strategy must cover all axes that exist today: `is_enabled()` true/false, `__Host-bff_session` cookie present/absent, cookie seal valid/invalid/expired, `SessionCache` hit/miss, `needs_refresh` yes/no, refresh-token rotation yes/no, slide warranted yes/no, absolute-lifetime cap passed yes/no, request method safe/unsafe (for CSRF interaction) + - **Observe behavior on UNFIXED code** for each non-buggy input and record: response status, `Set-Cookie` headers for `__Host-bff_session` and `__Host-bff_csrf` (including every attribute), `request.state.bff_session`, `request.state.bff_csrf_token`, DDB call counts, Cognito call counts, KMS/Secrets Manager call counts + - Write property-based tests capturing these observed behaviors as preservation invariants (from Preservation Requirements 3.1–3.11): + - **(3.1) Dormant pass-through**: for all requests, when `is_enabled() == False`, response == `call_next(request)` AND zero AWS calls + - **(3.2) No-cookie pass-through**: for all requests with no `__Host-bff_session` header, response == `call_next(request)` AND zero AWS calls + - **(3.3) Unrecoverable cookie clears both cookies**: for bad-seal / missing-row / expired-row / terminal-`CognitoRefreshError` inputs, `Set-Cookie` for both `__Host-bff_session` AND `__Host-bff_csrf` has `Max-Age=0` AND identical attribute set + - **(3.4) Max-Age re-emit contract**: when `_maybe_slide` returns non-None, the resulting `Set-Cookie` headers for both BFF cookies use that exact `Max-Age` and the exact attribute set from `_reemit_cookies` today + - **(3.5) Refresh-storm coalescing**: for 10 concurrent same-session requests crossing the refresh-leeway window, exactly one `cognito-idp:initiate_auth` call is observed + - **(3.6) Codec singleton**: across many requests, `get_default_codec()` returns the same instance identity; zero per-request `kms:GenerateDataKey` calls + - **(3.7) Client-secret cache**: across many requests, `resolve_bff_client_secret()` hits Secrets Manager exactly once per process + - **(3.8) CSRF path unchanged**: `CSRFMiddleware` accept/reject decision on unsafe-method requests is identical to unfixed; zero new I/O on the CSRF path + - **(3.9) Absolute-lifetime cap**: when `now > created_at + absolute_lifetime_seconds`, `_maybe_slide` returns `None`; no slide scheduled + - **(3.10) Fail-closed rotation**: when rotation triggers AND `_persist_refresh` exhausts retries, cache is invalidated AND both cookies are cleared + - **(3.11) Cookie decode uniformity**: every `CookieDecodeError` branch produces identical response shape and timing profile (no new oracle) + - Run tests on UNFIXED code + - **EXPECTED OUTCOME**: Tests PASS (this confirms baseline behavior to preserve) + - Mark task complete when tests are written, run, and passing on unfixed code + - _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, 3.11_ + +- [x] 3. Fix for BFF middleware event-loop blocking and fan-out amplification + + - [x] 3.1 Offload `SessionRepository` boto3 calls via `asyncio.to_thread` + - Edit `backend/src/apis/shared/sessions_bff/repository.py` + - For each of `get`, `touch_last_seen`, `update_tokens`, `put`, `delete`: extract the boto3 invocation into a nested sync helper and invoke it via `await asyncio.to_thread(helper, ...)` + - Keep method signatures, return types, and exception-handling branches identical + - Keep the post-decode TTL defense-in-depth check and `_item_to_record` translation on the calling coroutine + - Do NOT change the public API — every callsite in the middleware remains an `await` + - _Bug_Condition: isBugCondition(input) where sub-condition 1.1 holds (sync boto3 on event loop)_ + - _Expected_Behavior: expectedBehavior(result) per design Property 1 clause (a) — every `SessionRepository` boto3 call executes off the event loop thread_ + - _Preservation: 3.3, 3.10 — exception branches and fail-closed rotation unchanged_ + - _Requirements: 2.1_ + + - [x] 3.2 Offload `CognitoRefreshClient.refresh` via `asyncio.to_thread` + - Edit `backend/src/apis/shared/sessions_bff/refresh.py` + - Rename existing `refresh` to `_refresh_sync` (or equivalent private sync form) and add a new `async def refresh(...)` that calls `await asyncio.to_thread(self._refresh_sync, username=..., refresh_token=...)` + - Update the callsite in `SessionRefreshMiddleware._resolve_session` to `await self._refresh_client.refresh(...)` + - Preserve the `CognitoRefreshError` contract and `RefreshResult` return shape exactly + - Do NOT move or widen the `get_session_lock(session_id)` scope around the refresh exchange + - _Bug_Condition: isBugCondition(input) where sub-condition 1.2 holds (sync Cognito on event loop while session lock held)_ + - _Expected_Behavior: expectedBehavior(result) per design Property 1 clause (a) — Cognito `initiate_auth` executes off the event loop thread, other sessions' locks are acquirable_ + - _Preservation: 3.5 — refresh-storm coalescing preserved_ + - _Requirements: 2.2_ + + - [x] 3.3 Add per-session single-flight primitive module + - Create `backend/src/apis/shared/sessions_bff/single_flight.py` + - Export `async def resolve_once(session_id: str, loader_coro_factory: Callable[[], Awaitable[tuple[Optional[SessionRecord], bool]]]) -> tuple[Optional[SessionRecord], bool]` + - Internal state: module-level `dict[str, asyncio.Future[tuple[Optional[SessionRecord], bool]]]` guarded by a `threading.Lock` (mirroring the shape of `sessions_bff/lock.py`) + - Leader semantics: first caller for a given `session_id` creates an `asyncio.Future`, registers it under the session lock, runs the loader, sets the result or exception, removes the entry, and returns + - Follower semantics: any caller that finds an existing Future `await`s it and returns its value + - Exception propagation: an exception from the loader MUST propagate to all current waiters, and the registry entry MUST be removed so subsequent calls start a new leader + - Distinct `session_id`s MUST NOT share a Future + - Include unit tests alongside: two concurrent `resolve_once` calls share one loader invocation; exception propagation to all waiters; distinct sessions are independent + - _Bug_Condition: isBugCondition(input) where sub-condition 1.3 holds (N concurrent same-session resolves issue N `get_item` calls)_ + - _Expected_Behavior: expectedBehavior(result) per design Property 1 clause (b) — at most one DynamoDB `get_item` per `session_id` per cache window_ + - _Preservation: 3.5 — the existing `get_session_lock` scope around the Cognito exchange is unchanged (this is a separate primitive upstream)_ + - _Requirements: 2.3_ + + - [x] 3.4 Wire single-flight into `SessionRefreshMiddleware._resolve_session` + - Edit `backend/src/apis/shared/middleware/session_refresh.py` + - Wrap the `_cache.get → _repository.get → needs_refresh → (maybe refresh)` block in `_resolve_session` inside `resolve_once(session_id, loader_coro_factory)` where the loader factory builds the coroutine that performs today's cache/repo/refresh sequence and returns `(Optional[SessionRecord], clear_cookie: bool)` + - Ensure the existing `get_session_lock(session_id)` scope around the Cognito refresh exchange remains exactly where it is today — do NOT move or widen it + - Ensure the bad-seal / missing-row / expired-row / terminal-refresh-error paths still produce the same `clear_cookie=True` return and the same exception propagation to `dispatch` as today + - _Bug_Condition: isBugCondition(input) where sub-conditions 1.3 and 1.7 hold (fan-out amplification)_ + - _Expected_Behavior: expectedBehavior(result) per design Property 1 clause (b)_ + - _Preservation: 3.3, 3.5, 3.11 — unrecoverable cookie clearing, refresh-storm coalescing, uniform decode failure preserved_ + - _Requirements: 2.3, 2.7_ + + - [x] 3.5 Convert `_maybe_slide` to fire-and-forget DDB write + - Edit `backend/src/apis/shared/middleware/session_refresh.py` + - In `_maybe_slide`, update the local cache synchronously (`record.last_seen_at = now`, `record.ttl = new_ttl`, `self._cache.set(record)`) BEFORE scheduling the background task + - Replace `await self._repository.touch_last_seen(...)` with `asyncio.create_task(self._slide_write_task(...))` + - Introduce a private `async def _slide_write_task(self, record, ...)` helper that performs `await self._repository.touch_last_seen(...)` inside a `try/except` that logs on failure (preserving today's "swallow failures" semantics) + - Return the computed `new_max_age` synchronously from `_maybe_slide` + - Do NOT change `dispatch` structure or the cookie-clear / cookie-reemit branches + - Do NOT change the absolute-lifetime cap path — it must still return `None` + - _Bug_Condition: isBugCondition(input) where sub-condition 1.5 holds (response waits on inline slide-write)_ + - _Expected_Behavior: expectedBehavior(result) per design Property 1 clause (d) — response latency independent of `touch_last_seen` latency_ + - _Preservation: 3.4, 3.9 — Max-Age re-emit contract and absolute-lifetime cap preserved_ + - _Requirements: 2.5_ + + - [x] 3.6 De-align cache/leeway and throttle windows in config + - Edit `backend/src/apis/shared/sessions_bff/config.py` + - Change `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` from `60` to `60 * 5` (or explicit `300`) + - Verify `_DEFAULT_REFRESH_LEEWAY_SECONDS` remains `60` + - Confirm the strict-multiple relationship: `300 % 60 == 0` AND `300 > 60` + - Ensure the `BFF_SESSION_SLIDING_RENEWAL_THROTTLE_SECONDS` env var still overrides the default + - _Bug_Condition: isBugCondition(input) where sub-condition 1.4 holds (aligned windows force both writes on one request)_ + - _Expected_Behavior: expectedBehavior(result) per design Property 1 clause (c) — cache-miss does not imply slide-write_ + - _Preservation: none impacted (pure default-value change; overrides preserved)_ + - _Requirements: 2.4_ + + - [x] 3.7 Raise production `appApi.desiredCount` to 2 + - Edit `infrastructure/cdk.context.json` to set `appApi.desiredCount` to `2` in the production/non-test context + - Keep `appApi.maxCapacity` unchanged (4) + - Test fixtures under `infrastructure/test/` may stay at `1` if needed for CDK unit-test speed; only the top-level production context value must change + - Update or add CDK unit tests to assert `DesiredCount: 2` on the production `app-api` service synthesis + - _Bug_Condition: isBugCondition(input) where sub-condition 1.6 holds (no concurrency slack at deployment)_ + - _Expected_Behavior: expectedBehavior(result) per design Property 1 clause (e) — `desiredCount >= 2` in production configuration_ + - _Preservation: none impacted (deployment-config change; in-process singletons untouched)_ + - _Requirements: 2.6_ + + - [x] 3.8 Verify bug condition exploration test now passes + - **Property 1: Expected Behavior** - Event-Loop Non-Blocking, Coalesced, Window-Staggered, Fire-and-Forget BFF Middleware + - **IMPORTANT**: Re-run the SAME test from task 1 - do NOT write a new test + - The test from task 1 encodes the expected behavior from design Property 1 + - When this test passes, it confirms the expected behavior is satisfied across all seven sub-conditions + - Run: `cd backend && uv run python -m pytest tests/apis/shared/middleware/test_session_refresh_bug_condition.py -v` + - **EXPECTED OUTCOME**: Test PASSES (confirms bug is fixed): + - Marker coroutines complete while AWS stubs are still sleeping (1.1, 1.2) + - 8-fan-out produces exactly 1 `get_item` on the stub (1.3, 1.7) + - Aligned-boundary request produces at most one of `{get_item, update_item}` (1.4) + - Dispatch latency independent of `update_item` stub latency (1.5) + - `appApi.desiredCount >= 2` in production context (1.6) + - _Requirements: Expected Behavior Properties from design (2.1–2.7)_ + + - [x] 3.9 Verify preservation tests still pass + - **Property 2: Preservation** - BFF Middleware Contracts Unchanged for Non-Buggy Inputs + - **IMPORTANT**: Re-run the SAME tests from task 2 - do NOT write new tests + - Run: `cd backend && uv run python -m pytest tests/apis/shared/middleware/test_session_refresh_preservation.py -v` + - **EXPECTED OUTCOME**: Tests PASS (confirms no regressions): + - Dormant pass-through with zero AWS calls (3.1) + - No-cookie pass-through with zero AWS calls (3.2) + - Unrecoverable cookie clears both cookies with identical attributes (3.3) + - Max-Age re-emit contract preserved under fire-and-forget dispatch (3.4) + - Exactly one `initiate_auth` per `session_id` per leeway window (3.5) + - `get_default_codec()` and `resolve_bff_client_secret()` remain singletons (3.6, 3.7) + - `CSRFMiddleware` path unchanged (3.8) + - Absolute-lifetime cap preserved (3.9) + - Fail-closed rotation preserved (3.10) + - Uniform `CookieDecodeError` handling preserved (3.11) + - Confirm all tests still pass after fix (no regressions) + +- [x] 4. Checkpoint - Ensure all tests pass + - Run the full backend test suite: `cd backend && uv run python -m pytest tests/ -v` + - Run CDK unit tests: `cd infrastructure && npm run build && npm test` + - Confirm the bug condition exploration test (task 1) passes on fixed code + - Confirm the preservation property tests (task 2) pass on fixed code + - Confirm no unrelated tests regress + - Ensure all tests pass, ask the user if questions arise diff --git a/.kiro/steering/structure.md b/.kiro/steering/structure.md index 54606b6f..d5862e64 100644 --- a/.kiro/steering/structure.md +++ b/.kiro/steering/structure.md @@ -238,7 +238,7 @@ scripts/ - **Files**: snake_case (e.g., `turn_based_session_manager.py`) - **Classes**: PascalCase (e.g., `TurnBasedSessionManager`) -- **Functions**: snake_case (e.g., `get_current_user`) +- **Functions**: snake_case (e.g., `get_current_user_from_session`) - **Constants**: UPPER_SNAKE_CASE (e.g., `MAX_FILE_SIZE`) - **Private**: Leading underscore (e.g., `_internal_method`) @@ -266,7 +266,7 @@ All modules are properly packaged and can be imported directly: ```python # Shared utilities (canonical location for cross-service code) -from apis.shared.auth import get_current_user, User +from apis.shared.auth import get_current_user_from_session, User from apis.shared.rbac import RBACService from apis.shared.costs.calculator import CostCalculator from apis.shared.tools.models import ToolDefinition diff --git a/CHANGELOG.md b/CHANGELOG.md index 3756d72e..4b854a06 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,220 @@ All notable changes to this project are documented in this file. Format follows For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md). +## [1.0.0-beta.27] - 2026-05-20 + +The largest release since the BFF cutover. Two new user-facing surfaces (Artifacts and MCP Apps host-renderer) each backed by a new CDK stack, an admin shell redesign that replaces the 15-card grid with a persistent grouped sidebar, recoverable `max_tokens` truncation with a Continue affordance, model-aware adaptive thinking for Opus 4.7, an inference-API `/ping` reaper fix, and a pre-migration backup tool. `bedrock-agentcore` 1.6.4 → 1.9.1, `boto3` 1.42.96 → 1.43.9, `strands-agents` 1.39.0 → 1.40.0. + +### 🚀 Added + +- **Artifacts feature** — agent-authored versioned standalone documents (HTML, Markdown, code) that render in a sandboxed iframe in a docked side panel. Backed by a new `ArtifactsStack` (DDB `user-artifacts` heads + version log with session GSI; private S3 `artifacts-content` bucket; render Lambda; CloudFront on `artifacts.{domain}`) and short-lived HMAC-signed render-token JWTs minted by app-api. Two new built-in tools (`create_artifact`, `update_artifact`) registered as default public tools so the feature works on first deploy. Versions are immutable (no `s3:DeleteObject` on inference-api). HTML mode allows scripts from `cdn.tailwindcss.com`, `esm.sh`, `cdn.jsdelivr.net`, `unpkg.com`; `connect-src 'none'`. Markdown mode wraps GFM input in a self-contained HTML render harness server-side. Frontend: docked resizable panel, auto-open on first creation, skeleton loader, latest-version on update, per-version history cards, preview/code toggle with syntax-highlighted source view, download button (#306, #309, #310, #311, #312, #314, #316, #317, #318, #319, #321, #322, #323, #324, #325, #326, #334) +- **MCP Apps host-renderer** — third-party MCP servers can ship UI alongside their tools. New `McpSandboxStack` (CloudFront on `mcp-sandbox.{domain}` with a CloudFront Function emitting per-resource `frame-ancestors` CSP; outer mount-page S3 bucket). Agent advertises `experimental.ui` on MCP `initialize`, fetches `ui_resource` payloads via `resources/read`, emits a `ui_resource` SSE event with `uri`, `permissions`, and `sandboxOrigin`. Frontend `` Angular custom element renders Apps in a sandboxed iframe with a `postMessage` bridge that enforces allowed message types (`ui/message`, `ui/update-model-context`) and origin checks. App-initiated `tools/call` proxied through app-api over an event broker. Explicit user consent prompt on first frame, persisted across reloads via card store. Default-on this release (`Defaults.MCP_APPS_HOST_ENABLED` flips false → true) with `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN` wired into inference-api runtime env from SSM. Tools whose only output is a `ui_resource` are filtered out for non-capable clients. Committed `budget-allocator-server` example; runbooks updated (#296, #339, #342, #343, #344, #345, #346, #347, #348, #349, #352, #353, #355, #360) +- **Admin shell redesign** — persistent grouped sidebar nav (Usage & Spend / AI Configuration / Identity & Access / Customization) replaces the 15-card admin grid. `/admin` redirects to `/admin/costs`. Quotas (Tiers / Assignments / Overrides / Inspector / Events) collapses 5 sibling routes into a single tabbed page; Fine-Tuning (Access / Costs) collapses into one. "Back to Admin" link removed from 10 sub-pages. Cost summary cards restructured (title on its own row, icon as top-right corner accent) so "Cache Savings" / "Avg Cost/User" stop wrapping (#300) +- **Compact model browse + manage views** — manage-models and the Bedrock/Gemini/OpenAI browse pages redesigned as one-line scannable rows with expand-on-demand detail; slim inline filter toolbar; inline enable/disable toggle so status changes don't require opening the form; `rounded-2xl` matches the chat input (#332) +- **Compact tool catalog + form** — same redesign applied to admin tools list and create/edit form. Compact expandable rows; form flattened to shared list-page token set (`rounded-2xl`, `text-sm/6`, `text-2xl/8` header, `focus:ring-2`); no behavior changes (#335) +- **Admin-managed user-menu links** — new admin domain so org admins can curate the SPA user-menu links without code changes. Each link is either an external URL (new tab) or an in-app modal with admin-authored Markdown. New `user-menu-links` DDB table; admin CRUD at `/admin/user-menu-links` (`require_admin`); public enabled-only read at `/user-menu-links` (cookie-aware `get_current_user_from_session`) (#298) +- **Recoverable `max_tokens` truncation** — `MaxTokensReachedException` is classified specifically in the stream processor and emits a `max_tokens`-coded recoverable `stream_error` event. Continue is a resume, not a new turn: `continue_truncated` re-enters the agent loop with an empty-list prompt (assistant-prefill) bypassing quota / RAG / file-resolution. `lastTurnContinuable` marker on session metadata flows through `SessionMetadataResponse` so Continue reappears after a refresh. Frontend renders a compact inline "Response length limit reached" notice + Continue button (no verbose error bubble); continuation-aware message-map sync pins the partial and appends the continuation. `stream_error` is now an always-allowed parser event (#328) +- **Model-aware adaptive thinking + `effort` knob** — `_shape_thinking_value` is now model-aware. Opus 4.6/4.7, Sonnet 4.6, and Mythos emit `{type: "adaptive", display: "summarized"}` (the explicit `display` keeps the reasoning trace visible — Opus 4.7 defaults `display` to `"omitted"`); older models keep `{type: "enabled", budget_tokens: N}`. New `effort` canonical inference param wired through `additional_request_fields.output_config.effort` (NOT `additionalModelRequestFields`). Wired through the admin model form and the user-facing chat settings panel as a new select control with server-side allowed-set gating. Generic `allowed` enum on `ModelParamSpec` so the per-model effort-tier difference (Sonnet 4.6 vs Opus 4.7) is data, not a model-family branch (#331) +- **Pre-migration backup tool** — `scripts/backup-data/` produces a complete restore-friendly snapshot for a given `CDK_PROJECT_PREFIX`: all ~20 application DDB tables via `ExportTableToPointInTime`, user-content S3 buckets via `aws s3 sync`, full Cognito user pool config including identity providers and app clients with plaintext client secrets preserved, users / groups / group memberships, and best-effort AgentCore Memory events. Each run lands in a freshly-created versioned SSE-encrypted TLS-only `{prefix}-backup-{utc_timestamp}` bucket. `manifest.json` is the single source of truth for restore. Cognito password hashes are not exportable by AWS — documented prominently. Ephemeral session/state tables excluded by default. `workflow_dispatch` GitHub workflow wired via the existing OIDC composite action (#361) +- **Live tool output streamed into the tool rail** during artifact authoring (#316) +- **Markdown content-type support** in the artifact tool (#318) +- **Configurable extra CSP `frame-ancestors`** for the artifact origin (#314) +- **`` custom element + `postMessage` bridge** with origin- and type-enforcement (#346) +- **Tool result renderer registry** — signal-backed `ToolRendererRegistryService` keyed by tool name replaces the implicit text/JSON/image switch baked into `ToolUseComponent`. The default renderer reproduces the prior markup verbatim — zero visible change. `calculator`, `fetch_url_content`, and `create_visualization` migrated as proof points. Foundation for the MCP Apps `` renderer (#339) +- **Copy-to-clipboard button on chat code blocks** + Prism syntax-highlighting bundles for JavaScript, TypeScript, Python, and SQL alongside the existing C#/CSS bundles (#299) +- **Autofocus chat input on session load and switch** so the user can type immediately without clicking. Assistant-preview empty state opts out via a new `autoFocus` input (#333) +- **Denser session sidebar with skeleton + entry animation** — rows tighten from ~40px to ~32px (`py-2 → py-1.5`, `text-sm/6 → text-sm/5`); nested flex wrappers around the title removed; group gaps tightened. A 10-session list is ~25% shorter overall. Inactive items `font-normal`; active row `!font-medium` via `routerLinkActive` (#301) + +### ✨ Improved + +- **Spinners across admin / settings / fine-tuning / auth pages** — 24 loading spinners had been rendering as a uniform gray ring in dark mode (no visible motion); they now spin with the proper accent (#300) +- **Admin shell wider with sidebar label wrapping fixed** (#305) +- **User-menu links / in-app modals visually distinguished** in both modal preview and runtime rendering (#303) +- **`mcp-sandbox` outer CSP + inner mount aligned** with the upstream `ext-apps` basic-host reference; blob iframe rendering, first-class block element, Angular 21-specific fixes (#352, #353) +- **Dynamic per-resource CSP** for the sandbox proxy — CloudFront Function decodes a URL-encoded `?csp=` query param scoped to one resource and emits the per-request `Content-Security-Policy` header. Source loaded from `assets/mcp-sandbox/csp-function.js` with `frame-ancestors` JSON-injected at synth; substitution asserts the placeholder is present exactly once so a future refactor that loses it fails loudly at synth (#355) + +### 🐛 Fixed + +- **Critical:** `MaxTokensReachedException` surfaced as a generic leaky error (`...unrecoverable state... https://strandsagents.com/...`) and the only "recovery" re-sent the original prompt as a new user turn, so the model re-answered from scratch and re-truncated — an infinite loop. Continue is now a true resume (`continue_truncated` empty-list prompt, assistant-prefill on restored history) bypassing quota / RAG / file-resolution like the existing interrupt-resume path (#328) +- **Opus 4.7 400 on `thinking.type="enabled"`** — Opus 4.7 rejects the legacy thinking shape; model-aware `_shape_thinking_value` now emits `{type: "adaptive"}` for Opus 4.6/4.7, Sonnet 4.6, Mythos. Without this fix, Opus 4.7 turns failed at the SDK boundary (#331) +- **Float-typed `max_tokens` / `top_k` crashed boto3's Bedrock Converse client.** Untyped inference params (`Dict[str, Any]` from JSON) let a float reach the SDK, which rejects a float `maxTokens` with a hard validation error. Coerced to `int` at the single provider-translation chokepoint (covers fresh + resumed turns, all providers). The thinking-vs-`max_tokens` consistency guard previously used `isinstance(..., int)` and silently no-opped on float input; it now coerces first so an inconsistent request (`thinking >= max_tokens`) is rejected before reaching Anthropic. Model-ceiling cap protects against admin-configured `max_tokens` exceeding the model's hard limit (#329, #330) +- **Silent mid-stream microVM reaping on long generations.** AgentCore's idle reaper requires an integer `time_of_last_update` field alongside `status`; when absent, the platform reaps the microVM at `idleRuntimeSessionTimeout` regardless of reported status (`bedrock-agentcore-sdk-python#471`). Inference-api's `/ping` now emits a fresh timestamp on every call as the documented mitigation. Status casing also corrected to match `PingStatus`. Workaround until async-task busy tracking lands and we can report `HealthyBusy` (#338) +- **Frontend deploy bundles shipped the `'dev'` placeholder.** `scripts/stack-frontend/build.sh` invoked `ng build` directly, bypassing the npm `prebuild` lifecycle hook that runs `gen-version.js`. The user menu rendered "local" on `develop` and `main`. Build script now runs `gen-version.js` explicitly before the build (#336) +- **Chart.js artifacts loaded via `cdn.jsdelivr.net` rendered blank.** The artifact-origin CSP only permitted scripts from `cdn.tailwindcss.com` and `esm.sh`. Widened script-src to `cdn.jsdelivr.net` and `unpkg.com`, kept byte-identical across the render Lambda `CSP_SCRIPT_SRC` env var and the system-prompt allowlist (#326) +- **Admin user-menu-links resource fired a duplicate load request for non-admin users** — gated to admin-only (#315) +- **Artifact card z-index escapes its message row on focus** — scoped with `isolation: isolate` (#323) +- **`mcp-sandbox` CFN `Comment` overflowed AWS's 128-char cap** — twice, on the original RHP and the rebuild (#356, #357) +- **`mcp-sandbox` CSP not URL-decoded in CloudFront Function** — decoded properly; `x-csp-debug` diagnostic header added during the investigation (#358) and removed once the fix landed (#359) +- **Inner App iframe gained `allow-same-origin`** to match the upstream basic-host reference (#360) +- **Docker build hard-fail from rotated `curl` apt pin.** Debian rotated `curl 8.14.1-2+deb13u2` out of the trixie apt index (superseded by `+deb13u3`); the exact pin made every App API / Inference API Docker build on `develop` fail with `E: Version '8.14.1-2+deb13u2' for 'curl' was not found`. Pin bumped (#327) +- **Artifact env vars not passed to non-`ArtifactsStack` consumer workflows.** `validateConfig` runs on every stack synth (the `bin/` instantiates all enabled stacks), so consumer workflows need to pass `CDK_HOSTED_ZONE_DOMAIN`, `CDK_ARTIFACTS_ENABLED`, and `CDK_ARTIFACTS_CERTIFICATE_ARN` even though they don't synth `ArtifactsStack` directly. Five deploys failed on the develop merge before this fix (#307) +- **`infrastructure-stack` tests asserted a stale DDB count.** `resourceCountIs(18)` went red when `user-menu-links` landed (19 tables). Replaced the magic number with an enumerated, justified table list (#350) + +### 🔒 Security + +- **Artifacts isolation.** `artifacts.{domain}` is a different cookie-jar host from the SPA. CSP `connect-src 'none'` — artifacts cannot make outbound network calls. Render-token JWTs are scoped to one `(artifact_id, version)` and are HMAC-signed with a Secrets-Manager-managed key. S3 versions are immutable: there's no `s3:DeleteObject` grant on the inference-api role +- **MCP Apps isolation.** `mcp-sandbox.{domain}` is a separate origin from the SPA. Per-resource `frame-ancestors` CSP is emitted by a CloudFront Function on viewer-response. Inner App iframe carries `allow-same-origin` to match the basic-host reference. Explicit user consent (with reload persistence) gates first-time framing +- **Dead Bearer-only auth removed from app-api (#297).** A sweep of `app_api/` for `Depends(get_current_user)`, `Depends(security)`, `Depends(verify_token)`, and manual `Authorization` header reads turned up exactly two routes still on Bearer auth, both in `chat/routes.py`. Dead Bearer paths removed; `POST /chat/agent-stream` is documented as intentionally Bearer for non-SPA callers (API-key tooling, scripts). All other app-api routes are cookie-based BFF auth post-beta.24 + +### ⚠️ Breaking changes + +- **MCP Apps default-on.** `Defaults.MCP_APPS_HOST_ENABLED` flips false → true. To remain opt-in, set `AGENTCORE_MCP_APPS_HOST_ENABLED=false` in inference-api task env. If MCP Apps is enabled but `mcp-sandbox` isn't deployed, `ui_resource` events emit with empty `sandboxOrigin` and the SPA cannot frame the App (#349) +- **App-api Bearer-only auth removed (#297).** External integrations calling `apis/app_api/` routes with `Authorization: Bearer` must switch to the API-key feature (`auth/api_keys/`, `X-API-Key`) before deploying beta.27. `POST /chat/agent-stream` remains Bearer-acceptable for non-SPA callers + +### 🏗️ Infrastructure + +- **New `ArtifactsStack`** (gated by `config.artifacts.enabled`) — DDB `user-artifacts` table, private S3 `artifacts-content` bucket, render Lambda, CloudFront on `artifacts.{domain}`, Route53 alias. Consumes `/artifacts/render-token-key-arn` SSM (published by `InfrastructureStack`); publishes `/artifacts/bucket-name`, `/artifacts/bucket-arn`, `/artifacts/table-name`, `/artifacts/table-arn`, `/artifacts/origin`. Requires `CDK_HOSTED_ZONE_DOMAIN`, `CDK_ARTIFACTS_CERTIFICATE_ARN` (must be in `us-east-1`) +- **New `McpSandboxStack`** (gated by `config.mcpSandbox.enabled`) — S3 mount-page bucket, CloudFront on `mcp-sandbox.{domain}` with a CloudFront Function for dynamic per-resource CSP, Route53 alias. Publishes `/mcp-sandbox/origin` SSM, consumed by inference-api at runtime as `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN`. ACM cert must be in `us-east-1` +- **New `UserMenuLinksTable`** in `InfrastructureStack` + `/admin/user-menu-links-table-name` and `/admin/user-menu-links-table-arn` SSM parameters (#298) +- **New `ArtifactRenderTokenSecret`** in `InfrastructureStack` (Secrets Manager, AWS-managed encryption, `generateSecretString` 64-char) gated on `config.artifacts.enabled`. SSM `/artifacts/render-token-key-arn` publishes the ARN. Lives in `InfrastructureStack` (not `ArtifactsStack`) so app-api can read it without taking a stack-deploy-order dependency on `ArtifactsStack` +- **Inference-api conditionally consumes `mcp-sandbox` SSM** when `config.mcpSandbox.enabled` is true. Mirrors the artifacts conditional-SSM pattern; two synth tests cover present/absent (#349) + +### 🔧 CI/CD + +- **Backup workflow** wired as `workflow_dispatch` against the existing OIDC composite action (#361) +- **All five consumer workflows** now thread `CDK_HOSTED_ZONE_DOMAIN`, `CDK_ARTIFACTS_ENABLED`, `CDK_ARTIFACTS_CERTIFICATE_ARN` so synth-time validation doesn't fail on workflows that don't synth `ArtifactsStack` directly (#307) +- **Frontend build** runs `gen-version.js` explicitly before `ng build` so deployed bundles bake the real version (#336) +- **`infrastructure/test/infrastructure-stack.test.ts`** enumerates the 19 DDB tables instead of asserting `resourceCountIs(18)` (#350) +- **Docker `curl` pin** bumped to `8.14.1-2+deb13u3`; pin policy documented as "follow Debian point-releases" (#327) + +### 📦 Dependency upgrades + +- `bedrock-agentcore` 1.6.4 → 1.9.1 (with coupled `boto3` 1.42.96 → 1.43.9, `botocore` / `s3transfer` following). CHANGELOG audited end-to-end: no breaking changes for our memory/identity usage. Validated with a read-only dev smoke test (memory `get_memory_strategies` / `retrieve_memories` + identity `list_workload_identities`) and the full backend suite. Test-infra side effect: `botocore` 1.43 newly reads `Credentials.account_id` during endpoint construction; on a `RefreshableCredentials` (SSO) object that forces a refresh → `GetRoleCredentials`, which `moto` does not implement. Combined with `backend/src/.env`'s `AWS_PROFILE` leaking via `load_dotenv(override=True)`, this red-ed the suite order-dependently. Added per-test autouse scrub fixtures for `AWS_PROFILE` and the `DYNAMODB_*` / `COGNITO_*` config families, mirroring the existing `_clear_skip_auth_env` fixture for the same `.env`-bleed bug class (#337) +- `strands-agents` 1.39.0 → 1.40.0. Gated on a token-count audit and a compaction double-fire check. `use_native_token_count` default flipped true → false (Strands PR #2284) is inert for our token accounting — the flag gates only `BedrockModel.count_tokens()`, which Strands calls solely from `_estimate_input_tokens()` to populate `projected_input_tokens` on `BeforeModelCallEvent`. Our cost-badge / context-% / compaction-trigger plumbing reads from `inputTokens` + `cacheReadInputTokens` + `cacheWriteInputTokens` directly, so the flip is transparent (#340) + +### 🧪 Test Coverage + +- Backend + frontend regression coverage for `MaxTokensReachedException` classification, the `continue_truncated` resume path, `stream_error` always-allowed parser gating, and the `lastTurnContinuable` refresh-survival marker round-trip (#328) +- Backend regression coverage for adaptive thinking shape per model marker, `effort` allowed-set gating, and the float→int coercion path on `max_tokens` / `top_k` (#329, #330, #331) +- `infrastructure/test/mcp-sandbox-stack.test.ts` (264 lines) — synth + CFN unit coverage including the placeholder-substitution invariants (#343, #355) +- `infrastructure/test/mcp-sandbox-csp-function.test.ts` (357 lines) — `frame-ancestors` quote-escaping, including `'none'` (which would otherwise produce `''none''`, a JS syntax error) (#355) +- `infrastructure/test/inference-api-stack.test.ts` — two synth cases gating `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN` wiring on `config.mcpSandbox.enabled` (#349) +- `infrastructure/test/cors.test.ts` (53 lines) — new CORS test surface +- `infrastructure/test/infrastructure-stack.test.ts` — 19 DDB tables enumerated with one-line justifications instead of count assertion (#350) +- Frontend specs: `mcp-app-bridge`, `mcp-app-card-state.service`, `mcp-app-consent.service`, `mcp-app-message.service`, `mcp-app-proxy.service`, `mcp-app-state.service`, `proxy-url`, `artifact-http.service`, `artifact-state.service`, `artifact-source.component` + +### 📚 Docs + +- `docs/kaizen/scoping/mcp-apps-host-renderer.md` — initial scoping document for the MCP Apps Host Renderer initiative (#296) +- `step-04-deploy.md` — "Register an MCP-Apps-capable MCP server" section with `budget-allocator-server` example + committed `ToolCreateRequest` payload (no auto-seed; registration stays an explicit per-env opt-in) (#349) +- `step-05-verify.md` — manual e2e dogfood scenario exercising all six Definition-of-Done MCP Apps interactions (#349) +- `docs/artifacts/...` — corrected cert-reuse guidance for subdomain primaries (#308) +- `CLAUDE.md` — `ui_resource` SSE row + deploy-order line updated for the live flag and conditional `mcp-sandbox` SSM consumption (#349) +- `.env.example` — documents `BFF_COOKIE_DATA_KEY_SECRET_ARN` (carry-over from beta.25) (#276) +- Architecture rules surfaced for Copilot CLI: 3-package import boundary, inference-api Runtime 404 trap, deploy order, SSE error model. Points to `.kiro/steering` and `.claude/skills` for deeper dives (#361) +- Forward-looking A2A guard: if exposing an A2A server, `AgentCard.capabilities` must include `streaming=True` or clients hang ~40 min (`sample-strands-agent-with-agentcore` commit `50c9112`) (#338) +- Kaizen-2026-05-15 hygiene — replaced dead source URLs in `kaizen-research` (the `bedrock/whats-new/` 404, the `docs.claude.com` claude-code release-notes 301→404, and the inactive `anthropics/courses`); fixed `aws/amazon-bedrock-agentcore-{sdk-python,starter-toolkit}` repo-slug typos to the correct `aws/bedrock-agentcore-*` slugs (#338, #341, #302, #304) + +## [1.0.0-beta.26] - 2026-05-13 + +Small focused release. Multi-sheet XLSX support for the spreadsheet analysis tool, async refactor of the spreadsheet file-lookup path, user default model preference applied at chat time, nightly E2E pipeline restored, and upstream contribution governance (PRs restricted to collaborators, Dependabot version-update PRs disabled). + +### 🚀 Added + +- Multi-sheet XLSX support in the `analyze_spreadsheet` tool. Each sheet converts to its own deterministic CSV (`stem.sheetname.csv`) with a primary alias (`stem.csv`) for the first sheet. Defensive caps via env vars `MAX_SHEETS_TO_CONVERT` and `MAX_ROWS_PER_SHEET` prevent latency blowout and context-window exhaustion on pathological workbooks. Skipped/truncated sheets are surfaced to the model with markdown footers documenting per-sheet conversion status +- `_sanitize_sheet_name()` produces filesystem-safe deterministic CSV filenames; `_parse_sheet_inventory()` extracts structured sheet metadata from bootstrap stdout without `eval`-style evaluation; `_safe_int()` for defensive integer parsing; `_format_sheet_note()` for the per-call markdown footer + +### ✨ Improved + +- `analyze_spreadsheet`, `list_spreadsheets`, `_find_file`, `_get_kb_files`, and `_get_session_files` are now `async def`. Every DynamoDB call is offloaded via `asyncio.to_thread` so the event loop keeps scheduling other coroutines for the full round-trip duration +- `inference_api/chat/routes.py::_build_tabular_inventory` is now `async` and awaits the file-operation calls directly, replacing the nested `asyncio.run` + thread pool executor pattern that could deadlock under concurrent chat load. Closes the regression introduced in #260 +- `analyze_tool` code generation stashes the filename as a `_FNAME` variable inside the generated snippet to prevent f-string interpolation conflicts when filenames contain quotes or special characters (`repr()` indirection in `_build_preview_code`) +- `_clean_stderr` now respects the `MAX_ERROR_CHARS` budget strictly, accounting for ellipsis length + +### 🐛 Fixed + +- User-saved default model preference (`defaultModelId` in user settings) is now applied at chat time when the request doesn't specify a `model_id`. Previously the persisted preference was silently ignored and chat fell back to the hardcoded factory default. RBAC is re-checked on the resolved default to prevent access to permissions that have since been revoked. A missing user-settings table now surfaces as `503` instead of silently dropping the user choice. Fixes #161 +- Nightly E2E pipeline failures from cookie/JWT validation against the dynamic CloudFront URL, missing CDK certificate ARN in the nightly job, agent test timeouts on multi-tool turns, and cross-region Bedrock model routing flakes (switched the suite from global to US-region model IDs) (#290) + +### 📚 Docs + +- `backend/src/.env.example` — BFF cookie encryption documentation updated to reflect the beta.25 shift from direct KMS cookie encryption to Secrets Manager-mediated approach. Documents the new `BFF_COOKIE_DATA_KEY_SECRET_ARN` variable, the SHA-256 cross-task derivation, and the SSM parameter path with example ARN format + +### 🔧 CI/CD + +- Nightly E2E pipeline restored after multi-attempt fix (#290): CloudFront URL handling, CDK certificate ARN wiring, agent test timeout bumps, US-region Bedrock model IDs, rebase on develop to pick up #248 + +### 🛡️ Governance + +- **CONTRIBUTING.md** documents that pull requests are restricted to approved collaborators (GitHub "Collaborators only" setting). Issues remain open to everyone; maintainers triage and either implement upstream or coordinate next steps with the reporter. Adds collaborator checklist (link tracking issue, single logical change per PR, DCO sign-off, green CI, respect backend import boundaries enforced by `backend/tests/architecture/test_import_boundaries.py`) (#293) +- **`.github/dependabot.yml`** — `open-pull-requests-limit: 0` across all four ecosystems (pip, frontend npm, infrastructure npm, github-actions). Disables scheduled version-update PRs; security updates are unaffected and will still be raised when a CVE is published. Existing groups, labels, schedules retained for easy reversal (#293) + +### 🧪 Test Coverage + +- `backend/tests/agents/builtin_tools/spreadsheet_analysis/` — 2,800+ lines of new tests across 8 files. Notable: `test_analyze_tool_integration.py` (779 lines, multi-sheet XLSX + CSV workflows end-to-end), `test_sheet_inventory.py` (307 lines, parser robustness against malformed bootstrap output), `test_clean_stderr.py` (202 lines, strict error-char budget), `test_build_preview_code.py` (127 lines, filename escaping), plus `test_helpers.py`, `test_find_file.py`, `test_list_spreadsheets.py`, `test_strip_first_row.py` +- `frontend/ai.client/src/app/session/services/model/model.service.spec.ts` (56 lines) — default-model resolution flow +- `frontend/ai.client/src/app/settings/pages/chat-preferences/chat-preferences-settings.page.spec.ts` (101 lines) — Chat Preferences settings UI + +## [1.0.0-beta.25] - 2026-05-11 + +Production-readiness fix for the BFF Token Handler shipped in beta.24. Fixes three production-breaking bugs introduced by beta.24: event-loop-blocking sync boto3 on every cookie-bearing request, per-process AES-256 keys that can't round-trip cookies across ECS tasks, and an in-process-only refresh lock that races Cognito rotation across replicas. Also ships PDF thumbnails, rich attachment previews, spreadsheet analysis tools, centralized 401 handling, and a `SKIP_AUTH` local-dev bypass. + +### 🐛 Fixed + +- **Critical (beta.24 regression):** `SessionRefreshMiddleware` ran sync boto3 (DynamoDB + Cognito) on the uvicorn event loop so Angular's ~8-endpoint page-load fan-out produced ~16 serialized blocking AWS calls per user per minute. Observable as ALB 504s, 15.6s p-max `TargetResponseTime` at 0.7% CPU, `/files/quota` outliers reaching ~80s. Every boto3 call in `SessionRepository` and `CognitoRefreshClient.refresh` now offloads via `asyncio.to_thread`; `_resolve_session` is wrapped in a per-session `asyncio.Future` single-flight so N concurrent same-session callers share one loader invocation; `_maybe_slide` dispatches `touch_last_seen` as a detached `asyncio.Task` (with strong reference on the middleware to prevent GC); `_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS` raised 60s → 300s to de-align from the 60s refresh-leeway window (#264) +- **Critical (beta.24 regression):** `CookieCodec` called `kms:GenerateDataKey` on first use per process, so each app-api task minted its own random AES-256 key. Once `desiredCount` went above 1, cookies sealed on Task A failed as `bad seal` on Task B (~50% of requests). Data key is now generated once via Secrets Manager `generateSecretString` (44-char, ~261 bits entropy) encrypted at rest with the existing `BFFCookieSigningKey` CMK; `CookieCodec._ensure_cipher` reads the secret and derives the AES-256 key via SHA-256; `kms:GenerateDataKey` dropped from the runtime task role (#273, #274) +- **Critical (beta.24 regression):** In-process `single_flight` and `get_session_lock` only coalesce same-session callers within one Python process. Under multi-replica, two tasks could each call `cognito-idp:initiate_auth` with the same refresh token; Cognito rotates on the winner and the loser silently logs the user out. New DDB conditional-write lock (`try_acquire_refresh_lock` / `release_refresh_lock` on `BFFSessionsTable`, reusing the existing `dynamodb:UpdateItem` grant) elects exactly one leader fleet-wide; followers poll the row and adopt the leader's tokens. `update_tokens` gains strict-owner condition (`refresh_lock_owner = :owner`) that atomically `REMOVE`s the lock attrs on successful persist and rejects stale-leader stomps via `ConditionalCheckFailedException`. Absolute-lifetime guard added ahead of lock acquisition so we don't burn a Cognito refresh on a row that's about to TTL-evict (#273, #275) +- Per-message cost double-count on tool-use turns — Strands' `AgentResultEvent` cumulative `accumulated_usage` overwrote the last assistant message's per-call usage via `.update()`. Route the result-extracted cumulative on the `metadata_summary` turn-summary track instead of `metadata` (#270) +- Context-% inflation within a tool turn — Bedrock reports each per-LLM-call `inputTokens` as the full context sent on that call, so Strands' summed `accumulated_usage` over-reports. `stream_coordinator` no longer accumulates `metadata_summary` into `accumulated_metadata`; per-call `metadata` last-write-wins so the value equals the most recent call's full input = current context. Summed across `inputTokens` + `cacheReadInputTokens` + `cacheWriteInputTokens` since `AgentResult.context_size` under-reports by 99%+ under prompt caching (#270) +- `LatencyMetrics.time_to_first_token` changed from `int` (placeholder 0) to `Optional[int]` (placeholder `null`) — a real TTFT can't be 0ms and aggregations need to distinguish absence from a real value (#270) +- Session-expired mid-session left users stranded with a generic toast or no feedback on SSE. Every 401 now flows through `SessionService.handleUnauthorized()`, which dedupes concurrent calls and navigates once with preserved `returnUrl` (#277) +- Session loss not surfaced until the next HTTP call failed. Added cookie-presence fast-path (JS-readable `__Host-bff_csrf` cookie absence implies `__Host-bff_session` also gone) and visibility re-probe on tab refocus (#277) +- Login & first-boot lava-lamp backdrop dark-mode CSS never applied on cold load — `html.dark .X` selectors don't match under Angular's emulated view encapsulation, and `ThemeService` was never injected in the pre-auth tree. Switched to `:host-context(html.dark) .X` and forced `ThemeService` construction via `provideAppInitializer` (#271) +- XLSX→CSV filename mismatches in the Code Interpreter sandbox triggered retry loops. Targeted error hints, tolerant filename matching for CSV↔XLSX aliasing, schema footer preservation on errors + +### 🚀 Added + +- Server-rendered PDF page-1 thumbnails on attachment cards. New `ThumbnailRenderer` MIME-dispatcher (PDF today via `pypdfium2`, lazy-cached `_thumb.png` sibling in S3, render runs in `loop.run_in_executor`); new `GET /files/{upload_id}/thumbnail` returning a short-lived presigned URL; single-file + session-cascade deletes clean up thumbnails. Frontend: `FileUploadService.getThumbnail()` returns a typed `ready` / `unsupported` / `unavailable` result; PDF badge renders `object-cover` (#263) +- Rich previews in user messages — iMessage-style image mosaic (1-bubble / 2-col / 1+2 split / 2×2 / 5+ with `+N` overlay) with full-screen lightbox + arrow-key navigation; document-style cards for non-images with tinted header + folded corner + content excerpt. New `GET /files/{upload_id}/preview-url` and `GET /files/{upload_id}/text-snippet` (first 2KB UTF-8) (#254) +- Inline markdown preview for `.md` files in attachment cards; full-screen modal viewer via `ngx-markdown` instead of opening raw source in a new tab (#262) +- Spreadsheet analysis tools — `list_spreadsheets` enumerates CSV/XLSX across KB + attachments (with size + MIME metadata); `analyze_spreadsheet` runs Python analysis in Code Interpreter with schema detection (skiprows probing), cleaned pandas/numpy tracebacks, and 10K/600-char output/error truncation. Injected per-request via `extra_tools` (#f88ce7ec, #0ab90bb1) +- `SKIP_AUTH=true` local-dev bypass in `apis.shared.auth.dependencies` returns a fake admin user from all three auth dependencies. Optional tuning: `SKIP_AUTH_ROLES`, `SKIP_AUTH_USER_ID`, `SKIP_AUTH_EMAIL`. Startup guard in `app_api/main.lifespan` refuses to boot when `SKIP_AUTH=true` is paired with any non-localhost entry in `CORS_ORIGINS`. Inference-api intentionally not bypassed (all SPA traffic flows through app-api) (#272) +- New CI workflow `.github/workflows/skip-auth-guard.yml` greps CDK source, workflow files, and Dockerfiles for `SKIP_AUTH=true` / `SKIP_AUTH: true` patterns and fails the build if any leak into deployed config. SHA-pinned `actions/checkout`, `ubuntu-24.04` (#272) +- `SessionRepository.try_acquire_refresh_lock(session_id, owner, lock_ttl_seconds)` and `release_refresh_lock(session_id, owner)` for cross-task refresh coalescing (#273, #275) +- `apis/shared/sessions_bff/single_flight.py` — new `resolve_once(session_id, loader_coro_factory)` primitive for in-process coalescing of the session-resolve path (#264) +- CAUTION comment in `stream_coordinator` documenting that `AgentResult.context_size` / `EventLoopMetrics.latest_context_size` return only `inputTokens`, under-reporting by 99%+ under prompt caching (#270) + +### ✨ Improved + +- File metadata utilities (`backend/src/apis/shared/files/models.py`) for consistent attachment handling — `FileMetadata`, `FileContent`, size formatting, MIME-type inference — shared between routes and the chat-input component +- Spreadsheet-analysis system prompt clarifies filename vs. sandbox-path handling; tool docstrings expanded with critical guidance on retries +- Stream processor error handling for Code Interpreter responses is more defensive +- Updated `test_session_refresh_preservation.py`'s `InstrumentedTable` to differentiate lock-acquire / token-persist / slide writes so `update_item_side_effect` injection only fires on the persist path (preserving original test intent) (#273) + +### 🔒 Security + +- `kms:GenerateDataKey` and `kms:DescribeKey` dropped from the app-api runtime task role (least privilege). Only `kms:Decrypt` remains, invoked by Secrets Manager on the caller's behalf when reading the CMK-encrypted `BFFCookieDataKeySecret` (#274) +- `SKIP_AUTH=true` gated by boot-time CORS-origin allowlist + CI guard workflow; fails closed for any deploy target we haven't anticipated instead of blocklisting known cloud env vars (#272) + +### ⚡ Performance + +- `SessionRefreshMiddleware` resolve path now coalesces Angular's ~8-endpoint page-load fan-out to 1 `get_item` and 0 `update_item` on the critical path (previously ~16 serialized blocking AWS calls per user per minute). Response latency independent of `touch_last_seen` DDB latency after the `_maybe_slide` fire-and-forget refactor (#264) +- `CookieCodec` initialization dropped from `kms:GenerateDataKey` + per-cold-start round trip to a one-shot Secrets Manager `GetSecretValue` + local SHA-256. No more per-task cold-start KMS call (#274) +- Thumbnail render runs in `loop.run_in_executor` so the request worker isn't blocked; lazy `_thumb.png` sibling in S3 means steady-state thumbnails are a HEAD + presign, not a render (#263) + +### 🏗️ Infrastructure + +- New `BFFCookieDataKeySecret` (Secrets Manager, encrypted with `BFFCookieSigningKey` CMK); SSM parameter `/${projectPrefix}/auth/bff-cookie-data-key-secret-arn` publishes the ARN +- App-api task role: added `secretsmanager:GetSecretValue` on the new secret; removed `kms:GenerateDataKey` and `kms:DescribeKey` on `BFFCookieSigningKey`; kept `kms:Decrypt` +- `appApi.desiredCount` raised 1 → 2 — concurrency slack so a single blocked event loop can no longer halt all ingress + +### 📦 Dependencies + +- Backend: `strands-agents` 1.37.0 → 1.39.0, `strands-agents-tools` 0.5.1 → 0.5.2, new: `pypdfium2` (#265, #263) + +### 🧪 Test Coverage + +- `tests/apis/shared/middleware/test_session_refresh_bug_condition.py` (12 cases) — encodes the seven sub-conditions of the event-loop-blocking bug as Hypothesis properties. Fails on unfixed code (by design); passes on fixed code (#264) +- `tests/apis/shared/middleware/test_session_refresh_preservation.py` (19 cases) — locks in 11 preservation invariants that must remain unchanged for non-buggy inputs (#264) +- `tests/apis/shared/sessions_bff/test_single_flight.py` (6 cases) — primitive-level coverage for the new `resolve_once` module (#264) +- `tests/apis/shared/sessions_bff/test_session_refresh_cross_task.py` (480 lines) — two-task integration coverage over moto DDB for the cross-task refresh lock, follower-polling/adoption, TTL recovery, headline invariant that two tasks racing in parallel call Cognito at most once (#273) +- 8 new repository tests for the lock primitive (acquire on unlocked row, contention blocks peer, TTL recovery, distinct-session isolation, release-by-owner-only, atomic clear on token persist, condition fails when peer owns the lock, phantom-row-prevention on acquire, strict-owner release condition, absolute-lifetime guard ahead of refresh) (#273, #275) +- `tests/agents/main_agent/streaming/test_per_message_cost_attribution.py` — three regression cases for the `metadata` vs `metadata_summary` contract; two parametrized cases for `stream_coordinator` current-context semantics including all-three-buckets-summed under cache-read/write (#270) +- `tests/costs/test_calculator.py` — 26 cases of direct coverage for `CostCalculator` (per-bucket pricing, cache scenarios against Sonnet 4.5 rates, defensive missing-key / None handling, `calculate_cache_savings`, `validate_*` predicates) (#270) +- `tests/auth/test_skip_auth.py` — `SKIP_AUTH` dependency-bypass + env-override coverage, startup guard allowlist behavior, skip-auth-guard.yml regex matches (#272) +- Session-wide autouse fixture in `tests/conftest.py` scrubs `SKIP_AUTH_*` env so developer `.env` bleed doesn't silently turn on the bypass in test runs (#272) +- Infrastructure-stack tests: dropped bootstrap-custom-resource assertions; added negative lock that no `AwsCustomResource` emits `kms:GenerateDataKey` / `secretsmanager:PutSecretValue`; positive assertion on `generateSecretString` shape (44-char, no punctuation, no space); fixed two pre-existing stale resource-count assertions (16→18 DDB tables, 3→6 secrets) (#273, #274) + ## [1.0.0-beta.24] - 2026-05-06 ### 🚀 Added diff --git a/CLAUDE.MD b/CLAUDE.MD index f3f0f754..356e2143 100644 --- a/CLAUDE.MD +++ b/CLAUDE.MD @@ -32,7 +32,7 @@ npx cdk deploy --all ## Key Conventions -- **Deploy order:** Infrastructure → Gateway → Inference API → App API → Frontend (App API reads `runtime-workload-identity-name` from SSM, published by Inference API) +- **Deploy order:** Infrastructure → (Gateway, RAG Ingestion, SageMaker Fine-Tuning, Artifacts, MCP Sandbox — parallel-safe) → Inference API → App API → Frontend (App API reads `runtime-workload-identity-name` from SSM, published by Inference API; Inference API + App API + Frontend conditionally consume `/{prefix}/artifacts/*` SSM params when `CDK_ARTIFACTS_ENABLED=true`; Inference API conditionally consumes `/{prefix}/mcp-sandbox/origin` into `AGENTCORE_MCP_APPS_SANDBOX_ORIGIN` when `CDK_MCP_SANDBOX_ENABLED=true`) - **Admin endpoints** go under `/admin//`, user-facing under `//` - **Errors stream as assistant messages** via SSE (not HTTP error codes) - **Signal-based state** throughout frontend (`signal()`, `computed()`) @@ -47,6 +47,7 @@ npx cdk deploy --all | `content_block_start/delta/stop` | Streaming content | | `message_stop` | End of message | | `tool_use` / `tool_result` | Tool invocation and result | +| `ui_resource` | MCP App UI for a tool result (SEP-1865) — payload `{type, toolUseId, resourceUri, html, mimeType, csp, permissions, sandboxOrigin}`, emitted right after the correlated `tool_result` when the tool declared a `ui://` resource. HTML fetched server-side via `resources/read` and inlined; `sandboxOrigin` is the proxy.html origin the SPA frames it in (empty unless the mcp-sandbox stack is deployed — inference-api consumes its SSM origin only when `CDK_MCP_SANDBOX_ENABLED=true`; an empty origin means the SPA cannot frame the App). Gated by `AGENTCORE_MCP_APPS_HOST_ENABLED` (default true since PR #7; set `=false` to opt an environment out) | | `stream_error` | Conversational error | | `oauth_required` | External MCP tool needs user consent — payload `{providerId, authorizationUrl}`, one event per provider emitted after `message_stop` | | `compaction` | Backend rolled older turns into a summary on this turn — payload `{previousCheckpoint, newCheckpoint, summarizedTurns, inputTokens}`, emitted after the final `metadata` event so the badge updates first, before `done` | @@ -61,6 +62,8 @@ npx cdk deploy --all | MCP + SigV4 | Cloud Lambda (Gateway) | AWS SigV4 | | A2A | Cloud Runtime | AgentCore auth | +Today A2A is **client-only** — `A2AAgentConfig` (`apis/shared/tools/models.py`) describes remote agents we call out to; we do not yet expose an A2A server / `AgentCard`. **When the first A2A server construct lands** (Strands `agent.to_a2a()`, an `A2AServer`, or a hand-built `AgentCard`), its advertised `capabilities` MUST include `streaming=True`. Without it the A2A SDK client silently falls back to non-streaming, never receives a `completed` event, and hangs until its ~40-minute timeout (ref-repo `aws-samples/sample-strands-agent-with-agentcore` commit `50c9112`). + ## Cross-Package Contracts - Backend route handlers define the API shape; frontend TypeScript interfaces must match @@ -76,7 +79,8 @@ npx cdk deploy --all | New admin endpoint | `backend/src/apis/app_api/admin//` | | New agent tool | `backend/src/agents/main_agent/tools/` + register in `__init__.py` | | New Angular page | `frontend/ai.client/src/app//` | -| New CDK stack | `infrastructure/lib/-stack.ts` | +| New CDK stack | `infrastructure/lib/-stack.ts` (also: register in `test/stack-dependencies.test.ts` with a tier, add scripts in `scripts/stack-/`, add a workflow in `.github/workflows/`, update `step-04-deploy.md`) | +| New Lambda for an infra stack | `backend/src/lambdas//` (one folder per Lambda; not part of the `apis/` import boundary) | | Shared backend code | `backend/src/apis/shared//` | ### Inference API boundary @@ -87,6 +91,14 @@ The `inference-api` runs inside an AgentCore Runtime container. The runtime data If you're tempted to add to inference-api because of an existing route there (`converse_router`, `voice_router`), don't use them as templates without confirming their access path — they predate this rule and may rely on bypasses (API key, WebSocket upgrade, direct container reach in environments where the runtime isn't the only path). +### Auth dependency on app_api routes + +The SPA sends an httpOnly session cookie — not `Authorization: Bearer`. A route that declares a bare Bearer-only dependency on the SPA-facing surface causes a 401 → centralized redirect loop the moment the SPA hits it. + +**Rule:** New routes under `apis/app_api/` use `Depends(get_current_user_from_session)` from `apis.shared.auth.dependencies` for user authentication. Admin routes use `Depends(require_admin)` (which chains through the same cookie dependency). The only exceptions are the API-key feature (`auth/api_keys/`, uses `X-API-Key`) and voice mode (`voice/`, uses a voice-ticket cookie) — both handle auth on their own terms and should not be used as templates for ordinary user routes. + +Do not reintroduce a Bearer-only `Depends(...)` on any user-facing route. If you find one in older code, migrate it to `get_current_user_from_session`. + ## Debugging Quick Reference - **Tool not appearing:** Check `__init__.py` export, RBAC permissions, `enabled_tools`, ToolRegistry diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 18a4b50c..6ecfe347 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,5 +1,39 @@ # Contributing to AgentCore Public Stack +## Contribution Policy + +AgentCore Public Stack is maintained by Boise State University as a reference +implementation for academic and public-sector AgentCore deployments. It is +source-available under the PolyForm Noncommercial License 1.0.0 (see +[`LICENSE`](./LICENSE)). + +### Pull requests are restricted to approved collaborators + +To keep the reference architecture coherent and to let downstream deployments +stay in sync with a single, well-known upstream, this repository uses GitHub's +**"Collaborators only"** pull request setting. Only users with Write access or +higher can open a pull request. + +### Reporting issues and proposing changes + +If you are deploying this stack and find a bug, regression, or documentation +gap, please open a GitHub issue — issues are open to everyone. A maintainer +will triage the report, and if the change belongs upstream we will either +implement it or coordinate with the reporter on next steps. + +### For collaborators + +- Link the tracking issue in the PR description so changes stay discoverable. +- Keep each PR focused on a single logical change. +- Sign off your commits with `git commit -s` (Developer Certificate of Origin). +- Make sure CI is green before requesting review. +- Respect the backend import boundaries enforced by + `backend/tests/architecture/test_import_boundaries.py` — `app_api`, + `inference_api`, and `agents/` are independent consumers of `apis.shared` + and must not import from each other. + +--- + ## Prerequisites - **Node.js** 20+ (for frontend and infrastructure) diff --git a/README.md b/README.md index c9db526f..e0089660 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **An open-source, production-ready Generative AI platform for institutions** *Built by Boise State University, designed for everyone.* -[![Release](https://img.shields.io/badge/Release-v1.0.0--beta.24-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) +[![Release](https://img.shields.io/badge/Release-v1.0.0--beta.28-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) [![Nightly](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml) ![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=flat&logo=python&logoColor=white) @@ -204,6 +204,7 @@ The fastest path to production is the **GitHub Actions pipeline**, which automat |-----------|-------------|---------| | Networking | VPC, ALB, Security Groups | Isolated network with load balancing | | Fine-Tuning *(optional)* | SageMaker, S3, DynamoDB | Model training, batch inference, artifact storage | +| Artifacts *(optional)* | DynamoDB, S3, CloudFront, Lambda | Iframe-isolated rendering for agent-generated HTML/code artifacts | | RAG Ingestion | Lambda, S3 | Document ingestion for retrieval-augmented generation | | Inference API | Bedrock Agentcore | Agent orchestration with Bedrock | | App API | ECS Fargate | Authentication, admin, session management | @@ -248,7 +249,8 @@ agentcore-public-stack/ │ └── services/ # State management ├── infrastructure/ # AWS CDK stacks │ └── lib/ # Infra, App API, Inference API, Frontend, -│ # Gateway, RAG Ingestion, SageMaker Fine-Tuning +│ # Gateway, RAG Ingestion, SageMaker Fine-Tuning, +│ # Artifacts └── .github/ ├── workflows/ # CI/CD pipelines └── docs/deploy/ # Deployment guides @@ -260,7 +262,7 @@ agentcore-public-stack/ See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full changelog, including new features, bug fixes, platform upgrades, and deployment notes for each release. -**Current release:** v1.0.0-beta.24 +**Current release:** v1.0.0-beta.28 --- diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 46d94619..bd182350 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,13 +1,699 @@ -# Release Notes — v1.0.0-beta.24 +# Release Notes — v1.0.0-beta.27 -**Release Date:** May 6, 2026 -**Previous Release:** v1.0.0-beta.23 (April 29, 2026) +**Release Date:** May 20, 2026 +**Previous Release:** v1.0.0-beta.26 (May 13, 2026) --- ## Highlights -This release lands the **BFF Token Handler** — a ground-up rewrite of the SPA's auth surface. `localStorage` Bearer tokens are replaced with server-side Cognito session storage keyed by an opaque session id in a KMS-sealed AES-GCM cookie, the public PKCE Cognito client is decommissioned in favor of a confidential client whose secret never leaves the server, and same-origin `/api/*` routing via CloudFront enables `__Host-` cookies, double-submit CSRF, and eliminates the CORS preflight from every chat turn. **Voice mode returns** via a WebSocket-ticket proxy on app-api. The chat view gains a **per-conversation cost + context-window badge** with write-time aggregation, and **context compaction events** now surface inline with refresh-survival. Anthropic **extended thinking** is wired end-to-end via per-model inference parameters. The backend finishes its architecture cleanup: cost, tools, storage, and API-keys modules now live under `apis.shared` with AST-enforced import boundaries. +The largest release since the BFF cutover. Beta.27 lands two new user-visible surfaces, both built on top of brand-new CDK stacks, plus a major admin redesign and a handful of inference-API correctness fixes. + +- **Artifacts** — the agent can now produce versioned, iframe-isolated HTML, Markdown, and code artifacts that render in a docked side panel beside the chat. Backed by a new `ArtifactsStack` (S3 + DynamoDB + render Lambda + CloudFront on `artifacts.{domain}`) and short-lived JWT render tokens minted by app-api. +- **MCP Apps host renderer** — third-party MCP servers can ship UI alongside their tools. The agent advertises a UI extension on `initialize`, fetches `ui_resource` payloads via `resources/read`, and the SPA frames them in a sandboxed `` over a strict CSP, with an app-initiated `tools/call` proxy and explicit user consent. Backed by a new `McpSandboxStack` (CloudFront origin on `mcp-sandbox.{domain}` with dynamic per-resource CSP via a CloudFront Function). Default-on this release. +- **Admin shell redesign** — the 15-card admin grid is replaced with a persistent grouped sidebar, and dense list redesigns for models and tools turn cards into compact expandable rows. Quotas and Fine-Tuning collapse from seven sibling routes into two tabbed pages. +- **Recoverable `max_tokens` truncation** — what used to be a leaky, infinite-looping `MaxTokensReachedException` is now an inline "Response length limit reached" notice with a Continue button that resumes the truncated turn instead of resending the prompt. Survives a page refresh. +- **Model-aware adaptive thinking** — Opus 4.7's 400 on `thinking.type=enabled` is fixed: Opus 4.6/4.7, Sonnet 4.6, and Mythos now emit `{type: adaptive, display: summarized}` and depth is governed by a new admin- and user-configurable `effort` knob. Older models keep the legacy `enabled` shape. +- **`/ping` reaper fix** — fixes silent mid-stream microVM reaping by emitting the integer `time_of_last_update` field AgentCore's idle reaper requires. Workaround for `bedrock-agentcore-sdk-python#471` until async-task busy tracking lands. +- **Pre-migration backup tool** — `scripts/backup-data/` produces a complete, restore-friendly snapshot of all DynamoDB tables, user-content S3 buckets, and Cognito (config + users + groups + IdPs + plaintext app-client secrets) for a given `CDK_PROJECT_PREFIX`. Workflow-dispatch wired. +- **Dependency upgrades** — `bedrock-agentcore` 1.6.4 → 1.9.1 (with coupled `boto3` 1.42.96 → 1.43.9) and `strands-agents` 1.39.0 → 1.40.0. + +This release adds two new CDK stacks (`ArtifactsStack`, `McpSandboxStack`) and one new DynamoDB table (`user-menu-links`). Both new stacks are gated by config flags. Deploy order matters — see "Deployment notes" below. + +--- + +## Artifacts + +The agent can now author versioned standalone documents — HTML pages, charts, Markdown reports — that render in a sandboxed iframe alongside the chat. Artifacts solve two problems the existing `create_visualization` and Code Interpreter outputs couldn't: persistence (the user can re-open and download), and isolation (HTML/JS runs in a cross-origin sandbox so it can't read cookies or the SPA DOM). + +### Architecture + +A new leaf stack, `ArtifactsStack`, owns the rendering pipeline: + +- **DynamoDB `user-artifacts` table** — version log + HEAD pointer per artifact. PK `USER#{user_id}`, SK `ARTIFACT#{aid}#V#{version:05d}` for versions and `ARTIFACT#{aid}#HEAD` for the latest pointer. GSI1 indexes by `SESSION#{session_id}` so the SPA can list artifacts produced in the current chat. +- **S3 `artifacts-content` bucket** — private, no CORS. Layout `{user_id}/{aid}/v{n}/index.html`. Versions are immutable: there's no `s3:DeleteObject` grant on the inference-api role, so an `update_artifact` writes a new version and re-points HEAD instead of mutating. +- **Render Lambda** — validates a render-token JWT scoped to one `(artifact_id, version)`, fetches the blob from S3, and returns it with a strict per-origin CSP that allows inline ` + + +
Rendering…
+ + + + +""" + + +def _is_markdown(content_type: Optional[str]) -> bool: + """True for a Markdown MIME type, ignoring any `; charset=` suffix.""" + bare = (content_type or "").split(";")[0].strip().lower() + return bare in _MARKDOWN_MIME_TYPES + + +def _wrap_markdown(title: str, markdown: str) -> str: + """Render Markdown source into a self-contained HTML document.""" + md_b64 = base64.b64encode(markdown.encode("utf-8")).decode("ascii") + return _MARKDOWN_RENDER_TEMPLATE.replace( + "__ARTIFACT_TITLE__", html.escape(title or "Markdown document") + ).replace("__ARTIFACT_MD_B64__", md_b64) + + +_cached_bucket: Optional[str] = None +_cached_table: Optional[str] = None +_ssm_client = None +_s3_client = None +_ddb_resource = None + + +class ArtifactError(Exception): + """Base class for artifact write failures.""" + + +class ArtifactNotFoundError(ArtifactError): + """Update target does not exist for this user.""" + + +class ArtifactConfigError(ArtifactError): + """Artifacts feature is not configured for this environment.""" + + +def _reset_caches_for_tests() -> None: + global _cached_bucket, _cached_table, _ssm_client, _s3_client, _ddb_resource + _cached_bucket = None + _cached_table = None + _ssm_client = None + _s3_client = None + _ddb_resource = None + + +def _region() -> str: + return ( + os.environ.get("AWS_REGION") + or os.environ.get("AWS_DEFAULT_REGION") + or "us-west-2" + ) + + +def _resolve(env_var: str, ssm_suffix: str) -> str: + """Env var first, then SSM under the runtime's PROJECT_PREFIX. + + inference-api exposes PROJECT_PREFIX and holds ssm:GetParameter on + `/{prefix}/*`, so the artifacts params published by the artifacts + stack are readable without any extra wiring.""" + value = os.environ.get(env_var) + if value: + return value + global _ssm_client + prefix = os.environ.get("PROJECT_PREFIX") + if not prefix: + raise ArtifactConfigError( + f"{env_var} unset and PROJECT_PREFIX unavailable" + ) + if _ssm_client is None: + _ssm_client = boto3.client("ssm", region_name=_region()) + try: + resp = _ssm_client.get_parameter( + Name=f"/{prefix}/artifacts/{ssm_suffix}" + ) + except ClientError as exc: + raise ArtifactConfigError( + f"artifacts {ssm_suffix} parameter not found" + ) from exc + return resp["Parameter"]["Value"] + + +def _bucket_name() -> str: + global _cached_bucket + if _cached_bucket is None: + _cached_bucket = _resolve("S3_ARTIFACTS_BUCKET_NAME", "bucket-name") + return _cached_bucket + + +def _table(): + global _cached_table, _ddb_resource + if _cached_table is None: + _cached_table = _resolve("DYNAMODB_ARTIFACTS_TABLE_NAME", "table-name") + if _ddb_resource is None: + _ddb_resource = boto3.resource("dynamodb", region_name=_region()) + return _ddb_resource.Table(_cached_table) + + +def _s3(): + global _s3_client + if _s3_client is None: + _s3_client = boto3.client("s3", region_name=_region()) + return _s3_client + + +def _now_iso() -> str: + return datetime.now(timezone.utc).isoformat() + + +def _put_object(user_id: str, artifact_id: str, version: int, + content: str, content_type: str, title: str) -> str: + key = f"{user_id}/{artifact_id}/v{version}/index.html" + if _is_markdown(content_type): + body = _wrap_markdown(title, content) + object_content_type = _RENDERED_CONTENT_TYPE + else: + body = content + object_content_type = content_type + _s3().put_object( + Bucket=_bucket_name(), + Key=key, + Body=body.encode("utf-8"), + ContentType=object_content_type, + ) + return key + + +def create_artifact_record( + user_id: str, + session_id: str, + title: str, + content: str, + content_type: str, +) -> tuple[str, int]: + """Create v1 of a new artifact. Returns (artifact_id, version).""" + artifact_id = uuid.uuid4().hex + version = 1 + content_type = content_type or _DEFAULT_CONTENT_TYPE + now = _now_iso() + content_key = _put_object( + user_id, artifact_id, version, content, content_type, title + ) + + pk = f"USER#{user_id}" + common = { + "storage": "s3", + "content_key": content_key, + "content_type": content_type, + "version": version, + "artifact_id": artifact_id, + "user_id": user_id, + "session_id": session_id, + "title": title, + "created_at": now, + } + table = _table() + try: + table.put_item( + Item={ + **common, + "PK": pk, + "SK": f"ARTIFACT#{artifact_id}#V#{version:05d}", + "updated_at": now, + }, + ConditionExpression="attribute_not_exists(SK)", + ) + table.put_item( + Item={ + **common, + "PK": pk, + "SK": f"ARTIFACT#{artifact_id}#HEAD", + "updated_at": now, + "GSI1PK": f"SESSION#{session_id}", + "GSI1SK": f"ARTIFACT#{now}#{artifact_id}", + }, + ConditionExpression="attribute_not_exists(SK)", + ) + except ClientError as exc: + raise ArtifactError("failed to write artifact metadata") from exc + + logger.info( + "created artifact user=%s artifact=%s v=%s session=%s", + user_id, artifact_id, version, session_id, + ) + return artifact_id, version + + +def update_artifact_record( + user_id: str, + artifact_id: str, + content: str, + title: Optional[str], + content_type: Optional[str], +) -> int: + """Append a new immutable version and re-point HEAD. Returns version.""" + pk = f"USER#{user_id}" + table = _table() + try: + head = table.get_item( + Key={"PK": pk, "SK": f"ARTIFACT#{artifact_id}#HEAD"} + ).get("Item") + except ClientError as exc: + raise ArtifactError("artifact metadata lookup failed") from exc + if not head: + raise ArtifactNotFoundError(artifact_id) + + current = int(head["version"]) + version = current + 1 + title = title or head.get("title", "") + content_type = content_type or head.get("content_type") or _DEFAULT_CONTENT_TYPE + now = _now_iso() + content_key = _put_object( + user_id, artifact_id, version, content, content_type, title + ) + + common = { + "storage": "s3", + "content_key": content_key, + "content_type": content_type, + "version": version, + "artifact_id": artifact_id, + "user_id": user_id, + "session_id": head.get("session_id", ""), + "title": title, + "created_at": head.get("created_at", now), + } + try: + table.put_item( + Item={ + **common, + "PK": pk, + "SK": f"ARTIFACT#{artifact_id}#V#{version:05d}", + "updated_at": now, + }, + ConditionExpression="attribute_not_exists(SK)", + ) + # Optimistic lock: HEAD must still be at the version we read, so + # two concurrent updates can't silently clobber each other. + table.put_item( + Item={ + **common, + "PK": pk, + "SK": f"ARTIFACT#{artifact_id}#HEAD", + "updated_at": now, + "GSI1PK": f"SESSION#{head.get('session_id', '')}", + "GSI1SK": f"ARTIFACT#{now}#{artifact_id}", + }, + ConditionExpression="version = :cur", + ExpressionAttributeValues={":cur": current}, + ) + except ClientError as exc: + code = exc.response.get("Error", {}).get("Code", "") + if code == "ConditionalCheckFailedException": + raise ArtifactError( + "artifact changed concurrently; retry the update" + ) from exc + raise ArtifactError("failed to write artifact metadata") from exc + + logger.info( + "updated artifact user=%s artifact=%s v=%s", user_id, artifact_id, version + ) + return version + + +def set_produced_by_message_index( + user_id: str, artifact_id: str, version: int, message_index: int +) -> None: + """Stamp the artifact's version row (and HEAD) with the index of the + assistant message that produced this version this turn. + + Per-version linkage is what lets the SPA place every version's card + under the turn that produced it after a reload — the list endpoint + returns all version rows, not just HEAD. HEAD is stamped too so any + HEAD-only reader still sees the latest version's linkage. + + The artifact tool can't know this index at write time — the turn + isn't finished — so the stream coordinator writes it back post-turn + using the same odd-position index it already computes for per-message + metadata (`initial_message_count + 2*i + 1`). That index matches the + `idx` the messages endpoint enumerates on reload. + + Best-effort: a SET on a single attribute that deliberately does not + touch `version`, so it can never collide with the update_artifact + optimistic lock. Failures are swallowed by the caller (linkage is a + UX nicety, never worth breaking a turn over). + """ + table = _table() + for sk in ( + f"ARTIFACT#{artifact_id}#V#{version:05d}", + f"ARTIFACT#{artifact_id}#HEAD", + ): + table.update_item( + Key={"PK": f"USER#{user_id}", "SK": sk}, + UpdateExpression="SET produced_by_message_index = :idx", + ExpressionAttributeValues={":idx": message_index}, + ConditionExpression="attribute_exists(SK)", + ) + + +_SESSION_INDEX = "SessionIndex" + + +def list_session_artifacts(user_id: str, session_id: str) -> list[dict]: + """Current HEAD of every artifact written in a chat session. + + Read side of the same SessionIndex GSI the app-api list endpoint + uses; the stream coordinator calls this post-turn to emit the live + `artifact` SSE event. Only HEAD rows carry GSI1PK/GSI1SK, so the + query returns one row per artifact (its current version). GSI1PK is + SESSION#-scoped (not user-scoped) so every row is re-checked against + the authenticated user's id. + """ + table = _table() + items: list[dict] = [] + kwargs: dict = { + "IndexName": _SESSION_INDEX, + "KeyConditionExpression": Key("GSI1PK").eq(f"SESSION#{session_id}"), + "ScanIndexForward": False, # GSI1SK embeds updated_at → newest first + } + try: + while True: + resp = table.query(**kwargs) + items.extend(resp.get("Items", [])) + last = resp.get("LastEvaluatedKey") + if not last: + break + kwargs["ExclusiveStartKey"] = last + except ClientError as exc: + raise ArtifactError("artifact list query failed") from exc + + out: list[dict] = [] + for item in items: + if item.get("user_id") != user_id: + continue + out.append( + { + "artifact_id": item.get("artifact_id", ""), + "version": int(item.get("version", 0)), + "title": item.get("title", ""), + "content_type": item.get( + "content_type", _DEFAULT_CONTENT_TYPE + ), + "updated_at": item.get("updated_at", ""), + "created_at": item.get("created_at"), + "produced_by_message_index": item.get( + "produced_by_message_index" + ), + } + ) + return out diff --git a/backend/src/agents/builtin_tools/artifacts/tools.py b/backend/src/agents/builtin_tools/artifacts/tools.py new file mode 100644 index 00000000..a202e034 --- /dev/null +++ b/backend/src/agents/builtin_tools/artifacts/tools.py @@ -0,0 +1,140 @@ +"""Context-bound factories for the artifact authoring tools. + +Identity is captured by closure (the codebase has no tool-execution +contextvar) — same pattern as the spreadsheet_analysis tools. Blocking +boto3 work is offloaded with ``asyncio.to_thread`` so the chat event +loop stays responsive under concurrent load. +""" + +from __future__ import annotations + +import asyncio +import logging +from typing import Any, Optional + +from strands import tool + +from . import service + +logger = logging.getLogger(__name__) + + +def make_create_artifact_tool(session_id: str, user_id: str): + @tool + async def create_artifact( + title: str, + content: str, + content_type: str = "text/html; charset=utf-8", + ) -> dict[str, Any]: + """Save a standalone document as a versioned artifact the user can open. + + Use this when you produce a self-contained deliverable the user + will want to view, keep, or iterate on — an HTML page, a chart, + an interactive widget, a formatted report, or a written document. + + Two authoring modes: + + - HTML (default): `content` MUST be a complete standalone HTML + document (include `` and a full `` … + ``). It renders in a sandboxed iframe with a strict CSP: + inline `" + "" + "

Artifact unavailable

" + f"

{message}

" + "" + ) + + +def _response(status: int, body: str, content_type: str) -> dict[str, Any]: + return { + "statusCode": status, + "headers": _security_headers(content_type), + "body": body, + } + + +def _error_response(status: int, message: str) -> dict[str, Any]: + return _response(status, _error_html(message), "text/html; charset=utf-8") + + +def _b64url_decode(segment: str) -> bytes: + """Decode a base64url JWT segment, restoring the stripped padding.""" + padding = "=" * (-len(segment) % 4) + return base64.urlsafe_b64decode(segment + padding) + + +def _signing_key() -> str: + """Fetch and cache the HMAC signing key. The secret is a plain + string (Secrets Manager `generateSecretString`, no JSON wrapper) — + same shape as the BFF cookie data key. Cached for the container + lifetime; on rotation the container eventually recycles, which is + acceptable for short-lived render tokens.""" + global _secrets_client, _cached_signing_key + if _cached_signing_key is not None: + return _cached_signing_key + if not _RENDER_TOKEN_SECRET_ARN: + raise _RenderConfigError("RENDER_TOKEN_SECRET_ARN is not set") + if _secrets_client is None: + _secrets_client = boto3.client("secretsmanager") + try: + secret = _secrets_client.get_secret_value(SecretId=_RENDER_TOKEN_SECRET_ARN) + except ClientError as exc: + raise _RenderConfigError("could not read render token secret") from exc + key = secret.get("SecretString") + if not key: + raise _RenderConfigError("render token secret is empty") + _cached_signing_key = key + return key + + +def _verify_token(token: str) -> dict[str, Any]: + """Verify an HS256 render token and return its validated claims. + + Implemented against the stdlib rather than PyJWT so the Lambda asset + stays dependency-free. `alg` is pinned to HS256 explicitly to reject + the `none` algorithm and HS/RS confusion.""" + parts = token.split(".") + if len(parts) != 3: + raise _TokenError("malformed token") + header_b64, payload_b64, signature_b64 = parts + + try: + header = json.loads(_b64url_decode(header_b64)) + except (ValueError, json.JSONDecodeError) as exc: + raise _TokenError("unreadable header") from exc + if not isinstance(header, dict): + raise _TokenError("malformed header") + if header.get("alg") != "HS256": + raise _TokenError("unexpected token algorithm") + + expected_sig = hmac.new( + _signing_key().encode("utf-8"), + f"{header_b64}.{payload_b64}".encode("ascii"), + hashlib.sha256, + ).digest() + try: + provided_sig = _b64url_decode(signature_b64) + except ValueError as exc: + raise _TokenError("unreadable signature") from exc + # Constant-time compare — never short-circuit on the first byte. + if not hmac.compare_digest(expected_sig, provided_sig): + raise _TokenError("signature mismatch") + + try: + claims = json.loads(_b64url_decode(payload_b64)) + except (ValueError, json.JSONDecodeError) as exc: + raise _TokenError("unreadable payload") from exc + if not isinstance(claims, dict): + raise _TokenError("malformed payload") + + if claims.get("iss") != _EXPECTED_ISS: + raise _TokenError("unexpected issuer") + if claims.get("aud") != _EXPECTED_AUD: + raise _TokenError("unexpected audience") + + now = time.time() + exp = claims.get("exp") + if not isinstance(exp, (int, float)): + raise _TokenError("missing exp") + if now > exp + _LEEWAY_SECONDS: + raise _TokenError("token expired") + + # `iat` is mandatory: the lifetime cap is the blast-radius control for + # a minter bug, and it can only be enforced relative to `iat`. The + # cross-PR contract requires the minter to send it, so a missing `iat` + # is itself a contract violation — reject rather than skip the cap. + # `bool` is an `int` subclass — exclude it explicitly. + iat = claims.get("iat") + if not isinstance(iat, (int, float)) or isinstance(iat, bool): + raise _TokenError("missing iat") + if iat > now + _LEEWAY_SECONDS: + raise _TokenError("token issued in the future") + if exp - iat > _MAX_TOKEN_LIFETIME_SECONDS: + raise _TokenError("token lifetime too long") + + sub = claims.get("sub") + aid = claims.get("aid") + ver = claims.get("ver") + if not isinstance(sub, str) or not sub: + raise _TokenError("missing sub") + if not isinstance(aid, str) or not aid: + raise _TokenError("missing aid") + # `bool` is an `int` subclass — exclude it explicitly. + if not isinstance(ver, int) or isinstance(ver, bool) or ver < 1: + raise _TokenError("invalid ver") + + return claims + + +def _get_version_record(user_id: str, artifact_id: str, version: int) -> dict[str, Any]: + global _ddb_table + if not _ARTIFACTS_TABLE: + raise _RenderConfigError("ARTIFACTS_TABLE is not set") + if _ddb_table is None: + _ddb_table = boto3.resource("dynamodb").Table(_ARTIFACTS_TABLE) + sk = f"ARTIFACT#{artifact_id}#V#{version:05d}" + try: + result = _ddb_table.get_item(Key={"PK": f"USER#{user_id}", "SK": sk}) + except ClientError as exc: + raise _RenderConfigError("artifact metadata lookup failed") from exc + item = result.get("Item") + if not item: + raise _ArtifactNotFound("version record not found") + return item + + +def _fetch_content(content_key: str) -> str: + global _s3_client + if not _ARTIFACTS_BUCKET: + raise _RenderConfigError("ARTIFACTS_BUCKET is not set") + if _s3_client is None: + _s3_client = boto3.client("s3") + try: + obj = _s3_client.get_object(Bucket=_ARTIFACTS_BUCKET, Key=content_key) + except ClientError as exc: + code = exc.response.get("Error", {}).get("Code", "") + if code in ("NoSuchKey", "404"): + raise _ArtifactNotFound("content object missing") from exc + raise _RenderConfigError("content fetch failed") from exc + content_length = obj.get("ContentLength") + if isinstance(content_length, int) and content_length > _MAX_CONTENT_BYTES: + raise _UnsupportedStorage("content exceeds size limit") + raw = obj["Body"].read(_MAX_CONTENT_BYTES + 1) + if len(raw) > _MAX_CONTENT_BYTES: + raise _UnsupportedStorage("content exceeds size limit") + try: + return raw.decode("utf-8") + except UnicodeDecodeError as exc: + raise _UnsupportedStorage("content is not valid utf-8") from exc + + +def _extract_token(event: dict[str, Any]) -> str: + params = event.get("queryStringParameters") or {} + token = params.get("t") + if not token: + raw = event.get("rawQueryString") or "" + token = (parse_qs(raw).get("t") or [None])[0] + if not token: + raise _TokenError("missing render token") + return token + + +def _request_method(event: dict[str, Any]) -> str: + return ( + event.get("requestContext", {}) + .get("http", {}) + .get("method", "GET") + .upper() + ) + + +def handler(event: dict[str, Any], _context: Any) -> dict[str, Any]: + """Lambda Function URL handler. Payload format v2.0. + + SECURITY: never log `event`, `rawQueryString`, `queryStringParameters`, + or the raw token — the render token is a bearer credential carried in + the URL query string. Log identifiers (sub/aid/ver/sid) only. + """ + method = _request_method(event) + if method not in ("GET", "HEAD"): + return _error_response(405, "Method not allowed.") + + try: + token = _extract_token(event) + claims = _verify_token(token) + except _TokenError as exc: + logger.warning("render token rejected: %s", exc) + return _error_response(403, "This artifact link is invalid or has expired.") + except _RenderConfigError as exc: + logger.error("render config error during verification: %s", exc) + return _error_response(500, "The artifact service is misconfigured.") + + user_id = claims["sub"] + artifact_id = claims["aid"] + version = claims["ver"] + logger.info( + "render request user=%s artifact=%s v=%s sid=%s", + user_id, + artifact_id, + version, + claims.get("sid"), + ) + + try: + record = _get_version_record(user_id, artifact_id, version) + storage = record.get("storage") + if storage != "s3": + raise _UnsupportedStorage(f"storage class {storage!r} not supported") + content_key = record.get("content_key") + if not isinstance(content_key, str) or not content_key: + raise _ArtifactNotFound("version record has no content pointer") + stored_content_type = record.get("content_type") or _HTML_CONTENT_TYPE + content_type = _serve_content_type(stored_content_type) + raw_title = record.get("title") + title = raw_title if isinstance(raw_title, str) else "" + body = _fetch_content(content_key) + except _ArtifactNotFound as exc: + logger.warning( + "artifact not found user=%s artifact=%s v=%s: %s", + user_id, + artifact_id, + version, + exc, + ) + return _error_response(404, "This artifact could not be found.") + except _UnsupportedStorage as exc: + logger.error( + "unsupported artifact content user=%s artifact=%s v=%s: %s", + user_id, + artifact_id, + version, + exc, + ) + return _error_response(500, "This artifact could not be rendered.") + except _RenderConfigError as exc: + logger.error("render config error during fetch: %s", exc) + return _error_response(500, "The artifact service is misconfigured.") + + if _wants_download(event): + ext = _download_extension(stored_content_type) + headers = _download_headers( + content_type, _content_disposition(title, ext) + ) + return { + "statusCode": 200, + "headers": headers, + "body": "" if method == "HEAD" else body, + } + + if method == "HEAD": + return _response(200, "", content_type) + return _response(200, body, content_type) + + +# Local smoke test: `python handler.py` exercises the missing-token path +# (returns 403) with zero AWS calls — the token check precedes any client. +if __name__ == "__main__": + print(json.dumps(handler({}, None), indent=2)) diff --git a/backend/tests/agents/builtin_tools/__init__.py b/backend/tests/agents/builtin_tools/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/tests/agents/builtin_tools/artifacts/test_artifact_tools.py b/backend/tests/agents/builtin_tools/artifacts/test_artifact_tools.py new file mode 100644 index 00000000..23841c85 --- /dev/null +++ b/backend/tests/agents/builtin_tools/artifacts/test_artifact_tools.py @@ -0,0 +1,294 @@ +"""Tests for the artifact authoring tools. + +The headline guarantee is `test_record_satisfies_minter`: a row written +by this tool must be accepted byte-for-byte by #310's app-api minter +(the real downstream reader) and resolve to the S3 object #309's render +Lambda would serve. +""" + +from __future__ import annotations + +import base64 +import re + +import boto3 +import pytest +from moto import mock_aws + +from agents.builtin_tools.artifacts import service +from apis.inference_api.chat.routes import _build_artifact_tools + +REGION = "us-east-1" +TABLE = "test-user-artifacts" +BUCKET = "test-artifacts-content" +USER = "user-123" +SESSION = "sess-9" +DOC = "

hi

" +MD = "# Title\n\nSome **bold** text and a list:\n\n- one\n- two\n" + + +def _embedded_markdown(body: str) -> str: + """Decode the base64 source the render wrapper embeds (no `<` in + base64, so the cheap regex is safe).""" + m = re.search(r'id="md-src">([^<]+)', body) + assert m, "render wrapper is missing the embedded markdown block" + return base64.b64decode(m.group(1)).decode() + + +@pytest.fixture(autouse=True) +def _reset() -> None: + service._reset_caches_for_tests() + + +@pytest.fixture +def aws(monkeypatch: pytest.MonkeyPatch): + with mock_aws(): + monkeypatch.setenv("AWS_REGION", REGION) + monkeypatch.setenv("S3_ARTIFACTS_BUCKET_NAME", BUCKET) + monkeypatch.setenv("DYNAMODB_ARTIFACTS_TABLE_NAME", TABLE) + + boto3.client("s3", region_name=REGION).create_bucket(Bucket=BUCKET) + boto3.client("dynamodb", region_name=REGION).create_table( + TableName=TABLE, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + {"AttributeName": "GSI1PK", "AttributeType": "S"}, + {"AttributeName": "GSI1SK", "AttributeType": "S"}, + ], + GlobalSecondaryIndexes=[ + { + "IndexName": "SessionIndex", + "KeySchema": [ + {"AttributeName": "GSI1PK", "KeyType": "HASH"}, + {"AttributeName": "GSI1SK", "KeyType": "RANGE"}, + ], + "Projection": {"ProjectionType": "ALL"}, + } + ], + BillingMode="PAY_PER_REQUEST", + ) + yield boto3.resource("dynamodb", region_name=REGION), boto3.client( + "s3", region_name=REGION + ) + + +def _item(ddb, artifact_id: str, sk_suffix: str) -> dict: + return ddb.Table(TABLE).get_item( + Key={"PK": f"USER#{USER}", "SK": f"ARTIFACT#{artifact_id}#{sk_suffix}"} + ).get("Item") + + +def test_create_writes_s3_and_rows(aws) -> None: + ddb, s3 = aws + aid, ver = service.create_artifact_record(USER, SESSION, "My Art", DOC, "") + assert ver == 1 + + key = f"{USER}/{aid}/v1/index.html" + assert s3.get_object(Bucket=BUCKET, Key=key)["Body"].read().decode() == DOC + + vrow = _item(ddb, aid, "V#00001") + assert vrow["storage"] == "s3" + assert vrow["content_key"] == key + assert vrow["content_type"] == "text/html; charset=utf-8" + + head = _item(ddb, aid, "HEAD") + assert head["version"] == 1 + assert head["GSI1PK"] == f"SESSION#{SESSION}" + assert head["GSI1SK"].startswith("ARTIFACT#") and head["GSI1SK"].endswith(aid) + + +def test_update_increments_and_preserves_old(aws) -> None: + ddb, s3 = aws + aid, _ = service.create_artifact_record(USER, SESSION, "T", DOC, "") + new_doc = "v2" + ver = service.update_artifact_record(USER, aid, new_doc, None, None) + assert ver == 2 + + # Old version object is immutable / still present. + assert s3.get_object( + Bucket=BUCKET, Key=f"{USER}/{aid}/v1/index.html" + )["Body"].read().decode() == DOC + assert s3.get_object( + Bucket=BUCKET, Key=f"{USER}/{aid}/v2/index.html" + )["Body"].read().decode() == new_doc + + assert _item(ddb, aid, "V#00002")["content_key"] == f"{USER}/{aid}/v2/index.html" + head = _item(ddb, aid, "HEAD") + assert head["version"] == 2 + assert head["title"] == "T" # carried forward + + +def test_update_unknown_artifact_raises(aws) -> None: + with pytest.raises(service.ArtifactNotFoundError): + service.update_artifact_record(USER, "nope", DOC, None, None) + + +def test_update_foreign_artifact_raises(aws) -> None: + aid, _ = service.create_artifact_record(USER, SESSION, "T", DOC, "") + with pytest.raises(service.ArtifactNotFoundError): + service.update_artifact_record("someone-else", aid, DOC, None, None) + + +def test_content_type_default(aws) -> None: + ddb, _ = aws + aid, _ = service.create_artifact_record(USER, SESSION, "T", DOC, "") + assert _item(ddb, aid, "V#00001")["content_type"] == "text/html; charset=utf-8" + + +def test_markdown_create_wraps_and_preserves_type(aws) -> None: + ddb, s3 = aws + aid, ver = service.create_artifact_record( + USER, SESSION, "Notes", MD, "text/markdown" + ) + assert ver == 1 + + # DDB keeps the authored Markdown type — drives the SPA card badge + # and list; the render Lambda maps it to text/html when serving. + assert _item(ddb, aid, "V#00001")["content_type"] == "text/markdown" + + body = s3.get_object( + Bucket=BUCKET, Key=f"{USER}/{aid}/v1/index.html" + )["Body"].read().decode() + assert body.lstrip().startswith("") + assert "https://esm.sh/marked@14.1.4" in body + # Source is base64-embedded, never inlined raw (escaping/XSS-safe). + assert MD not in body + assert _embedded_markdown(body) == MD + + +def test_markdown_charset_suffix_still_markdown(aws) -> None: + _, s3 = aws + aid, _ = service.create_artifact_record( + USER, SESSION, "Doc", MD, "text/markdown; charset=utf-8" + ) + body = s3.get_object( + Bucket=BUCKET, Key=f"{USER}/{aid}/v1/index.html" + )["Body"].read().decode() + assert _embedded_markdown(body) == MD + + +def test_markdown_update_rewraps_inherited_type(aws) -> None: + _, s3 = aws + aid, _ = service.create_artifact_record( + USER, SESSION, "Doc", MD, "text/markdown" + ) + new_md = "## v2\n\nrevised body\n" + # content_type omitted → inherits Markdown from HEAD, must re-wrap. + ver = service.update_artifact_record(USER, aid, new_md, None, None) + assert ver == 2 + body = s3.get_object( + Bucket=BUCKET, Key=f"{USER}/{aid}/v2/index.html" + )["Body"].read().decode() + assert body.lstrip().startswith("") + assert _embedded_markdown(body) == new_md + + +def test_html_artifact_not_wrapped(aws) -> None: + _, s3 = aws + aid, _ = service.create_artifact_record(USER, SESSION, "Page", DOC, "text/html") + assert s3.get_object( + Bucket=BUCKET, Key=f"{USER}/{aid}/v1/index.html" + )["Body"].read().decode() == DOC + + +def test_ssm_fallback(aws, monkeypatch: pytest.MonkeyPatch) -> None: + """Env unset → resolve bucket/table from /{PROJECT_PREFIX}/artifacts/*.""" + monkeypatch.delenv("S3_ARTIFACTS_BUCKET_NAME", raising=False) + monkeypatch.delenv("DYNAMODB_ARTIFACTS_TABLE_NAME", raising=False) + monkeypatch.setenv("PROJECT_PREFIX", "myproj") + ssm = boto3.client("ssm", region_name=REGION) + ssm.put_parameter(Name="/myproj/artifacts/bucket-name", Value=BUCKET, Type="String") + ssm.put_parameter(Name="/myproj/artifacts/table-name", Value=TABLE, Type="String") + service._reset_caches_for_tests() + + aid, ver = service.create_artifact_record(USER, SESSION, "T", DOC, "") + assert ver == 1 and aid + + +def test_record_satisfies_minter(aws) -> None: + """Cross-PR contract: the written version row must be accepted by + #310's app-api minter and resolve to the S3 object #309 serves.""" + _, s3 = aws + aid, ver = service.create_artifact_record(USER, SESSION, "T", DOC, "") + + from apis.app_api.artifacts import service as minter + + minter._reset_caches_for_tests() + # Minter reads its own table handle from the same env we set. + minter._assert_version_exists(USER, aid, ver) # must not raise + + # And the content_key the readers trust actually points at content. + vrow = _item(boto3.resource("dynamodb", region_name=REGION), aid, "V#00001") + assert s3.get_object( + Bucket=BUCKET, Key=vrow["content_key"] + )["Body"].read().decode() == DOC + + +@pytest.mark.parametrize( + "enabled,expected", + [(None, 0), ([], 0), (["other"], 0), (["create_artifact"], 1), + (["create_artifact", "update_artifact"], 2)], +) +def test_routes_gating(enabled, expected) -> None: + tools = _build_artifact_tools(enabled, SESSION, USER) + assert len(tools) == expected + + +def test_list_session_artifacts_returns_heads_newest_first(aws) -> None: + a1, _ = service.create_artifact_record(USER, SESSION, "First", DOC, "") + a2, _ = service.create_artifact_record(USER, SESSION, "Second", DOC, "") + # Bump a1 to v2 so it becomes the most-recently-updated HEAD. + service.update_artifact_record(USER, a1, DOC, None, None) + + rows = service.list_session_artifacts(USER, SESSION) + by_id = {r["artifact_id"]: r for r in rows} + assert set(by_id) == {a1, a2} + assert by_id[a1]["version"] == 2 # reflects current HEAD, not v1 + assert by_id[a2]["title"] == "Second" + # Newest-first: a1 (just updated) precedes a2. + assert [r["artifact_id"] for r in rows] == [a1, a2] + + +def test_list_session_artifacts_scopes_to_user(aws) -> None: + mine, _ = service.create_artifact_record(USER, SESSION, "Mine", DOC, "") + service.create_artifact_record("someone-else", SESSION, "Theirs", DOC, "") + + rows = service.list_session_artifacts(USER, SESSION) + assert [r["artifact_id"] for r in rows] == [mine] + + +def test_list_session_artifacts_empty_session(aws) -> None: + assert service.list_session_artifacts(USER, "no-such-session") == [] + + +def test_set_produced_by_message_index_stamps_version_and_head(aws) -> None: + ddb, _ = aws + aid, _ = service.create_artifact_record(USER, SESSION, "Doc", DOC, "") + assert service.list_session_artifacts(USER, SESSION)[0][ + "produced_by_message_index" + ] is None + + service.set_produced_by_message_index(USER, aid, 1, 7) + + # Per-version linkage: the v1 row itself carries the index — this is + # what survives reload via the all-versions list endpoint. + assert _item(ddb, aid, "V#00001")["produced_by_message_index"] == 7 + # HEAD is stamped too, so the writer's HEAD-based live list still sees it. + rows = service.list_session_artifacts(USER, SESSION) + assert rows[0]["produced_by_message_index"] == 7 + # The stamp must leave the optimistic-lock `version` untouched so a + # later update_artifact still re-points HEAD cleanly. + assert service.update_artifact_record(USER, aid, DOC, None, None) == 2 + + +def test_set_produced_by_message_index_requires_existing_rows(aws) -> None: + from botocore.exceptions import ClientError + + # No version/HEAD rows for "nope": the conditional update fails closed. + with pytest.raises(ClientError): + service.set_produced_by_message_index(USER, "nope", 1, 1) diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/__init__.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/conftest.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/conftest.py new file mode 100644 index 00000000..c6d41284 --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/conftest.py @@ -0,0 +1,664 @@ +"""Shared fixtures for spreadsheet_analysis unit tests. + +Central place to assemble the stack of mocks the analyze_spreadsheet tool +requires: the Code Interpreter client, the S3 client, and the file +resolution helpers (_get_kb_files / _get_session_files / _find_file). + +Each fixture is small and composable so individual tests can swap in +exactly the behavior they want to assert on. + +S3 and DynamoDB are handled with moto (see ``tests/shared/conftest.py``) +so that tests exercise real boto3 call paths rather than ad-hoc mocks. +The Code Interpreter client has no moto equivalent — it's an AgentCore +service — so ``FakeCodeInterpreter`` below is a hand-rolled stand-in. +""" + +from __future__ import annotations + +import asyncio +from dataclasses import dataclass, field +from typing import Any, Callable +from unittest.mock import patch + +import boto3 +import pytest +from moto import mock_aws + + +# --------------------------------------------------------------------------- +# AWS mocks (moto) — S3 + DynamoDB tables +# --------------------------------------------------------------------------- + + +AWS_REGION = "us-east-1" +SESSIONS_BUCKET = "test-sessions-bucket" +KB_BUCKET = "test-kb-bucket" + + +@pytest.fixture +def aws_mocked(monkeypatch): + """Activate moto's ``mock_aws`` for the duration of the test. + + Sets the minimum env vars boto3 clients expect. Any S3 / DynamoDB + calls made by analyze_tool._download_file or _get_kb_files during + the test execute against moto's in-process fakes, not real AWS. + + ``AWS_REGION`` is set alongside ``AWS_DEFAULT_REGION`` because some + helpers (``_get_kb_files``, ``_download_file``) read ``AWS_REGION`` + explicitly and fall back to ``us-west-2`` — which would land on a + different moto region than the fixtures use. + """ + monkeypatch.setenv("AWS_DEFAULT_REGION", AWS_REGION) + monkeypatch.setenv("AWS_REGION", AWS_REGION) + monkeypatch.setenv("AWS_ACCESS_KEY_ID", "testing") + monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "testing") + monkeypatch.setenv("AWS_SECURITY_TOKEN", "testing") + monkeypatch.setenv("AWS_SESSION_TOKEN", "testing") + with mock_aws(): + yield + + +@pytest.fixture +def sessions_bucket(aws_mocked): + """Create the session-attachments S3 bucket. Tests push real objects + in and analyze_tool downloads them through real boto3 calls. + """ + s3 = boto3.client("s3", region_name=AWS_REGION) + s3.create_bucket(Bucket=SESSIONS_BUCKET) + return SESSIONS_BUCKET + + +@pytest.fixture +def kb_bucket(aws_mocked, monkeypatch): + """Create the assistant-KB S3 bucket and point the env var at it so + ``_download_file`` can resolve the bucket for KB-source files. + """ + s3 = boto3.client("s3", region_name=AWS_REGION) + s3.create_bucket(Bucket=KB_BUCKET) + monkeypatch.setenv("S3_ASSISTANTS_DOCUMENTS_BUCKET_NAME", KB_BUCKET) + return KB_BUCKET + + +@pytest.fixture +def assistants_table(aws_mocked, monkeypatch): + """Create the DynamoDB assistants table with the schema + ``_get_kb_files`` queries against. Tests can ``put_item`` real + document records and see them flow through the filter. + """ + ddb = boto3.client("dynamodb", region_name=AWS_REGION) + name = "test-assistants" + monkeypatch.setenv("DYNAMODB_ASSISTANTS_TABLE_NAME", name) + ddb.create_table( + TableName=name, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + return boto3.resource("dynamodb", region_name=AWS_REGION).Table(name) + + +@pytest.fixture +def files_table(aws_mocked, monkeypatch): + """Create the user-files DynamoDB table with the SessionIndex GSI + that ``FileUploadRepository.list_session_files`` queries. + """ + ddb = boto3.client("dynamodb", region_name=AWS_REGION) + name = "test-user-files" + monkeypatch.setenv("DYNAMODB_USER_FILES_TABLE_NAME", name) + ddb.create_table( + TableName=name, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + {"AttributeName": "GSI1PK", "AttributeType": "S"}, + {"AttributeName": "GSI1SK", "AttributeType": "S"}, + ], + GlobalSecondaryIndexes=[{ + "IndexName": "SessionIndex", + "KeySchema": [ + {"AttributeName": "GSI1PK", "KeyType": "HASH"}, + {"AttributeName": "GSI1SK", "KeyType": "RANGE"}, + ], + "Projection": {"ProjectionType": "ALL"}, + }], + BillingMode="PAY_PER_REQUEST", + ) + return boto3.resource("dynamodb", region_name=AWS_REGION).Table(name) + + +@pytest.fixture +def file_repository(files_table): + """A real ``FileUploadRepository`` pointed at the moto-backed table.""" + from apis.shared.files.repository import FileUploadRepository + + return FileUploadRepository(table_name="test-user-files") + + +# --------------------------------------------------------------------------- +# Seed helpers — write KB docs / session files into moto-backed stores +# --------------------------------------------------------------------------- + + +def put_kb_doc( + table, + *, + assistant_id: str, + filename: str, + content_type: str, + status: str = "complete", + size_bytes: int = 1024, + document_id: str | None = None, + s3_key: str | None = None, + use_snake_case: bool = False, +) -> None: + """Write a completed (or failed) KB document row to the assistants + table in the shape ``_get_kb_files`` queries. + + ``use_snake_case`` lets tests pin the legacy field-name behavior — + some older items store ``content_type`` / ``size_bytes`` / ``s3_key`` + / ``document_id`` instead of the camelCase defaults. + """ + doc_id = document_id or f"doc-{filename}" + key = s3_key or f"assistants/{assistant_id}/{filename}" + item = { + "PK": f"AST#{assistant_id}", + "SK": f"DOC#{doc_id}", + "status": status, + "filename": filename, + } + if use_snake_case: + item.update({ + "content_type": content_type, + "size_bytes": size_bytes, + "document_id": doc_id, + "s3_key": key, + }) + else: + item.update({ + "contentType": content_type, + "sizeBytes": size_bytes, + "documentId": doc_id, + "s3Key": key, + }) + table.put_item(Item=item) + + +async def put_session_file( + file_repository, + *, + session_id: str, + user_id: str = "u1", + upload_id: str, + filename: str, + mime_type: str, + size_bytes: int = 1024, + s3_bucket: str = SESSIONS_BUCKET, + s3_key: str | None = None, +) -> None: + """Create a READY file record in the files repository so + ``FileUploadRepository.list_session_files`` returns it. + """ + from apis.shared.files.models import FileMetadata, FileStatus + + key = s3_key or f"sessions/{session_id}/{filename}" + await file_repository.create_file(FileMetadata( + upload_id=upload_id, + user_id=user_id, + session_id=session_id, + filename=filename, + mime_type=mime_type, + size_bytes=size_bytes, + s3_key=key, + s3_bucket=s3_bucket, + status=FileStatus.READY, + )) + + +@pytest.fixture +def seed_kb_doc(assistants_table): + """Tiny helper so tests read like ``seed_kb_doc(filename=..., ...)`` + without threading the table fixture through every call site. + """ + def _seed(**kwargs): + put_kb_doc(assistants_table, **kwargs) + return _seed + + +@pytest.fixture +def seed_session_file(file_repository): + """Async-aware helper; tests should ``await seed_session_file(...)``.""" + async def _seed(**kwargs): + await put_session_file(file_repository, **kwargs) + return _seed + + +# --------------------------------------------------------------------------- +# Fake CodeInterpreter +# --------------------------------------------------------------------------- + + +@dataclass +class InvocationRecord: + """One call to the fake CodeInterpreter's ``invoke`` method.""" + + name: str + payload: dict + + +@dataclass +class FakeCodeInterpreter: + """Drop-in stand-in for bedrock_agentcore's CodeInterpreter client. + + Tests can: + - install a ``reply_for`` callback that returns the canned stream + response for a given (invocation_name, payload) pair; or + - rely on the default empty-success behavior (``executeCode`` returns + an empty stdout non-error stream; ``writeFiles`` / ``readFiles`` + return empty streams). + + The ``invocations`` list preserves call order so tests can assert on + the full sequence, not just the last call. + """ + + reply_for: Callable[[str, dict], dict] | None = None + invocations: list[InvocationRecord] = field(default_factory=list) + started: bool = False + stopped: bool = False + + # Inputs the test doesn't care about — bedrock_agentcore exposes these + # as construction / lifecycle hooks. We keep no-op stubs. + def __init__(self, *_args, reply_for=None, **_kwargs): + self.reply_for = reply_for + self.invocations = [] + self.started = False + self.stopped = False + + def start(self, identifier: str) -> None: # noqa: D401 — mock signature + self.started = True + + def stop(self) -> None: + self.stopped = True + + def invoke(self, name: str, payload: dict) -> dict: + self.invocations.append(InvocationRecord(name=name, payload=payload)) + if self.reply_for is not None: + return self.reply_for(name, payload) + # Default: empty successful response. + return {"stream": [{"result": {"isError": False, "structuredContent": {"stdout": ""}}}]} + + # --- Handy helpers for test assertions --- + + def bootstrap_payload(self) -> str | None: + """Return the code string passed to the first executeCode call + (the XLSX bootstrap), or None if nothing was executed. + """ + for rec in self.invocations: + if rec.name == "executeCode": + return rec.payload.get("code") + return None + + def executed_codes(self) -> list[str]: + return [r.payload.get("code", "") for r in self.invocations if r.name == "executeCode"] + + +def _stream_response(stdout: str = "", *, is_error: bool = False, stderr: str = "") -> dict: + """Build a minimally valid stream response from CodeInterpreter.""" + return { + "stream": [ + { + "result": { + "isError": is_error, + "structuredContent": {"stdout": stdout, "stderr": stderr}, + } + } + ] + } + + +@pytest.fixture +def fake_code_interpreter(): + """Return a FakeCodeInterpreter instance + a patch context that + substitutes it for the real client used by analyze_tool. + + Usage: + def test_it(fake_code_interpreter): + fake, patcher = fake_code_interpreter + with patcher: + ... + """ + fake = FakeCodeInterpreter() + + def _factory(*_args, **_kwargs): + return fake + + patcher = patch( + "bedrock_agentcore.tools.code_interpreter_client.CodeInterpreter", + side_effect=_factory, + ) + return fake, patcher + + +# --------------------------------------------------------------------------- +# S3 object helpers (moto-backed) +# --------------------------------------------------------------------------- + + +def put_s3_object(bucket: str, key: str, body: bytes) -> None: + """Push a real object into a moto-backed bucket.""" + s3 = boto3.client("s3", region_name=AWS_REGION) + s3.put_object(Bucket=bucket, Key=key, Body=body) + + +@pytest.fixture +def seed_s3_object(sessions_bucket): + """Drop an object into the sessions bucket. ``analyze_tool._download_file`` + will pick it up via real boto3 through moto's interceptor. + """ + def _seed(key: str, body: bytes = b"fake bytes", bucket: str = SESSIONS_BUCKET): + put_s3_object(bucket, key, body) + return _seed + + +# --------------------------------------------------------------------------- +# File sources (KB + session) +# --------------------------------------------------------------------------- + + +@pytest.fixture +def file_sources(): + """Patch ``_get_kb_files`` and ``_get_session_files`` together. + + Patches both modules that import these helpers — analyze_tool + (for _find_file and the tabular inventory) and list_spreadsheets_tool + (for the tool factory's direct calls). Tests can configure both + sides cleanly: + + def test_it(file_sources): + set_kb, set_session = file_sources + set_session([{...}]) + + The helpers are ``async def`` (see #260 — sync boto3 was blocking + the event loop), so the patches install async side-effects. Returning + a plain list from an ``async def`` gives the callers an awaitable + they can ``await`` exactly like the real helpers. + """ + kb_files: list[dict[str, Any]] = [] + session_files: list[dict[str, Any]] = [] + + def set_kb(files): + kb_files[:] = list(files) + + def set_session(files): + session_files[:] = list(files) + + async def _kb_side_effect(_aid): + return list(kb_files) + + async def _session_side_effect(_sid): + return list(session_files) + + patchers = [ + patch( + "agents.builtin_tools.spreadsheet_analysis.analyze_tool._get_kb_files", + side_effect=_kb_side_effect, + ), + patch( + "agents.builtin_tools.spreadsheet_analysis.analyze_tool._get_session_files", + side_effect=_session_side_effect, + ), + patch( + "agents.builtin_tools.spreadsheet_analysis.list_spreadsheets_tool._get_kb_files", + side_effect=_kb_side_effect, + ), + patch( + "agents.builtin_tools.spreadsheet_analysis.list_spreadsheets_tool._get_session_files", + side_effect=_session_side_effect, + ), + ] + for p in patchers: + p.start() + try: + yield set_kb, set_session + finally: + for p in patchers: + p.stop() + + +@pytest.fixture +def code_interpreter_id(monkeypatch): + """Set a sentinel Code Interpreter id so ``_get_code_interpreter_id`` + short-circuits to the env branch (avoiding the SSM fallback). + """ + monkeypatch.setenv("AGENTCORE_CODE_INTERPRETER_ID", "ci-test-123") + + +# --------------------------------------------------------------------------- +# Canned file records +# --------------------------------------------------------------------------- + + +def make_session_csv(filename: str = "data.csv", size: int = 1024) -> dict: + return { + "filename": filename, + "source": "chat_attachment", + "content_type": "text/csv", + "size_bytes": size, + "document_id": f"upload-{filename}", + "s3_key": f"sessions/{filename}", + "s3_bucket": SESSIONS_BUCKET, + } + + +def make_session_xlsx(filename: str = "workbook.xlsx", size: int = 1024 * 500) -> dict: + return { + "filename": filename, + "source": "chat_attachment", + "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + "size_bytes": size, + "document_id": f"upload-{filename}", + "s3_key": f"sessions/{filename}", + "s3_bucket": SESSIONS_BUCKET, + } + + +def make_kb_xlsx(filename: str = "kb_workbook.xlsx", size: int = 1024 * 200) -> dict: + return { + "filename": filename, + "source": "knowledge_base", + "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + "size_bytes": size, + "document_id": f"doc-{filename}", + "s3_key": f"assistants/ast-1/{filename}", + } + + +@pytest.fixture +def file_factories(): + """Expose the canned-file helpers to tests without an import dance.""" + return { + "session_csv": make_session_csv, + "session_xlsx": make_session_xlsx, + "kb_xlsx": make_kb_xlsx, + } + + +# --------------------------------------------------------------------------- +# Tool invocation helper +# --------------------------------------------------------------------------- + + +def _unwrap(tool_obj: Any) -> Callable[..., Any]: + """Strands' @tool wraps the original function; our tests call the raw + function so we bypass framework marshalling. + """ + return getattr(tool_obj, "__wrapped__", None) or tool_obj + + +@pytest.fixture +def call_analyze(): + """Shortcut for ``unwrap(tool)(**kwargs)`` — builds the analyze tool + via the factory and invokes it in one go. + + ``analyze_spreadsheet`` is ``async def`` (see #260) so we run the + returned coroutine to completion here; tests stay sync and assert + on the resolved result, keeping the call-site unchanged from the + pre-refactor shape. + + def test_it(call_analyze, ...): + result = call_analyze( + filename="x.csv", + python_code="print(1)", + assistant_id=None, session_id="s1", user_id="u1", + ) + """ + from agents.builtin_tools.spreadsheet_analysis.analyze_tool import ( + make_analyze_tool, + ) + + def _call(*, filename, python_code, output_filename=None, + assistant_id=None, session_id="s1", user_id="u1"): + tool = make_analyze_tool(assistant_id, session_id, user_id) + fn = _unwrap(tool) + return asyncio.run(fn(filename=filename, python_code=python_code, + output_filename=output_filename)) + + return _call + + +# --------------------------------------------------------------------------- +# Bootstrap stdout builder (multi-sheet / single-sheet) +# --------------------------------------------------------------------------- + + +def build_bootstrap_stdout( + *, + total: int, + sheets: list[tuple[str, str, int, bool, str]], + skipped_names: list[str] | None = None, +) -> str: + """Build the stdout the XLSX bootstrap emits inside its ``[__SHEETS__]`` + block. + + Each ``sheets`` entry is ``(name, path, rows, truncated, alias)``. + The function assembles the block in the exact shape the real + bootstrap writes, so ``_parse_sheet_inventory`` can round-trip it. + """ + from agents.builtin_tools.spreadsheet_analysis.analyze_tool import _SHEETS_MARKER + + lines = [ + _SHEETS_MARKER, + f"total: {total}", + f"converted: {len(sheets)}", + f"skipped: {total - len(sheets)}", + ] + if skipped_names: + lines.append(f"skipped_names: {skipped_names!r}") + for name, path, rows, truncated, alias in sheets: + flag = "1" if truncated else "0" + lines.append(f"sheet|{name}|{path}|{rows}|{flag}|{alias}") + lines.append(_SHEETS_MARKER) + return "\n".join(lines) + "\n" + + +@pytest.fixture +def bootstrap_stdout(): + """Expose ``build_bootstrap_stdout`` to tests.""" + return build_bootstrap_stdout + + +# --------------------------------------------------------------------------- +# Schema-preview stdout builder +# --------------------------------------------------------------------------- + + +def build_schema_stdout( + *, + file: str, + rows: int = 100, + cols: int = 3, + load: str | None = None, + columns: str = "a, b, c", + first_row: str = "{'a': 1, 'b': 2, 'c': 3}", +) -> str: + """Build the stdout the schema-preview probe emits inside its + ``[__SCHEMA__]`` block. + """ + from agents.builtin_tools.spreadsheet_analysis.analyze_tool import _SCHEMA_MARKER + + load_line = load or f"pd.read_csv('{file}', low_memory=False)" + return "\n".join([ + _SCHEMA_MARKER, + f"file: {file} ({rows} rows x {cols} cols)", + f"load: {load_line}", + f"columns: {columns}", + f"first_row: {first_row}", + _SCHEMA_MARKER, + ]) + "\n" + + +@pytest.fixture +def schema_stdout(): + return build_schema_stdout + + +# --------------------------------------------------------------------------- +# Default stream reply dispatcher +# --------------------------------------------------------------------------- + + +def default_reply_factory( + *, + bootstrap_out: str = "", + schema_out: str = "", + user_out: str = "", + user_err: str = "", + user_is_error: bool = False, +) -> Callable[[str, dict], dict]: + """Return a ``reply_for`` callback suitable for ``FakeCodeInterpreter``. + + Reads the invocation ordering the tool performs — ``writeFiles`` for + the base64 blob / raw CSV, then executeCode for the bootstrap, + executeCode for the schema probe, executeCode for the user code — + and emits the matching stdout/stderr. + + Ignores ``readFiles`` (used for chart downloads) unless a caller + explicitly overrides. + """ + state = {"execute_calls": 0} + + def _reply(name: str, _payload: dict) -> dict: + if name == "executeCode": + state["execute_calls"] += 1 + # Order: 1) XLSX bootstrap (or none for CSV), 2) schema probe, + # 3) user code. For CSV inputs, the bootstrap is skipped so + # call #1 is schema, call #2 is user code. + call_idx = state["execute_calls"] + if bootstrap_out and call_idx == 1: + return _stream_response(bootstrap_out) + if bootstrap_out and call_idx == 2: + return _stream_response(schema_out) + if bootstrap_out and call_idx == 3: + return _stream_response(user_out, is_error=user_is_error, stderr=user_err) + # CSV path — no bootstrap. + if not bootstrap_out and call_idx == 1: + return _stream_response(schema_out) + if not bootstrap_out and call_idx == 2: + return _stream_response(user_out, is_error=user_is_error, stderr=user_err) + return _stream_response() + + return _reply + + +@pytest.fixture +def reply_factory(): + return default_reply_factory diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_analyze_tool_integration.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_analyze_tool_integration.py new file mode 100644 index 00000000..e4b4aae4 --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_analyze_tool_integration.py @@ -0,0 +1,779 @@ +"""Integration-style tests for the analyze_spreadsheet tool, exercising +the full factory → file lookup → download → CodeInterpreter → response +path with every external dependency mocked. + +Covers the behaviors the issue (#261) specifically called out as "subtle +logic worth pinning down": + +- CSV fast-path (no bootstrap, direct writeFiles + schema probe + user code) +- XLSX bootstrap: base64 push, sheet inventory round-trip, CSV rename +- Single-sheet vs. multi-sheet response shape +- Filename alias fallback (foo.csv ↔ foo.xlsx) via ``_find_file`` +- Error-path hints: wrong-filename retry, schema-footer attached +- Truncation warnings when sheets hit MAX_ROWS_PER_SHEET +- Skipped-sheet warning when workbook exceeds MAX_SHEETS_TO_CONVERT +- Missing Code Interpreter → friendly error, no interpreter calls +- File not found → friendly error with list_spreadsheets hint +- S3 download failure → friendly error, interpreter still stopped + +S3 and DynamoDB go through moto so tests exercise the real boto3 call +paths. Only the CodeInterpreter is hand-mocked (no moto equivalent for +AgentCore). All tests run offline; no AWS credentials required and the +backend env doesn't need pandas installed. +""" + +from __future__ import annotations + +from unittest.mock import patch + + +# --------------------------------------------------------------------------- +# Happy path: CSV end-to-end +# --------------------------------------------------------------------------- + + +class TestCsvHappyPath: + def test_csv_end_to_end_success( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + schema_stdout, + reply_factory, + ): + set_kb, set_session = file_sources + set_session([file_factories["session_csv"]("data.csv")]) + seed_s3_object(key="sessions/data.csv", body=b"col1,col2\n1,2\n") + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + schema_out=schema_stdout(file="data.csv"), + user_out="Total: 42\n", + ) + + with ci_patch: + result = call_analyze( + filename="data.csv", + python_code="print('Total: 42')", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "success" + text = result["content"][0]["text"] + assert "Total: 42" in text + # Schema footer attached to success responses. + assert "Dataset" in text + assert "data.csv" in text + # No XLSX bootstrap: one writeFiles (raw CSV) + two executeCode + # calls (schema probe, user code). + assert fake.started and fake.stopped + write_calls = [r for r in fake.invocations if r.name == "writeFiles"] + exec_calls = [r for r in fake.invocations if r.name == "executeCode"] + assert len(write_calls) == 1 + assert len(exec_calls) == 2 + + def test_csv_writes_raw_text_to_sandbox( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + schema_stdout, + reply_factory, + ): + """CSV fast-path pushes the raw text directly — no base64, no + bootstrap. Regression guard against a future "always run the + bootstrap" refactor. + """ + set_kb, set_session = file_sources + set_session([file_factories["session_csv"]("data.csv")]) + seed_s3_object(key="sessions/data.csv", body=b"col1,col2\n1,2\n") + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + schema_out=schema_stdout(file="data.csv"), + user_out="done\n", + ) + + with ci_patch: + call_analyze( + filename="data.csv", + python_code="print('done')", + session_id="s1", + user_id="u1", + ) + + write_call = next(r for r in fake.invocations if r.name == "writeFiles") + files = write_call.payload["content"] + assert len(files) == 1 + # Pushed as text, not base64. + assert files[0]["path"] == "data.csv" + assert files[0]["text"] == "col1,col2\n1,2\n" + + +# --------------------------------------------------------------------------- +# XLSX happy path — single-sheet and multi-sheet +# --------------------------------------------------------------------------- + + +XLSX_BYTES = b"\x50\x4b\x03\x04" + b"fake xlsx binary" # PK... magic + payload + + +class TestXlsxSingleSheet: + def test_single_sheet_xlsx_success( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + bootstrap_stdout, + schema_stdout, + reply_factory, + ): + set_kb, set_session = file_sources + set_session([file_factories["session_xlsx"]("Budget.xlsx")]) + seed_s3_object(key="sessions/Budget.xlsx", body=XLSX_BYTES) + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + bootstrap_out=bootstrap_stdout( + total=1, + sheets=[("Sheet1", "Budget.csv", 100, False, "")], + ), + schema_out=schema_stdout(file="Budget.csv", rows=100), + user_out="sum=9999\n", + ) + + with ci_patch: + result = call_analyze( + filename="Budget.xlsx", + python_code="print('sum=9999')", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "success" + text = result["content"][0]["text"] + assert "sum=9999" in text + # Single-sheet path does not emit the multi-sheet inventory. + assert "Available sheets" not in text + + def test_xlsx_bootstrap_pushes_base64_blob( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + bootstrap_stdout, + schema_stdout, + reply_factory, + ): + set_kb, set_session = file_sources + set_session([file_factories["session_xlsx"]("Budget.xlsx")]) + seed_s3_object(key="sessions/Budget.xlsx", body=b"xlsx-binary-bytes") + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + bootstrap_out=bootstrap_stdout( + total=1, + sheets=[("Sheet1", "Budget.csv", 10, False, "")], + ), + schema_out=schema_stdout(file="Budget.csv"), + user_out="", + ) + + with ci_patch: + call_analyze( + filename="Budget.xlsx", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + write_call = next(r for r in fake.invocations if r.name == "writeFiles") + # Encoded blob written as text under _encoded.b64. + entries = write_call.payload["content"] + assert any(e["path"] == "_encoded.b64" for e in entries) + + +class TestXlsxMultiSheet: + def test_multi_sheet_response_includes_inventory( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + bootstrap_stdout, + schema_stdout, + reply_factory, + ): + set_kb, set_session = file_sources + set_session([file_factories["session_xlsx"]("Budget.xlsx")]) + seed_s3_object(key="sessions/Budget.xlsx", body=XLSX_BYTES) + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + bootstrap_out=bootstrap_stdout( + total=3, + sheets=[ + ("Summary", "Budget.summary.csv", 12, False, "Budget.csv"), + ("Transactions", "Budget.transactions.csv", 18_551, False, ""), + ("Notes", "Budget.notes.csv", 5, False, ""), + ], + ), + schema_out=schema_stdout(file="Budget.csv"), + user_out="analyzed\n", + ) + + with ci_patch: + result = call_analyze( + filename="Budget.xlsx", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + text = result["content"][0]["text"] + assert "Available sheets" in text + assert "Summary" in text + assert "Transactions" in text + assert "Notes" in text + assert "Budget.summary.csv" in text + # Row counts are formatted with commas for readability. + assert "18,551" in text + + def test_skipped_sheets_warning_surfaces( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + bootstrap_stdout, + schema_stdout, + reply_factory, + ): + set_kb, set_session = file_sources + set_session([file_factories["session_xlsx"]("Many.xlsx")]) + seed_s3_object(key="sessions/Many.xlsx", body=XLSX_BYTES) + + fake, ci_patch = fake_code_interpreter + sheets = [ + (f"S{i}", f"Many.s{i}.csv", 10, False, "" if i > 1 else "Many.csv") + for i in range(1, 26) + ] + fake.reply_for = reply_factory( + bootstrap_out=bootstrap_stdout( + total=30, + sheets=sheets, + skipped_names=["S26", "S27", "S28", "S29", "S30"], + ), + schema_out=schema_stdout(file="Many.csv"), + user_out="ok\n", + ) + + with ci_patch: + result = call_analyze( + filename="Many.xlsx", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + text = result["content"][0]["text"] + assert "30 sheets" in text + assert "first 25" in text + assert "S26" in text + assert "S30" in text + + def test_truncated_sheet_warning_surfaces( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + bootstrap_stdout, + schema_stdout, + reply_factory, + ): + """A sheet truncated at MAX_ROWS_PER_SHEET must be flagged in the + inventory list so the user knows the analysis may be partial. + """ + from agents.builtin_tools.spreadsheet_analysis.analyze_tool import ( + MAX_ROWS_PER_SHEET, + ) + + set_kb, set_session = file_sources + set_session([file_factories["session_xlsx"]("Huge.xlsx")]) + seed_s3_object(key="sessions/Huge.xlsx", body=XLSX_BYTES) + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + bootstrap_out=bootstrap_stdout( + total=2, + sheets=[ + ("BigSheet", "Huge.bigsheet.csv", + MAX_ROWS_PER_SHEET, True, "Huge.csv"), + ("SmallSheet", "Huge.smallsheet.csv", 10, False, ""), + ], + ), + schema_out=schema_stdout(file="Huge.csv"), + user_out="done\n", + ) + + with ci_patch: + result = call_analyze( + filename="Huge.xlsx", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + text = result["content"][0]["text"] + # Truncation tag for the big sheet; none for the small one. + big_line = next(line for line in text.splitlines() if "BigSheet" in line) + small_line = next(line for line in text.splitlines() if "SmallSheet" in line) + assert "truncated" in big_line.lower() + assert "truncated" not in small_line.lower() + + +# --------------------------------------------------------------------------- +# Filename aliasing +# --------------------------------------------------------------------------- + + +class TestFilenameAliasing: + def test_csv_request_resolves_xlsx_source( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + bootstrap_stdout, + schema_stdout, + reply_factory, + ): + """Model asks for ``Budget.csv`` (sandbox filename) when only + ``Budget.xlsx`` was uploaded. ``_find_file`` aliases to the XLSX + source; end-to-end the tool should still succeed. + """ + set_kb, set_session = file_sources + set_session([file_factories["session_xlsx"]("Budget.xlsx")]) + seed_s3_object(key="sessions/Budget.xlsx", body=XLSX_BYTES) + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + bootstrap_out=bootstrap_stdout( + total=1, + sheets=[("Sheet1", "Budget.csv", 10, False, "")], + ), + schema_out=schema_stdout(file="Budget.csv"), + user_out="ok\n", + ) + + with ci_patch: + result = call_analyze( + filename="Budget.csv", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "success" + + +class TestKnowledgeBaseDownload: + def test_kb_source_downloads_from_kb_bucket( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + kb_bucket, + code_interpreter_id, + schema_stdout, + reply_factory, + ): + """KB-sourced files resolve their bucket from the + ``S3_ASSISTANTS_DOCUMENTS_BUCKET_NAME`` env var (set by the + ``kb_bucket`` fixture) rather than from the file record's + ``s3_bucket`` field. Covers the knowledge_base branch of + ``_download_file`` which the session-attachment tests never + exercise. + """ + import boto3 + + from tests.agents.builtin_tools.spreadsheet_analysis.conftest import ( + AWS_REGION, + ) + + # Seed the KB bucket directly (the kb_xlsx factory points s3_key + # at assistants/ast-1/..., which is what _download_file reads). + kb_file = file_factories["kb_xlsx"]("Ledger.csv") + kb_file["content_type"] = "text/csv" # simpler path — no XLSX bootstrap + s3 = boto3.client("s3", region_name=AWS_REGION) + s3.put_object( + Bucket=kb_bucket, + Key=kb_file["s3_key"], + Body=b"a,b,c\n1,2,3\n", + ) + + set_kb, set_session = file_sources + set_kb([kb_file]) + set_session([]) + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + schema_out=schema_stdout(file="Ledger.csv"), + user_out="kb-analyzed\n", + ) + + with ci_patch: + result = call_analyze( + filename="Ledger.csv", + python_code="pass", + assistant_id="ast-1", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "success" + assert "kb-analyzed" in result["content"][0]["text"] + + def test_kb_source_missing_env_var_surfaces_friendly_error( + self, + call_analyze, + file_sources, + file_factories, + aws_mocked, + monkeypatch, + code_interpreter_id, + ): + """``_download_file`` raises ``ValueError`` when a KB file has no + resolvable bucket. Tool wraps that in a graceful error rather + than propagating the exception. + """ + monkeypatch.delenv("S3_ASSISTANTS_DOCUMENTS_BUCKET_NAME", raising=False) + + kb_file = file_factories["kb_xlsx"]("Ledger.csv") + kb_file["content_type"] = "text/csv" + + set_kb, set_session = file_sources + set_kb([kb_file]) + set_session([]) + + result = call_analyze( + filename="Ledger.csv", + python_code="pass", + assistant_id="ast-1", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "error" + assert "Failed to download" in result["content"][0]["text"] + + +# --------------------------------------------------------------------------- +# Error paths +# --------------------------------------------------------------------------- + + +class TestFileNotFound: + def test_unknown_file_returns_list_spreadsheets_hint( + self, + call_analyze, + file_sources, + code_interpreter_id, + ): + set_kb, set_session = file_sources + set_session([]) + + # No Code Interpreter patching because we never get that far. + result = call_analyze( + filename="missing.csv", + python_code="print(1)", + session_id="s1", + user_id="u1", + ) + assert result["status"] == "error" + text = result["content"][0]["text"] + assert "not found" in text + assert "list_spreadsheets" in text + + +class TestS3DownloadFailure: + def test_s3_error_surfaces_friendly_message( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + code_interpreter_id, + ): + """File metadata points at a key that doesn't exist in the bucket. + moto returns a NoSuchKey ClientError, which ``_download_file`` + wraps in a friendly message rather than crashing. + """ + set_kb, set_session = file_sources + set_session([file_factories["session_csv"]("data.csv")]) + # Note: no seed_s3_object — the object doesn't exist, so + # get_object raises. + + fake, ci_patch = fake_code_interpreter + with ci_patch: + result = call_analyze( + filename="data.csv", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "error" + assert "Failed to download" in result["content"][0]["text"] + # The interpreter should never have been started for a download + # failure — start() happens after _download_file succeeds. + assert not fake.started + + +class TestCodeInterpreterUnavailable: + def test_no_ci_id_returns_friendly_error( + self, + call_analyze, + file_sources, + file_factories, + monkeypatch, + ): + """When ``_get_code_interpreter_id`` resolves to None (env unset, + SSM lookup fails), the tool bails out with a contact-admin + message instead of crashing. + """ + monkeypatch.delenv("AGENTCORE_CODE_INTERPRETER_ID", raising=False) + + set_kb, set_session = file_sources + set_session([file_factories["session_csv"]("data.csv")]) + + with patch( + "agents.builtin_tools.spreadsheet_analysis.analyze_tool._get_code_interpreter_id", + return_value=None, + ): + result = call_analyze( + filename="data.csv", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "error" + assert "Code Interpreter is not configured" in result["content"][0]["text"] + + +class TestUserCodeError: + def test_wrong_xlsx_filename_injects_hint( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + bootstrap_stdout, + schema_stdout, + reply_factory, + ): + """Classic failure: model wrote ``pd.read_csv('Budget.xlsx', ...)`` + but the sandbox has ``Budget.csv``. Error response must include + the targeted retry hint naming the correct filename, not just + dump the FileNotFoundError. + """ + set_kb, set_session = file_sources + set_session([file_factories["session_xlsx"]("Budget.xlsx")]) + seed_s3_object(key="sessions/Budget.xlsx", body=XLSX_BYTES) + + err_traceback = ( + "Traceback (most recent call last):\n" + " File \"/tmp/ipykernel_1/code.py\", line 1, in \n" + " df = pd.read_csv('Budget.xlsx', low_memory=False)\n" + "FileNotFoundError: [Errno 2] No such file or directory: " + "'Budget.xlsx'\n" + ) + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + bootstrap_out=bootstrap_stdout( + total=1, + sheets=[("Sheet1", "Budget.csv", 10, False, "")], + ), + schema_out=schema_stdout(file="Budget.csv"), + user_out="", + user_err=err_traceback, + user_is_error=True, + ) + + with ci_patch: + result = call_analyze( + filename="Budget.xlsx", + python_code="df = pd.read_csv('Budget.xlsx')", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "error" + text = result["content"][0]["text"] + assert "FileNotFoundError" in text + # The retry hint names the sandbox filename explicitly. + assert "loaded as" in text + assert "Budget.csv" in text + # Schema footer should also be attached so the retry has the + # load line. + assert "Dataset info" in text or "use the `load:` line" in text + + def test_generic_user_error_attaches_schema( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + schema_stdout, + reply_factory, + ): + """A KeyError on a CSV — no XLSX hint needed, but the schema + footer with column list should land so the model can fix its + column reference on retry. + """ + set_kb, set_session = file_sources + set_session([file_factories["session_csv"]("data.csv")]) + seed_s3_object(key="sessions/data.csv", body=b"a,b,c\n1,2,3\n") + + err_traceback = ( + "Traceback (most recent call last):\n" + " File \"/tmp/ipykernel_1/code.py\", line 1, in \n" + " print(df['WRONG_COL'].sum())\n" + "KeyError: 'WRONG_COL'\n" + ) + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + schema_out=schema_stdout(file="data.csv", columns="a, b, c"), + user_out="", + user_err=err_traceback, + user_is_error=True, + ) + + with ci_patch: + result = call_analyze( + filename="data.csv", + python_code="print(df['WRONG_COL'].sum())", + session_id="s1", + user_id="u1", + ) + + assert result["status"] == "error" + text = result["content"][0]["text"] + assert "KeyError" in text + assert "Dataset info" in text + assert "columns: a, b, c" in text + # The xlsx hint must NOT appear on a CSV error. + assert "loaded as" not in text + + +class TestInterpreterLifecycle: + def test_interpreter_stopped_on_success( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + schema_stdout, + reply_factory, + ): + set_kb, set_session = file_sources + set_session([file_factories["session_csv"]("data.csv")]) + seed_s3_object(key="sessions/data.csv", body=b"a,b\n1,2\n") + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + schema_out=schema_stdout(file="data.csv"), + user_out="done\n", + ) + + with ci_patch: + call_analyze( + filename="data.csv", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + assert fake.started + assert fake.stopped + + def test_interpreter_stopped_on_user_error( + self, + call_analyze, + file_sources, + file_factories, + fake_code_interpreter, + sessions_bucket, + seed_s3_object, + code_interpreter_id, + schema_stdout, + reply_factory, + ): + """The finally: stop() must run even when user code fails. + Otherwise we'd leak interpreter sessions on every bad query. + """ + set_kb, set_session = file_sources + set_session([file_factories["session_csv"]("data.csv")]) + seed_s3_object(key="sessions/data.csv", body=b"a,b\n1,2\n") + + fake, ci_patch = fake_code_interpreter + fake.reply_for = reply_factory( + schema_out=schema_stdout(file="data.csv"), + user_out="", + user_err="KeyError: 'x'\n", + user_is_error=True, + ) + + with ci_patch: + call_analyze( + filename="data.csv", + python_code="pass", + session_id="s1", + user_id="u1", + ) + + assert fake.stopped diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_build_preview_code.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_build_preview_code.py new file mode 100644 index 00000000..b6915d9d --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_build_preview_code.py @@ -0,0 +1,127 @@ +"""Tests for ``_build_preview_code`` — the schema-probe Python template +that runs inside the Code Interpreter sandbox. + +Scope note: the sandbox runs in an AWS-managed container with pandas +preinstalled; the backend's own test environment does NOT bundle pandas. +That means we can't execute the template in-process here without pulling +pandas into backend dependencies — which nothing else needs. So these +tests focus on the template's **shape**: it must parse as valid Python, +quote the filename safely (including filenames with apostrophes or +double quotes), and include the expected scorer/marker scaffolding so +regressions to the template structure are caught. + +Execution-level coverage of the scorer (does it correctly prescribe +``skiprows=4`` for a 4-row title preamble?) will land in a follow-up +issue to extract the scorer into a pure, directly-testable helper. See +#261. +""" + +import ast + +from agents.builtin_tools.spreadsheet_analysis.analyze_tool import ( + _SCHEMA_MARKER, + _build_preview_code, +) + + +class TestPreviewCodeParsesAsValidPython: + def test_simple_filename(self): + ast.parse(_build_preview_code("data.csv")) + + def test_filename_with_apostrophe(self): + """Regression: before the ``_FNAME`` indirection, a filename like + ``O'Brien data.csv`` produced invalid Python because repr() emits + double quotes around strings containing single quotes, conflicting + with the template's outer f-string quoting. + """ + ast.parse(_build_preview_code("O'Brien data.csv")) + + def test_filename_with_double_quote(self): + """Double quotes in filenames should also survive — repr() picks + single quotes when the string contains doubles. + """ + ast.parse(_build_preview_code('say "hello".csv')) + + def test_filename_with_backslashes(self): + ast.parse(_build_preview_code("path\\with\\backslashes.csv")) + + def test_filename_with_tabs_and_newlines(self): + """Whitespace escapes — Python's repr uses \\t / \\n so the + generated source stays on one line. + """ + ast.parse(_build_preview_code("file\twith\ttabs.csv")) + ast.parse(_build_preview_code("file\nwith\nnewlines.csv")) + + def test_filename_with_unicode(self): + ast.parse(_build_preview_code("Ñiño.csv")) + + def test_filename_with_braces(self): + """Curly braces in filenames must not be interpreted as f-string + placeholders. ``_FNAME`` indirection sidesteps the issue. + """ + ast.parse(_build_preview_code("{templated}.csv")) + + def test_empty_filename(self): + """Empty strings should produce valid (if useless) Python — we + don't want to fail tool construction on a bad filename; that's + the call site's job. + """ + ast.parse(_build_preview_code("")) + + +class TestPreviewCodeShape: + def test_contains_schema_markers(self): + code = _build_preview_code("x.csv") + # The marker appears at least twice — once to open, once to close. + assert code.count(repr(_SCHEMA_MARKER)) >= 2 + + def test_emits_marker_on_failure_branch(self): + """The template wraps its probe in try/except and emits the marker + on the except path too, so a probe failure doesn't leave the + outer parser hanging on a half-emitted schema. + """ + code = _build_preview_code("x.csv") + # Look for the failure branch's signature text. Resilience against + # template churn: use a stable keyword rather than exact wording. + assert "schema preview unavailable" in code + + def test_scorer_iterates_skiprows_0_to_8(self): + """Regression: the probe range is deliberate. If someone shortens + it, the scorer can't find the right header on deeply-nested + report exports. + """ + code = _build_preview_code("x.csv") + assert "range(9)" in code + + def test_references_pandas(self): + code = _build_preview_code("x.csv") + assert "pandas" in code + assert "pd.read_csv" in code + + def test_stores_filename_in_local_once(self): + """The template references ``_FNAME`` rather than re-interpolating + the raw filename into every usage. Pin this to keep the quoting + bug from regressing if someone "simplifies" the template. + """ + code = _build_preview_code("whatever.csv") + # Exactly one assignment of _FNAME. + assert code.count("_FNAME = ") == 1 + # All file operations use the local, not a re-interpolated literal. + for expected in ( + "open(_FNAME", + "pd.read_csv(_FNAME, nrows=0", + "pd.read_csv(_FNAME, skiprows=", + ): + assert expected in code + + def test_confidence_gate_still_present(self): + """The ``_prescribe`` gate is what prevents over-eager skiprows + recommendations. If it disappears, the scorer will happily point + the model at a data-row-as-header and regressions become silent. + """ + code = _build_preview_code("x.csv") + assert "_prescribe" in code + # The gate checks all three conditions — drop any and we're + # back to pre-gate behavior. + assert "_best_skip > 0" in code + assert "_win_clean_ratio" in code diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_clean_stderr.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_clean_stderr.py new file mode 100644 index 00000000..84d214e7 --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_clean_stderr.py @@ -0,0 +1,202 @@ +"""Tests for ``_clean_stderr`` — strips pandas internal frames and warning +noise from Code Interpreter tracebacks, keeping only the user-code frame +and the final exception line. + +Fixtures model real tracebacks surfaced by the interpreter: KeyError from +a missing column, ValueError from a bad dtype cast, FileNotFoundError from +an incorrect filename, a SyntaxError from malformed python_code, and a +malformed blob that doesn't match the expected traceback shape. + +These tests are important because ``_clean_stderr`` output is what the +model sees on retry. Regressions here either flood the model with +irrelevant noise (bad retries, wasted tokens) or swallow the real error +(stuck retries). See #261. +""" + +from agents.builtin_tools.spreadsheet_analysis.analyze_tool import ( + MAX_ERROR_CHARS, + _clean_stderr, +) + + +class TestCleanStderrEmptyInput: + def test_empty_string_returns_placeholder(self): + assert _clean_stderr("") == "Unknown error" + + def test_none_returns_placeholder(self): + assert _clean_stderr(None) == "Unknown error" # type: ignore[arg-type] + + +class TestCleanStderrKeyError: + """pandas KeyError — the most common failure: wrong column name.""" + + TRACEBACK = """Traceback (most recent call last): + File "/tmp/ipykernel_42/user_code.py", line 3, in + total = df['NET_AMOUNT_MISSPELLED'].sum() + ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/opt/venv/lib/python3.12/site-packages/pandas/core/frame.py", line 4090, in __getitem__ + indexer = self.columns.get_loc(key) + File "/opt/venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc + raise KeyError(key) from err +KeyError: 'NET_AMOUNT_MISSPELLED' +""" + + def test_keeps_user_frame(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "user_code.py" in cleaned + assert "NET_AMOUNT_MISSPELLED" in cleaned + + def test_drops_pandas_internal_frames(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "site-packages/pandas/" not in cleaned + assert "pandas/core/frame.py" not in cleaned + assert "get_loc" not in cleaned + + def test_includes_final_exception(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "KeyError" in cleaned + # The actual missing key should survive + assert "'NET_AMOUNT_MISSPELLED'" in cleaned + + def test_within_budget(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert len(cleaned) <= MAX_ERROR_CHARS + + +class TestCleanStderrValueError: + TRACEBACK = """Traceback (most recent call last): + File "/tmp/ipykernel_99/script.py", line 7, in + df['amount'] = df['amount'].astype(int) + ~~~~~~~~~~~~~~~~~~~~^^^^^ + File "/opt/venv/lib/python3.12/site-packages/pandas/core/generic.py", line 6534, in astype + new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors) + File "/opt/venv/lib/python3.12/site-packages/pandas/core/internals/managers.py", line 414, in astype + return self.apply( +ValueError: invalid literal for int() with base 10: '$1,234.56' +""" + + def test_user_frame_kept(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "script.py" in cleaned + assert "astype(int)" in cleaned + + def test_exception_kept(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "ValueError" in cleaned + assert "'$1,234.56'" in cleaned + + def test_pandas_internals_dropped(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "generic.py" not in cleaned + assert "managers.py" not in cleaned + + +class TestCleanStderrFileNotFoundError: + """The XLSX/CSV mismatch path — model points at the wrong filename.""" + + TRACEBACK = """Traceback (most recent call last): + File "/tmp/ipykernel_1/user_code.py", line 2, in + df = pd.read_csv('FY_27_Ledger.xlsx', low_memory=False) + File "/opt/venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv + return _read(filepath_or_buffer, kwds) +FileNotFoundError: [Errno 2] No such file or directory: 'FY_27_Ledger.xlsx' +""" + + def test_filename_preserved_for_targeted_hint(self): + """The outer tool matches on ``filename in error_msg`` to trigger + the xlsx→csv retry hint — the cleaner must keep the source + filename readable. + """ + cleaned = _clean_stderr(self.TRACEBACK) + assert "FY_27_Ledger.xlsx" in cleaned + + def test_exception_name_preserved(self): + assert "FileNotFoundError" in _clean_stderr(self.TRACEBACK) + + def test_pandas_reader_frame_dropped(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "readers.py" not in cleaned + + +class TestCleanStderrSyntaxError: + """Model wrote broken python_code — no useful stack, just the syntax + error line and caret. + """ + + TRACEBACK = """ File "/tmp/ipykernel_5/broken.py", line 2 + df = pd.read_csv('x.csv' + ^ +SyntaxError: '(' was never closed +""" + + def test_syntax_error_surfaced(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "SyntaxError" in cleaned + assert "never closed" in cleaned + + def test_user_frame_preserved(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "broken.py" in cleaned + + +class TestCleanStderrMalformed: + """If the traceback doesn't match the expected shape, we should still + return *something* useful (a tail of the raw stderr) rather than blank. + """ + + def test_no_exception_line_returns_tail(self): + weird = "line1\nline2\nline3\n\nunexpected output without traceback" + cleaned = _clean_stderr(weird) + assert cleaned != "" + assert len(cleaned) > 0 + + def test_tail_bounded_by_budget(self): + """Malformed output should not exceed the error budget — prevents a + multi-kilobyte dump of unrelated stderr from eating tool result + space on retries. + """ + weird = "\n".join(f"random noise line {i}" for i in range(200)) + cleaned = _clean_stderr(weird) + assert len(cleaned) <= MAX_ERROR_CHARS + + +class TestCleanStderrWarnings: + """DtypeWarning / FutureWarning / UserWarning are pandas noise that + appear *above* the real error. The cleaner drops the warning line and + its call-site follow-up. + """ + + TRACEBACK = """/opt/venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py:622: DtypeWarning: Columns (17) have mixed types. Specify dtype option on import or set low_memory=False. + return _read(filepath_or_buffer, kwds) +Traceback (most recent call last): + File "/tmp/ipykernel_7/code.py", line 4, in + print(df['NET'].sum()) + ~~^^^^^^^ +KeyError: 'NET' +""" + + def test_warning_dropped(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "DtypeWarning" not in cleaned + assert "mixed types" not in cleaned + + def test_real_error_preserved(self): + cleaned = _clean_stderr(self.TRACEBACK) + assert "KeyError" in cleaned + assert "'NET'" in cleaned + assert "code.py" in cleaned + + +class TestCleanStderrTruncation: + def test_output_clamped_to_max_error_chars(self): + """A long user-code frame shouldn't push the cleaned output past + MAX_ERROR_CHARS. Truncation appends an ellipsis marker. + """ + long_traceback = ( + "Traceback (most recent call last):\n" + f" File \"/tmp/ipykernel_1/code.py\", line 1, in \n" + f" {'x' * 2000}\n" + "ValueError: super long error message " + "y" * 1000 + "\n" + ) + cleaned = _clean_stderr(long_traceback) + assert len(cleaned) <= MAX_ERROR_CHARS diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_find_file.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_find_file.py new file mode 100644 index 00000000..dfe26b61 --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_find_file.py @@ -0,0 +1,212 @@ +"""Tests for ``_find_file`` — file lookup used by analyze_spreadsheet to +resolve a model-supplied filename to an S3-backed file record. + +The lookup pulls from two sources: the assistant's knowledge base +(``_get_kb_files``) and the session's attachments (``_get_session_files``). +The twist is an alias pass: XLSX↔CSV for tabular files, so +``analyze_spreadsheet(filename="foo.csv", ...)`` resolves to the backing +``foo.xlsx`` (and vice versa). Without this, the model's "retry with the +sandbox filename" guess — which the docstring asks for — would fail at +the tool boundary (#206). + +These tests pin down: +- exact-match wins over the alias pass +- aliasing only triggers for tabular extensions (no foo.pdf ↔ foo.docx) +- both sources contribute candidates +- case-insensitive exact match + +After #260, ``_find_file`` and both helpers are ``async def``; the +``patch`` calls install ``AsyncMock`` side-effects so awaiting them +yields the configured return values. +""" + +from unittest.mock import AsyncMock, patch + +import pytest + +from agents.builtin_tools.spreadsheet_analysis.analyze_tool import _find_file + + +def _kb_file(filename: str, content_type: str = "") -> dict: + return { + "filename": filename, + "source": "knowledge_base", + "content_type": content_type, + "size_bytes": 1234, + "document_id": "doc-1", + "s3_key": f"kb/{filename}", + } + + +def _session_file(filename: str, content_type: str = "") -> dict: + return { + "filename": filename, + "source": "chat_attachment", + "content_type": content_type, + "size_bytes": 1234, + "document_id": "upload-1", + "s3_key": f"session/{filename}", + "s3_bucket": "test-bucket", + } + + +def _patch_sources(*, kb=None, session=None): + """Install AsyncMock patches for both file-source helpers. + + Returns a tuple of (kb_patch, session_patch) that callers apply via + ``with`` so the mocks tear down cleanly between tests. + """ + kb_value = list(kb or []) + session_value = list(session or []) + return ( + patch( + "agents.builtin_tools.spreadsheet_analysis.analyze_tool._get_kb_files", + new=AsyncMock(return_value=kb_value), + ), + patch( + "agents.builtin_tools.spreadsheet_analysis.analyze_tool._get_session_files", + new=AsyncMock(return_value=session_value), + ), + ) + + +class TestExactMatchWins: + @pytest.mark.asyncio + async def test_exact_xlsx_match_in_session(self): + kb_p, sess_p = _patch_sources( + session=[_session_file("Report.xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")], + ) + with kb_p, sess_p: + result = await _find_file("Report.xlsx", assistant_id=None, session_id="s1") + assert result is not None + assert result["filename"] == "Report.xlsx" + + @pytest.mark.asyncio + async def test_exact_csv_match_in_kb(self): + kb_p, sess_p = _patch_sources(kb=[_kb_file("Q1.csv", "text/csv")]) + with kb_p, sess_p: + result = await _find_file("Q1.csv", assistant_id="ast-1", session_id="s1") + assert result is not None + assert result["filename"] == "Q1.csv" + assert result["source"] == "knowledge_base" + + @pytest.mark.asyncio + async def test_exact_match_case_insensitive(self): + kb_p, sess_p = _patch_sources( + session=[_session_file("Budget.XLSX", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")], + ) + with kb_p, sess_p: + result = await _find_file("budget.xlsx", assistant_id=None, session_id="s1") + assert result is not None + assert result["filename"] == "Budget.XLSX" + + @pytest.mark.asyncio + async def test_exact_match_preferred_over_alias(self): + """If both ``foo.xlsx`` and ``foo.csv`` exist and the model asks + for ``foo.csv``, exact match should win — no surprise aliasing. + """ + kb_p, sess_p = _patch_sources( + session=[ + _session_file("Data.xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"), + _session_file("Data.csv", "text/csv"), + ], + ) + with kb_p, sess_p: + result = await _find_file("Data.csv", assistant_id=None, session_id="s1") + assert result is not None + assert result["filename"] == "Data.csv" + + +class TestAliasPass: + @pytest.mark.asyncio + async def test_csv_request_resolves_xlsx_source(self): + """Model asked for ``foo.csv`` (sandbox filename), only ``foo.xlsx`` + is attached. Alias pass finds it. + """ + kb_p, sess_p = _patch_sources( + session=[_session_file("FY_27_Ledger.xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")], + ) + with kb_p, sess_p: + result = await _find_file("FY_27_Ledger.csv", assistant_id=None, session_id="s1") + assert result is not None + assert result["filename"] == "FY_27_Ledger.xlsx" + + @pytest.mark.asyncio + async def test_xlsx_request_resolves_csv_source(self): + """Reverse direction — model asked for ``foo.xlsx`` but only + ``foo.csv`` is attached (rare but handled). + """ + kb_p, sess_p = _patch_sources( + session=[_session_file("Q3.csv", "text/csv")], + ) + with kb_p, sess_p: + result = await _find_file("Q3.xlsx", assistant_id=None, session_id="s1") + assert result is not None + assert result["filename"] == "Q3.csv" + + @pytest.mark.asyncio + async def test_alias_only_applies_to_tabular(self): + """``foo.pdf`` must not alias to ``foo.docx``. The alias pass is + gated on target being a tabular extension. + """ + kb_p, sess_p = _patch_sources( + session=[_session_file("report.docx", "application/vnd.openxmlformats-officedocument.wordprocessingml.document")], + ) + with kb_p, sess_p: + result = await _find_file("report.pdf", assistant_id=None, session_id="s1") + assert result is None + + @pytest.mark.asyncio + async def test_alias_skips_non_tabular_candidate(self): + """Even if the target is tabular, candidates with non-tabular + content/type shouldn't match. Prevents e.g. alias bleeding + ``.docx`` into a ``.csv`` request. + """ + kb_p, sess_p = _patch_sources( + session=[_session_file("data.pdf", "application/pdf")], + ) + with kb_p, sess_p: + result = await _find_file("data.csv", assistant_id=None, session_id="s1") + assert result is None + + +class TestSourceOrder: + @pytest.mark.asyncio + async def test_kb_checked_before_session(self): + """When assistant_id is set, KB files are consulted first. This + matches behavior documented in the tool: the KB is the + authoritative source for assistants. + """ + kb_p, sess_p = _patch_sources( + kb=[_kb_file("shared.csv", "text/csv")], + session=[_session_file("shared.csv", "text/csv")], + ) + with kb_p, sess_p: + result = await _find_file("shared.csv", assistant_id="ast-1", session_id="s1") + assert result is not None + assert result["source"] == "knowledge_base" + + @pytest.mark.asyncio + async def test_no_assistant_skips_kb_lookup(self): + """With ``assistant_id=None``, KB is not queried — only session + files. Avoids spurious DynamoDB calls on non-assistant chats. + """ + kb_mock = AsyncMock(return_value=[_kb_file("only-in-kb.csv", "text/csv")]) + sess_mock = AsyncMock(return_value=[_session_file("only-in-session.csv", "text/csv")]) + with patch( + "agents.builtin_tools.spreadsheet_analysis.analyze_tool._get_kb_files", + new=kb_mock, + ), patch( + "agents.builtin_tools.spreadsheet_analysis.analyze_tool._get_session_files", + new=sess_mock, + ): + result = await _find_file("only-in-kb.csv", assistant_id=None, session_id="s1") + kb_mock.assert_not_called() + # KB file isn't visible; only session files considered. + assert result is None + + @pytest.mark.asyncio + async def test_returns_none_when_not_found(self): + kb_p, sess_p = _patch_sources() + with kb_p, sess_p: + assert await _find_file("nope.csv", assistant_id="ast-1", session_id="s1") is None diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_helpers.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_helpers.py new file mode 100644 index 00000000..c723a8bf --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_helpers.py @@ -0,0 +1,149 @@ +"""Unit tests for the small pure helpers in analyze_tool.py. + +These cover the boring-but-critical glue: output truncation, schema-marker +extraction, sheet-name sanitization, and safe int parsing. The logic is +simple so the tests are small — their job is to lock in the current +behavior so the async refactor doesn't regress the happy paths (#261). +""" + +from agents.builtin_tools.spreadsheet_analysis.analyze_tool import ( + MAX_OUTPUT_CHARS, + _extract_schema_preview, + _safe_int, + _sanitize_sheet_name, + _truncate_output, + _SCHEMA_MARKER, +) + + +class TestTruncateOutput: + def test_empty_returns_empty(self): + assert _truncate_output("") == "" + + def test_none_returns_none(self): + # The helper short-circuits on falsy inputs. Preserve that. + assert _truncate_output(None) is None # type: ignore[arg-type] + + def test_under_cap_unchanged(self): + text = "x" * (MAX_OUTPUT_CHARS - 1) + assert _truncate_output(text) == text + + def test_at_cap_unchanged(self): + text = "x" * MAX_OUTPUT_CHARS + assert _truncate_output(text) == text + + def test_over_cap_truncated_with_marker(self): + text = "x" * (MAX_OUTPUT_CHARS + 500) + out = _truncate_output(text) + assert out.startswith("x" * MAX_OUTPUT_CHARS) + assert "truncated" in out + assert f"{MAX_OUTPUT_CHARS:,}" in out + assert f"{len(text):,}" in out + + +class TestExtractSchemaPreview: + def test_no_marker_returns_empty_block_and_full_stdout(self): + stdout = "some tool output\nwith no marker\n" + schema, remaining = _extract_schema_preview(stdout) + assert schema == "" + assert remaining == stdout + + def test_full_block_between_markers(self): + stdout = ( + f"{_SCHEMA_MARKER}\n" + "file: data.csv (10 rows x 3 cols)\n" + "columns: a, b, c\n" + f"{_SCHEMA_MARKER}\n" + ) + schema, remaining = _extract_schema_preview(stdout) + assert "file: data.csv" in schema + assert "columns: a, b, c" in schema + # The remaining stdout should be empty (or a stripped empty string) + assert remaining == "" or remaining.strip() == "" + + def test_schema_surrounded_by_user_output(self): + """User code may print before AND after the schema block. + + The helper should pull out just the schema and preserve both sides + of the user stdout — important because the tool concatenates the + two halves back together when rendering the final response. + """ + stdout = ( + "Hello from user code\n" + f"{_SCHEMA_MARKER}\n" + "file: data.csv\n" + f"{_SCHEMA_MARKER}\n" + "After schema user output\n" + ) + schema, remaining = _extract_schema_preview(stdout) + assert "file: data.csv" in schema + assert "Hello from user code" in remaining + assert "After schema user output" in remaining + + def test_marker_present_only_once_returns_empty_block(self): + """A single marker (not bracketed) is malformed — treat as no schema. + + Prevents us from accidentally surfacing half of a stream as a + "schema" when the bootstrap failed mid-emit. + """ + stdout = f"partial {_SCHEMA_MARKER}\ntruncated" + schema, remaining = _extract_schema_preview(stdout) + assert schema == "" + # Original stdout returned on the malformed path + assert remaining == stdout + + +class TestSafeInt: + def test_parses_int(self): + assert _safe_int("42") == 42 + + def test_parses_large_int(self): + assert _safe_int("1000000") == 1_000_000 + + def test_strips_whitespace(self): + assert _safe_int(" 7 ") == 7 + + def test_returns_zero_for_empty(self): + assert _safe_int("") == 0 + + def test_returns_zero_for_garbage(self): + assert _safe_int("not-a-number") == 0 + + def test_returns_zero_for_none(self): + assert _safe_int(None) == 0 # type: ignore[arg-type] + + def test_parses_negative(self): + assert _safe_int("-5") == -5 + + +class TestSanitizeSheetName: + def test_simple_name_lowercased(self): + assert _sanitize_sheet_name("Summary") == "summary" + + def test_spaces_become_underscore(self): + assert _sanitize_sheet_name("Q1 2026") == "q1_2026" + + def test_multiple_non_alnum_collapse_to_single_underscore(self): + assert _sanitize_sheet_name("Q1 --- 2026") == "q1_2026" + + def test_slashes_replaced(self): + assert _sanitize_sheet_name("Sales/2026") == "sales_2026" + + def test_unicode_replaced(self): + # Non-ASCII characters aren't in [A-Za-z0-9] so they all collapse. + assert _sanitize_sheet_name("Ñiño") == "i_o" + + def test_leading_trailing_punctuation_stripped(self): + assert _sanitize_sheet_name("--Budget--") == "budget" + + def test_empty_returns_fallback(self): + assert _sanitize_sheet_name("") == "sheet" + + def test_all_punctuation_returns_fallback(self): + # Everything collapses to "" post-strip, fallback kicks in. + assert _sanitize_sheet_name("---") == "sheet" + + def test_deterministic(self): + # Same input always yields same output — callers rely on this + # to predict filenames. + assert _sanitize_sheet_name("Q1 2026") == _sanitize_sheet_name("Q1 2026") diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_list_spreadsheets.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_list_spreadsheets.py new file mode 100644 index 00000000..6acc21cc --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_list_spreadsheets.py @@ -0,0 +1,388 @@ +"""Tests for the ``list_spreadsheets`` tool factory and its two private +helpers (``_get_kb_files``, ``_get_session_files``). + +The factory (``make_list_spreadsheets_tool``) builds a closure-bound tool +the agent can invoke. Helper-level tests exercise the real boto3 / +DynamoDB paths via moto so the query / filter / field-mapping logic is +under test, not mocked out. + +After #260 the helpers and the tool itself are ``async def``; tests that +invoke them are marked with ``@pytest.mark.asyncio`` and use ``await``. + +See #261. +""" + +import pytest + +from agents.builtin_tools.spreadsheet_analysis.list_spreadsheets_tool import ( + _get_kb_files, + _get_session_files, + _is_tabular_file, + make_list_spreadsheets_tool, +) + + +# --------------------------------------------------------------------------- +# _is_tabular_file — thin wrapper; delegate to shared is_tabular_file +# --------------------------------------------------------------------------- + + +class TestIsTabularFile: + def test_csv_by_extension(self): + assert _is_tabular_file("data.csv", "") is True + + def test_csv_by_mime(self): + assert _is_tabular_file("anything", "text/csv") is True + + def test_xlsx_by_extension(self): + assert _is_tabular_file("data.xlsx", "") is True + + def test_xlsx_by_mime(self): + mime = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + assert _is_tabular_file("anything", mime) is True + + def test_pdf_rejected(self): + assert _is_tabular_file("report.pdf", "application/pdf") is False + + def test_docx_rejected(self): + docx_mime = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + assert _is_tabular_file("report.docx", docx_mime) is False + + +# --------------------------------------------------------------------------- +# list_spreadsheets tool — factory + invocation +# --------------------------------------------------------------------------- + + +def _call_tool(tool) -> dict: + """Invoke a Strands-decorated async tool and unwrap the result. + + ``@tool`` returns a wrapper that exposes the original coroutine + function via ``__wrapped__``. We ``await`` it from the test, which + must be marked ``@pytest.mark.asyncio``. + """ + fn = getattr(tool, "__wrapped__", None) or tool + return fn() + + +class TestMakeListSpreadsheetsTool: + @pytest.mark.asyncio + async def test_empty_state_returns_helpful_message(self, file_sources): + set_kb, set_session = file_sources + set_kb([]) + set_session([]) + + tool = make_list_spreadsheets_tool( + assistant_id="ast-1", session_id="s1", user_id="u1" + ) + result = await _call_tool(tool) + + assert result["status"] == "success" + text = result["content"][0]["text"] + assert "No spreadsheet files" in text + # "files" key should NOT be present on the empty path so the model + # doesn't loop on an empty list. + assert "files" not in result + + @pytest.mark.asyncio + async def test_kb_and_session_files_merged(self, file_sources, file_factories): + set_kb, set_session = file_sources + set_kb([file_factories["kb_xlsx"]("Budget.xlsx")]) + set_session([file_factories["session_csv"]("notes.csv")]) + + tool = make_list_spreadsheets_tool( + assistant_id="ast-1", session_id="s1", user_id="u1" + ) + result = await _call_tool(tool) + + assert result["status"] == "success" + filenames = [f["filename"] for f in result["files"]] + assert filenames == ["Budget.xlsx", "notes.csv"] + + text = result["content"][0]["text"] + assert "Budget.xlsx" in text + assert "knowledge_base" in text + assert "notes.csv" in text + assert "chat_attachment" in text + + @pytest.mark.asyncio + async def test_no_assistant_skips_kb_call(self, file_sources): + """Without an assistant_id, KB files aren't queried — locks in the + conditional branch so we don't regress and start spamming DynamoDB + on non-assistant chats. + """ + from unittest.mock import patch + + kb_calls = [] + + async def _track(_aid): + kb_calls.append(_aid) + return [] + + set_kb, set_session = file_sources + set_session([]) + with patch( + "agents.builtin_tools.spreadsheet_analysis.list_spreadsheets_tool._get_kb_files", + side_effect=_track, + ): + tool = make_list_spreadsheets_tool( + assistant_id=None, session_id="s1", user_id="u1" + ) + await _call_tool(tool) + + assert kb_calls == [], "KB lookup should be skipped when assistant_id is None" + + @pytest.mark.asyncio + async def test_size_formatted_in_kb(self, file_sources, file_factories): + """Files are rendered with their size in KB for the preview text. + Pinning this so the formatter change doesn't silently regress. + """ + set_kb, set_session = file_sources + set_kb([]) + set_session([file_factories["session_csv"]("tiny.csv", size=2560)]) + + tool = make_list_spreadsheets_tool( + assistant_id=None, session_id="s1", user_id="u1" + ) + result = await _call_tool(tool) + text = result["content"][0]["text"] + # 2560 bytes → 3 KB with the current round-to-nearest formatter. + assert "3 KB" in text or "2 KB" in text # allow either rounding + + +# --------------------------------------------------------------------------- +# _get_kb_files — DynamoDB query with status filter, via moto +# --------------------------------------------------------------------------- + + +XLSX_MIME = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + + +class TestGetKbFilesDynamoDB: + """Exercise the real DynamoDB query path against a moto-backed table. + + This replaces the earlier MagicMock-based tests: those verified that + ``table.query`` was called, but didn't actually check the schema + (attribute names, key-condition expression) matches what production + writes. Moto does. + + ``_get_kb_files`` is ``async def`` (see #260) so each test awaits it. + """ + + @pytest.mark.asyncio + async def test_no_table_env_returns_empty(self, monkeypatch): + """Helper bails out cleanly when the env var isn't set at all, + rather than crashing on a missing table. + """ + monkeypatch.delenv("DYNAMODB_ASSISTANTS_TABLE_NAME", raising=False) + assert await _get_kb_files("ast-1") == [] + + @pytest.mark.asyncio + async def test_completed_tabular_file_included(self, assistants_table, seed_kb_doc): + seed_kb_doc( + assistant_id="ast-1", + filename="Budget.xlsx", + content_type=XLSX_MIME, + size_bytes=1024, + ) + files = await _get_kb_files("ast-1") + assert len(files) == 1 + assert files[0]["filename"] == "Budget.xlsx" + assert files[0]["source"] == "knowledge_base" + assert files[0]["size_bytes"] == 1024 + + @pytest.mark.asyncio + async def test_non_tabular_file_filtered_out(self, assistants_table, seed_kb_doc): + seed_kb_doc( + assistant_id="ast-1", + filename="report.pdf", + content_type="application/pdf", + ) + assert await _get_kb_files("ast-1") == [] + + @pytest.mark.asyncio + async def test_incomplete_status_filtered_out(self, assistants_table, seed_kb_doc): + seed_kb_doc( + assistant_id="ast-1", + filename="Pending.xlsx", + content_type=XLSX_MIME, + status="processing", # not "complete" + ) + assert await _get_kb_files("ast-1") == [] + + @pytest.mark.asyncio + async def test_mixed_statuses_filters_correctly(self, assistants_table, seed_kb_doc): + seed_kb_doc(assistant_id="ast-1", filename="done.csv", + content_type="text/csv", status="complete") + seed_kb_doc(assistant_id="ast-1", filename="broken.csv", + content_type="text/csv", status="failed") + seed_kb_doc(assistant_id="ast-1", filename="notes.txt", + content_type="text/plain", status="complete") + + files = await _get_kb_files("ast-1") + assert len(files) == 1 + assert files[0]["filename"] == "done.csv" + + @pytest.mark.asyncio + async def test_isolates_by_assistant_id(self, assistants_table, seed_kb_doc): + """The ``PK = AST#`` key condition partitions by assistant. + Documents under a different assistant must not leak through. + """ + seed_kb_doc(assistant_id="ast-1", filename="mine.csv", + content_type="text/csv") + seed_kb_doc(assistant_id="ast-other", filename="theirs.csv", + content_type="text/csv") + + files = await _get_kb_files("ast-1") + assert [f["filename"] for f in files] == ["mine.csv"] + + @pytest.mark.asyncio + async def test_dynamodb_exception_returns_empty( + self, aws_mocked, monkeypatch, caplog + ): + """Graceful degradation: a query failure shouldn't crash the + tool. Points the helper at a table that doesn't exist *within + moto* so the failure mode is the production-realistic + ``ResourceNotFoundException`` rather than a credentials error + (which would mask a real graceful-degradation regression). + """ + import logging + + monkeypatch.setenv("DYNAMODB_ASSISTANTS_TABLE_NAME", "nonexistent-table") + with caplog.at_level(logging.ERROR): + files = await _get_kb_files("ast-1") + assert files == [] + # Verify we actually hit the exception branch — passing solely + # because the early-return fired would be a silent regression + # of the graceful-degradation contract the next refactor + # (#260) needs to preserve. + assert any( + "ResourceNotFoundException" in record.getMessage() + or "not found" in record.getMessage().lower() + for record in caplog.records + ), f"expected error log, got: {[r.getMessage() for r in caplog.records]}" + + @pytest.mark.asyncio + async def test_legacy_snake_case_fields_supported(self, assistants_table, seed_kb_doc): + """The repo stores camelCase but some legacy items use snake_case + aliases. Both must work. + """ + seed_kb_doc( + assistant_id="ast-1", + filename="legacy.xlsx", + content_type=XLSX_MIME, + size_bytes=500, + use_snake_case=True, + ) + files = await _get_kb_files("ast-1") + assert len(files) == 1 + assert files[0]["filename"] == "legacy.xlsx" + assert files[0]["size_bytes"] == 500 + + +# --------------------------------------------------------------------------- +# _get_session_files — async repo, via moto +# --------------------------------------------------------------------------- + + +class TestGetSessionFiles: + """Real repository queries against the moto-backed files table. + + After #260, ``_get_session_files`` awaits the repository directly + instead of running ``asyncio.run`` inside a thread-pool. These + tests exercise the straightened-out async path end-to-end. + """ + + @pytest.mark.asyncio + async def test_returns_tabular_files_only( + self, file_repository, seed_session_file + ): + await seed_session_file( + session_id="s1", upload_id="u-xlsx", + filename="Budget.xlsx", mime_type=XLSX_MIME, + ) + await seed_session_file( + session_id="s1", upload_id="u-md", + filename="README.md", mime_type="text/markdown", + ) + await seed_session_file( + session_id="s1", upload_id="u-csv", + filename="data.csv", mime_type="text/csv", + ) + + files = await _get_session_files("s1") + filenames = {f["filename"] for f in files} + assert filenames == {"Budget.xlsx", "data.csv"} + assert "README.md" not in filenames + + @pytest.mark.asyncio + async def test_empty_session_returns_empty(self, file_repository): + # No files seeded — list_session_files returns []. + assert await _get_session_files("s1") == [] + + @pytest.mark.asyncio + async def test_missing_table_env_returns_empty(self, aws_mocked, monkeypatch, caplog): + """Pointing the repo at a table that doesn't exist exercises the + exception path inside the async helper. Tool should return an + empty list, not crash. + + Uses ``caplog`` to confirm the error was actually logged — + otherwise this test could regress silently if a future refactor + made the helper return ``[]`` without ever reaching the + exception branch. + """ + import logging + + # Reset the module-level singleton so the new env var is picked + # up on the next ``get_file_upload_repository()`` call — otherwise + # we inherit the repo bound to whatever table name another test + # happened to set first. + import apis.shared.files.repository as repo_module + monkeypatch.setattr(repo_module, "_repository_instance", None) + monkeypatch.setenv("DYNAMODB_USER_FILES_TABLE_NAME", "no-such-table") + + with caplog.at_level(logging.ERROR): + files = await _get_session_files("s1") + assert files == [] + assert any( + "ResourceNotFoundException" in record.getMessage() + or "not found" in record.getMessage().lower() + for record in caplog.records + ), f"expected error log, got: {[r.getMessage() for r in caplog.records]}" + + @pytest.mark.asyncio + async def test_record_structure( + self, file_repository, seed_session_file + ): + """Session records need specific keys so analyze_tool._download_file + can find the S3 bucket/key. Lock the contract. + """ + await seed_session_file( + session_id="s1", upload_id="u-1", + filename="Q1.csv", mime_type="text/csv", + ) + files = await _get_session_files("s1") + assert files[0].keys() >= { + "filename", "source", "content_type", "size_bytes", + "document_id", "s3_key", "s3_bucket", + } + assert files[0]["source"] == "chat_attachment" + + @pytest.mark.asyncio + async def test_isolates_by_session_id( + self, file_repository, seed_session_file + ): + """The session index must partition: a file attached to session + A should not appear in session B's list. + """ + await seed_session_file( + session_id="s1", upload_id="u-a", + filename="a.csv", mime_type="text/csv", + ) + await seed_session_file( + session_id="s2", upload_id="u-b", + filename="b.csv", mime_type="text/csv", + ) + + files = await _get_session_files("s1") + assert [f["filename"] for f in files] == ["a.csv"] diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_sheet_inventory.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_sheet_inventory.py new file mode 100644 index 00000000..1b71a694 --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_sheet_inventory.py @@ -0,0 +1,307 @@ +"""Tests for ``_parse_sheet_inventory`` and ``_format_sheet_note`` — the +parser for the XLSX bootstrap's pipe-delimited sheet inventory, and the +markdown footer builder that surfaces it to the model. + +The inventory flows from the sandbox's stdout back to the tool response, +so regressions here would either silently drop multi-sheet support or +mis-report which sheets were included/skipped. See #261. +""" + +from agents.builtin_tools.spreadsheet_analysis.analyze_tool import ( + MAX_ROWS_PER_SHEET, + _SHEETS_MARKER, + _format_sheet_note, + _parse_sheet_inventory, +) + + +def _wrap_block(lines: list[str]) -> str: + """Helper: wrap inventory lines with the sheet markers as the bootstrap + would emit them. + """ + body = "\n".join(lines) + return f"{_SHEETS_MARKER}\n{body}\n{_SHEETS_MARKER}\n" + + +class TestParseSheetInventoryEmpty: + def test_no_marker_returns_empty_inventory(self): + result = _parse_sheet_inventory("some unrelated stdout") + assert result["total"] == 0 + assert result["sheets"] == [] + assert result["skipped"] == 0 + assert result["has_primary_alias"] is False + + def test_empty_string_returns_empty_inventory(self): + result = _parse_sheet_inventory("") + assert result["sheets"] == [] + + def test_single_marker_returns_empty_inventory(self): + """Malformed emission with only one marker — don't try to parse.""" + result = _parse_sheet_inventory(f"partial {_SHEETS_MARKER}\nsheet|x|x|0|0|") + # Behavior: parser splits on marker; only one marker means no + # bracketed block. Should still return a safe empty structure. + # Whether it returns data or empty is implementation-defined, but + # it must not raise. + assert isinstance(result, dict) + + +class TestParseSheetInventorySingleSheet: + def test_single_sheet_no_truncation(self): + stdout = _wrap_block([ + "total: 1", + "converted: 1", + "skipped: 0", + "sheet|Summary|Budget.csv|100|0|", + ]) + result = _parse_sheet_inventory(stdout) + assert result["total"] == 1 + assert result["converted"] == 1 + assert result["skipped"] == 0 + assert len(result["sheets"]) == 1 + assert result["sheets"][0]["name"] == "Summary" + assert result["sheets"][0]["path"] == "Budget.csv" + assert result["sheets"][0]["rows"] == 100 + assert result["sheets"][0]["truncated"] is False + assert result["sheets"][0]["primary_alias"] is None + assert result["has_primary_alias"] is False + + def test_truncation_flag_parsed(self): + stdout = _wrap_block([ + "total: 1", + "converted: 1", + "skipped: 0", + "sheet|BigSheet|data.csv|500000|1|", + ]) + result = _parse_sheet_inventory(stdout) + assert result["sheets"][0]["truncated"] is True + + +class TestParseSheetInventoryMultiSheet: + def test_multi_sheet_with_primary_alias(self): + stdout = _wrap_block([ + "total: 3", + "converted: 3", + "skipped: 0", + "sheet|Summary|Budget.summary.csv|12|0|Budget.csv", + "sheet|Transactions|Budget.transactions.csv|18551|0|", + "sheet|Notes|Budget.notes.csv|5|0|", + ]) + result = _parse_sheet_inventory(stdout) + assert result["total"] == 3 + assert result["converted"] == 3 + assert len(result["sheets"]) == 3 + assert result["has_primary_alias"] is True + assert result["sheets"][0]["primary_alias"] == "Budget.csv" + # Sibling sheets don't carry the alias. + assert result["sheets"][1]["primary_alias"] is None + assert result["sheets"][2]["primary_alias"] is None + + def test_skipped_sheets_preview(self): + stdout = _wrap_block([ + "total: 30", + "converted: 25", + "skipped: 5", + "skipped_names: ['Sheet26', 'Sheet27', 'Sheet28', 'Sheet29', 'Sheet30']", + "sheet|Sheet1|data.sheet1.csv|10|0|", + ]) + result = _parse_sheet_inventory(stdout) + assert result["skipped"] == 5 + assert result["skipped_preview"] == [ + "Sheet26", "Sheet27", "Sheet28", "Sheet29", "Sheet30", + ] + + def test_sheet_names_with_special_chars_via_literal_eval(self): + """Sheet names can contain commas, apostrophes, etc. The skipped_names + field is a Python list literal — ast.literal_eval handles quoting + correctly. This locks in the contract. + """ + stdout = _wrap_block([ + "total: 5", + "converted: 3", + "skipped: 2", + 'skipped_names: ["O\'Brien, J.", "Q1, 2026"]', + "sheet|Main|data.main.csv|10|0|", + ]) + result = _parse_sheet_inventory(stdout) + # Both names survive round-trip. + assert "O'Brien, J." in result["skipped_preview"] + assert "Q1, 2026" in result["skipped_preview"] + + def test_malformed_skipped_names_gracefully_ignored(self): + """If the literal is invalid, we don't crash — we just skip it.""" + stdout = _wrap_block([ + "total: 10", + "converted: 5", + "skipped: 5", + "skipped_names: not-a-valid-literal", + "sheet|Main|data.main.csv|10|0|", + ]) + result = _parse_sheet_inventory(stdout) + assert result["skipped_preview"] == [] + # Other fields still populated. + assert result["total"] == 10 + + +class TestParseSheetInventoryMalformedSheetLines: + def test_truncated_sheet_line_skipped(self): + """A sheet line with fewer than 6 pipe-delimited fields is + skipped rather than crashing the parser. + """ + stdout = _wrap_block([ + "total: 2", + "converted: 2", + "skipped: 0", + "sheet|Valid|data.csv|10|0|", + "sheet|Broken|truncated", # too few fields + ]) + result = _parse_sheet_inventory(stdout) + # Only the valid sheet is kept. + assert len(result["sheets"]) == 1 + assert result["sheets"][0]["name"] == "Valid" + + def test_integer_fields_with_whitespace(self): + """``_safe_int`` handles surrounding whitespace — regression + guard: the parser strips on its own too. + """ + stdout = _wrap_block([ + "total: 42 ", + "converted: 10 ", + "skipped: 32", + "sheet|S|p.csv| 500 | 0 |", + ]) + result = _parse_sheet_inventory(stdout) + assert result["total"] == 42 + assert result["converted"] == 10 + assert result["skipped"] == 32 + assert result["sheets"][0]["rows"] == 500 + + +# --------------------------------------------------------------------------- +# _format_sheet_note +# --------------------------------------------------------------------------- + + +class TestFormatSheetNoteSingleSheet: + def test_single_sheet_no_truncation_returns_empty(self): + """Single-sheet workbook without truncation is the boring case — + no message needed. + """ + inventory = { + "total": 1, + "converted": 1, + "skipped": 0, + "skipped_preview": [], + "sheets": [ + {"name": "Sheet1", "path": "data.csv", "rows": 100, + "truncated": False, "primary_alias": None}, + ], + "has_primary_alias": False, + } + assert _format_sheet_note(inventory) == "" + + def test_single_sheet_truncated_surfaces_warning(self): + inventory = { + "total": 1, + "converted": 1, + "skipped": 0, + "skipped_preview": [], + "sheets": [ + {"name": "BigSheet", "path": "data.csv", + "rows": MAX_ROWS_PER_SHEET, "truncated": True, + "primary_alias": None}, + ], + "has_primary_alias": False, + } + note = _format_sheet_note(inventory) + assert note != "" + assert "truncated" in note.lower() + assert "BigSheet" in note + assert f"{MAX_ROWS_PER_SHEET:,}" in note + + +class TestFormatSheetNoteMultiSheet: + def test_all_sheets_converted(self): + inventory = { + "total": 3, + "converted": 3, + "skipped": 0, + "skipped_preview": [], + "sheets": [ + {"name": "Summary", "path": "Budget.summary.csv", "rows": 12, + "truncated": False, "primary_alias": "Budget.csv"}, + {"name": "Transactions", "path": "Budget.transactions.csv", + "rows": 18551, "truncated": False, "primary_alias": None}, + {"name": "Notes", "path": "Budget.notes.csv", "rows": 5, + "truncated": False, "primary_alias": None}, + ], + "has_primary_alias": True, + } + note = _format_sheet_note(inventory) + # Full inventory listed so the model can pick or combine. + assert "Available sheets" in note + assert "Summary" in note + assert "Transactions" in note + assert "Notes" in note + assert "Budget.summary.csv" in note + assert "Budget.transactions.csv" in note + assert "18,551" in note # row count formatted with commas + + def test_skipped_sheets_surfaced_with_names(self): + inventory = { + "total": 30, + "converted": 25, + "skipped": 5, + "skipped_preview": ["Q6", "Q7", "Q8", "Q9", "Q10"], + "sheets": [ + {"name": f"Q{i + 1}", "path": f"Budget.q{i + 1}.csv", + "rows": 100, "truncated": False, "primary_alias": None} + for i in range(25) + ], + "has_primary_alias": False, + } + note = _format_sheet_note(inventory) + assert "30 sheets" in note + assert "first 25" in note + assert "Q6" in note + assert "Q10" in note + # Tells the user what to do about it. + assert "split" in note.lower() or "export" in note.lower() + + def test_skipped_many_includes_more_suffix(self): + inventory = { + "total": 100, + "converted": 25, + "skipped": 75, + "skipped_preview": ["A", "B", "C", "D", "E"], + "sheets": [ + {"name": f"S{i}", "path": f"d.s{i}.csv", "rows": 1, + "truncated": False, "primary_alias": None} + for i in range(25) + ], + "has_primary_alias": False, + } + note = _format_sheet_note(inventory) + assert "+70 more" in note # 75 skipped - 5 shown = 70 + + def test_truncated_sheet_annotation_in_list(self): + inventory = { + "total": 2, + "converted": 2, + "skipped": 0, + "skipped_preview": [], + "sheets": [ + {"name": "Huge", "path": "wb.huge.csv", + "rows": MAX_ROWS_PER_SHEET, "truncated": True, + "primary_alias": None}, + {"name": "Small", "path": "wb.small.csv", "rows": 10, + "truncated": False, "primary_alias": None}, + ], + "has_primary_alias": False, + } + note = _format_sheet_note(inventory) + # The truncated row should have a specific tag; the other shouldn't. + lines = note.splitlines() + huge_line = next(line for line in lines if "Huge" in line) + small_line = next(line for line in lines if "Small" in line) + assert "truncated" in huge_line.lower() + assert "truncated" not in small_line.lower() diff --git a/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_strip_first_row.py b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_strip_first_row.py new file mode 100644 index 00000000..505f043b --- /dev/null +++ b/backend/tests/agents/builtin_tools/spreadsheet_analysis/test_strip_first_row.py @@ -0,0 +1,70 @@ +"""Tests for ``_strip_first_row`` — drops the ``first_row:`` line from a +schema footer on the error path to keep retry responses token-efficient. + +Simple helper but load-bearing: every analyze_spreadsheet error retry goes +through it, and a bug here silently bloats every follow-up turn by ~1K +tokens (#261). +""" + +from agents.builtin_tools.spreadsheet_analysis.analyze_tool import _strip_first_row + + +class TestStripFirstRow: + def test_drops_first_row_line(self): + schema = ( + "file: data.csv (100 rows x 5 cols)\n" + "load: pd.read_csv('data.csv', low_memory=False)\n" + "columns: a, b, c, d, e\n" + "first_row: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}\n" + ) + result = _strip_first_row(schema) + assert "first_row:" not in result + assert "file: data.csv" in result + assert "load:" in result + assert "columns:" in result + + def test_no_first_row_line_unchanged(self): + """If the schema footer doesn't have a first_row line (malformed or + schema-preview-failed path), return it as-is. Don't lose structure + trying to remove something that isn't there. + """ + schema = ( + "file: data.csv (100 rows x 5 cols)\n" + "load: pd.read_csv('data.csv', low_memory=False)\n" + "columns: a, b, c, d, e" + ) + result = _strip_first_row(schema) + assert result.count("\n") == schema.count("\n") + assert "file: data.csv" in result + assert "columns:" in result + + def test_empty_input_returns_empty_string(self): + assert _strip_first_row("") == "" + + def test_only_first_row_line_returns_empty(self): + assert _strip_first_row("first_row: {'a': 1}") == "" + + def test_first_row_with_leading_whitespace_not_stripped(self): + """The helper is strict: only lines whose raw text starts with + ``first_row:`` are dropped. Indented variants (which we don't emit) + should pass through. Pinning this so a future "be more lenient" + change is deliberate. + """ + schema = ( + "file: data.csv\n" + " first_row: {'indented': True}\n" + "columns: a, b" + ) + result = _strip_first_row(schema) + assert "indented" in result + + def test_preserves_line_ordering(self): + schema = ( + "file: a\n" + "first_row: x\n" + "columns: z\n" + "note: extra\n" + ) + lines = _strip_first_row(schema).splitlines() + # Only the first_row line should be gone; relative order preserved. + assert lines == ["file: a", "columns: z", "note: extra"] diff --git a/backend/tests/agents/main_agent/core/test_model_config.py b/backend/tests/agents/main_agent/core/test_model_config.py index 9489f909..5dfeafb2 100644 --- a/backend/tests/agents/main_agent/core/test_model_config.py +++ b/backend/tests/agents/main_agent/core/test_model_config.py @@ -98,9 +98,10 @@ def test_explicit_gemini_overrides_gpt_model_id(self): class TestToBedrockConfig: """Validates: Requirements 1.6, 1.7""" - def test_bedrock_config_with_caching_disabled_due_to_bedrock_limitation(self): - """Req 1.6 — caching_enabled=True but cache_config omitted due to - Bedrock limitation with non-PDF document blocks. See model_config.py TODO.""" + def test_bedrock_config_with_caching_enabled_currently_omits_cache_config(self): + """Req 1.6 — caching_enabled=True but cache_config omitted while + Bedrock prompt caching rollout is deferred. The SDK-side blocker is + resolved in strands 1.39.0; see model_config.py for the deferral note.""" cfg = ModelConfig(caching_enabled=True) result = cfg.to_bedrock_config() @@ -179,6 +180,109 @@ def test_bedrock_config_thinking_disabled_passes_sampling_params_through(self): assert result["temperature"] == 0.5 assert result["top_p"] == 0.8 + @pytest.mark.parametrize( + "model_id", + [ + "us.anthropic.claude-opus-4-7-20260115-v1:0", + "us.anthropic.claude-opus-4-6", + "us.anthropic.claude-sonnet-4-6", + "claude-mythos-preview", + ], + ) + def test_bedrock_thinking_uses_adaptive_shape_on_newer_models(self, model_id): + """Opus 4.6/4.7, Sonnet 4.6 and Mythos require/recommend adaptive + thinking. Opus 4.7 rejects `{type:"enabled"}` with a 400, so the + int budget only signals "on" and the shape is `{type:"adaptive"}`. + `display:"summarized"` keeps the reasoning trace from going blank + (Opus 4.7 defaults display to "omitted").""" + cfg = ModelConfig(model_id=model_id, inference_params={"thinking": 4096}) + result = cfg.to_bedrock_config() + + assert result["additional_request_fields"]["thinking"] == { + "type": "adaptive", + "display": "summarized", + } + assert "budget_tokens" not in result["additional_request_fields"]["thinking"] + + @pytest.mark.parametrize( + "model_id", + [ + "us.anthropic.claude-sonnet-4-5-20250101-v1:0", + "claude-3-opus", + "us.anthropic.claude-haiku-4-5-20251001-v1:0", + ], + ) + def test_bedrock_thinking_keeps_legacy_enabled_shape_on_older_models(self, model_id): + """Older models (Sonnet 4.5, Claude 3, Haiku 4.5) still take the + legacy `{type:"enabled", budget_tokens:N}` shape — unchanged.""" + cfg = ModelConfig(model_id=model_id, inference_params={"thinking": 4096}) + result = cfg.to_bedrock_config() + + assert result["additional_request_fields"]["thinking"] == { + "type": "enabled", + "budget_tokens": 4096, + } + + def test_bedrock_adaptive_thinking_still_suppresses_sampling_params(self): + """Anthropic rejects temperature/top_p/top_k while extended thinking + is on regardless of mode — suppression still fires for adaptive.""" + cfg = ModelConfig( + model_id="us.anthropic.claude-opus-4-7-20260115-v1:0", + inference_params={"thinking": 2048, "temperature": 0.7, "top_p": 0.9}, + ) + result = cfg.to_bedrock_config() + + assert "temperature" not in result + assert "top_p" not in result + assert result["additional_request_fields"]["thinking"]["type"] == "adaptive" + + def test_bedrock_effort_maps_to_output_config(self): + """`effort` rides through additional_request_fields as Anthropic's + top-level `output_config.effort` — not a Converse standard field.""" + cfg = ModelConfig( + model_id="us.anthropic.claude-opus-4-7-20260115-v1:0", + inference_params={"effort": "xhigh"}, + ) + result = cfg.to_bedrock_config() + + assert result["additional_request_fields"]["output_config"]["effort"] == "xhigh" + assert "effort" not in result + assert "output_config" not in result + + def test_bedrock_effort_and_adaptive_thinking_coexist(self): + """effort and adaptive thinking are independent knobs — both land + under additional_request_fields together.""" + cfg = ModelConfig( + model_id="us.anthropic.claude-opus-4-7-20260115-v1:0", + inference_params={"thinking": 2048, "effort": "high"}, + ) + result = cfg.to_bedrock_config() + + arf = result["additional_request_fields"] + assert arf["thinking"] == {"type": "adaptive", "display": "summarized"} + assert arf["output_config"]["effort"] == "high" + + def test_bedrock_config_coerces_float_max_tokens_to_int(self): + """JSON-sourced inference params can carry a float (100000.0); the + Bedrock SDK rejects a float maxTokens, so it must be coerced to int.""" + cfg = ModelConfig(inference_params={"max_tokens": 100000.0, "top_k": 40.0}) + result = cfg.to_bedrock_config() + + assert result["max_tokens"] == 100000 + assert isinstance(result["max_tokens"], int) + assert result["additional_request_fields"]["top_k"] == 40 + assert isinstance(result["additional_request_fields"]["top_k"], int) + + def test_gemini_config_coerces_float_max_tokens_to_int(self): + """Coercion applies across providers — Gemini max_output_tokens too.""" + cfg = ModelConfig( + model_id="gemini-pro", inference_params={"max_tokens": 2048.0} + ) + result = cfg.to_gemini_config() + + assert result["params"]["max_output_tokens"] == 2048 + assert isinstance(result["params"]["max_output_tokens"], int) + def test_bedrock_config_drops_unknown_canonical_param(self): """Provider translation table silently drops keys it doesn't know.""" cfg = ModelConfig(inference_params={"reasoning_effort": "high"}) diff --git a/backend/tests/agents/main_agent/integrations/test_mcp_apps.py b/backend/tests/agents/main_agent/integrations/test_mcp_apps.py new file mode 100644 index 00000000..46063cae --- /dev/null +++ b/backend/tests/agents/main_agent/integrations/test_mcp_apps.py @@ -0,0 +1,500 @@ +"""Tests for MCP Apps host support (PR #2 of the host-renderer initiative). + +Covers the PR #2 acceptance criteria from +`docs/kaizen/scoping/mcp-apps-host-renderer.md`: + + (a) `io.modelcontextprotocol/ui` is advertised on the outbound MCP + `initialize` when the host flag is on, and absent when it is off. + (b) A tool whose `_meta.ui.visibility` excludes `"model"` is filtered out + of the Strands tool list (external client + gateway filtered client). + (c) `_meta.ui.resourceUri` survives the round-trip into our tool catalog. + (d) Ordinary tools and default-visibility (`["model", "app"]`) tools are + unaffected. + +The fake-MCP-server surface is a `super().list_tools_sync()` stub returning +UI-bearing tools, mirroring the mock-the-boundary style already used in +`test_external_mcp_client.py`. +""" + +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import anyio +import mcp.types as mcp_types +import pytest +import strands.tools.mcp.mcp_client as strands_mcp_client_mod +from mcp.shared.session import BaseSession +from strands.types import PaginatedList + +from agents.main_agent.integrations import mcp_apps +from agents.main_agent.integrations.mcp_apps import ( + MCP_APPS_UI_EXTENSION_KEY, + MCP_APPS_UI_MIME_TYPE, + UICapableMCPClient, + _UIExtensionClientSession, + ensure_ui_extension_session_patch, + fetch_ui_resource, + get_ui_tool_catalog, + record_and_filter_ui_tools, +) +from agents.main_agent.integrations.gateway_mcp_client import FilteredMCPClient +from apis.shared.tools.models import DEFAULT_TOOL_VISIBILITY, ToolUIMetadata + +_ENV_FLAG = "AGENTCORE_MCP_APPS_HOST_ENABLED" +_ENV_SANDBOX_ORIGIN = "AGENTCORE_MCP_APPS_SANDBOX_ORIGIN" + + +@pytest.fixture +def mcp_apps_clean(monkeypatch): + """Isolate the global catalog and the strands ClientSession symbol.""" + get_ui_tool_catalog().clear() + original_session = strands_mcp_client_mod.ClientSession + monkeypatch.delenv(_ENV_FLAG, raising=False) + monkeypatch.delenv(_ENV_SANDBOX_ORIGIN, raising=False) + try: + yield + finally: + strands_mcp_client_mod.ClientSession = original_session + get_ui_tool_catalog().clear() + + +def _fake_tool(tool_name, ui=None, mcp_name=None): + """An MCPAgentTool stand-in: it carries the raw mcp tool with `_meta`.""" + meta = {"ui": ui} if ui is not None else None + return SimpleNamespace( + tool_name=tool_name, + mcp_tool=SimpleNamespace(name=mcp_name or tool_name, meta=meta), + ) + + +# ── ToolUIMetadata.from_meta ────────────────────────────────────────────────── + + +class TestToolUIMetadataParsing: + def test_returns_none_for_non_ui_tool(self): + assert ToolUIMetadata.from_meta(None) is None + assert ToolUIMetadata.from_meta({}) is None + assert ToolUIMetadata.from_meta({"other": 1}) is None + + def test_absent_visibility_defaults_to_spec_default(self): + ui = ToolUIMetadata.from_meta({"ui": {"resourceUri": "ui://x/y"}}) + assert ui is not None + assert ui.resource_uri == "ui://x/y" + assert ui.visibility == DEFAULT_TOOL_VISIBILITY + assert ui.visible_to_model() is True + + def test_app_only_visibility_hides_from_model(self): + ui = ToolUIMetadata.from_meta( + {"ui": {"resourceUri": "ui://x/y", "visibility": ["app"]}} + ) + assert ui.visibility == ["app"] + assert ui.visible_to_model() is False + + def test_raw_payload_is_retained_verbatim(self): + raw = { + "resourceUri": "ui://x/y", + "visibility": ["model", "app"], + "csp": {"connectDomains": ["https://example.com"]}, + } + ui = ToolUIMetadata.from_meta({"ui": raw}) + assert ui.raw == raw + + +# ── (a) initialize advertises the UI extension ──────────────────────────────── + + +async def _run_initialize(monkeypatch, *, enabled): + """Drive _UIExtensionClientSession.initialize() with I/O stubbed out and + return the ClientCapabilities that went onto the wire.""" + if enabled: + monkeypatch.setenv(_ENV_FLAG, "true") + else: + monkeypatch.setenv(_ENV_FLAG, "false") + + captured: dict = {} + + async def fake_send_request(request, result_type, *a, **k): + captured["request"] = request + return mcp_types.InitializeResult( + protocolVersion=mcp_types.LATEST_PROTOCOL_VERSION, + capabilities=mcp_types.ServerCapabilities(), + serverInfo=mcp_types.Implementation(name="fake-server", version="1"), + ) + + send_a, recv_a = anyio.create_memory_object_stream(1) + send_b, recv_b = anyio.create_memory_object_stream(1) + session = _UIExtensionClientSession(recv_a, send_b) + + with patch.object( + BaseSession, "send_request", new=AsyncMock(side_effect=fake_send_request) + ), patch.object(BaseSession, "send_notification", new=AsyncMock()): + await session.initialize() + + request = captured["request"] + caps = request.root.params.capabilities + return caps.model_dump(by_alias=True, exclude_none=True) + + +@pytest.mark.asyncio +async def test_initialize_advertises_ui_extension_when_enabled( + mcp_apps_clean, monkeypatch +): + caps = await _run_initialize(monkeypatch, enabled=True) + + assert caps.get("extensions", {}).get(MCP_APPS_UI_EXTENSION_KEY) == { + "mimeTypes": [MCP_APPS_UI_MIME_TYPE] + } + assert MCP_APPS_UI_MIME_TYPE == "text/html;profile=mcp-app" + + +@pytest.mark.asyncio +async def test_initialize_omits_ui_extension_when_disabled( + mcp_apps_clean, monkeypatch +): + caps = await _run_initialize(monkeypatch, enabled=False) + + assert MCP_APPS_UI_EXTENSION_KEY not in caps.get("extensions", {}) + + +# ── ClientSession symbol patch ──────────────────────────────────────────────── + + +class TestSessionPatch: + def test_ensure_patch_substitutes_strands_client_session(self, mcp_apps_clean): + ensure_ui_extension_session_patch() + assert ( + strands_mcp_client_mod.ClientSession is _UIExtensionClientSession + ) + + def test_constructing_ui_capable_client_installs_patch(self, mcp_apps_clean): + strands_mcp_client_mod.ClientSession = ( + mcp_apps._UIExtensionClientSession.__bases__[0] + ) + UICapableMCPClient(lambda: None) + assert ( + strands_mcp_client_mod.ClientSession is _UIExtensionClientSession + ) + + +# ── (b)(c)(d) record + visibility filter ───────────────────────────────────── + + +class TestRecordAndFilter: + def test_passthrough_when_flag_disabled(self, mcp_apps_clean, monkeypatch): + monkeypatch.setenv(_ENV_FLAG, "false") + tools = [ + _fake_tool("app_only", ui={"resourceUri": "ui://a", "visibility": ["app"]}), + _fake_tool("plain"), + ] + + result = record_and_filter_ui_tools(tools) + + # Inert: nothing filtered, nothing recorded. + assert result == tools + assert get_ui_tool_catalog().snapshot() == {} + + def test_filters_app_only_and_records_metadata( + self, mcp_apps_clean, monkeypatch + ): + monkeypatch.setenv(_ENV_FLAG, "true") + tools = [ + _fake_tool( + "app_widget", + ui={"resourceUri": "ui://app/widget", "visibility": ["app"]}, + ), + _fake_tool( + "panel", + ui={"resourceUri": "ui://app/panel"}, # default visibility + ), + _fake_tool( + "dual", + ui={"resourceUri": "ui://app/dual", "visibility": ["model", "app"]}, + ), + _fake_tool("plain"), # ordinary, no _meta.ui + ] + + result = record_and_filter_ui_tools(tools) + + kept = {t.tool_name for t in result} + # (b) app-only hidden from the model; (d) the rest unaffected. + assert kept == {"panel", "dual", "plain"} + + catalog = get_ui_tool_catalog() + # (c) resourceUri survives the round-trip into our tool catalog, + # including for the app-only tool we hide from the model. + assert catalog.get("app_widget").resource_uri == "ui://app/widget" + assert catalog.get("app_widget").visible_to_model() is False + assert catalog.get("panel").resource_uri == "ui://app/panel" + assert catalog.get("panel").visibility == DEFAULT_TOOL_VISIBILITY + assert catalog.get("dual").resource_uri == "ui://app/dual" + # Ordinary tools are never recorded. + assert catalog.get("plain") is None + + +# ── external client: UICapableMCPClient.list_tools_sync ─────────────────────── + + +class TestUICapableMCPClientListTools: + @pytest.mark.asyncio + async def test_list_tools_sync_filters_and_preserves_pagination( + self, mcp_apps_clean, monkeypatch + ): + monkeypatch.setenv(_ENV_FLAG, "true") + client = UICapableMCPClient(lambda: None) + + fake_page = PaginatedList( + [ + _fake_tool( + "app_only", + ui={"resourceUri": "ui://srv/app", "visibility": ["app"]}, + ), + _fake_tool("normal"), + ], + token="next-page", + ) + + with patch.object( + strands_mcp_client_mod.MCPClient, + "list_tools_sync", + return_value=fake_page, + ): + result = client.list_tools_sync() + + assert [t.tool_name for t in result] == ["normal"] + assert result.pagination_token == "next-page" + assert ( + get_ui_tool_catalog().get("app_only").resource_uri == "ui://srv/app" + ) + + +# ── gateway client: FilteredMCPClient applies the same filter ───────────────── + + +class TestFilteredGatewayClientUIFilter: + @pytest.mark.asyncio + async def test_gateway_filtered_client_hides_app_only_tool( + self, mcp_apps_clean, monkeypatch + ): + monkeypatch.setenv(_ENV_FLAG, "true") + client = FilteredMCPClient( + lambda: None, + enabled_tool_ids=["app_only", "normal"], + ) + + fake_page = PaginatedList( + [ + _fake_tool( + "app_only", + ui={"resourceUri": "ui://gw/app", "visibility": ["app"]}, + ), + _fake_tool("normal"), + ], + token=None, + ) + + # Patch the grandparent MCPClient.list_tools_sync so FilteredMCPClient's + # own override runs (enabled-id filter -> UI visibility filter). + with patch.object( + strands_mcp_client_mod.MCPClient, + "list_tools_sync", + return_value=fake_page, + ): + result = client.list_tools_sync() + + assert [t.tool_name for t in result] == ["normal"] + assert get_ui_tool_catalog().get("app_only").resource_uri == "ui://gw/app" + + +# ── PR #3: resources/read fetch path + ui_resource payload ─────────────────── + + +class _FakeMCPClient: + """Stand-in for a Strands MCPClient at the `resources/read` boundary. + + Mirrors the mock-the-boundary style in `test_external_mcp_client.py`: + the unit under test never starts a real session — it only calls + `read_resource_sync`, which we record and stub. + """ + + def __init__(self, result=None, raises: Exception | None = None) -> None: + self._result = result + self._raises = raises + self.read_calls: list = [] + + def read_resource_sync(self, uri): + self.read_calls.append(uri) + if self._raises is not None: + raise self._raises + return self._result + + +def _html_resource( + *, text="

widget

", mime=MCP_APPS_UI_MIME_TYPE, ui_meta=None +): + """A real `mcp.types.ReadResourceResult` — proves our extraction works + against the actual MCP SDK shape, not just a duck-typed fake.""" + kwargs = {"uri": "ui://srv/widget", "mimeType": mime, "text": text} + if ui_meta is not None: + kwargs["_meta"] = {MCP_APPS_UI_EXTENSION_KEY: ui_meta} + return mcp_types.ReadResourceResult( + contents=[mcp_types.TextResourceContents(**kwargs)] + ) + + +def _seed_catalog(monkeypatch, *, ui, client): + """Record a UI tool + its hosting client exactly the way a live + `list_tools_sync` would (so the client-passing path is exercised too).""" + monkeypatch.setenv(_ENV_FLAG, "true") + record_and_filter_ui_tools([_fake_tool("widget", ui=ui)], client=client) + + +class TestFetchUIResource: + def test_fetches_via_resources_read_and_inlines_html( + self, mcp_apps_clean, monkeypatch + ): + client = _FakeMCPClient( + result=_html_resource( + ui_meta={ + "csp": {"connectDomains": ["https://api.test"]}, + # SEP-1865: permissions is an OBJECT, not a list. + "permissions": {"clipboardWrite": {}}, + } + ) + ) + _seed_catalog( + monkeypatch, + ui={"resourceUri": "ui://srv/widget"}, + client=client, + ) + + payload = fetch_ui_resource("widget", "tu-1") + + # Spec MUST: the resource is fetched via resources/read against the + # hosting client, addressed by the `ui://` URI — never inlined by us. + assert client.read_calls == ["ui://srv/widget"] + assert payload == { + "type": "ui_resource", + "toolUseId": "tu-1", + "resourceUri": "ui://srv/widget", + "html": "

widget

", + "mimeType": MCP_APPS_UI_MIME_TYPE, + "csp": {"connectDomains": ["https://api.test"]}, + "permissions": {"clipboardWrite": {}}, + # Empty when the mcp-sandbox stack origin isn't wired into env. + "sandboxOrigin": "", + } + + def test_carries_sandbox_origin_from_env( + self, mcp_apps_clean, monkeypatch + ): + client = _FakeMCPClient(result=_html_resource()) + _seed_catalog( + monkeypatch, ui={"resourceUri": "ui://srv/widget"}, client=client + ) + monkeypatch.setenv( + _ENV_SANDBOX_ORIGIN, "https://mcp-sandbox.example.com" + ) + + payload = fetch_ui_resource("widget", "tu-1") + assert payload["sandboxOrigin"] == "https://mcp-sandbox.example.com" + + def test_inert_when_flag_disabled(self, mcp_apps_clean, monkeypatch): + client = _FakeMCPClient(result=_html_resource()) + _seed_catalog( + monkeypatch, ui={"resourceUri": "ui://srv/widget"}, client=client + ) + # Flag flipped off *after* catalog seeding: the fetch path itself + # must stay inert regardless of catalog contents. + monkeypatch.setenv(_ENV_FLAG, "false") + + assert fetch_ui_resource("widget", "tu-1") is None + assert client.read_calls == [] + + def test_none_for_unknown_or_non_ui_tool(self, mcp_apps_clean, monkeypatch): + monkeypatch.setenv(_ENV_FLAG, "true") + assert fetch_ui_resource("never-seen", "tu-1") is None + + def test_none_when_no_hosting_client_recorded( + self, mcp_apps_clean, monkeypatch + ): + # Metadata recorded without a client (e.g. PR #2's catalog-only + # path) → we cannot issue resources/read, so no event. + _seed_catalog( + monkeypatch, ui={"resourceUri": "ui://srv/widget"}, client=None + ) + assert fetch_ui_resource("widget", "tu-1") is None + + def test_resources_read_failure_is_swallowed( + self, mcp_apps_clean, monkeypatch + ): + client = _FakeMCPClient(raises=RuntimeError("session not running")) + _seed_catalog( + monkeypatch, ui={"resourceUri": "ui://srv/widget"}, client=client + ) + assert fetch_ui_resource("widget", "tu-1") is None + assert client.read_calls == ["ui://srv/widget"] + + def test_none_when_resource_has_no_inline_html( + self, mcp_apps_clean, monkeypatch + ): + blob = mcp_types.ReadResourceResult( + contents=[ + mcp_types.BlobResourceContents( + uri="ui://srv/widget", + mimeType="application/octet-stream", + blob="AAAA", + ) + ] + ) + client = _FakeMCPClient(result=blob) + _seed_catalog( + monkeypatch, ui={"resourceUri": "ui://srv/widget"}, client=client + ) + assert fetch_ui_resource("widget", "tu-1") is None + + def test_csp_permissions_fall_back_to_tool_meta( + self, mcp_apps_clean, monkeypatch + ): + # Resource carries no `_meta.ui`; the tool's `tools/list` `_meta.ui` + # (retained verbatim by PR #2 in ToolUIMetadata.raw) supplies them. + client = _FakeMCPClient(result=_html_resource(ui_meta=None)) + _seed_catalog( + monkeypatch, + ui={ + "resourceUri": "ui://srv/widget", + "csp": {"frameDomains": ["https://embed.test"]}, + "permissions": {"geolocation": {}}, + }, + client=client, + ) + + payload = fetch_ui_resource("widget", "tu-9") + assert payload is not None + assert payload["csp"] == {"frameDomains": ["https://embed.test"]} + assert payload["permissions"] == {"geolocation": {}} + + def test_prefers_mcp_app_mime_when_multiple_text_contents( + self, mcp_apps_clean, monkeypatch + ): + result = mcp_types.ReadResourceResult( + contents=[ + mcp_types.TextResourceContents( + uri="ui://srv/widget", + mimeType="text/plain", + text="ignored", + ), + mcp_types.TextResourceContents( + uri="ui://srv/widget", + mimeType=MCP_APPS_UI_MIME_TYPE, + text="
chosen
", + ), + ] + ) + client = _FakeMCPClient(result=result) + _seed_catalog( + monkeypatch, ui={"resourceUri": "ui://srv/widget"}, client=client + ) + + payload = fetch_ui_resource("widget", "tu-2") + assert payload["html"] == "
chosen
" + assert payload["mimeType"] == MCP_APPS_UI_MIME_TYPE diff --git a/backend/tests/agents/main_agent/streaming/test_artifact_events.py b/backend/tests/agents/main_agent/streaming/test_artifact_events.py new file mode 100644 index 00000000..c55f7554 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_artifact_events.py @@ -0,0 +1,198 @@ +"""Tests for StreamCoordinator._extract_artifact_events. + +Covers the post-turn `artifact` SSE emit: turn-window filtering (only +artifacts touched this turn), action derivation, fail-closed behavior, +and the no-session guard. +""" + +from __future__ import annotations + +import json +from datetime import datetime, timezone + +import pytest + +from agents.builtin_tools.artifacts import service as artifact_service +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + +SESSION = "sess-9" +USER = "user-123" + + +def _parse_sse(raw: str) -> dict: + assert raw.startswith("event: artifact\ndata: ") + assert raw.endswith("\n\n") + return json.loads(raw[len("event: artifact\ndata: ") :].strip()) + + +@pytest.fixture +def turn_start() -> datetime: + return datetime(2026, 5, 15, 12, 0, 0, tzinfo=timezone.utc) + + +@pytest.fixture +def coord() -> StreamCoordinator: + return StreamCoordinator() + + +def _row(**kw) -> dict: + base = { + "artifact_id": "art-1", + "version": 1, + "title": "Doc", + "content_type": "text/html; charset=utf-8", + "updated_at": "2026-05-15T12:00:05+00:00", + "created_at": "2026-05-15T12:00:05+00:00", + } + base.update(kw) + return base + + +@pytest.mark.asyncio +async def test_emits_created_for_v1(coord, turn_start, monkeypatch) -> None: + monkeypatch.setattr( + artifact_service, "list_session_artifacts", lambda u, s: [_row()] + ) + out = await coord._extract_artifact_events(SESSION, USER, turn_start) + assert len(out) == 1 + payload = _parse_sse(out[0]) + assert payload == { + "type": "artifact", + "artifactId": "art-1", + "version": 1, + "title": "Doc", + "contentType": "text/html; charset=utf-8", + "sessionId": SESSION, + "updatedAt": "2026-05-15T12:00:05+00:00", + "action": "created", + "producedByMessageIndex": None, + } + + +@pytest.mark.asyncio +async def test_stamps_and_emits_produced_by_message_index( + coord, turn_start, monkeypatch +) -> None: + monkeypatch.setattr( + artifact_service, + "list_session_artifacts", + lambda u, s: [_row(artifact_id="a"), _row(artifact_id="b")], + ) + stamped: list[tuple] = [] + monkeypatch.setattr( + artifact_service, + "set_produced_by_message_index", + lambda u, aid, ver, idx: stamped.append((u, aid, ver, idx)), + ) + out = await coord._extract_artifact_events( + SESSION, USER, turn_start, produced_by_message_index=7 + ) + assert {_parse_sse(e)["producedByMessageIndex"] for e in out} == {7} + # Each artifact's own version is threaded to the stamp so the right + # version row is linked (both rows are v1 here). + assert stamped == [(USER, "a", 1, 7), (USER, "b", 1, 7)] + + +@pytest.mark.asyncio +async def test_stamp_failure_is_swallowed( + coord, turn_start, monkeypatch +) -> None: + monkeypatch.setattr( + artifact_service, "list_session_artifacts", lambda u, s: [_row()] + ) + + def _boom(u, aid, ver, idx): + raise RuntimeError("ddb down") + + monkeypatch.setattr( + artifact_service, "set_produced_by_message_index", _boom + ) + out = await coord._extract_artifact_events( + SESSION, USER, turn_start, produced_by_message_index=3 + ) + # Stamp failure must not drop the live event. + assert _parse_sse(out[0])["producedByMessageIndex"] == 3 + + +@pytest.mark.asyncio +async def test_version_gt_1_is_updated(coord, turn_start, monkeypatch) -> None: + monkeypatch.setattr( + artifact_service, + "list_session_artifacts", + lambda u, s: [_row(version=4)], + ) + out = await coord._extract_artifact_events(SESSION, USER, turn_start) + assert _parse_sse(out[0])["action"] == "updated" + assert _parse_sse(out[0])["version"] == 4 + + +@pytest.mark.asyncio +async def test_filters_artifacts_from_earlier_turns( + coord, turn_start, monkeypatch +) -> None: + stale = _row(artifact_id="old", updated_at="2026-05-15T11:59:59+00:00") + fresh = _row(artifact_id="new", updated_at="2026-05-15T12:00:30+00:00") + monkeypatch.setattr( + artifact_service, + "list_session_artifacts", + lambda u, s: [fresh, stale], + ) + out = await coord._extract_artifact_events(SESSION, USER, turn_start) + ids = [_parse_sse(e)["artifactId"] for e in out] + assert ids == ["new"] + + +@pytest.mark.asyncio +async def test_unparseable_updated_at_is_included( + coord, turn_start, monkeypatch +) -> None: + monkeypatch.setattr( + artifact_service, + "list_session_artifacts", + lambda u, s: [_row(updated_at="")], + ) + out = await coord._extract_artifact_events(SESSION, USER, turn_start) + assert len(out) == 1 + + +@pytest.mark.asyncio +async def test_config_error_is_swallowed(coord, turn_start, monkeypatch) -> None: + def _raise(u, s): + raise artifact_service.ArtifactConfigError("not configured") + + monkeypatch.setattr(artifact_service, "list_session_artifacts", _raise) + assert await coord._extract_artifact_events(SESSION, USER, turn_start) == [] + + +@pytest.mark.asyncio +async def test_unexpected_error_is_swallowed( + coord, turn_start, monkeypatch +) -> None: + def _raise(u, s): + raise RuntimeError("ddb down") + + monkeypatch.setattr(artifact_service, "list_session_artifacts", _raise) + assert await coord._extract_artifact_events(SESSION, USER, turn_start) == [] + + +@pytest.mark.asyncio +async def test_no_session_or_user_is_noop(coord, turn_start) -> None: + assert await coord._extract_artifact_events(None, USER, turn_start) == [] + assert await coord._extract_artifact_events(SESSION, None, turn_start) == [] + + +@pytest.mark.asyncio +async def test_multiple_artifacts_one_turn(coord, turn_start, monkeypatch) -> None: + monkeypatch.setattr( + artifact_service, + "list_session_artifacts", + lambda u, s: [ + _row(artifact_id="a", version=1), + _row(artifact_id="b", version=2), + ], + ) + out = await coord._extract_artifact_events(SESSION, USER, turn_start) + actions = { + _parse_sse(e)["artifactId"]: _parse_sse(e)["action"] for e in out + } + assert actions == {"a": "created", "b": "updated"} diff --git a/backend/tests/agents/main_agent/streaming/test_compaction_sse_emit_once.py b/backend/tests/agents/main_agent/streaming/test_compaction_sse_emit_once.py new file mode 100644 index 00000000..32223b61 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_compaction_sse_emit_once.py @@ -0,0 +1,215 @@ +"""Regression: the `compaction` SSE event emits exactly once per compaction event. + +The `compaction` SSE event (frontend inline "earlier messages summarized" +divider, landed in PR #243) is emitted by ``StreamCoordinator.stream_response`` +from inside the single terminal ``done`` handler, gated solely on +``TurnBasedSessionManager.update_after_turn`` returning a ``CompactionResult``. + +``process_agent_stream`` yields exactly one ``done`` event per turn (STEP 9, +after the raw agent stream is exhausted), so ``update_after_turn`` is awaited +exactly once and the SSE frame is yielded at most once. + +This module locks that once-per-turn invariant against the *real* pipeline +(``stream_response`` → ``process_agent_stream`` → coordinator emit code), +stubbing only two narrow seams: ``agent.stream_async`` (raw Strands events) +and ``session_manager.update_after_turn`` (the compaction decision). + +It is also the explicit non-regression guard for the Strands 1.40 bump. +Strands 1.40 ships proactive context compression (strands PR #2239) and feeds +``EventLoopMetrics.accumulated_usage`` on the ``AgentResult`` event — which +``_handle_metadata_events`` surfaces on the ``metadata_summary`` track. Neither +is a second emit path: proactive compression is opt-in via a +``ConversationManager``'s ``proactive_compression`` (default ``None`` → the +``BeforeModelCallEvent`` handler early-returns), and our compaction lives in a +``SessionManager`` (``TurnBasedSessionManager.update_after_turn``), a different +abstraction. The third test drives the ``metadata_summary``/accumulated-usage +surface explicitly and asserts the emit count stays at one. +""" + +from typing import Any, AsyncIterator, Dict, List, Optional + +import pytest + +from agents.main_agent.session.compaction_models import CompactionResult +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + + +# Per-call metadata raw event: Bedrock's `metadata` chunk wrapped inside +# Strands' ModelStreamChunkEvent. Same shape as the cost-attribution suite. +def _raw_metadata_event(usage: Dict[str, int]) -> Dict[str, Any]: + return {"event": {"metadata": {"usage": usage, "metrics": {"latencyMs": 100}}}} + + +# Strands AgentResult event. EventLoopMetrics.accumulated_usage is summed +# across all LLM calls in the turn; _handle_metadata_events extracts it onto +# the `metadata_summary` track. This is the surface Strands 1.40's proactive +# compression also reads from — included here to prove it is not a second +# compaction-emit path. +class _FakeEventLoopMetrics: + def __init__(self, accumulated_usage: Dict[str, int]) -> None: + self.accumulated_usage = accumulated_usage + self.accumulated_metrics = {"latencyMs": 250} + + +class _FakeAgentResult: + def __init__(self, accumulated_usage: Dict[str, int]) -> None: + self.metrics = _FakeEventLoopMetrics(accumulated_usage) + + +def _raw_agent_result_event(accumulated_usage: Dict[str, int]) -> Dict[str, Any]: + return {"result": _FakeAgentResult(accumulated_usage)} + + +class _FakeAgent: + """Minimal agent: a message list and a controllable raw event stream. + + No ``_interrupt_state`` so the coordinator's paused-turn snapshot and + OAuth / tool-approval extractors all early-return on the ``done`` event. + """ + + def __init__(self, raw_events: List[Dict[str, Any]]) -> None: + self.messages = [{"role": "user", "content": [{"text": "hi"}]}] + self._raw_events = raw_events + + def stream_async(self, prompt: Any) -> AsyncIterator[Dict[str, Any]]: + async def _gen() -> AsyncIterator[Dict[str, Any]]: + for ev in self._raw_events: + yield ev + + return _gen() + + +class _RecordingSessionManager: + """Stub session manager that records ``update_after_turn`` invocations. + + Only the seam the coordinator depends on is implemented; the real + threshold/checkpoint math is covered by the TurnBasedSessionManager + suite. This isolates the coordinator-level once-per-turn invariant. + """ + + def __init__(self, result: Optional[CompactionResult]) -> None: + self._result = result + self.calls: List[int] = [] + + async def update_after_turn( + self, + input_tokens: int, + current_messages: Optional[List[Dict]] = None, + ) -> Optional[CompactionResult]: + self.calls.append(input_tokens) + return self._result + + +async def _collect_sse( + agent: _FakeAgent, session_manager: _RecordingSessionManager +) -> List[str]: + coordinator = StreamCoordinator() + frames: List[str] = [] + async for sse in coordinator.stream_response( + agent=agent, + prompt="hi", + session_manager=session_manager, + session_id="sess-1", + user_id="user-1", + main_agent_wrapper=None, + ): + frames.append(sse) + return frames + + +def _compaction_frames(frames: List[str]) -> List[str]: + return [f for f in frames if f.startswith("event: compaction\n")] + + +# A turn whose summed input buckets exceed any threshold — guarantees the +# coordinator's `total_input_tokens > 0` guard passes so update_after_turn +# is consulted. +_TURN_USAGE = {"inputTokens": 150_000, "outputTokens": 80, "totalTokens": 150_080} + + +@pytest.mark.asyncio +async def test_compaction_sse_emitted_exactly_once_when_checkpoint_advances(): + """Checkpoint advances → exactly one `event: compaction` frame.""" + result = CompactionResult( + previous_checkpoint=0, + new_checkpoint=4, + summarized_turns=2, + input_tokens=150_000, + ) + agent = _FakeAgent([_raw_metadata_event(_TURN_USAGE)]) + sm = _RecordingSessionManager(result) + + frames = await _collect_sse(agent, sm) + compaction = _compaction_frames(frames) + + # update_after_turn consulted exactly once (one terminal `done`). + assert sm.calls == [150_000] + assert len(compaction) == 1, ( + f"expected exactly one compaction SSE frame, got {len(compaction)}: " + f"{compaction}" + ) + + import json + + payload = json.loads(compaction[0][len("event: compaction\ndata: ") :].strip()) + assert payload == { + "type": "compaction", + "previousCheckpoint": 0, + "newCheckpoint": 4, + "summarizedTurns": 2, + "inputTokens": 150_000, + } + + +@pytest.mark.asyncio +async def test_no_compaction_sse_when_checkpoint_does_not_advance(): + """update_after_turn returns None → zero compaction frames, still one call.""" + agent = _FakeAgent([_raw_metadata_event(_TURN_USAGE)]) + sm = _RecordingSessionManager(None) + + frames = await _collect_sse(agent, sm) + + assert sm.calls == [150_000] + assert _compaction_frames(frames) == [] + + +@pytest.mark.asyncio +async def test_strands_result_metadata_track_does_not_double_fire(): + """Strands 1.40 non-regression guard. + + Interleave per-call `metadata` events with a Strands ``AgentResult`` + (the ``EventLoopMetrics.accumulated_usage`` / ``metadata_summary`` + surface that 1.40's proactive compression also reads). There is still + exactly one terminal ``done`` → update_after_turn is consulted exactly + once → exactly one compaction frame. The accumulated-usage track is not + a second emit path. + """ + call_0 = {"inputTokens": 80_000, "outputTokens": 40, "totalTokens": 80_040} + call_1 = {"inputTokens": 150_000, "outputTokens": 60, "totalTokens": 150_060} + turn_cumulative = { + "inputTokens": 230_000, # Strands sums across calls — must not re-trigger + "outputTokens": 100, + "totalTokens": 230_100, + } + agent = _FakeAgent( + [ + _raw_metadata_event(call_0), + _raw_metadata_event(call_1), + _raw_agent_result_event(turn_cumulative), + ] + ) + result = CompactionResult( + previous_checkpoint=4, + new_checkpoint=8, + summarized_turns=2, + input_tokens=150_000, + ) + sm = _RecordingSessionManager(result) + + frames = await _collect_sse(agent, sm) + + # Consulted exactly once. The compaction trigger reads "current context" + # (last per-call usage via last-write-wins), NOT Strands' summed + # accumulated_usage — so the input is call_1's 150_000, not 230_000. + assert sm.calls == [150_000] + assert len(_compaction_frames(frames)) == 1 diff --git a/backend/tests/agents/main_agent/streaming/test_per_message_cost_attribution.py b/backend/tests/agents/main_agent/streaming/test_per_message_cost_attribution.py new file mode 100644 index 00000000..0c24bcc8 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_per_message_cost_attribution.py @@ -0,0 +1,310 @@ +"""Regression test for per-message cost attribution on multi-LLM-call turns. + +Strands emits two sources of usage during a tool-use turn: + 1. Per-LLM-call metadata via ``ModelStreamChunkEvent`` (one per assistant + message), carrying just that call's tokens. + 2. A final ``AgentResultEvent`` whose ``AgentResult.metrics`` is an + ``EventLoopMetrics`` with ``accumulated_usage`` summed across every call + in the turn. + +``stream_processor._handle_metadata_events`` extracts both. The stream +coordinator routes any ``metadata`` event into +``per_message_metadata[current_assistant_message_index]["usage"].update(...)``. +Because the AgentResult event arrives *after* every ``message_stop`` (so the +index still points at the last assistant message), a naive ``.update()`` on +the same key overwrites the last message's per-call usage with the +turn-cumulative usage. Pricing each per-message entry and summing then +double-counts every earlier message's input tokens. + +This module locks the contract: + - The per-call metadata events stay typed ``metadata`` (per-message track). + - The result-extracted cumulative metadata is typed ``metadata_summary`` + (turn-summary track), so it never lands in per_message_metadata. + +If the contract regresses, simulating the dispatch loop will reproduce the +double-count and these assertions will fail. +""" + +from typing import Any, Dict, List + +from agents.main_agent.streaming.stream_processor import _handle_metadata_events + + +# Realistic per-call metadata chunk shape: Bedrock's `metadata` chunk wrapped +# inside Strands' ModelStreamChunkEvent (`{"event": chunk}`). +def _per_call_metadata_event(usage: Dict[str, int]) -> Dict[str, Any]: + return {"event": {"metadata": {"usage": usage, "metrics": {"latencyMs": 100}}}} + + +# Realistic AgentResultEvent shape. EventLoopMetrics has accumulated_usage +# summed across all calls; _handle_metadata_events extracts it via __dict__. +class _FakeEventLoopMetrics: + def __init__(self, accumulated_usage: Dict[str, int]) -> None: + self.accumulated_usage = accumulated_usage + self.accumulated_metrics = {"latencyMs": 250} + + +class _FakeAgentResult: + def __init__(self, accumulated_usage: Dict[str, int]) -> None: + self.metrics = _FakeEventLoopMetrics(accumulated_usage) + + +def _agent_result_event(accumulated_usage: Dict[str, int]) -> Dict[str, Any]: + return {"result": _FakeAgentResult(accumulated_usage)} + + +def _dispatch_to_per_message( + processed_events: List[Dict[str, Any]], + per_message_metadata: List[Dict[str, Any]], + current_index: int, +) -> None: + """Mimic stream_coordinator's per-message routing for a single source event. + + Only ``metadata`` events flow into ``per_message_metadata`` — the + ``metadata_summary`` track is for the turn-level accumulator and is + intentionally not routed here. + """ + for processed in processed_events: + if processed.get("type") != "metadata": + continue + usage = processed.get("data", {}).get("usage") + if not usage: + continue + per_message_metadata[current_index]["usage"].update(usage) + + +class TestPerMessageAttributionTwoCallTurn: + """Reproduce the dispatch sequence of a 2-call tool-use turn.""" + + CALL_0_USAGE = {"inputTokens": 1000, "outputTokens": 50, "totalTokens": 1050} + CALL_1_USAGE = {"inputTokens": 1300, "outputTokens": 80, "totalTokens": 1380} + TURN_CUMULATIVE = { + "inputTokens": CALL_0_USAGE["inputTokens"] + CALL_1_USAGE["inputTokens"], + "outputTokens": CALL_0_USAGE["outputTokens"] + CALL_1_USAGE["outputTokens"], + "totalTokens": CALL_0_USAGE["totalTokens"] + CALL_1_USAGE["totalTokens"], + } + + def test_per_call_metadata_routes_to_per_message_track(self): + """Each per-call metadata event carries one message's tokens, no more.""" + events = _handle_metadata_events(_per_call_metadata_event(self.CALL_0_USAGE)) + metadata_events = [e for e in events if e["type"] == "metadata"] + assert len(metadata_events) == 1 + assert metadata_events[0]["data"]["usage"] == self.CALL_0_USAGE + + def test_result_cumulative_does_not_route_to_per_message_track(self): + """The AgentResult cumulative must not be a `metadata` event. + + If it is, the dispatch loop overwrites the last per-message entry + with cumulative usage, double-counting earlier messages' input + tokens at pricing time. + """ + events = _handle_metadata_events(_agent_result_event(self.TURN_CUMULATIVE)) + per_message_typed = [e for e in events if e["type"] == "metadata"] + assert per_message_typed == [], ( + "AgentResult cumulative usage was emitted as a `metadata` event; " + "it would clobber the last per-message entry. Expected " + "`metadata_summary` so it stays on the turn-summary track only." + ) + + def test_result_cumulative_emitted_on_summary_track(self): + """Result-extracted cumulative is still emitted — just on metadata_summary.""" + events = _handle_metadata_events(_agent_result_event(self.TURN_CUMULATIVE)) + summary_events = [e for e in events if e["type"] == "metadata_summary"] + assert len(summary_events) == 1 + assert summary_events[0]["data"]["usage"] == self.TURN_CUMULATIVE + + def test_full_turn_dispatch_preserves_per_call_attribution(self): + """Drive the full event sequence and assert no double-counting.""" + per_message_metadata = [ + {"usage": {}, "metrics": {}}, + {"usage": {}, "metrics": {}}, + ] + + # Message 0's per-call metadata fires while index = 0. + _dispatch_to_per_message( + _handle_metadata_events(_per_call_metadata_event(self.CALL_0_USAGE)), + per_message_metadata, + current_index=0, + ) + # Message 1's per-call metadata fires while index = 1. + _dispatch_to_per_message( + _handle_metadata_events(_per_call_metadata_event(self.CALL_1_USAGE)), + per_message_metadata, + current_index=1, + ) + # AgentResult cumulative fires last, with index still at 1. If this + # leaks onto the `metadata` track, msg 1's usage gets clobbered with + # the turn cumulative — input tokens for msg 0 would be summed twice + # when pricing each entry independently. + _dispatch_to_per_message( + _handle_metadata_events(_agent_result_event(self.TURN_CUMULATIVE)), + per_message_metadata, + current_index=1, + ) + + assert per_message_metadata[0]["usage"] == self.CALL_0_USAGE + assert per_message_metadata[1]["usage"] == self.CALL_1_USAGE + + # Pricing each entry independently must equal the cumulative input, + # not 2× msg 0's input + msg 1's input. + summed_input = ( + per_message_metadata[0]["usage"]["inputTokens"] + + per_message_metadata[1]["usage"]["inputTokens"] + ) + assert summed_input == self.TURN_CUMULATIVE["inputTokens"] + + +class TestSummaryAccumulatorAcceptsBothTracks: + """The stream_processor main loop must keep `accumulated_metadata` cumulative. + + Per-call events accumulate via ``.update()`` (last-write-wins), so before + the cumulative arrives the accumulator only holds the last call's usage — + which is *not* cumulative. The accumulator must therefore consume both + `metadata` and `metadata_summary` events for the final summary emission + to carry true turn totals. + """ + + def test_accumulator_processes_both_tracks(self): + """Walk the same sequence the main loop does and check the final state.""" + accumulated: Dict[str, Any] = {"usage": {}, "metrics": {}} + + sequence = [ + _per_call_metadata_event(TestPerMessageAttributionTwoCallTurn.CALL_0_USAGE), + _per_call_metadata_event(TestPerMessageAttributionTwoCallTurn.CALL_1_USAGE), + _agent_result_event(TestPerMessageAttributionTwoCallTurn.TURN_CUMULATIVE), + ] + + for raw in sequence: + for processed in _handle_metadata_events(raw): + if processed.get("type") in ("metadata", "metadata_summary"): + data = processed.get("data", {}) + if "usage" in data: + accumulated["usage"].update(data["usage"]) + if "metrics" in data: + accumulated["metrics"].update(data["metrics"]) + + assert accumulated["usage"] == TestPerMessageAttributionTwoCallTurn.TURN_CUMULATIVE + + +class TestStreamCoordinatorContextOccupancy: + """The final SSE `usage` field must reflect current context, not sums. + + Bedrock reports each LLM call's `inputTokens` as the FULL context size + sent on that call. For a 2-call tool turn: + call_1.input = 1000 (system + user_msg) + call_2.input = 2500 (system + user_msg + tool_use + tool_result) + + Strands' EventLoopMetrics.accumulated_usage sums these into 3500 — but + the actual context occupancy is 2500, the size of the most recent call. + The frontend uses the SSE metadata `usage` to drive the context-% + badge, and the backend uses it to decide whether to trigger + compaction; both need "current context size", not the cross-call sum. + + This locks in the contract that stream_coordinator's accumulated_metadata + (which feeds the final SSE metadata) takes per-call values via + last-write-wins from `metadata` events and IGNORES the cross-call + cumulative carried on `metadata_summary`. + """ + + CALL_0_USAGE = {"inputTokens": 1000, "outputTokens": 50, "totalTokens": 1050} + CALL_1_USAGE = {"inputTokens": 2500, "outputTokens": 100, "totalTokens": 2600} + TURN_CUMULATIVE = { + "inputTokens": 3500, # 1000 + 2500 — Strands' accumulated_usage + "outputTokens": 150, + "totalTokens": 3650, + } + + def _simulate_stream_coordinator_accumulator( + self, events: List[Dict[str, Any]] + ) -> Dict[str, Any]: + """Mirror stream_coordinator's accumulator branches for a sequence of + already-processed events. Returns the resulting accumulated_metadata. + + - `metadata` events → update accumulated_metadata.usage/metrics. + - `metadata_summary` events → first_token_time only; usage/metrics ignored. + """ + accumulated: Dict[str, Any] = {"usage": {}, "metrics": {}} + for processed in events: + event_type = processed.get("type") + event_data = processed.get("data", {}) + if event_type == "metadata": + if "usage" in event_data: + accumulated["usage"].update(event_data["usage"]) + if "metrics" in event_data: + accumulated["metrics"].update(event_data["metrics"]) + # metadata_summary intentionally does NOT touch usage/metrics here + return accumulated + + def test_final_usage_reflects_last_call_not_sum(self): + """End of a 2-call tool turn — usage should be call_2's, not the sum.""" + # Drive the realistic event order through _handle_metadata_events + # exactly as stream_processor would, then through the coordinator's + # accumulator branches. + raw_events = [ + _per_call_metadata_event(self.CALL_0_USAGE), + _per_call_metadata_event(self.CALL_1_USAGE), + _agent_result_event(self.TURN_CUMULATIVE), + ] + processed: List[Dict[str, Any]] = [] + for raw in raw_events: + processed.extend(_handle_metadata_events(raw)) + + result = self._simulate_stream_coordinator_accumulator(processed) + + assert result["usage"] == self.CALL_1_USAGE, ( + "Final accumulated usage must equal the last per-call's full input " + "(current context size), not Strands' summed-across-calls value. " + "If this regresses, the context-% badge and compaction trigger " + "will inflate by ~the size of every prior call in the turn." + ) + + def test_compaction_input_tokens_match_current_context(self): + """The trigger threshold computation in stream_coordinator uses + `usage.inputTokens + cacheReadInputTokens + cacheWriteInputTokens`.""" + call_with_cache = { + "inputTokens": 200, + "outputTokens": 80, + "totalTokens": 280, + "cacheReadInputTokens": 2000, + "cacheWriteInputTokens": 300, + } + prior_call = { + "inputTokens": 100, + "outputTokens": 40, + "totalTokens": 140, + "cacheReadInputTokens": 0, + "cacheWriteInputTokens": 800, + } + cumulative_after_two_calls = { + "inputTokens": 300, # would be summed by Strands + "outputTokens": 120, + "totalTokens": 420, + "cacheReadInputTokens": 2000, + "cacheWriteInputTokens": 1100, # would be summed by Strands + } + + raw_events = [ + _per_call_metadata_event(prior_call), + _per_call_metadata_event(call_with_cache), + _agent_result_event(cumulative_after_two_calls), + ] + processed: List[Dict[str, Any]] = [] + for raw in raw_events: + processed.extend(_handle_metadata_events(raw)) + + result = self._simulate_stream_coordinator_accumulator(processed) + usage = result["usage"] + + # Compaction sums all three input buckets — must equal call_with_cache's + # totals (current context), not the summed-across-calls totals. + compaction_input = ( + usage.get("inputTokens", 0) + + usage.get("cacheReadInputTokens", 0) + + usage.get("cacheWriteInputTokens", 0) + ) + expected_current_context = ( + call_with_cache["inputTokens"] + + call_with_cache["cacheReadInputTokens"] + + call_with_cache["cacheWriteInputTokens"] + ) + assert compaction_input == expected_current_context diff --git a/backend/tests/agents/main_agent/streaming/test_stream_processor.py b/backend/tests/agents/main_agent/streaming/test_stream_processor.py index 04a99318..2848c0e1 100644 --- a/backend/tests/agents/main_agent/streaming/test_stream_processor.py +++ b/backend/tests/agents/main_agent/streaming/test_stream_processor.py @@ -19,6 +19,7 @@ _handle_reasoning_events, _handle_tool_events, _serialize_object, + process_agent_stream, ) @@ -608,7 +609,13 @@ def test_empty_event_returns_empty(self): assert _handle_metadata_events({}) == [] def test_result_with_accumulated_usage(self): - """result.metrics.accumulated_usage produces a metadata event.""" + """result.metrics.accumulated_usage rides the metadata_summary track. + + It must NOT be emitted as a `metadata` event — those land in + per_message_metadata in the stream coordinator and would clobber + the last assistant message's per-call usage with a turn-cumulative + value, double-counting earlier messages at pricing time. + """ raw = { "result": { "metrics": { @@ -621,9 +628,11 @@ def test_result_with_accumulated_usage(self): } } events = _handle_metadata_events(raw) - m = [e for e in events if e["type"] == "metadata"] - assert len(m) >= 1 - assert m[0]["data"]["usage"]["inputTokens"] == 500 + per_message_typed = [e for e in events if e["type"] == "metadata"] + summary_typed = [e for e in events if e["type"] == "metadata_summary"] + assert per_message_typed == [] + assert len(summary_typed) == 1 + assert summary_typed[0]["data"]["usage"]["inputTokens"] == 500 # --------------------------------------------------------------------------- @@ -779,3 +788,52 @@ def test_metadata_structure(self): "usage": {"inputTokens": 1, "outputTokens": 1, "totalTokens": 2}, }) self._assert_structure(events) + + +class TestProcessAgentStreamMaxTokens: + """MaxTokensReachedException is classified as a recoverable max_tokens + error event (not the generic stream_error) and never leaks the raw SDK + message/URL.""" + + @pytest.mark.asyncio + async def test_max_tokens_emits_recoverable_error_event(self): + from strands.types.exceptions import MaxTokensReachedException + + async def mock_stream(): + yield {"start_event_loop": True} + raise MaxTokensReachedException( + "Agent has reached an unrecoverable state due to max_tokens " + "limit. For more information see: https://strandsagents.com/x" + ) + + events = [] + async for ev in process_agent_stream(mock_stream()): + events.append(ev) + + error_events = [e for e in events if e.get("type") == "error"] + assert len(error_events) == 1 + data = error_events[0]["data"] + assert data["code"] == "max_tokens" + assert data["recoverable"] is True + # detail is None and excluded — no leaked SDK URL/raw exception text. + assert "strandsagents.com" not in str(data) + assert "unrecoverable" not in str(data).lower() + + @pytest.mark.asyncio + async def test_generic_exception_still_stream_error(self): + """Regression: a non-max_tokens exception still maps to the + non-recoverable generic stream_error.""" + + async def mock_stream(): + yield {"start_event_loop": True} + raise RuntimeError("totally unrelated boom") + + events = [] + async for ev in process_agent_stream(mock_stream()): + events.append(ev) + + error_events = [e for e in events if e.get("type") == "error"] + assert len(error_events) == 1 + data = error_events[0]["data"] + assert data["code"] == "stream_error" + assert data["recoverable"] is False diff --git a/backend/tests/agents/main_agent/streaming/test_ui_resource_events.py b/backend/tests/agents/main_agent/streaming/test_ui_resource_events.py new file mode 100644 index 00000000..883abd3a --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_ui_resource_events.py @@ -0,0 +1,226 @@ +"""Tests for StreamCoordinator._extract_ui_resource_events. + +PR #3 of the MCP Apps host-renderer initiative +(`docs/kaizen/scoping/mcp-apps-host-renderer.md`). Covers the per-tool-result +`ui_resource` SSE emit: it fires only for UI-bearing tools, fetches the +resource via the hosting client's `resources/read` and inlines the HTML, +correlates by toolUseId, dedupes, stays inert behind the host flag, and +never breaks the stream on failure. + +Mirrors the helper-level style of `test_artifact_events.py` (drive the +coordinator method directly) and the mock-the-boundary catalog seeding from +`tests/agents/main_agent/integrations/test_mcp_apps.py`. +""" + +from __future__ import annotations + +import json + +import mcp.types as mcp_types +import pytest + +from agents.main_agent.integrations import mcp_apps +from agents.main_agent.integrations.mcp_apps import ( + MCP_APPS_UI_EXTENSION_KEY, + MCP_APPS_UI_MIME_TYPE, + get_ui_tool_catalog, + record_and_filter_ui_tools, +) +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + +_ENV_FLAG = "AGENTCORE_MCP_APPS_HOST_ENABLED" +_ENV_SANDBOX_ORIGIN = "AGENTCORE_MCP_APPS_SANDBOX_ORIGIN" + + +@pytest.fixture +def coord() -> StreamCoordinator: + return StreamCoordinator() + + +@pytest.fixture +def catalog_clean(monkeypatch): + get_ui_tool_catalog().clear() + monkeypatch.delenv(_ENV_FLAG, raising=False) + monkeypatch.delenv(_ENV_SANDBOX_ORIGIN, raising=False) + try: + yield + finally: + get_ui_tool_catalog().clear() + + +class _FakeMCPClient: + def __init__(self, result): + self._result = result + self.read_calls: list = [] + + def read_resource_sync(self, uri): + self.read_calls.append(uri) + return self._result + + +def _fake_tool(tool_name, ui): + from types import SimpleNamespace + + return SimpleNamespace( + tool_name=tool_name, + mcp_tool=SimpleNamespace(name=tool_name, meta={"ui": ui}), + ) + + +def _html_result(text="

hi

"): + return mcp_types.ReadResourceResult( + contents=[ + mcp_types.TextResourceContents( + uri="ui://srv/widget", + mimeType=MCP_APPS_UI_MIME_TYPE, + text=text, + _meta={ + MCP_APPS_UI_EXTENSION_KEY: { + "csp": {"connectDomains": ["https://api.test"]}, + "permissions": {"clipboardWrite": {}}, + } + }, + ) + ] + ) + + +def _seed(monkeypatch, client): + monkeypatch.setenv(_ENV_FLAG, "true") + record_and_filter_ui_tools( + [_fake_tool("widget", {"resourceUri": "ui://srv/widget"})], + client=client, + ) + + +def _tool_result_event(tool_use_id="tu-1"): + return { + "type": "tool_result", + "data": { + "tool_result": { + "toolUseId": tool_use_id, + "status": "success", + "content": [{"text": "ok"}], + } + }, + } + + +def _parse(raw: str) -> dict: + assert raw.startswith("event: ui_resource\ndata: ") + assert raw.endswith("\n\n") + return json.loads(raw[len("event: ui_resource\ndata: ") :].strip()) + + +@pytest.mark.asyncio +async def test_emits_ui_resource_with_inline_html( + coord, catalog_clean, monkeypatch +): + client = _FakeMCPClient(_html_result("
app
")) + _seed(monkeypatch, client) + + out = await coord._extract_ui_resource_events( + _tool_result_event("tu-1"), {"tu-1": "widget"}, set() + ) + + assert client.read_calls == ["ui://srv/widget"] + assert len(out) == 1 + payload = _parse(out[0]) + assert payload == { + "type": "ui_resource", + "toolUseId": "tu-1", + "resourceUri": "ui://srv/widget", + "html": "
app
", + "mimeType": MCP_APPS_UI_MIME_TYPE, + "csp": {"connectDomains": ["https://api.test"]}, + "permissions": {"clipboardWrite": {}}, + "sandboxOrigin": "", + } + + +@pytest.mark.asyncio +async def test_dedupes_per_tool_use_id(coord, catalog_clean, monkeypatch): + client = _FakeMCPClient(_html_result()) + _seed(monkeypatch, client) + emitted: set = set() + + first = await coord._extract_ui_resource_events( + _tool_result_event("tu-1"), {"tu-1": "widget"}, emitted + ) + second = await coord._extract_ui_resource_events( + _tool_result_event("tu-1"), {"tu-1": "widget"}, emitted + ) + + assert len(first) == 1 + assert second == [] + assert emitted == {"tu-1"} + # The dedupe must short-circuit before a second resources/read. + assert client.read_calls == ["ui://srv/widget"] + + +@pytest.mark.asyncio +async def test_inert_when_flag_disabled(coord, catalog_clean, monkeypatch): + client = _FakeMCPClient(_html_result()) + _seed(monkeypatch, client) + monkeypatch.setenv(_ENV_FLAG, "false") + + out = await coord._extract_ui_resource_events( + _tool_result_event("tu-1"), {"tu-1": "widget"}, set() + ) + assert out == [] + assert client.read_calls == [] + + +@pytest.mark.asyncio +async def test_noop_for_untracked_tool_use_id( + coord, catalog_clean, monkeypatch +): + client = _FakeMCPClient(_html_result()) + _seed(monkeypatch, client) + + # No name learned for this toolUseId → cannot map to the catalog. + out = await coord._extract_ui_resource_events( + _tool_result_event("tu-unknown"), {}, set() + ) + assert out == [] + assert client.read_calls == [] + + +@pytest.mark.asyncio +async def test_noop_when_tool_result_has_no_tool_use_id( + coord, catalog_clean, monkeypatch +): + client = _FakeMCPClient(_html_result()) + _seed(monkeypatch, client) + + event = {"type": "tool_result", "data": {"tool_result": {"status": "ok"}}} + out = await coord._extract_ui_resource_events( + event, {"tu-1": "widget"}, set() + ) + assert out == [] + + +@pytest.mark.asyncio +async def test_noop_for_non_ui_tool(coord, catalog_clean, monkeypatch): + # Flag on, but the tool has no `_meta.ui` in the catalog at all. + monkeypatch.setenv(_ENV_FLAG, "true") + out = await coord._extract_ui_resource_events( + _tool_result_event("tu-1"), {"tu-1": "plain_tool"}, set() + ) + assert out == [] + + +@pytest.mark.asyncio +async def test_failure_is_swallowed(coord, catalog_clean, monkeypatch): + _seed(monkeypatch, _FakeMCPClient(_html_result())) + + def _boom(tool_name, tool_use_id): + raise RuntimeError("catalog exploded") + + monkeypatch.setattr(mcp_apps, "fetch_ui_resource", _boom) + + # A failure in the fetch path must not propagate into the live stream. + out = await coord._extract_ui_resource_events( + _tool_result_event("tu-1"), {"tu-1": "widget"}, set() + ) + assert out == [] diff --git a/backend/tests/agents/main_agent/test_chat_agent_continue.py b/backend/tests/agents/main_agent/test_chat_agent_continue.py new file mode 100644 index 00000000..645d2b65 --- /dev/null +++ b/backend/tests/agents/main_agent/test_chat_agent_continue.py @@ -0,0 +1,70 @@ +"""ChatAgent.stream_async continuation-after-max_tokens behavior. + +A `continue_truncated=True` call must NOT synthesize a new user prompt: it +forwards an empty-list prompt so Strands appends no message and the model +resumes the truncated assistant message already in restored history. +""" + +import pytest + +from agents.main_agent.chat_agent import ChatAgent + + +class _RecordingCoordinator: + """Captures the prompt stream_async forwards to the coordinator.""" + + def __init__(self): + self.captured = {} + + async def stream_response(self, **kwargs): + self.captured = kwargs + if False: # pragma: no cover - make this an async generator + yield "" + + +class _ExplodingMultimodalBuilder: + """build_prompt must never be called on the continuation path.""" + + def build_prompt(self, message, files): # noqa: D401 + raise AssertionError("multimodal build_prompt called on continuation path") + + +def _bare_chat_agent(coordinator, multimodal): + agent = object.__new__(ChatAgent) + agent.agent = object() # truthy so _create_agent() is skipped + agent.stream_coordinator = coordinator + agent.multimodal_builder = multimodal + agent.session_manager = object() + agent.session_id = "sess-1" + agent.user_id = "user-1" + return agent + + +@pytest.mark.asyncio +async def test_continue_truncated_forwards_empty_list_prompt(): + coordinator = _RecordingCoordinator() + agent = _bare_chat_agent(coordinator, _ExplodingMultimodalBuilder()) + + async for _ in agent.stream_async( + "this message text must be ignored", + continue_truncated=True, + ): + pass + + assert coordinator.captured.get("prompt") == [] + + +@pytest.mark.asyncio +async def test_normal_turn_still_uses_multimodal_builder(): + coordinator = _RecordingCoordinator() + + class _Builder: + def build_prompt(self, message, files): + return f"built:{message}" + + agent = _bare_chat_agent(coordinator, _Builder()) + + async for _ in agent.stream_async("hello", continue_truncated=False): + pass + + assert coordinator.captured.get("prompt") == "built:hello" diff --git a/backend/tests/apis/app_api/admin/auth_providers/test_cognito_redirect_uri.py b/backend/tests/apis/app_api/admin/auth_providers/test_cognito_redirect_uri.py index e71a9dd1..671ace8a 100644 --- a/backend/tests/apis/app_api/admin/auth_providers/test_cognito_redirect_uri.py +++ b/backend/tests/apis/app_api/admin/auth_providers/test_cognito_redirect_uri.py @@ -7,7 +7,7 @@ from fastapi.testclient import TestClient from apis.shared.auth.models import User -from apis.shared.rbac.system_admin import require_system_admin +from apis.shared.auth import require_admin @pytest.fixture @@ -35,7 +35,7 @@ def _create_app(admin_user: User) -> FastAPI: admin_router = APIRouter(prefix="/admin") admin_router.include_router(router) app.include_router(admin_router) - app.dependency_overrides[require_system_admin] = lambda: admin_user + app.dependency_overrides[require_admin] = lambda: admin_user return app diff --git a/backend/tests/apis/app_api/artifacts/test_artifact_content.py b/backend/tests/apis/app_api/artifacts/test_artifact_content.py new file mode 100644 index 00000000..200385fd --- /dev/null +++ b/backend/tests/apis/app_api/artifacts/test_artifact_content.py @@ -0,0 +1,203 @@ +"""Tests for the app-api artifact content endpoint (panel code view). + +Covers ownership scoping, the Markdown unwrap (+ its fallback), the +inline size cap, and the fail-closed config behavior. +""" + +from __future__ import annotations + +import base64 + +import boto3 +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from moto import mock_aws + +from apis.app_api.artifacts import service as artifact_service +from apis.app_api.artifacts.routes import router as artifacts_router +from apis.shared.auth import User, get_current_user_from_session + +TABLE = "test-user-artifacts" +BUCKET = "test-artifacts-bucket" +REGION = "us-east-1" +USER_ID = "user-123" + + +@pytest.fixture(autouse=True) +def _reset_caches() -> None: + artifact_service._reset_caches_for_tests() + + +@pytest.fixture +def client(monkeypatch: pytest.MonkeyPatch): + with mock_aws(): + monkeypatch.setenv("AWS_REGION", REGION) + + ddb = boto3.client("dynamodb", region_name=REGION) + ddb.create_table( + TableName=TABLE, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + s3 = boto3.client("s3", region_name=REGION) + s3.create_bucket(Bucket=BUCKET) + + monkeypatch.setenv("DYNAMODB_ARTIFACTS_TABLE_NAME", TABLE) + monkeypatch.setenv("S3_ARTIFACTS_BUCKET_NAME", BUCKET) + + app = FastAPI() + app.include_router(artifacts_router) + app.dependency_overrides[get_current_user_from_session] = ( + lambda: User( + email="u@x.com", user_id=USER_ID, name="U", roles=[] + ) + ) + yield ( + TestClient(app), + boto3.resource("dynamodb", region_name=REGION), + s3, + ) + + +def _put( + ddb, + s3, + *, + user_id: str = USER_ID, + artifact: str = "art-1", + version: int = 1, + content_type: str = "text/html; charset=utf-8", + body: bytes = b"

hi

", + write_object: bool = True, + content_key: str | None = None, +) -> None: + key = content_key + if key is None: + key = f"{user_id}/{artifact}/v{version}/index.html" + ddb.Table(TABLE).put_item( + Item={ + "PK": f"USER#{user_id}", + "SK": f"ARTIFACT#{artifact}#V#{version:05d}", + "storage": "s3", + "content_key": key, + "content_type": content_type, + } + ) + if write_object: + s3.put_object(Bucket=BUCKET, Key=key, Body=body) + + +def _markdown_wrapper(md: str) -> bytes: + b64 = base64.b64encode(md.encode("utf-8")).decode("ascii") + return ( + "
Rendering…
" + '' + "" + "" + ).encode("utf-8") + + +def test_happy_path_returns_raw_source(client) -> None: + tc, ddb, s3 = client + _put(ddb, s3, body=b"

Hello

") + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 200 + body = resp.json() + assert body["content"] == "

Hello

" + assert body["content_type"] == "text/html; charset=utf-8" + assert body["version"] == 1 + + +def test_markdown_is_unwrapped_to_authored_source(client) -> None: + tc, ddb, s3 = client + md = "# Title\n\nSome **bold** text.\n" + _put( + ddb, + s3, + content_type="text/markdown", + body=_markdown_wrapper(md), + ) + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 200 + body = resp.json() + assert body["content"] == md + assert body["content_type"] == "text/markdown" + + +def test_markdown_without_src_tag_falls_back_to_raw(client) -> None: + """A Markdown row whose object lacks the embed (legacy / future + template) returns the raw stored bytes + real type, not an error.""" + tc, ddb, s3 = client + _put( + ddb, + s3, + content_type="text/markdown", + body=b"no embed here", + ) + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 200 + body = resp.json() + assert "no embed here" in body["content"] + assert body["content_type"] == "text/markdown" + + +def test_unknown_version_is_404(client) -> None: + tc, _, _ = client + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 404 + + +def test_other_users_artifact_is_404(client) -> None: + tc, ddb, s3 = client + _put(ddb, s3, user_id="someone-else") + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 404 + + +def test_missing_s3_object_is_404(client) -> None: + tc, ddb, s3 = client + _put(ddb, s3, write_object=False) + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 404 + + +def test_oversized_artifact_is_413(client, monkeypatch) -> None: + tc, ddb, s3 = client + monkeypatch.setattr(artifact_service, "_MAX_CONTENT_BYTES", 16) + _put(ddb, s3, body=b"x" * 64) + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 413 + + +def test_missing_bucket_is_500(client, monkeypatch) -> None: + tc, ddb, s3 = client + _put(ddb, s3) + monkeypatch.delenv("S3_ARTIFACTS_BUCKET_NAME", raising=False) + artifact_service._reset_caches_for_tests() + resp = tc.get("/artifacts/art-1/content", params={"version": 1}) + assert resp.status_code == 500 + + +def test_version_must_be_positive(client) -> None: + tc, ddb, s3 = client + _put(ddb, s3) + resp = tc.get("/artifacts/art-1/content", params={"version": 0}) + assert resp.status_code == 422 + + +def test_requires_authentication() -> None: + app = FastAPI() + app.include_router(artifacts_router) + resp = TestClient(app).get( + "/artifacts/art-1/content", params={"version": 1} + ) + assert resp.status_code == 401 diff --git a/backend/tests/apis/app_api/artifacts/test_list_artifacts.py b/backend/tests/apis/app_api/artifacts/test_list_artifacts.py new file mode 100644 index 00000000..82e25200 --- /dev/null +++ b/backend/tests/apis/app_api/artifacts/test_list_artifacts.py @@ -0,0 +1,365 @@ +"""Tests for the app-api session artifacts list endpoint. + +The endpoint returns *every version* of every artifact in a session via +a two-step query: SessionIndex (HEAD rows only) to discover the +artifacts, then a per-artifact main-table `SK begins_with #V#` query for +all immutable version rows. The SPA renders one card per version, +anchored to the turn that produced it via the per-version +`produced_by_message_index` the writer stamps. +""" + +from __future__ import annotations + +import boto3 +import pytest +from botocore.exceptions import ClientError +from fastapi import FastAPI +from fastapi.testclient import TestClient +from moto import mock_aws + +from apis.app_api.artifacts import service as artifact_service +from apis.app_api.artifacts.routes import router as artifacts_router +from apis.app_api.artifacts.service import ( + ArtifactListService, + ArtifactQueryError, + RenderTokenConfigError, + get_artifact_list_service, +) +from apis.shared.auth import User, get_current_user_from_session + +TABLE = "test-user-artifacts" +REGION = "us-east-1" +USER_ID = "user-123" +SESSION = "sess-9" + + +@pytest.fixture(autouse=True) +def _reset_caches() -> None: + artifact_service._reset_caches_for_tests() + + +@pytest.fixture +def client(monkeypatch: pytest.MonkeyPatch): + with mock_aws(): + monkeypatch.setenv("AWS_REGION", REGION) + ddb = boto3.client("dynamodb", region_name=REGION) + ddb.create_table( + TableName=TABLE, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + {"AttributeName": "GSI1PK", "AttributeType": "S"}, + {"AttributeName": "GSI1SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + GlobalSecondaryIndexes=[ + { + "IndexName": "SessionIndex", + "KeySchema": [ + {"AttributeName": "GSI1PK", "KeyType": "HASH"}, + {"AttributeName": "GSI1SK", "KeyType": "RANGE"}, + ], + "Projection": {"ProjectionType": "ALL"}, + } + ], + ) + + monkeypatch.setenv("DYNAMODB_ARTIFACTS_TABLE_NAME", TABLE) + + app = FastAPI() + app.include_router(artifacts_router) + app.dependency_overrides[get_current_user_from_session] = lambda: User( + email="u@x.com", user_id=USER_ID, name="U", roles=[] + ) + yield TestClient(app), boto3.resource("dynamodb", region_name=REGION) + + +def _put_version( + ddb, + *, + artifact: str, + version: int, + user_id: str = USER_ID, + session_id: str = SESSION, + title: str = "Doc", + updated_at: str | None = "2026-05-15T10:00:00+00:00", + created_at: str = "2026-05-15T10:00:00+00:00", + produced_by: int | None = None, +) -> None: + """One immutable version row, mirroring the writer. `updated_at` / + `produced_by` left None models a pre-per-version-linkage row.""" + item = { + "PK": f"USER#{user_id}", + "SK": f"ARTIFACT#{artifact}#V#{version:05d}", + "storage": "s3", + "content_key": f"{user_id}/{artifact}/v{version}/index.html", + "content_type": "text/html; charset=utf-8", + "version": version, + "artifact_id": artifact, + "user_id": user_id, + "session_id": session_id, + "title": title, + "created_at": created_at, + } + if updated_at is not None: + item["updated_at"] = updated_at + if produced_by is not None: + item["produced_by_message_index"] = produced_by + ddb.Table(TABLE).put_item(Item=item) + + +def _put_head( + ddb, + *, + artifact: str, + head_version: int, + user_id: str = USER_ID, + session_id: str = SESSION, + updated_at: str = "2026-05-15T10:00:00+00:00", + title: str = "Doc", +) -> None: + """The HEAD pointer row — carries the SessionIndex GSI keys used for + step-1 artifact discovery.""" + ddb.Table(TABLE).put_item( + Item={ + "PK": f"USER#{user_id}", + "SK": f"ARTIFACT#{artifact}#HEAD", + "GSI1PK": f"SESSION#{session_id}", + "GSI1SK": f"ARTIFACT#{updated_at}#{artifact}", + "storage": "s3", + "content_key": f"{user_id}/{artifact}/v{head_version}/index.html", + "content_type": "text/html; charset=utf-8", + "version": head_version, + "artifact_id": artifact, + "user_id": user_id, + "session_id": session_id, + "title": title, + "created_at": "2026-05-15T10:00:00+00:00", + "updated_at": updated_at, + } + ) + + +def _put_artifact( + ddb, + *, + artifact: str, + versions: list[dict], + user_id: str = USER_ID, + session_id: str = SESSION, +) -> None: + """N immutable version rows plus a HEAD at the latest — exactly what + the writer leaves after a create + updates sequence.""" + for v in versions: + _put_version( + ddb, + artifact=artifact, + user_id=user_id, + session_id=session_id, + **v, + ) + last = max(versions, key=lambda v: v["version"]) + _put_head( + ddb, + artifact=artifact, + head_version=last["version"], + user_id=user_id, + session_id=session_id, + updated_at=last.get("updated_at") or "2026-05-15T10:00:00+00:00", + title=last.get("title", "Doc"), + ) + + +def test_empty_session_is_empty_list(client) -> None: + tc, _ = client + resp = tc.get("/artifacts", params={"session_id": SESSION}) + assert resp.status_code == 200 + assert resp.json() == {"artifacts": []} + + +def test_returns_every_version_newest_artifact_first(client) -> None: + tc, ddb = client + _put_artifact( + ddb, + artifact="old", + versions=[ + {"version": 1, "updated_at": "2026-05-15T10:00:00+00:00", "title": "Old"} + ], + ) + _put_artifact( + ddb, + artifact="new", + versions=[ + {"version": 1, "updated_at": "2026-05-15T11:00:00+00:00", "title": "New"}, + {"version": 2, "updated_at": "2026-05-15T11:30:00+00:00", "title": "New"}, + {"version": 3, "updated_at": "2026-05-15T12:00:00+00:00", "title": "New"}, + ], + ) + + arts = tc.get("/artifacts", params={"session_id": SESSION}).json()[ + "artifacts" + ] + # Every version of every artifact is present. + assert {(a["artifact_id"], a["version"]) for a in arts} == { + ("new", 1), + ("new", 2), + ("new", 3), + ("old", 1), + } + # Step-1 discovery is HEAD-newest-first, so all of "new"'s versions + # come before "old"'s. + ids = [a["artifact_id"] for a in arts] + assert set(ids[:3]) == {"new"} + assert ids[-1] == "old" + + +def test_per_version_produced_by_index(client) -> None: + """Each version row carries its own linkage index so the SPA can + anchor every version's card under the turn that produced it. A row + without one (pre-linkage) is null → SPA end-of-conversation strip.""" + tc, ddb = client + _put_artifact( + ddb, + artifact="art-1", + versions=[ + {"version": 1, "updated_at": "2026-05-15T11:00:00+00:00", "produced_by": 3}, + {"version": 2, "updated_at": "2026-05-15T12:00:00+00:00", "produced_by": 7}, + {"version": 3, "updated_at": "2026-05-15T12:30:00+00:00"}, + ], + ) + arts = tc.get("/artifacts", params={"session_id": SESSION}).json()[ + "artifacts" + ] + by_v = {a["version"]: a for a in arts} + assert by_v[1]["produced_by_message_index"] == 3 + assert by_v[2]["produced_by_message_index"] == 7 + assert by_v[3]["produced_by_message_index"] is None + + +def test_legacy_version_rows_degrade_gracefully(client) -> None: + """Version rows written before per-version linkage lack updated_at / + produced_by_message_index. They must still be returned (empty/null) + so the SPA shows them in the strip rather than dropping them.""" + tc, ddb = client + _put_artifact( + ddb, + artifact="legacy", + versions=[ + {"version": 1, "updated_at": None}, + {"version": 2, "updated_at": None}, + ], + ) + arts = tc.get("/artifacts", params={"session_id": SESSION}).json()[ + "artifacts" + ] + assert {a["version"] for a in arts} == {1, 2} + for a in arts: + assert a["updated_at"] == "" + assert a["produced_by_message_index"] is None + + +def test_created_at_present_on_each_version(client) -> None: + tc, ddb = client + _put_artifact( + ddb, + artifact="art-1", + versions=[ + {"version": 1}, + {"version": 2, "updated_at": "2026-05-15T12:00:00+00:00"}, + ], + ) + arts = tc.get("/artifacts", params={"session_id": SESSION}).json()[ + "artifacts" + ] + assert all( + a["created_at"] == "2026-05-15T10:00:00+00:00" for a in arts + ) + + +def test_other_users_artifact_is_filtered(client) -> None: + """Step 1 drops a HEAD owned by another user that happens to share + the queried session id; step 2 is PK=USER#{caller}, so their version + rows are never read even if a HEAD leaked.""" + tc, ddb = client + _put_artifact(ddb, artifact="mine", versions=[{"version": 1}]) + _put_artifact( + ddb, + artifact="theirs", + versions=[{"version": 1}], + user_id="someone-else", + ) + + arts = tc.get("/artifacts", params={"session_id": SESSION}).json()[ + "artifacts" + ] + assert {a["artifact_id"] for a in arts} == {"mine"} + + +def test_session_id_required(client) -> None: + tc, _ = client + resp = tc.get("/artifacts") + assert resp.status_code == 422 + + +def test_requires_authentication() -> None: + app = FastAPI() + app.include_router(artifacts_router) + resp = TestClient(app).get("/artifacts", params={"session_id": SESSION}) + assert resp.status_code == 401 + + +def test_transient_query_error_is_not_a_config_error( + client, monkeypatch +) -> None: + """A transient DynamoDB ClientError is a runtime query failure, not a + misconfiguration — it must surface as ArtifactQueryError so the route + can distinguish a configured-but-throttled feature from a broken one.""" + + class _ThrottlingTable: + def query(self, **_): + raise ClientError( + {"Error": {"Code": "ThrottlingException", "Message": "slow down"}}, + "Query", + ) + + monkeypatch.setattr(artifact_service, "_table", lambda: _ThrottlingTable()) + with pytest.raises(ArtifactQueryError): + ArtifactListService().list_for_session( + user_id=USER_ID, session_id=SESSION + ) + + +def test_route_maps_transient_query_failure_to_503(client) -> None: + """ArtifactQueryError → 503 (retryable), distinct from the 500 a real + RenderTokenConfigError misconfiguration produces.""" + tc, _ = client + + class _FailingService: + def list_for_session(self, **_): + raise ArtifactQueryError("artifact list query failed") + + tc.app.dependency_overrides[get_artifact_list_service] = _FailingService + try: + resp = tc.get("/artifacts", params={"session_id": SESSION}) + finally: + tc.app.dependency_overrides.pop(get_artifact_list_service, None) + assert resp.status_code == 503 + + +def test_route_maps_misconfig_to_500(client) -> None: + tc, _ = client + + class _MisconfiguredService: + def list_for_session(self, **_): + raise RenderTokenConfigError("DYNAMODB_ARTIFACTS_TABLE_NAME is not set") + + tc.app.dependency_overrides[get_artifact_list_service] = _MisconfiguredService + try: + resp = tc.get("/artifacts", params={"session_id": SESSION}) + finally: + tc.app.dependency_overrides.pop(get_artifact_list_service, None) + assert resp.status_code == 500 diff --git a/backend/tests/apis/app_api/artifacts/test_render_token.py b/backend/tests/apis/app_api/artifacts/test_render_token.py new file mode 100644 index 00000000..0aaca0ed --- /dev/null +++ b/backend/tests/apis/app_api/artifacts/test_render_token.py @@ -0,0 +1,182 @@ +"""Tests for the app-api render-token minter. + +The headline test (`test_token_verifies_against_render_lambda`) mints a +token and feeds it straight through #309's Lambda verifier with a shared +signing key — that is the real cross-PR contract guarantee. +""" + +from __future__ import annotations + +import jwt +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from moto import mock_aws +import boto3 + +from apis.app_api.artifacts import service as token_service +from apis.app_api.artifacts.routes import router as artifacts_router +from apis.shared.auth import User, get_current_user_from_session +from lambdas.artifact_render import handler as render_lambda + +KEY = "test-render-key-44-chars-of-entropy-aaaaaaaa" +SECRET_NAME = "test-artifact-render-token-key" +TABLE = "test-user-artifacts" +ORIGIN = "https://artifacts.test.example.com" +REGION = "us-east-1" +USER_ID = "user-123" + + +@pytest.fixture(autouse=True) +def _reset_caches(monkeypatch: pytest.MonkeyPatch) -> None: + token_service._reset_caches_for_tests() + # The verifier caches its own signing key separately. + monkeypatch.setattr(render_lambda, "_cached_signing_key", None) + + +@pytest.fixture +def client(monkeypatch: pytest.MonkeyPatch): + with mock_aws(): + monkeypatch.setenv("AWS_REGION", REGION) + sm = boto3.client("secretsmanager", region_name=REGION) + arn = sm.create_secret(Name=SECRET_NAME, SecretString=KEY)["ARN"] + + ddb = boto3.client("dynamodb", region_name=REGION) + ddb.create_table( + TableName=TABLE, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + + monkeypatch.setenv("ARTIFACTS_RENDER_TOKEN_SECRET_ARN", arn) + monkeypatch.setenv("DYNAMODB_ARTIFACTS_TABLE_NAME", TABLE) + monkeypatch.setenv("ARTIFACTS_ORIGIN", ORIGIN) + + app = FastAPI() + app.include_router(artifacts_router) + app.dependency_overrides[get_current_user_from_session] = ( + lambda: User(email="u@x.com", user_id=USER_ID, name="U", roles=[]) + ) + yield TestClient(app), boto3.resource("dynamodb", region_name=REGION) + + +def _put_version(ddb, *, user_id: str = USER_ID, artifact="art-1", version=1) -> None: + ddb.Table(TABLE).put_item( + Item={ + "PK": f"USER#{user_id}", + "SK": f"ARTIFACT#{artifact}#V#{version:05d}", + "storage": "s3", + "content_key": f"{user_id}/{artifact}/v{version}/index.html", + "content_type": "text/html; charset=utf-8", + } + ) + + +def _token_from_url(url: str) -> str: + assert url.startswith(f"{ORIGIN}/?t=") + return url.split("?t=", 1)[1] + + +def test_happy_path_mints_valid_token(client) -> None: + tc, ddb = client + _put_version(ddb) + resp = tc.post( + "/artifacts/art-1/render-token", json={"version": 1, "sessionId": "sess-9"} + ) + assert resp.status_code == 200 + body = resp.json() + claims = jwt.decode( + _token_from_url(body["url"]), + KEY, + algorithms=["HS256"], + audience="artifact-render", + ) + assert claims["iss"] == "app-api" + assert claims["sub"] == USER_ID + assert claims["aid"] == "art-1" + assert claims["ver"] == 1 + assert claims["sid"] == "sess-9" + assert claims["exp"] - claims["iat"] == 120 + assert body["expires_at"].endswith("+00:00") + + +def test_token_verifies_against_render_lambda(client, monkeypatch) -> None: + """The cross-PR contract: a freshly minted token must pass the + actual #309 verifier byte-for-byte with the same signing key.""" + tc, ddb = client + _put_version(ddb) + monkeypatch.setattr(render_lambda, "_cached_signing_key", KEY) + + resp = tc.post("/artifacts/art-1/render-token", json={"version": 1}) + token = _token_from_url(resp.json()["url"]) + + verified = render_lambda._verify_token(token) + assert verified["sub"] == USER_ID + assert verified["aid"] == "art-1" + assert verified["ver"] == 1 + + +def test_unknown_version_is_404(client) -> None: + tc, _ = client + resp = tc.post("/artifacts/art-1/render-token", json={"version": 1}) + assert resp.status_code == 404 + + +def test_other_users_artifact_is_404(client) -> None: + """Ownership scoping: a record owned by someone else is invisible + because the PK is built from the authenticated user's id.""" + tc, ddb = client + _put_version(ddb, user_id="someone-else") + resp = tc.post("/artifacts/art-1/render-token", json={"version": 1}) + assert resp.status_code == 404 + + +def test_version_must_be_positive(client) -> None: + tc, ddb = client + _put_version(ddb) + resp = tc.post("/artifacts/art-1/render-token", json={"version": 0}) + assert resp.status_code == 422 + + +def test_session_id_optional(client) -> None: + tc, ddb = client + _put_version(ddb) + resp = tc.post("/artifacts/art-1/render-token", json={"version": 1}) + assert resp.status_code == 200 + claims = jwt.decode( + _token_from_url(resp.json()["url"]), + KEY, + algorithms=["HS256"], + audience="artifact-render", + ) + assert claims["sid"] == "" + + +def test_missing_origin_is_500(client, monkeypatch) -> None: + """Fail-closed config: with ARTIFACTS_ORIGIN unset the service must + 500 before minting — never hand back a usable token embedded in a + relative, unloadable URL. The artifact row exists, so a 500 (not a + 404) proves the origin check fires first.""" + tc, ddb = client + _put_version(ddb) + monkeypatch.delenv("ARTIFACTS_ORIGIN", raising=False) + resp = tc.post("/artifacts/art-1/render-token", json={"version": 1}) + assert resp.status_code == 500 + + +def test_requires_authentication() -> None: + """No dependency override and no session cookie → the route is + blocked by the session dependency, never reaching mint logic.""" + app = FastAPI() + app.include_router(artifacts_router) + resp = TestClient(app).post( + "/artifacts/art-1/render-token", json={"version": 1} + ) + assert resp.status_code == 401 diff --git a/backend/tests/apis/app_api/test_mcp_apps_proxy_call.py b/backend/tests/apis/app_api/test_mcp_apps_proxy_call.py new file mode 100644 index 00000000..ea514e03 --- /dev/null +++ b/backend/tests/apis/app_api/test_mcp_apps_proxy_call.py @@ -0,0 +1,134 @@ +"""Tests for the cookie-authenticated MCP App tools/call proxy (PR #5). + +Mirrors `test_proxy_routes.py`: the upstream client seam +(`proxy_routes._build_upstream_client`) is swapped for a MockTransport so +the relay to inference-api `/invocations` is asserted without a network. +""" + +from __future__ import annotations + +import json +from typing import Callable, Optional + +import httpx +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from apis.app_api.chat import proxy_routes +from apis.app_api.mcp_apps.routes import router as mcp_apps_router +from apis.shared.auth.dependencies import get_current_user_from_session +from apis.shared.auth.models import User + + +def _user(raw_token: str = "access.token.value") -> User: + user = User( + email="alice@example.com", + user_id="user-sub", + name="Alice", + roles=["user"], + ) + user.raw_token = raw_token + return user + + +def _build_app(*, user_override: Optional[User] = None) -> FastAPI: + app = FastAPI() + app.include_router(mcp_apps_router) + if user_override is not None: + app.dependency_overrides[get_current_user_from_session] = ( + lambda: user_override + ) + return app + + +def _patch_upstream( + monkeypatch: pytest.MonkeyPatch, + handler: Callable[[httpx.Request], httpx.Response], +) -> None: + transport = httpx.MockTransport(handler) + monkeypatch.setattr( + proxy_routes, + "_build_upstream_client", + lambda: httpx.AsyncClient(transport=transport), + ) + + +_BODY = { + "sessionId": "sess-1", + "toolUseId": "tu-1", + "toolName": "widget_tool", + "arguments": {"q": "x"}, + "enabledTools": ["gateway_widget"], + "modelId": "m1", +} + + +def test_requires_session() -> None: + # No auth override → get_current_user_from_session rejects. + resp = TestClient(_build_app()).post("/mcp-apps/proxy-call", json=_BODY) + assert resp.status_code == 401 + + +def test_relays_directive_and_bearer_then_returns_result( + monkeypatch: pytest.MonkeyPatch, +) -> None: + seen: dict = {} + + def handler(request: httpx.Request) -> httpx.Response: + seen["url"] = str(request.url) + seen["auth"] = request.headers.get("Authorization") + seen["body"] = json.loads(request.content) + return httpx.Response( + 200, + json={ + "toolUseId": "tu-1", + "result": {"content": [{"type": "text", "text": "ok"}], "isError": False}, + }, + ) + + _patch_upstream(monkeypatch, handler) + app = _build_app(user_override=_user("tok-abc")) + + resp = TestClient(app).post("/mcp-apps/proxy-call", json=_BODY) + + assert resp.status_code == 200 + assert resp.json()["result"]["content"][0]["text"] == "ok" + assert seen["url"].endswith("/invocations") + assert seen["auth"] == "Bearer tok-abc" + # The conversation binding + directive are forwarded verbatim. + assert seen["body"]["session_id"] == "sess-1" + assert seen["body"]["enabled_tools"] == ["gateway_widget"] + assert seen["body"]["app_tool_call"] == { + "tool_use_id": "tu-1", + "tool_name": "widget_tool", + "arguments": {"q": "x"}, + } + + +def test_relays_inference_error_status_verbatim( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # inference-api rejected the tool as not app-visible (spec MUST gate). + def handler(_request: httpx.Request) -> httpx.Response: + return httpx.Response(403, json={"error": "not app-visible"}) + + _patch_upstream(monkeypatch, handler) + app = _build_app(user_override=_user()) + + resp = TestClient(app).post("/mcp-apps/proxy-call", json=_BODY) + assert resp.status_code == 403 + assert resp.json()["error"] == "not app-visible" + + +def test_maps_unreachable_inference_to_502( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def handler(_request: httpx.Request) -> httpx.Response: + raise httpx.ConnectError("refused") + + _patch_upstream(monkeypatch, handler) + app = _build_app(user_override=_user()) + + resp = TestClient(app).post("/mcp-apps/proxy-call", json=_BODY) + assert resp.status_code == 502 diff --git a/backend/tests/apis/app_api/test_mcp_apps_update_context.py b/backend/tests/apis/app_api/test_mcp_apps_update_context.py new file mode 100644 index 00000000..a0b42861 --- /dev/null +++ b/backend/tests/apis/app_api/test_mcp_apps_update_context.py @@ -0,0 +1,125 @@ +"""Tests for the cookie-authenticated ui/update-model-context relay (PR #6). + +Mirrors `test_mcp_apps_proxy_call.py`: the upstream client seam is swapped +for a MockTransport so the relay to inference-api `/invocations` is +asserted without a network. +""" + +from __future__ import annotations + +import json +from typing import Callable, Optional + +import httpx +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from apis.app_api.chat import proxy_routes +from apis.app_api.mcp_apps.routes import router as mcp_apps_router +from apis.shared.auth.dependencies import get_current_user_from_session +from apis.shared.auth.models import User + + +def _user(raw_token: str = "access.token.value") -> User: + user = User( + email="alice@example.com", + user_id="user-sub", + name="Alice", + roles=["user"], + ) + user.raw_token = raw_token + return user + + +def _build_app(*, user_override: Optional[User] = None) -> FastAPI: + app = FastAPI() + app.include_router(mcp_apps_router) + if user_override is not None: + app.dependency_overrides[get_current_user_from_session] = ( + lambda: user_override + ) + return app + + +def _patch_upstream( + monkeypatch: pytest.MonkeyPatch, + handler: Callable[[httpx.Request], httpx.Response], +) -> None: + transport = httpx.MockTransport(handler) + monkeypatch.setattr( + proxy_routes, + "_build_upstream_client", + lambda: httpx.AsyncClient(transport=transport), + ) + + +_BODY = { + "sessionId": "sess-1", + "resourceUri": "ui://srv/widget", + "content": [{"type": "text", "text": "user picked X"}], + "structuredContent": {"selection": "X"}, + "enabledTools": ["gateway_widget"], + "modelId": "m1", +} + + +def test_requires_session() -> None: + resp = TestClient(_build_app()).post("/mcp-apps/update-context", json=_BODY) + assert resp.status_code == 401 + + +def test_relays_directive_and_bearer(monkeypatch: pytest.MonkeyPatch) -> None: + seen: dict = {} + + def handler(request: httpx.Request) -> httpx.Response: + seen["url"] = str(request.url) + seen["auth"] = request.headers.get("Authorization") + seen["body"] = json.loads(request.content) + return httpx.Response( + 200, json={"resourceUri": "ui://srv/widget", "status": "stored"} + ) + + _patch_upstream(monkeypatch, handler) + app = _build_app(user_override=_user("tok-xyz")) + + resp = TestClient(app).post("/mcp-apps/update-context", json=_BODY) + + assert resp.status_code == 200 + assert resp.json()["status"] == "stored" + assert seen["url"].endswith("/invocations") + assert seen["auth"] == "Bearer tok-xyz" + assert seen["body"]["session_id"] == "sess-1" + assert seen["body"]["enabled_tools"] == ["gateway_widget"] + assert seen["body"]["app_context_update"] == { + "resource_uri": "ui://srv/widget", + "content": [{"type": "text", "text": "user picked X"}], + "structured_content": {"selection": "X"}, + } + + +def test_relays_inference_error_status_verbatim( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def handler(_request: httpx.Request) -> httpx.Response: + return httpx.Response(400, json={"error": "needs content"}) + + _patch_upstream(monkeypatch, handler) + app = _build_app(user_override=_user()) + + resp = TestClient(app).post("/mcp-apps/update-context", json=_BODY) + assert resp.status_code == 400 + assert resp.json()["error"] == "needs content" + + +def test_maps_unreachable_inference_to_502( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def handler(_request: httpx.Request) -> httpx.Response: + raise httpx.ConnectError("refused") + + _patch_upstream(monkeypatch, handler) + app = _build_app(user_override=_user()) + + resp = TestClient(app).post("/mcp-apps/update-context", json=_BODY) + assert resp.status_code == 502 diff --git a/backend/tests/apis/inference_api/test_app_context_dispatch.py b/backend/tests/apis/inference_api/test_app_context_dispatch.py new file mode 100644 index 00000000..fdefc1be --- /dev/null +++ b/backend/tests/apis/inference_api/test_app_context_dispatch.py @@ -0,0 +1,137 @@ +"""Tests for app-pushed model context dispatch (MCP Apps PR #6). + +Uses a fake that faithfully mimics strands 1.40 `AgentState`: `.get()` +returns a deep copy (so the read-modify-write path is genuinely +exercised) and `.set()` enforces JSON-serializability. No live agent. +""" + +import copy +import json + +import pytest + +from apis.inference_api.chat.app_context_dispatch import ( + STATE_KEY, + AppContextUpdateError, + dispatch_app_context_update, + merge_and_clear_pending_context, +) + + +class _FakeState: + """Mimics strands.agent.state.AgentState get/set semantics.""" + + def __init__(self) -> None: + self._data: dict = {} + + def get(self, key=None): + if key is None: + return copy.deepcopy(self._data) + return copy.deepcopy(self._data.get(key)) + + def set(self, key: str, value) -> None: + json.dumps(value) # raises TypeError/ValueError if not serializable + self._data[key] = copy.deepcopy(value) + + +class _FakeStrands: + def __init__(self) -> None: + self.state = _FakeState() + + +class _FakeAgent: + """BaseAgent wrapper — inner Strands agent is `.agent`.""" + + def __init__(self) -> None: + self.agent = _FakeStrands() + + +def test_dispatch_writes_under_resource_uri_and_acks(): + agent = _FakeAgent() + ack = dispatch_app_context_update( + agent, + resource_uri="ui://srv/widget", + content=[{"type": "text", "text": "hello"}], + structured_content={"count": 2}, + ) + assert ack == { + "resourceUri": "ui://srv/widget", + "status": "stored", + "pending": 1, + } + bag = agent.agent.state.get(STATE_KEY) + entry = bag["context"]["ui://srv/widget"] + assert entry["content"] == [{"type": "text", "text": "hello"}] + assert entry["structuredContent"] == {"count": 2} + assert "updatedAt" in entry + + +def test_last_write_wins_per_resource_uri(): + agent = _FakeAgent() + dispatch_app_context_update( + agent, resource_uri="ui://a", content=None, structured_content={"v": 1} + ) + ack = dispatch_app_context_update( + agent, resource_uri="ui://a", content=None, structured_content={"v": 2} + ) + assert ack["pending"] == 1 # same uri overwrote, not appended + bag = agent.agent.state.get(STATE_KEY) + assert bag["context"]["ui://a"]["structuredContent"] == {"v": 2} + + +def test_requires_content_or_structured(): + with pytest.raises(AppContextUpdateError) as ei: + dispatch_app_context_update( + _FakeAgent(), resource_uri="ui://a", content=None, structured_content=None + ) + assert ei.value.code == 400 + + +def test_missing_agent_state_is_409(): + class _NoState: + agent = None + + with pytest.raises(AppContextUpdateError) as ei: + dispatch_app_context_update( + _NoState(), resource_uri="ui://a", content=None, + structured_content={"x": 1}, + ) + assert ei.value.code == 409 + + +def test_merge_drains_clears_and_dedupes_by_uri(): + agent = _FakeAgent() + dispatch_app_context_update( + agent, resource_uri="ui://a", content=None, + structured_content={"a": 1}, + ) + dispatch_app_context_update( + agent, + resource_uri="ui://b", + content=[{"type": "text", "text": "note-b"}], + structured_content=None, + ) + + block = merge_and_clear_pending_context(agent) + assert block is not None + assert "" in block and "" in block + assert 'resource="ui://a"' in block + assert 'resource="ui://b"' in block + assert "note-b" in block + assert '"a": 1' in block + + # Cleared: a second merge with no new updates yields nothing. + assert merge_and_clear_pending_context(agent) is None + + +def test_merge_empty_returns_none(): + assert merge_and_clear_pending_context(_FakeAgent()) is None + + +def test_merge_never_raises_on_bad_agent(): + class _Broken: + agent = None + + # _strands_agent would raise AppContextUpdateError(409); merge swallows + # it (context is best-effort and must never break a turn). + assert merge_and_clear_pending_context(_Broken()) is None diff --git a/backend/tests/apis/inference_api/test_app_tool_dispatch.py b/backend/tests/apis/inference_api/test_app_tool_dispatch.py new file mode 100644 index 00000000..d270b52c --- /dev/null +++ b/backend/tests/apis/inference_api/test_app_tool_dispatch.py @@ -0,0 +1,167 @@ +"""Tests for app-initiated tools/call dispatch (MCP Apps PR #5). + +Mocks the boundary the way the PR #3 tests do: a fake MCP client + +`UIToolCatalog`, no live agent. Asserts the spec-MUST app-visibility gate +at the inference-api dispatch, and that a successful call publishes +synthesized tool_use/tool_result into the per-session broker. +""" + +import pytest + +from apis.inference_api.chat.app_tool_dispatch import ( + AppToolCallError, + dispatch_app_tool_call, +) +from apis.shared.mcp_apps.broker import get_app_tool_event_broker +from apis.shared.tools.models import ToolUIMetadata +from agents.main_agent.integrations import mcp_apps as mcp_apps_mod + + +class _FakeContent: + def __init__(self, text: str) -> None: + self._text = text + + def model_dump(self, **_: object) -> dict: + return {"type": "text", "text": self._text} + + +class _FakeResult: + def __init__(self, text: str = "ok", is_error: bool = False) -> None: + self.content = [_FakeContent(text)] + self.isError = is_error + + +class _FakeClient: + def __init__(self, result=None, raises: Exception | None = None) -> None: + self._result = result if result is not None else _FakeResult() + self._raises = raises + self.calls: list = [] + + def call_tool_sync(self, tool_use_id, name, arguments=None): + self.calls.append((tool_use_id, name, arguments)) + if self._raises is not None: + raise self._raises + return self._result + + +class _FakeCatalog: + def __init__(self, meta=None, client=None) -> None: + self._meta = meta + self._client = client + + def get(self, _name): + return self._meta + + def get_client(self, _name): + return self._client + + +def _patch(monkeypatch, *, enabled=True, meta=None, client=None): + monkeypatch.setattr( + mcp_apps_mod, "is_mcp_apps_host_enabled", lambda: enabled + ) + monkeypatch.setattr( + mcp_apps_mod, + "get_ui_tool_catalog", + lambda: _FakeCatalog(meta=meta, client=client), + ) + + +def _ui(visibility): + return ToolUIMetadata(resource_uri="ui://srv/w", visibility=visibility) + + +async def _call(session_id="disp-s1", tool_name="widget_tool"): + return await dispatch_app_tool_call( + agent=None, + session_id=session_id, + user_id="u1", + tool_use_id="tu-1", + tool_name=tool_name, + arguments={"q": "x"}, + ) + + +@pytest.mark.asyncio +async def test_rejects_when_host_flag_disabled(monkeypatch): + _patch(monkeypatch, enabled=False) + with pytest.raises(AppToolCallError) as ei: + await _call() + assert ei.value.code == 403 + + +@pytest.mark.asyncio +async def test_rejects_unknown_tool(monkeypatch): + _patch(monkeypatch, enabled=True, meta=None, client=_FakeClient()) + with pytest.raises(AppToolCallError) as ei: + await _call() + assert ei.value.code == 403 + + +@pytest.mark.asyncio +async def test_rejects_tool_not_app_visible(monkeypatch): + # visibility=["model"] → callable by the model, NOT by an app. + _patch( + monkeypatch, + enabled=True, + meta=_ui(["model"]), + client=_FakeClient(), + ) + with pytest.raises(AppToolCallError) as ei: + await _call() + assert ei.value.code == 403 + + +@pytest.mark.asyncio +async def test_rejects_when_no_live_client(monkeypatch): + _patch(monkeypatch, enabled=True, meta=_ui(["model", "app"]), client=None) + with pytest.raises(AppToolCallError) as ei: + await _call() + assert ei.value.code == 409 + + +@pytest.mark.asyncio +async def test_dispatch_failure_maps_to_502(monkeypatch): + _patch( + monkeypatch, + enabled=True, + meta=_ui(["app"]), + client=_FakeClient(raises=RuntimeError("boom")), + ) + with pytest.raises(AppToolCallError) as ei: + await _call() + assert ei.value.code == 502 + + +@pytest.mark.asyncio +async def test_success_returns_result_and_publishes_thread_events(monkeypatch): + client = _FakeClient(_FakeResult("hello")) + _patch(monkeypatch, enabled=True, meta=_ui(["model", "app"]), client=client) + + broker = get_app_tool_event_broker() + q = broker.add_subscriber("disp-ok") + try: + payload = await dispatch_app_tool_call( + agent=None, + session_id="disp-ok", + user_id="u1", + tool_use_id="tu-9", + tool_name="widget_tool", + arguments={"q": "x"}, + ) + finally: + events = broker.drain(q) + broker.remove_subscriber("disp-ok", q) + + assert payload["toolUseId"] == "tu-9" + assert payload["result"]["isError"] is False + assert payload["result"]["content"] == [{"type": "text", "text": "hello"}] + # The MCP client was called with a synthesized (distinct) id. + assert client.calls[0][1] == "widget_tool" + assert client.calls[0][0] != "tu-9" + # Both thread events were published, tool_use before tool_result. + types = [e["type"] for e in events] + assert types == ["tool_use", "tool_result"] + assert events[0]["data"]["tool_use"]["name"] == "widget_tool" + assert events[0]["data"]["tool_use"]["origin"] == "mcp_app" + assert events[1]["data"]["tool_result"]["status"] == "success" diff --git a/backend/tests/apis/inference_api/test_inference_param_merge.py b/backend/tests/apis/inference_api/test_inference_param_merge.py new file mode 100644 index 00000000..60801937 --- /dev/null +++ b/backend/tests/apis/inference_api/test_inference_param_merge.py @@ -0,0 +1,123 @@ +"""Tests for the inference-param merge guard in ``apis.inference_api.chat.routes``. + +Focus: the cross-param safety check that drops ``thinking`` when +``thinking >= max_tokens`` (Anthropic rejects that request outright). Inference +params arrive untyped (``Dict[str, Any]`` from JSON), so an int bound can show +up as a float — an ``isinstance(..., int)`` gate used to silently skip the +check on float input and let the bad request through. +""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from apis.inference_api.chat.routes import _as_int_or_none, _merge_inference_params +from apis.shared.models.models import ModelParamSpec, SupportedParams + + +def _model(**specs: ModelParamSpec) -> SimpleNamespace: + """Minimal managed-model stand-in: only ``supported_params`` + ``model_id``.""" + return SimpleNamespace( + model_id="test-model", + supported_params=SupportedParams(params=dict(specs)), + ) + + +# Wide bounds so request values pass through unclamped (and keep their +# original float type), reproducing the JSON-sourced-float scenario. +_WIDE_MAX_TOKENS = ModelParamSpec(supported=True, min=1, max=200000) +_WIDE_THINKING = ModelParamSpec(supported=True, min=1024, max=None) + + +class TestAsIntOrNone: + @pytest.mark.parametrize( + "value,expected", + [ + (8192, 8192), + (8192.0, 8192), + (100000.0, 100000), + (True, None), + (False, None), + (None, None), + ("8192", None), + ({"type": "enabled"}, None), + ], + ) + def test_coercion(self, value, expected): + assert _as_int_or_none(value) == expected + + +class TestThinkingGuardFloatInput: + def test_float_thinking_ge_float_max_tokens_drops_thinking(self): + """The original bug: both arrive as floats, thinking >= max_tokens. + The guard must still fire and drop thinking.""" + model = _model(max_tokens=_WIDE_MAX_TOKENS, thinking=_WIDE_THINKING) + merged = _merge_inference_params( + model, {"max_tokens": 2048.0, "thinking": 4096.0} + ) + + assert "thinking" not in merged + assert merged["max_tokens"] == 2048.0 + + def test_float_thinking_below_float_max_tokens_is_retained(self): + """Guard must not over-drop when the float values are consistent.""" + model = _model(max_tokens=_WIDE_MAX_TOKENS, thinking=_WIDE_THINKING) + merged = _merge_inference_params( + model, {"max_tokens": 8192.0, "thinking": 2048.0} + ) + + assert merged["thinking"] == 2048.0 + assert merged["max_tokens"] == 8192.0 + + def test_int_inputs_still_guarded(self): + """Pre-existing int path must keep working.""" + model = _model(max_tokens=_WIDE_MAX_TOKENS, thinking=_WIDE_THINKING) + merged = _merge_inference_params( + model, {"max_tokens": 2048, "thinking": 4096} + ) + + assert "thinking" not in merged + + +class TestEffortAllowedGating: + """`effort` is enum-gated: a request override must be a member of the + admin-declared `allowed` set, else it falls back to the default. The + per-model effort-tier difference (Sonnet 4.6 vs Opus 4.7) is data on + `ModelParamSpec.allowed`, not model-family code.""" + + _SONNET_EFFORT = ModelParamSpec( + supported=True, allowed=["low", "medium", "high"], default="high" + ) + _OPUS_EFFORT = ModelParamSpec( + supported=True, allowed=["low", "medium", "high", "xhigh", "max"], default="high" + ) + + def test_in_domain_override_is_kept(self): + model = _model(effort=self._SONNET_EFFORT) + merged = _merge_inference_params(model, {"effort": "low"}) + assert merged["effort"] == "low" + + def test_out_of_domain_override_falls_back_to_default(self): + # `xhigh` is Opus-4.7-only; on a Sonnet-4.6-shaped spec it's rejected + # and the admin default wins instead of erroring mid-stream. + model = _model(effort=self._SONNET_EFFORT) + merged = _merge_inference_params(model, {"effort": "xhigh"}) + assert merged["effort"] == "high" + + def test_xhigh_allowed_on_opus_spec(self): + model = _model(effort=self._OPUS_EFFORT) + merged = _merge_inference_params(model, {"effort": "xhigh"}) + assert merged["effort"] == "xhigh" + + def test_no_override_uses_default(self): + model = _model(effort=self._SONNET_EFFORT) + merged = _merge_inference_params(model, {}) + assert merged["effort"] == "high" + + def test_out_of_domain_with_no_default_is_dropped(self): + spec = ModelParamSpec(supported=True, allowed=["low", "medium", "high"]) + model = _model(effort=spec) + merged = _merge_inference_params(model, {"effort": "max"}) + assert "effort" not in merged diff --git a/backend/tests/apis/shared/middleware/test_session_refresh_bug_condition.py b/backend/tests/apis/shared/middleware/test_session_refresh_bug_condition.py new file mode 100644 index 00000000..3445180c --- /dev/null +++ b/backend/tests/apis/shared/middleware/test_session_refresh_bug_condition.py @@ -0,0 +1,739 @@ +"""Bug condition exploration property tests for SessionRefreshMiddleware event-loop blocking. + +Property 1: Bug Condition — Event-Loop Non-Blocking, Coalesced, Window-Staggered, Fire-and-Forget + +This file encodes the EXPECTED behavior (Property 1 / Expected Behavior 2.1–2.7) from +the design document. Each sub-condition test surfaces a counterexample that demonstrates +the corresponding sub-condition (1.1–1.7) of `isBugCondition` from design.md. + +CRITICAL: These tests MUST FAIL on unfixed code — failure confirms the bug exists. +They will PASS after the fix (task 3 series) is implemented: + - Repository/Cognito offload via asyncio.to_thread (2.1, 2.2) + - Per-session single-flight for the resolve path (2.3) + - Strict-multiple windows (throttle=300s, leeway=60s) (2.4) + - Fire-and-forget slide-write (2.5) + - appApi.desiredCount >= 2 (2.6) + - Bounded blocking DDB calls across fan-out (2.7) + +Scoped PBT Approach: each sub-condition is reproduced by a concrete, deterministic +scenario under pytest-asyncio. Hypothesis is used on the two sub-conditions that +generalize over a family of inputs (fan-out size for 1.3 / 1.7). + +Validates: Requirements 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7 +""" + +from __future__ import annotations + +import asyncio +import json +import secrets +import time +from pathlib import Path +from typing import Any, Optional +from unittest.mock import MagicMock + +import httpx +import pytest +from cryptography.hazmat.primitives.ciphers.aead import AESGCM +from fastapi import FastAPI, Request +from hypothesis import HealthCheck, given, settings +from hypothesis import strategies as st + +from apis.shared.middleware.session_refresh import SessionRefreshMiddleware +from apis.shared.sessions_bff import lock as lock_module +from apis.shared.sessions_bff.cache import SessionCache +from apis.shared.sessions_bff.config import ( + BFFConfig, + SESSION_COOKIE_NAME, + _DEFAULT_REFRESH_LEEWAY_SECONDS, + _DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS, +) +from apis.shared.sessions_bff.cookie import CookieCodec +from apis.shared.sessions_bff.lock import get_session_lock +from apis.shared.sessions_bff.models import CookiePayload, SessionRecord +from apis.shared.sessions_bff.refresh import ( + CognitoRefreshClient, + _reset_secret_cache_for_tests, +) +from apis.shared.sessions_bff.repository import SessionRepository + + +# ═══════════════════════════════════════════════════════════════════════════ +# Fixtures and helpers +# ═══════════════════════════════════════════════════════════════════════════ + + +class InstrumentedTable: + """Synchronous fake of a boto3 DynamoDB Table. + + Records call counts and can inject a `time.sleep` delay to block the + event loop thread on unfixed code, letting us prove whether the caller + yielded to the loop while the boto3 call was in flight. + + Mirrors the tiny subset of the Table API that `SessionRepository` uses: + `get_item`, `update_item`, `put_item`, `delete_item`. + """ + + def __init__( + self, + *, + record: Optional[SessionRecord] = None, + delay_s: float = 0.0, + ) -> None: + self._delay_s = delay_s + self._record = record + self.get_item_calls = 0 + self.update_item_calls = 0 + self.put_item_calls = 0 + self.delete_item_calls = 0 + + def _sleep(self) -> None: + if self._delay_s > 0: + time.sleep(self._delay_s) + + def get_item(self, Key: dict) -> dict: + self.get_item_calls += 1 + self._sleep() + if self._record is None: + return {} + return {"Item": _record_to_item(self._record)} + + def update_item(self, **kwargs: Any) -> dict: + self.update_item_calls += 1 + self._sleep() + return {} + + def put_item(self, Item: dict) -> dict: + self.put_item_calls += 1 + self._sleep() + return {} + + def delete_item(self, Key: dict) -> dict: + self.delete_item_calls += 1 + self._sleep() + return {} + + +def _record_to_item(r: SessionRecord) -> dict: + return { + "PK": f"SESSION#{r.session_id}", + "SK": "META", + "session_id": r.session_id, + "user_id": r.user_id, + "username": r.username, + "cognito_access_token": r.cognito_access_token, + "cognito_refresh_token": r.cognito_refresh_token, + "id_token": r.id_token, + "access_token_exp": r.access_token_exp, + "csrf_secret": r.csrf_secret, + "created_at": r.created_at, + "last_seen_at": r.last_seen_at, + "ttl": r.ttl, + } + + +def _make_repo(table: InstrumentedTable) -> SessionRepository: + """Build a SessionRepository backed by an InstrumentedTable. + + Bypasses boto3.resource() initialization by starting disabled, then + flipping `_enabled` and injecting the fake table. Exercises the real + SessionRepository async-method bodies — which is the point for + sub-condition 1.1 (offload). + """ + repo = SessionRepository(table_name="") + repo._enabled = True + repo._table = table # type: ignore[assignment] + repo._table_name = "test-bff-sessions" + return repo + + +def _make_codec() -> CookieCodec: + codec = CookieCodec(kms_key_arn="arn:aws:kms:fake") + # Pre-inject an AES-GCM cipher so no KMS call is attempted. + codec._cipher = AESGCM(secrets.token_bytes(32)) + return codec + + +def _make_record( + *, + session_id: str = "sess-001", + access_token_exp: Optional[int] = None, + last_seen_at: Optional[int] = None, + created_at: Optional[int] = None, +) -> SessionRecord: + now = int(time.time()) + return SessionRecord( + session_id=session_id, + user_id="user-sub-001", + username="alice", + cognito_access_token="access.original", + cognito_refresh_token="refresh.original", + id_token="id.original", + access_token_exp=access_token_exp if access_token_exp is not None else now + 3600, + csrf_secret="csrf-secret-deadbeef", + created_at=created_at if created_at is not None else now, + last_seen_at=last_seen_at if last_seen_at is not None else now, + ttl=now + 28800, + ) + + +def _enabled_config(**overrides: Any) -> BFFConfig: + defaults: dict[str, Any] = dict( + sessions_table_name="tbl", + cookie_signing_key_arn="arn:aws:kms:fake", + session_ttl_seconds=28800, + refresh_leeway_seconds=_DEFAULT_REFRESH_LEEWAY_SECONDS, + cognito_bff_app_client_id="client-id", + cognito_bff_app_client_secret_arn="arn:secret", + inference_api_url=None, + absolute_lifetime_seconds=30 * 24 * 3600, + sliding_renewal_throttle_seconds=_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS, + ) + defaults.update(overrides) + return BFFConfig(**defaults) + + +def _build_app( + *, + config: BFFConfig, + repository: SessionRepository, + codec: CookieCodec, + refresh_client: Any, + cache: Optional[SessionCache] = None, +) -> FastAPI: + app = FastAPI() + app.add_middleware( + SessionRefreshMiddleware, + config=config, + repository=repository, + cookie_codec=codec, + refresh_client=refresh_client, + cache=cache or SessionCache(ttl_seconds=60), + ) + + @app.get("/echo") + async def echo(request: Request) -> dict: + record = getattr(request.state, "bff_session", None) + return { + "has_session": record is not None, + "session_id": record.session_id if record else None, + } + + return app + + +@pytest.fixture(autouse=True) +def _reset_session_state() -> Any: + """Clear process-wide state between tests so storm/coalescing behavior + stays independent across cases.""" + lock_module._reset_for_tests() + _reset_secret_cache_for_tests() + yield + lock_module._reset_for_tests() + _reset_secret_cache_for_tests() + + +# ═══════════════════════════════════════════════════════════════════════════ +# Sub-condition 1.1 — SessionRepository.* must offload sync boto3 to a threadpool +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "method_name", + ["get", "touch_last_seen", "update_tokens", "put", "delete"], +) +async def test_1_1_session_repository_methods_offload_sync_boto3( + method_name: str, +) -> None: + """(1.1) Repository offload. + + Each SessionRepository async method that wraps boto3 must execute its + boto3 call off the event loop thread. We prove this by running the + method concurrently with a 50ms marker coroutine against a 500ms + slow-stubbed table. + + - Fixed code: marker completes in ~0.05s while repo call is still in flight. + - Unfixed code: sync boto3 freezes the loop for the full 500ms, starving + the marker so it only completes once the method returns. + + Expected Behavior 2.1 (design.md). + """ + record = _make_record(session_id=f"sess-1-1-{method_name}") + table = InstrumentedTable(record=record, delay_s=0.5) + repo = _make_repo(table) + + now = int(time.time()) + if method_name == "get": + op = repo.get(record.session_id) + elif method_name == "touch_last_seen": + op = repo.touch_last_seen(record.session_id, last_seen_at=now) + elif method_name == "update_tokens": + op = repo.update_tokens( + session_id=record.session_id, + access_token="access.rotated", + refresh_token="refresh.rotated", + id_token=None, + access_token_exp=now + 3600, + last_seen_at=now, + ) + elif method_name == "put": + op = repo.put(record) + elif method_name == "delete": + op = repo.delete(record.session_id) + else: + pytest.fail(f"unknown method_name: {method_name}") + + marker_elapsed: dict[str, float] = {} + + async def marker(start: float) -> None: + await asyncio.sleep(0.05) + marker_elapsed["t"] = time.monotonic() - start + + t0 = time.monotonic() + marker_task = asyncio.create_task(marker(t0)) + await op + op_elapsed = time.monotonic() - t0 + await marker_task + + # Sanity: the stubbed boto3 call really took ~500ms. + assert op_elapsed >= 0.4, ( + f"[1.1/{method_name}] Sanity: stubbed {method_name} should take ~500ms, " + f"got {op_elapsed:.3f}s — the InstrumentedTable delay may not be wired." + ) + # Counterexample: on unfixed code, the marker sits behind the frozen loop. + assert "t" in marker_elapsed, ( + f"[1.1/{method_name}] Marker coroutine never completed — " + f"event loop fully frozen by sync boto3." + ) + assert marker_elapsed["t"] < 0.25, ( + f"[1.1/{method_name}] Marker coroutine starved by sync boto3: " + f"marker elapsed={marker_elapsed['t']:.3f}s, " + f"op elapsed={op_elapsed:.3f}s. " + f"SessionRepository.{method_name} must offload its boto3 call via " + "asyncio.to_thread so the event loop continues scheduling other " + "coroutines for the round-trip duration." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Sub-condition 1.2 — CognitoRefreshClient.refresh must offload initiate_auth +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.asyncio +async def test_1_2_cognito_refresh_offloads_sync_initiate_auth() -> None: + """(1.2) Cognito offload. + + CognitoRefreshClient.refresh must execute cognito-idp:initiate_auth + off the event loop thread, including while the per-session + get_session_lock(session_id) is held. We prove this by running + refresh concurrently with: + (a) a 50ms marker coroutine; + (b) an unrelated get_session_lock(other_session_id) acquisition. + + - Fixed code: both complete promptly while refresh is still in flight. + - Unfixed code: the sync initiate_auth freezes the loop, starving + the marker and delaying the unrelated lock acquisition. + + Expected Behavior 2.2 (design.md). + """ + slow_cognito = MagicMock() + + def slow_initiate_auth(**_kwargs: Any) -> dict: + time.sleep(0.5) + return { + "AuthenticationResult": { + "AccessToken": "access.fresh", + "RefreshToken": "refresh.fresh", + "IdToken": "id.fresh", + "ExpiresIn": 3600, + } + } + + slow_cognito.initiate_auth.side_effect = slow_initiate_auth + + slow_secrets = MagicMock() + slow_secrets.get_secret_value.return_value = {"SecretString": "client-secret"} + + client = CognitoRefreshClient( + app_client_id="client-id", + app_client_secret_arn="arn:secret", + cognito_idp_client=slow_cognito, + secrets_manager_client=slow_secrets, + ) + + marker_elapsed: dict[str, float] = {} + lock_elapsed: dict[str, float] = {} + refresh_elapsed: dict[str, float] = {} + + async def call_refresh(start: float) -> None: + result = client.refresh(username="alice", refresh_token="refresh.original") + # Support both the unfixed (sync) and fixed (coroutine) shape. + if asyncio.iscoroutine(result): + result = await result + refresh_elapsed["t"] = time.monotonic() - start + + async def marker(start: float) -> None: + await asyncio.sleep(0.05) + marker_elapsed["t"] = time.monotonic() - start + + async def acquire_other_lock(start: float) -> None: + other_lock = get_session_lock("other-session-id") + async with other_lock: + pass + lock_elapsed["t"] = time.monotonic() - start + + t0 = time.monotonic() + marker_task = asyncio.create_task(marker(t0)) + other_lock_task = asyncio.create_task(acquire_other_lock(t0)) + await call_refresh(t0) + await marker_task + await other_lock_task + + # Sanity: the stubbed initiate_auth really took ~500ms. + assert refresh_elapsed.get("t", 0.0) >= 0.4, ( + f"[1.2] Sanity: stubbed refresh should take ~500ms, " + f"got {refresh_elapsed.get('t', 0.0):.3f}s — stub not wired." + ) + assert "t" in marker_elapsed, ( + "[1.2] Marker coroutine never completed — loop fully frozen." + ) + assert marker_elapsed["t"] < 0.25, ( + f"[1.2] Marker coroutine starved by sync Cognito initiate_auth: " + f"marker elapsed={marker_elapsed['t']:.3f}s, " + f"refresh elapsed={refresh_elapsed['t']:.3f}s. " + "CognitoRefreshClient.refresh must offload initiate_auth via " + "asyncio.to_thread so other coroutines — including those for " + "different session_ids — make progress while the per-session " + "asyncio.Lock is held." + ) + assert lock_elapsed["t"] < 0.25, ( + f"[1.2] Unrelated get_session_lock('other-session-id') acquisition " + f"starved by sync Cognito call: lock elapsed={lock_elapsed['t']:.3f}s, " + f"refresh elapsed={refresh_elapsed['t']:.3f}s. " + "Even uncontended locks for different sessions block when the " + "event loop thread is frozen." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Sub-condition 1.3 — Resolve-path coalescing: N concurrent reqs → 1 get_item +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.asyncio +@pytest.mark.parametrize("fanout", [8]) +async def test_1_3_concurrent_same_session_fanout_coalesces_to_one_get_item( + fanout: int, +) -> None: + """(1.3) Resolve-path coalescing. + + N concurrent SessionRefreshMiddleware.dispatch calls for the same + session_id with a cold SessionCache and a valid sealed cookie must + result in exactly ONE DynamoDB get_item invocation. The upstream + unseal → SessionCache.get → SessionRepository.get path needs + coalescing via a per-session single-flight primitive. + + - Fixed code: 1 get_item (single-flight leader + followers). + - Unfixed code: N get_item calls — the existing get_session_lock only + wraps the Cognito exchange, not the resolve path. + + Expected Behavior 2.3 (design.md). + """ + record = _make_record(session_id="sess-1-3") + # Small delay so concurrent dispatches overlap long enough for each + # to observe cache-miss independently on unfixed code. + table = InstrumentedTable(record=record, delay_s=0.05) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + cache = SessionCache(ttl_seconds=60) # cold → cache miss + app = _build_app( + config=_enabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + cache=cache, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + transport = httpx.ASGITransport(app=app) + + async with httpx.AsyncClient( + transport=transport, base_url="http://test" + ) as client: + client.cookies.set(SESSION_COOKIE_NAME, sealed) + responses = await asyncio.gather( + *(client.get("/echo") for _ in range(fanout)) + ) + + for r in responses: + assert r.status_code == 200 + + assert table.get_item_calls == 1, ( + f"[1.3] Fan-out of {fanout} concurrent same-session requests against " + f"a cold cache must coalesce to exactly one get_item call. " + f"Observed: {table.get_item_calls} get_item calls (bug target: {fanout}). " + "A per-session asyncio.Future single-flight is required upstream of " + "SessionRepository.get." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Sub-condition 1.4 — Cache window and slide throttle must be de-aligned +# ═══════════════════════════════════════════════════════════════════════════ + + +@given( + throttle=st.just(_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS), + leeway=st.just(_DEFAULT_REFRESH_LEEWAY_SECONDS), +) +@settings(max_examples=1, deadline=None, suppress_health_check=[HealthCheck.function_scoped_fixture]) +def test_1_4a_default_throttle_is_strict_multiple_of_leeway( + throttle: int, leeway: int +) -> None: + """(1.4) Window de-alignment — config invariant. + + _DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS must be a strict multiple of + _DEFAULT_REFRESH_LEEWAY_SECONDS AND strictly greater. This de-aligns + cache-expiry (TTL = leeway) from slide-throttle expiry so a single + request crossing one boundary does not also cross the other. + + - Fixed code: throttle=300, leeway=60 → 300 > 60 and 300 % 60 == 0. + - Unfixed code: both default to 60 → 60 > 60 is False. + + Expected Behavior 2.4 (design.md). + """ + assert throttle > leeway, ( + f"[1.4a] Sliding-renewal throttle ({throttle}s) must be strictly " + f"greater than refresh leeway ({leeway}s) to de-align boundaries." + ) + assert throttle % leeway == 0, ( + f"[1.4a] Sliding-renewal throttle ({throttle}s) must be a strict " + f"multiple of refresh leeway ({leeway}s)." + ) + + +@pytest.mark.asyncio +async def test_1_4b_single_request_at_boundary_skips_slide_write() -> None: + """(1.4) Window de-alignment — runtime behavior. + + A single request with SessionCache TTL just elapsed AND + (now - last_seen_at) == refresh_leeway_seconds must issue AT MOST ONE + of {get_item, update_item} on the critical path. On unfixed code the + aligned 60s windows guarantee BOTH writes on the same request (the + cache miss drives get_item AND the past-throttle state drives + update_item). + + Expected Behavior 2.4 (design.md). + """ + now = int(time.time()) + record = _make_record( + session_id="sess-1-4b", + last_seen_at=now - _DEFAULT_REFRESH_LEEWAY_SECONDS, + ) + table = InstrumentedTable(record=record, delay_s=0.01) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + cache = SessionCache(ttl_seconds=60) # cold → cache miss + # Use the real default throttle so the test fails on unfixed code + # (throttle == leeway == 60s) and passes on fixed code (throttle=300s, + # leeway=60s). + app = _build_app( + config=_enabled_config( + sliding_renewal_throttle_seconds=_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS, + ), + repository=repo, + codec=codec, + refresh_client=refresh_client, + cache=cache, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + transport = httpx.ASGITransport(app=app) + + async with httpx.AsyncClient( + transport=transport, base_url="http://test" + ) as client: + client.cookies.set(SESSION_COOKIE_NAME, sealed) + response = await client.get("/echo") + assert response.status_code == 200 + + ddb_calls = table.get_item_calls + table.update_item_calls + assert ddb_calls <= 1, ( + f"[1.4b] Single request at cache/throttle boundary issued " + f"{table.get_item_calls} get_item + {table.update_item_calls} " + f"update_item = {ddb_calls} DDB calls on critical path. " + "Windows must be de-aligned (throttle > leeway, strict multiple) " + "so a cache miss never also triggers a slide write." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Sub-condition 1.5 — _maybe_slide must fire-and-forget the DDB write +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.asyncio +async def test_1_5_slide_write_is_fire_and_forget() -> None: + """(1.5) Fire-and-forget slide. + + When a slide is warranted, the response path must NOT wait on + touch_last_seen. Stubbing update_item with a 500ms delay, the total + dispatch elapsed must stay well under 500ms. + + - Fixed code: _maybe_slide schedules touch_last_seen as an + asyncio.Task and returns synchronously → elapsed ~= handler time. + - Unfixed code: _maybe_slide awaits touch_last_seen inline → + elapsed >= 500ms. + + Expected Behavior 2.5 (design.md). + """ + now = int(time.time()) + record = _make_record( + session_id="sess-1-5", + last_seen_at=now - 3600, # past any reasonable throttle window + ) + table = InstrumentedTable(record=record, delay_s=0.5) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + + # Pre-seed the cache so repo.get is not on the path — this test isolates + # the slide-write-on-response-path question from the coalescing question. + cache = SessionCache(ttl_seconds=60) + cache.set(record) + + # Use a small throttle so the slide is warranted (last_seen == now-3600). + app = _build_app( + config=_enabled_config(sliding_renewal_throttle_seconds=60), + repository=repo, + codec=codec, + refresh_client=refresh_client, + cache=cache, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + transport = httpx.ASGITransport(app=app) + + async with httpx.AsyncClient( + transport=transport, base_url="http://test" + ) as client: + client.cookies.set(SESSION_COOKIE_NAME, sealed) + t0 = time.monotonic() + response = await client.get("/echo") + elapsed = time.monotonic() - t0 + + assert response.status_code == 200 + # Sanity: the slide write was in fact requested (fires exactly once; + # in the fixed scenario it's still counted on the fake table — it just + # doesn't block the response path). + assert table.update_item_calls >= 1, ( + f"[1.5] Sanity: the slide path should have fired update_item at least " + f"once, got {table.update_item_calls}. Check last_seen_at setup." + ) + assert elapsed < 0.25, ( + f"[1.5] Dispatch elapsed={elapsed:.3f}s; the response waited on the " + "500ms stubbed update_item. _maybe_slide must dispatch the DDB write " + "as a detached asyncio.Task so the response returns without blocking." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Sub-condition 1.6 — Production deployment must have concurrency slack +# ═══════════════════════════════════════════════════════════════════════════ + + +def test_1_6_cdk_app_api_desired_count_at_least_two() -> None: + """(1.6) Concurrency slack at deployment. + + infrastructure/cdk.context.json must set appApi.desiredCount >= 2 so + a single blocked event loop on one ECS task cannot stall all ingress. + + Expected Behavior 2.6 (design.md). + """ + cdk_context_path = ( + Path(__file__).resolve().parents[5] / "infrastructure" / "cdk.context.json" + ) + assert cdk_context_path.exists(), ( + f"[1.6] Expected cdk.context.json at {cdk_context_path}" + ) + ctx = json.loads(cdk_context_path.read_text()) + app_api = ctx.get("appApi", {}) + desired = app_api.get("desiredCount") + assert isinstance(desired, int) and desired >= 2, ( + f"[1.6] appApi.desiredCount must be >= 2 in the production context " + f"(found: {desired!r}). Single-task deployment cannot absorb a " + "blocked event loop — a slow AWS call on one task halts every " + "concurrent request." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Sub-condition 1.7 — Fan-out at cache boundary must not amplify to N*2 DDB calls +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.asyncio +@pytest.mark.parametrize("fanout", [8]) +async def test_1_7_fanout_at_boundary_bounded_blocking_ddb_calls( + fanout: int, +) -> None: + """(1.7) Fan-out amplification. + + N concurrent requests for the same session at a cache-boundary moment + must produce AT MOST 2 blocking DDB calls across the entire fan-out + (ideally 1 get_item and 0 slide-writes when windows are de-aligned). + + - Fixed code: single-flight + de-aligned windows → ≤ 1 get_item + + ≤ 1 update_item = ≤ 2. + - Unfixed code: each coroutine observes cache miss + past-throttle + independently on its local SessionRecord copy and issues its own + get_item + update_item → 2*N blocking calls. + + Expected Behavior 2.7 (design.md). + """ + now = int(time.time()) + record = _make_record( + session_id="sess-1-7", + last_seen_at=now - _DEFAULT_REFRESH_LEEWAY_SECONDS, # past aligned throttle on unfixed + ) + table = InstrumentedTable(record=record, delay_s=0.01) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + cache = SessionCache(ttl_seconds=60) # cold → cache miss + app = _build_app( + config=_enabled_config( + sliding_renewal_throttle_seconds=_DEFAULT_SLIDING_RENEWAL_THROTTLE_SECONDS, + ), + repository=repo, + codec=codec, + refresh_client=refresh_client, + cache=cache, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + transport = httpx.ASGITransport(app=app) + + async with httpx.AsyncClient( + transport=transport, base_url="http://test" + ) as client: + client.cookies.set(SESSION_COOKIE_NAME, sealed) + responses = await asyncio.gather( + *(client.get("/echo") for _ in range(fanout)) + ) + + for r in responses: + assert r.status_code == 200 + + blocking_calls = table.get_item_calls + table.update_item_calls + assert blocking_calls <= 2, ( + f"[1.7] Fan-out of {fanout} concurrent same-session requests at a " + f"cache-boundary moment produced {table.get_item_calls} get_item + " + f"{table.update_item_calls} update_item = {blocking_calls} blocking " + f"DDB calls (bug: ~{2 * fanout}). Single-flight coalescing AND " + "window de-alignment are required." + ) diff --git a/backend/tests/apis/shared/middleware/test_session_refresh_preservation.py b/backend/tests/apis/shared/middleware/test_session_refresh_preservation.py new file mode 100644 index 00000000..4f42c8db --- /dev/null +++ b/backend/tests/apis/shared/middleware/test_session_refresh_preservation.py @@ -0,0 +1,1213 @@ +"""Preservation property tests for SessionRefreshMiddleware. + +Property 2: BFF Middleware Contracts Unchanged for Non-Buggy Inputs. + +This file encodes the observable contracts (Preservation Requirements 3.1–3.11) +that the event-loop-blocking fix MUST preserve. Tests are run on UNFIXED code +first and MUST PASS — confirming the baseline behavior to lock in. After the +fix lands (task 3.x series) these same tests must continue to pass with no +modifications. + +Observation-first methodology: each preservation test encodes behavior +OBSERVED on today's code — response status, `Set-Cookie` headers (including +every attribute), `request.state.bff_session`, `request.state.bff_csrf_token`, +DDB call counts, Cognito call counts, KMS/Secrets Manager call counts — rather +than re-derived from the spec. + +The hypothesis strategies cover the axes that exist today: `is_enabled()` +true/false, `__Host-bff_session` cookie present/absent, cookie seal +valid/invalid/expired, `SessionCache` hit/miss, `needs_refresh` yes/no, +refresh-token rotation yes/no, slide warranted yes/no, absolute-lifetime cap +passed yes/no, request method safe/unsafe. Inputs that themselves reproduce +an isBugCondition sub-condition (fan-outs at aligned boundaries, slide timing +vs response timing, etc.) are avoided — preservation is about the externally +observable contract, not about how many DDB calls happen under bug-triggering +inputs. + +Validates: Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, 3.11 +""" + +from __future__ import annotations + +import asyncio +import secrets +import time +from typing import Any, Optional +from unittest.mock import AsyncMock, MagicMock + +import httpx +import pytest +from cryptography.hazmat.primitives.ciphers.aead import AESGCM +from fastapi import FastAPI, Request +from fastapi.testclient import TestClient +from hypothesis import HealthCheck, given, settings +from hypothesis import strategies as st + +from apis.shared.middleware.csrf import CSRFMiddleware +from apis.shared.middleware.session_refresh import SessionRefreshMiddleware +from apis.shared.sessions_bff import cache as cache_module +from apis.shared.sessions_bff import cookie as cookie_module +from apis.shared.sessions_bff import lock as lock_module +from apis.shared.sessions_bff import refresh as refresh_module +from apis.shared.sessions_bff.cache import SessionCache +from apis.shared.sessions_bff.config import ( + BFFConfig, + CSRF_COOKIE_NAME, + CSRF_HEADER_NAME, + SESSION_COOKIE_NAME, + _DEFAULT_REFRESH_LEEWAY_SECONDS, +) +from apis.shared.sessions_bff.cookie import CookieCodec, get_default_codec +from apis.shared.sessions_bff.csrf import CSRFHelper +from apis.shared.sessions_bff.models import CookiePayload, SessionRecord +from apis.shared.sessions_bff.refresh import ( + CognitoRefreshClient, + CognitoRefreshError, + RefreshResult, + _reset_secret_cache_for_tests, + resolve_bff_client_secret, +) +from apis.shared.sessions_bff.repository import SessionRepository + + +# ═══════════════════════════════════════════════════════════════════════════ +# Shared helpers — duplicated from test_session_refresh_bug_condition.py for +# test-file isolation. Keep the two files' helper shapes in sync. +# ═══════════════════════════════════════════════════════════════════════════ + + +class InstrumentedTable: + """Synchronous fake of a boto3 DynamoDB Table. + + Records call counts so preservation tests can assert "zero AWS calls" + for dormant / no-cookie pass-through paths, and "exactly one get_item" + for the refresh-storm coalescing contract. + + `update_item` writes are classified into three kinds by inspecting the + `UpdateExpression`: + - `lock_acquire_calls`: cross-task refresh-lock acquisition (writes + `refresh_lock_owner` + `refresh_lock_until`, no token columns). + - `token_persist_calls`: token rotation write (sets + `cognito_access_token` etc., usually also REMOVE-ing the lock). + - `slide_calls`: sliding-renewal touch (writes only `last_seen_at` + and optionally `ttl`). + `update_item_calls` remains the total (sum) so existing assertions on + "any update_item issued" continue to hold. The injected side-effect is + applied only to the token-persist path so tests that simulate "DDB + throttled during persist" don't accidentally fail at the lock-acquire + write — that's a different code path with different recovery semantics. + """ + + def __init__( + self, + *, + record: Optional[SessionRecord] = None, + delay_s: float = 0.0, + update_item_side_effect: Optional[Exception] = None, + ) -> None: + self._delay_s = delay_s + self._record = record + self._update_item_side_effect = update_item_side_effect + self.get_item_calls = 0 + self.update_item_calls = 0 + self.lock_acquire_calls = 0 + self.token_persist_calls = 0 + self.slide_calls = 0 + self.put_item_calls = 0 + self.delete_item_calls = 0 + + def _sleep(self) -> None: + if self._delay_s > 0: + time.sleep(self._delay_s) + + def get_item(self, Key: dict) -> dict: + self.get_item_calls += 1 + self._sleep() + if self._record is None: + return {} + return {"Item": _record_to_item(self._record)} + + @staticmethod + def _classify_update(update_expr: str) -> str: + """Classify which middleware path issued this update_item. + + Token persist writes always set `cognito_access_token`. Pure lock + acquires write `refresh_lock_owner` without touching tokens. Slide + writes touch only `last_seen_at` (+ optionally `ttl`). + """ + if "cognito_access_token" in update_expr: + return "token_persist" + if "refresh_lock_owner" in update_expr: + return "lock_acquire" + return "slide" + + def update_item(self, **kwargs: Any) -> dict: + self.update_item_calls += 1 + kind = self._classify_update(kwargs.get("UpdateExpression", "")) + if kind == "token_persist": + self.token_persist_calls += 1 + elif kind == "lock_acquire": + self.lock_acquire_calls += 1 + else: + self.slide_calls += 1 + self._sleep() + # Side-effect injection applies only to the token-persist path — + # tests that simulate "rotation persist exhausted" mean exactly + # that write, not the upstream lock-acquire. + if self._update_item_side_effect is not None and kind == "token_persist": + raise self._update_item_side_effect + return {} + + def put_item(self, Item: dict) -> dict: + self.put_item_calls += 1 + self._sleep() + return {} + + def delete_item(self, Key: dict) -> dict: + self.delete_item_calls += 1 + self._sleep() + return {} + + +def _record_to_item(r: SessionRecord) -> dict: + return { + "PK": f"SESSION#{r.session_id}", + "SK": "META", + "session_id": r.session_id, + "user_id": r.user_id, + "username": r.username, + "cognito_access_token": r.cognito_access_token, + "cognito_refresh_token": r.cognito_refresh_token, + "id_token": r.id_token, + "access_token_exp": r.access_token_exp, + "csrf_secret": r.csrf_secret, + "created_at": r.created_at, + "last_seen_at": r.last_seen_at, + "ttl": r.ttl, + } + + +def _make_repo(table: InstrumentedTable) -> SessionRepository: + """SessionRepository backed by an InstrumentedTable. + + Bypasses boto3.resource() by starting disabled, then flipping `_enabled` + and injecting the fake table. Exercises the real repository async-method + bodies so preservation tests see the production code path. + """ + repo = SessionRepository(table_name="") + repo._enabled = True + repo._table = table # type: ignore[assignment] + repo._table_name = "test-bff-sessions" + return repo + + +def _make_codec() -> CookieCodec: + codec = CookieCodec(kms_key_arn="arn:aws:kms:fake") + codec._cipher = AESGCM(secrets.token_bytes(32)) + return codec + + +def _make_record( + *, + session_id: str = "sess-pres-001", + access_token_exp: Optional[int] = None, + last_seen_at: Optional[int] = None, + created_at: Optional[int] = None, + ttl: Optional[int] = None, +) -> SessionRecord: + now = int(time.time()) + return SessionRecord( + session_id=session_id, + user_id="user-sub-001", + username="alice", + cognito_access_token="access.original", + cognito_refresh_token="refresh.original", + id_token="id.original", + access_token_exp=access_token_exp if access_token_exp is not None else now + 3600, + csrf_secret="csrf-secret-deadbeef", + created_at=created_at if created_at is not None else now, + last_seen_at=last_seen_at if last_seen_at is not None else now, + ttl=ttl if ttl is not None else now + 28800, + ) + + +def _enabled_config(**overrides: Any) -> BFFConfig: + defaults: dict[str, Any] = dict( + sessions_table_name="tbl", + cookie_signing_key_arn="arn:aws:kms:fake", + session_ttl_seconds=28800, + refresh_leeway_seconds=_DEFAULT_REFRESH_LEEWAY_SECONDS, + cognito_bff_app_client_id="client-id", + cognito_bff_app_client_secret_arn="arn:secret", + inference_api_url=None, + absolute_lifetime_seconds=30 * 24 * 3600, + sliding_renewal_throttle_seconds=60, + ) + defaults.update(overrides) + return BFFConfig(**defaults) + + +def _disabled_config() -> BFFConfig: + return BFFConfig( + sessions_table_name=None, + cookie_signing_key_arn=None, + session_ttl_seconds=28800, + refresh_leeway_seconds=60, + cognito_bff_app_client_id=None, + cognito_bff_app_client_secret_arn=None, + inference_api_url=None, + ) + + +def _build_app( + *, + config: BFFConfig, + repository: Any, + codec: CookieCodec, + refresh_client: Any, + cache: Optional[SessionCache] = None, + include_csrf: bool = False, +) -> FastAPI: + app = FastAPI() + if include_csrf: + # Added first → innermost relative to SessionRefreshMiddleware. + # Request order: SessionRefresh → CSRF → route. + app.add_middleware(CSRFMiddleware) + app.add_middleware( + SessionRefreshMiddleware, + config=config, + repository=repository, + cookie_codec=codec, + refresh_client=refresh_client, + cache=cache or SessionCache(ttl_seconds=60), + ) + + @app.get("/echo") + async def echo_get(request: Request) -> dict: + record = getattr(request.state, "bff_session", None) + csrf = getattr(request.state, "bff_csrf_token", None) + return { + "has_session": record is not None, + "session_id": record.session_id if record else None, + "access_token": record.cognito_access_token if record else None, + "csrf_token": csrf, + } + + @app.post("/submit") + async def submit_post(request: Request) -> dict: + record = getattr(request.state, "bff_session", None) + return { + "has_session": record is not None, + "session_id": record.session_id if record else None, + } + + return app + + +@pytest.fixture(autouse=True) +def _reset_session_state() -> Any: + """Clear process-wide state between tests.""" + lock_module._reset_for_tests() + _reset_secret_cache_for_tests() + cache_module._reset_default_cache_for_tests() + cookie_module._reset_default_codec_for_tests() + yield + lock_module._reset_for_tests() + _reset_secret_cache_for_tests() + cache_module._reset_default_cache_for_tests() + cookie_module._reset_default_codec_for_tests() + + +# ═══════════════════════════════════════════════════════════════════════════ +# Set-Cookie parsing helpers — the preservation contract on cookie attributes +# is observed from the raw `Set-Cookie` header, so we parse it here. +# ═══════════════════════════════════════════════════════════════════════════ + + +def _parse_set_cookie(header: str) -> dict[str, Any]: + """Parse a raw Set-Cookie header into {name, value, attributes}. + + Attributes are keyed case-folded for reliable membership checks. + Boolean attributes (HttpOnly, Secure) map to True. + """ + parts = [p.strip() for p in header.split(";")] + name, _, value = parts[0].partition("=") + attrs: dict[str, Any] = {} + for attr in parts[1:]: + if "=" in attr: + k, _, v = attr.partition("=") + attrs[k.strip().lower()] = v.strip() + else: + attrs[attr.strip().lower()] = True + return {"name": name.strip(), "value": value.strip(), "attrs": attrs} + + +def _find_set_cookies( + response_headers: Any, cookie_name: str +) -> list[dict[str, Any]]: + """Return every parsed Set-Cookie for a given cookie name.""" + parsed = [] + for header in response_headers.get_list("set-cookie"): + pc = _parse_set_cookie(header) + if pc["name"] == cookie_name: + parsed.append(pc) + return parsed + + +def _wait_for(predicate: Any, *, timeout_s: float = 1.0, interval_s: float = 0.01) -> bool: + """Poll ``predicate`` until it returns truthy or ``timeout_s`` elapses. + + The slide-write path became fire-and-forget in task 3.5 — `_maybe_slide` + schedules the DDB `touch_last_seen` on a detached `asyncio.create_task` + and returns the Max-Age synchronously. `TestClient` returns the response + before the scheduled task has a chance to run on slower CI schedulers, + so assertions about `update_item_calls == 1` must poll rather than + sample immediately. The observable external contract (cookie attributes, + Max-Age, response body) is unchanged — only the internal timing of the + background write moves. + """ + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(interval_s) + return predicate() + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.1 — Dormant pass-through with zero AWS calls +# ═══════════════════════════════════════════════════════════════════════════ + + +# Cookie-safe ASCII: printable, no semicolons/commas/whitespace/control chars — +# httpx's cookiejar only accepts ASCII values and rejects the RFC 6265 separators. +_COOKIE_SAFE_ALPHABET = st.characters( + min_codepoint=0x21, + max_codepoint=0x7E, + blacklist_characters=";, \t\"\\", +) + + +@given( + method=st.sampled_from(["GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"]), + path=st.sampled_from(["/echo", "/submit"]), + with_cookie=st.booleans(), + cookie_value=st.text(alphabet=_COOKIE_SAFE_ALPHABET, min_size=0, max_size=64), +) +@settings( + max_examples=30, + deadline=None, + suppress_health_check=[HealthCheck.function_scoped_fixture], +) +def test_3_1_dormant_passthrough_zero_aws_calls( + method: str, path: str, with_cookie: bool, cookie_value: str +) -> None: + """(3.1) Dormant pass-through. + + When `BFFConfig.is_enabled() == False`, every request shape (method, + path, cookie present/absent) short-circuits through `call_next(request)` + with zero DDB calls and zero Cognito calls. + """ + table = InstrumentedTable() + repo = _make_repo(table) + # Force the repo into the "enabled" posture so we'd observe a call if + # the middleware mistakenly went past its `is_enabled()` guard. + codec = _make_codec() + refresh_client = MagicMock() + app = _build_app( + config=_disabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + ) + + cookies: dict[str, str] = {} + if with_cookie: + cookies[SESSION_COOKIE_NAME] = cookie_value + + with TestClient(app) as client: + response = client.request(method, path, cookies=cookies) + + # OPTIONS/HEAD may be allowed or not depending on route — we only care + # that the middleware did not touch AWS regardless of status. + assert response.status_code < 500, ( + f"[3.1] dormant pass-through produced 5xx for {method} {path}: " + f"{response.status_code}" + ) + assert table.get_item_calls == 0, ( + f"[3.1] dormant middleware issued {table.get_item_calls} get_item " + f"calls — must be zero when is_enabled() == False" + ) + assert table.update_item_calls == 0, ( + f"[3.1] dormant middleware issued {table.update_item_calls} " + "update_item calls — must be zero when is_enabled() == False" + ) + assert table.put_item_calls == 0 + assert table.delete_item_calls == 0 + refresh_client.refresh.assert_not_called() + # No Set-Cookie emitted by the middleware when dormant. + assert response.headers.get_list("set-cookie") == [] + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.2 — No-cookie pass-through with zero AWS calls +# ═══════════════════════════════════════════════════════════════════════════ + + +@given( + method=st.sampled_from(["GET", "POST", "PUT", "PATCH", "DELETE"]), + path=st.sampled_from(["/echo", "/submit"]), +) +@settings( + max_examples=20, + deadline=None, + suppress_health_check=[HealthCheck.function_scoped_fixture], +) +def test_3_2_no_cookie_passthrough_zero_aws_calls( + method: str, path: str +) -> None: + """(3.2) No-cookie pass-through. + + When `is_enabled() == True` but no `__Host-bff_session` cookie is present + (Bearer-token requests, anonymous endpoints), the middleware must pass + through with zero AWS calls and no `request.state.bff_session`. + """ + table = InstrumentedTable() + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + app = _build_app( + config=_enabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + ) + + with TestClient(app) as client: + response = client.request(method, path) + + assert response.status_code < 500 + # When the call returned 200 with body, the handler reports has_session=False. + if response.status_code == 200 and response.headers.get( + "content-type", "" + ).startswith("application/json"): + body = response.json() + assert body["has_session"] is False, ( + "[3.2] state.bff_session must NOT be set when no cookie is present" + ) + assert table.get_item_calls == 0, ( + f"[3.2] no-cookie path issued {table.get_item_calls} get_item calls" + ) + assert table.update_item_calls == 0 + assert table.put_item_calls == 0 + assert table.delete_item_calls == 0 + refresh_client.refresh.assert_not_called() + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.3 — Unrecoverable cookie clears BOTH cookies with matching attrs +# ═══════════════════════════════════════════════════════════════════════════ + + +def _assert_clear_cookie_attrs(parsed: dict[str, Any]) -> None: + """Attributes observed today on a cleared BFF cookie: + + Max-Age=0; Path=/; SameSite=lax; Secure + + HttpOnly is present on the session cookie only (intentional: the CSRF + cookie is JS-readable). All other attributes are identical across both + cookies. + """ + attrs = parsed["attrs"] + assert attrs.get("max-age") == "0", ( + f"[3.3] clear must set Max-Age=0; got attrs={attrs}" + ) + assert attrs.get("path") == "/", ( + f"[3.3] clear must set Path=/; got attrs={attrs}" + ) + assert attrs.get("samesite") == "lax", ( + f"[3.3] clear must set SameSite=lax; got attrs={attrs}" + ) + assert attrs.get("secure") is True, ( + f"[3.3] clear must set Secure; got attrs={attrs}" + ) + + +@pytest.mark.parametrize( + "scenario", + ["bad_seal", "missing_row", "expired_row", "terminal_refresh_error"], +) +def test_3_3_unrecoverable_cookie_clears_both_cookies_with_matching_attrs( + scenario: str, +) -> None: + """(3.3) Unrecoverable cookie → clear both. + + Bad-seal, missing-row, expired-row, and terminal-`CognitoRefreshError` + inputs all produce Set-Cookie for both `__Host-bff_session` and + `__Host-bff_csrf` with `Max-Age=0` and the today-observed attribute set. + The HttpOnly attribute intentionally differs between the two (session + is HttpOnly; CSRF is JS-readable by design); all other attrs match. + """ + codec = _make_codec() + refresh_client = MagicMock() + + if scenario == "bad_seal": + table = InstrumentedTable() + cookie_value = "not-a-sealed-cookie" + elif scenario == "missing_row": + # No record on the table — get_item returns {} → record None. + table = InstrumentedTable(record=None) + cookie_value = codec.seal(CookiePayload(session_id="sess-gone")) + elif scenario == "expired_row": + # TTL in the past — repository treats as missing (defense in depth). + expired = _make_record(ttl=int(time.time()) - 10) + table = InstrumentedTable(record=expired) + cookie_value = codec.seal(CookiePayload(session_id=expired.session_id)) + elif scenario == "terminal_refresh_error": + # Access token within leeway → refresh path → Cognito raises. + rec = _make_record(access_token_exp=int(time.time()) + 5) + table = InstrumentedTable(record=rec) + cookie_value = codec.seal(CookiePayload(session_id=rec.session_id)) + refresh_client.refresh.side_effect = CognitoRefreshError("rotated-dead") + else: + pytest.fail(f"unknown scenario: {scenario}") + + repo = _make_repo(table) + app = _build_app( + config=_enabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + ) + + with TestClient(app) as client: + response = client.get("/echo", cookies={SESSION_COOKIE_NAME: cookie_value}) + + assert response.status_code == 200 + assert response.json()["has_session"] is False, ( + f"[3.3/{scenario}] state.bff_session must NOT be set after clear" + ) + + session_clears = _find_set_cookies(response.headers, SESSION_COOKIE_NAME) + csrf_clears = _find_set_cookies(response.headers, CSRF_COOKIE_NAME) + assert len(session_clears) == 1, ( + f"[3.3/{scenario}] expected exactly one Set-Cookie for " + f"{SESSION_COOKIE_NAME}; got {len(session_clears)}" + ) + assert len(csrf_clears) == 1, ( + f"[3.3/{scenario}] expected exactly one Set-Cookie for " + f"{CSRF_COOKIE_NAME}; got {len(csrf_clears)}" + ) + + # Each cleared cookie carries Max-Age=0 and the shared attribute set. + _assert_clear_cookie_attrs(session_clears[0]) + _assert_clear_cookie_attrs(csrf_clears[0]) + + # HttpOnly is the one documented difference between the two cookies. + assert session_clears[0]["attrs"].get("httponly") is True, ( + f"[3.3/{scenario}] session cookie must remain HttpOnly on clear" + ) + assert csrf_clears[0]["attrs"].get("httponly") is not True, ( + f"[3.3/{scenario}] CSRF cookie must NOT be HttpOnly (JS must read it)" + ) + + # Shared (non-HttpOnly) attribute set is identical across the two clears. + shared_keys = {"max-age", "path", "samesite", "secure"} + sess_shared = {k: session_clears[0]["attrs"].get(k) for k in shared_keys} + csrf_shared = {k: csrf_clears[0]["attrs"].get(k) for k in shared_keys} + assert sess_shared == csrf_shared, ( + f"[3.3/{scenario}] shared clear attrs diverge: " + f"session={sess_shared}, csrf={csrf_shared}" + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.4 — Max-Age re-emit contract (slide path) +# ═══════════════════════════════════════════════════════════════════════════ + + +@given( + # Session TTL bounded so it always fits well within the absolute cap. + session_ttl=st.integers(min_value=120, max_value=28800), + # Time since the last touch — past the throttle so a slide is warranted. + seconds_since_last_seen=st.integers(min_value=61, max_value=3600), +) +@settings( + max_examples=15, + deadline=None, + suppress_health_check=[HealthCheck.function_scoped_fixture], +) +def test_3_4_slide_max_age_matches_on_both_cookies( + session_ttl: int, seconds_since_last_seen: int +) -> None: + """(3.4) Max-Age re-emit contract. + + When `_maybe_slide` returns a non-None Max-Age, the Set-Cookie headers + for BOTH `__Host-bff_session` and `__Host-bff_csrf` carry that exact + Max-Age and the attribute set observed today on `_reemit_cookies`: + + Session: HttpOnly; Max-Age=; Path=/; SameSite=lax; Secure + CSRF: Max-Age=; Path=/; SameSite=lax; Secure + """ + now = int(time.time()) + record = _make_record(last_seen_at=now - seconds_since_last_seen) + table = InstrumentedTable(record=record) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + # Large absolute lifetime so the slide is not capped — the Max-Age we + # get back must equal session_ttl_seconds exactly. + app = _build_app( + config=_enabled_config( + session_ttl_seconds=session_ttl, + absolute_lifetime_seconds=30 * 24 * 3600, + sliding_renewal_throttle_seconds=60, + ), + repository=repo, + codec=codec, + refresh_client=refresh_client, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + with TestClient(app) as client: + response = client.get("/echo", cookies={SESSION_COOKIE_NAME: sealed}) + # Slide-write is fire-and-forget (task 3.5) — drive the event + # loop with a second request to let the background task from the + # first request flush. MUST happen inside the `with TestClient` + # block because TestClient tears down its anyio portal (and the + # event loop) on `__exit__`, which cancels any pending tasks. + _wait_for(lambda: table.update_item_calls >= 1) + if table.update_item_calls == 0: + # A no-op second request keeps the event loop alive long + # enough for the pending slide task to run. + client.get("/echo") + _wait_for(lambda: table.update_item_calls >= 1) + + assert response.status_code == 200 + # Slide must have fired exactly once (one DDB update_item). + assert table.update_item_calls == 1, ( + f"[3.4] slide must issue exactly one update_item; got " + f"{table.update_item_calls}" + ) + + session_emits = _find_set_cookies(response.headers, SESSION_COOKIE_NAME) + csrf_emits = _find_set_cookies(response.headers, CSRF_COOKIE_NAME) + assert len(session_emits) == 1, ( + f"[3.4] expected exactly one Set-Cookie for {SESSION_COOKIE_NAME}" + ) + assert len(csrf_emits) == 1, ( + f"[3.4] expected exactly one Set-Cookie for {CSRF_COOKIE_NAME}" + ) + + sess_attrs = session_emits[0]["attrs"] + csrf_attrs = csrf_emits[0]["attrs"] + + # Max-Age equals session_ttl_seconds on BOTH cookies (no absolute cap). + assert sess_attrs.get("max-age") == str(session_ttl), ( + f"[3.4] session cookie Max-Age mismatch: expected {session_ttl}, " + f"got {sess_attrs.get('max-age')}" + ) + assert csrf_attrs.get("max-age") == str(session_ttl), ( + f"[3.4] csrf cookie Max-Age mismatch: expected {session_ttl}, " + f"got {csrf_attrs.get('max-age')}" + ) + + # Attribute set observed on today's _reemit_cookies: + assert sess_attrs.get("path") == "/" + assert sess_attrs.get("samesite") == "lax" + assert sess_attrs.get("secure") is True + assert sess_attrs.get("httponly") is True + + assert csrf_attrs.get("path") == "/" + assert csrf_attrs.get("samesite") == "lax" + assert csrf_attrs.get("secure") is True + # CSRF is JS-readable → MUST NOT be HttpOnly. + assert csrf_attrs.get("httponly") is not True + + # Shared (non-HttpOnly) attribute set is identical. + shared = {"max-age", "path", "samesite", "secure"} + assert {k: sess_attrs.get(k) for k in shared} == { + k: csrf_attrs.get(k) for k in shared + } + + # The sealed value on the session cookie is the exact same value the + # browser already held — slide doesn't mint a new seal. + assert session_emits[0]["value"] == sealed, ( + "[3.4] slide must re-emit the same sealed session value, not a new seal" + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.5 — Refresh-storm coalescing preserved (one initiate_auth per +# session per leeway window) +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.asyncio +async def test_3_5_refresh_storm_coalesces_to_single_initiate_auth() -> None: + """(3.5) Refresh-storm coalescing. + + 10 concurrent same-session requests crossing the refresh-leeway window + must drive exactly ONE `cognito-idp:initiate_auth` call (the existing + per-session lock coalescing contract). The fix MUST preserve this. + """ + now = int(time.time()) + record = _make_record(access_token_exp=now + 5) # within 60s leeway + table = InstrumentedTable(record=record) + repo = _make_repo(table) + codec = _make_codec() + + refresh_call_count = {"n": 0} + + async def _refresh(*, username: str, refresh_token: str) -> RefreshResult: + refresh_call_count["n"] += 1 + return RefreshResult( + access_token=f"access.fresh.{refresh_call_count['n']}", + refresh_token="refresh.original", # no rotation + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) + + refresh_client = MagicMock() + refresh_client.refresh = AsyncMock(side_effect=_refresh) + + # After the first refresh lands, later repo.get calls should observe + # a record that no longer needs refresh (the update_item write is a + # no-op on the fake, so we pre-refresh the in-memory record copy). + fresh = _make_record( + session_id=record.session_id, access_token_exp=now + 3600 + ) + fresh.cognito_access_token = "access.fresh.1" + # Sequential responses: first few see the stale record, then the fresh one. + table._record = record # starts stale + original_get_item = table.get_item + + get_item_counter = {"n": 0} + + def counting_get_item(Key: dict) -> dict: + get_item_counter["n"] += 1 + # After the leader's update_item bumps tokens, followers arriving + # late should see the fresh record. Flip after 2 calls so both + # pre-lock and post-lock rechecks on the leader path see the stale row. + if get_item_counter["n"] > 2: + table._record = fresh + return original_get_item(Key) + + table.get_item = counting_get_item # type: ignore[assignment] + + app = _build_app( + config=_enabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + transport = httpx.ASGITransport(app=app) + + async with httpx.AsyncClient( + transport=transport, base_url="http://test" + ) as client: + client.cookies.set(SESSION_COOKIE_NAME, sealed) + responses = await asyncio.gather( + *(client.get("/echo") for _ in range(10)) + ) + + for r in responses: + assert r.status_code == 200 + + assert refresh_call_count["n"] == 1, ( + f"[3.5] 10 concurrent same-session requests drove " + f"{refresh_call_count['n']} Cognito initiate_auth calls — exactly " + "one is required per session per leeway window (existing " + "get_session_lock coalescing)." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.6 — Codec singleton, zero per-request KMS GenerateDataKey +# ═══════════════════════════════════════════════════════════════════════════ + + +def test_3_6_get_default_codec_is_singleton_with_no_per_request_kms() -> None: + """(3.6) Codec singleton. + + `get_default_codec()` returns the same instance across calls. The + underlying `secretsmanager:GetSecretValue` call happens at most once + per process. Hot seal/unseal traffic must not re-fetch. + + (This contract held under the original `kms:GenerateDataKey`-per-process + design and the interim KMS-wrap design too; only the underlying AWS + APIs and KDF changed when the codec was moved to a shared + Secrets-Manager-generated secret for cross-task seal/unseal.) + """ + sm_client = MagicMock() + sm_client.get_secret_value.return_value = { + "SecretString": "secret-3-6-high-entropy-1234567890ABCDEFGHIJ" + } + + codec = CookieCodec( + kms_key_arn="arn:aws:kms:fake-3.6", + data_key_secret_arn="arn:aws:secretsmanager:fake-3.6", + secrets_manager_client=sm_client, + ) + cookie_module._set_default_codec_for_tests(codec) + + first = get_default_codec() + for _ in range(25): + other = get_default_codec() + assert other is first, ( + "[3.6] get_default_codec() must return the same instance each call" + ) + + payload = CookiePayload(session_id="sess-3-6") + for _ in range(20): + sealed = first.seal(payload) + roundtripped = first.unseal(sealed) + assert roundtripped.session_id == "sess-3-6" + + assert sm_client.get_secret_value.call_count <= 1, ( + f"[3.6] Secrets Manager get_secret_value invoked " + f"{sm_client.get_secret_value.call_count} times — must be at most " + "one per process." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.7 — Client-secret cache, one Secrets Manager hit per process +# ═══════════════════════════════════════════════════════════════════════════ + + +def test_3_7_client_secret_cache_one_secrets_manager_hit_per_process() -> None: + """(3.7) Client-secret cache. + + `resolve_bff_client_secret()` must hit Secrets Manager exactly once per + process regardless of how many times it is called. + """ + sm_client = MagicMock() + sm_client.get_secret_value.return_value = {"SecretString": "client-secret-A"} + + first = resolve_bff_client_secret( + secret_arn="arn:secret", + region="us-east-1", + secrets_manager_client=sm_client, + ) + assert first == "client-secret-A" + + # Many subsequent calls — even with a fresh SM client — must not drive + # a new GetSecretValue, because the first call populated the cache. + for _ in range(50): + value = resolve_bff_client_secret( + secret_arn="arn:secret", + region="us-east-1", + secrets_manager_client=sm_client, + ) + assert value == "client-secret-A" + + assert sm_client.get_secret_value.call_count == 1, ( + f"[3.7] Secrets Manager get_secret_value called " + f"{sm_client.get_secret_value.call_count} times — must be exactly one." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.8 — CSRFMiddleware accept/reject unchanged, no new I/O +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.parametrize( + "case", + ["matching", "mismatched", "header_only", "cookie_only", "forged_pair", "missing"], +) +def test_3_8_csrf_decision_unchanged_with_zero_new_io(case: str) -> None: + """(3.8) CSRF path unchanged. + + With `SessionRefreshMiddleware` upstream populating `state.bff_session`, + the `CSRFMiddleware` accept/reject decision on unsafe-method requests + matches today's observed behavior across all five CSRF token cases. + No new DDB / Cognito / KMS / Secrets Manager I/O is introduced on the + CSRF path. + """ + record = _make_record() + table = InstrumentedTable(record=record) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + app = _build_app( + config=_enabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + include_csrf=True, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + valid_token = CSRFHelper.derive_token(record.csrf_secret, record.session_id) + forged_token = "0" * 32 + + headers: dict[str, str] = {} + cookies: dict[str, str] = {SESSION_COOKIE_NAME: sealed} + + if case == "matching": + headers[CSRF_HEADER_NAME] = valid_token + cookies[CSRF_COOKIE_NAME] = valid_token + expected_status = 200 + elif case == "mismatched": + headers[CSRF_HEADER_NAME] = valid_token + cookies[CSRF_COOKIE_NAME] = "different-value" + expected_status = 403 + elif case == "header_only": + headers[CSRF_HEADER_NAME] = valid_token + expected_status = 403 + elif case == "cookie_only": + cookies[CSRF_COOKIE_NAME] = valid_token + expected_status = 403 + elif case == "forged_pair": + headers[CSRF_HEADER_NAME] = forged_token + cookies[CSRF_COOKIE_NAME] = forged_token + expected_status = 403 + elif case == "missing": + expected_status = 403 + else: + pytest.fail(f"unknown case: {case}") + + # Snapshot AWS call counters BEFORE the CSRF-exercising request. + # (Session resolve may have happened on-open via middleware init; we + # expect exactly one get_item for the resolve, and zero writes.) + initial_refresh_calls = refresh_client.refresh.call_count + initial_update_calls = table.update_item_calls + + with TestClient(app) as client: + response = client.post("/submit", headers=headers, cookies=cookies) + + assert response.status_code == expected_status, ( + f"[3.8/{case}] unexpected CSRF decision: expected {expected_status}, " + f"got {response.status_code}" + ) + # Zero NEW Cognito / DDB write I/O on the CSRF path itself. + assert refresh_client.refresh.call_count == initial_refresh_calls, ( + f"[3.8/{case}] CSRF path triggered an unexpected Cognito refresh" + ) + # CSRF itself never writes to DDB. + assert table.update_item_calls - initial_update_calls <= 1, ( + f"[3.8/{case}] more than one update_item observed — at most the " + "preceding session-resolve slide is expected." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.9 — Absolute-lifetime cap returns None from _maybe_slide +# ═══════════════════════════════════════════════════════════════════════════ + + +@pytest.mark.asyncio +async def test_3_9_maybe_slide_returns_none_past_absolute_cap() -> None: + """(3.9) Absolute-lifetime cap. + + When `now > created_at + absolute_lifetime_seconds`, `_maybe_slide` + returns `None` (no cookie re-emit, no DDB write). + """ + now = int(time.time()) + # Session was created 200s ago with an absolute lifetime of 100s → cap + # was reached 100s ago. last_seen_at is past the throttle so otherwise + # a slide would be warranted. + record = _make_record( + created_at=now - 200, + last_seen_at=now - 120, + ) + table = InstrumentedTable(record=record) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + config = _enabled_config( + absolute_lifetime_seconds=100, + sliding_renewal_throttle_seconds=60, + ) + + # Build the middleware directly so we can invoke _maybe_slide in + # isolation — the preservation contract is specifically that the + # method returns None past the cap. + middleware = SessionRefreshMiddleware( + app=FastAPI(), + config=config, + repository=repo, + cookie_codec=codec, + refresh_client=refresh_client, + cache=SessionCache(ttl_seconds=60), + ) + middleware._ensure_collaborators() + + result = await middleware._maybe_slide(record) + assert result is None, ( + f"[3.9] _maybe_slide must return None past the absolute cap; " + f"got {result!r}" + ) + assert table.update_item_calls == 0, ( + f"[3.9] _maybe_slide must NOT schedule a DDB write past the cap; " + f"observed {table.update_item_calls} update_item calls." + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.10 — Fail-closed rotation: cache invalidated AND cookies cleared +# ═══════════════════════════════════════════════════════════════════════════ + + +def test_3_10_rotation_persist_exhausts_invalidates_cache_and_clears_cookies() -> None: + """(3.10) Fail-closed rotation. + + When refresh-token rotation kicks in AND `_persist_refresh` exhausts all + retries (update_item fails every time), the middleware MUST: + (a) invalidate the cache entry for this session + (b) clear BOTH BFF cookies on the response + so the user is forced to re-authenticate before their next request + hits a dead refresh token. + """ + now = int(time.time()) + # Access token within leeway → refresh path. + record = _make_record(access_token_exp=now + 5) + table = InstrumentedTable( + record=record, + update_item_side_effect=RuntimeError("DDB throttled"), + ) + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + # Rotation kicks in — refresh_token differs from current. + refresh_client.refresh = AsyncMock( + return_value=RefreshResult( + access_token="access.fresh", + refresh_token="refresh.ROTATED", + id_token="id.fresh", + access_token_exp=now + 3600, + ) + ) + + cache = SessionCache(ttl_seconds=60) + # Pre-seed the cache so we can verify invalidation. + cache.set(record) + assert cache.get(record.session_id) is not None + + app = _build_app( + config=_enabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + cache=cache, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + with TestClient(app) as client: + response = client.get("/echo", cookies={SESSION_COOKIE_NAME: sealed}) + + assert response.status_code == 200 + assert response.json()["has_session"] is False, ( + "[3.10] state.bff_session must NOT be set after fail-closed rotation" + ) + + # (a) Cache entry invalidated. + assert cache.get(record.session_id) is None, ( + "[3.10] cache entry must be invalidated after exhausted rotation persist" + ) + + # (b) Both cookies cleared. + session_clears = _find_set_cookies(response.headers, SESSION_COOKIE_NAME) + csrf_clears = _find_set_cookies(response.headers, CSRF_COOKIE_NAME) + assert len(session_clears) == 1 and len(csrf_clears) == 1, ( + f"[3.10] both BFF cookies must be cleared; got " + f"session={len(session_clears)}, csrf={len(csrf_clears)}" + ) + _assert_clear_cookie_attrs(session_clears[0]) + _assert_clear_cookie_attrs(csrf_clears[0]) + + # Sanity: update_tokens was retried 3 times on rotation. Use the + # token_persist sub-counter so we measure persist attempts only, + # not the (also-incrementing) lock_acquire write that precedes them. + assert table.token_persist_calls == 3, ( + f"[3.10] rotation must retry update_tokens 3 times; got " + f"{table.token_persist_calls}" + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Requirement 3.11 — Cookie decode uniformity (no new timing/shape oracle) +# ═══════════════════════════════════════════════════════════════════════════ + + +@given( + garbage=st.one_of( + # Arbitrary non-empty ASCII cookie-safe strings — typical "bad seal" + # wire shape. Excludes '' because an empty cookie value is treated + # as "no cookie present" by the middleware (requirement 3.2), not + # as a decode failure. + st.text(alphabet=_COOKIE_SAFE_ALPHABET, min_size=1, max_size=64), + # Hex-encoded random bytes — invalid base64url alphabet and length. + st.binary(min_size=1, max_size=48).map(lambda b: b.hex()), + ), +) +@settings( + max_examples=25, + deadline=None, + suppress_health_check=[HealthCheck.function_scoped_fixture], +) +def test_3_11_cookie_decode_failure_produces_uniform_response_shape( + garbage: str, +) -> None: + """(3.11) Cookie decode uniformity. + + Every `CookieDecodeError` branch — bad base64, bad tag, truncated blob, + wrong version, non-JSON body — produces the SAME externally observable + response shape: identical status, identical Set-Cookie clearing pattern + for both BFF cookies, identical handler body (has_session=False). + + The middleware must NOT surface any oracle that lets a caller + distinguish decode failure modes. + """ + table = InstrumentedTable() + repo = _make_repo(table) + codec = _make_codec() + refresh_client = MagicMock() + app = _build_app( + config=_enabled_config(), + repository=repo, + codec=codec, + refresh_client=refresh_client, + ) + + with TestClient(app) as client: + response = client.get( + "/echo", cookies={SESSION_COOKIE_NAME: garbage} + ) + + assert response.status_code == 200, ( + f"[3.11] bad-seal path must return 200 with cleared cookie; " + f"got {response.status_code}" + ) + assert response.json() == { + "has_session": False, + "session_id": None, + "access_token": None, + "csrf_token": None, + }, ( + f"[3.11] handler body diverges for garbage cookie {garbage!r}: " + f"{response.json()}" + ) + + # Both cookies cleared with the same attribute set. + session_clears = _find_set_cookies(response.headers, SESSION_COOKIE_NAME) + csrf_clears = _find_set_cookies(response.headers, CSRF_COOKIE_NAME) + assert len(session_clears) == 1, ( + f"[3.11] expected one session-cookie clear; got {len(session_clears)}" + ) + assert len(csrf_clears) == 1, ( + f"[3.11] expected one csrf-cookie clear; got {len(csrf_clears)}" + ) + _assert_clear_cookie_attrs(session_clears[0]) + _assert_clear_cookie_attrs(csrf_clears[0]) + + # Zero AWS calls — decode failure is caught before any DDB / Cognito I/O. + assert table.get_item_calls == 0, ( + f"[3.11] bad-seal path must NOT reach DDB; observed " + f"{table.get_item_calls} get_item calls." + ) + refresh_client.refresh.assert_not_called() diff --git a/backend/tests/apis/shared/sessions_bff/test_cookie.py b/backend/tests/apis/shared/sessions_bff/test_cookie.py index afeaf61a..49f4d5bc 100644 --- a/backend/tests/apis/shared/sessions_bff/test_cookie.py +++ b/backend/tests/apis/shared/sessions_bff/test_cookie.py @@ -1,21 +1,33 @@ """Tests for the AES-GCM cookie codec. -Uses an injected `AESGCM` cipher to avoid mocking KMS — `CookieCodec` exposes -the `_cipher` attribute which we set directly. (Production callers always go -through `_ensure_cipher`, which is what the KMS-integration test exercises.) +Two layers of coverage: + + 1. Round-trip / decode tests — use an injected `AESGCM` cipher (set on + `_cipher` directly) so we don't need to mock Secrets Manager. + 2. `_ensure_cipher` path — exercises the deploy-time-bootstrapped data + key flow (`secretsmanager:GetSecretValue` -> SHA-256 -> AESGCM cipher) + with mock clients. This is the path that runs in production every + time a task starts. + +The cross-task seal/unseal regression — a cookie sealed by one process +unsealing on a *different* process — is locked in by +`test_two_codecs_with_same_secret_derive_the_same_cipher`. """ from __future__ import annotations import base64 +import hashlib import os import secrets +from unittest.mock import MagicMock import pytest from cryptography.hazmat.primitives.ciphers.aead import AESGCM from apis.shared.sessions_bff.cookie import ( CookieCodec, + CookieDataKeyUnavailable, CookieDecodeError, _reset_default_codec_for_tests, _set_default_codec_for_tests, @@ -110,17 +122,24 @@ def test_seal_preserves_extras() -> None: def test_default_codec_is_a_singleton() -> None: """The auth/callback route seals with this codec and the `SessionRefreshMiddleware` unseals with it on the next request — they - must be the *same* instance, since each `CookieCodec` derives its own - random AES key. A second instance would fail every unseal as 'bad seal'. + must be the *same* instance within a process so we don't refetch the + data-key secret on every cookie operation. + + Cross-process consistency (Task A's seal unsealing on Task B) is locked + in by `test_two_codecs_with_same_secret_derive_the_same_cipher`. """ _reset_default_codec_for_tests() try: os.environ["BFF_COOKIE_SIGNING_KEY_ARN"] = "arn:aws:kms:fake" + os.environ["BFF_COOKIE_DATA_KEY_SECRET_ARN"] = ( + "arn:aws:secretsmanager:us-east-1:0:secret:bff-data-key" + ) first = get_default_codec() second = get_default_codec() assert first is second finally: os.environ.pop("BFF_COOKIE_SIGNING_KEY_ARN", None) + os.environ.pop("BFF_COOKIE_DATA_KEY_SECRET_ARN", None) _reset_default_codec_for_tests() @@ -142,14 +161,136 @@ def test_default_codec_round_trip_seals_and_unseals() -> None: _reset_default_codec_for_tests() -def test_unseal_propagates_kms_infrastructure_errors() -> None: - """KMS unavailable is not a decode error — it must surface so the caller - can return 5xx instead of clearing the cookie and forcing re-login.""" - from unittest.mock import MagicMock +# ===================================================================== +# `_ensure_cipher` — Secrets Manager fetch + SHA-256 derivation path. +# ===================================================================== + +KMS_KEY_ARN = "arn:aws:kms:us-east-1:0:key/test" +DATA_KEY_SECRET_ARN = "arn:aws:secretsmanager:us-east-1:0:secret:bff-data-key" + + +def _make_sm_mock(secret_string: str) -> MagicMock: + sm = MagicMock() + sm.get_secret_value.return_value = {"SecretString": secret_string} + return sm + + +def test_ensure_cipher_fetches_secret_and_derives_key() -> None: + """Happy path: codec fetches the secret from Secrets Manager, derives + a 32-byte AES-256 key with SHA-256, then seals/unseals successfully.""" + secret_string = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKL012345" # 44 chars + sm = _make_sm_mock(secret_string) + + codec = CookieCodec( + kms_key_arn=KMS_KEY_ARN, + data_key_secret_arn=DATA_KEY_SECRET_ARN, + secrets_manager_client=sm, + ) + sealed = codec.seal(CookiePayload(session_id="sess-bootstrapped")) + assert codec.unseal(sealed).session_id == "sess-bootstrapped" + + sm.get_secret_value.assert_called_once_with(SecretId=DATA_KEY_SECRET_ARN) - fake_kms = MagicMock() - fake_kms.generate_data_key.side_effect = RuntimeError("KMS unreachable") - codec = CookieCodec(kms_key_arn="arn:aws:kms:fake", kms_client=fake_kms) - with pytest.raises(RuntimeError, match="KMS unreachable"): - codec.unseal("doesnt-matter") +def test_ensure_cipher_derived_key_matches_sha256_of_secret() -> None: + """Lock the KDF: a future change must keep the same derivation, or + every cookie sealed by an old task fails to unseal on a new task + after deploy.""" + secret_string = "deterministic-secret-for-kdf-pinning-test-1234" + sm = _make_sm_mock(secret_string) + + codec = CookieCodec( + kms_key_arn=KMS_KEY_ARN, + data_key_secret_arn=DATA_KEY_SECRET_ARN, + secrets_manager_client=sm, + ) + # Force initialization without exposing _cipher's key directly: use a + # parallel cipher with the expected key, encrypt, and decrypt with the + # codec. If the codec didn't derive via SHA-256, decrypt fails. + codec.seal(CookiePayload(session_id="x")) + expected_key = hashlib.sha256(secret_string.encode("utf-8")).digest() + expected_cipher = AESGCM(expected_key) + nonce = secrets.token_bytes(12) + ciphertext = expected_cipher.encrypt(nonce, b'{"sid":"y"}', bytes([1])) + blob = bytes([1]) + nonce + ciphertext + sealed = base64.urlsafe_b64encode(blob).rstrip(b"=").decode("ascii") + decoded = codec.unseal(sealed) + assert decoded.session_id == "y" + + +def test_ensure_cipher_caches_after_first_call() -> None: + """Hot-path requirement: only one Secrets Manager call per process.""" + sm = _make_sm_mock("a" * 44) + codec = CookieCodec( + kms_key_arn=KMS_KEY_ARN, + data_key_secret_arn=DATA_KEY_SECRET_ARN, + secrets_manager_client=sm, + ) + for _ in range(5): + codec.seal(CookiePayload(session_id="x")) + assert sm.get_secret_value.call_count == 1 + + +def test_two_codecs_with_same_secret_derive_the_same_cipher() -> None: + """Regression lock for the dev `bad seal` 401 storm. + + Two independent `CookieCodec` instances simulate two ECS tasks. Both + fetch the SAME secret string from Secrets Manager and derive the same + 32-byte key via SHA-256. A cookie sealed on `task_a` MUST unseal on + `task_b`. Pre-fix, each task generated its own random data key and + this failed. + """ + secret_string = "shared-secret-across-tasks-1234567890ABCDEFGH" + sm_a = _make_sm_mock(secret_string) + sm_b = _make_sm_mock(secret_string) + + task_a = CookieCodec( + kms_key_arn=KMS_KEY_ARN, + data_key_secret_arn=DATA_KEY_SECRET_ARN, + secrets_manager_client=sm_a, + ) + task_b = CookieCodec( + kms_key_arn=KMS_KEY_ARN, + data_key_secret_arn=DATA_KEY_SECRET_ARN, + secrets_manager_client=sm_b, + ) + + sealed_on_a = task_a.seal(CookiePayload(session_id="sess-cross-task")) + decoded_on_b = task_b.unseal(sealed_on_a) + assert decoded_on_b.session_id == "sess-cross-task" + + +def test_ensure_cipher_propagates_secrets_manager_failure() -> None: + """Secrets Manager unreachable must surface as `CookieDataKeyUnavailable` + so the request returns 5xx — never as a decode error that clears the + user's cookie.""" + sm = MagicMock() + sm.get_secret_value.side_effect = RuntimeError("Secrets Manager unreachable") + codec = CookieCodec( + kms_key_arn=KMS_KEY_ARN, + data_key_secret_arn=DATA_KEY_SECRET_ARN, + secrets_manager_client=sm, + ) + with pytest.raises(CookieDataKeyUnavailable): + codec.unseal("anything") + + +def test_ensure_cipher_rejects_empty_secret_string() -> None: + """Bootstrap not yet completed (or secret manually wiped) — fail loud + rather than silently invalidate every active session.""" + sm = _make_sm_mock("") + codec = CookieCodec( + kms_key_arn=KMS_KEY_ARN, + data_key_secret_arn=DATA_KEY_SECRET_ARN, + secrets_manager_client=sm, + ) + with pytest.raises(CookieDataKeyUnavailable, match="bootstrap missing"): + codec.unseal("anything") + + +def test_ensure_cipher_missing_config_surfaces_as_decode_error() -> None: + """No KMS ARN or no secret ARN — same shape as today's "BFF disabled" + path. Treated as `bad seal` so the middleware clears the cookie.""" + codec = CookieCodec(kms_key_arn="", data_key_secret_arn="") + with pytest.raises(CookieDecodeError): + codec.unseal("anything") diff --git a/backend/tests/apis/shared/sessions_bff/test_repository.py b/backend/tests/apis/shared/sessions_bff/test_repository.py index ec6c771b..b20c33cf 100644 --- a/backend/tests/apis/shared/sessions_bff/test_repository.py +++ b/backend/tests/apis/shared/sessions_bff/test_repository.py @@ -77,3 +77,302 @@ async def test_disabled_repository_is_inert() -> None: # All ops succeed silently — no exceptions, no AWS calls. assert await repo.get("any") is None await repo.delete("any") + + +# ===================================================================== +# Cross-task refresh lock — try_acquire_refresh_lock / release_refresh_lock +# ===================================================================== + + +@pytest.mark.asyncio +async def test_try_acquire_refresh_lock_succeeds_on_unlocked_row( + repository, sample_record +) -> None: + """The first contender claims the lock when no peer is holding one.""" + record = sample_record() + await repository.put(record) + + acquired = await repository.try_acquire_refresh_lock( + session_id=record.session_id, + owner="task-A", + lock_ttl_seconds=30, + ) + assert acquired is True + + +@pytest.mark.asyncio +async def test_try_acquire_refresh_lock_blocks_concurrent_peer( + repository, sample_record +) -> None: + """While task-A's lock is fresh, task-B's acquisition MUST fail. + + This is the cross-task coalescing primitive — without it, two tasks + would each call cognito-idp:initiate_auth with the same refresh token + under desiredCount > 1. + """ + record = sample_record() + await repository.put(record) + + a = await repository.try_acquire_refresh_lock( + session_id=record.session_id, + owner="task-A", + lock_ttl_seconds=30, + ) + b = await repository.try_acquire_refresh_lock( + session_id=record.session_id, + owner="task-B", + lock_ttl_seconds=30, + ) + assert a is True + assert b is False + + +@pytest.mark.asyncio +async def test_try_acquire_refresh_lock_takes_over_after_ttl_expires( + repository, sample_record +) -> None: + """A leader that crashed mid-refresh strands the lock for at most + `lock_ttl_seconds`. After that, any peer can re-acquire — no manual + cleanup required, no permanent stuck state.""" + record = sample_record() + await repository.put(record) + + # task-A acquires with a 0-second TTL → lock_until = now, so any + # contender at a later second sees `refresh_lock_until < :now`. + a = await repository.try_acquire_refresh_lock( + session_id=record.session_id, + owner="task-A", + lock_ttl_seconds=0, + ) + assert a is True + + # Sleep 1s so the next contender's :now is strictly greater. + time.sleep(1) + + b = await repository.try_acquire_refresh_lock( + session_id=record.session_id, + owner="task-B", + lock_ttl_seconds=30, + ) + assert b is True + + +@pytest.mark.asyncio +async def test_try_acquire_refresh_lock_distinct_sessions_dont_block( + repository, sample_record +) -> None: + rec_a = sample_record(session_id="sess-A") + rec_b = sample_record(session_id="sess-B") + await repository.put(rec_a) + await repository.put(rec_b) + + a = await repository.try_acquire_refresh_lock( + session_id=rec_a.session_id, owner="task-1", lock_ttl_seconds=30 + ) + b = await repository.try_acquire_refresh_lock( + session_id=rec_b.session_id, owner="task-1", lock_ttl_seconds=30 + ) + assert a is True + assert b is True + + +@pytest.mark.asyncio +async def test_release_refresh_lock_clears_attrs_for_owner( + repository, sample_record +) -> None: + record = sample_record() + await repository.put(record) + await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="task-A", lock_ttl_seconds=30 + ) + + await repository.release_refresh_lock(record.session_id, owner="task-A") + + # After release a peer can immediately acquire. + b = await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="task-B", lock_ttl_seconds=30 + ) + assert b is True + + +@pytest.mark.asyncio +async def test_release_refresh_lock_is_no_op_for_non_owner( + repository, sample_record +) -> None: + """Best-effort release: if a peer has already taken over the lock + (because ours TTL'd), the release MUST NOT clear their lock attrs.""" + record = sample_record() + await repository.put(record) + await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="task-A", lock_ttl_seconds=30 + ) + + # task-B (who never held the lock) calls release — must not blow away + # task-A's lock. + await repository.release_refresh_lock(record.session_id, owner="task-B") + + # task-A's lock is still in force; a third contender can't acquire. + c = await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="task-C", lock_ttl_seconds=30 + ) + assert c is False + + +@pytest.mark.asyncio +async def test_update_tokens_with_lock_owner_clears_lock_atomically( + repository, sample_record +) -> None: + """Successful refresh persist clears the lock attributes in the same + write so peers don't have to wait for the TTL to retry.""" + record = sample_record() + await repository.put(record) + await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="task-A", lock_ttl_seconds=30 + ) + + await repository.update_tokens( + session_id=record.session_id, + access_token="access.fresh", + refresh_token="refresh.rotated", + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + last_seen_at=int(time.time()), + expected_lock_owner="task-A", + ) + + # Lock cleared → another contender can acquire immediately. + b = await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="task-B", lock_ttl_seconds=30 + ) + assert b is True + + +@pytest.mark.asyncio +async def test_update_tokens_rejects_persist_when_peer_owns_the_lock( + repository, sample_record +) -> None: + """Stale-leader guard: if our lock TTL'd and a peer took over, we must + NOT overwrite their freshly persisted tokens. ConditionalCheckFailed + propagates so the caller can re-read DDB and adopt the peer's state.""" + from botocore.exceptions import ClientError + + record = sample_record() + await repository.put(record) + # Peer task acquired the lock. + await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="peer-task", lock_ttl_seconds=30 + ) + + with pytest.raises(ClientError) as exc_info: + await repository.update_tokens( + session_id=record.session_id, + access_token="access.stale", + refresh_token="refresh.stale", + id_token="id.stale", + access_token_exp=int(time.time()) + 3600, + last_seen_at=int(time.time()), + expected_lock_owner="our-task", # ≠ peer-task + ) + assert ( + exc_info.value.response.get("Error", {}).get("Code") + == "ConditionalCheckFailedException" + ) + + +@pytest.mark.asyncio +async def test_update_tokens_rejects_persist_when_peer_already_cleared_the_lock( + repository, sample_record +) -> None: + """The other half of the stale-leader guard: a peer whose lock TTL'd, + took over, refreshed, and successfully persisted (which atomically + REMOVEs the lock attrs) — the row now has NO lock attributes at all. + A stale leader trying to persist with `expected_lock_owner=our-task` + must still fail closed; otherwise our older Cognito tokens would + silently overwrite the peer's freshly rotated ones, and the next + request would get NotAuthorizedException from Cognito (our refresh + token was revoked when the peer's rotation was issued). + + Sequence: + 1. Task A acquires lock at T0. + 2. Task A's Cognito call hangs. + 3. Task B sees lock TTL'd, acquires, refreshes, persists (clears). + 4. Task A's Cognito finally returns; A tries to persist. + => MUST fail with ConditionalCheckFailedException. + """ + from botocore.exceptions import ClientError + + record = sample_record() + await repository.put(record) + + # Peer acquired the lock and successfully persisted (clearing it). + await repository.try_acquire_refresh_lock( + session_id=record.session_id, owner="peer-task", lock_ttl_seconds=30 + ) + await repository.update_tokens( + session_id=record.session_id, + access_token="access.peer-fresh", + refresh_token="refresh.peer-rotated", + id_token="id.peer", + access_token_exp=int(time.time()) + 3600, + last_seen_at=int(time.time()), + expected_lock_owner="peer-task", + ) + + # Stale leader (our-task) — never owned a lock that's still on the + # row, but holds an old `lock_owner` from before the TTL. Must fail. + with pytest.raises(ClientError) as exc_info: + await repository.update_tokens( + session_id=record.session_id, + access_token="access.stale-leader", + refresh_token="refresh.stale-leader", + id_token="id.stale-leader", + access_token_exp=int(time.time()) + 3600, + last_seen_at=int(time.time()), + expected_lock_owner="our-task", + ) + assert ( + exc_info.value.response.get("Error", {}).get("Code") + == "ConditionalCheckFailedException" + ) + + # Peer's tokens are still intact on the row. + fetched = await repository.get(record.session_id) + assert fetched is not None + assert fetched.cognito_access_token == "access.peer-fresh" + assert fetched.cognito_refresh_token == "refresh.peer-rotated" + + +@pytest.mark.asyncio +async def test_try_acquire_refresh_lock_does_not_create_phantom_row( + repository, moto_bff_dynamodb +) -> None: + """Logout-during-refresh guard: if the session row was deleted between + `repository.get()` and `try_acquire_refresh_lock`, UpdateItem would + upsert a phantom row containing only the lock attrs (and crucially no + `ttl`, so DDB TTL would never reap it). The `attribute_exists(PK)` + guard turns that into a clean False return. + + Asserts via raw DDB get_item — `repository.get` would mask a phantom + behind its post-read TTL check (a row with no `ttl` attribute reads + as `int(item.get("ttl", 0)) <= now`, treated as missing), so we + bypass that and look at the raw item. + """ + # Session row never existed (or was just deleted by a logout from + # another task between this request's repository.get() and here). + acquired = await repository.try_acquire_refresh_lock( + session_id="never-existed", + owner="task-A", + lock_ttl_seconds=30, + ) + assert acquired is False + + # No phantom row was created — check the raw table, since + # repository.get() would also return None for a phantom (no `ttl`). + table = moto_bff_dynamodb.Table("test-bff-sessions") + response = table.get_item( + Key={"PK": "SESSION#never-existed", "SK": "META"} + ) + assert "Item" not in response, ( + "try_acquire_refresh_lock created a phantom row with no `ttl` — " + "DDB TTL would never reap it" + ) diff --git a/backend/tests/apis/shared/sessions_bff/test_session_refresh_cross_task.py b/backend/tests/apis/shared/sessions_bff/test_session_refresh_cross_task.py new file mode 100644 index 00000000..946e357c --- /dev/null +++ b/backend/tests/apis/shared/sessions_bff/test_session_refresh_cross_task.py @@ -0,0 +1,480 @@ +"""Cross-task refresh-coalescing tests for SessionRefreshMiddleware. + +Locks in the regression that PR #264 created and the cookie-codec fix would +*expose* once dev started working again: with `desiredCount: 2`, two +`SessionRefreshMiddleware` instances running in two ECS tasks would each +see a cookie crossing the refresh-leeway boundary, each call +`cognito-idp:initiate_auth` with the same refresh token, and one of them +would lose the rotation race — Cognito revokes the original token on the +winner's exchange, the loser gets `NotAuthorizedException`, the loser's +middleware clears the user's cookie. Page-load fan-outs become routine +silent logouts. + +The fix coalesces the refresh exchange across tasks via a DynamoDB +conditional-write lock (`refresh_lock_owner` + `refresh_lock_until` on +the session row). These tests instantiate two repository + middleware +pairs against ONE moto-backed DDB table so we can drive the leader and +follower paths deterministically without spinning real ECS tasks. + +What's covered: + - Leader-only Cognito refresh under same-time contention from two tasks + - Follower adoption of the leader's persisted tokens (no Cognito call) + - Leader crash (Cognito error) releases the lock so peers can retry + - Lock TTL recovery: a crashed leader's lock unblocks peers after TTL + - Refresh-token rotation: peer's rotated tokens propagate to follower +""" + +from __future__ import annotations + +import asyncio +import secrets +import time +from typing import Optional +from unittest.mock import AsyncMock, MagicMock + +import boto3 +import pytest +from cryptography.hazmat.primitives.ciphers.aead import AESGCM +from fastapi import FastAPI, Request +from fastapi.testclient import TestClient +from moto import mock_aws + +from apis.shared.middleware.session_refresh import SessionRefreshMiddleware +from apis.shared.sessions_bff import lock as lock_module +from apis.shared.sessions_bff import single_flight as single_flight_module +from apis.shared.sessions_bff.cache import SessionCache +from apis.shared.sessions_bff.config import ( + BFFConfig, + SESSION_COOKIE_NAME, +) +from apis.shared.sessions_bff.cookie import CookieCodec +from apis.shared.sessions_bff.models import CookiePayload, SessionRecord +from apis.shared.sessions_bff.refresh import ( + CognitoRefreshError, + RefreshResult, +) +from apis.shared.sessions_bff.repository import SessionRepository + +# Single shared DDB table — both "tasks" attach to the same backing store, +# matching production where two ECS tasks read/write one BFFSessionsTable. +TABLE_NAME = "test-bff-sessions" + + +@pytest.fixture(autouse=True) +def _reset_module_state(): + """Drop process-wide locks + single-flight registries between tests so + a leftover Future or asyncio lock from one test can't influence the + next case's contention behavior.""" + lock_module._reset_for_tests() + single_flight_module._reset_for_tests() + yield + lock_module._reset_for_tests() + single_flight_module._reset_for_tests() + + +@pytest.fixture +def two_task_setup(monkeypatch): + """Spin up two `SessionRefreshMiddleware` instances over one moto DDB + table so each represents a distinct ECS task in the same fleet.""" + monkeypatch.setenv("AWS_DEFAULT_REGION", "us-east-1") + monkeypatch.setenv("AWS_ACCESS_KEY_ID", "testing") + monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "testing") + + with mock_aws(): + dynamodb = boto3.resource("dynamodb", region_name="us-east-1") + dynamodb.create_table( + TableName=TABLE_NAME, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + + # Both tasks share the data-key secret (otherwise the cookie sealed + # by Task A would unseal as `bad seal` on Task B — that's the OTHER + # bug in this branch, exercised by test_cookie). We pre-inject one + # AES key here to keep the test focused on the refresh-lock path. + shared_aes_key = secrets.token_bytes(32) + + def _make_codec() -> CookieCodec: + codec = CookieCodec( + kms_key_arn="arn:aws:kms:fake", + data_key_secret_arn="arn:aws:secretsmanager:fake", + ) + codec._cipher = AESGCM(shared_aes_key) + return codec + + def _make_task(*, refresh_client) -> dict: + repo = SessionRepository(table_name=TABLE_NAME) + codec = _make_codec() + cache = SessionCache(ttl_seconds=60) + config = _enabled_config() + + app = FastAPI() + app.add_middleware( + SessionRefreshMiddleware, + config=config, + repository=repo, + cookie_codec=codec, + refresh_client=refresh_client, + cache=cache, + refresh_lock_ttl_seconds=2, # short for tests + ) + + @app.get("/echo") + async def echo(request: Request): + record = getattr(request.state, "bff_session", None) + return { + "has_session": record is not None, + "access_token": ( + record.cognito_access_token if record else None + ), + "refresh_token": ( + record.cognito_refresh_token if record else None + ), + } + + return { + "app": app, + "repository": repo, + "codec": codec, + "cache": cache, + "refresh_client": refresh_client, + } + + yield { + "make_task": _make_task, + "table_name": TABLE_NAME, + "shared_aes_key": shared_aes_key, + "make_codec": _make_codec, + } + + +def _enabled_config() -> BFFConfig: + return BFFConfig( + sessions_table_name="tbl", + cookie_signing_key_arn="arn:aws:kms:fake", + session_ttl_seconds=28800, + refresh_leeway_seconds=60, + cognito_bff_app_client_id="client-id", + cognito_bff_app_client_secret_arn="arn:secret", + inference_api_url=None, + absolute_lifetime_seconds=30 * 24 * 3600, + sliding_renewal_throttle_seconds=300, + ) + + +def _seed_session_in_refresh_window(repository: SessionRepository) -> SessionRecord: + """Persist a session whose access token is inside the refresh leeway, + so the middleware MUST hit the refresh path.""" + now = int(time.time()) + record = SessionRecord( + session_id="sess-cross-task", + user_id="user-001", + username="alice", + cognito_access_token="access.original", + cognito_refresh_token="refresh.original", + id_token="id.original", + access_token_exp=now + 5, # within 60s leeway + csrf_secret="csrf-secret", + created_at=now, + last_seen_at=now, + ttl=now + 28800, + ) + asyncio.run(repository.put(record)) + return record + + +def test_only_the_leader_calls_cognito_under_cross_task_contention( + two_task_setup, +) -> None: + """Two tasks see the same cookie in the refresh window. Exactly one + calls Cognito (the leader). The other adopts the leader's tokens + from DDB without ever calling Cognito. + + Pre-fix: BOTH tasks would call Cognito with the same refresh token, + and the loser would get NotAuthorizedException → clear cookie → 401. + """ + # Refresh client A is the leader's; refresh client B simulates the + # follower's. We assert that B is NEVER called. + leader_refresh = AsyncMock( + return_value=RefreshResult( + access_token="access.fresh-from-leader", + refresh_token="refresh.rotated-by-leader", + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) + ) + follower_refresh = AsyncMock( + side_effect=AssertionError( + "Follower MUST NOT call Cognito — peer holds the refresh lock" + ) + ) + + task_a = two_task_setup["make_task"](refresh_client=MagicMock(refresh=leader_refresh)) + task_b = two_task_setup["make_task"](refresh_client=MagicMock(refresh=follower_refresh)) + + record = _seed_session_in_refresh_window(task_a["repository"]) + sealed = task_a["codec"].seal(CookiePayload(session_id=record.session_id)) + + # Drive task_a first (it'll grab the lock and refresh). Then drive + # task_b — it must observe the lock as held (or just released, with + # tokens already rotated on the row) and adopt rather than refresh. + with TestClient(task_a["app"]) as client_a: + response_a = client_a.get( + "/echo", cookies={SESSION_COOKIE_NAME: sealed} + ) + with TestClient(task_b["app"]) as client_b: + response_b = client_b.get( + "/echo", cookies={SESSION_COOKIE_NAME: sealed} + ) + + assert response_a.status_code == 200 + assert response_b.status_code == 200 + assert response_a.json()["has_session"] is True + assert response_b.json()["has_session"] is True + # Both tasks see the leader's freshly rotated tokens. + assert response_a.json()["access_token"] == "access.fresh-from-leader" + assert response_b.json()["access_token"] == "access.fresh-from-leader" + assert response_b.json()["refresh_token"] == "refresh.rotated-by-leader" + + leader_refresh.assert_called_once() + follower_refresh.assert_not_called() + + +def test_follower_polls_until_leader_persists_then_adopts( + two_task_setup, +) -> None: + """Simulates near-simultaneous arrival: task_a gets the lock just + before task_b runs. Task_b's `_wait_for_peer_refresh` polls DDB + and adopts task_a's tokens once they land. + + To force the follower to actually poll (rather than fall through + a fully-completed leader path), we make the leader's Cognito refresh + take a measurable amount of time and start the follower while the + leader is still in flight. + """ + leader_done = asyncio.Event() + follower_started = asyncio.Event() + + async def slow_leader_refresh(*args, **kwargs) -> RefreshResult: + # Wait for the follower to be inside its poll loop, then complete. + await follower_started.wait() + await asyncio.sleep(0.05) + leader_done.set() + return RefreshResult( + access_token="access.fresh-leader", + refresh_token="refresh.rotated-leader", + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) + + leader_refresh = AsyncMock(side_effect=slow_leader_refresh) + follower_refresh = AsyncMock( + side_effect=AssertionError("Follower must NOT call Cognito") + ) + + task_a = two_task_setup["make_task"](refresh_client=MagicMock(refresh=leader_refresh)) + task_b = two_task_setup["make_task"](refresh_client=MagicMock(refresh=follower_refresh)) + record = _seed_session_in_refresh_window(task_a["repository"]) + sealed = task_a["codec"].seal(CookiePayload(session_id=record.session_id)) + + async def drive_both() -> tuple[dict, dict]: + async def hit(client_app): + from httpx import ASGITransport, AsyncClient + + async with AsyncClient( + transport=ASGITransport(app=client_app), base_url="http://t" + ) as client: + response = await client.get( + "/echo", cookies={SESSION_COOKIE_NAME: sealed} + ) + return response.json() + + async def driven_follower(): + # Start the follower a tick later, so the leader has the lock. + await asyncio.sleep(0.02) + follower_started.set() + return await hit(task_b["app"]) + + a, b = await asyncio.gather(hit(task_a["app"]), driven_follower()) + return a, b + + a_body, b_body = asyncio.run(drive_both()) + + assert a_body["has_session"] is True + assert b_body["has_session"] is True + assert a_body["access_token"] == "access.fresh-leader" + assert b_body["access_token"] == "access.fresh-leader" + leader_refresh.assert_called_once() + follower_refresh.assert_not_called() + + +def test_lock_ttl_lets_a_peer_retry_after_a_dead_leader( + two_task_setup, +) -> None: + """Leader's Cognito call fails → lock is released → peer can refresh + on its next request without waiting for the full TTL. + + This guards against the worst case where a leader crashes mid-refresh + and never persists tokens. We don't want every subsequent request to + fail closed for the duration of the lock TTL. + """ + leader_refresh = AsyncMock(side_effect=CognitoRefreshError("Cognito down")) + follower_refresh = AsyncMock( + return_value=RefreshResult( + access_token="access.peer-fresh", + refresh_token="refresh.peer-rotated", + id_token="id.peer", + access_token_exp=int(time.time()) + 3600, + ) + ) + + task_a = two_task_setup["make_task"]( + refresh_client=MagicMock(refresh=leader_refresh) + ) + task_b = two_task_setup["make_task"]( + refresh_client=MagicMock(refresh=follower_refresh) + ) + record = _seed_session_in_refresh_window(task_a["repository"]) + sealed = task_a["codec"].seal(CookiePayload(session_id=record.session_id)) + + # Task A: leader fails. The middleware clears its cookie for THIS + # request but releases the lock (so a peer can retry). + with TestClient(task_a["app"]) as client_a: + response_a = client_a.get( + "/echo", cookies={SESSION_COOKIE_NAME: sealed} + ) + assert response_a.status_code == 200 + assert response_a.json()["has_session"] is False + set_cookies_a = response_a.headers.get_list("set-cookie") + assert any( + "__Host-bff_session=" in c and "Max-Age=0" in c for c in set_cookies_a + ), "Task A must clear cookie after its own refresh failed" + + # Task B (different request): lock is released; peer becomes the new + # leader and refreshes successfully. + with TestClient(task_b["app"]) as client_b: + response_b = client_b.get( + "/echo", cookies={SESSION_COOKIE_NAME: sealed} + ) + assert response_b.status_code == 200 + assert response_b.json()["has_session"] is True + assert response_b.json()["access_token"] == "access.peer-fresh" + leader_refresh.assert_called_once() + follower_refresh.assert_called_once() + + +def test_follower_falls_back_terminal_when_leader_disappears_mid_refresh( + two_task_setup, +) -> None: + """Pathological case: leader holds the lock but never persists tokens + AND never releases (e.g. process killed). The follower's poll deadline + is bounded by `refresh_lock_ttl_seconds`; after that, this request + fails closed (clear cookie). The user re-auths. + + The next request after this one will see the lock TTL'd and can + re-acquire — that path is covered by + `test_lock_ttl_lets_a_peer_retry_after_a_dead_leader`. + """ + follower_refresh = AsyncMock( + side_effect=AssertionError("Follower must NOT call Cognito while a peer holds the lock") + ) + task_b = two_task_setup["make_task"]( + refresh_client=MagicMock(refresh=follower_refresh) + ) + record = _seed_session_in_refresh_window(task_b["repository"]) + + # Manually park a lock on the row as if some other task is mid-refresh + # but hasn't persisted yet (and won't, for the duration of this test). + asyncio.run( + task_b["repository"].try_acquire_refresh_lock( + session_id=record.session_id, + owner="ghost-task", + lock_ttl_seconds=2, # matches make_task's middleware TTL + ) + ) + + sealed = task_b["codec"].seal(CookiePayload(session_id=record.session_id)) + with TestClient(task_b["app"]) as client_b: + response = client_b.get( + "/echo", cookies={SESSION_COOKIE_NAME: sealed} + ) + + assert response.status_code == 200 + assert response.json()["has_session"] is False + set_cookies = response.headers.get_list("set-cookie") + assert any( + "__Host-bff_session=" in c and "Max-Age=0" in c for c in set_cookies + ), "Follower must clear cookie after polling timed out on a stuck leader" + follower_refresh.assert_not_called() + + +def test_two_tasks_in_parallel_call_cognito_at_most_once( + two_task_setup, +) -> None: + """Pure asyncio gather of one request per task at the same instant. + Whichever wins the conditional UpdateItem becomes the leader; the + other adopts. Combined Cognito call count must be exactly 1. + + This is the closest analogue to the page-load fan-out behavior we + care about in production — two tasks each receive their share of + the 8-endpoint fan-out at the moment the cookie crosses the leeway + window. + """ + refresh_count = {"calls": 0} + + async def counted_refresh(*args, **kwargs): + refresh_count["calls"] += 1 + await asyncio.sleep(0.05) + return RefreshResult( + access_token="access.fresh", + refresh_token="refresh.rotated", + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) + + refresh_a = AsyncMock(side_effect=counted_refresh) + refresh_b = AsyncMock(side_effect=counted_refresh) + + task_a = two_task_setup["make_task"]( + refresh_client=MagicMock(refresh=refresh_a) + ) + task_b = two_task_setup["make_task"]( + refresh_client=MagicMock(refresh=refresh_b) + ) + record = _seed_session_in_refresh_window(task_a["repository"]) + sealed = task_a["codec"].seal(CookiePayload(session_id=record.session_id)) + + async def drive() -> tuple[dict, dict]: + from httpx import ASGITransport, AsyncClient + + async def hit(app): + async with AsyncClient( + transport=ASGITransport(app=app), base_url="http://t" + ) as client: + response = await client.get( + "/echo", cookies={SESSION_COOKIE_NAME: sealed} + ) + return response.json() + + return await asyncio.gather(hit(task_a["app"]), hit(task_b["app"])) + + a_body, b_body = asyncio.run(drive()) + + # Both succeeded. + assert a_body["has_session"] is True + assert b_body["has_session"] is True + # Both got the same fresh tokens (one set, sourced from the leader). + assert a_body["access_token"] == b_body["access_token"] == "access.fresh" + assert a_body["refresh_token"] == b_body["refresh_token"] == "refresh.rotated" + # CRITICAL: across BOTH tasks, Cognito refresh was called at most once. + assert refresh_count["calls"] == 1, ( + f"Cross-task coalescing violated — Cognito refresh was called " + f"{refresh_count['calls']} times across two tasks" + ) diff --git a/backend/tests/apis/shared/sessions_bff/test_session_refresh_middleware.py b/backend/tests/apis/shared/sessions_bff/test_session_refresh_middleware.py index cdc68f7e..7e64f02d 100644 --- a/backend/tests/apis/shared/sessions_bff/test_session_refresh_middleware.py +++ b/backend/tests/apis/shared/sessions_bff/test_session_refresh_middleware.py @@ -226,12 +226,16 @@ def test_near_expiry_session_triggers_refresh_once() -> None: repo = AsyncMock() repo.get.return_value = record codec = _make_codec() + # `refresh_client.refresh` is now `async` (task 3.2) — use AsyncMock so + # `await self._refresh_client.refresh(...)` in the middleware resolves. refresh = MagicMock() - refresh.refresh.return_value = RefreshResult( - access_token="access.fresh", - refresh_token="refresh.fresh", - id_token="id.fresh", - access_token_exp=int(time.time()) + 3600, + refresh.refresh = AsyncMock( + return_value=RefreshResult( + access_token="access.fresh", + refresh_token="refresh.fresh", + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) ) app = _build_app( config=_enabled_config(), repository=repo, codec=codec, refresh_client=refresh @@ -245,7 +249,7 @@ def test_near_expiry_session_triggers_refresh_once() -> None: assert body["has_session"] is True # The refreshed token should be exposed downstream. assert body["access_token"] == "access.fresh" - refresh.refresh.assert_called_once_with( + refresh.refresh.assert_awaited_once_with( username="alice", refresh_token="refresh.original" ) repo.update_tokens.assert_awaited_once() @@ -259,7 +263,7 @@ def test_refresh_failure_clears_cookie() -> None: repo.get.return_value = record codec = _make_codec() refresh = MagicMock() - refresh.refresh.side_effect = CognitoRefreshError("rotated") + refresh.refresh = AsyncMock(side_effect=CognitoRefreshError("rotated")) app = _build_app( config=_enabled_config(), repository=repo, codec=codec, refresh_client=refresh ) @@ -298,7 +302,7 @@ def slow_refresh(*, username: str, refresh_token: str) -> RefreshResult: ) refresh = MagicMock() - refresh.refresh.side_effect = slow_refresh + refresh.refresh = AsyncMock(side_effect=slow_refresh) # After the first refresh, repo.get returns the *fresh* record so other # waiters short-circuit out of the refresh branch. @@ -381,7 +385,13 @@ def test_slide_within_throttle_window_does_not_write_or_reemit() -> None: def test_slide_past_throttle_writes_ddb_and_reemits_cookie() -> None: """Once `last_seen_at` is older than the throttle window, the slide fires: one DDB touch with a fresh ttl, plus a Set-Cookie carrying a - fresh Max-Age = session_ttl_seconds.""" + fresh Max-Age = session_ttl_seconds. + + The slide-write is fire-and-forget (task 3.5) — we poll for the + background task's side effect rather than sample immediately. The + observable external contract (Set-Cookie Max-Age) is unchanged; only + the internal timing of the write moves off the request path. + """ record = _make_record() record.last_seen_at = int(time.time()) - 120 # past the 60s throttle repo = AsyncMock() @@ -393,7 +403,21 @@ def test_slide_past_throttle_writes_ddb_and_reemits_cookie() -> None: ) sealed = codec.seal(CookiePayload(session_id=record.session_id)) - response = TestClient(app).get("/echo", cookies={SESSION_COOKIE_NAME: sealed}) + with TestClient(app) as client: + response = client.get("/echo", cookies={SESSION_COOKIE_NAME: sealed}) + # Poll for the fire-and-forget slide-write (task 3.5) INSIDE the + # `with` block — TestClient tears down its anyio portal (and the + # event loop) on `__exit__`, cancelling any unfinished tasks. + # Drive the loop with a second GET if the first request's + # background task hasn't flushed yet. + deadline = time.monotonic() + 1.0 + while time.monotonic() < deadline and repo.touch_last_seen.await_count == 0: + time.sleep(0.01) + if repo.touch_last_seen.await_count == 0: + client.get("/echo") + deadline = time.monotonic() + 1.0 + while time.monotonic() < deadline and repo.touch_last_seen.await_count == 0: + time.sleep(0.01) assert response.status_code == 200 # Exactly one slide-write, and it carries a ttl bumped by ~session_ttl_seconds. @@ -472,6 +496,44 @@ def test_slide_max_age_capped_by_remaining_absolute_lifetime() -> None: assert 350 <= max_age <= 400 +def test_refresh_path_past_absolute_cap_clears_cookie_without_calling_cognito() -> None: + """The refresh path must mirror the slide path's absolute-cap behavior: + once `created_at + absolute_lifetime` has passed, do NOT mint fresh + tokens. Persisting them would also write a past-dated `ttl` + (`min(now + session_ttl_seconds, absolute_cap)` is `< now` past the + cap), which would instantly TTL-evict the row right after the write + and silently log the user out one request later. Failing closed up + front avoids burning a Cognito refresh-token rotation we'd just + throw away.""" + record = _make_record(access_token_exp=int(time.time()) + 5) # within leeway + record.created_at = int(time.time()) - 200 # past 100s absolute cap + repo = AsyncMock() + repo.get.return_value = record + codec = _make_codec() + refresh = MagicMock() + refresh.refresh = AsyncMock( + side_effect=AssertionError( + "Cognito refresh MUST NOT be called past absolute lifetime" + ) + ) + app = _build_app( + config=_enabled_config(absolute_lifetime_seconds=100), + repository=repo, + codec=codec, + refresh_client=refresh, + ) + + sealed = codec.seal(CookiePayload(session_id=record.session_id)) + response = TestClient(app).get("/echo", cookies={SESSION_COOKIE_NAME: sealed}) + + assert response.status_code == 200 + assert response.json()["has_session"] is False + refresh.refresh.assert_not_called() + repo.update_tokens.assert_not_called() + cleared = " ".join(response.headers.get_list("set-cookie")) + assert SESSION_COOKIE_NAME in cleared and "Max-Age=0" in cleared + + def test_refresh_path_bumps_ttl_when_persisting_tokens() -> None: """The token-rotation write must also slide the row's ttl forward — otherwise a session that just refreshed could still expire moments @@ -481,11 +543,13 @@ def test_refresh_path_bumps_ttl_when_persisting_tokens() -> None: repo.get.return_value = record codec = _make_codec() refresh = MagicMock() - refresh.refresh.return_value = RefreshResult( - access_token="access.fresh", - refresh_token="refresh.original", # no rotation - id_token="id.fresh", - access_token_exp=int(time.time()) + 3600, + refresh.refresh = AsyncMock( + return_value=RefreshResult( + access_token="access.fresh", + refresh_token="refresh.original", # no rotation + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) ) app = _build_app( config=_enabled_config(), repository=repo, codec=codec, refresh_client=refresh @@ -517,11 +581,13 @@ def test_rotation_persist_failure_invalidates_session() -> None: repo.update_tokens.side_effect = RuntimeError("DDB throttled") codec = _make_codec() refresh = MagicMock() - refresh.refresh.return_value = RefreshResult( - access_token="access.fresh", - refresh_token="refresh.ROTATED", # rotation kicked in - id_token="id.fresh", - access_token_exp=int(time.time()) + 3600, + refresh.refresh = AsyncMock( + return_value=RefreshResult( + access_token="access.fresh", + refresh_token="refresh.ROTATED", # rotation kicked in + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) ) app = _build_app( config=_enabled_config(), repository=repo, codec=codec, refresh_client=refresh @@ -550,11 +616,13 @@ def test_non_rotation_persist_failure_does_not_invalidate() -> None: repo.update_tokens.side_effect = RuntimeError("DDB throttled") codec = _make_codec() refresh = MagicMock() - refresh.refresh.return_value = RefreshResult( - access_token="access.fresh", - refresh_token="refresh.original", # SAME — no rotation - id_token="id.fresh", - access_token_exp=int(time.time()) + 3600, + refresh.refresh = AsyncMock( + return_value=RefreshResult( + access_token="access.fresh", + refresh_token="refresh.original", # SAME — no rotation + id_token="id.fresh", + access_token_exp=int(time.time()) + 3600, + ) ) app = _build_app( config=_enabled_config(), repository=repo, codec=codec, refresh_client=refresh diff --git a/backend/tests/apis/shared/sessions_bff/test_single_flight.py b/backend/tests/apis/shared/sessions_bff/test_single_flight.py new file mode 100644 index 00000000..e2159765 --- /dev/null +++ b/backend/tests/apis/shared/sessions_bff/test_single_flight.py @@ -0,0 +1,211 @@ +"""Unit tests for the per-session single-flight primitive. + +Covers the contract documented in +`backend/src/apis/shared/sessions_bff/single_flight.py`: + +1. Two concurrent `resolve_once` calls for the same `session_id` share one + loader invocation; both receive the same result. +2. An exception raised by the loader propagates to every current waiter + (leader + all followers). +3. After a loader exception the registry entry is removed, so a subsequent + call starts a fresh leader. +4. Distinct `session_id`s are independent (two different sessions produce two + loader invocations). +5. Happy path: a single caller's result is returned correctly. +""" + +from __future__ import annotations + +import asyncio +import time +from typing import Optional, Tuple + +import pytest + +from apis.shared.sessions_bff import single_flight +from apis.shared.sessions_bff.models import SessionRecord + + +def _make_record(session_id: str = "sess-sf-001") -> SessionRecord: + now = int(time.time()) + return SessionRecord( + session_id=session_id, + user_id=f"user-for-{session_id}", + username="alice", + cognito_access_token="access.token.value", + cognito_refresh_token="refresh.token.value", + id_token="id.token.value", + access_token_exp=now + 3600, + csrf_secret="csrf-secret-deadbeef", + created_at=now, + last_seen_at=now, + ttl=now + 28800, + ) + + +@pytest.fixture(autouse=True) +def _reset_registry(): + """Drop any residual in-flight Futures between tests.""" + single_flight._reset_for_tests() + yield + single_flight._reset_for_tests() + + +@pytest.mark.asyncio +async def test_happy_path_single_caller_returns_loader_result(): + """A lone caller receives the loader's exact return value.""" + record = _make_record() + call_count = 0 + + async def loader() -> Tuple[Optional[SessionRecord], bool]: + nonlocal call_count + call_count += 1 + return record, False + + result = await single_flight.resolve_once("sess-sf-001", loader) + + assert result == (record, False) + assert call_count == 1 + # Registry is clean after success. + assert "sess-sf-001" not in single_flight._inflight + + +@pytest.mark.asyncio +async def test_concurrent_same_session_share_one_loader_invocation(): + """N concurrent `resolve_once` calls on the same session call loader once.""" + record = _make_record() + call_count = 0 + gate = asyncio.Event() + + async def loader() -> Tuple[Optional[SessionRecord], bool]: + nonlocal call_count + call_count += 1 + # Hold the leader open long enough for followers to attach. + await gate.wait() + return record, False + + async def release_after_followers_attach() -> None: + # Give followers a chance to see the existing Future. + await asyncio.sleep(0.05) + gate.set() + + tasks = [ + asyncio.create_task(single_flight.resolve_once("sess-sf-002", loader)) + for _ in range(8) + ] + releaser = asyncio.create_task(release_after_followers_attach()) + + results = await asyncio.gather(*tasks) + await releaser + + assert call_count == 1, "loader must be invoked exactly once for shared session" + for result in results: + assert result == (record, False) + assert "sess-sf-002" not in single_flight._inflight + + +@pytest.mark.asyncio +async def test_loader_exception_propagates_to_all_waiters(): + """An exception from the loader reaches the leader and every follower.""" + call_count = 0 + gate = asyncio.Event() + + class LoaderBoom(RuntimeError): + pass + + async def loader() -> Tuple[Optional[SessionRecord], bool]: + nonlocal call_count + call_count += 1 + await gate.wait() + raise LoaderBoom("cognito exploded") + + async def release_after_followers_attach() -> None: + await asyncio.sleep(0.05) + gate.set() + + tasks = [ + asyncio.create_task(single_flight.resolve_once("sess-sf-003", loader)) + for _ in range(5) + ] + releaser = asyncio.create_task(release_after_followers_attach()) + + results = await asyncio.gather(*tasks, return_exceptions=True) + await releaser + + assert call_count == 1 + assert len(results) == 5 + for outcome in results: + assert isinstance(outcome, LoaderBoom) + assert str(outcome) == "cognito exploded" + + +@pytest.mark.asyncio +async def test_registry_entry_removed_after_exception_so_next_call_is_fresh_leader(): + """After a loader failure, the next call must start a new leader.""" + attempts = 0 + + async def failing_loader() -> Tuple[Optional[SessionRecord], bool]: + nonlocal attempts + attempts += 1 + raise ValueError("transient ddb failure") + + with pytest.raises(ValueError): + await single_flight.resolve_once("sess-sf-004", failing_loader) + + # Registry entry must be gone so the next call is a new leader. + assert "sess-sf-004" not in single_flight._inflight + + record = _make_record("sess-sf-004") + + async def succeeding_loader() -> Tuple[Optional[SessionRecord], bool]: + nonlocal attempts + attempts += 1 + return record, False + + result = await single_flight.resolve_once("sess-sf-004", succeeding_loader) + + assert result == (record, False) + assert attempts == 2, "both loaders ran; the failure did not sticky-cache" + assert "sess-sf-004" not in single_flight._inflight + + +@pytest.mark.asyncio +async def test_distinct_sessions_are_independent(): + """Two different `session_id`s run two independent loader invocations.""" + calls: list[str] = [] + record_a = _make_record("sess-A") + record_b = _make_record("sess-B") + + async def loader_for(session_id: str, record: SessionRecord): + async def _loader() -> Tuple[Optional[SessionRecord], bool]: + calls.append(session_id) + # Small sleep to encourage interleaving. + await asyncio.sleep(0.01) + return record, False + + return _loader + + loader_a = await loader_for("sess-A", record_a) + loader_b = await loader_for("sess-B", record_b) + + result_a, result_b = await asyncio.gather( + single_flight.resolve_once("sess-A", loader_a), + single_flight.resolve_once("sess-B", loader_b), + ) + + assert result_a == (record_a, False) + assert result_b == (record_b, False) + assert sorted(calls) == ["sess-A", "sess-B"], "each session's loader runs exactly once" + assert "sess-A" not in single_flight._inflight + assert "sess-B" not in single_flight._inflight + + +@pytest.mark.asyncio +async def test_clear_cookie_flag_is_preserved(): + """`resolve_once` must faithfully propagate the `clear_cookie` bool.""" + + async def loader_none_clear() -> Tuple[Optional[SessionRecord], bool]: + return None, True + + result = await single_flight.resolve_once("sess-sf-005", loader_none_clear) + assert result == (None, True) diff --git a/backend/tests/auth/test_dependencies.py b/backend/tests/auth/test_dependencies.py index c336d115..5e4fcf89 100644 --- a/backend/tests/auth/test_dependencies.py +++ b/backend/tests/auth/test_dependencies.py @@ -1,22 +1,18 @@ """Tests for FastAPI auth dependencies. Covers: -- get_current_user: Bearer token validation via CognitoJWTValidator - get_current_user_trusted: JWT decode without signature verification - get_current_user_id: convenience wrapper returning user_id string Requirements: 10.5, 10.6 """ -import time -from unittest.mock import AsyncMock, MagicMock, patch +from unittest.mock import MagicMock, patch -import jwt as pyjwt import pytest from fastapi import HTTPException from apis.shared.auth.dependencies import ( - get_current_user, get_current_user_id, get_current_user_trusted, ) @@ -34,107 +30,6 @@ def _bearer(token: str): return creds -# --------------------------------------------------------------------------- -# get_current_user tests -# --------------------------------------------------------------------------- - - -class TestGetCurrentUser: - """Tests for the get_current_user dependency (Cognito-based).""" - - @pytest.mark.asyncio - async def test_valid_bearer_token(self, make_jwt, make_user): - """Req 10.5: valid Bearer token validated by CognitoJWTValidator, returns User with raw_token.""" - token = make_jwt() - expected_user = make_user(raw_token=None) - - mock_validator = MagicMock() - mock_validator.validate_token = MagicMock(return_value=expected_user) - - with patch( - "apis.shared.auth.dependencies._get_cognito_validator", - return_value=mock_validator, - ), patch( - "apis.shared.auth.dependencies._get_user_sync_service", - return_value=None, - ): - user = await get_current_user(credentials=_bearer(token)) - - assert isinstance(user, User) - assert user.raw_token == token - assert user.user_id == expected_user.user_id - mock_validator.validate_token.assert_called_once_with(token) - - @pytest.mark.asyncio - async def test_no_credentials_401(self): - """Req 10.5: None credentials raises 401 with WWW-Authenticate header.""" - with pytest.raises(HTTPException) as exc_info: - await get_current_user(credentials=None) - - assert exc_info.value.status_code == 401 - assert "WWW-Authenticate" in (exc_info.value.headers or {}) - - @pytest.mark.asyncio - async def test_failed_validation_401(self, make_jwt): - """Req 10.5: token that fails Cognito validation raises 401.""" - token = make_jwt() - - mock_validator = MagicMock() - mock_validator.validate_token = MagicMock( - side_effect=HTTPException(status_code=401, detail="Invalid token signature.") - ) - - with patch( - "apis.shared.auth.dependencies._get_cognito_validator", - return_value=mock_validator, - ), patch( - "apis.shared.auth.dependencies._get_user_sync_service", - return_value=None, - ): - with pytest.raises(HTTPException) as exc_info: - await get_current_user(credentials=_bearer(token)) - - assert exc_info.value.status_code == 401 - - @pytest.mark.asyncio - async def test_no_validator_500(self, make_jwt): - """Req 10.6: no Cognito validator available raises 500.""" - token = make_jwt() - - with patch( - "apis.shared.auth.dependencies._get_cognito_validator", - return_value=None, - ): - with pytest.raises(HTTPException) as exc_info: - await get_current_user(credentials=_bearer(token)) - - assert exc_info.value.status_code == 500 - assert "Authentication service not configured" in exc_info.value.detail - - @pytest.mark.asyncio - async def test_unexpected_exception_401(self, make_jwt): - """Unexpected exception during validation raises 401.""" - token = make_jwt() - - mock_validator = MagicMock() - mock_validator.validate_token = MagicMock( - side_effect=RuntimeError("unexpected") - ) - - with patch( - "apis.shared.auth.dependencies._get_cognito_validator", - return_value=mock_validator, - ), patch( - "apis.shared.auth.dependencies._get_user_sync_service", - return_value=None, - ): - with pytest.raises(HTTPException) as exc_info: - await get_current_user(credentials=_bearer(token)) - - assert exc_info.value.status_code == 401 - assert exc_info.value.detail == "Authentication failed." - - # --------------------------------------------------------------------------- # get_current_user_trusted tests # --------------------------------------------------------------------------- @@ -260,24 +155,11 @@ class TestGetCurrentUserId: """Tests for the get_current_user_id dependency.""" @pytest.mark.asyncio - async def test_returns_string(self, make_jwt, make_user): - """get_current_user_id returns the user_id string.""" - token = make_jwt() + async def test_returns_string(self, make_user): + """get_current_user_id returns the resolved user's user_id.""" expected_user = make_user(user_id="uid-42") - mock_validator = MagicMock() - mock_validator.validate_token = MagicMock(return_value=expected_user) - - with patch( - "apis.shared.auth.dependencies._get_cognito_validator", - return_value=mock_validator, - ), patch( - "apis.shared.auth.dependencies._get_user_sync_service", - return_value=None, - ): - user_id = await get_current_user_id( - user=await get_current_user(credentials=_bearer(token)) - ) + user_id = await get_current_user_id(user=expected_user) assert user_id == "uid-42" assert isinstance(user_id, str) diff --git a/backend/tests/auth/test_skip_auth.py b/backend/tests/auth/test_skip_auth.py new file mode 100644 index 00000000..bbf3a299 --- /dev/null +++ b/backend/tests/auth/test_skip_auth.py @@ -0,0 +1,252 @@ +"""Tests for the SKIP_AUTH=true local-dev bypass. + +Covers: +- `_skip_auth_user()` returns None when disabled, fake User when enabled +- All three auth dependencies bypass when SKIP_AUTH=true +- `_validate_skip_auth_or_raise()` accepts localhost-only CORS_ORIGINS, + rejects empty CORS_ORIGINS, rejects any non-localhost origin +- The CI-guard regex matches realistic leak strings and skips the + legitimate references in dependencies.py / main.py +""" + +import importlib +import re +from unittest.mock import MagicMock, patch + +import pytest +from fastapi import HTTPException + +from apis.shared.auth.dependencies import ( + _skip_auth_user, + get_current_user_from_session, + get_current_user_trusted, +) +from apis.shared.auth.models import User + + +# --------------------------------------------------------------------------- +# Env helpers +# --------------------------------------------------------------------------- + +@pytest.fixture +def clean_skip_auth_env(monkeypatch): + """Clear all SKIP_AUTH_* env vars so each test starts from a known state.""" + for key in ( + "SKIP_AUTH", + "SKIP_AUTH_ROLES", + "SKIP_AUTH_USER_ID", + "SKIP_AUTH_EMAIL", + "CORS_ORIGINS", + ): + monkeypatch.delenv(key, raising=False) + + +# --------------------------------------------------------------------------- +# _skip_auth_user +# --------------------------------------------------------------------------- + + +class TestSkipAuthUser: + """Tests for the `_skip_auth_user()` helper.""" + + def test_returns_none_when_unset(self, clean_skip_auth_env): + assert _skip_auth_user() is None + + @pytest.mark.parametrize("value", ["false", "0", "", "no", "FALSE"]) + def test_returns_none_when_falsey(self, clean_skip_auth_env, monkeypatch, value): + monkeypatch.setenv("SKIP_AUTH", value) + assert _skip_auth_user() is None + + @pytest.mark.parametrize("value", ["true", "TRUE", "True"]) + def test_returns_user_when_true(self, clean_skip_auth_env, monkeypatch, value): + monkeypatch.setenv("SKIP_AUTH", value) + user = _skip_auth_user() + assert isinstance(user, User) + assert user.user_id == "local-dev" + assert user.email == "dev@local" + assert user.name == "Local Dev" + assert user.roles == ["admin"] + + def test_overrides_via_env(self, clean_skip_auth_env, monkeypatch): + monkeypatch.setenv("SKIP_AUTH", "true") + monkeypatch.setenv("SKIP_AUTH_USER_ID", "phil") + monkeypatch.setenv("SKIP_AUTH_EMAIL", "phil@example.com") + monkeypatch.setenv("SKIP_AUTH_ROLES", "admin,DotNetDevelopers, ,QA ") + + user = _skip_auth_user() + assert user is not None + assert user.user_id == "phil" + assert user.email == "phil@example.com" + # whitespace-only entries filtered, surrounding whitespace stripped + assert user.roles == ["admin", "DotNetDevelopers", "QA"] + + +# --------------------------------------------------------------------------- +# Dependency bypass +# --------------------------------------------------------------------------- + + +class TestDependencyBypass: + """Tests that the bypass short-circuits each auth dependency.""" + + @pytest.mark.asyncio + async def test_get_current_user_from_session_bypassed( + self, clean_skip_auth_env, monkeypatch + ): + monkeypatch.setenv("SKIP_AUTH", "true") + # Build a request whose state.bff_session is unset; without the + # bypass this would 401, with the bypass we get the fake user. + request = MagicMock() + request.state = MagicMock(spec=[]) # no bff_session attr + + user = await get_current_user_from_session(request) + assert user.user_id == "local-dev" + + @pytest.mark.asyncio + async def test_get_current_user_trusted_bypassed( + self, clean_skip_auth_env, monkeypatch + ): + monkeypatch.setenv("SKIP_AUTH", "true") + # No credentials supplied — without the bypass this 401s. + user = await get_current_user_trusted(credentials=None) + assert user.user_id == "local-dev" + + @pytest.mark.asyncio + async def test_get_current_user_from_session_still_401_when_disabled( + self, clean_skip_auth_env + ): + """Sanity check: with SKIP_AUTH unset, missing session still 401.""" + request = MagicMock() + request.state = MagicMock(spec=[]) # no bff_session attr + with pytest.raises(HTTPException) as exc: + await get_current_user_from_session(request) + assert exc.value.status_code == 401 + + +# --------------------------------------------------------------------------- +# Startup guard +# --------------------------------------------------------------------------- + + +def _import_main_module(): + """Import (and reload) apis.app_api.main so it picks up current env. + + The module calls `load_dotenv(..., override=True)` at import time, + which would clobber monkeypatched env vars on reload. Patch the + upstream symbol so the `from dotenv import load_dotenv` re-binding + inside the module reload also picks up the no-op. + """ + with patch("dotenv.load_dotenv", lambda *a, **kw: None): + import apis.app_api.main as m + return importlib.reload(m) + + +class TestStartupGuard: + """Tests for `_validate_skip_auth_or_raise()` in app_api/main.py.""" + + def test_noop_when_skip_auth_off(self, clean_skip_auth_env): + m = _import_main_module() + # Doesn't raise even with no CORS_ORIGINS — guard is a no-op. + m._validate_skip_auth_or_raise() + + @pytest.mark.parametrize( + "origins", + [ + "http://localhost:4200", + "http://localhost:4200,http://127.0.0.1:8000", + "http://[::1]:4200", + "http://0.0.0.0:4200", + "http://localhost:4200, http://127.0.0.1:8000 ", # whitespace tolerated + ], + ) + def test_accepts_localhost_origins( + self, clean_skip_auth_env, monkeypatch, origins + ): + monkeypatch.setenv("SKIP_AUTH", "true") + monkeypatch.setenv("CORS_ORIGINS", origins) + m = _import_main_module() + m._validate_skip_auth_or_raise() # no raise + + def test_rejects_empty_cors_origins(self, clean_skip_auth_env, monkeypatch): + monkeypatch.setenv("SKIP_AUTH", "true") + monkeypatch.setenv("CORS_ORIGINS", "") + m = _import_main_module() + with pytest.raises(RuntimeError, match="localhost"): + m._validate_skip_auth_or_raise() + + def test_rejects_unset_cors_origins(self, clean_skip_auth_env, monkeypatch): + monkeypatch.setenv("SKIP_AUTH", "true") + # CORS_ORIGINS deliberately unset + m = _import_main_module() + with pytest.raises(RuntimeError, match="localhost"): + m._validate_skip_auth_or_raise() + + @pytest.mark.parametrize( + "origins", + [ + "https://app.example.com", + "http://localhost:4200,https://app.example.com", # one bad apple + "https://prod.boisestate.edu", + ], + ) + def test_rejects_non_localhost( + self, clean_skip_auth_env, monkeypatch, origins + ): + monkeypatch.setenv("SKIP_AUTH", "true") + monkeypatch.setenv("CORS_ORIGINS", origins) + m = _import_main_module() + with pytest.raises(RuntimeError, match="localhost"): + m._validate_skip_auth_or_raise() + + +# --------------------------------------------------------------------------- +# CI-guard regex +# --------------------------------------------------------------------------- + + +class TestCIGuardPattern: + """Tests that mirror the grep pattern in skip-auth-guard.yml. + + The CI workflow uses `grep -E` with this pattern; we validate the + same regex against representative leak strings and the legitimate + references in our own source so a future refactor of the workflow + has a behavioral spec to test against. + """ + + # Mirrors the PATTERN in .github/workflows/skip-auth-guard.yml + PATTERN = re.compile(r"""SKIP_AUTH[ \t]*[:=][ \t]*["']*true""") + + @pytest.mark.parametrize( + "leak", + [ + "SKIP_AUTH=true", + 'SKIP_AUTH: "true"', + "SKIP_AUTH: true", + "SKIP_AUTH:true", + "SKIP_AUTH: 'true'", + " SKIP_AUTH = true", + 'SKIP_AUTH="true"', + "ENV SKIP_AUTH=true", # Dockerfile + " SKIP_AUTH: 'true' # in some yaml", + ], + ) + def test_matches_leak_strings(self, leak): + assert self.PATTERN.search(leak) is not None, f"missed leak: {leak!r}" + + @pytest.mark.parametrize( + "benign", + [ + 'SKIP_AUTH = "false"', + "SKIP_AUTH=false", + "# Document SKIP_AUTH behaviour", + 'os.environ.get("SKIP_AUTH", "")', + 'if os.environ.get("SKIP_AUTH", "").lower() == "true":', + ], + ) + def test_skips_benign_strings(self, benign): + # The legitimate dependencies.py / main.py references compare against + # "true" but don't *assign* SKIP_AUTH=true, so they shouldn't match. + # The one exception is the inline comparison string "== \"true\"" — + # which the workflow excludes via path-based filtering, not regex. + # We only assert the pattern itself doesn't trip on these forms. + assert self.PATTERN.search(benign) is None, f"false positive: {benign!r}" diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index 5be3a462..1bb58882 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -4,10 +4,26 @@ import sys from pathlib import Path +import pytest + # Ensure AWS region is set so that module-level boto3 calls don't fail # during import (e.g. agents.main_agent.quota -> boto3.resource('dynamodb')) os.environ.setdefault("AWS_DEFAULT_REGION", "us-east-1") +# botocore >= 1.43 accesses Credentials.account_id during endpoint +# construction. On a RefreshableCredentials object (e.g. resolved from a +# real SSO profile) that property forces a credential _refresh() → +# GetRoleCredentials, which moto does not implement, so mocked AWS calls +# fail. Pin static dummy credentials so the chain builds a non-refreshable +# Credentials object instead. The matching AWS_PROFILE scrub is done +# per-test below (a process-wide pop here is not enough: tests that reload +# `apis.app_api.main` run load_dotenv(override=True), which re-injects +# AWS_PROFILE from backend/src/.env mid-suite). Mirrors moto's practice. +os.environ.setdefault("AWS_ACCESS_KEY_ID", "testing") +os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "testing") +os.environ.setdefault("AWS_SESSION_TOKEN", "testing") +os.environ.setdefault("AWS_SECURITY_TOKEN", "testing") + # Add backend/src to Python path for imports # This file is in backend/tests/, so we need to go up one level to backend/ BACKEND_DIR = Path(__file__).parent.parent @@ -16,3 +32,89 @@ if str(SRC_DIR) not in sys.path: sys.path.insert(0, str(SRC_DIR)) + +# Scrub SKIP_AUTH bleed from local .env. Some tests reload +# `apis.app_api.main`, which calls `load_dotenv(override=True)` and +# clobbers process env with whatever `backend/src/.env` has set — +# typically `SKIP_AUTH=true` for local dev. Without this fixture every +# auth-aware test downstream of that reload returns the fake bypass +# user. Tests that need SKIP_AUTH on can still set it via monkeypatch +# (test-local setenv runs after this autouse delenv). +# +# Manages os.environ directly rather than depending on monkeypatch so +# this autouse fixture doesn't perturb fixture-teardown ordering for +# tests that already use monkeypatch + their own autouse fixtures +# (e.g. tests/apis/app_api/test_connectors_routes.py). +_SKIP_AUTH_ENV_KEYS = ( + "SKIP_AUTH", + "SKIP_AUTH_ROLES", + "SKIP_AUTH_USER_ID", + "SKIP_AUTH_EMAIL", +) + + +@pytest.fixture(autouse=True) +def _clear_skip_auth_env(): + saved = {k: os.environ.pop(k, None) for k in _SKIP_AUTH_ENV_KEYS} + try: + yield + finally: + for k, v in saved.items(): + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + + +# Same load_dotenv(override=True) bleed as above, but for AWS_PROFILE. +# backend/src/.env sets a real SSO profile for local dev; once a test +# reloads `apis.app_api.main` it lands in process env and every later +# test that builds a boto3 client resolves SSO credentials. Under +# botocore >= 1.43 that fails all mocked AWS calls (see import-time note). +# Scrub per-test so the static dummy credentials win the provider chain. +_AWS_PROFILE_ENV_KEYS = ( + "AWS_PROFILE", + "AWS_DEFAULT_PROFILE", +) + + +@pytest.fixture(autouse=True) +def _clear_aws_profile_env(): + saved = {k: os.environ.pop(k, None) for k in _AWS_PROFILE_ENV_KEYS} + try: + yield + finally: + for k, v in saved.items(): + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + + +# Same load_dotenv(override=True) bleed again, for the infra-resource +# config families. backend/src/.env sets real DYNAMODB_*_TABLE_NAME and +# COGNITO_* identifiers for local dev; once a test reloads +# `apis.app_api.main` they land in process env. Repositories/services gate +# their "configured" flag on `param or os.getenv("DYNAMODB_..."/"COGNITO_...")`, +# so a leaked value makes "disabled when unconfigured" tests construct a +# live client and attempt real AWS calls. Tests always inject their own +# resource names via moto fixtures, so scrub the whole family per-test. +_ENV_CONFIG_BLEED_PREFIXES = ( + "DYNAMODB_", + "COGNITO_", +) + + +@pytest.fixture(autouse=True) +def _clear_env_config_bleed(): + saved = { + k: os.environ.pop(k) + for k in list(os.environ) + if k.startswith(_ENV_CONFIG_BLEED_PREFIXES) + } + try: + yield + finally: + for k, v in saved.items(): + os.environ[k] = v + diff --git a/backend/tests/costs/test_calculator.py b/backend/tests/costs/test_calculator.py new file mode 100644 index 00000000..0d6e8120 --- /dev/null +++ b/backend/tests/costs/test_calculator.py @@ -0,0 +1,282 @@ +"""Unit tests for CostCalculator — the source-of-truth for all USD math. + +These tests pin the per-bucket pricing formula, the cache-savings derivation, +and the input-validation predicates. The aggregator and storage tests cover +this code transitively, but only through mocks; this module is the only +place the math itself is asserted directly. + +Conventions for cases: + - "Sonnet 4.5 pricing" reflects Bedrock's published rates so a regression + in the formula would be visible in dollar terms a reader can sanity-check. + - Floats are compared with ``pytest.approx`` to avoid 1e-15 drift. +""" + +import pytest + +from apis.shared.costs.calculator import CostCalculator +from apis.shared.costs.models import CostBreakdown + + +# Bedrock rates for Claude Sonnet 4.5 ($/Mtok). Used as the "realistic" +# baseline so dollar amounts in tests can be compared to a published source. +SONNET_45_PRICING = { + "inputPricePerMtok": 3.0, + "outputPricePerMtok": 15.0, + "cacheWritePricePerMtok": 3.75, + "cacheReadPricePerMtok": 0.30, +} + + +class TestCalculateMessageCostBasic: + """Core formula: per-bucket pricing summed into total.""" + + def test_input_only(self): + usage = {"inputTokens": 1_000_000, "outputTokens": 0} + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + assert total == pytest.approx(3.0) + assert breakdown.input_cost == pytest.approx(3.0) + assert breakdown.output_cost == 0.0 + assert breakdown.cache_read_cost == 0.0 + assert breakdown.cache_write_cost == 0.0 + + def test_output_only(self): + usage = {"inputTokens": 0, "outputTokens": 1_000_000} + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + assert total == pytest.approx(15.0) + assert breakdown.output_cost == pytest.approx(15.0) + assert breakdown.input_cost == 0.0 + + def test_input_and_output_no_cache(self): + """Realistic short turn: 1k input + 500 output on Sonnet 4.5.""" + usage = {"inputTokens": 1_000, "outputTokens": 500} + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + # 1000/1M * 3.00 + 500/1M * 15.00 = 0.003 + 0.0075 = 0.0105 + assert total == pytest.approx(0.0105) + assert breakdown.input_cost == pytest.approx(0.003) + assert breakdown.output_cost == pytest.approx(0.0075) + + def test_breakdown_components_sum_to_total(self): + """The total in the breakdown must equal the sum of its parts.""" + usage = { + "inputTokens": 1_234, + "outputTokens": 567, + "cacheReadInputTokens": 8_910, + "cacheWriteInputTokens": 2_345, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + component_sum = ( + breakdown.input_cost + + breakdown.output_cost + + breakdown.cache_read_cost + + breakdown.cache_write_cost + ) + assert breakdown.total_cost == pytest.approx(component_sum) + assert total == pytest.approx(component_sum) + + +class TestCalculateMessageCostWithCache: + """Cache buckets price separately and add to the total.""" + + def test_cache_read_only(self): + """A subsequent turn hitting the prompt cache.""" + usage = { + "inputTokens": 100, # uncached suffix + "outputTokens": 200, + "cacheReadInputTokens": 5_000, # cached prefix + "cacheWriteInputTokens": 0, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + # input: 100/1M * 3 = 0.0003 + # output: 200/1M * 15 = 0.003 + # cache_read: 5000/1M * 0.30 = 0.0015 + assert breakdown.input_cost == pytest.approx(0.0003) + assert breakdown.output_cost == pytest.approx(0.003) + assert breakdown.cache_read_cost == pytest.approx(0.0015) + assert breakdown.cache_write_cost == 0.0 + assert total == pytest.approx(0.0048) + + def test_cache_write_only(self): + """The first turn that establishes the cache pays the write premium.""" + usage = { + "inputTokens": 0, + "outputTokens": 100, + "cacheReadInputTokens": 0, + "cacheWriteInputTokens": 5_000, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + # cache_write: 5000/1M * 3.75 = 0.01875 + # output: 100/1M * 15 = 0.0015 + assert breakdown.cache_write_cost == pytest.approx(0.01875) + assert breakdown.output_cost == pytest.approx(0.0015) + assert breakdown.cache_read_cost == 0.0 + assert total == pytest.approx(0.02025) + + def test_cache_read_and_write_mixed(self): + """A turn that hits part of the cache and writes a new section.""" + usage = { + "inputTokens": 200, + "outputTokens": 300, + "cacheReadInputTokens": 10_000, + "cacheWriteInputTokens": 2_000, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + assert breakdown.input_cost == pytest.approx(200 / 1_000_000 * 3.0) + assert breakdown.output_cost == pytest.approx(300 / 1_000_000 * 15.0) + assert breakdown.cache_read_cost == pytest.approx(10_000 / 1_000_000 * 0.30) + assert breakdown.cache_write_cost == pytest.approx(2_000 / 1_000_000 * 3.75) + assert total == pytest.approx( + breakdown.input_cost + + breakdown.output_cost + + breakdown.cache_read_cost + + breakdown.cache_write_cost + ) + + def test_docstring_example_holds(self): + """The docstring example must match the implementation.""" + usage = { + "inputTokens": 1_000, + "outputTokens": 500, + "cacheReadInputTokens": 200, + "cacheWriteInputTokens": 100, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + assert breakdown.input_cost == pytest.approx(0.003) + assert breakdown.output_cost == pytest.approx(0.0075) + assert breakdown.cache_read_cost == pytest.approx(0.00006) + assert breakdown.cache_write_cost == pytest.approx(0.000375) + assert total == pytest.approx(0.010935) + + +class TestCalculateMessageCostDefensive: + """Missing or None fields should degrade to 0, never raise.""" + + def test_missing_pricing_fields_default_to_zero(self): + """Cache prices may be absent for non-Bedrock providers.""" + pricing = {"inputPricePerMtok": 1.0, "outputPricePerMtok": 2.0} + usage = { + "inputTokens": 1_000_000, + "outputTokens": 1_000_000, + "cacheReadInputTokens": 1_000_000, + "cacheWriteInputTokens": 1_000_000, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, pricing) + assert breakdown.input_cost == pytest.approx(1.0) + assert breakdown.output_cost == pytest.approx(2.0) + assert breakdown.cache_read_cost == 0.0 + assert breakdown.cache_write_cost == 0.0 + assert total == pytest.approx(3.0) + + def test_none_pricing_values_default_to_zero(self): + """A managed-model row with explicit None for cache prices must not raise.""" + pricing = { + "inputPricePerMtok": 3.0, + "outputPricePerMtok": 15.0, + "cacheReadPricePerMtok": None, + "cacheWritePricePerMtok": None, + } + usage = { + "inputTokens": 1_000, + "outputTokens": 500, + "cacheReadInputTokens": 1_000, + "cacheWriteInputTokens": 1_000, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, pricing) + assert breakdown.cache_read_cost == 0.0 + assert breakdown.cache_write_cost == 0.0 + + def test_none_usage_values_default_to_zero(self): + usage = { + "inputTokens": None, + "outputTokens": None, + "cacheReadInputTokens": None, + "cacheWriteInputTokens": None, + } + total, breakdown = CostCalculator.calculate_message_cost(usage, SONNET_45_PRICING) + assert total == 0.0 + assert breakdown.input_cost == 0.0 + assert breakdown.output_cost == 0.0 + assert breakdown.cache_read_cost == 0.0 + assert breakdown.cache_write_cost == 0.0 + + def test_empty_usage_and_pricing(self): + total, breakdown = CostCalculator.calculate_message_cost({}, {}) + assert total == 0.0 + assert isinstance(breakdown, CostBreakdown) + + +class TestCalculateCacheSavings: + """Cache savings = (input_price - cache_read_price) * read_tokens / 1M.""" + + def test_typical_savings(self): + """200 read tokens at Sonnet 4.5 rates.""" + savings = CostCalculator.calculate_cache_savings(200, 3.0, 0.30) + # standard: 200/1M * 3 = 0.0006; cached: 200/1M * 0.30 = 0.00006 + assert savings == pytest.approx(0.00054) + + def test_zero_reads_returns_zero(self): + assert CostCalculator.calculate_cache_savings(0, 3.0, 0.30) == 0.0 + + def test_none_reads_returns_zero(self): + """``None`` is the realistic shape from a model that didn't hit cache.""" + assert CostCalculator.calculate_cache_savings(None, 3.0, 0.30) == 0.0 + + def test_none_prices_default_to_zero(self): + """None prices must not raise — the formula collapses cleanly to 0.""" + assert CostCalculator.calculate_cache_savings(1_000, None, None) == 0.0 + + def test_savings_equals_full_input_cost_when_cache_is_free(self): + """If cache reads are priced at 0, savings is the full input cost.""" + savings = CostCalculator.calculate_cache_savings(1_000_000, 3.0, 0.0) + assert savings == pytest.approx(3.0) + + +class TestValidatePricing: + """validate_pricing requires inputPricePerMtok and outputPricePerMtok with non-None values.""" + + def test_complete_pricing_is_valid(self): + assert CostCalculator.validate_pricing(SONNET_45_PRICING) is True + + def test_minimal_pricing_is_valid(self): + """Cache fields are not required.""" + assert CostCalculator.validate_pricing({ + "inputPricePerMtok": 1.0, + "outputPricePerMtok": 2.0, + }) is True + + def test_missing_input_price_is_invalid(self): + assert CostCalculator.validate_pricing({"outputPricePerMtok": 2.0}) is False + + def test_missing_output_price_is_invalid(self): + assert CostCalculator.validate_pricing({"inputPricePerMtok": 1.0}) is False + + def test_none_value_is_invalid(self): + assert CostCalculator.validate_pricing({ + "inputPricePerMtok": None, + "outputPricePerMtok": 2.0, + }) is False + + +class TestValidateUsage: + """validate_usage requires inputTokens and outputTokens with non-None values.""" + + def test_complete_usage_is_valid(self): + assert CostCalculator.validate_usage({ + "inputTokens": 100, + "outputTokens": 50, + }) is True + + def test_zero_values_are_valid(self): + """Zero is a real measurement, not an absence.""" + assert CostCalculator.validate_usage({ + "inputTokens": 0, + "outputTokens": 0, + }) is True + + def test_missing_input_tokens_is_invalid(self): + assert CostCalculator.validate_usage({"outputTokens": 50}) is False + + def test_none_value_is_invalid(self): + assert CostCalculator.validate_usage({ + "inputTokens": None, + "outputTokens": 50, + }) is False diff --git a/backend/tests/lambdas/test_artifact_render.py b/backend/tests/lambdas/test_artifact_render.py new file mode 100644 index 00000000..5817c760 --- /dev/null +++ b/backend/tests/lambdas/test_artifact_render.py @@ -0,0 +1,550 @@ +"""Tests for the artifact render Lambda. + +Two layers: + * Token verification matrix — pure stdlib HS256 logic, no AWS. + * Handler integration — full request flow against moto-backed + Secrets Manager, DynamoDB, and S3. +""" + +from __future__ import annotations + +import base64 +import hashlib +import hmac +import json +import time +from typing import Any + +import boto3 +import pytest +from moto import mock_aws + +from lambdas.artifact_render import handler + +KEY = "test-signing-key-44-chars-of-entropy-aaaaaaa" + + +def _b64url(data: bytes) -> str: + return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii") + + +def _mint( + claims: dict[str, Any], + *, + key: str = KEY, + alg: str = "HS256", + tamper_sig: bool = False, +) -> str: + header = _b64url(json.dumps({"alg": alg, "typ": "JWT"}).encode()) + payload = _b64url(json.dumps(claims).encode()) + signing_input = f"{header}.{payload}".encode("ascii") + sig = hmac.new(key.encode(), signing_input, hashlib.sha256).digest() + sig_b64 = _b64url(sig) + if tamper_sig: + sig_b64 = ("A" if sig_b64[0] != "A" else "B") + sig_b64[1:] + return f"{header}.{payload}.{sig_b64}" + + +def _valid_claims(**overrides: Any) -> dict[str, Any]: + now = int(time.time()) + base = { + "sub": "user-123", + "aid": "artifact-abc", + "ver": 1, + "sid": "session-xyz", + "iss": "app-api", + "aud": "artifact-render", + "iat": now, + "exp": now + 90, + } + base.update(overrides) + return base + + +@pytest.fixture(autouse=True) +def _reset_module_state(monkeypatch: pytest.MonkeyPatch) -> None: + """Each test starts from clean module-scoped caches.""" + monkeypatch.setattr(handler, "_cached_signing_key", None) + monkeypatch.setattr(handler, "_secrets_client", None) + monkeypatch.setattr(handler, "_s3_client", None) + monkeypatch.setattr(handler, "_ddb_table", None) + + +# -------------------------------------------------------------------------- +# Token verification matrix (no AWS — signing key injected directly). +# -------------------------------------------------------------------------- + + +@pytest.fixture +def _injected_key(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(handler, "_cached_signing_key", KEY) + + +def test_valid_token_returns_claims(_injected_key: None) -> None: + claims = handler._verify_token(_mint(_valid_claims())) + assert claims["sub"] == "user-123" + assert claims["aid"] == "artifact-abc" + assert claims["ver"] == 1 + + +def test_tampered_signature_rejected(_injected_key: None) -> None: + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(), tamper_sig=True)) + + +def test_wrong_signing_key_rejected(_injected_key: None) -> None: + forged = _mint(_valid_claims(), key="a-different-key") + with pytest.raises(handler._TokenError): + handler._verify_token(forged) + + +def test_alg_none_rejected(_injected_key: None) -> None: + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(), alg="none")) + + +def test_alg_confusion_rejected(_injected_key: None) -> None: + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(), alg="HS512")) + + +def test_expired_token_rejected(_injected_key: None) -> None: + now = int(time.time()) + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(iat=now - 200, exp=now - 100))) + + +def test_expiry_within_leeway_accepted(_injected_key: None) -> None: + now = int(time.time()) + claims = handler._verify_token(_mint(_valid_claims(iat=now - 4, exp=now - 3))) + assert claims["sub"] == "user-123" + + +def test_future_iat_rejected(_injected_key: None) -> None: + now = int(time.time()) + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(iat=now + 100, exp=now + 200))) + + +def test_overlong_lifetime_rejected(_injected_key: None) -> None: + now = int(time.time()) + over = handler._MAX_TOKEN_LIFETIME_SECONDS + 60 + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(iat=now, exp=now + over))) + + +def test_wrong_issuer_rejected(_injected_key: None) -> None: + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(iss="evil"))) + + +def test_wrong_audience_rejected(_injected_key: None) -> None: + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(aud="some-other-service"))) + + +def test_missing_exp_rejected(_injected_key: None) -> None: + claims = _valid_claims() + del claims["exp"] + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(claims)) + + +def test_missing_iat_rejected(_injected_key: None) -> None: + # `iat` is mandatory — without it the lifetime cap can't be enforced. + claims = _valid_claims() + del claims["iat"] + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(claims)) + + +@pytest.mark.parametrize("bad_iat", ["123", True]) +def test_non_numeric_iat_rejected(_injected_key: None, bad_iat: Any) -> None: + # A string or bool `iat` must not slip past the numeric guard + # (bool is an int subclass). + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(iat=bad_iat))) + + +@pytest.mark.parametrize("bad_ver", [0, -1, True, "1", 1.0]) +def test_invalid_version_rejected(_injected_key: None, bad_ver: Any) -> None: + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(_valid_claims(ver=bad_ver))) + + +@pytest.mark.parametrize("missing", ["sub", "aid"]) +def test_missing_identity_claim_rejected(_injected_key: None, missing: str) -> None: + claims = _valid_claims() + del claims[missing] + with pytest.raises(handler._TokenError): + handler._verify_token(_mint(claims)) + + +@pytest.mark.parametrize("token", ["", "a.b", "a.b.c.d", "not-a-token"]) +def test_malformed_token_rejected(_injected_key: None, token: str) -> None: + with pytest.raises(handler._TokenError): + handler._verify_token(token) + + +def test_non_dict_header_rejected(_injected_key: None) -> None: + # Header decodes to a JSON array, not an object. + header = _b64url(json.dumps(["HS256"]).encode()) + payload = _b64url(json.dumps(_valid_claims()).encode()) + sig = _b64url( + hmac.new( + KEY.encode(), f"{header}.{payload}".encode("ascii"), hashlib.sha256 + ).digest() + ) + with pytest.raises(handler._TokenError): + handler._verify_token(f"{header}.{payload}.{sig}") + + +def test_non_dict_payload_rejected(_injected_key: None) -> None: + header = _b64url(json.dumps({"alg": "HS256", "typ": "JWT"}).encode()) + payload = _b64url(json.dumps(["not", "an", "object"]).encode()) + sig = _b64url( + hmac.new( + KEY.encode(), f"{header}.{payload}".encode("ascii"), hashlib.sha256 + ).digest() + ) + with pytest.raises(handler._TokenError): + handler._verify_token(f"{header}.{payload}.{sig}") + + +# -------------------------------------------------------------------------- +# Handler integration (moto-backed Secrets Manager + DynamoDB + S3). +# -------------------------------------------------------------------------- + +SECRET_ARN_NAME = "test-artifact-render-token-key" +TABLE = "test-user-artifacts" +BUCKET = "test-artifacts-content" +CONTENT_KEY = "user-123/artifact-abc/v1/index.html" +DOC = "

hi

" + + +@pytest.fixture +def aws_env(monkeypatch: pytest.MonkeyPatch): + with mock_aws(): + sm = boto3.client("secretsmanager", region_name="us-east-1") + secret = sm.create_secret(Name=SECRET_ARN_NAME, SecretString=KEY) + + ddb = boto3.client("dynamodb", region_name="us-east-1") + ddb.create_table( + TableName=TABLE, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + + s3 = boto3.client("s3", region_name="us-east-1") + s3.create_bucket(Bucket=BUCKET) + s3.put_object(Bucket=BUCKET, Key=CONTENT_KEY, Body=DOC.encode()) + + monkeypatch.setattr(handler, "_RENDER_TOKEN_SECRET_ARN", secret["ARN"]) + monkeypatch.setattr(handler, "_ARTIFACTS_TABLE", TABLE) + monkeypatch.setattr(handler, "_ARTIFACTS_BUCKET", BUCKET) + monkeypatch.setattr(handler, "_FRAME_ANCESTOR", "https://app.example.com") + + yield {"ddb": boto3.resource("dynamodb", region_name="us-east-1")} + + +def _put_record(ddb, **overrides: Any) -> None: + item = { + "PK": "USER#user-123", + "SK": "ARTIFACT#artifact-abc#V#00001", + "storage": "s3", + "content_key": CONTENT_KEY, + "content_type": "text/html; charset=utf-8", + } + item.update(overrides) + ddb.Table(TABLE).put_item(Item=item) + + +def _event( + token: str | None, method: str = "GET", *, download: bool = False +) -> dict[str, Any]: + qsp: dict[str, str] = {} + raw_parts: list[str] = [] + if token: + qsp["t"] = token + raw_parts.append(f"t={token}") + if download: + qsp["download"] = "1" + raw_parts.append("download=1") + return { + "requestContext": {"http": {"method": method}}, + "queryStringParameters": qsp, + "rawQueryString": "&".join(raw_parts), + } + + +def test_happy_path_returns_content(aws_env) -> None: + _put_record(aws_env["ddb"]) + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 200 + assert resp["body"] == DOC + assert resp["headers"]["cache-control"] == "no-store" + assert "frame-ancestors https://app.example.com" in ( + resp["headers"]["content-security-policy"] + ) + + +def test_secret_fetched_from_secrets_manager(aws_env) -> None: + # _cached_signing_key is None (reset fixture) so this exercises the + # real Secrets Manager round-trip, not an injected key. + _put_record(aws_env["ddb"]) + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 200 + + +def test_head_request_omits_body(aws_env) -> None: + _put_record(aws_env["ddb"]) + resp = handler.handler(_event(_mint(_valid_claims()), method="HEAD"), None) + assert resp["statusCode"] == 200 + assert resp["body"] == "" + + +def test_token_from_raw_query_string(aws_env) -> None: + _put_record(aws_env["ddb"]) + token = _mint(_valid_claims()) + event = { + "requestContext": {"http": {"method": "GET"}}, + "queryStringParameters": None, + "rawQueryString": f"t={token}", + } + assert handler.handler(event, None)["statusCode"] == 200 + + +def test_missing_token_is_403(aws_env) -> None: + resp = handler.handler(_event(None), None) + assert resp["statusCode"] == 403 + + +def test_tampered_token_is_403(aws_env) -> None: + _put_record(aws_env["ddb"]) + bad = _mint(_valid_claims(), tamper_sig=True) + assert handler.handler(_event(bad), None)["statusCode"] == 403 + + +def test_non_get_method_is_405(aws_env) -> None: + resp = handler.handler(_event(_mint(_valid_claims()), method="POST"), None) + assert resp["statusCode"] == 405 + + +def test_missing_version_record_is_404(aws_env) -> None: + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 404 + + +def test_unsupported_storage_is_500(aws_env) -> None: + _put_record(aws_env["ddb"], storage="inline") + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 500 + + +def test_record_without_content_key_is_404(aws_env) -> None: + ddb = aws_env["ddb"] + ddb.Table(TABLE).put_item( + Item={ + "PK": "USER#user-123", + "SK": "ARTIFACT#artifact-abc#V#00001", + "storage": "s3", + } + ) + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 404 + + +def test_missing_s3_object_is_404(aws_env) -> None: + _put_record(aws_env["ddb"], content_key="user-123/artifact-abc/v1/gone.html") + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 404 + + +def test_version_pins_exact_sk(aws_env) -> None: + # Token asks for v2; only v1 exists → must 404, never fall back to HEAD. + _put_record(aws_env["ddb"]) + resp = handler.handler(_event(_mint(_valid_claims(ver=2))), None) + assert resp["statusCode"] == 404 + + +@pytest.mark.parametrize( + "stored,served", + [ + ("text/markdown", "text/html; charset=utf-8"), + ("text/markdown; charset=utf-8", "text/html; charset=utf-8"), + ("text/x-markdown", "text/html; charset=utf-8"), + ("TEXT/MARKDOWN", "text/html; charset=utf-8"), + ("text/html; charset=utf-8", "text/html; charset=utf-8"), + ("image/svg+xml", "image/svg+xml"), + ("application/json", "application/json"), + ], +) +def test_serve_content_type_mapping(stored: str, served: str) -> None: + assert handler._serve_content_type(stored) == served + + +def test_markdown_record_served_as_html(aws_env) -> None: + # S3 holds the writer's HTML render wrapper; the row is typed + # text/markdown so the SPA card/list stay truthful. The Lambda must + # serve the exact bytes but with a text/html HTTP content type. + wrapper = "rendered md" + boto3.client("s3", region_name="us-east-1").put_object( + Bucket=BUCKET, Key=CONTENT_KEY, Body=wrapper.encode() + ) + _put_record(aws_env["ddb"], content_type="text/markdown; charset=utf-8") + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 200 + assert resp["headers"]["content-type"] == "text/html; charset=utf-8" + assert resp["body"] == wrapper # bytes are an exact pass-through + + +def test_non_markdown_content_type_served_verbatim(aws_env) -> None: + _put_record(aws_env["ddb"], content_type="image/svg+xml") + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 200 + assert resp["headers"]["content-type"] == "image/svg+xml" + + +def test_oversized_content_is_500(aws_env, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(handler, "_MAX_CONTENT_BYTES", 16) + boto3.client("s3", region_name="us-east-1").put_object( + Bucket=BUCKET, Key=CONTENT_KEY, Body=b"x" * 64 + ) + _put_record(aws_env["ddb"]) + resp = handler.handler(_event(_mint(_valid_claims())), None) + assert resp["statusCode"] == 500 + + +# -------------------------------------------------------------------------- +# Download mode (`?download=1`): attachment disposition, no CSP, gated by +# the same token as render. +# -------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "stored,ext", + [ + ("text/html; charset=utf-8", "html"), + ("text/markdown", "html"), # S3 body is the HTML render wrapper + ("text/x-markdown", "html"), + ("TEXT/MARKDOWN; charset=utf-8", "html"), + ("image/svg+xml", "svg"), + ("application/json", "json"), + ("text/css", "css"), + ("application/javascript", "js"), + ("text/plain", "txt"), + ("application/x-weird", "bin"), + ], +) +def test_download_extension_mapping(stored: str, ext: str) -> None: + assert handler._download_extension(stored) == ext + + +def test_content_disposition_sanitizes_title() -> None: + cd = handler._content_disposition("Q3 / Report: ", "html") + assert cd.startswith('attachment; filename="') + # The ASCII fallback must not carry path/header-hostile characters. + fallback = cd.split('filename="', 1)[1].split('"', 1)[0] + assert fallback.endswith(".html") + for bad in ("/", ":", "<", ">", "\\", "\r", "\n"): + assert bad not in fallback + # The original title is preserved in the RFC 5987 form. + assert "filename*=UTF-8''" in cd + + +def test_content_disposition_defaults_when_title_blank() -> None: + # A whitespace-only title is treated as absent: both forms fall back + # to "artifact" (never a file literally named " .json"). + assert handler._content_disposition(" ", "json") == ( + "attachment; filename=\"artifact.json\"; " + "filename*=UTF-8''artifact.json" + ) + + +def test_content_disposition_preserves_unicode_title() -> None: + cd = handler._content_disposition("résumé", "txt") + # Non-ASCII collapses to '_' in the fallback but survives in filename*. + assert 'filename="r' in cd + assert "filename*=UTF-8''r%C3%A9sum%C3%A9.txt" in cd + + +def test_download_returns_attachment(aws_env) -> None: + _put_record(aws_env["ddb"], title="My Page") + resp = handler.handler( + _event(_mint(_valid_claims()), download=True), None + ) + assert resp["statusCode"] == 200 + assert resp["body"] == DOC + cd = resp["headers"]["content-disposition"] + assert cd.startswith("attachment; ") + assert 'filename="My Page.html"' in cd + assert resp["headers"]["content-type"] == "text/html; charset=utf-8" + assert resp["headers"]["x-content-type-options"] == "nosniff" + assert resp["headers"]["cache-control"] == "no-store" + # An attachment is saved, never framed — no CSP/frame-ancestors. + assert "content-security-policy" not in resp["headers"] + + +def test_download_markdown_uses_html_extension(aws_env) -> None: + wrapper = "rendered md" + boto3.client("s3", region_name="us-east-1").put_object( + Bucket=BUCKET, Key=CONTENT_KEY, Body=wrapper.encode() + ) + _put_record( + aws_env["ddb"], + content_type="text/markdown; charset=utf-8", + title="Notes", + ) + resp = handler.handler( + _event(_mint(_valid_claims()), download=True), None + ) + assert resp["statusCode"] == 200 + assert resp["body"] == wrapper + assert resp["headers"]["content-type"] == "text/html; charset=utf-8" + assert 'filename="Notes.html"' in resp["headers"]["content-disposition"] + + +def test_download_default_filename_when_title_missing(aws_env) -> None: + _put_record(aws_env["ddb"], content_type="application/json") + resp = handler.handler( + _event(_mint(_valid_claims()), download=True), None + ) + assert 'filename="artifact.json"' in resp["headers"]["content-disposition"] + + +def test_head_download_omits_body_keeps_disposition(aws_env) -> None: + _put_record(aws_env["ddb"], title="Doc") + resp = handler.handler( + _event(_mint(_valid_claims()), method="HEAD", download=True), None + ) + assert resp["statusCode"] == 200 + assert resp["body"] == "" + assert resp["headers"]["content-disposition"].startswith("attachment; ") + + +def test_download_still_requires_valid_token(aws_env) -> None: + _put_record(aws_env["ddb"]) + bad = _mint(_valid_claims(), tamper_sig=True) + resp = handler.handler(_event(bad, download=True), None) + assert resp["statusCode"] == 403 + assert "content-disposition" not in resp["headers"] + + +def test_download_flag_from_raw_query_string(aws_env) -> None: + _put_record(aws_env["ddb"], title="Doc") + token = _mint(_valid_claims()) + event = { + "requestContext": {"http": {"method": "GET"}}, + "queryStringParameters": None, + "rawQueryString": f"t={token}&download=1", + } + resp = handler.handler(event, None) + assert resp["statusCode"] == 200 + assert resp["headers"]["content-disposition"].startswith("attachment; ") diff --git a/backend/tests/routes/conftest.py b/backend/tests/routes/conftest.py index 850b2b4d..b023dc3d 100644 --- a/backend/tests/routes/conftest.py +++ b/backend/tests/routes/conftest.py @@ -25,7 +25,7 @@ from fastapi import FastAPI, HTTPException, status from fastapi.testclient import TestClient -from apis.shared.auth.dependencies import get_current_user, get_current_user_from_session +from apis.shared.auth.dependencies import get_current_user_from_session from apis.shared.auth.models import User @@ -96,26 +96,16 @@ def mock_auth_user(app: FastAPI, user: User) -> None: Requirement 1.1: authenticated TestClient with Auth_Dependency overridden. - Overrides BOTH `get_current_user` (Bearer-only, retained for the - `/chat/agent-stream` external-caller route) and - `get_current_user_from_session` (cookie auth — the SPA-facing - surface) so routes can be exercised regardless of which dep they - pull in. Without the cookie override they'd hit the real session - resolution path and 401. + Overrides `get_current_user_from_session` (cookie auth — the only + user-facing auth dependency in `app_api/` after the BFF migration). """ - app.dependency_overrides[get_current_user] = lambda: user app.dependency_overrides[get_current_user_from_session] = lambda: user def mock_no_auth(app: FastAPI) -> None: - """Override the auth dependencies to raise HTTP 401. + """Override the auth dependency to raise HTTP 401. Requirement 1.2: unauthenticated TestClient behaviour. - - Both Bearer (`get_current_user`) and cookie - (`get_current_user_from_session`) dependencies are overridden so the - "no auth provided" assertion holds regardless of which dep the route - uses. """ def _raise_401(): @@ -124,7 +114,6 @@ def _raise_401(): detail="Not authenticated", ) - app.dependency_overrides[get_current_user] = _raise_401 app.dependency_overrides[get_current_user_from_session] = _raise_401 diff --git a/backend/tests/routes/test_chat.py b/backend/tests/routes/test_chat.py index 54c6a444..f8d3fa0e 100644 --- a/backend/tests/routes/test_chat.py +++ b/backend/tests/routes/test_chat.py @@ -3,20 +3,16 @@ Endpoints under test: - POST /chat/generate-title → 200 with generated title - POST /chat/generate-title → 401 for unauthenticated request -- POST /chat/agent-stream → streaming response with text/event-stream -- POST /chat/multimodal → streaming response -Requirements: 5.1, 5.2, 5.3, 5.4 +Requirements: 5.1, 5.2 """ -from unittest.mock import AsyncMock, MagicMock, patch +from unittest.mock import AsyncMock, patch import pytest from fastapi import FastAPI -from fastapi.testclient import TestClient from apis.app_api.chat.routes import router -from tests.routes.conftest import mock_auth_user, mock_no_auth # --------------------------------------------------------------------------- @@ -92,124 +88,3 @@ def test_returns_401_for_unauthenticated(self, app, unauthenticated_client): json={"session_id": "sess-001", "input": "Hello"}, ) assert resp.status_code == 401 - - -# --------------------------------------------------------------------------- -# Requirement 5.3: POST /chat/agent-stream returns streaming response -# --------------------------------------------------------------------------- - - -class TestChatStream: - """POST /chat/agent-stream returns a streaming response.""" - - def test_returns_streaming_response(self, app, make_user, authenticated_client): - """Req 5.3: Should return streaming response with text/event-stream.""" - user = make_user() - client = authenticated_client(app, user) - - # Mock the agent returned by get_agent - mock_agent = MagicMock() - - async def fake_stream(*args, **kwargs): - yield 'event: message_start\ndata: {"role": "assistant"}\n\n' - yield "event: done\ndata: {}\n\n" - - mock_agent.stream_async = fake_stream - mock_agent.session_manager = MagicMock() - mock_agent.session_manager.flush = MagicMock() - - with patch( - "apis.app_api.chat.routes.get_agent", - return_value=mock_agent, - ), patch( - "apis.app_api.chat.routes.get_tool_access_service", - ) as mock_tool_svc, patch( - "apis.app_api.chat.routes.is_quota_enforcement_enabled", - return_value=False, - ), patch( - "apis.app_api.chat.routes.get_session_metadata", - new_callable=AsyncMock, - return_value=None, - ): - mock_tool_access = AsyncMock() - mock_tool_access.check_access_and_filter = AsyncMock( - return_value=(["tool1"], []) - ) - mock_tool_svc.return_value = mock_tool_access - - resp = client.post( - "/chat/agent-stream", - json={ - "session_id": "sess-001", - "message": "Hello, how are you?", - }, - ) - - assert resp.status_code == 200 - assert "text/event-stream" in resp.headers["content-type"] - - def test_returns_401_for_unauthenticated(self, app, unauthenticated_client): - """Req 5.3: Should return 401 when no auth is provided.""" - client = unauthenticated_client(app) - resp = client.post( - "/chat/agent-stream", - json={"session_id": "sess-001", "message": "Hello"}, - ) - assert resp.status_code == 401 - - -# --------------------------------------------------------------------------- -# Requirement 5.4: POST /chat/multimodal returns streaming response -# --------------------------------------------------------------------------- - - -class TestChatMultimodal: - """POST /chat/multimodal returns a streaming response.""" - - def test_returns_streaming_response(self, app, make_user, authenticated_client): - """Req 5.4: Should return streaming response for multimodal input.""" - user = make_user() - client = authenticated_client(app, user) - - resp = client.post( - "/chat/multimodal", - json={ - "session_id": "sess-001", - "message": "Describe this image", - "files": [ - { - "filename": "test.png", - "content_type": "image/png", - "bytes": "aGVsbG8=", - } - ], - }, - ) - - assert resp.status_code == 200 - assert "text/event-stream" in resp.headers["content-type"] - - def test_returns_streaming_response_without_files(self, app, make_user, authenticated_client): - """Req 5.4: Should return streaming response even without files.""" - user = make_user() - client = authenticated_client(app, user) - - resp = client.post( - "/chat/multimodal", - json={ - "session_id": "sess-001", - "message": "Just a text message", - }, - ) - - assert resp.status_code == 200 - assert "text/event-stream" in resp.headers["content-type"] - - def test_returns_401_for_unauthenticated(self, app, unauthenticated_client): - """Req 5.4: Should return 401 when no auth is provided.""" - client = unauthenticated_client(app) - resp = client.post( - "/chat/multimodal", - json={"session_id": "sess-001", "message": "Hello"}, - ) - assert resp.status_code == 401 diff --git a/backend/tests/routes/test_inference.py b/backend/tests/routes/test_inference.py index 1e08ef0b..c0daf0c6 100644 --- a/backend/tests/routes/test_inference.py +++ b/backend/tests/routes/test_inference.py @@ -65,11 +65,17 @@ def test_ping_returns_200(self, app): assert resp.status_code == 200 def test_ping_response_contains_status(self, app): - """Req 15.1: /ping response should contain status field.""" + """Req 15.1: /ping returns the AgentCore health contract. + + Status must be a valid AgentCore PingStatus value, and the response + must carry an integer ``time_of_last_update``; without that field the + platform idle-reaps the microVM mid-stream + (bedrock-agentcore-sdk-python#471). + """ client = TestClient(app) body = client.get("/ping").json() - assert "status" in body - assert body["status"] == "healthy" + assert body["status"] in {"Healthy", "HealthyBusy"} + assert isinstance(body["time_of_last_update"], int) # --------------------------------------------------------------------------- diff --git a/backend/tests/routes/test_pbt_auth_sweep.py b/backend/tests/routes/test_pbt_auth_sweep.py index f34247b0..2f966221 100644 --- a/backend/tests/routes/test_pbt_auth_sweep.py +++ b/backend/tests/routes/test_pbt_auth_sweep.py @@ -18,7 +18,7 @@ from hypothesis import given, settings, HealthCheck from hypothesis import strategies as st -from apis.shared.auth.dependencies import get_current_user, get_current_user_from_session +from apis.shared.auth.dependencies import get_current_user_from_session from apis.shared.auth.models import User from apis.shared.auth.rbac import require_admin @@ -157,16 +157,13 @@ def test_non_admin_roles_get_403(self, roles): app = FastAPI() app.include_router(admin_router) - # Override get_current_user to return a user with the generated roles + # Override the session dependency to return a user with the generated roles user = User( email="prop4@example.com", user_id="prop4-user", name="Property 4 User", roles=roles, ) - # `require_admin` is cookie-only since Phase 7; the Bearer override - # is kept too for any test that still relies on it. - app.dependency_overrides[get_current_user] = lambda: user app.dependency_overrides[get_current_user_from_session] = lambda: user # Mock AppRoleService to return no admin AppRoles (simulates @@ -231,7 +228,7 @@ def test_unauthenticated_request_returns_401(self, method, path): ) # Clean up the override so it doesn't leak to other tests - app.dependency_overrides.pop(get_current_user, None) + app.dependency_overrides.pop(get_current_user_from_session, None) def test_health_endpoint_accessible_without_auth(self): """Requirement 17.3: Health endpoint remains accessible without auth. @@ -240,7 +237,7 @@ def test_health_endpoint_accessible_without_auth(self): """ app = _FULL_APP # Ensure no auth override is set - app.dependency_overrides.pop(get_current_user, None) + app.dependency_overrides.pop(get_current_user_from_session, None) client = TestClient(app, raise_server_exceptions=False) resp = client.get("/health") diff --git a/backend/tests/shared/test_managed_models.py b/backend/tests/shared/test_managed_models.py index 1064d8bd..f4144a13 100644 --- a/backend/tests/shared/test_managed_models.py +++ b/backend/tests/shared/test_managed_models.py @@ -92,3 +92,89 @@ async def test_supports_caching_default_openai(self): from apis.shared.models.managed_models import create_managed_model model = await create_managed_model(_make_model_data("gpt4", provider="openai")) assert model.supports_caching is False + + +class TestMaxTokensCeiling: + """max_tokens spec must not exceed the model's declared output ceiling.""" + + def test_default_above_ceiling_rejected(self): + # Default 8192 is within the (absent) row bounds but exceeds the + # model's 4096 ceiling — only the cross-field rule should fire. + with pytest.raises(Exception): + _make_model_data( + maxOutputTokens=4096, + supportedParams={"params": {"max_tokens": {"supported": True, "default": 8192}}}, + ) + + def test_max_above_ceiling_rejected(self): + with pytest.raises(Exception): + _make_model_data( + maxOutputTokens=4096, + supportedParams={"params": {"max_tokens": {"supported": True, "max": 8192}}}, + ) + + def test_within_ceiling_ok(self): + m = _make_model_data( + maxOutputTokens=8192, + supportedParams={"params": {"max_tokens": {"supported": True, "max": 8192, "default": 8192}}}, + ) + assert m.max_output_tokens == 8192 + + def test_unsupported_row_not_ceiling_checked(self): + m = _make_model_data( + maxOutputTokens=4096, + supportedParams={"params": {"max_tokens": {"supported": False, "max": 999999, "default": 999999}}}, + ) + assert m.max_output_tokens == 4096 + + def test_update_payload_enforced(self): + from apis.shared.models.models import ManagedModelUpdate + with pytest.raises(Exception): + ManagedModelUpdate( + maxOutputTokens=4096, + supportedParams={"params": {"max_tokens": {"supported": True, "default": 8192}}}, + ) + + +class TestEffortAllowed: + """Enum params carry an `allowed` set; `default` must be a member. + + This is the per-model representation of the effort-tier difference + (Sonnet 4.6 vs Opus 4.7) — data, not model-family branching in code. + """ + + def test_default_in_allowed_ok(self): + m = _make_model_data( + supportedParams={"params": {"effort": { + "supported": True, "allowed": ["low", "medium", "high"], "default": "high", + }}}, + ) + spec = m.supported_params.params["effort"] + assert spec.allowed == ["low", "medium", "high"] + assert spec.default == "high" + + def test_default_not_in_allowed_rejected(self): + with pytest.raises(Exception): + _make_model_data( + supportedParams={"params": {"effort": { + "supported": True, "allowed": ["low", "medium", "high"], "default": "xhigh", + }}}, + ) + + def test_empty_allowed_rejected(self): + with pytest.raises(Exception): + _make_model_data( + supportedParams={"params": {"effort": { + "supported": True, "allowed": [], "default": None, + }}}, + ) + + def test_allowed_without_default_ok(self): + # No default is valid — runtime sends nothing, model uses its own + # API default (effort "high"). + m = _make_model_data( + supportedParams={"params": {"effort": { + "supported": True, "allowed": ["low", "medium", "high", "xhigh", "max"], + }}}, + ) + assert m.supported_params.params["effort"].default is None diff --git a/backend/tests/shared/test_mcp_apps_broker.py b/backend/tests/shared/test_mcp_apps_broker.py new file mode 100644 index 00000000..e4e8ef03 --- /dev/null +++ b/backend/tests/shared/test_mcp_apps_broker.py @@ -0,0 +1,102 @@ +"""Tests for the per-conversation app-tool event broker (MCP Apps PR #5).""" + +import asyncio + +import pytest + +from apis.shared.mcp_apps.broker import ( + AppToolEventBroker, + get_app_tool_event_broker, +) + + +def _ev(tag: str) -> dict: + return {"type": "tool_use", "data": {"tag": tag}} + + +def test_singleton_accessor(): + assert get_app_tool_event_broker() is get_app_tool_event_broker() + + +def test_publish_to_active_subscriber_is_live(): + b = AppToolEventBroker() + q = b.add_subscriber("s1") + b.publish("s1", _ev("a")) + b.publish("s1", _ev("b")) + assert [e["data"]["tag"] for e in b.drain(q)] == ["a", "b"] + assert b.drain(q) == [] + b.remove_subscriber("s1", q) + + +def test_publish_with_no_subscriber_buffers_then_flushes_on_subscribe(): + b = AppToolEventBroker() + # No active stream — buffered. + b.publish("s1", _ev("early")) + q = b.add_subscriber("s1") + # The next stream to open drains what it missed. + assert [e["data"]["tag"] for e in b.drain(q)] == ["early"] + b.remove_subscriber("s1", q) + + +def test_pending_ring_is_bounded(): + b = AppToolEventBroker() + for i in range(150): + b.publish("s1", _ev(str(i))) + q = b.add_subscriber("s1") + drained = b.drain(q) + # Capped at 100, oldest dropped → tail retained. + assert len(drained) == 100 + assert drained[0]["data"]["tag"] == "50" + assert drained[-1]["data"]["tag"] == "149" + + +def test_sessions_are_isolated(): + b = AppToolEventBroker() + qa = b.add_subscriber("a") + qb = b.add_subscriber("b") + b.publish("a", _ev("for-a")) + assert [e["data"]["tag"] for e in b.drain(qa)] == ["for-a"] + assert b.drain(qb) == [] + b.remove_subscriber("a", qa) + b.remove_subscriber("b", qb) + + +def test_fan_out_to_multiple_active_subscribers(): + b = AppToolEventBroker() + q1 = b.add_subscriber("s1") + q2 = b.add_subscriber("s1") + b.publish("s1", _ev("x")) + assert b.drain(q1)[0]["data"]["tag"] == "x" + assert b.drain(q2)[0]["data"]["tag"] == "x" + b.remove_subscriber("s1", q1) + b.remove_subscriber("s1", q2) + + +def test_remove_subscriber_prunes_session_then_buffers_again(): + b = AppToolEventBroker() + q = b.add_subscriber("s1") + b.remove_subscriber("s1", q) + # With the subscriber gone the session falls back to buffering. + b.publish("s1", _ev("after")) + q2 = b.add_subscriber("s1") + assert [e["data"]["tag"] for e in b.drain(q2)] == ["after"] + b.remove_subscriber("s1", q2) + + +def test_publish_empty_session_is_noop(): + b = AppToolEventBroker() + b.publish("", _ev("x")) # must not raise + + +@pytest.mark.asyncio +async def test_subscribe_context_manager_pairs_add_remove(): + b = AppToolEventBroker() + b.publish("s1", _ev("buffered")) + async with b.subscribe("s1") as q: + assert [e["data"]["tag"] for e in b.drain(q)] == ["buffered"] + b.publish("s1", _ev("live")) + assert [e["data"]["tag"] for e in b.drain(q)] == ["live"] + # Context exit unsubscribed → back to buffering. + b.publish("s1", _ev("after")) + async with b.subscribe("s1") as q2: + assert [e["data"]["tag"] for e in b.drain(q2)] == ["after"] diff --git a/backend/tests/shared/test_mcp_apps_card_store.py b/backend/tests/shared/test_mcp_apps_card_store.py new file mode 100644 index 00000000..69ce17f2 --- /dev/null +++ b/backend/tests/shared/test_mcp_apps_card_store.py @@ -0,0 +1,128 @@ +"""Tests for the app-initiated tool-card store (MCP Apps PR #6, Option A). + +The store reuses the existing `sessions-metadata` table. No DynamoDB in +tests — the no-table path is a silent no-op (matches dev), and a fake +table asserts the record shape, the ownership re-check, and the size cap. +""" + +from __future__ import annotations + +from decimal import Decimal + +from apis.shared.mcp_apps.card_store import AppCardStore + + +class _FakeTable: + def __init__(self, items=None) -> None: + self.items = items or [] + self.puts: list = [] + + def put_item(self, Item): # noqa: N803 - boto3 kwarg name + self.puts.append(Item) + + def query(self, **kwargs): + return {"Items": self.items} + + +def _store_with(table) -> AppCardStore: + s = AppCardStore() # __init__ sets _table=None without the env var + s._table = table + return s + + +def test_no_table_is_silent_noop(): + s = AppCardStore() + assert s.enabled is False + # Must not raise. + s.store( + user_id="u1", + session_id="s1", + tool_use_id="tu1", + tool_name="t", + arguments={}, + content=[], + is_error=False, + ) + assert s.list_for_session(session_id="s1", user_id="u1") == [] + + +def test_store_writes_appcard_record_shape(): + table = _FakeTable() + s = _store_with(table) + s.store( + user_id="u1", + session_id="s1", + tool_use_id="tu1", + tool_name="widget_tool", + arguments={"q": "x", "n": 1.5}, + content=[{"type": "text", "text": "ok"}], + is_error=False, + ) + assert len(table.puts) == 1 + item = table.puts[0] + assert item["PK"] == "USER#u1" + assert item["SK"].startswith("APPCARD#") + assert item["GSI_PK"] == "SESSION#s1" + assert item["GSI_SK"].startswith("APPCARD#") + assert item["toolName"] == "widget_tool" + assert item["isError"] is False + # floats are stored as Decimal for DynamoDB. + assert item["arguments"]["n"] == Decimal("1.5") + assert "ttl" in item + + +def test_store_caps_oversized_content(): + table = _FakeTable() + s = _store_with(table) + huge = [{"type": "text", "text": "z" * 300_000}] + s.store( + user_id="u1", + session_id="s1", + tool_use_id="tu1", + tool_name="t", + arguments={}, + content=huge, + is_error=False, + ) + stored = table.puts[0]["content"] + assert stored == [ + {"type": "text", "text": "[result omitted from history — too large to persist]"} + ] + + +def test_list_filters_by_owner_and_cleans_record(): + items = [ + { + "PK": "USER#u1", + "SK": "APPCARD#2026-01-01T00:00:00#aaa", + "GSI_PK": "SESSION#s1", + "GSI_SK": "APPCARD#2026-01-01T00:00:00", + "ttl": 123, + "userId": "u1", + "sessionId": "s1", + "toolName": "mine", + "isError": False, + "producedByMessageIndex": Decimal("4"), + }, + { + "PK": "USER#someone-else", + "SK": "APPCARD#2026-01-01T00:00:01#bbb", + "GSI_PK": "SESSION#s1", + "GSI_SK": "APPCARD#2026-01-01T00:00:01", + "userId": "other", + "toolName": "not-mine", + "isError": False, + }, + ] + s = _store_with(_FakeTable(items)) + cards = s.list_for_session(session_id="s1", user_id="u1") + + assert len(cards) == 1 + card = cards[0] + assert card["toolName"] == "mine" + # Key attributes are stripped from the returned card. + for k in ("PK", "SK", "GSI_PK", "GSI_SK", "ttl"): + assert k not in card + # Decimals are converted back to native ints/floats. + assert card["producedByMessageIndex"] == 4 + assert isinstance(card["producedByMessageIndex"], int) diff --git a/backend/tests/shared/test_models_and_utils.py b/backend/tests/shared/test_models_and_utils.py index 90e38a83..4c5a6432 100644 --- a/backend/tests/shared/test_models_and_utils.py +++ b/backend/tests/shared/test_models_and_utils.py @@ -244,6 +244,27 @@ def test_build_conversational_error_service_unavailable(self): evt = build_conversational_error_event(ErrorCode.SERVICE_UNAVAILABLE, Exception("down")) assert "unavailable" in evt.message.lower() + def test_build_conversational_error_max_tokens(self): + from apis.shared.errors import build_conversational_error_event, ErrorCode + # The raw SDK message carries a strandsagents.com URL we must NOT leak. + raw = Exception( + "Agent has reached an unrecoverable state due to max_tokens limit. " + "For more information see: https://strandsagents.com/x" + ) + evt = build_conversational_error_event( + ErrorCode.MAX_TOKENS, raw, session_id="s1", recoverable=True + ) + # Concise, not rendered as a bubble — the UI owns the wording. + assert "limit" in evt.message.lower() + # No leaked SDK URL or raw exception text. + assert "strandsagents.com" not in evt.message + assert "unrecoverable" not in evt.message.lower() + # Recoverable + machine-readable hint for the frontend affordance. + assert evt.recoverable is True + assert evt.code == ErrorCode.MAX_TOKENS + assert evt.metadata["error_kind"] == "max_tokens" + assert evt.metadata["session_id"] == "s1" + # =================================================================== # sessions/metadata.py — pure functions diff --git a/backend/tests/shared/test_sessions_metadata.py b/backend/tests/shared/test_sessions_metadata.py index ac6027c2..7b80f10b 100644 --- a/backend/tests/shared/test_sessions_metadata.py +++ b/backend/tests/shared/test_sessions_metadata.py @@ -74,6 +74,51 @@ async def test_get_nonexistent(self, sessions_metadata_table): assert result is None +class TestTruncatedTurnMarker: + """Refresh-survival marker for the max_tokens 'Continue' affordance.""" + + @pytest.mark.asyncio + async def test_set_then_clear(self, sessions_metadata_table): + from apis.shared.sessions.metadata import ( + store_session_metadata, + get_session_metadata, + set_truncated_turn, + clear_truncated_turn, + ) + await store_session_metadata(session_id="s1", user_id="u1", session_metadata=_make_session_metadata()) + + # Default: not continuable. + result = await get_session_metadata("s1", "u1") + assert not result.last_turn_continuable + + await set_truncated_turn("s1", "u1") + result = await get_session_metadata("s1", "u1") + assert result.last_turn_continuable is True + + await clear_truncated_turn("s1", "u1") + result = await get_session_metadata("s1", "u1") + assert not result.last_turn_continuable + + @pytest.mark.asyncio + async def test_survives_response_round_trip(self, sessions_metadata_table): + # Exact contract the metadata endpoint uses: + # SessionMetadataResponse.model_validate(metadata.model_dump(by_alias=True)) + from apis.shared.sessions.metadata import ( + store_session_metadata, + get_session_metadata, + set_truncated_turn, + ) + from apis.shared.sessions.models import SessionMetadataResponse + + await store_session_metadata(session_id="s2", user_id="u1", session_metadata=_make_session_metadata(session_id="s2")) + await set_truncated_turn("s2", "u1") + meta = await get_session_metadata("s2", "u1") + + resp = SessionMetadataResponse.model_validate(meta.model_dump(by_alias=True)) + assert resp.last_turn_continuable is True + assert resp.model_dump(by_alias=True)["lastTurnContinuable"] is True + + class TestGetAllMessageMetadata: @pytest.mark.asyncio async def test_get_cost_records(self, sessions_metadata_table): diff --git a/backend/tests/shared/test_user_menu_links.py b/backend/tests/shared/test_user_menu_links.py new file mode 100644 index 00000000..c75682c6 --- /dev/null +++ b/backend/tests/shared/test_user_menu_links.py @@ -0,0 +1,259 @@ +"""Tests for the user-menu links shared module (repository + service).""" + +import boto3 +import pytest +from pydantic import ValidationError + +from apis.shared.user_menu_links.models import ( + UserMenuLink, + UserMenuLinkCreate, + UserMenuLinkUpdate, +) +from apis.shared.user_menu_links.repository import UserMenuLinksRepository +from apis.shared.user_menu_links.service import UserMenuLinksService + +AWS_REGION = "us-west-2" + + +@pytest.fixture() +def user_menu_links_table(aws, monkeypatch): + ddb = boto3.client("dynamodb", region_name=AWS_REGION) + name = "test-user-menu-links" + ddb.create_table( + TableName=name, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + monkeypatch.setenv("DYNAMODB_USER_MENU_LINKS_TABLE_NAME", name) + return boto3.resource("dynamodb", region_name=AWS_REGION).Table(name) + + +@pytest.fixture() +def repo(user_menu_links_table): + return UserMenuLinksRepository(table_name="test-user-menu-links", region=AWS_REGION) + + +@pytest.fixture() +def service(repo): + return UserMenuLinksService(repo) + + +def _external(**kw): + defaults = dict(label="Privacy policy", kind="external", url="https://x.example/p") + defaults.update(kw) + return UserMenuLinkCreate(**defaults) + + +def _modal(**kw): + defaults = dict(label="About", kind="modal", body_markdown="# Hi") + defaults.update(kw) + return UserMenuLinkCreate(**defaults) + + +# ---------------------------------------------------------------------- +# Pydantic validation +# ---------------------------------------------------------------------- + + +class TestCreateValidation: + def test_external_requires_url(self): + with pytest.raises(ValidationError): + UserMenuLinkCreate(label="X", kind="external") + + def test_modal_requires_body_markdown(self): + with pytest.raises(ValidationError): + UserMenuLinkCreate(label="X", kind="modal") + + def test_external_ok_with_url(self): + link = UserMenuLinkCreate(label="X", kind="external", url="https://x.example") + assert link.kind == "external" + + def test_modal_ok_with_body(self): + link = UserMenuLinkCreate(label="X", kind="modal", body_markdown="hi") + assert link.kind == "modal" + + def test_order_bounds(self): + with pytest.raises(ValidationError): + UserMenuLinkCreate(label="X", kind="external", url="https://x.example", order=-1) + with pytest.raises(ValidationError): + UserMenuLinkCreate(label="X", kind="external", url="https://x.example", order=10_001) + + @pytest.mark.parametrize( + "bad_url", + [ + "javascript:alert(1)", + "data:text/html,", + "file:///etc/passwd", + "ftp://example.com/x", + "//example.com/x", + "example.com/x", + ], + ) + def test_external_rejects_non_http_url(self, bad_url): + with pytest.raises(ValidationError): + UserMenuLinkCreate(label="X", kind="external", url=bad_url) + + def test_external_accepts_http_and_https(self): + UserMenuLinkCreate(label="X", kind="external", url="http://x.example") + UserMenuLinkCreate(label="X", kind="external", url="HTTPS://X.example") + + def test_update_rejects_non_http_url(self): + with pytest.raises(ValidationError): + UserMenuLinkUpdate(url="javascript:alert(1)") + + +# ---------------------------------------------------------------------- +# Repository +# ---------------------------------------------------------------------- + + +class TestRepository: + @pytest.mark.asyncio + async def test_create_and_get(self, repo): + created = await repo.create_link(_external(), created_by="admin@x") + assert created.link_id + fetched = await repo.get_link(created.link_id) + assert fetched is not None + assert fetched.label == "Privacy policy" + assert fetched.created_by == "admin@x" + + @pytest.mark.asyncio + async def test_get_missing_returns_none(self, repo): + assert await repo.get_link("nope") is None + + @pytest.mark.asyncio + async def test_list_returns_all_then_enabled_only(self, repo): + a = await repo.create_link(_external(label="A", order=2)) + b = await repo.create_link(_modal(label="B", order=1, enabled=False)) + all_links = await repo.list_links() + assert {link.link_id for link in all_links} == {a.link_id, b.link_id} + enabled = await repo.list_links(enabled_only=True) + assert [link.link_id for link in enabled] == [a.link_id] + + @pytest.mark.asyncio + async def test_list_sorted_by_order_then_label(self, repo): + await repo.create_link(_external(label="Beta", order=10)) + await repo.create_link(_external(label="Alpha", order=10)) + await repo.create_link(_external(label="First", order=0)) + labels = [link.label for link in await repo.list_links()] + assert labels == ["First", "Alpha", "Beta"] + + @pytest.mark.asyncio + async def test_update_partial_merges(self, repo): + created = await repo.create_link(_external()) + updated = await repo.update_link( + created.link_id, UserMenuLinkUpdate(label="Renamed") + ) + assert updated is not None + assert updated.label == "Renamed" + assert updated.url == "https://x.example/p" # untouched + + @pytest.mark.asyncio + async def test_update_to_modal_requires_body(self, repo): + created = await repo.create_link(_external()) + # Switching to modal without supplying body_markdown should raise: + # the merged record has kind=modal but no body. + with pytest.raises(ValueError, match="modal links require body_markdown"): + await repo.update_link(created.link_id, UserMenuLinkUpdate(kind="modal")) + + @pytest.mark.asyncio + async def test_update_to_modal_with_body_succeeds(self, repo): + created = await repo.create_link(_external()) + updated = await repo.update_link( + created.link_id, + UserMenuLinkUpdate(kind="modal", body_markdown="hi"), + ) + assert updated is not None + assert updated.kind == "modal" + assert updated.body_markdown == "hi" + + @pytest.mark.asyncio + async def test_update_missing_returns_none(self, repo): + assert await repo.update_link("nope", UserMenuLinkUpdate(label="x")) is None + + @pytest.mark.asyncio + async def test_delete(self, repo): + created = await repo.create_link(_external()) + assert await repo.delete_link(created.link_id) is True + assert await repo.get_link(created.link_id) is None + + @pytest.mark.asyncio + async def test_delete_missing_returns_false(self, repo): + assert await repo.delete_link("nope") is False + + @pytest.mark.asyncio + async def test_update_rejects_non_http_url_on_merge(self, repo): + """Defense-in-depth: even if a bypass somehow stored a bad URL, + an update that merges it back through the repo must reject it. + We bypass Pydantic by mutating the existing record directly.""" + created = await repo.create_link(_external()) + # Force a bad URL into the persisted item to simulate corruption / + # an out-of-band write. + repo._table.update_item( + Key={"PK": "USER_MENU_LINKS", "SK": f"LINK#{created.link_id}"}, + UpdateExpression="SET #u = :u", + ExpressionAttributeNames={"#u": "url"}, + ExpressionAttributeValues={":u": "javascript:alert(1)"}, + ) + with pytest.raises(ValueError, match="url must start with"): + await repo.update_link( + created.link_id, UserMenuLinkUpdate(label="Renamed") + ) + + +# ---------------------------------------------------------------------- +# Service +# ---------------------------------------------------------------------- + + +class TestService: + @pytest.mark.asyncio + async def test_create_then_list(self, service): + await service.create_link(_external()) + links = await service.list_links() + assert len(links) == 1 + + @pytest.mark.asyncio + async def test_enabled_only_filter(self, service): + await service.create_link(_external(label="A")) + await service.create_link(_external(label="B", enabled=False)) + enabled = await service.list_links(enabled_only=True) + labels = [link.label for link in enabled] + assert labels == ["A"] + + +# ---------------------------------------------------------------------- +# Dataclass round-trip +# ---------------------------------------------------------------------- + + +class TestDynamoRoundTrip: + def test_to_and_from_dynamo_item(self): + original = UserMenuLink( + link_id="abc", + label="Test", + kind="modal", + enabled=True, + order=5, + body_markdown="# Hi", + created_at="2026-05-14T00:00:00Z", + updated_at="2026-05-14T00:00:00Z", + ) + item = original.to_dynamo_item() + round_tripped = UserMenuLink.from_dynamo_item(item) + assert round_tripped == original + + def test_from_dynamo_item_requires_timestamps(self): + # Corrupted records (missing createdAt/updatedAt) should raise rather + # than silently substituting "now". + with pytest.raises(ValueError, match="missing required timestamp"): + UserMenuLink.from_dynamo_item( + {"linkId": "x", "label": "x", "kind": "external", "url": "https://x.example"} + ) diff --git a/backend/tests/shared/test_user_menu_links_routes.py b/backend/tests/shared/test_user_menu_links_routes.py new file mode 100644 index 00000000..d283a806 --- /dev/null +++ b/backend/tests/shared/test_user_menu_links_routes.py @@ -0,0 +1,229 @@ +"""Route tests for user-menu links endpoints (admin CRUD + public read).""" + +import boto3 +import pytest +from fastapi import APIRouter, FastAPI, HTTPException +from fastapi.testclient import TestClient + +from apis.shared.auth import get_current_user_from_session, require_admin +from apis.shared.auth.models import User +from apis.shared.user_menu_links import repository as repo_module +from apis.shared.user_menu_links import service as service_module + +AWS_REGION = "us-east-1" +TABLE_NAME = "test-user-menu-links-routes" + + +def _make_user(email: str = "user@example.com", roles=None) -> User: + return User( + email=email, + user_id="user-001", + name="Test User", + roles=roles if roles is not None else ["User"], + ) + + +@pytest.fixture() +def user_menu_links_table(aws, monkeypatch): + """Moto-backed DynamoDB table + module-singleton reset so the routes pick + up this fresh table on first call inside each test.""" + ddb = boto3.client("dynamodb", region_name=AWS_REGION) + ddb.create_table( + TableName=TABLE_NAME, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + monkeypatch.setenv("DYNAMODB_USER_MENU_LINKS_TABLE_NAME", TABLE_NAME) + monkeypatch.setenv("AWS_REGION", AWS_REGION) + # The service + repo are module-level singletons; reset them so the + # next get_*() call constructs a fresh instance against the moto table. + monkeypatch.setattr(repo_module, "_repository", None) + monkeypatch.setattr(service_module, "_service", None) + return boto3.resource("dynamodb", region_name=AWS_REGION).Table(TABLE_NAME) + + +def _build_admin_app() -> FastAPI: + """Mount the admin router under /admin to mirror the real app.""" + from apis.app_api.admin.user_menu_links.routes import router as admin_router + + app = FastAPI() + parent = APIRouter(prefix="/admin") + parent.include_router(admin_router) + app.include_router(parent) + return app + + +def _build_public_app() -> FastAPI: + from apis.app_api.user_menu_links.routes import router as public_router + + app = FastAPI() + app.include_router(public_router) + return app + + +# ---------------------------------------------------------------------- +# Admin routes +# ---------------------------------------------------------------------- + + +class TestAdminRoutes: + def test_create_returns_201(self, user_menu_links_table): + app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + app.dependency_overrides[require_admin] = lambda: admin + + client = TestClient(app) + resp = client.post( + "/admin/user-menu-links/", + json={"label": "Privacy", "kind": "external", "url": "https://x.example"}, + ) + assert resp.status_code == 201 + body = resp.json() + assert body["label"] == "Privacy" + assert body["created_by"] == "admin@example.com" + + def test_create_rejects_non_http_url_with_422(self, user_menu_links_table): + app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + app.dependency_overrides[require_admin] = lambda: admin + + client = TestClient(app) + resp = client.post( + "/admin/user-menu-links/", + json={"label": "Bad", "kind": "external", "url": "javascript:alert(1)"}, + ) + assert resp.status_code == 422 + + def test_create_missing_url_for_external_returns_422(self, user_menu_links_table): + app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + app.dependency_overrides[require_admin] = lambda: admin + + client = TestClient(app) + resp = client.post( + "/admin/user-menu-links/", + json={"label": "X", "kind": "external"}, + ) + assert resp.status_code == 422 + + def test_non_admin_gets_403(self, user_menu_links_table): + app = _build_admin_app() + + def _forbid(): + raise HTTPException(status_code=403, detail="Forbidden") + + app.dependency_overrides[require_admin] = _forbid + + client = TestClient(app) + resp = client.get("/admin/user-menu-links/") + assert resp.status_code == 403 + + def test_list_then_get_round_trips(self, user_menu_links_table): + app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + app.dependency_overrides[require_admin] = lambda: admin + + client = TestClient(app) + created = client.post( + "/admin/user-menu-links/", + json={"label": "About", "kind": "modal", "body_markdown": "# Hi"}, + ).json() + + list_resp = client.get("/admin/user-menu-links/") + assert list_resp.status_code == 200 + assert list_resp.json()["total"] == 1 + + get_resp = client.get(f"/admin/user-menu-links/{created['link_id']}") + assert get_resp.status_code == 200 + assert get_resp.json()["label"] == "About" + + def test_get_missing_returns_404(self, user_menu_links_table): + app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + app.dependency_overrides[require_admin] = lambda: admin + + client = TestClient(app) + resp = client.get("/admin/user-menu-links/does-not-exist") + assert resp.status_code == 404 + + def test_update_returns_400_on_invariant_violation(self, user_menu_links_table): + app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + app.dependency_overrides[require_admin] = lambda: admin + + client = TestClient(app) + created = client.post( + "/admin/user-menu-links/", + json={"label": "Privacy", "kind": "external", "url": "https://x.example"}, + ).json() + + # PATCH kind=modal without supplying body_markdown — the merged record + # fails the kind/body invariant in the repository, which raises + # ValueError → mapped to 400 by the handler. + resp = client.patch( + f"/admin/user-menu-links/{created['link_id']}", + json={"kind": "modal"}, + ) + assert resp.status_code == 400 + + def test_delete_returns_204_then_404(self, user_menu_links_table): + app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + app.dependency_overrides[require_admin] = lambda: admin + + client = TestClient(app) + created = client.post( + "/admin/user-menu-links/", + json={"label": "X", "kind": "external", "url": "https://x.example"}, + ).json() + + del_resp = client.delete(f"/admin/user-menu-links/{created['link_id']}") + assert del_resp.status_code == 204 + + again = client.delete(f"/admin/user-menu-links/{created['link_id']}") + assert again.status_code == 404 + + +# ---------------------------------------------------------------------- +# Public read route +# ---------------------------------------------------------------------- + + +class TestPublicRoute: + def test_returns_only_enabled_links(self, user_menu_links_table): + admin_app = _build_admin_app() + admin = _make_user(email="admin@example.com", roles=["system_admin"]) + admin_app.dependency_overrides[require_admin] = lambda: admin + admin_client = TestClient(admin_app) + + # Seed one enabled + one disabled link via the admin API. + admin_client.post( + "/admin/user-menu-links/", + json={"label": "Visible", "kind": "external", "url": "https://x.example"}, + ) + admin_client.post( + "/admin/user-menu-links/", + json={ + "label": "Hidden", + "kind": "external", + "url": "https://y.example", + "enabled": False, + }, + ) + + public_app = _build_public_app() + public_app.dependency_overrides[get_current_user_from_session] = ( + lambda: _make_user() + ) + public_client = TestClient(public_app) + resp = public_client.get("/user-menu-links/") + assert resp.status_code == 200 + body = resp.json() + assert [link["label"] for link in body["links"]] == ["Visible"] diff --git a/backend/tests/test_seed_system_admin_jwt.py b/backend/tests/test_seed_system_admin_jwt.py index 2b605f9c..5ad50c90 100644 --- a/backend/tests/test_seed_system_admin_jwt.py +++ b/backend/tests/test_seed_system_admin_jwt.py @@ -117,7 +117,7 @@ def test_creates_default_tools(self, dynamodb_table): """Creates the default tool entries.""" result = seed_default_tools(TABLE_NAME, REGION) - assert result.created == 4 + assert result.created == 6 assert result.failed == 0 # Verify fetch_url_content @@ -167,13 +167,39 @@ def test_creates_default_tools(self, dynamodb_table): assert item["category"] == "code" assert item["protocol"] == "local" + # Verify create_artifact + resp = dynamodb_table.get_item( + Key={"PK": "TOOL#create_artifact", "SK": "METADATA"} + ) + item = resp["Item"] + assert item["toolId"] == "create_artifact" + assert item["displayName"] == "Create Artifact" + assert item["category"] == "document" + assert item["protocol"] == "local" + assert item["enabledByDefault"] is True + assert item["isPublic"] is True + assert item["GSI1PK"] == "CATEGORY#document" + assert item["GSI1SK"] == "TOOL#create_artifact" + + # Verify update_artifact + resp = dynamodb_table.get_item( + Key={"PK": "TOOL#update_artifact", "SK": "METADATA"} + ) + item = resp["Item"] + assert item["toolId"] == "update_artifact" + assert item["displayName"] == "Update Artifact" + assert item["category"] == "document" + assert item["protocol"] == "local" + assert item["enabledByDefault"] is True + assert item["isPublic"] is True + def test_skips_existing_tools(self, dynamodb_table): """Skips tools that already exist.""" seed_default_tools(TABLE_NAME, REGION) result = seed_default_tools(TABLE_NAME, REGION) - assert result.skipped == 4 + assert result.skipped == 6 assert result.created == 0 def test_partial_skip(self, dynamodb_table): @@ -187,5 +213,5 @@ def test_partial_skip(self, dynamodb_table): result = seed_default_tools(TABLE_NAME, REGION) - assert result.created == 3 + assert result.created == 5 assert result.skipped == 1 diff --git a/backend/tests/test_system_admin.py b/backend/tests/test_system_admin.py deleted file mode 100644 index 2cdf5637..00000000 --- a/backend/tests/test_system_admin.py +++ /dev/null @@ -1,90 +0,0 @@ -"""Tests for require_system_admin dependency.""" - -import pytest -from unittest.mock import AsyncMock, patch -from datetime import datetime, timezone - -from fastapi import HTTPException - -from apis.shared.auth.models import User -from apis.shared.rbac.models import UserEffectivePermissions -from apis.shared.rbac.system_admin import require_system_admin - - -def _user(roles: list | None = None) -> User: - return User( - user_id="u-1", - email="test@example.com", - name="Test", - roles=roles or [], - ) - - -def _perms(app_roles: list) -> UserEffectivePermissions: - return UserEffectivePermissions( - user_id="u-1", - app_roles=app_roles, - tools=[], - models=[], - quota_tier=None, - resolved_at=datetime.now(timezone.utc).isoformat() + "Z", - ) - - -class TestRequireSystemAdmin: - @pytest.mark.asyncio - async def test_grants_access_when_system_admin_role_present(self): - mock_service = AsyncMock() - mock_service.resolve_user_permissions.return_value = _perms(["system_admin"]) - - with patch( - "apis.shared.rbac.service.get_app_role_service", - return_value=mock_service, - ): - result = await require_system_admin(user=_user(["Admin"])) - - assert result.user_id == "u-1" - mock_service.resolve_user_permissions.assert_called_once() - - @pytest.mark.asyncio - async def test_denies_access_without_system_admin_role(self): - mock_service = AsyncMock() - mock_service.resolve_user_permissions.return_value = _perms(["default"]) - - with patch( - "apis.shared.rbac.service.get_app_role_service", - return_value=mock_service, - ): - with pytest.raises(HTTPException) as exc_info: - await require_system_admin(user=_user(["Faculty"])) - - assert exc_info.value.status_code == 403 - - @pytest.mark.asyncio - async def test_denies_access_on_service_error(self): - """Fail-closed: if AppRoleService raises, deny access.""" - mock_service = AsyncMock() - mock_service.resolve_user_permissions.side_effect = Exception("DynamoDB down") - - with patch( - "apis.shared.rbac.service.get_app_role_service", - return_value=mock_service, - ): - with pytest.raises(HTTPException) as exc_info: - await require_system_admin(user=_user(["Admin"])) - - assert exc_info.value.status_code == 403 - - @pytest.mark.asyncio - async def test_denies_access_with_empty_app_roles(self): - mock_service = AsyncMock() - mock_service.resolve_user_permissions.return_value = _perms([]) - - with patch( - "apis.shared.rbac.service.get_app_role_service", - return_value=mock_service, - ): - with pytest.raises(HTTPException) as exc_info: - await require_system_admin(user=_user([])) - - assert exc_info.value.status_code == 403 diff --git a/backend/uv.lock b/backend/uv.lock index df65cc44..8afa59de 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = ">=3.10" resolution-markers = [ "python_full_version >= '3.15'", @@ -12,7 +12,7 @@ resolution-markers = [ [[package]] name = "agentcore-stack" -version = "1.0.0b24" +version = "1.0.0b28" source = { editable = "." } dependencies = [ { name = "aiofiles" }, @@ -25,6 +25,7 @@ dependencies = [ { name = "httpx" }, { name = "pillow" }, { name = "pyjwt", extra = ["crypto"] }, + { name = "pypdfium2" }, { name = "python-dotenv" }, { name = "python-multipart" }, { name = "starlette" }, @@ -83,9 +84,9 @@ requires-dist = [ { name = "aiohttp", specifier = "==3.13.5" }, { name = "authlib", specifier = "==1.7.0" }, { name = "aws-opentelemetry-distro", marker = "extra == 'agentcore'", specifier = "==0.17.0" }, - { name = "bedrock-agentcore", marker = "extra == 'agentcore'", specifier = "==1.6.4" }, + { name = "bedrock-agentcore", marker = "extra == 'agentcore'", specifier = "==1.9.1" }, { name = "black", marker = "extra == 'dev'", specifier = "==26.3.1" }, - { name = "boto3", specifier = "==1.42.96" }, + { name = "boto3", specifier = "==1.43.9" }, { name = "cachetools", specifier = "==6.2.4" }, { name = "cryptography", specifier = "==47.0.0" }, { name = "fastapi", specifier = "==0.136.1" }, @@ -98,6 +99,7 @@ requires-dist = [ { name = "openai", marker = "extra == 'agentcore'", specifier = "==2.32.0" }, { name = "pillow", specifier = "==12.2.0" }, { name = "pyjwt", extras = ["crypto"], specifier = "==2.12.1" }, + { name = "pypdfium2", specifier = "==4.30.0" }, { name = "pytest", marker = "extra == 'dev'", specifier = "==9.0.3" }, { name = "pytest-asyncio", marker = "extra == 'dev'", specifier = "==1.3.0" }, { name = "pytest-cov", marker = "extra == 'dev'", specifier = "==7.1.0" }, @@ -105,11 +107,11 @@ requires-dist = [ { name = "python-multipart", specifier = "==0.0.27" }, { name = "ruff", marker = "extra == 'dev'", specifier = "==0.15.12" }, { name = "starlette", specifier = "==1.0.0" }, - { name = "strands-agents", marker = "extra == 'agentcore'", specifier = "==1.37.0" }, - { name = "strands-agents", extras = ["bidi"], marker = "extra == 'bidi'", specifier = "==1.37.0" }, - { name = "strands-agents-tools", marker = "extra == 'agentcore'", specifier = "==0.5.1" }, + { name = "strands-agents", marker = "extra == 'agentcore'", specifier = "==1.40.0" }, + { name = "strands-agents", extras = ["bidi"], marker = "extra == 'bidi'", specifier = "==1.40.0" }, + { name = "strands-agents-tools", marker = "extra == 'agentcore'", specifier = "==0.5.2" }, { name = "tiktoken", marker = "extra == 'dev'", specifier = "==0.12.0" }, - { name = "types-aiofiles", marker = "extra == 'dev'", specifier = "==25.1.0.20251011" }, + { name = "types-aiofiles", marker = "extra == 'dev'", specifier = "==25.1.0.20260409" }, { name = "uvicorn", extras = ["standard"], specifier = "==0.46.0" }, ] provides-extras = ["agentcore", "bidi", "dev", "all"] @@ -513,7 +515,7 @@ wheels = [ [[package]] name = "bedrock-agentcore" -version = "1.6.4" +version = "1.9.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "boto3" }, @@ -525,9 +527,9 @@ dependencies = [ { name = "uvicorn" }, { name = "websockets" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/54/6d/6dfd3e9f05fb3fff256312cf7a9cee11d849281dfd4d32fa7aaf3cea87e9/bedrock_agentcore-1.6.4.tar.gz", hash = "sha256:7b3e12361ca432ab1cada5e191e6f3cfa9536cd5cedafc37058f670b263bdabf", size = 521801, upload-time = "2026-04-23T20:08:25.401Z" } +sdist = { url = "https://files.pythonhosted.org/packages/08/ba/91b6ec49558755cccc5bfa5a64916995baed5490768bee33581b370a1e4e/bedrock_agentcore-1.9.1.tar.gz", hash = "sha256:f0e69b41c32c12e395d698299c96981d34035dafa90e0e79fcbd743574315c6a", size = 692593, upload-time = "2026-05-12T21:50:47.639Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/49/cc/298426f7601172fab91a7c4fe6c0f7a07ecbdaeb2413f1e8dc5d79aacbd7/bedrock_agentcore-1.6.4-py3-none-any.whl", hash = "sha256:a20f76f23cf08f4c081704eeb85c1899340163066b1612458c93963055a5e3dd", size = 168734, upload-time = "2026-04-23T20:08:23.467Z" }, + { url = "https://files.pythonhosted.org/packages/34/05/a5fbaa2320c34f8df196c105ca1938848845216cacc36850c73d116f28a9/bedrock_agentcore-1.9.1-py3-none-any.whl", hash = "sha256:f323c3d943dfe1defd52febd1409f8c4d04c0fc37848dd100ede692c2a6addd2", size = 262193, upload-time = "2026-05-12T21:50:45.506Z" }, ] [[package]] @@ -576,30 +578,30 @@ wheels = [ [[package]] name = "boto3" -version = "1.42.96" +version = "1.43.9" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "botocore" }, { name = "jmespath" }, { name = "s3transfer" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/a6/2d/69fb3acd50bab83fb295c167d33c4b653faeb5fb0f42bfca4d9b69d6fb68/boto3-1.42.96.tar.gz", hash = "sha256:b38a9e4a3fbbee9017252576f1379780d0a5814768676c08df2f539d31fcdd68", size = 113203, upload-time = "2026-04-24T19:47:18.677Z" } +sdist = { url = "https://files.pythonhosted.org/packages/b4/cc/42d798fc5305e4636170b50cdfb305ff0a81f470e35131f4a0d2641976ae/boto3-1.43.9.tar.gz", hash = "sha256:37dac72f2921095378c0200caf07918d5e10a82b7c1f611abb70e44f69d0b962", size = 113135, upload-time = "2026-05-15T19:28:31.167Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2b/9d/b3f617d011c42eb804d993103b8fa9acdce153e181a3042f58bfe33d7cb4/boto3-1.42.96-py3-none-any.whl", hash = "sha256:2f4566da2c209a98bdbfc874d813ef231c84ad24e4f815e9bc91de5f63351a24", size = 140557, upload-time = "2026-04-24T19:47:15.824Z" }, + { url = "https://files.pythonhosted.org/packages/f4/dc/51286e9551f7852a79ce5d2a57468d9d905c30d32bcace55204551db202d/boto3-1.43.9-py3-none-any.whl", hash = "sha256:5e967292d361482793471bd80fad1e714515b7401f65a0d5b4aa6ef9d009c030", size = 140523, upload-time = "2026-05-15T19:28:28.948Z" }, ] [[package]] name = "botocore" -version = "1.42.97" +version = "1.43.9" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "jmespath" }, { name = "python-dateutil" }, { name = "urllib3" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/c6/95/c37edb602948fad2253ffd1bb3dba5b938645bd1845ee4160350136a0f41/botocore-1.42.97.tar.gz", hash = "sha256:5c0bb00e32d16ff6d278cc8c9e10dc3672d9c1d569031635ac3c908a60de8310", size = 15269348, upload-time = "2026-04-27T20:39:05.625Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ca/e8/f696c80982685a4cdb3df5f0781919afa50262f40e1aac7066c9c2520deb/botocore-1.43.9.tar.gz", hash = "sha256:93e91c7160678182860f5902ee4cfe6d643cac0d9ee84d3eb65becc9f4c00228", size = 15357963, upload-time = "2026-05-15T19:28:19.342Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/e3/d2/8e025ba1a4e257879af72d06913272311af79673d82fa2581a351b924317/botocore-1.42.97-py3-none-any.whl", hash = "sha256:77d2c8ce1bc592d3fbd7c01c35836f4a5b0cac2ca03ccdf6ffc60faa16b5fadc", size = 14950367, upload-time = "2026-04-27T20:39:01.261Z" }, + { url = "https://files.pythonhosted.org/packages/77/c9/a1b51a74d476f5cb2f555ce8274f0f6b9fb21d75cc3f57b87dd0632ee17a/botocore-1.43.9-py3-none-any.whl", hash = "sha256:b9bdcd9c87fc552aad30006f00167d9ebb3480e1b06f1902bac5b2c41014fdab", size = 15039827, upload-time = "2026-05-15T19:28:14.543Z" }, ] [[package]] @@ -3668,6 +3670,26 @@ crypto = [ { name = "cryptography" }, ] +[[package]] +name = "pypdfium2" +version = "4.30.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a1/14/838b3ba247a0ba92e4df5d23f2bea9478edcfd72b78a39d6ca36ccd84ad2/pypdfium2-4.30.0.tar.gz", hash = "sha256:48b5b7e5566665bc1015b9d69c1ebabe21f6aee468b509531c3c8318eeee2e16", size = 140239, upload-time = "2024-05-09T18:33:17.552Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c7/9a/c8ff5cc352c1b60b0b97642ae734f51edbab6e28b45b4fcdfe5306ee3c83/pypdfium2-4.30.0-py3-none-macosx_10_13_x86_64.whl", hash = "sha256:b33ceded0b6ff5b2b93bc1fe0ad4b71aa6b7e7bd5875f1ca0cdfb6ba6ac01aab", size = 2837254, upload-time = "2024-05-09T18:32:48.653Z" }, + { url = "https://files.pythonhosted.org/packages/21/8b/27d4d5409f3c76b985f4ee4afe147b606594411e15ac4dc1c3363c9a9810/pypdfium2-4.30.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:4e55689f4b06e2d2406203e771f78789bd4f190731b5d57383d05cf611d829de", size = 2707624, upload-time = "2024-05-09T18:32:51.458Z" }, + { url = "https://files.pythonhosted.org/packages/11/63/28a73ca17c24b41a205d658e177d68e198d7dde65a8c99c821d231b6ee3d/pypdfium2-4.30.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e6e50f5ce7f65a40a33d7c9edc39f23140c57e37144c2d6d9e9262a2a854854", size = 2793126, upload-time = "2024-05-09T18:32:53.581Z" }, + { url = "https://files.pythonhosted.org/packages/d1/96/53b3ebf0955edbd02ac6da16a818ecc65c939e98fdeb4e0958362bd385c8/pypdfium2-4.30.0-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3d0dd3ecaffd0b6dbda3da663220e705cb563918249bda26058c6036752ba3a2", size = 2591077, upload-time = "2024-05-09T18:32:55.99Z" }, + { url = "https://files.pythonhosted.org/packages/ec/ee/0394e56e7cab8b5b21f744d988400948ef71a9a892cbeb0b200d324ab2c7/pypdfium2-4.30.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cc3bf29b0db8c76cdfaac1ec1cde8edf211a7de7390fbf8934ad2aa9b4d6dfad", size = 2864431, upload-time = "2024-05-09T18:32:57.911Z" }, + { url = "https://files.pythonhosted.org/packages/65/cd/3f1edf20a0ef4a212a5e20a5900e64942c5a374473671ac0780eaa08ea80/pypdfium2-4.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f1f78d2189e0ddf9ac2b7a9b9bd4f0c66f54d1389ff6c17e9fd9dc034d06eb3f", size = 2812008, upload-time = "2024-05-09T18:32:59.886Z" }, + { url = "https://files.pythonhosted.org/packages/c8/91/2d517db61845698f41a2a974de90762e50faeb529201c6b3574935969045/pypdfium2-4.30.0-py3-none-musllinux_1_1_aarch64.whl", hash = "sha256:5eda3641a2da7a7a0b2f4dbd71d706401a656fea521b6b6faa0675b15d31a163", size = 6181543, upload-time = "2024-05-09T18:33:02.597Z" }, + { url = "https://files.pythonhosted.org/packages/ba/c4/ed1315143a7a84b2c7616569dfb472473968d628f17c231c39e29ae9d780/pypdfium2-4.30.0-py3-none-musllinux_1_1_i686.whl", hash = "sha256:0dfa61421b5eb68e1188b0b2231e7ba35735aef2d867d86e48ee6cab6975195e", size = 6175911, upload-time = "2024-05-09T18:33:05.376Z" }, + { url = "https://files.pythonhosted.org/packages/7a/c4/9e62d03f414e0e3051c56d5943c3bf42aa9608ede4e19dc96438364e9e03/pypdfium2-4.30.0-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:f33bd79e7a09d5f7acca3b0b69ff6c8a488869a7fab48fdf400fec6e20b9c8be", size = 6267430, upload-time = "2024-05-09T18:33:08.067Z" }, + { url = "https://files.pythonhosted.org/packages/90/47/eda4904f715fb98561e34012826e883816945934a851745570521ec89520/pypdfium2-4.30.0-py3-none-win32.whl", hash = "sha256:ee2410f15d576d976c2ab2558c93d392a25fb9f6635e8dd0a8a3a5241b275e0e", size = 2775951, upload-time = "2024-05-09T18:33:10.567Z" }, + { url = "https://files.pythonhosted.org/packages/25/bd/56d9ec6b9f0fc4e0d95288759f3179f0fcd34b1a1526b75673d2f6d5196f/pypdfium2-4.30.0-py3-none-win_amd64.whl", hash = "sha256:90dbb2ac07be53219f56be09961eb95cf2473f834d01a42d901d13ccfad64b4c", size = 2892098, upload-time = "2024-05-09T18:33:13.107Z" }, + { url = "https://files.pythonhosted.org/packages/be/7a/097801205b991bc3115e8af1edb850d30aeaf0118520b016354cf5ccd3f6/pypdfium2-4.30.0-py3-none-win_arm64.whl", hash = "sha256:119b2969a6d6b1e8d55e99caaf05290294f2d0fe49c12a3f17102d01c441bd29", size = 2752118, upload-time = "2024-05-09T18:33:15.489Z" }, +] + [[package]] name = "pytest" version = "9.0.3" @@ -4195,14 +4217,14 @@ wheels = [ [[package]] name = "s3transfer" -version = "0.16.0" +version = "0.17.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "botocore" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/05/04/74127fc843314818edfa81b5540e26dd537353b123a4edc563109d8f17dd/s3transfer-0.16.0.tar.gz", hash = "sha256:8e990f13268025792229cd52fa10cb7163744bf56e719e0b9cb925ab79abf920", size = 153827, upload-time = "2025-12-01T02:30:59.114Z" } +sdist = { url = "https://files.pythonhosted.org/packages/9b/ec/7c692cde9125b77e84b307354d4fb705f98b8ccad59a036d5957ca75bfc3/s3transfer-0.17.0.tar.gz", hash = "sha256:9edeb6d1c3c2f89d6050348548834ad8289610d886e5bf7b7207728bd43ce33a", size = 155337, upload-time = "2026-04-29T22:07:36.33Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/fc/51/727abb13f44c1fcf6d145979e1535a35794db0f6e450a0cb46aa24732fe2/s3transfer-0.16.0-py3-none-any.whl", hash = "sha256:18e25d66fed509e3868dc1572b3f427ff947dd2c56f844a5bf09481ad3f3b2fe", size = 86830, upload-time = "2025-12-01T02:30:57.729Z" }, + { url = "https://files.pythonhosted.org/packages/87/72/c6c32d2b657fa3dad1de340254e14390b1e334ce38268b7ad51abda3c8c2/s3transfer-0.17.0-py3-none-any.whl", hash = "sha256:ce3801712acf4ad3e89fb9990df97b4972e93f4b3b0004d214be5bce12814c20", size = 86811, upload-time = "2026-04-29T22:07:34.966Z" }, ] [[package]] @@ -4362,7 +4384,7 @@ wheels = [ [[package]] name = "strands-agents" -version = "1.37.0" +version = "1.40.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "boto3" }, @@ -4378,9 +4400,9 @@ dependencies = [ { name = "typing-extensions" }, { name = "watchdog" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/03/88/cf23aa713ea68c8a0ad5144341da7ee022e88ce6206512aeafddba257b75/strands_agents-1.37.0.tar.gz", hash = "sha256:3fe6821f730f0468eee91e1ff38eb27a5244046893ffba63e8f5345288096509", size = 824168, upload-time = "2026-04-22T19:18:01.378Z" } +sdist = { url = "https://files.pythonhosted.org/packages/07/fa/b5fdfa099b122fea98fc64b9923237077ed6b7c2a90f2c3a65cba00d7202/strands_agents-1.40.0.tar.gz", hash = "sha256:5d867c1255f8449f0030a9a9c085106c15b1704e871d0fea56d3c20b2309a4d3", size = 878176, upload-time = "2026-05-14T13:48:28.812Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/5f/ff/bede1b8d5fe1c776bd5ed33575505681b3b65ab20889fe6b8344b92fc82d/strands_agents-1.37.0-py3-none-any.whl", hash = "sha256:2fa12e22ed1dac228aa93e91c2ea5381d9b3f08416ed8162222b61b255fee0b1", size = 404526, upload-time = "2026-04-22T19:17:59.634Z" }, + { url = "https://files.pythonhosted.org/packages/9e/ca/ce4c061d0fa007738f0ce4ebdb234969d9343322a089c24d5986620faa66/strands_agents-1.40.0-py3-none-any.whl", hash = "sha256:40c04f411e4082a6eb78b22d5b421757b27aac1f9a42e8198ff3db7fd4fcc13f", size = 432744, upload-time = "2026-05-14T13:48:26.639Z" }, ] [package.optional-dependencies] @@ -4391,7 +4413,7 @@ bidi = [ [[package]] name = "strands-agents-tools" -version = "0.5.1" +version = "0.5.2" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "aiohttp" }, @@ -4412,9 +4434,9 @@ dependencies = [ { name = "tzdata", marker = "sys_platform == 'win32'" }, { name = "watchdog" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/b4/fc/8a9da78b5c4a8802367a8eeec046f98eda742b1ee1b2fff568c81c1b3479/strands_agents_tools-0.5.1.tar.gz", hash = "sha256:616ba88b5849d9fd495da057ccb670108580320b8cb0fc4faac5fc327f2622aa", size = 483123, upload-time = "2026-04-22T20:01:13.305Z" } +sdist = { url = "https://files.pythonhosted.org/packages/63/32/710a49ffd32b0a232ec1731620ee6105c045e9a77ecee1f3ecaa1a80a6cd/strands_agents_tools-0.5.2.tar.gz", hash = "sha256:96763c8ae75933c5dd327cca87561f573aed720c9c0f3d17fd20835910d11381", size = 483164, upload-time = "2026-04-30T17:08:13.151Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/64/59/79360f718683ae15cefeb8b0ca1e6d96608c4581280fb12b0f502375a705/strands_agents_tools-0.5.1-py3-none-any.whl", hash = "sha256:790865d073410e9a16ac44ce3a46c169b98e1f89844ce8670472b869257b7686", size = 316122, upload-time = "2026-04-22T20:01:11.599Z" }, + { url = "https://files.pythonhosted.org/packages/59/ef/fe73b6d25d095784d2e1f6f33419265e796143100fb2f32a6e86f8ae68af/strands_agents_tools-0.5.2-py3-none-any.whl", hash = "sha256:8f85e4cb28d9411e62e1f159aa7e300d3a0f4b1d2b878a7cdfd5d746d9333343", size = 316178, upload-time = "2026-04-30T17:08:11.416Z" }, ] [[package]] @@ -4567,11 +4589,11 @@ wheels = [ [[package]] name = "types-aiofiles" -version = "25.1.0.20251011" +version = "25.1.0.20260409" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/84/6c/6d23908a8217e36704aa9c79d99a620f2fdd388b66a4b7f72fbc6b6ff6c6/types_aiofiles-25.1.0.20251011.tar.gz", hash = "sha256:1c2b8ab260cb3cd40c15f9d10efdc05a6e1e6b02899304d80dfa0410e028d3ff", size = 14535, upload-time = "2025-10-11T02:44:51.237Z" } +sdist = { url = "https://files.pythonhosted.org/packages/6c/66/9e62a2692792bc96c0f423f478149f4a7b84720704c546c8960b0a047c89/types_aiofiles-25.1.0.20260409.tar.gz", hash = "sha256:49e67d72bdcf9fe406f5815758a78dc34a1249bb5aa2adba78a80aec0a775435", size = 14812, upload-time = "2026-04-09T04:22:35.308Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/71/0f/76917bab27e270bb6c32addd5968d69e558e5b6f7fb4ac4cbfa282996a96/types_aiofiles-25.1.0.20251011-py3-none-any.whl", hash = "sha256:8ff8de7f9d42739d8f0dadcceeb781ce27cd8d8c4152d4a7c52f6b20edb8149c", size = 14338, upload-time = "2025-10-11T02:44:50.054Z" }, + { url = "https://files.pythonhosted.org/packages/27/d0/28236f869ba4dfb223ecdbc267eb2bdb634b81a561dd992230a4f9ec48fa/types_aiofiles-25.1.0.20260409-py3-none-any.whl", hash = "sha256:923fedb532c772cc0f62e0ce4282725afa82ca5b41cabd9857f06b55e5eee8de", size = 14372, upload-time = "2026-04-09T04:22:34.328Z" }, ] [[package]] diff --git a/docs/feature-summaries/MULTIMODAL_FILE_ATTACHMENTS.md b/docs/feature-summaries/MULTIMODAL_FILE_ATTACHMENTS.md index 52a593a9..49fb58f3 100644 --- a/docs/feature-summaries/MULTIMODAL_FILE_ATTACHMENTS.md +++ b/docs/feature-summaries/MULTIMODAL_FILE_ATTACHMENTS.md @@ -42,7 +42,7 @@ Users can attach files to chat messages. Files are uploaded to S3 via pre-signed │ Frontend Backend AWS │ │ ──────── ─────── ─── │ │ │ -│ 1. POST /chat/agent-stream 2. FileResolver.resolve_files() │ +│ 1. POST /chat/stream (BFF) 2. FileResolver.resolve_files() │ │ {message, file_upload_ids} ─────────────────────────────► S3 │ │ ─────────────────────────► - Fetch each file from S3 │ │ - Base64 encode content │ diff --git a/docs/feature-summaries/RBAC_IMPLEMENTATION.md b/docs/feature-summaries/RBAC_IMPLEMENTATION.md index 2b0f185b..0339f70d 100644 --- a/docs/feature-summaries/RBAC_IMPLEMENTATION.md +++ b/docs/feature-summaries/RBAC_IMPLEMENTATION.md @@ -116,10 +116,10 @@ async def critical_endpoint(user: User = Depends(require_all_roles("Admin", "Sec ### Conditional Features ```python -from apis.shared.auth import get_current_user, has_any_role +from apis.shared.auth import get_current_user_from_session, has_any_role @router.get("/dashboard") -async def dashboard(user: User = Depends(get_current_user)): +async def dashboard(user: User = Depends(get_current_user_from_session)): """All authenticated users can access, but admins see extra data.""" response = {"user": user.email} @@ -303,7 +303,7 @@ The dependency automatically: 1. **Always use dependencies** - Never manually check roles 2. **Log admin actions** - Audit trail for compliance -3. **Use specific roles** - Prefer `require_admin` over `get_current_user` for sensitive operations +3. **Use specific roles** - Prefer `require_admin` over `get_current_user_from_session` for sensitive operations 4. **Never disable auth in production** - `ENABLE_AUTHENTICATION=false` is for development only 5. **Validate on every request** - Stateless authentication, no sessions 6. **Use HTTPS in production** - Protect tokens in transit diff --git a/docs/kaizen/decisions.md b/docs/kaizen/decisions.md new file mode 100644 index 00000000..ddd0eebe --- /dev/null +++ b/docs/kaizen/decisions.md @@ -0,0 +1,26 @@ +# Kaizen Decisions Log + +Declined proposals and corrected premises. `kaizen-research` and `kaizen-review-prep` +**must not re-propose** anything here without *materially new context* (a new capability, +a changed upstream constraint, or a new exploit/failure path). Each entry records what +the new context would have to be to re-open it. + +--- + +### [2026-05-18] Declined — Add Reddit `.rss` or Reddit MCP to `kaizen-research` +- **Origin**: review-queue.md (open since 2026-05-10) ▸ research/2026-05-10.md Risks; recommended Decline in reviews/2026-05-15.md ▸ Retirement Candidates. +- **Decision**: Decline. +- **Reasoning**: research/2026-05-15.md confirmed Reddit is blocked at the **domain level** via WebFetch — not just the HTML path. The proposal as scoped (add a Reddit `.rss` source to the research skill) is infeasible with current tooling. +- **Re-open only if**: a Reddit MCP server becomes available, or a `curl`-via-Bash path with a custom User-Agent header is whitelisted. Absent one of those, do not re-surface. + +### [2026-05-18] Premise corrected — "Close #266 / #267 as phantom tech debt" +- **Origin**: review-queue.md (open since 2026-05-10) ▸ reviews/2026-05-15.md ▸ Proposal #7 ("close phantom tech debt; features already in our Strands 1.39 pin"). Actioned via PR #338. +- **Decision**: Premise rejected. Issues **#266** (large tool-result offload) and **#267** (context-window lookup fallback) are **not** phantom debt — they are live, well-specified Strands adoption/wiring tasks whose 1.39 precondition is now met. PR #338 posted "unblocked, keep open" comments on both rather than closing them. +- **Reasoning**: The kaizen review assumed the upstream features being present in our pinned Strands version made the issues obsolete. They are not obsolete — they track the *wiring* work to actually adopt those features. Closing them would have silently dropped real, scoped backlog. +- **Re-open only if**: never re-propose *closing* #266/#267 on the "already in our pin" basis. They are valid open work; treat as normal backlog, not kaizen retirement candidates. (Proposing to *implement* them is fine — that is the opposite of this decision.) + +### [2026-05-18] Scope note — "Adopt Strands built-in proactive compression, retire our custom `TurnBasedSessionManager` compaction" +- **Origin**: the review-queue Strands-bump entry framed Strands 1.40 proactive compression (PR #2239) as a "library-native subtraction" reducing our custom session-manager compaction surface. Surfaced concretely in PR #340's "Subtraction opportunity (noted, NOT acted on)". +- **Decision**: **Not a drop-in replacement.** Do not propose retiring our custom compaction on a bare "Strands now does this" basis. +- **Reasoning**: Strands' built-in proactive compression operates on `ConversationManager` and only summarizes. Our `TurnBasedSessionManager` compaction additionally does: (1) tool-content truncation, (2) AgentCore-Memory long-term-summary retrieval, (3) DynamoDB-persisted checkpoint state — and drives the PR #243 `compaction` SSE event. The built-in managers do none of (1)–(3). +- **Re-open only if**: a concrete migration design accounts for tool-content truncation, LTM summary retrieval, DynamoDB checkpoint persistence, and the `compaction` SSE-once invariant. A bare "adopt the built-in, delete ours" proposal is out of scope and should not be re-surfaced. diff --git a/docs/kaizen/research/2026-05-10.md b/docs/kaizen/research/2026-05-10.md new file mode 100644 index 00000000..b009ba2e --- /dev/null +++ b/docs/kaizen/research/2026-05-10.md @@ -0,0 +1,265 @@ +# Kaizen Research — Sunday, May 10, 2026 +> Scan window: May 3 – May 10, 2026 (7 days; reference repo + UI/UX scan extended to 30 days for first-run baseline) +> Web budget: 64/50 used (target — UX-lens scan added 10 requests post-initial-run). Frontier-models also went over the sub-budget by ~5 due to two OpenAI WebFetch 403s. +> **Bootstrap run** — first execution of the kaizen-research skill. Subsequent runs cover only the prior 7 days for the reference repo + UX sources too. + +## TL;DR + +Three converging signals this week: +1. **MCP Apps is now the de-facto agentic UI standard**, and we don't host it yet. The spec (SEP-1865) is production-ready: tool results can declare a `ui://` resource that the host renders in a sandboxed iframe alongside the chat. Claude Desktop, ChatGPT, VS Code Copilot, Goose, Postman all ship support. Every third-party MCP server we connect could be shipping richer UX than text+JSON; we're leaving that on the table. +2. **Upstream is shrinking our backlog for free**: our open issues #266/#267 were quietly solved in Strands v1.37/v1.38 (now in our 1.39 pin from #265); `bedrock-agentcore` is 3 minor versions behind (1.6.4 → 1.9.0, latest published May 7 — inside the scan window) with likely fixes for two open SDK issues we feel. +3. **CI is broken**: 9 Nightly Build & Test failures + 6+ Deploy failures in 7 days, untriaged. + +**Recommended #1**: scope an MCP Apps host renderer in our chat (multi-PR initiative). It's the highest-leverage agentic-UX investment this week per the scan. **Recommended quick-win**: bump `bedrock-agentcore` 1.6.4 → 1.9.0. + +## External Scan + +### What's moving this week + +The week converged on two themes worth our attention. First, AWS shipped two AgentCore capabilities that map cleanly onto things we already do: **AgentCore Runtime BYO filesystem from S3/EFS** (cross-session filesystem persistence without custom mount code) and **AgentCore Memory metadata** (structured tags on long-term memory records for filtered retrieval). Both are direct value-adds to our `inference-api` and our `TurnBasedSessionManager` layer. Second, Strands has been cleaning up the long tail: v1.37 added a context-window lookup table (closes our open issue #267), v1.38 added large tool result offload (closes our open issue #266), and v1.39 — which we just pinned in #265 — added AWS-profile support for the OpenAI provider. We're caught up to the head, but we haven't yet *used* the v1.37/v1.38 features the upgrade unlocked. + +The reference repo (`aws-samples/sample-strands-agent-with-agentcore`) has diverged from us in one major direction (CDK → Terraform on Apr 19) and converged in several minor ones — most notably moving compaction state and per-message metadata onto Strands' own `agent.state` and `message.metadata` instead of a custom DynamoDB table. They also abandoned the `enabledTools` whitelist pattern that's still embedded in our CLAUDE.md, in favor of a `disabled_skills` blacklist read from DDB per-request. Those are architectural calls, not direct ports. + +The MCP spec is heading toward stateless transport (SEP-2567 sessionless MCP merged May 7), which is a strong fit for our SigV4 Gateway model — but our Python `mcp` library hasn't picked it up yet (current 1.27.1). Watch. + +### Notable items by source + +#### AWS Bedrock / AgentCore +- **AgentCore Runtime BYO file system from S3 and EFS** — Attach S3/EFS to runtimes for cross-session persistence without custom mount code — https://aws.amazon.com/about-aws/whats-new/2026/05/amazon-bedrock-agentcore-runtime/ — *relevance*: directly applicable to `inference-api`; could replace future filesystem-staging glue +- **AgentCore Memory adds metadata for long-term memory** — Long-term memory records now support structured metadata for filtered retrieval — https://aws.amazon.com/about-aws/whats-new/2026/05/agentcore-longterm-memory-metadata — *relevance*: `TurnBasedSessionManager` long-term flush could carry user/RBAC/conversation-type metadata for richer recall +- **Secure AI agents with AgentCore Identity on Amazon ECS** — OAuth federation walkthrough for ECS-hosted agents — https://aws.amazon.com/blogs/machine-learning/secure-ai-agents-with-amazon-bedrock-agentcore-identity-on-amazon-ecs/ — *relevance*: useful reference; pattern mirrors our `apis/shared/oauth/agentcore_identity.py` mint-fallback +- **OS-Level Actions in AgentCore Browser** — OS-level control for native UI agents — https://aws.amazon.com/blogs/machine-learning/introducing-os-level-actions-in-amazon-bedrock-agentcore-browser/ — *relevance*: informational; we don't use AgentCore Browser +- **AgentCore Payments preview** — Wallet/auth/governance for transactional agents (Coinbase + Stripe partners) — https://aws.amazon.com/blogs/machine-learning/agents-that-transact-introducing-amazon-bedrock-agentcore-payments-built-with-coinbase-and-stripe/ — *relevance*: informational; no commerce path today + +**Open AgentCore SDK issues affecting us:** +- **#456 — OTEL context detached across asyncio/thread boundaries in memory client + Strands session_manager** — https://github.com/aws/bedrock-agentcore-sdk-python/issues/456 — *applicability*: HIGH — we use Strands 1.39 + AgentCore Memory + `TurnBasedSessionManager`; X-Ray/OTEL traces likely show broken spans on memory writes +- **#452 — AgentCoreMemorySessionManager: add `async_mode` to prevent event-loop blocking** — https://github.com/aws/bedrock-agentcore-sdk-python/issues/452 — *applicability*: HIGH — `inference-api` is FastAPI/async; sync flush on the loop could be hurting concurrency +- **#453 — Auto-populate AgentCard.skills[] from ToolRegistry in serve_a2a** — *applicability*: medium; relevant if/when we expose A2A endpoints + +#### Strands Agents +- **v1.39.0 (current pin)** — AWS profile support for OpenAI, MCP init error messaging, Bedrock token-counting enhancements, A2A task-lifecycle states — https://github.com/strands-agents/sdk-python/releases/tag/v1.39.0 — *informational*: just landed in #265 +- **v1.38.0 — large tool result offload + `CachePoint` TTL for prompt caching** — https://github.com/strands-agents/sdk-python/releases/tag/v1.38.0 — *closes our issue #266* +- **v1.37.0 — context-window limit lookup tables + experimental checkpoint API** — https://github.com/strands-agents/sdk-python/releases/tag/v1.37.0 — *closes our issue #267* +- **#2266 — `BedrockModel.stream` leaks inner task on outer cancellation (May 9, open)** — https://github.com/strands-agents/sdk-python/issues/2266 — *applicability*: HIGH — we cancel SSE streams on client disconnect; check for "Task exception was never retrieved" in stream_coordinator logs +- **#2271 — Support dual cache prefixes in Bedrock auto caching strategy (May 10)** — https://github.com/strands-agents/sdk-python/issues/2271 — *applicability*: medium; pairs with issue #269 (prompt caching) if we move to Strands' built-in caching strategy +- **#2243 — Tool-level suspend/resume for external async callbacks** — https://github.com/strands-agents/sdk-python/issues/2243 — *applicability*: medium; could simplify our `oauth_required` SSE handoff +- **PR #2239 — Proactive Context Compression (merged May 8)** — https://github.com/strands-agents/sdk-python/pull/2239 — *applicability*: medium; could complement our SSE compaction surfacing + +#### Reference repo: aws-samples/sample-strands-agent-with-agentcore (last 30 days — bootstrap baseline) +- **CDK → Terraform migration (Apr 19, c422fbf)** — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/c422fbf — *applicability*: NOT relevant for porting; we're CDK-native. **Implication**: the reference repo is no longer a usable CDK template going forward. Anything CDK-shaped historically pulled from them is frozen at pre-Apr-19 state. +- **Compaction state + metrics moved from custom DynamoDB to SDK `agent.state` + `message.metadata` (Apr 27, 2b1a13d)** — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/2b1a13d — *applicability*: HIGH — our `TurnBasedSessionManager` could shed code by piggybacking on `agent.state` rather than maintaining parallel state; potential subtraction +- **Force re-auth on OAuth 401/403 mid-tool-call (Apr 22, 9fcdb4c)** — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/9fcdb4c — *applicability*: HIGH — verify our `oauth_required` SSE flow handles mid-conversation 401/403 from Google etc. by re-emitting `oauth_required` rather than streaming an error +- **Supersede stale executions instead of 409-rejecting (May 6, d6c9516)** — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/d6c9516 — *applicability*: medium; check how app-api handles concurrent submissions on the same conversation +- **Use SDK `agent.cancel()` for stop-signal handling (May 6, fd9acec)** — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/fd9acec — *applicability*: medium; if we have custom cancellation code, may simplify +- **`enabledTools` whitelist replaced with `disabled_skills` blacklist (May 3, 092aa33)** — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/092aa33 — *applicability*: monitor; our CLAUDE.md still mentions `enabled_tools` as a debug step. Inversion has UX upside but RBAC implications + +#### MCP ecosystem +- **SEP-2567 Sessionless MCP merged (May 7)** — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2567 — *implications*: drops `Mcp-Session-Id` and `session/create`; list endpoints become cacheable. Strong fit for our SigV4 Gateway model. Watch python `mcp` library for adoption. +- **SEP-2575 init-removal track (companion)** — same thread — *implications*: stateless HTTP transport simplifies Lambda-backed Gateway servers +- **Schema rename: `IncompleteResult` → `InputRequiredResult`** — typed-API break on next `mcp` lib bump +- **MCPSafe — security scanner for MCP servers** — https://github.com/orgs/modelcontextprotocol/discussions — could scan our Gateway-hosted servers +- **MCP servers repo (no new servers this week)** — discovery has moved to `registry.modelcontextprotocol.io` + +#### FastMCP (used by our externally hosted MCP servers, behind AgentCore Gateway) +- **Latest release: 3.2.4** — published 2026-04-14 (~26 days ago) — https://pypi.org/project/fastmcp/ — *applicability*: cross-reference against our MCP server repos' pinned FastMCP version; if any are behind 3.x, evaluate the migration path. +- **Bootstrap-run note**: FastMCP source category was added mid-bootstrap based on follow-up feedback. Full release-notes + issues scan (https://github.com/jlowin/fastmcp) deferred to the first regular Friday run (2026-05-15). For this bootstrap, only the PyPI version snapshot is captured. + +#### Agentic UI/UX patterns (30-day baseline scan for bootstrap) + +- **MCP Apps extension is production-ready (SEP-1865)** — https://modelcontextprotocol.io/extensions/apps/overview | https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/ — *what it is*: spec letting MCP tools return a `_meta.ui.resourceUri` pointing to a `ui://` resource; host fetches the HTML and renders it in a sandboxed iframe alongside the chat with bidirectional `ui/`-prefixed JSON-RPC via `postMessage`. Claude Desktop, ChatGPT, VS Code Copilot, Goose, Postman, MCPJam already ship support. — *fit*: **direct port (high impact, high effort)** — this is the standard our chat is going to be measured against in 2026. — *where it'd land*: new SSE event (`ui_resource` carrying `{resourceUri, csp, permissions}`), Angular `` sandboxed-iframe component implementing the `ui/` host bridge, branch in tool-result rendering pipeline. +- **MCP Apps host security model — sandboxed iframe + opt-in capabilities** — https://modelcontextprotocol.io/extensions/apps/overview — *what it is*: hosts declare capabilities (`sendOpenLink`, mic, camera) a given app can request; tool-call proxying goes through the host with user consent. — *fit*: **direct port** — maps cleanly onto our existing `oauth_required` consent pattern. — *where it'd land*: extend `oauth_required` SSE event family with `ui_consent_required`; reuse per-provider consent badge UI. +- **MCP Apps example servers** — https://github.com/modelcontextprotocol/ext-apps/tree/main/examples — *what it is*: starter servers for data exploration (cohort heatmap, customer segmentation), forms (scenario modeler, budget allocator), media (PDF, video, sheet music), 3D (Cesium, Three.js). — *fit*: pattern-only — templates are React/Vue/Svelte but the protocol is framework-agnostic. — *informs*: the kinds of internal tools we'd expose as MCP Apps once we host. +- **AI SDK "Render Visual Interface in Chat" recipe** — https://ai-sdk.dev/cookbook — *what it is*: pattern where tool results map to specific UI components on the client, model drives which component renders. — *fit*: pattern-only (React hook). — *Angular equivalent*: a `toolRenderers` registry keyed by tool name, with a signal-driven `` component doing `@switch (toolName())` over registered renderers. We do a coarse version today; the pattern argues for making it a first-class extension point so per-tool components live next to the tool definition rather than in a god-switch. +- **AI SDK "Call Tools in Multiple Steps" / `streamText` multi-step** — https://ai-sdk.dev/cookbook — *fit*: pattern-only. — *Angular equivalent*: keep `signal()`-backed tool-call state mutable across the conversation (don't freeze at `tool_result`), so prior tool-call cards stay interactive as new steps stream in. +- **assistant-ui @0.14.0 (2026-05-07)** — https://github.com/Yonom/assistant-ui/releases — API consolidation (`useAui` replaces deprecated naming). Also: `mcp-app-studio` package updated alongside — assistant-ui is shipping first-party MCP Apps authoring/preview tooling. — *signal*: **MCP Apps is the assumed UI surface** for serious agentic chat shells going forward. +- **"Output isn't design" — Karri Saarinen, Linear (2026-04-17)** — https://linear.app/now/output-isn-t-design — *takeaway*: pointed pushback on generative-UI hype. "Plausible-looking generated interfaces unravel the moment you actually use them" because the work of resolving tensions and edge cases hasn't happened. — *implication for us*: when we add MCP Apps, treat the iframe as a vehicle for *purpose-built* UIs (forms, viewers), not as a "let the model generate a UI" shortcut. +- **"Interact with agent-created visualizations in canvases" — Cursor (2026-04-15)** — https://www.cursor.com/blog/canvas — *takeaway*: agent output that's interactive (charts you can drill into, plots you can re-parameterize) is now table stakes in agentic IDEs. Maps to our PDF/markdown/spreadsheet preview surface — direction is "previews become interactive viewers," not static thumbnails. +- **Linear Agent as named participant** — https://linear.app/now/how-we-use-linear-agent-at-linear (2026-04-10) + https://linear.app/changelog/2026-04-23-linear-agent-mcp-support — *pattern*: Linear's agent reads context via MCP and posts back as a structured agent identity in the issue thread (not as a chat message). **Agents as named participants with distinct affordances**, not just a stream of assistant text. — *fit for us*: worth considering for our multi-agent A2A flows — A2A sub-agents could render as distinct attributed turns rather than nested tool calls. +- **"Claude Design by Anthropic Labs" (2026-04-17)** — https://www.anthropic.com/news/claude-design-anthropic-labs — *takeaway*: "collaborate with Claude to produce polished visual work" as a first-class output type. Validates investing in artifact-style rendering surfaces beyond plain markdown. +- **NN/g "Designing AI Agents: 4 Lessons from China's Qwen Agent" (2026-05-08)** — https://www.nngroup.com/articles/designing-ai-agents/ — *evidence-based principles*: support discoverability, reuse familiar patterns, handle personal data carefully, protect user autonomy. — *applicability*: **discoverability** — tool-call rendering should surface available tools *before* the user has to phrase the right prompt (slash menu, suggestions from `enabled_tools`). **Autonomy** — our `oauth_required` consent event is on-pattern; extend the same explicit-consent model to MCP-Apps-initiated tool calls. +- **OpenAI AgentKit / Agent Builder visual canvas** — https://openai.com/index/introducing-agentkit/ — *takeaway*: agent *authoring* is moving to visual node-graphs. Not directly applicable to our runtime chat, but a signal that **agent-state visibility** (which agent is running, which tool, what step) is increasingly expected at runtime too — relevant to how we render A2A and multi-step tool flows. + +#### Frontier model announcements +- **Anthropic — higher Opus rate limits (May 6)** — https://www.anthropic.com/news/higher-limits-spacex — informational; we use Bedrock-hosted, not first-party +- **Anthropic — finance-agents pack (May 5)** — https://www.anthropic.com/news/finance-agents — Moody's MCP server is a concrete public MCP we could register in Gateway if a finance use case emerges +- **OpenAI — GPT-5.5 Instant displaces GPT-5.3 Instant (May 5)** — https://openai.com/index/gpt-5-5-instant/ — *risk*: confirm our model selector doesn't expose a deprecation-path 5.3 ID +- **Google / Gemini** — quiet week (no new model/API deltas) +- **Meta / Llama** — quiet week + +#### Agent harness patterns +- **Claude Code 2.1.136 — skills-under-plugins fix + MCP content-block fix** — https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md — *relevance*: skill loading + MCP tool result rendering +- **Claude Code 2.1.133 — hooks receive `effort.level` + `worktree.baseRef` setting** — same URL — pattern worth mirroring in Strands hook payloads +- **Claude Code 2.1.132 — `CLAUDE_CODE_SESSION_ID` env into Bash subprocess** — same URL — session-id-everywhere pattern we already do loosely +- **CMA_coordinate_specialist_team.ipynb (May 6)** — https://github.com/anthropics/claude-cookbooks/tree/main/managed_agents — coordinator + 3 specialists with per-role tool scoping +- **CMA_verify_with_outcome_grader.ipynb (May 6)** — same repo — writer/grader loop with `user.define_outcome` rubrics; could bolt onto SSE for tool-result fact-checking +- **Agent Development Lifecycle (LangChain blog, May 9)** — https://www.langchain.com/blog/the-agent-development-lifecycle — our kaizen cadence already covers most of this; gap is "online evals" + +#### Pricing / quota +- No detected Bedrock or AgentCore pricing changes this week +- Note: `https://aws.amazon.com/bedrock/whats-new/` returned **404** — page appears retired. Skill source URL needs replacement. + +#### Community + GitHub issues +- HN: 0 hits for stack keywords (`bedrock`, `agentcore`, `strands`, `mcp`, `claude code`) in the 7-day window — quiet +- Reddit: blocked from WebFetch in this environment — gap to address (`.rss` or Reddit MCP) + +#### Cookbook / courses +- 4 new managed-agent cookbooks landed May 5–8 (vulnerability detection, coordinator/specialists, outcome grading, registry category) +- `anthropics/courses` quiet (last commit Nov 2025) — candidate to drop from weekly scan + +#### Seasonal +- Out of window — no re:Invent or NeurIPS items + +### Patterns worth considering + +- **Online evals via grader sub-agent** — sample N% of conversation turns, run a stateless grader, persist outcomes. Fits LangChain's Agent Development Lifecycle framing and the CMA outcome-grader cookbook. **Verdict**: monitor — interesting once we've shipped the core cleanups below. +- **Brain/hands separation** (Anthropic Managed Agents direction) — push session/checkpoint store outside the agent process. We already do this via AgentCore Memory; fully aligned. **Verdict**: aligned, no action. +- **Sessionless MCP** (SEP-2567) — list endpoints cacheable per (deployment, auth). Direct fit for SigV4 Gateway. **Verdict**: monitor; act when python `mcp` library adopts. + +## Internal Audit + +### Activity (last 7 days) +- **Commits on develop**: 8 (all from squash-merged PRs) +- **PRs opened**: 5 (4 dependabot — #237/#239/#241 still open, plus #276 docs) +- **PRs merged**: 8 +- **PRs reverted**: 0 +- **Issues opened**: 4 (#266, #267, #268, #269 on May 9 — Strands-features and prompt caching) +- **CI failures (workflow → count)**: Nightly Build & Test 9, Deploy Inference API 5, Deploy App API 6, Deploy Frontend 1, Version Check 6, Deploy Infrastructure 2 + +### Repeated friction signals +- **Nightly Build & Test failing 9× since May 6** — concentrated cluster; no signal it's been investigated. Could be the test flakiness from issue #220 (order-dependent flakiness in `test_cognito_idp_service`, `test_oauth_repositories`, `test_auth_providers*`) compounding, or a different cause. *Hypothesis*: untriaged. *Fix candidate*: triage one failure end-to-end; promote to a blocking issue if not already on the board. +- **Deploy workflows failing 6+ times May 6–9** — Inference API, App API, Frontend deploys all hit failures. *Hypothesis*: BFF migration shipped this week (#272–#277) introduced env-var or stack drift not caught in synth. *Fix candidate*: cross-check most recent failed deploy log against beta.24 ↔ post-beta.24 stack diff. +- **5 of 8 commits this week are BFF/auth fixes** (#270, #271, #273, #274, #275, #277) — the BFF migration shipped in beta.24 is still being patched. Healthy iteration, but the pace says "treat BFF as not-done-yet" before declaring beta.25. + +### Version-pin lag + +| Dep | Pinned | Latest | Lag | Notes | +|---|---|---|---|---| +| `bedrock-agentcore` | 1.6.4 | **1.9.0** | 3 minor / latest 2026-05-07 | Open issues #456 (OTEL detach) and #452 (event-loop blocking) may already be addressed | +| `boto3` | 1.42.96 | 1.43.6 | 1 minor / ~10 patches | Routine bump | +| `aws-cdk-lib` | 2.251.0 | 2.253.1 | 2 patch | Routine | +| `aws-cdk` | 2.1120.0 | 2.1121.0 | 1 patch | Routine | +| `@angular/core` | 21.2.11 | 21.2.12 | 1 patch | Routine | +| `strands-agents` | 1.39.0 | 1.39.0 | current | Just upgraded in #265 | +| `fastapi` | 0.136.1 | 0.136.1 | current | — | +| `mcp` | (transitive) | 1.27.1 | n/a | Watch for SEP-2567 adoption | + +### Retirement candidates + +- **`enabled_tools` whitelist debug guidance in `CLAUDE.md`** — Reference repo abandoned this pattern May 3 (`092aa33`) for `disabled_skills` blacklist. Not urgent retirement, but worth a re-evaluation if we touch tool-enablement code. +- **`anthropics/courses` source in `kaizen-research`** — Last commit Nov 2025; subagent reported "quiet". Drop from weekly scan list. +- **`https://aws.amazon.com/bedrock/whats-new/` URL in `kaizen-research`** — 404'd on this run. Replace with the AWS What's New RSS feed only, or a different filtered URL. +- **`https://docs.claude.com/en/docs/claude-code/release-notes` URL in `kaizen-research`** — 301→404. Replace with `https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md`. +- **6 of 9 skills not modified in 60+ days** (angualar-best-practices, tailwind-ui, frontend-design, cdk-infrastructure, versioning, cors-deployment) — modification freshness alone is a weak signal for skills since they encode stable conventions. **No retirement recommended without invocation telemetry.** + +### Risks introduced this week + +- **`bedrock-agentcore` 3 minor versions behind** with a release in the scan window — issues #456 (OTEL trace detach in Memory + Strands session_manager) and #452 (async-mode for AgentCoreMemorySessionManager) may be already-fixed in 1.7-1.9. *What breaks if ignored*: silent observability gaps (broken spans on memory writes); concurrency degradation under load. — https://pypi.org/project/bedrock-agentcore/ +- **OpenAI displaces GPT-5.3 Instant with GPT-5.5 Instant (May 5)** — our model selector exposes per-model IDs. *What breaks if ignored*: customers using a 5.3 default may hit a deprecation window. — https://openai.com/index/gpt-5-5-instant/ +- **Strands #2266 — `BedrockModel.stream` leaks inner task on outer cancellation** — we cancel SSE streams on client disconnect. *What breaks if ignored*: orphaned tasks, "Task exception was never retrieved" log noise, possible memory pressure under churn. — https://github.com/strands-agents/sdk-python/issues/2266 +- **Reddit blocked from WebFetch** in the kaizen-research environment — community-signal scan is half-blind. — *Fix*: switch to `https://www.reddit.com/r//.rss` or a configured Reddit MCP server. + +## Ideas — Top 6 (ranked) + +> Bootstrap exceptionally lists 6 (vs the skill's nominal 5) because the UI/UX lens was added mid-run and surfaced an MCP Apps initiative worth ranking. Regular runs target 5. + +| # | Idea | Surface | Effort | Impact | Subtracts? | +|---|---|---|---|---|---| +| 1 | Scope an MCP Apps host renderer in our chat (multi-PR initiative) | frontend + backend (SSE event + component) | H | H | no — additive, but unlocks every future MCP server shipping a UI | +| 2 | Bump `bedrock-agentcore` 1.6.4 → 1.9.0; verify SDK issues #456/#452 are addressed | backend | L | M | no — pure dep bump (justified: 3 versions of upstream fixes, latest in scan window) | +| 3 | Promote tool-result rendering to a per-tool renderer registry (signal-backed) | frontend | M | M-H | partial — replaces an implicit switch with an explicit registry; bridges toward MCP Apps | +| 4 | Audit `BedrockModel.stream` cancellation path against Strands #2266 | backend | L | M-H | no — defensive; SSE-disconnect path is hot | +| 5 | Close issues #266 and #267 — features already in our Strands 1.39 pin; replace with smaller "wire upstream feature" tasks | cross-cutting | L | M | **yes — retires 2 build-from-scratch tickets (library-native subtraction)** | +| 6 | Triage Nightly Build & Test failure cluster (9× since May 6) | cross-cutting / CI | L-M | M-H | possibly — if root is issue #220, fixing it simplifies suite | + +### 1. Scope an MCP Apps host renderer in our chat +- **Source**: research/2026-05-10.md ▸ Agentic UI/UX ▸ MCP Apps (SEP-1865, production-ready); cross-confirmed by assistant-ui's `mcp-app-studio` direction +- **Surface area**: frontend (new `` Angular component, tool-result rendering pipeline branch) + backend (new SSE event `ui_resource`; possibly extend `oauth_required` family with `ui_consent_required`) +- **Change**: implement the host side of the MCP Apps spec — sandboxed iframe rendering `ui://` resources returned by MCP tools, with the `ui/`-prefixed JSON-RPC dialect over `postMessage`. Consent UX reuses the existing `oauth_required` pattern. Treat as a multi-PR initiative: (a) SSE event + plumbing, (b) iframe component + postMessage bridge, (c) consent UI, (d) end-to-end with one example MCP App from `ext-apps/examples`. +- **Subtracts**: no — pure addition. Justified because: every major host already ships this; without it, third-party MCP servers we connect can't deliver UI beyond text+JSON. We become the platform less-than the rest of the ecosystem. +- **Effort × Impact**: High × High +- **Verdict**: Worth scoping (formal scoping doc before any code). Could comfortably be a 3-4 week initiative spanning multiple sprints. + +### 2. Bump `bedrock-agentcore` 1.6.4 → 1.9.0 +- **Source**: PyPI (https://pypi.org/project/bedrock-agentcore/) + open SDK issues #456, #452 +- **Surface area**: `backend/pyproject.toml`, `backend/uv.lock` +- **Change**: pin update + smoke-test memory + identity flows in dev; verify CHANGELOG between 1.6 and 1.9 for any breaking changes +- **Subtracts**: addition only — justified by 3 versions of upstream fixes including likely-relevant OTEL trace detach (#456) and event-loop blocking (#452) +- **Effort × Impact**: Low × Medium +- **Verdict**: Worth trying + +### 3. Promote tool-result rendering to a per-tool renderer registry +- **Source**: research/2026-05-10.md ▸ Agentic UI/UX ▸ AI SDK generative-UI recipes + Cursor canvases +- **Surface area**: frontend (`` component or equivalent, plus a new `ToolRendererRegistry` service) +- **Change**: today our tool-result rendering is (implicitly) a switch in one place. Promote to a signal-backed registry keyed by tool name; per-tool renderers live next to the tool definition. Bridges naturally toward MCP Apps (which would just be "another registered renderer that emits an iframe"). Lifts a chunk of switch-like code into a declarative table. +- **Subtracts**: partial — replaces an implicit switch with an explicit registry; the registry's existence is more code, but it absorbs scattered tool-specific UI logic into one place. +- **Effort × Impact**: Medium × Medium-High +- **Verdict**: Worth trying — independently valuable AND pre-work for proposal #1. + +### 4. Audit `BedrockModel.stream` cancellation path +- **Source**: Strands open issue #2266 (filed May 9) +- **Surface area**: `backend/src/agents/main_agent/` stream coordinator + SSE handler +- **Change**: locate where we cancel `BedrockModel.stream`; ensure we `await task` on cancel paths so tasks don't orphan; add a log assertion in dev to detect "Task exception was never retrieved" +- **Subtracts**: addition only — defensive +- **Effort × Impact**: Low × Medium-High +- **Verdict**: Worth trying + +### 5. Close issues #266 and #267 — features already in our Strands 1.39 pin +- **Source**: Strands v1.37 (PR #2249, context-window lookup) + v1.38 (large tool result offload) +- **Surface area**: GitHub issues + small wiring in `stream_coordinator` and tool-result handling for spreadsheet/Code Interpreter outputs +- **Change**: close #266 and #267 with comments pointing at upstream PRs; replace with smaller "wire context-window lookup" and "wire large tool result offload" tasks if the wiring isn't automatic +- **Subtracts**: **yes — retires 2 "build from scratch" issues; replaces with at-most 2 "use upstream feature" tasks. Library-native subtraction.** +- **Effort × Impact**: Low × Medium +- **Verdict**: Worth trying + +### 6. Audit `oauth_required` SSE flow against ref-repo's mid-tool-call 401/403 handling +- **Source**: aws-samples/sample-strands-agent-with-agentcore commit `9fcdb4c` +- **Surface area**: `backend/src/apis/shared/oauth/agentcore_identity.py`, SSE event emission in `inference-api`, MCP/A2A tool wrappers +- **Change**: ensure mid-conversation 401/403 from Google/external OAuth providers re-emits `oauth_required` (consent-resume) rather than streaming a tool error to the user +- **Subtracts**: addition only — defensive; closes a real UX gap when upstream tokens revoke mid-stream +- **Effort × Impact**: Medium × High +- **Verdict**: Worth trying + +### 7. Triage Nightly Build & Test failure cluster +- **Source**: 9 failures since May 6 in `gh run list --status=failure` +- **Surface area**: `.github/workflows/nightly-*.yml`, possibly `tests/shared/test_cognito_idp_service.py` + adjacent (per issue #220) +- **Change**: pull the most recent failure log; trace to root cause; fix flakiness OR document why it's failing if it's a real regression; promote to blocking issue +- **Subtracts**: possibly — if root is issue #220 (test isolation), fixing it materially simplifies the suite +- **Effort × Impact**: Low-Medium × Medium-High +- **Verdict**: Worth trying + +## Take + +Two big themes this week. **First, agentic UI/UX has shifted under us.** MCP Apps shipped to production with adoption from Claude Desktop, ChatGPT, VS Code Copilot, Goose, and Postman; assistant-ui is building first-party MCP Apps tooling; the design conversation has moved from "what should an agent chat look like" to "how do we host other people's UIs in our chat." Our text+JSON tool-result rendering is now the baseline competitors are extending past. **Second, library-native subtraction is the kaizen loop's clearest win** — Strands 1.37/1.38 quietly closes our open issues #266/#267, and `bedrock-agentcore` 1.6.4 → 1.9.0 likely closes two open SDK issues we already feel. The single change that would matter most this week if scoped is **proposal #1 (MCP Apps host renderer)** — high effort, but the right strategic investment. Quick wins: **#2 (`bedrock-agentcore` bump)** and **#5 (close #266/#267)**. **#3 (renderer registry)** is the natural mid-ground that delivers value standalone AND pre-paves proposal #1. + +--- + +## Sources Scanned + +| # | Source | URL | Accessed | Items | +|---|---|---|---|---| +| 1 | AWS Bedrock + AgentCore (RSS, blog, pricing, SDK issues) | aws.amazon.com / github.com/aws/bedrock-agentcore-* | 2026-05-10 | 5 announcements + 5 issues | +| 2 | Strands Agents (releases, issues, PRs) | github.com/strands-agents/sdk-python | 2026-05-10 | 3 releases + 5 issues | +| 3 | Reference repo | github.com/aws-samples/sample-strands-agent-with-agentcore | 2026-05-10 | 12 commits in 30-day window | +| 4 | MCP ecosystem | modelcontextprotocol.io / github.com/modelcontextprotocol | 2026-05-10 | 4 spec items + 3 discussions | +| 4a | FastMCP (bootstrap: PyPI snapshot only — full scan deferred to 2026-05-15) | pypi.org/project/fastmcp + github.com/jlowin/fastmcp | 2026-05-10 | latest 3.2.4 | +| 4b | Agentic UI/UX (MCP Apps, AI SDK, assistant-ui, Linear/Cursor/Anthropic, NN/g) — 30-day baseline | modelcontextprotocol.io + ai-sdk.dev + assistant-ui.com + linear.app + cursor.com + anthropic.com + nngroup.com | 2026-05-10 | 11 items across MCP Apps spec, AI SDK patterns, assistant-ui, vendor product blogs, NN/g research | +| 5 | Frontier models (Anthropic, OpenAI, Google, Meta) | anthropic.com / openai.com / blog.google / ai.meta.com | 2026-05-10 | 3 Anthropic + 1 OpenAI + 0 others | +| 6 | Agent harness | github.com/anthropics + langchain.com + pydantic.dev | 2026-05-10 | 3 CC releases + 4 cookbook items | +| 7 | Community (HN Algolia + Reddit) | hn.algolia.com + reddit.com | 2026-05-10 | 0 HN hits, Reddit blocked | +| 8 | Version-pin diff | pypi.org / npmjs.com | 2026-05-10 | 8 deps checked, 4 lag | + +## Web Budget + +Used: 64 / 50 requests (target — UX-lens scan added 10 to the original 54). + +Skipped (unreachable / rate-limited): +- Reddit (`r/LocalLLaMA`, `r/MachineLearning`) — WebFetch blocked from this environment. Switch to `.rss` endpoint or configured Reddit MCP next run. +- `https://aws.amazon.com/bedrock/whats-new/` — 404 (page appears retired). Drop or replace. +- `https://docs.claude.com/en/docs/claude-code/release-notes` — 301→404. Replace with `github.com/anthropics/claude-code/blob/main/CHANGELOG.md`. +- OpenAI blog returned 403 twice; backfilled via search. + +Skipped (other): Security advisories (external) and Internal security posture (Dependabot + CodeQL) sources were initially included in this bootstrap run but **removed per scope refinement** — security signals are handled by Dependabot and CodeQL directly and don't need a weekly kaizen lens. Future runs won't scan them. + +Notes: +- Frontier-models sub-budget exceeded (11 vs ~6 target) due to two OpenAI WebFetch 403s requiring search backfill. +- This is a **bootstrap run**: reference-repo + UX-lens scope extended to 30 days for baseline; Carried Over and prior-decisions sections in the review-prep doc are necessarily empty. diff --git a/docs/kaizen/research/2026-05-15.md b/docs/kaizen/research/2026-05-15.md new file mode 100644 index 00000000..bdc3c06d --- /dev/null +++ b/docs/kaizen/research/2026-05-15.md @@ -0,0 +1,253 @@ +# Kaizen Research — Friday, May 15, 2026 +> Scan window: May 8 – May 15, 2026 (7 days). +> Web budget: ~46/50 used (target). Modest overshoot only on the frontier-models sub-budget (OpenAI 403 backfilled via search). +> First regular run after the 2026-05-10 bootstrap. + +## TL;DR + +Quiet external week with two outsized upstream events: **Strands v1.40.0 shipped (proactive context compression PR #2239 landed)** and carries a **breaking default flip** (`use_native_token_count` true → false) that affects our token accounting; and **`bedrock-agentcore` is now 4 releases behind** (1.6.4 → 1.9.1, latest May 12) with PR #478 in flight that directly resolves last week's flagged issue #452 (AgentCoreMemorySessionManager event-loop blocking). Internally the BFF parade settled into the beta.25 + beta.26 release pair, but **#293 disabled Dependabot version-update PRs on May 13** — the kaizen loop is now the *only* mechanism catching version-pin lag. **Recommended #1**: re-prioritize the `bedrock-agentcore` bump from last week's queue — it's stale and the lag is widening. + +## External Scan + +### What's moving this week + +Two ecosystem currents converged. **First, the upstream-shrinks-our-backlog pattern continued** but with a wrinkle: Strands v1.40 (May 14) lands the proactive context compression we'd noted as PR #2239, but it also flips `use_native_token_count` from `true` to `false` per PR #2284 — a silent latency regression fix for multimodal, but a behavior change for anyone reading Bedrock-native input-token counts. We'd need to audit our token-metric reads (recall last week's #270 bugfix touched per-message-cost + context-% semantics) before promoting. **Second, the MCP "stateless" track keeps consolidating**: SEP-2575 merged May 11 paired with SEP-2567 from last week's window — together they remove `Mcp-Session-Id` and the mandatory `initialize` handshake. Python `mcp` SDK 1.27.1 is patch-only and doesn't adopt either yet, so the watch-and-wait posture from last week stands. + +The reference repo (`aws-samples/sample-strands-agent-with-agentcore`) had 12 commits in window. The architecturally interesting cluster is around a Progressive-Disclosure "skill_executor" SSE wire-name unwrap (which doesn't map to us — we don't have a wrapper-tool layer) and two defensive fixes that *do* generalize: (a) `tool_use` SSE was being emitted twice per call (empty-args then populated-args), and (b) A2A AgentCard needed an explicit `capabilities={"streaming": True}` or the SDK silently fell back to non-streaming → 40-minute timeouts. Both are 30-second checks for us with non-trivial silent-failure modes. + +AgentCore SDK shipped **1.9.1 (May 12)** with a runtime parse-error fix and an entrypoint registration fix; **PR #478** adds the long-requested `async_mode` to `MemorySessionManager` — the fix for issue #452 we flagged HIGH last week. Likely lands in 1.10.0. Anthropic shipped the **`/goal` command + per-tool `duration_ms` on PostToolUse hooks** in Claude Code 2.1.139/141 — both directly inspire pieces of work we already have on deck (the planned context-attribution prototype + a long-arc-objective UX). Frontier-model side was quiet: only OpenAI's GPT-5.2/5.3 snapshot deprecation notice (no Anthropic / Google / Meta model releases). Agentic UI/UX was a lean week — only Linear Code Intelligence (validating the named-participant pattern with a 5× growth datapoint) and Cursor's pre-run elicitation wizard (informs MCP elicitation UX whenever we tackle it). HN and Reddit yielded nothing in window; Reddit `.rss` confirmed blocked at the domain level via WebFetch, not just the HTML path. + +### Notable items by source + +> **Annotation conventions:** +> - `*relevance*:` — impact-on-existing-code lens. What construct/file does this affect? What does it replace, simplify, or obsolete? +> - `*unlocks*:` — capability-unlock lens (where applicable). What net-new capability or enhancement does this make possible? + +#### AWS Bedrock / AgentCore +- **AgentCore Browser + Code Interpreter — Chrome enterprise policies + custom root CA certificates** — `BrowserClient.create_browser(enterprise_policies=...)` and `CodeInterpreter.create_code_interpreter(certificates=[Certificate.from_secret_arn(...)])` for governed/corporate-proxy scenarios. — https://aws.amazon.com/blogs/machine-learning/control-where-your-ai-agents-can-browse-with-chrome-enterprise-policies-on-amazon-bedrock-agentcore/ — *relevance*: informational; we don't use AgentCore Browser or Code Interpreter primitives today (tool layer is direct-call + AWS-SDK + Gateway-MCP + A2A) — *unlocks*: domain-restricted browser agents (e.g. `*.boisestate.edu`) and code-exec tools that reach internal HTTPS endpoints behind a corporate CA without disabling cert verification — parking-lot for whenever we add a browsing or sandboxed-exec tool +- **Bedrock advanced prompt optimizer + migration tool** — console feature that refines a prompt across multiple models and shows comparative perf + cost. — https://aws.amazon.com/about-aws/whats-new/2026/05/amazon-bedrock-advanced-prompt-optimization-migration-tool/ — *relevance*: informational; we author agent prompts in `backend/src/agents/main_agent/` (version-controlled), not Bedrock console. Could be a one-off tuning aid for the main-agent system prompt against a candidate model swap. No code impact. +- **Real-time voice agents with Stream Vision Agents + Nova 2 Sonic** — integration walkthrough for Nova 2 Sonic on Bedrock. — https://aws.amazon.com/blogs/machine-learning/real-time-voice-agents-with-stream-vision-agents-and-amazon-nova-2-sonic/ — *relevance*: informational; we already have a voice mode (`apis/app_api/voice/` with voice-ticket cookie auth). Useful comp point if we ever evaluate Nova 2 Sonic. +- **No AgentCore platform-level GA/preview announcements this week.** No movement on BYO filesystem, Memory metadata, Identity-on-ECS, or Payments (all flagged last week as new). Quietest AgentCore week in recent memory. + +#### Strands Agents +- **v1.40.0 released 2026-05-14** — *applicability*: HIGH — touches `backend/src/agents/main_agent/` agent setup and our custom `TurnBasedSessionManager`. — https://github.com/strands-agents/sdk-python/releases/tag/v1.40.0 + - Proactive context compression (PR #2239) landed — adjacent to our compaction-surfacing SSE path (`compaction` event). Verify it doesn't double-fire with our existing flush. + - Bedrock `count_tokens` AccessDenied caching (PR #2279). + - Swarm OTel context-detach fix (PR #2281) — same family as bedrock-agentcore SDK issue #456 we flagged last week. +- **BREAKING — `use_native_token_count` default flipped to `false` (PR #2284, in v1.40.0)** — fixes #2277 silent-latency regression for image/multimodal workloads; previous default added a per-turn Bedrock `count_tokens` call. — *applicability*: HIGH — if we read native input-token counts anywhere in `apis/shared/` token metrics or compaction triggers, we now get heuristic counts unless we explicitly set `BedrockModel(use_native_token_count=True)`. Audit before bumping. — https://github.com/strands-agents/sdk-python/pull/2284 — *closes our issues?* no, but interacts with last week's #270 per-message-cost fix. +- **Issue #2266 still open** — `BedrockModel.stream` cancel leak ("Task exception was never retrieved"); last updated 2026-05-09, no PR linked yet. — *applicability*: MEDIUM-HIGH (this is the same item we already queued as last week's #4 audit). — https://github.com/strands-agents/sdk-python/issues/2266 +- **PR #2290 — MCP progress notifications in MCPClient (open, updated 2026-05-14)** — adds `notifications/progress` to `MCPClient`. — *applicability*: MEDIUM — once merged, long-running MCP tools (Gateway-SigV4 + external OAuth servers) can stream progress to the user; would slot into our SSE event stream as a new event type. — *unlocks*: long-running-tool progress UX, e.g. for spreadsheet-analysis or future browsing tools. — https://github.com/strands-agents/sdk-python/pull/2290 +- **Issue #2286 — Strands SDK repos consolidating into a monorepo** — maintainer announcement opened 2026-05-12. — *applicability*: LOW today, HIGH later — import paths may move in a future major; pin and watch for v2.x messaging. — https://github.com/strands-agents/sdk-python/issues/2286 + +#### Reference repo (aws-samples/sample-strands-agent-with-agentcore) +- **fix(a2a): enable streaming capability on AgentCard** (`50c9112`) — one-line fix: `capabilities={"streaming": True}` on the A2A AgentCard. Without it, the A2A SDK client silently falls back to non-streaming and never receives the completed event → ~40-minute timeouts. — *applicability*: HIGH if we have any A2A AgentCard config. **Defensive 30-second check on our A2A construct(s)** (likely `backend/src/agents/` or infra). Failure mode is silent — exactly the kind of bug we don't want to discover in prod. — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/50c9112cbc83a4517462d9e77d73e2239b22a004 +- **refactor(sse): emit TOOL_CALL_START once per tool_use, after unwrap** (`d668685`) — they discovered Strands' tool-use processor was firing `_format_tool_use` twice per call (registration with empty input, then again with populated args). They gated START until the populated call. — *applicability*: MEDIUM — quick audit of our SSE `tool_use` emission. If we emit twice (empty + populated), the frontend tool chip may flicker or render an "unknown args" intermediate state. — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/d668685ddfd2e6164093ca2cca0e91852ad19876 +- **fix(skill): filter disabled skills out of the L1 catalog** (`a0753dc`) — generalizes to: *if a tool is filtered at execution, it must also be filtered out of any catalog/listing that gets injected into the system prompt*. Otherwise the model hallucinates availability. — *applicability*: MEDIUM — our system-prompt tool list should derive from the same post-RBAC filtered set the executor uses. Quick consistency check. — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/a0753dca5ade0e613f44608c3824e002dc02bc03 +- **test(e2e): add L4 protocol-path integration suite** (`8d111d9`) — 13 protocol paths (local/builtin/gateway/a2a/skill/memory) tested via deployed BFF with event-based SSE assertions. — *applicability*: MEDIUM — we have unit + architecture tests but limited deployed-BFF e2e of the multi-protocol matrix. Event-based assertions sidestep LLM flakiness. — https://github.com/aws-samples/sample-strands-agent-with-agentcore/commit/8d111d9bb79b7b1d88cc03cfb223ef23e037a32e + +#### MCP ecosystem +- **SEP-2575 "Make MCP Stateless" merged 2026-05-11** — companion to SEP-2567 (sessionless transport, last week). Removes the mandatory `initialize` handshake; replaces with per-request `MCP-Protocol-Version` header + `_meta` capability bits, a new optional `server/discover` RPC, and `messages/listen` for client-initiated streaming. — *implications*: directly affects our Lambda+Gateway servers. Combined with SEP-2567 this formalizes the "stateless wire protocol" direction. **No action yet** — wait for python `mcp` SDK adoption. — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2575 +- **python-sdk v1.27.1 released 2026-05-08 — NO SEP-2567/2575 adoption** — patch release only (pydantic 2.13 compat, OAuth client-metadata coercion, `httpx<1.0.0` pin, SSEError import refactor). — *implications*: safe transitive bump from 1.27.0; the `httpx<1.0.0` pin is worth noting if anything in `backend/pyproject.toml` is reaching toward httpx 1.x. — https://github.com/modelcontextprotocol/python-sdk/releases/tag/v1.27.1 +- **awslabs/mcp PR #3491 — auth-conflict detection + structured recovery (merged May 13)** — multi-tenant OAuth/MCP pattern. — *implications*: worth reading before our next Gateway-SigV4/OAuth bridging round in `backend/src/apis/shared/oauth/agentcore_identity.py` — same problem space (token vault + multiple identities). Reference pattern, not a blocker. — https://github.com/awslabs/mcp/pull/3491 +- **SEP-2577 "Deprecate Roots, Sampling, and Logging" still open** (active May 13) — signal the spec is consolidating around tools+prompts+resources only. — *implications*: informational; our Lambda servers don't implement those client-side features. — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2577 + +#### FastMCP +- **v3.3.0 "Slim Reaper" released 2026-05-15** — headline: `fastmcp-slim`, a client-only distribution that drops Starlette/Uvicorn for installs that only consume MCP. Import namespace unchanged. — https://github.com/PrefectHQ/fastmcp/releases/tag/v3.3.0 — *implications for our MCP servers*: **informational, not breaking**. Our externally-hosted MCP servers are servers, so they stay on the full `fastmcp`. But anything in `apis.shared` that acts as a *client*, or future scripts/agents that just consume MCP, could shrink their install footprint by switching to `fastmcp-slim[client]`. No API changes. +- **No transport / SEP-2567 / Lambda-adapter changes this week.** Active PRs are server hardening (docstring caching #4136, proactive token refresh #4142, event-store eviction crash #4144, cancelled-tool-call forwarding #4145) — nothing protocol-level. — https://github.com/PrefectHQ/fastmcp/pulls +- **Latest FastMCP pin moved 3.2.4 → 3.3.0.** Update the kaizen tracker. + +#### Agentic UI/UX patterns +- **Linear Agent — Code Intelligence (May 14)** — Linear Agent can now read codebases and answer technical questions; invoked via `@Linear` mention. **Usage jumped ~5× Feb→May (1,055 → ~5,200 queries/mo)** — strong validation of the named-participant pattern flagged last week. — https://linear.app/now/code-intelligence-for-linear-agent — *what it is*: agent-as-mention pattern, addressable by `@`, scoped to a thread, inline answers. — *fit*: pattern-only (Angular equivalent: render assistant turns with a distinct avatar + handle; support `@mention` addressing when multi-agent lands). — *where it'd land*: `chat-message` component (agent identity slot + mention-token renderer); no SSE change needed yet. **Strengthens the existing queue item for named A2A participants.** +- **Cursor — Cloud Agent Development Environments (May 12)** — pre-run setup wizard where the agent gathers config/secrets via structured prompts before any task runs. — https://www.cursor.com/blog/cloud-agent-development-environments — *what it is*: productized elicitation flow, distinct from in-conversation tool consent. — *fit*: pattern-only; closest MCP analog is SEP-elicitation. Bookmark for whenever we tackle MCP elicitation UX. Not on the near-term roadmap. +- **assistant-ui 0.14.2 (May 13)** — CI plumbing only (disable OIDC pre-flight verification). The substantive 0.14.0 release fell May 7 (one day outside window). — *fit*: not applicable this week; watch for an `mcp-app-studio` follow-up. +- **blog.modelcontextprotocol.io — quiet (no new posts in window).** Last MCP-blog post was April 8. +- **NN/Group AI topic URL 404'd** — try `/articles/?topic=ai` or a search path next week. + +#### Frontier model announcements +- **OpenAI — deprecation notice for `gpt-5.2-chat-latest` and `gpt-5.3-chat-latest` snapshots (May 8)** — pairs with last week's GPT-5.5 Instant displacing 5.3 flag. — https://community.openai.com/t/deprecation-notice-upcoming-model-shutdowns-in-2026/1379553 — *relevance*: informational (we don't ship OpenAI in the selector); useful as a comp pattern for our own model-lifecycle UX in the model-selector. — *risk*: low; only material if we proxy OpenAI. +- **OpenAI — Realtime API Beta removed (May 12)**, **DALL·E 2/3 snapshots removed (May 12)** — informational; no impact on `backend/src/apis/app_api/voice/` (own ticket flow) or any of our image paths. +- **Anthropic — Claude for Small Business (May 13)** — packaging/connectors announcement; QuickBooks, PayPal, HubSpot, Canva, DocuSign, Google Workspace, Microsoft 365. — https://www.anthropic.com/news/claude-for-small-business — *relevance*: informational; foreshadows which integrations Anthropic considers "table stakes." Hints at first-party MCP tools we might otherwise build. +- **Anthropic — $200M Gates Foundation partnership (May 14)** — non-technical; no capability deltas. — https://www.anthropic.com/news/gates-foundation-partnership +- **Google DeepMind + Meta — quiet (two weeks running).** No Gemini or Llama posts in window. Frontier signal is now coming exclusively from Anthropic + OpenAI. + +#### Agent harness patterns +- **Claude Code 2.1.139 — `/goal` command + persistent agent view** — sets a turn-spanning completion condition with live overlay (elapsed time / turns / tokens). Hooks declared in agent frontmatter fire when that agent runs as the main thread via `--agent`. — *applicability*: HIGH — maps cleanly onto our SSE pipeline (we already stream `metadata` tokens and emit `compaction`). *Inspires*: a `goal` SSE event + sidebar overlay showing user-declared objective + turn count + token-budget burn-down, particularly useful for long agentic threads. — https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md +- **Claude Code 2.1.141 — `duration_ms` on PostToolUse hooks** — per-tool wall-clock cost without instrumenting each tool. — *applicability*: HIGH — Strands has a hook system we already lean on (planned context-attribution prototype is hook-based per our memory). *Inspires*: emit per-tool `duration_ms` in our `tool_result` SSE payload + a faint inline timing badge on tool blocks. Feeds the planned context-attribution work by separating tool latency from token cost. +- **Claude Code 2.1.142 — `MCP_TOOL_TIMEOUT` per-request override** — removes the hardcoded 60s ceiling on remote MCP. — *applicability*: MEDIUM — our Gateway-MCP path has the 15-min Lambda cap as the real ceiling, but our app_api → inference_api SSE has its own 600s timeout. *Inspires*: per-MCP-target timeout in our Gateway target registry (today one global) + surfacing remaining-time in the `tool_use` event so the UI can show a progress hint on slow MCP calls. +- **Anthropic engineering blog — no posts in window.** + +#### Pricing / quota +- **AgentCore Payments launch (May 11)** — preview only; AWS charges $0 for the service, wallet-provider passes through (Coinbase CDP ~$0.005/wallet op). — *impact*: none on inference_api model selection or cost-badge values today; relevant only if we wire an agent to autonomously pay for external APIs/MCP servers. — https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-bedrock-agentcore-payments-agent-toolkit-for-aws-and-more-may-11-2026/ +- **AgentCore base pricing unchanged.** Runtime/Browser/Code-Interpreter still $0.0895/vCPU-hr + $0.00945/GB-hr; Memory $0.25/1K events; Identity $0.010/1K token requests (free via Runtime/Gateway); Gateway $0.005/1K invocations. +- **Bedrock model pricing page** — WebFetch extraction returned only Claude 3.5 entries (same limitation as last week). No model-pricing announcements detected. Worth a manual eyeball or direct curl next week. — https://aws.amazon.com/bedrock/pricing/ + +#### Open AgentCore SDK issues affecting us +- **PR #478 — feat: add async support to MemorySessionManager** — opt-in `async_mode: bool` on `AgentCoreMemoryConfig`; when true, wraps sync methods in `asyncio.to_thread()`. **Directly resolves last week's #452**. Likely lands in 1.10.0. — https://github.com/aws/bedrock-agentcore-sdk-python/pull/478 — *applicability*: HIGH +- **Issue #471 — docs: `/ping` response requires undocumented `time_of_last_update` field** — without that field, AgentCore's idle reaper kills microVMs even while `/ping` returns `HealthyBusy`; AWS docs only show `{"status": "HealthyBusy"}`. — *applicability*: HIGH — we run long-running streaming responses on the inference-api runtime. If our `/ping` doesn't emit `time_of_last_update`, we may be experiencing silent microVM reaping on long generations. **Grep our ping handler today.** — https://github.com/aws/bedrock-agentcore-sdk-python/issues/471 +- **Issue #468 — BedrockAgentCoreApp SDK needs update for expanded Request Header Allowlist** — AgentCore CP now allows arbitrary HTTP headers; SDK doesn't yet surface them to the handler. — *applicability*: MEDIUM — unlocks passing trace IDs / tenant hints from BFF/app-api into inference-api without the `X-Amzn-...-Custom-` prefix dance. Not blocking. — https://github.com/aws/bedrock-agentcore-sdk-python/issues/468 +- **Issue #467 — Multi-agent (Graph) session support missing from AgentCoreMemorySessionManager** — `create_multi_agent` / `read_multi_agent` / `update_multi_agent` on Strands `SessionRepository` aren't implemented; Strands `Graph`-based flows fail at graph creation when paired with AgentCore Memory. — *applicability*: LOW-MEDIUM today (single-agent), HIGH if/when we adopt Strands Graph for sub-agent decomposition. — https://github.com/aws/bedrock-agentcore-sdk-python/issues/467 + +#### Cookbook / courses +- **Linear ↔ Managed Agents stateless webhook bridge** (May 13, `9644291`) — TS/Bun template: no held SSE, no session-map DB. `session.metadata` carries `linear_session_id` + `org_id`; uses `beta.webhooks.unwrap` with retrieve-then-filter and 10s ack. — https://github.com/anthropics/claude-cookbooks/commit/96442914bfee9842faa97b1d45ee7b43317f7391 — *what we'd borrow*: the stateless-bridge pattern is the cleanest pattern we've seen for wrapping streaming backends behind webhook-style async callers. Worth comparing to how `apis.shared` correlates async tool callbacks today. *Inspires*: stash external correlation IDs in `session.metadata` instead of a session-map table; future Slack/Teams entry points slot in cheaply. +- **CMA Sessions API as an MCP server (stdio + HTTP)** (May 13, `a090206`) — thin MCP wrapper with two entrypoints: `server.ts` for Claude Desktop stdio, `server-http.ts` for Streamable HTTP + bearer auth (claude.ai Connectors). `wait_for_idle` is an SSE→request/response shim. Authoring/destructive endpoints deliberately omitted. — *what we'd borrow*: the `wait_for_idle` SSE-to-request/response shim is a clean template for wrapping streaming backends as MCP tools — relevant if we ever expose our Strands agent loop as an MCP tool another agent can call. Also: deliberate-omission stance (no authoring/secrets) is a good template for our admin-endpoint MCP boundaries. +- **Registry: "Claude Managed Agents" category** (May 13, `c8b30f3`) — schema addition + retag of managed_agents notebooks under a new top-level category. — *takeaway*: Anthropic is consolidating CMA as a first-class product surface. Strategic data point on whether to keep building bridges *to* CMA vs. treat it as a peer to Bedrock AgentCore. + +#### Community + GitHub issues +- **HN Algolia (May 8–15) — 0 in-window hits** for `bedrock`, `agentcore`, `strands`, `mcp`, `claude code`. Index is functioning — `agentcore` returns 1,490 lifetime results, with the most recent (Payments) at May 7, one day before window. Quiet week confirmed. +- **Reddit `.rss` blocked at the domain level via WebFetch** (not just the HTML path). Last week's open queue item to "add Reddit `.rss`" is **infeasible via WebFetch** — needs a different transport (`curl` with UA header via Bash, or an RSS-fetching MCP). Recommend closing the queue item as infeasible. + +#### Seasonal +- Out of window — no re:Invent / NeurIPS / ICLR / EMNLP items. + +### Patterns worth considering + +- **Per-tool timing as a first-class hook output** (Claude Code 2.1.141) — directly enables our planned context-attribution prototype to separate tool latency from token cost. Cheap, mechanical, signal-rich. +- **`session.metadata` correlation-id pattern** (Linear↔CMA cookbook) — replaces dedicated session-map tables with a JSON blob the external system writes. Lifts a future infra ask (Slack/Teams entry points → schema work) to a zero-table approach. +- **Stateless MCP transport** (SEP-2567 + SEP-2575) — direct fit for our SigV4 Gateway model. Watch python `mcp` SDK for adoption; act when 1.28.0+ ships with the changes. +- **Tool-catalog ↔ executor consistency** (ref repo `a0753dc`) — generalizes the principle: any tool list emitted to the model must derive from the same post-RBAC filtered set the executor uses. Worth a quick audit on our system-prompt assembly. + +## Internal Audit + +### Activity (last 7 days) +- **Commits on develop**: 33 (across multiple squash-merged PRs and direct release/develop merges) +- **PRs merged into develop in window**: 8 user-facing (#290, #293, #296, #297, #298, #299, #300, #301) + 2 release back-merges (beta.25, beta.26) + 1 docs (#276 — pre-existing branch) +- **PRs opened**: 1 (#301 — session-list polish; the working branch this run is on) +- **PRs reverted**: 0 +- **Issues opened**: 1 (#288 — inference-api deploy: new images reach ECR but live AgentCore Runtime isn't rolled) +- **Releases**: beta.25 (May 12), beta.26 (May 14) +- **CI failures (workflow → count, last 7 days)**: + - Version Check: 7 (all Dependabot — see below) + - Deploy App API: 5 + - Deploy Inference API: 3 + - Deploy Infrastructure: 2 + - **Nightly Build & Test: 0** (last week's #6 idea — fixed via #290 on May 12) + +### Repeated friction signals +- **Inference API deploy not rolling new Runtime versions** (issue #288, May 12) — new container images reach ECR but the live AgentCore Runtime isn't picked up. Correlates with the 3 Deploy Inference API failures this week. *Hypothesis*: deploy step is publishing the image but not bumping the Runtime version pointer or invoking `update_agent_runtime`. *Fix candidate*: trace the deploy workflow against the AgentCore Runtime versioning model — verify it calls the SDK's `update_agent_runtime` (or equivalent) after the ECR push. +- **Deploy App API failures clustered (5 in 7 days)** — three on the `feature/fix-default-model-persistence` branch on May 12, two on `feature/skip-auth-local-bypass` on May 9 (pre-`#272` merge). Branch-level, not a `develop` regression — most are pre-merge breakages getting resolved by the author. +- **Nightly Build & Test cluster from last week — RESOLVED.** Last failure was May 8; PR #290 landed May 12 and the cluster has been silent since. *Last week's #6 worked.* + +### Version-pin lag + +| Dep | Pinned | Latest | Lag | Notes | +|---|---|---|---|---| +| `bedrock-agentcore` | 1.6.4 | **1.9.1** | 4 minor+patch / latest 2026-05-12 | **Widening from last week (was 3).** PR #478 in flight adds `async_mode` to `MemorySessionManager` — directly resolves last week's flagged #452. | +| `strands-agents` | 1.39.0 | **1.40.0** | 1 minor / latest 2026-05-14 | **New release in window.** Headline: proactive context compression (PR #2239). **Carries breaking default flip on `use_native_token_count` (true → false)** — audit token-metric reads before bumping. | +| `fastmcp` (transitive on external MCP servers) | n/a in this repo | **3.3.0** | n/a | New `fastmcp-slim` client distribution. Informational. | +| `mcp` (transitive) | (transitive) | 1.27.1 | 1 patch | Patch-only release. NO SEP-2567/2575 adoption — watch 1.28.0. | +| `boto3` | 1.42.96 | (not re-fetched this week) | — | Last week 1.43.6 was latest; routine bumps continue. | +| `aws-cdk-lib` | 2.251.0 | (not re-fetched this week) | — | Routine. | +| `@angular/core` | 21.2.11 | (not re-fetched this week) | — | Routine. | + +> **Note**: #293 disabled Dependabot version-update PRs on 2026-05-13. The kaizen loop is now the only mechanism catching version-pin lag — version-pin diff has been promoted from "routine" to "load-bearing." + +### Retirement candidates +- **Last week's queue item "Add Reddit `.rss` or Reddit MCP to `kaizen-research`"** — confirmed Reddit is blocked at the domain level via WebFetch (not just the HTML path). Close the queue item as **infeasible via WebFetch**; revisit only if a different transport becomes available (Reddit MCP, or `curl`-via-Bash with UA header). +- **`kaizen-research/SKILL.md` AgentCore starter-toolkit URL is wrong** — current URL references `aws/amazon-bedrock-agentcore-starter-toolkit`; correct slug is `aws/bedrock-agentcore-starter-toolkit` (no `amazon-` prefix). Fold into the existing queue item "Replace dead source URLs in `kaizen-research` skill." +- **`anthropics/courses` source** — already queued for removal last week; confirmed quiet again this week. Leave queued. + +### Risks introduced this week +- **Dependabot version-update PRs disabled (#293, May 13)** — no automated bump pressure. Kaizen loop is now the *only* mechanism catching version-pin lag. *What breaks if ignored*: version-pin lag silently widens; security patches in transitive deps may not arrive until something else surfaces them. — *mitigation*: tighten the version-pin diff section of this skill (already in scope); add direct fetches for `boto3`, `aws-cdk-lib`, `@angular/core`, `pydantic` every run. +- **`bedrock-agentcore` now 4 releases behind, with a release in the scan window** (1.9.1, May 12) — *what breaks if ignored*: still-open OTEL trace detach (#456), event-loop blocking (#452 — fix in PR #478 likely 1.10.0). Same risk as last week but worse — 3rd week of carry-over would be embarrassing. +- **Strands v1.40 `use_native_token_count` default flip** — if we bump without auditing, token-accounting code reading native counts gets heuristic values instead. — https://github.com/strands-agents/sdk-python/pull/2284 +- **AgentCore `/ping` may not emit `time_of_last_update`** (SDK issue #471) — silent microVM reaping on long generations. — *what breaks if ignored*: long agent turns get killed by the idle reaper even while we're streaming. **Grep our ping handler.** — https://github.com/aws/bedrock-agentcore-sdk-python/issues/471 +- **Inference-api deploy doesn't roll the live AgentCore Runtime** (our #288) — manual-redeploy band-aid; eventually a security patch or model-version bump will need to ship and won't. + +## Ideas — Top 5 (ranked) + +| # | Idea | Surface | Effort | Impact | Subtracts? | Unlocks? | +|---|---|---|---|---|---|---| +| 1 | Bump `bedrock-agentcore` 1.6.4 → 1.9.1 (now 4 releases behind — re-prioritized from last week) | backend | L | M-H | no — addition only, but retires open queue carry-over; sets up adoption of PR #478 / `async_mode` once 1.10.0 ships | — | +| 2 | Audit and fix `/ping` to emit `time_of_last_update` per AgentCore SDK issue #471 | backend (inference-api `/ping`) | L | M-H | no — defensive against silent microVM reaping on long generations | — | +| 3 | Strands 1.39 → 1.40 bump, gated on `use_native_token_count` audit + proactive-compression double-fire check | backend | M | M-H | **yes — adopting upstream proactive context compression (PR #2239) reduces our custom compaction surface area** | — | +| 4 | Defensive A2A AgentCard `capabilities={"streaming": True}` check (ref repo `50c9112`) | backend (A2A construct) | L | M | no — defensive; silent-failure mode is 40-min timeouts | — | +| 5 | Wire per-tool `duration_ms` into our `tool_result` SSE payload (Claude Code 2.1.141 pattern) | backend (Strands hook) + frontend (faint inline badge) | L-M | M-H | partial — replaces ad-hoc per-tool timing with a single hook-driven field; pre-paves the planned context-attribution prototype | per-tool timing visibility in the UI; the data substrate for context-attribution that separates latency from token cost | + +### 1. Bump `bedrock-agentcore` 1.6.4 → 1.9.1 +- **Source**: PyPI (https://pypi.org/project/bedrock-agentcore/) + open SDK issues #456, #452, #471. *Re-prioritized from last week's queue — lag widened 3 → 4 releases and Dependabot version-updates are now disabled (#293), so this won't get there on its own.* +- **Surface area**: `backend/pyproject.toml`, `backend/uv.lock` +- **Change**: pin update + smoke-test memory + identity flows in dev. Verify CHANGELOG between 1.6.4 and 1.9.1 for breaking changes. Verify whether 1.9.1 already addresses #456 (OTEL trace detach across asyncio boundaries) — if so, close. Track PR #478 (`async_mode`) for the 1.10.0 follow-up. +- **Subtracts**: addition only — justified by 4 versions of upstream fixes and the kaizen-loop accountability for catching version-pin lag now that Dependabot is off +- **Effort × Impact**: Low × Medium-High +- **Verdict**: Worth trying. *This is a re-prioritization, not a new idea — the bootstrap-week version is still open in the queue and is now stale.* + +### 2. Audit and fix `/ping` to emit `time_of_last_update` +- **Source**: AgentCore SDK issue #471 (https://github.com/aws/bedrock-agentcore-sdk-python/issues/471) +- **Surface area**: `backend/src/apis/inference_api/` (the `/ping` handler) — this is one of the two routes the AgentCore Runtime data plane actually serves (per CLAUDE.md inference-api boundary section). +- **Change**: grep our ping response shape; if it's just `{"status": "Healthy" | "HealthyBusy"}`, extend it to also emit `{"time_of_last_update": }`. Without that field AgentCore's idle reaper can kill microVMs mid-long-generation even while we're streaming. +- **Subtracts**: addition only — defensive; silent failure mode +- **Effort × Impact**: Low × Medium-High +- **Verdict**: Worth trying — cheap, surface area is tiny, failure mode is silent and bad + +### 3. Strands 1.39 → 1.40 bump +- **Source**: Strands v1.40.0 release notes + PR #2284 (breaking) +- **Surface area**: `backend/pyproject.toml`, `backend/uv.lock`, our compaction-surfacing SSE path, any code reading native Bedrock input-token counts +- **Change**: (a) audit `apis/shared/` and `agents/main_agent/streaming/` for native-token-count reads — if we depend on them, either pin `BedrockModel(use_native_token_count=True)` explicitly OR re-route through the heuristic and verify our cost-badge math (recall last week's #270 already touched this); (b) bump pin; (c) verify proactive context compression (PR #2239) doesn't double-fire with our existing `TurnBasedSessionManager` flush — our compaction SSE event should still emit cleanly. (d) Smoke-test cost-badge values across a tool turn before promoting. +- **Subtracts**: **yes — library-native subtraction.** Strands' proactive context compression replaces compaction logic we'd otherwise grow. Reduces the surface area in our custom session manager. +- **Effort × Impact**: Medium × Medium-High +- **Verdict**: Worth trying. The bump is straightforward; the audit is the careful part. + +### 4. Defensive A2A AgentCard `capabilities={"streaming": True}` check +- **Source**: aws-samples/sample-strands-agent-with-agentcore commit `50c9112` +- **Surface area**: wherever we construct A2A AgentCards in this repo (search `AgentCard`, `capabilities=`, `agent_card`) +- **Change**: 30-second grep + read; if the field is missing or `False`, set to `True`. Silent failure mode otherwise: A2A SDK client falls back to non-streaming, never receives `completed`, hangs 40 minutes. +- **Subtracts**: addition only — defensive +- **Effort × Impact**: Low × Medium +- **Verdict**: Worth trying. Cheapest item in the list with a real silent-failure mode. + +### 5. Wire per-tool `duration_ms` into `tool_result` SSE +- **Source**: Claude Code 2.1.141 hook pattern (https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md) +- **Surface area**: backend Strands hook (PostToolUse equivalent) emitting into our SSE `tool_result` payload; frontend `` component renders a faint inline timing badge +- **Change**: register a Strands `AfterToolCall` hook that captures `(end - start)` wall-clock per tool invocation; emit on the existing `tool_result` SSE event as `duration_ms`. Frontend renders inline timing only if `> 250ms` (avoid noise). +- **Subtracts**: partial — replaces ad-hoc per-tool timing (if any) with a single hook-driven field; *more importantly, it's the data substrate for the planned context-attribution prototype* (per our memory) — separating tool latency from token cost +- **Unlocks**: per-tool timing visibility in the UI (which slow tool is the bottleneck on this turn?); the data substrate for context-attribution that distinguishes latency from token cost +- **Effort × Impact**: Low-Medium × Medium-High +- **Verdict**: Worth trying. Pairs naturally with the planned context-attribution work — landing this first means the prototype starts with cleaner inputs. + +## Take + +Two patterns are quietly reshaping the kaizen loop. **First, the upstream-shrinks-our-backlog play keeps paying off**: this week Strands shipped the proactive compression we'd flagged, AgentCore SDK has the `async_mode` fix for our #452 in flight, and the ref repo's `50c9112` A2A streaming-capability fix is a 30-second port. **Second, #293 disabled Dependabot version-update PRs** — the kaizen loop is now the only mechanism catching version-pin lag, which makes the *re-prioritization* of last week's bedrock-agentcore bump (now 4 releases behind, not 3) the single most important change this week. The agentic-UI/MCP-Apps storyline that dominated bootstrap week has gone quiet — slow week on UX, normal noise on ecosystem. Idea #1 is a carry-over but earns its #1 by being stale; idea #2 (the `/ping` fix) is the most consequential silent-failure mitigation we can land cheaply. If Phil ships only two this week, those are the two. + +--- + +## Sources Scanned + +| # | Source | URL | Accessed | Items | +|---|---|---|---|---| +| 1 | AWS Bedrock + AgentCore (RSS, blog, AWS weekly roundup) | aws.amazon.com / github.com/aws/bedrock-agentcore-* | 2026-05-15 | 3 announcements + 4 SDK items (PR #478, issues #467/#468/#471) | +| 2 | Strands Agents (releases, issues, PRs) | github.com/strands-agents/sdk-python | 2026-05-15 | 1 release (v1.40.0) + 4 issues/PRs (#2266, #2284, #2286, #2290) | +| 3 | Reference repo (12 commits in window) | github.com/aws-samples/sample-strands-agent-with-agentcore | 2026-05-15 | 4 architecturally relevant commits (50c9112, d668685, a0753dc, 8d111d9) | +| 4 | MCP ecosystem (SEPs, python-sdk, awslabs/mcp) | modelcontextprotocol.io / github.com/modelcontextprotocol / github.com/awslabs/mcp | 2026-05-15 | SEP-2575 merge + python-sdk v1.27.1 + 2 awslabs PRs + SEP-2577 | +| 4a | FastMCP (v3.3.0 + active PRs) | github.com/PrefectHQ/fastmcp + pypi.org/project/fastmcp | 2026-05-15 | 1 release (3.3.0) + 4 active PRs | +| 4b | Agentic UI/UX (MCP blog, assistant-ui, Linear, Cursor, Anthropic, NN/g) | blog.modelcontextprotocol.io + linear.app + cursor.com + anthropic.com + nngroup.com | 2026-05-15 | 2 in-window posts (Linear Code Intelligence, Cursor cloud envs) + 1 assistant-ui CI release + NN/g 404 noted | +| 5 | Frontier models (Anthropic, OpenAI, Google, Meta) | anthropic.com / openai.com / blog.google / ai.meta.com | 2026-05-15 | 2 Anthropic non-technical + 3 OpenAI deprecations + 0 Google/Meta | +| 6 | Agent harness (Claude Code CHANGELOG, claude-cookbooks, Anthropic engineering) | github.com/anthropics/* | 2026-05-15 | 3 Claude Code releases (2.1.139/141/142) + 3 new cookbook artifacts | +| 7 | AWS Bedrock pricing + quota | aws.amazon.com/bedrock/pricing | 2026-05-15 | 0 detected changes; AgentCore Payments launch context only | +| 8 | Community (HN Algolia + Reddit) | hn.algolia.com + reddit.com | 2026-05-15 | 0 HN in-window; Reddit `.rss` confirmed domain-blocked via WebFetch | + +## Web Budget + +Used: ~46 / 50 requests (target). +- AWS Bedrock/AgentCore: 4–5 (one 404 on a `/whats-new/2026/05/` aggregator) +- Strands: 4 (mostly `gh api`) +- Reference repo: 4 (`gh api`) +- MCP ecosystem: 4 (mostly `gh api`, 1 web) +- FastMCP: 2 `gh api` +- Agentic UI/UX: 6 (one 404 on NN/g topic URL) +- Frontier models: 6 (1 over sub-budget — OpenAI 403 backfilled via search) +- Agent harness: 4 +- Pricing: 3 +- AgentCore SDK: 4 (mostly `gh api`) +- Community: 3 (HN + 2 Reddit attempts that failed fast) +- Cookbook: 2 `gh api` (no web) + +Skipped (unreachable): +- Reddit `.rss` — domain-blocked via WebFetch (confirmed). Closing the existing queue item as infeasible-via-WebFetch. +- NN/Group AI topic URL `https://www.nngroup.com/topic/artificial-intelligence/` 404'd. Try `/articles/?topic=ai` or search next week. +- `https://aws.amazon.com/about-aws/whats-new/2026/05/` 404'd. RSS feed covers this anyway. + +Skipped (other): seasonal sources (out of window). + +Notes: +- Frontier-models sub-budget overshot by 1 (OpenAI 403 required search backfill — same failure mode as bootstrap). +- Bedrock pricing-page extraction returned only Claude 3.5 entries for the second week running. If a third scan misses, switch to `curl` + grep instead of WebFetch summarization. diff --git a/docs/kaizen/review-queue.md b/docs/kaizen/review-queue.md new file mode 100644 index 00000000..d97d94cd --- /dev/null +++ b/docs/kaizen/review-queue.md @@ -0,0 +1,114 @@ +# Kaizen Review Queue + +Items added by `kaizen-research`, consumed by `kaizen-review-prep`. + +## Open + +### [2026-05-15] Wire per-tool `duration_ms` into `tool_result` SSE +- **Source**: research/2026-05-15.md ▸ Top 5 #5 — Claude Code 2.1.141 hook pattern +- **Surface**: backend (Strands `AfterToolCall` hook) + frontend (`` component — inline timing badge for `> 250ms`) +- **Effort × Impact**: L-M × M-H +- **Subtracts**: partial — single hook-driven field replaces any ad-hoc per-tool timing; pre-paves the planned context-attribution prototype +- **Unlocks**: + - Per-tool timing visibility in the UI (which slow tool is the bottleneck on this turn?) + - Data substrate for the planned context-attribution prototype — separates tool latency from token cost +- **Status**: open — surfaced in reviews/2026-05-15.md ▸ Proposal #3 (Ship); no decision logged yet + +### [2026-05-15] Investigate inference-api deploy — new images reach ECR but Runtime isn't rolled (issue #288) +- **Source**: reviews/2026-05-15.md ▸ Proposal #10 (new from internal friction, issue #288 May 12). Pairs with the 1.6.4 → 1.9.1 bump (same SDK package owns `update_agent_runtime`). +- **Surface**: cross-cutting — `.github/workflows/deploy-inference-api.yml` + bedrock-agentcore SDK `update_agent_runtime` call shape +- **Effort × Impact**: L-M × M-H +- **Subtracts**: possibly — removes the manual-redeploy band-aid that's been the workaround +- **Status**: open — surfaced in reviews/2026-05-15.md ▸ Proposal #10 (Ship — recommended ship-first); no decision logged yet. **Friction intensifying**: 6+ "Deploy Inference API" failures May 15–17; a new "Deploy App API" failure cluster (8× May 16–17) may share a root cause. + +### [2026-05-10] Scope AgentCore Runtime BYO filesystem (S3 Files / EFS) for persistent agent workspaces +- **Source**: research/2026-05-10.md ▸ AWS Bedrock / AgentCore (re-evaluated 2026-05-10 via strategic-lens follow-up — original framing under-weighted the capability-unlock angle) +- **Surface**: backend (`inference-api` invocation handler reads/writes mount) + infrastructure (VPC config, IAM mount permissions, S3 Files or EFS access points, per-user prefix/access-point layout for RBAC); ADR-worthy +- **Effort × Impact**: H × H +- **Subtracts**: no — pure capability addition +- **Unlocks**: + - Code-interpreter / persistent agent workspace (artifacts survive turn and session boundaries) + - Cross-session file uploads — PDFs/spreadsheets persist between conversations instead of re-staging per session + - Shared skill/template/prompt hot-swap without redeploying the runtime container + - A2A multi-agent intermediate-result handoff via shared mount + - Persistent vector indexes / embedding caches — avoids cold-start rebuild +- **Open questions**: GA vs preview status (March 2026 managed session storage was preview; May 2026 BYO needs verification); VPC requirement is a new architectural surface for the runtime; multi-tenancy isolation strategy (per-user S3 prefix vs per-user EFS access point); RBAC mount-path layout; runtime data plane still only proxies `/invocations` + `/ping` so this doesn't unlock new HTTP routes +- **Status**: open — deferred 4 weeks in reviews/2026-05-15.md (revisit 2026-06-12). MCP Apps host renderer is the dominant strategic initiative this cycle; layering another ADR-worthy bet on top would double the open architectural surface. + +### [2026-05-10] Audit `BedrockModel.stream` cancellation path against Strands #2266 +- **Source**: research/2026-05-10.md ▸ Top 6 #4 +- **Surface**: backend +- **Effort × Impact**: L × M-H +- **Subtracts**: no — defensive (SSE-disconnect path is hot) +- **Status**: open — surfaced in reviews/2026-05-15.md ▸ Proposal #8 (Ship); no decision logged yet + +### [2026-05-10] Audit `oauth_required` SSE flow against ref-repo's mid-tool-call 401/403 handling +- **Source**: research/2026-05-10.md ▸ Risks +- **Surface**: backend +- **Effort × Impact**: M × H +- **Subtracts**: no — defensive +- **Status**: open — deferred 2026-05-10 until 2026-05-24. BFF parade declared done via #297 (May 14), so deferral conditions have cleared a week early; reviews/2026-05-15.md holds to original revisit date to give one stable week. + +### [2026-05-10] Named A2A agent participants in the chat UI +- **Source**: research/2026-05-10.md ▸ Agentic UI/UX ▸ Linear Agent pattern. Reinforced by research/2026-05-15.md Linear Code Intelligence 5× usage-growth datapoint. +- **Surface**: frontend (extend message model with `agent_identity`, distinct avatar/name/styling) +- **Effort × Impact**: L-M × M +- **Subtracts**: no — additive but pattern-validated across Linear/ChatGPT/Cursor +- **Status**: open — deferred 4 weeks in reviews/2026-05-15.md (revisit 2026-06-12). Earns its keep when an A2A construct lands. + +## Resolved + +### [2026-05-15] Strands 1.39 → 1.40 bump (token-count audit + compaction double-fire check) → RESOLVED — shipped +- **Decision**: Ship — reviews/2026-05-15.md ▸ Proposal #6 +- **Reasoning**: Shipped in PR #340 (`chore(deps): bump strands-agents 1.39.0 → 1.40.0`, merged 2026-05-18). Audit outcome: **accept the new `use_native_token_count=False` default** — the flag gates only `BedrockModel.count_tokens()`, which nothing in our cost / context-% paths reads (those read native Bedrock Converse `usage`); pinning `True` would add a redundant CountTokens API call per invocation. Compaction double-fire **confirmed absent** — Strands proactive compression is opt-in (`proactive_compression=None` default), operates on `ConversationManager` not our `TurnBasedSessionManager`; the `compaction` SSE event still emits exactly once (PR #243 invariant preserved; new regression test `test_compaction_sse_emit_once.py`). Full local backend suite: 2887 passed / 3 skipped on 1.40. +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #6 + +### [2026-05-10] Promote tool-result rendering to a per-tool renderer registry (MCP Apps PR #0) → RESOLVED — shipped +- **Decision**: Ship — reviews/2026-05-15.md ▸ Proposal #5 +- **Reasoning**: Shipped in PR #339 (`refactor(chat): tool-result renderer registry (MCP Apps PR #0)`, merged 2026-05-18). Pure refactor — implicit text/JSON/image switch lifted into a signal-backed `ToolRendererRegistryService` keyed by tool name; `DefaultToolResultComponent` reproduces prior markup verbatim (zero user-visible change); `calculator` / `fetch_url_content` / `create_visualization` migrated as proof points. 1014/1014 frontend tests green (14 new, DI-token overrides not `vi.mock`). Unblocks MCP Apps PR #1; the PR #4 MCP App renderer now plugs in as just-another-registered-renderer. +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #5 + +### [2026-05-15] Bump `bedrock-agentcore` 1.6.4 → 1.9.1 → RESOLVED — shipped +- **Decision**: Ship — reviews/2026-05-15.md ▸ Proposal #1 +- **Reasoning**: Shipped in PR #337 (`chore(deps): bump bedrock-agentcore 1.6.4 → 1.9.1 (+ coupled boto3 1.43.9)`, merged 2026-05-18). Closes the structural version-pin lag now that Dependabot version-updates are disabled (#293); first proof the kaizen loop catches lag without Dependabot. +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #1 + +### [2026-05-15] Audit and fix `/ping` to emit `time_of_last_update` (#471) → RESOLVED — shipped +- **Decision**: Ship — reviews/2026-05-15.md ▸ Proposal #2 +- **Reasoning**: Shipped in PR #338 (kaizen bundle, merged 2026-05-18). `/ping` now emits an integer `time_of_last_update` + corrected `Healthy` casing. Accepted trade-off documented in the PR: a fresh per-ping timestamp disables ping-based idle reaping for this runtime — we can't report `HealthyBusy` without async-task busy tracking (deferred `async_mode` work). +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #2 + +### [2026-05-15] Defensive A2A AgentCard `capabilities={"streaming": True}` check → RESOLVED — guard documented +- **Decision**: Ship (docs-only) — reviews/2026-05-15.md ▸ Proposal #4 +- **Reasoning**: Resolved in PR #338 (merged 2026-05-18). A2A is client-only today (no server `AgentCard` exists), so there is no code site to patch. Added a forward-looking guard to `CLAUDE.md`: the first A2A server construct MUST advertise `capabilities` with `streaming=True`, else A2A clients hang ~40 min (ref-repo `50c9112`). +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #4 + +### [2026-05-10] Close issues #266 and #267 — features already in our Strands 1.39 pin → RESOLVED — decided (NOT closed; premise corrected) +- **Decision**: Decided, premise corrected — reviews/2026-05-15.md ▸ Proposal #7 (via PR #338) +- **Reasoning**: The review's "phantom tech debt — close them" framing was **wrong**. #266 (large tool-result offload) and #267 (context-window lookup fallback) are live, well-specified Strands adoption/wiring tasks whose 1.39 precondition is now met. Decision (PR #338, GitHub-only): posted "unblocked, keep open" comments on both — NOT closed. Logged in decisions.md so future research does not re-propose closing them. +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #7 + +### [2026-05-10] Replace dead source URLs in `kaizen-research` skill (+ starter-toolkit slug) → RESOLVED — shipped +- **Decision**: Ship — reviews/2026-05-15.md ▸ Proposal #9 +- **Reasoning**: Shipped in PR #338 (merged 2026-05-18). Replaced/dropped dead source URLs in `kaizen-research/SKILL.md`; fixed `aws/amazon-bedrock-agentcore-*` → `aws/bedrock-agentcore-*` slug — the review flagged the starter-toolkit; the sdk-python line had the same typo and was also fixed. +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #9 + +### [2026-05-10] Add Reddit `.rss` or Reddit MCP to `kaizen-research` → RESOLVED — declined +- **Decision**: Decline — reviews/2026-05-15.md ▸ Retirement Candidates +- **Reasoning**: research/2026-05-15.md confirmed Reddit is blocked at the *domain* level via WebFetch (not just the HTML path), so the proposal as scoped is infeasible. Logged in decisions.md; revisit only if a Reddit MCP or `curl`-via-Bash-with-UA-header path becomes available. +- **Reviewed-in**: reviews/2026-05-15.md ▸ Retirement Candidates + +### [2026-05-10] Scope an MCP Apps host renderer in our chat (multi-PR initiative) → RESOLVED — scoping landed +- **Decision**: Ship (scope only) — reviews/2026-05-10.md ▸ Proposal #1 +- **Reasoning**: Scoping doc `docs/kaizen/scoping/mcp-apps-host-renderer.md` landed in PR #296 (May 14, 2026). Four open architectural questions locked: sandbox-proxy origin, app-initiated `tools/call` plumbing, `ui/update-model-context` storage in Strands `agent.state`, full v1 method scope. PR #0 → PR #6 sequence defined; build work is now tracked via the renderer-registry queue item (PR #0 of that sequence). +- **Reviewed-in**: reviews/2026-05-10.md ▸ Proposal #1 + +### [2026-05-10] Triage Nightly Build & Test failure cluster (9× since May 6) → RESOLVED — fixed +- **Decision**: Ship — reviews/2026-05-10.md ▸ Proposal #6 +- **Reasoning**: PR #290 (`Fix e2e testing in nightly`, May 12) landed. The Nightly Build & Test workflow has been silent since — research/2026-05-15.md confirms 0 failures in the May 10–15 window. Loop caught and resolved CI hygiene. +- **Reviewed-in**: reviews/2026-05-10.md ▸ Proposal #6 + +### [2026-05-10] Bump `bedrock-agentcore` 1.6.4 → 1.9.0 → RESOLVED — superseded +- **Decision**: Superseded +- **Reasoning**: Replaced by the 2026-05-15 re-prioritized entry (`1.6.4 → 1.9.1`) — lag widened from 3 → 4 versions in window, and Dependabot version-updates were disabled by #293 (May 13), so the lag is now structural rather than incidental. The re-prioritized entry shipped in PR #337. +- **Reviewed-in**: reviews/2026-05-15.md ▸ Proposal #1 diff --git a/docs/kaizen/reviews/2026-05-10.md b/docs/kaizen/reviews/2026-05-10.md new file mode 100644 index 00000000..e48f395b --- /dev/null +++ b/docs/kaizen/reviews/2026-05-10.md @@ -0,0 +1,207 @@ +# Kaizen Review — Sunday, May 10, 2026 +> Prepared 11:00am MT. Review window: May 3 – May 10, 2026 (7 days). +> Source: research/2026-05-10.md + review-queue.md (8 open items). +> **Bootstrap run** — no prior reviews, no prior-week POC findings, no Carried Over items. Scope evolved mid-bootstrap: added FastMCP, library-native subtraction lens, and Agentic UI/UX lens; removed security posture lens (security is handled by Dependabot/CodeQL and doesn't need a weekly kaizen pass). + +## Week in Review + +Two themes braid this week. **Agentic UI/UX has shifted under us**: MCP Apps shipped to production with adoption from Claude Desktop, ChatGPT, VS Code Copilot, Goose, and Postman; the design conversation has moved from "what should an agent chat look like" to "how do we host other people's UIs in our chat." Our text+JSON tool-result rendering is now the baseline competitors are extending past. **Upstream-shrinks-our-backlog**: Strands v1.37/v1.38 quietly closed our open issues #266 and #267, `bedrock-agentcore` is 3 minor versions behind with likely fixes for two SDK issues we feel. The BFF migration is still v1 (5 of 8 commits this week are auth follow-ups), CI is unreliable. Net read: a "scope the big UX investment, harvest the upstream gains, stabilize hygiene" week. + +## Friction — the week's signal + +### Repeated patterns (≥2 occurrences) +- **CI deploy failures (6+ since May 6)** across Inference API, App API, and Frontend deploys. + - *Hypothesis*: BFF migration introduced env-var or stack drift not caught in synth (most failures cluster around the same auth/config landscape that's still being patched). + - *Candidate fix*: cross-check the most recent failed deploy log against the beta.23 → beta.24 stack diff; the most likely suspects are missing/renamed SSM parameters from the public-PKCE-client decommission (`/auth/cognito/app-client-id` removed) or the new `BFF_*` env vars. +- **Nightly Build & Test failures (9× since May 6)** — concentrated, untriaged. + - *Hypothesis*: known flakiness from issue #220 (`test_cognito_idp_service`, `test_oauth_repositories`, `test_auth_providers*` order-dependent) compounding under the BFF-heavy week's churn. + - *Candidate fix*: triage one failure log end-to-end. Either the root is #220 (then #220 needs to land) or it's a real regression masked by the noise. +- **BFF/auth fix churn (5 of 8 commits this week)** — #270, #271, #273, #274, #275, #277. + - *Hypothesis*: BFF migration is a v1, not a v1.1; expect 1–2 more weeks of follow-ups before declaring beta.25. + - *Candidate fix*: not a fix per se — adjust release-cut timing for beta.25 to wait for the churn to settle. + +### One-offs worth watching +- **`bedrock-agentcore` 1.6.4 → 1.9.0 lag** with a release published *inside* the scan window (May 7) — see proposal #1. +- **OpenAI displaces GPT-5.3 Instant with GPT-5.5 Instant** — model selector audit needed (proposal #6 — declined-by-default below; check first then decide). +- **Strands #2266 `BedrockModel.stream` cancel leak** — filed May 9; see proposal #2. + +### Silence that matters +- **No invocations of 6 of 9 skills in 60+ days** (angualar-best-practices, tailwind-ui, frontend-design, cdk-infrastructure, versioning, cors-deployment) — modification freshness is a weak signal for skills since they encode stable conventions; **not enough to act**, but worth instrumenting invocation telemetry if we want to make this a reliable retirement signal in the future. +- **HN was quiet on stack keywords this week** (0 hits in Algolia 7-day window) — not a problem; just a confirmation that this is an internal-momentum week, not a community-momentum week. +- **`anthropics/courses` quiet since Nov 2025** — proposal #6 below proposes dropping it from the scan list. + +## Proposals — ranked + +### 1. Scope an MCP Apps host renderer in our chat (multi-PR initiative) +- **Source**: research/2026-05-10.md ▸ Top 6 #1 ▸ Agentic UI/UX | review-queue.md (open) +- **Surface area**: frontend (new `` Angular component, tool-result rendering pipeline) + backend (new SSE event `ui_resource`; likely a `ui_consent_required` cousin of `oauth_required`) +- **Change**: implement the host side of MCP Apps (SEP-1865) — sandboxed iframe rendering `ui://` resources returned by MCP tools, with the `ui/`-prefixed JSON-RPC dialect over `postMessage`. Treat as a multi-PR initiative: (a) scoping/architecture doc + spec checklist, (b) SSE event + plumbing, (c) iframe component + postMessage bridge, (d) consent UX, (e) end-to-end demo with one example from `ext-apps/examples`. +- **Subtracts**: no — pure addition. Justified: Claude Desktop, ChatGPT, VS Code Copilot, Goose, and Postman ship this. Every third-party MCP server we connect could be shipping UIs richer than text+JSON; without a host, we leave that on the table. +- **Effort**: High (multi-PR; new SSE protocol event; sandboxed-iframe component; consent UX) +- **Impact**: High (strategic — agentic UI standard our chat is being measured against) +- **POC findings**: not POCed. +- **Ship means**: **scope this week, build over 3-4 weeks.** Open a scoping issue + architecture doc this week — not a code PR yet. Code PRs follow in subsequent sprints. +- **Decline means**: stay on text+JSON tool results; revisit when a third-party MCP server we connect ships an MCP App and we can't render it (the forcing function). +- **Recommendation**: **Ship (scope this week).** Highest strategic value of any item. Pre-paves with proposal #3. + +### 2. Bump `bedrock-agentcore` 1.6.4 → 1.9.0 +- **Source**: research/2026-05-10.md ▸ Top 6 #2 | review-queue.md (open) +- **Surface area**: backend (`backend/pyproject.toml`, `backend/uv.lock`) +- **Change**: pin update + smoke-test memory + identity flows in dev; verify upstream CHANGELOG between 1.6 and 1.9 for any breaking changes; close out open SDK issues #456 (OTEL trace detach) and #452 (event-loop blocking) if 1.9 addresses them. +- **Subtracts**: addition only — justified by 3 versions of upstream fixes including likely-relevant ones to OTEL trace detach and AgentCoreMemorySessionManager event-loop blocking. +- **Effort**: Low +- **Impact**: Medium (observability + concurrency) +- **POC findings**: not POCed — bootstrap run. +- **Ship means**: open a PR updating `pyproject.toml` and `uv.lock`; smoke-test memory write/read + identity OAuth flows in dev; if smoke passes, merge. +- **Decline means**: keep at 1.6.4 for another week; revisit after 1.10 ships or after we observe a memory/identity issue. +- **Recommendation**: **Ship.** Lowest effort × medium impact; clean upstream-harvest win. + +### 3. Promote tool-result rendering to a per-tool renderer registry (signal-backed) +- **Source**: research/2026-05-10.md ▸ Top 6 #3 ▸ Agentic UI/UX (AI SDK + Cursor canvases) | review-queue.md (open) +- **Surface area**: frontend (`` component + new `ToolRendererRegistry` service) +- **Change**: today our tool-result rendering is (implicitly) a switch in one place. Promote to a signal-backed registry keyed by tool name; per-tool renderers live next to the tool definition. Independently valuable AND the natural extension point for proposal #1 (MCP Apps becomes "just another registered renderer that emits an iframe"). +- **Subtracts**: partial — replaces an implicit switch with an explicit registry; the registry itself is new code, but it absorbs scattered tool-specific UI logic into one declarative table. +- **Effort**: Medium +- **Impact**: Medium-High (improves current tool-result UX AND pre-paves MCP Apps host) +- **POC findings**: not POCed. +- **Ship means**: open a PR that introduces the registry service + migrates 2-3 current tool renderers as a proof point. +- **Decline means**: keep the implicit switch; pay the cost when proposal #1 lands. +- **Recommendation**: **Ship.** Best risk-adjusted UX investment — value standalone AND scaffolding for proposal #1. + +### 4. Audit `BedrockModel.stream` cancellation path against Strands #2266 +- **Source**: research/2026-05-10.md ▸ Top 6 #4 | review-queue.md (open) +- **Surface area**: backend (`backend/src/agents/main_agent/` stream coordinator + SSE handler) +- **Change**: locate every `BedrockModel.stream` cancellation path; ensure each `await`s the inner task on cancel so it doesn't orphan; add a dev-only assertion / log filter to detect "Task exception was never retrieved" before it reaches prod. +- **Subtracts**: addition only — defensive. +- **Effort**: Low +- **Impact**: Medium-High (SSE-disconnect path is hot; orphan-task pressure is silent until it isn't) +- **POC findings**: not POCed. +- **Ship means**: open a PR with the audit + fixes + a regression test that triggers cancel + asserts no orphan tasks. +- **Decline means**: log a tech-debt issue; revisit if "Task exception was never retrieved" appears in CloudWatch. +- **Recommendation**: **Ship.** Pairs naturally with proposal #2 (same backend area, same week's Strands signal). Cheap insurance. + +### 5. Close issues #266 and #267 — features already in our Strands 1.39 pin +- **Source**: research/2026-05-10.md ▸ Top 6 #5 | review-queue.md (open) +- **Surface area**: cross-cutting — GitHub issues + small wiring in `stream_coordinator` and Code Interpreter / spreadsheet tool-result handling +- **Change**: + 1. Comment on #266 + #267 pointing at upstream PRs (Strands #2249 for context-window lookup; v1.38.0 release notes for large tool result offload). + 2. Verify the upstream features are *automatically* active under our 1.39 pin — if not, file replacement issues for the wiring work and link them. + 3. Close #266 + #267. +- **Subtracts**: **yes — library-native subtraction. Retires 2 "build from scratch" tickets; replaces with at-most 2 "wire upstream feature" tasks.** +- **Effort**: Low +- **Impact**: Medium (closes phantom tech debt; clears the issue list) +- **POC findings**: not POCed. +- **Ship means**: 30-minute issue-grooming pass; comment + close + (if needed) file 2 smaller follow-ups. +- **Decline means**: leave #266 + #267 open; future kaizen runs will re-flag them. +- **Recommendation**: **Ship.** Highest *subtraction* yield this week. The clearest demonstration of the kaizen loop earning its keep. + +### 6. Triage Nightly Build & Test failure cluster (9× since May 6) +- **Source**: research/2026-05-10.md ▸ Top 6 #6 | review-queue.md (open) +- **Surface area**: cross-cutting / CI (`.github/workflows/nightly-*.yml`, possibly `tests/shared/test_cognito_idp_service.py` per issue #220) +- **Change**: pull the most recent failure log; trace to root cause; if root is issue #220 (test isolation), bump #220 to blocking and land it; if it's a different cause, file and resolve. +- **Subtracts**: possibly — if root is #220, fixing it materially simplifies the suite (removes a tech-debt entry). +- **Effort**: Low-Medium (worst case: a real regression hiding under the noise) +- **Impact**: Medium-High (CI signal is currently unreliable; that affects *every* PR review) +- **POC findings**: not POCed. +- **Ship means**: 1-2 hour triage pass; either fix in a small PR or bump #220 to blocking. +- **Decline means**: continue ignoring nightly failures; eventually a real regression will hide here. +- **Recommendation**: **Ship.** This is independent of the kaizen-loop work above; it's hygiene. If the kaizen review is the venue that finally surfaces it, that's a kaizen win. + +### 7. Audit `oauth_required` SSE flow against ref-repo's mid-tool-call 401/403 handling +- **Source**: research/2026-05-10.md ▸ Risks | review-queue.md (open, deferred) +- **Surface area**: backend (`apis/shared/oauth/agentcore_identity.py`, SSE event emission in `inference-api`, MCP/A2A tool wrappers) +- **Change**: walk through the code paths where an external OAuth provider (Google etc.) returns 401/403 mid-stream; confirm the response is `oauth_required` SSE re-emission (consent-resume), not a streamed tool error to the user. Add a regression test if missing. +- **Subtracts**: addition only — defensive; closes a real UX gap when upstream tokens revoke. +- **Effort**: Medium (audit + likely 1-2 small fixes) +- **Impact**: High (user-visible UX; OAuth token revocation does happen) +- **POC findings**: not POCed. +- **Ship means**: open a tech-debt issue with the audit findings; fix in a follow-up PR if anything is broken. +- **Decline means**: assume current behavior is correct; revisit if a user reports a stuck-OAuth conversation. +- **Recommendation**: **Defer 2 weeks (revisit 2026-05-24).** Highest-impact backend proposal but BFF auth is still settling — landing this in the middle of the BFF patch parade risks compounding the churn. Wait for BFF to stabilize (likely beta.25), then audit cleanly. + +### 8. Named A2A agent participants in the chat UI +- **Source**: research/2026-05-10.md ▸ Agentic UI/UX ▸ Linear Agent pattern | review-queue.md (open) +- **Surface area**: frontend (extend message model with `agent_identity`; distinct avatar / name / styling for A2A sub-agent turns instead of nesting under generic `tool_use` cards) +- **Change**: when an A2A sub-agent produces output, render it as a distinctly attributed turn (named, avatar'd) rather than as a nested tool result. SSE already carries enough information; this is mostly a rendering change. +- **Subtracts**: no — additive but pattern-validated across Linear, ChatGPT agents, and Cursor multi-agent flows. +- **Effort**: Low-Medium +- **Impact**: Medium (legibility of multi-agent runs; user understanding of "who did what") +- **POC findings**: not POCed. +- **Ship means**: small PR extending the message model + a `` Angular component variant. +- **Decline means**: A2A sub-agent activity continues to be nested under tool cards; users can't easily tell when a different "actor" is responding. +- **Recommendation**: **Ship.** Low-effort UX win that future-proofs the chat for the A2A direction we're already heading. + +### 9. Replace dead source URLs in `kaizen-research` skill + drop `anthropics/courses` +- **Source**: research/2026-05-10.md ▸ Retirement candidates | review-queue.md (open) +- **Surface area**: skills (`.claude/skills/kaizen-research/SKILL.md`) +- **Change**: + - Replace `https://aws.amazon.com/bedrock/whats-new/` (404) with the AWS What's New RSS feed (already in the skill — drop the dead URL). + - Replace `https://docs.claude.com/en/docs/claude-code/release-notes` (301→404) with `https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md`. + - Drop `https://github.com/anthropics/courses` from the cookbook source (quiet since Nov 2025). +- **Subtracts**: **yes — replaces 2 broken URLs with working ones; drops 1 stale source.** +- **Effort**: Low +- **Impact**: Low (skill quality) +- **POC findings**: not POCed. +- **Ship means**: 5-minute edit to `kaizen-research/SKILL.md`. +- **Decline means**: leave dead URLs; future runs waste budget on them. +- **Recommendation**: **Ship.** Trivial, pure subtraction. Bundle with proposal #10 in one skill-update PR. + +### 10. Add Reddit `.rss` or Reddit MCP to `kaizen-research` +- **Source**: research/2026-05-10.md ▸ Risks ▸ "Reddit blocked from WebFetch" | review-queue.md (open) +- **Surface area**: skills (`.claude/skills/kaizen-research/SKILL.md`) +- **Change**: switch the community-signal subagent from raw `reddit.com` URLs to `https://www.reddit.com/r//.rss` (try first — `.rss` may be allowed where the HTML page isn't), or wire a Reddit MCP server if available. +- **Subtracts**: no — restores a half-blind source. +- **Effort**: Low (try `.rss` first; fall back to MCP if blocked) +- **Impact**: Low-Medium (community signal is one of 11 sources; useful but not load-bearing) +- **POC findings**: not POCed. +- **Ship means**: edit the skill's source list. +- **Decline means**: keep accepting "Reddit blocked" as a known gap. +- **Recommendation**: **Ship.** Bundle with proposal #9 in a single skill-update PR. + +## Carried Over From Prior Reviews + +Bootstrap run — none. + +## Retirement Candidates + +- **`enabled_tools` whitelist debug guidance in `CLAUDE.md`** — Reference repo abandoned this pattern May 3; ours isn't *wrong*, just diverging from the reference. **Recommendation**: monitor; revisit if/when we touch tool-enablement code. Not retire-this-week. +- **Skills not modified in 60+ days (6 of 9)** — modification freshness alone isn't enough signal to retire skills that encode stable conventions (e.g. `tailwind-ui`, `cdk-infrastructure`). **Recommendation**: **don't retire.** If we want this to be a reliable retirement signal, we'd need invocation telemetry — that's a separate proposal worth filing for a future week. + +## Risks Acknowledged But Not Acted On + +- **OpenAI GPT-5.3 → 5.5 Instant displacement** — https://openai.com/index/gpt-5-5-instant/ — *what breaks if ignored*: customers using a 5.3 default may hit a deprecation window. **Recommendation**: **Watch until 2026-06-01.** Quick check next week to confirm whether OpenAI is publishing a deprecation date for 5.3; if yes, file a model-selector audit. +- **MCP Apps adoption window** — every major host shipped support in Q1 2026. The longer we wait, the more we're shaping our tool-result UI in a direction that doesn't compose with where the ecosystem is going. **Recommendation**: scope this week (proposal #1); first code PR by 2026-05-31. + +## What Shipped This Week + +- **#277 — feat(auth): centralize 401 redirect + proactive session detection** (May 10) — *closes a real refresh-edge UX hole* +- **#275 — fix(bff): tighten cross-task refresh-lock release + absolute-lifetime guard** — *prevents zombie refresh attempts after lock release* +- **#274 — fix(bff): replace KMS-wrap data-key bootstrap with Secrets-Manager-generated secret** — *removes a bootstrap race the AES-GCM codec couldn't recover from* +- **#273 — fix(bff): cross-task cookie-codec & refresh-lock correctness** — *cleanup* +- **#272 — feat(auth): add SKIP_AUTH=true local-dev bypass with allowlist guard** — *unblocks local dev when Cognito is offline* +- **#271 — fix(auth): make lava-lamp backdrop dark-mode aware** — *visual polish* +- **#270 — fix(token-accounting): correct per-message cost and context-window semantics** — *fixes the cost badge accuracy* +- **#265 — chore(deps): upgrade strands-agents to 1.39.0** — *the upgrade that quietly closes #266 and #267* + +## Take + +The week's most valuable shipping is the strands-agents 1.39.0 bump (#265) — the team probably doesn't yet know it closed two of our open issues. That's the kaizen loop earning its keep on the upstream-harvest side. The new UI/UX lens — added mid-bootstrap — earned its keep too: it surfaced **MCP Apps** as a production-ready agentic UI standard that every major host already ships, and that our chat doesn't. The most consequential change this week if scoped is **proposal #1 (MCP Apps host renderer)** — high effort but high strategic value. The best risk-adjusted move is **proposal #3 (per-tool renderer registry)** — independently valuable AND pre-paves #1. Quick wins: **#2 (`bedrock-agentcore` bump)** and **#5 (close #266/#267)** demonstrate library-native subtraction. CI (proposal #6) is the loudest non-kaizen problem; surface it here but fix it as hygiene. + +--- + +## Review Protocol (for Phil) + +1. Read Friction (2 min). +2. Mark each Proposal ✅ Ship / ❌ Decline / ⏸ Defer (4-6 min). **10 proposals**; my recommendations: 9 Ship, 1 Defer, 0 Decline (proposal #7 — `oauth_required` audit — is the defer until 2026-05-24). +3. Same for Risks Acknowledged. +4. Pick 1-3 to ship this week. Suggested if you only do 3: **#1 (scope MCP Apps host — scoping doc only this week), #2 (bedrock-agentcore bump), #5 (close #266/#267)** — covers strategic, quick-win, and subtraction. If 4: add **#3 (renderer registry)** as the bridge investment. + +Target: 12-17 minutes (slightly more than the nominal 10-15 because the bootstrap is larger than a normal weekly review). + +## Post-review (separate PRs) + +- ✅ Ship items → individual feature PRs over the week. The decision is logged in this doc; the implementation lives elsewhere. +- ❌ Decline items → appended to `docs/kaizen/decisions.md` with reason so future research doesn't re-propose. +- ⏸ Defer items → kept open in `review-queue.md` with a "revisit by" date; surface again in the next review when due. + +This skill produces the agenda. Implementation never happens here. diff --git a/docs/kaizen/reviews/2026-05-15.md b/docs/kaizen/reviews/2026-05-15.md new file mode 100644 index 00000000..610bf82a --- /dev/null +++ b/docs/kaizen/reviews/2026-05-15.md @@ -0,0 +1,207 @@ +# Kaizen Review — Friday, May 15, 2026 +> Prepared 09:50am MT. Review window: May 10 – May 15, 2026 (5 days since the bootstrap review). +> Source: research/2026-05-15.md + review-queue.md (15 open items at run start). +> First *regular* review after the 2026-05-10 bootstrap. Two prior-week proposals already landed (#1 MCP Apps scoping → PR #296, #6 nightly-CI triage → PR #290). + +## Week in Review + +The week's defining move came not from external ecosystem noise but from a single internal decision: **#293 disabled Dependabot version-update PRs on May 13**, which silently promotes this kaizen loop from "nice-to-have radar" to "the only mechanism catching version-pin lag." The first cost of that promotion arrived the same week — `bedrock-agentcore` widened from 3 → 4 releases behind (1.6.4 → 1.9.1, latest May 12), Strands shipped v1.40 with a *breaking* default-flip on `use_native_token_count`, and four ref-repo and SDK signals pointed at silent-failure modes in our code (`/ping` reaping, A2A streaming capability, double-fired `tool_use` SSE, tool-catalog/executor RBAC drift). Externally the agentic-UI storyline that dominated bootstrap week went quiet, but the upstream-shrinks-our-backlog play kept paying off — Strands' proactive context compression and the AgentCore SDK `async_mode` PR #478 both directly intersect work we'd otherwise build. Net read: a "harvest the upstream gains, defuse the silent-failure modes, take responsibility for dependency drift" week. + +## Friction — the week's signal + +### Repeated patterns (≥2 occurrences) +- **Inference-API deploy doesn't roll new AgentCore Runtime versions** (issue #288, May 12; 3 Deploy Inference API failures in window) — new container images reach ECR but the live Runtime isn't picked up. + - *Hypothesis*: the deploy workflow pushes the image but does not call `update_agent_runtime` (or whatever the SDK equivalent is) to bump the Runtime version pointer. Manual redeploys have been the band-aid. + - *Candidate fix*: trace the deploy workflow against the AgentCore Runtime versioning model; verify the post-ECR-push step invokes the SDK's `update_agent_runtime`. Pair with the bedrock-agentcore 1.9.1 bump (the SDK we'd need to call against has moved 4 versions in that workflow's blind spot). +- **Version Check workflow failures clustered on Dependabot branches before #293 landed** (5 of 7 in window, May 11) — these are the failures that *prompted* the decision to disable Dependabot version-update PRs. + - *Hypothesis*: the Version Check workflow expects authored-by-human PRs to bump a project VERSION file; Dependabot PRs only bump pinned dependencies and tripped the check. + - *Candidate fix*: not actionable here — Phil's call to disable Dependabot version-updates (#293) was the decision; this review just inherits its consequence (load-bearing kaizen loop). + +### One-offs worth watching +- **Nightly Build & Test silent since May 8 (last week's #6 worked)** — PR #290 landed May 12 and the cluster has been quiet since. Confirmation that the loop catches and resolves CI hygiene. +- **BFF parade declared done with #297** (May 14) — `chore(auth): remove dead Bearer-only auth from app_api post-BFF migration` retires the parallel-auth path; beta.26 cut May 14 settles the BFF v1. + +### Silence that matters +- **Zero merged work on prior-review proposals #2, #3, #4, #5, #8, #9** in the 5-day window despite 9 "Ship" recommendations. Phil shipped the two highest-leverage ones (#1 scoping doc, #6 nightly-CI), then the rest of the week's PR throughput (#297–#301) was on admin-shell + UX polish + dead-code removal. *Read*: the loop produced more "Ship" recommendations than the week absorbed. Either accept that recommendation density is meant to give Phil optionality and re-surface the unshipped items here (this review's approach), or the next research run trims to top-3 Ship suggestions. Pick one — drifting between both reads is the failure mode. +- **AgentCore platform announcements** — zero new GA/preview items this week (last week had BYO filesystem, Memory metadata, Identity-on-ECS, Payments). The capability-unlock pipeline is shallow this week; lean on the carry-overs. + +## Proposals — ranked + +### 1. Bump `bedrock-agentcore` 1.6.4 → 1.9.1 (re-prioritized; lag widened 3 → 4) +- **Source**: research/2026-05-15.md ▸ Top 5 #1 | review-queue.md (open since 2026-05-15, supersedes 2026-05-10 entry) +- **Surface area**: backend (`backend/pyproject.toml`, `backend/uv.lock`) +- **Change**: pin update + smoke-test memory + identity flows in dev. Verify CHANGELOG between 1.6.4 → 1.9.1 (May 12) for breaking changes. Verify whether 1.9.1 already addresses #456 (OTEL trace detach across asyncio boundaries) — if so, close. Track PR #478 (`async_mode`) for the 1.10.0 follow-up. +- **Subtracts**: no — pure dep bump. Justified: 4 versions of upstream fixes; Dependabot version-updates are now off (#293), so this won't get there on its own; sets up adoption of PR #478 `async_mode` once 1.10.0 ships (which resolves last week's flagged #452). +- **Effort**: Low +- **Impact**: Medium-High +- **POC findings**: not POCed; recommended in bootstrap review but no PR opened. +- **Ship means**: open a PR updating `pyproject.toml` and `uv.lock`; smoke-test memory write/read + identity OAuth flows in dev; if smoke passes, merge. +- **Decline means**: lag widens to 5 next week; eventually a security patch or `async_mode` adoption forces a multi-version jump. +- **Recommendation**: **Ship.** Highest-priority cleanup of the week. Embarrassing on a third week of carry-over. Bundle with proposal #10 (deploy-rolls-runtime fix) — same surface area, the SDK methods are in the same package. + +### 2. Audit and fix `/ping` to emit `time_of_last_update` (AgentCore SDK issue #471) +- **Source**: research/2026-05-15.md ▸ Top 5 #2 | review-queue.md (open since 2026-05-15) — https://github.com/aws/bedrock-agentcore-sdk-python/issues/471 +- **Surface area**: backend (`backend/src/apis/inference_api/` `/ping` handler — one of the two routes the AgentCore Runtime data plane actually serves per CLAUDE.md inference-api boundary) +- **Change**: grep our ping response shape; if it's just `{"status": "Healthy" | "HealthyBusy"}`, extend to `{"status": ..., "time_of_last_update": }`. Without that field AgentCore's idle reaper can kill microVMs mid-long-generation even while we're streaming. +- **Subtracts**: no — defensive against silent microVM reaping on long generations. +- **Effort**: Low +- **Impact**: Medium-High (silent failure mode; long agent turns get killed mid-stream) +- **POC findings**: not POCed. +- **Ship means**: 15-minute PR — grep the handler, add the field, smoke-test against a long-running tool turn in dev. +- **Decline means**: keep accepting potential silent microVM reaping on long generations; revisit if a user reports a stuck-mid-stream conversation. +- **Recommendation**: **Ship.** Cheapest item in the list with a real silent-failure mode. This is exactly the kind of cheap-but-load-bearing fix the kaizen loop should be catching. + +### 3. Wire per-tool `duration_ms` into `tool_result` SSE (Claude Code 2.1.141 pattern) +- **Source**: research/2026-05-15.md ▸ Top 5 #5 | review-queue.md (open since 2026-05-15) +- **Surface area**: backend (new Strands `AfterToolCall` hook → SSE event emitter) + frontend (`` component: faint inline timing badge when `duration_ms > 250`) +- **Change**: register a Strands `AfterToolCall` hook capturing `(end - start)` wall-clock per tool invocation; emit on the existing `tool_result` SSE event as `duration_ms`. Frontend renders inline timing badge only above a noise threshold (default 250ms). +- **Subtracts**: partial — single hook-driven field replaces any ad-hoc per-tool timing. +- **Unlocks**: + - Per-tool timing visibility in the UI (which slow tool is the bottleneck on this turn?) + - Data substrate for the planned context-attribution prototype — separates tool latency from token cost +- **Effort**: Low-Medium +- **Impact**: Medium-High +- **POC findings**: not POCed. +- **Ship means**: one PR: backend hook + SSE field + frontend badge + a unit test that asserts the hook fires on tool completion. +- **Decline means**: tool-latency stays invisible; context-attribution prototype starts with murkier inputs. +- **Recommendation**: **Ship.** Pairs naturally with the planned context-attribution work — landing this first means the prototype starts clean. The "unlocks new UI surface" lens applies cleanly here. + +### 4. Defensive A2A AgentCard `capabilities={"streaming": True}` check +- **Source**: research/2026-05-15.md ▸ Top 5 #4 | review-queue.md (open since 2026-05-15) — ref-repo commit `50c9112` +- **Surface area**: backend (wherever we construct A2A AgentCards — search `AgentCard`, `capabilities=`, `agent_card`) +- **Change**: 30-second grep + read; if the field is missing or `False`, set to `True`. Silent failure mode otherwise: A2A SDK client falls back to non-streaming, never receives `completed`, hangs ~40 minutes. +- **Subtracts**: no — defensive; silent-failure mode of 40-min timeouts. +- **Effort**: Low +- **Impact**: Medium +- **POC findings**: not POCed. +- **Ship means**: 30-min grep + PR; if no A2A AgentCard exists in the repo today (single-agent baseline), file a note that it must be set on first A2A construct landed. +- **Decline means**: leave a silent-failure mode active in any future A2A AgentCard work. +- **Recommendation**: **Ship.** Cheapest defensive item this week. + +### 5. Promote tool-result rendering to a per-tool renderer registry (PR #0 of MCP Apps host sequence) +- **Source**: review-queue.md (open since 2026-05-10) | scoping doc `docs/kaizen/scoping/mcp-apps-host-renderer.md` ▸ PR #0 (pre-work) +- **Surface area**: frontend (`` / `tool-use.component.ts` + new `tool-renderer-registry.service.ts`) +- **Change**: lift the implicit tool-result switch into a signal-backed registry keyed by tool name. Default renderer is today's behavior. Migrate 2–3 existing renderers as proof points. +- **Subtracts**: partial — replaces implicit switch with declarative registry; absorbs scattered tool-specific UI logic into one table. Pre-paves MCP Apps PR #4 (the iframe renderer slots in as just-another-registered-renderer). +- **Effort**: Medium +- **Impact**: Medium-High (standalone UX value + scaffolding for the MCP Apps initiative) +- **POC findings**: not POCed — but Phil locked it in as PR #0 of the MCP Apps sequence on May 14 (#296 scoping doc). +- **Ship means**: open the PR #0 from the scoping doc — registry service + 2–3 migrated renderers, all existing tool rendering unchanged. +- **Decline means**: MCP Apps PR sequence is blocked; the implicit switch grows further. +- **Recommendation**: **Ship.** The scoping decision is already made; this is execution. Best risk-adjusted UX investment of the week. + +### 6. Strands 1.39 → 1.40 bump, gated on `use_native_token_count` audit + proactive-compression double-fire check +- **Source**: research/2026-05-15.md ▸ Top 5 #3 | review-queue.md (open since 2026-05-15) +- **Surface area**: backend (`backend/pyproject.toml`, `uv.lock`, `apis/shared/` token-metric reads, `agents/main_agent/streaming/`, `TurnBasedSessionManager`) +- **Change**: (a) audit `apis/shared/` and `agents/main_agent/streaming/` for native-token-count reads — if we depend on them, either pin `BedrockModel(use_native_token_count=True)` explicitly OR re-route through the heuristic and verify the cost-badge math (recall #270 just touched this); (b) bump pin; (c) verify proactive context compression (PR #2239) doesn't double-fire with our existing `TurnBasedSessionManager` flush — our `compaction` SSE event should still emit cleanly. (d) Smoke-test cost-badge values across a tool turn before promoting. +- **Subtracts**: **yes — library-native subtraction.** Strands' proactive context compression reduces the surface area of our custom session-manager compaction logic. +- **Effort**: Medium +- **Impact**: Medium-High +- **POC findings**: not POCed. +- **Ship means**: PR with the audit + pin bump + a regression test that asserts the `compaction` SSE event still emits exactly once per compaction event. +- **Decline means**: stay on 1.39 for another week; revisit when 1.41 ships or after a token-accounting issue surfaces. +- **Recommendation**: **Ship,** but second-priority of the bumps (after #1) — the audit is the careful part. + +### 7. Close issues #266 and #267 — features already in our Strands 1.39 pin +- **Source**: review-queue.md (open since 2026-05-10; carry-over from bootstrap review proposal #5) +- **Surface area**: cross-cutting — GitHub issues + minor wiring checks in `stream_coordinator` (#267) and large tool-result paths (#266) +- **Change**: (1) comment on #266 + #267 pointing at upstream PRs; (2) verify the upstream features are *automatically* active under our 1.39 pin — if not, file replacement issues for the wiring work and link them; (3) close #266 + #267. +- **Subtracts**: **yes — library-native subtraction.** Retires 2 build-from-scratch tickets; replaces with at-most 2 "wire upstream feature" tasks. +- **Effort**: Low +- **Impact**: Medium (closes phantom tech debt; clears the issue list) +- **POC findings**: not POCed — recommended Ship in the bootstrap review but didn't get to it in window. +- **Ship means**: 30-minute issue-grooming pass. +- **Decline means**: leave #266 + #267 open; future kaizen runs will re-flag them. +- **Recommendation**: **Ship.** Highest subtraction yield this week. Trivial. + +### 8. Audit `BedrockModel.stream` cancellation path against Strands #2266 +- **Source**: review-queue.md (open since 2026-05-10; carry-over from bootstrap review proposal #4) +- **Surface area**: backend (`backend/src/agents/main_agent/` stream coordinator + SSE handler) +- **Change**: locate every `BedrockModel.stream` cancellation path; ensure each `await`s the inner task on cancel so it doesn't orphan; add a dev-only assertion / log filter to detect "Task exception was never retrieved" before it reaches prod. +- **Subtracts**: no — defensive. +- **Effort**: Low +- **Impact**: Medium-High (SSE-disconnect path is hot) +- **POC findings**: not POCed. +- **Ship means**: PR with the audit + fixes + a regression test that triggers cancel + asserts no orphan tasks. +- **Decline means**: log a tech-debt issue; revisit if "Task exception was never retrieved" appears in CloudWatch. +- **Recommendation**: **Ship.** Pairs naturally with proposal #1 (same backend area). Cheap insurance. + +### 9. Replace dead source URLs in `kaizen-research` skill + AgentCore starter-toolkit slug typo +- **Source**: review-queue.md (open since 2026-05-10; carry-over from bootstrap review proposal #9) + research/2026-05-15.md ▸ Retirement candidates +- **Surface area**: skills (`.claude/skills/kaizen-research/SKILL.md`) +- **Change**: (a) replace `https://aws.amazon.com/bedrock/whats-new/` (404) with the AWS What's New RSS feed; (b) replace `https://docs.claude.com/en/docs/claude-code/release-notes` (301→404) with `https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md`; (c) drop `https://github.com/anthropics/courses` (quiet since Nov 2025); (d) fix `aws/amazon-bedrock-agentcore-starter-toolkit` → `aws/bedrock-agentcore-starter-toolkit` slug (new this week). +- **Subtracts**: **yes — replaces 2 broken URLs with working ones; drops 1 stale source; fixes 1 slug typo.** +- **Effort**: Low +- **Impact**: Low (skill quality; reduces web-budget waste) +- **POC findings**: not POCed. +- **Ship means**: 10-minute edit to `kaizen-research/SKILL.md`. +- **Decline means**: keep wasting web-budget on dead URLs; future runs continue to miss the starter-toolkit repo via the wrong slug. +- **Recommendation**: **Ship.** Trivial subtraction. + +### 10. Investigate inference-api deploy issue #288 — new images reach ECR but Runtime isn't rolled +- **Source**: research/2026-05-15.md ▸ Internal Audit ▸ Repeated friction (3 Deploy Inference API failures in window) | issue #288 (May 12, open) +- **Surface area**: cross-cutting — `.github/workflows/deploy-inference-api.yml` + the SDK call shape against the post-bump bedrock-agentcore 1.9.x API +- **Change**: trace the deploy workflow end-to-end; confirm whether it calls `update_agent_runtime` (or equivalent) after the ECR push; if missing, add it; if present but failing, surface the failure. Naturally pairs with proposal #1 since the SDK that owns this call is the package we're bumping. +- **Subtracts**: possibly — if the fix removes the manual-redeploy band-aid that's been the workaround. +- **Effort**: Low-Medium (worst case: an IAM/permission gap on the deploy role) +- **Impact**: Medium-High (eventually a security patch or model-version bump must ship via this path) +- **POC findings**: not POCed. +- **Ship means**: 1–2 hour triage; small PR fixing the workflow + closing #288 when verified. +- **Decline means**: continue running manual redeploys; eventually something time-critical needs to ship through this path. +- **Recommendation**: **Ship.** Pair with #1 for shared context on the bedrock-agentcore SDK surface area. + +## Carried Over From Prior Reviews + +- **`oauth_required` SSE flow audit against ref-repo's mid-tool-call 401/403 handling** (deferred 2026-05-10 until 2026-05-24) — original context: BFF auth was still settling. **Status this week**: BFF parade declared done with #297 (May 14) — the cleanup PR removed the parallel Bearer path entirely. Conditions for the original deferral have cleared a week early. *Recommendation*: keep deferred until 2026-05-24 per original commitment (one more stable week eliminates regression risk) — but surface here as an explicit hold rather than silently leaving on the queue. +- **Named A2A agent participants in the chat UI** (open since bootstrap, recommended Ship) — not shipped in window. *Recommendation*: defer 4 weeks (revisit 2026-06-12) — single-agent mode is still our baseline; this earns its keep when an A2A construct lands. No re-rank without a trigger. +- **Scope AgentCore Runtime BYO filesystem (S3 Files / EFS)** (open since bootstrap, no decision recorded) — high-effort high-impact capability unlock. *Recommendation*: defer 4 weeks (revisit 2026-06-12) — MCP Apps host renderer is the dominant strategic initiative; layering another VPC + IAM + storage-architecture push on top doubles the open ADR-worthy bets. + +## Retirement Candidates + +- **Queue item "Add Reddit `.rss` or Reddit MCP to `kaizen-research`"** — research/2026-05-15.md confirmed Reddit is blocked at the domain level via WebFetch, not just the HTML path. *Recommendation*: **Decline.** Move to `decisions.md` with reason "infeasible via WebFetch; revisit only if Reddit MCP or `curl`-via-Bash with UA header becomes available." +- **(Soft) Bootstrap-era queue entry "Bump `bedrock-agentcore` 1.6.4 → 1.9.0"** — superseded by re-prioritized 2026-05-15 entry (1.9.1 + lag widened). *Recommendation*: mark Resolved as "Superseded by 1.9.1 entry" when moving queue items. +- **(System-health check)** Three retirements landed across two weeks (kaizen-research lens churn, dead URLs queued for replacement, Reddit RSS declined this week). The subtraction muscle is exercising. No additional retirements needed *this* week. + +## Risks Acknowledged But Not Acted On + +- **Dependabot version-update PRs disabled (#293)** — https://github.com/Boise-State-Development/agentcore-public-stack/pull/293 — *what breaks if ignored*: version-pin lag silently widens; security patches in transitive deps don't arrive until something else surfaces them. The kaizen loop is the only mechanism left catching this. — *Recommendation*: **Address now via #1** (bedrock-agentcore bump validates the loop catches lag) + tighten the version-pin diff section of the research skill (direct fetches for `boto3`, `aws-cdk-lib`, `@angular/core`, `pydantic` every run — file as a kaizen-research skill update follow-up). +- **Strands v1.40 `use_native_token_count` default flip** — https://github.com/strands-agents/sdk-python/pull/2284 — *what breaks if ignored*: if we bump without auditing, token-accounting reading native counts gets heuristic values instead, and #270's per-message-cost / context-% math may quietly drift. — *Recommendation*: **Address now via #6** (the audit is the careful part of the bump). +- **AgentCore SDK PR #478 (`async_mode`) still in flight; #452 event-loop blocking unfixed in 1.9.1** — *what breaks if ignored*: under sustained load, AgentCoreMemorySessionManager can block the event loop. — *Recommendation*: **Watch until 1.10.0 ships** (likely lands the fix). Re-evaluate in the 2026-05-22 review. +- **Strands SDK monorepo consolidation announced (issue #2286, May 12)** — *what breaks if ignored*: import paths likely move in a future major; no near-term impact at 1.40. — *Recommendation*: **Watch until v2.x messaging emerges.** Re-evaluate at next minor (1.41). + +## What Shipped This Week + +- **#301 — feat(sidebar): denser session list with skeleton and entry animation** (May 15) — *UX polish* +- **#300 — feat(admin): persistent shell layout with grouped sidebar nav** (May 15) — *admin restructure* +- **#299 — feat(frontend): copy-to-clipboard button on chat code blocks** (May 14) — *UX* +- **#298 — feat(admin): admin-managed user-menu links** (May 14) — *new admin capability* +- **#297 — chore(auth): remove dead Bearer-only auth from app_api post-BFF migration** (May 14) — *BFF v1 cleanup; declares the BFF parade done* +- **#296 — chore(kaizen): add initial scoping document for MCP Apps Host Renderer** (May 14) — *kaizen proposal #1 from 2026-05-10 review; PR #0 → PR #6 sequence locked* +- **#293 — chore: restrict contributions and disable Dependabot version-update PRs** (May 13) — *the load-bearing decision of the week* +- **#290 — Fix e2e testing in nightly** (May 12) — *kaizen proposal #6 from 2026-05-10 review; nightly cluster silent since* +- **beta.26** (May 14) + **beta.25** (May 12) — *two release cuts in window* + +## Take + +The system is trending toward trust on the loop side (two prior-review proposals shipped, both high-leverage) and toward friction on the dependency-management side (lag widened the same week the only-mechanism guard came on). **The single change that matters most this week is proposal #1 (`bedrock-agentcore` bump)** — not because the bump itself is interesting, but because it's the proof that the kaizen loop can catch lag now that Dependabot can't. **The best risk-adjusted move this week is the bundle of proposals #2 + #4 + #7 + #9** — four under-30-minute items that collectively close 1 silent-failure mode, 1 latent-A2A-trap, 2 phantom GitHub issues, and 4 dead/typo URLs. Phil should ship all four in a single afternoon and feel the kaizen loop earn its keep. The slower-burn items (#3 per-tool duration_ms, #5 renderer registry, #6 Strands 1.40 audit, #8 stream cancel audit, #10 deploy/runtime fix) are the week's real engineering work — pick 1–2 of those to ride the week. **Do not** add new items to the queue this week — the queue is full, the carry-over count is healthy, and the loop is producing more recommendations than the week is absorbing; that imbalance is fine for now but worth watching. + +--- + +## Review Protocol (for Phil) + +1. Read Friction (2 min). +2. Mark each Proposal ✅ Ship / ❌ Decline / ⏸ Defer (4-6 min). **10 proposals**; my recommendations: 10 Ship, 0 Defer, 0 Decline. +3. Mark Carried Over: keep oauth audit deferred to 2026-05-24; defer A2A participants + BYO filesystem 4 weeks (1-2 min). +4. Confirm the Reddit RSS retirement → `decisions.md` (1 min). +5. Resolve Risks block. +6. **Suggested 1–3 to ship**: **#1 (bedrock-agentcore bump)**, **the bundle #2+#4+#7+#9** counted as one afternoon (4 cheap subtractions), and **#3 (per-tool `duration_ms`)** for the week's real UX investment. + +Target: 12-15 minutes. + +## Post-review (separate PRs) + +- ✅ Ship items → individual feature PRs over the week. The decision is logged here; the implementation lives elsewhere. +- ❌ Decline items → appended to `docs/kaizen/decisions.md` with reason so future research doesn't re-propose. +- ⏸ Defer items → kept open in `review-queue.md` with a "revisit by" date; surface again in the next review when due. + +This skill produces the agenda. Implementation never happens here. diff --git a/docs/kaizen/scoping/mcp-apps-budget-allocator.tool.json b/docs/kaizen/scoping/mcp-apps-budget-allocator.tool.json new file mode 100644 index 00000000..52640ac0 --- /dev/null +++ b/docs/kaizen/scoping/mcp-apps-budget-allocator.tool.json @@ -0,0 +1,16 @@ +{ + "_comment": "Example ToolCreateRequest body for registering the modelcontextprotocol/ext-apps `budget-allocator-server` as an MCP-Apps-capable external MCP tool. POST this to `/admin/tools/` on the App API (requires an admin session). The form-style App exercises ui/update-model-context + app-initiated tools/call without 3D/charting backend infra. Run the example server in HTTP mode first (see the runbook in .github/docs/deploy/step-04-deploy.md): `cd examples/budget-allocator-server && npm run start:http` → http://localhost:3001/mcp. Adjust serverUrl for a deployed server. authType `none` is only appropriate for an unauthenticated local/dev server — use aws-iam / api-key / oauth2 for anything real.", + "toolId": "budget_allocator", + "displayName": "Budget Allocator", + "description": "Interactive budget-allocation MCP App (modelcontextprotocol/ext-apps example) — sliders, presets, and benchmarks. Dogfood server for the MCP Apps host renderer.", + "category": "data", + "protocol": "mcp_external", + "status": "active", + "isPublic": false, + "enabledByDefault": false, + "mcpConfig": { + "serverUrl": "http://localhost:3001/mcp", + "transport": "streamable-http", + "authType": "none" + } +} diff --git a/docs/kaizen/scoping/mcp-apps-host-renderer.md b/docs/kaizen/scoping/mcp-apps-host-renderer.md new file mode 100644 index 00000000..66d583c0 --- /dev/null +++ b/docs/kaizen/scoping/mcp-apps-host-renderer.md @@ -0,0 +1,170 @@ +# Scoping — MCP Apps Host Renderer + +> Status: Scoping (no code yet) +> Owner: Phil Merrell +> Source: research/2026-05-10.md ▸ Top 6 #1 ▸ Agentic UI/UX | reviews/2026-05-10.md ▸ Proposal #1 (Ship — scope this week) | review-queue.md (open) +> Spec read: `specification/2026-01-26/apps.mdx` (normative). Pre-merge step: diff `specification/draft/apps.mdx` against the dated version to catch any movement before PR #1 lands. + +## Goal + +Implement the host side of the MCP Apps extension (SEP-1865) end-to-end and to spec, so that any MCP server we connect — Gateway-hosted or external — can return interactive UIs alongside text/JSON tool results, and so our chat sits on the agentic-UI standard that Claude Desktop, ChatGPT, VS Code Copilot, Goose, Postman, and MCPJam already meet. + +**Out of scope:** authoring MCP Apps (we are a host, not a server-of-apps), MCP-UI / `@mcp-ui/client` framework adoption (we implement the postMessage protocol directly), and any non-MCP-Apps "generative UI" pattern. + +## Architectural decisions (locked) + +These four were the open ones from scoping. Decisions, with rationale. + +### 1. Sandbox origin — new subdomain (Sandbox Proxy pattern) + +Stand up a dedicated origin for the outer "sandbox proxy" iframe so `allow-same-origin` does not give iframe content access to the main `ai.client` origin. Pattern matches Claude.ai's web-host implementation. + +- **Origin:** `mcp-sandbox.` (exact name TBD in PR #1 — see CDK work). +- **What it serves:** a single static `proxy.html` shell that itself creates the inner content iframe via `srcdoc` (the inner iframe is where the MCP App HTML actually runs). The outer page is what `ai.client` `postMessage`s to. +- **Why two iframes:** the spec's "Sandbox Proxy pattern" for web hosts — the inner iframe takes the strict CSP from `_meta.ui.csp`, the outer iframe gives us a stable cross-origin boundary against the host page. +- **CDK:** new stack `infrastructure/lib/mcp-sandbox-stack.ts` — CloudFront distribution, S3 bucket for `proxy.html`, ACM cert. Flowed through the `cors-deployment` skill for origin allowlisting. + +### 2. App-initiated `tools/call` — pipe through inference-api dispatch + +When the iframe calls `tools/call`, we surface it as a `tool_use` / `tool_result` event in the active conversation stream. Provenance is preserved — the chat history is a complete audit trail of what the embedded app ran on the user's behalf. + +- **Path:** iframe `postMessage` → frontend `mcp-app-frame` → app-api (new endpoint `POST /mcp-apps/proxy-call`) → inference-api → MCP server → reverse path. The inference-api side synthesizes a `tool_use` event into the conversation's SSE stream so it lands in the user's chat thread. +- **Conversation correlation:** the iframe is bound to the originating `toolUseId` and conversation session at render time; proxied calls inherit that binding. +- **Visibility enforcement:** the proxy endpoint MUST reject calls for tools whose `visibility` does not include `"app"` — at both the app-api boundary and the inference-api dispatch. + +### 3. `ui/update-model-context` storage — Strands `agent.state` + +App-supplied context (the structured/text payload from `ui/update-model-context`) lives in Strands `agent.state` under a dedicated key (e.g., `mcp_apps.context[resourceUri]`). This is where the upstream reference repo moved its compaction state on Apr 27 (commit `2b1a13d`) and it's where Strands is heading. + +- **Read path:** before each inference turn, merge any pending `agent.state.mcp_apps.context.*` entries into the prompt context, then clear them. +- **Spec semantics honored:** "host MAY defer context until next user message" and "host SHOULD only send last update if multiple arrive before next user message" — we dedupe by `resourceUri` and apply last-write-wins between turns. + +### 4. v1 method scope — full set, no deferrals + +Implement every `ui/` method the spec defines and every standard MCP method it permits inside the postMessage channel. Rationale: the user-facing payoff of MCP Apps is highest when the app can both *receive* context (host→app) and *push* it back (app→host) — half-implementing either side cuts off the workflows the spec exists to enable (`ui/message`, `ui/update-model-context`). One feature flag (`MCP_APPS_HOST_ENABLED`) gates the whole surface during rollout. + +## Spec compliance checklist + +Normative requirements from `apps.mdx` (2026-01-26). Items prefixed with `MUST` are spec-mandated; `SHOULD`/`MAY` items captured in the PR-level acceptance criteria below. + +- **MUST** fetch UI resources via `resources/read` against the `ui://` URI from `_meta.ui.resourceUri` — never inline. +- **MUST** treat `text/html;profile=mcp-app` as the resource MIME type. +- **MUST** advertise `capabilities.extensions["io.modelcontextprotocol/ui"]` with `{ mimeTypes: ["text/html;profile=mcp-app"] }` on every outbound MCP `initialize` (Gateway client + external MCP client). +- **MUST** filter tools whose `_meta.ui.visibility` excludes `"model"` from the agent's tool list (Strands tool registry filter). +- **MUST** reject `tools/call` proxied from the iframe for tools whose visibility excludes `"app"`. +- **MUST** set iframe `sandbox="allow-scripts allow-same-origin"` minimum; add `allow-camera` / `allow-microphone` / `allow-geolocation` / `allow-clipboard-write` only if the resource declares them in `_meta.ui.permissions`. +- **MUST** build CSP from `_meta.ui.csp.{connectDomains, resourceDomains, frameDomains, baseUriDomains}` and apply the spec's deny-by-default defaults. **MUST NOT** allow undeclared domains. +- **MUST** wait for `ui/notifications/initialized` from the app before sending any request or notification. +- **MUST** send `ui/notifications/tool-input` with the complete arguments exactly once (before `tool-result`). +- **MUST** send `ui/resource-teardown` before tearing the iframe down. +- **MUST** accept `event.origin === "null"` from the sandbox iframe and rely on a per-frame nonce instead of origin matching. +- **MUST** correlate JSON-RPC over postMessage using request `id` (standard JSON-RPC 2.0 envelope: `{jsonrpc, id, method, params}` / `{jsonrpc, id, result|error}`). + +## PR sequence + +Targets `develop`. Each PR is independently mergeable behind `MCP_APPS_HOST_ENABLED=false` until PR #6 flips it on. + +### PR #0 — Tool-renderer registry (pre-work; proposal #3 from reviews/2026-05-10.md) + +- **Files:** [tool-use.component.ts](frontend/ai.client/src/app/session/components/message-list/components/tool-use/tool-use.component.ts) + new `tool-renderer-registry.service.ts`. +- **Change:** lift the implicit tool-result switch in `ToolUseComponent` into a signal-backed registry keyed by tool name. Default renderer is today's behavior (text/JSON/image). Registry exposes `register(toolName, component)`. +- **Why first:** the MCP App renderer in PR #4 plugs in as just-another-registered-renderer — no special-case branches in `tool-use.component.html`. +- **Acceptance:** all existing tool renderings work unchanged; no MCP App code yet. + +### PR #1 — Sandbox-proxy origin (CDK) + +- **Files:** new `infrastructure/lib/mcp-sandbox-stack.ts`, updates to [`bin/agentcore-public-stack.ts`](infrastructure/bin/agentcore-public-stack.ts) and `cors-deployment` workflow env vars. +- **Change:** CloudFront + S3 + ACM for `mcp-sandbox.`; deploy a static `proxy.html` shell implementing the outer-iframe half of the Sandbox Proxy pattern. CSP `frame-ancestors` permits the `ai.client` origin only. +- **Acceptance:** `mcp-sandbox./proxy.html` serves; `ai.client` can `postMessage` to it; no MCP server wiring yet. +- **Coordinates with** the [cors-deployment skill](.) — every new env var that names this origin flows through that skill. + +### PR #2 — Backend: MCP `initialize` extension advertisement + tool-visibility filter + +- **Files:** [external_mcp_client.py](backend/src/agents/main_agent/integrations/external_mcp_client.py), [gateway_mcp_client.py](backend/src/agents/main_agent/integrations/gateway_mcp_client.py), [models.py](backend/src/apis/shared/tools/models.py) (add `visibility` to `ToolDefinition`), Strands tool registry adapter. +- **Change:** advertise `io.modelcontextprotocol/ui` on outbound MCP `initialize`; parse `_meta.ui` off `tools/list` responses onto `ToolDefinition`; filter model-invisible tools out of the Strands agent's tool list. +- **Acceptance:** unit tests covering a fake MCP server returning a UI-bearing tool — confirm visibility filtering, confirm `_meta.ui.resourceUri` survives the round-trip into our tool catalog. + +### PR #3 — Backend: SSE `ui_resource` event + `resources/read` fetch path + +- **Files:** [event_formatter.py](backend/src/agents/main_agent/streaming/event_formatter.py), [tool_result_processor.py](backend/src/agents/main_agent/streaming/tool_result_processor.py), [stream_processor.py](backend/src/agents/main_agent/streaming/stream_processor.py), and a new helper for `resources/read` against the MCP server hosting the tool. +- **Change:** when a tool result references `_meta.ui.resourceUri`, fetch the resource via `resources/read` and emit a new `ui_resource` SSE event: `{type, toolUseId, resourceUri, html, mimeType, csp, permissions}`. Update [CLAUDE.md](CLAUDE.md) SSE event table. +- **Acceptance:** integration test — fake MCP server returns `_meta.ui.resourceUri`; backend emits `ui_resource` event with HTML body inline. **Spec note:** we still call `resources/read` (spec MUST); we just inline the HTML in the SSE event so the frontend doesn't need its own MCP client. + +### PR #4 — Frontend: `` component + postMessage bridge + +- **Files:** new `mcp-app-frame.component.ts`, [stream-parser-types.ts](frontend/ai.client/src/app/shared/utils/stream-parser/stream-parser-types.ts), [stream-parser-core.ts](frontend/ai.client/src/app/shared/utils/stream-parser/stream-parser-core.ts), [stream-parser.service.ts](frontend/ai.client/src/app/session/services/chat/stream-parser.service.ts), wire-in via PR #0's renderer registry. +- **Change:** Angular component that: + - Renders the outer iframe pointed at `mcp-sandbox./proxy.html` with the spec-mandated `sandbox` attribute. + - Posts a `sandbox-resource-ready` notification to the proxy with `{html, sandbox, csp, permissions}` from the SSE `ui_resource` event. + - Implements the host half of the JSON-RPC 2.0 envelope over postMessage with a per-frame nonce. + - Handles `ui/initialize`, `ui/notifications/initialized`, `ui/notifications/size-changed`, `ui/open-link`, `ui/request-display-mode` (inline/fullscreen/pip), `ui/notifications/host-context-changed`, `ui/resource-teardown`. + - Wires `ui/notifications/tool-input` + `tool-input-partial` + `tool-result` + `tool-cancelled` from the active SSE stream. +- **Acceptance:** load the [basic-host](https://github.com/modelcontextprotocol/ext-apps/tree/main/examples/basic-host) reference's QR-server-style example against our component end-to-end in dev. + +### PR #5 — Backend + frontend: app-initiated `tools/call` proxying (decision #2) + +- **Files:** new `POST /mcp-apps/proxy-call` route in [app_api](backend/src/apis/app_api), tool-dispatch hook in inference-api to inject a synthesized `tool_use` into the active SSE stream, frontend wiring in `mcp-app-frame.component.ts`. +- **Change:** iframe `tools/call` → app-api → inference-api → MCP server → result → synthesized `tool_use`/`tool_result` events on the conversation SSE stream → frontend pushes `ui/notifications/tool-result` back to the iframe. +- **Acceptance:** clicking a button inside a hosted MCP App that triggers a server tool — the call shows up as a tool-use card in the chat *and* the iframe gets the result via `ui/notifications/tool-result`. +- **Open implementation question:** how to inject a synthesized event into a *closed* SSE stream (i.e., when the iframe lives past the originating turn). Likely a per-conversation event broker the active SSE handler subscribes to; if no handler is active, the call still runs but the chat thread shows it when the user next opens a stream. Detailed design lives in the PR. + +### PR #6 — Backend: `ui/message`, `ui/update-model-context`, `ui/open-link` consent, capability gating + +- **Files:** [oauth_consent.py](backend/src/apis/shared/oauth/oauth_consent.py) (pattern model), new `ui_capability_consent.py` hook, Strands `agent.state` integration for `mcp_apps.context.*`, conversation-message injection for `ui/message`. +- **Change:** + - `ui/update-model-context` writes to `agent.state.mcp_apps.context[resourceUri]`, merged into the next turn's context. + - `ui/message` injects a user-role message into the conversation (treated identically to a typed message). + - `ui/open-link` is gated by an `openLinks` capability declared in `hostCapabilities`; per-link consent reuses the [oauth-consent-prompt.component.ts](frontend/ai.client/src/app/session/components/message-list/components/oauth-consent-prompt/oauth-consent-prompt.component.ts) pattern (new `ui_consent_required` SSE event family). + - Camera / microphone / geolocation / clipboard-write capability gating wired through `hostCapabilities.sandbox.permissions`. +- **Acceptance:** an MCP App can mutate model context, post a user message, request to open a link, and request mic access — each triggers the correct host behavior (deferred merge, conversation message, consent prompt). + +### PR #7 — Dogfood + enable flag flip + +- **Files:** documentation, example MCP App registration, [CLAUDE.md](CLAUDE.md) update, feature flag default. +- **Change:** register one of the [ext-apps/examples](https://github.com/modelcontextprotocol/ext-apps/tree/main/examples) servers (recommended: `scenario-modeler-server` or `budget-allocator-server` — form-style, exercises `update-model-context` and `tools/call` proxying without 3D/charting infra). Flip `MCP_APPS_HOST_ENABLED=true`. Add a runbook entry to docs explaining how to register a new MCP App server. +- **Acceptance:** end-to-end conversation in dev that invokes the example tool, renders the iframe, drives the form, calls back into MCP, mutates model context, and the model picks up the context on the next turn. + +## Defaults applied without explicit user call + +These were small enough that I'm noting them here rather than putting them in the question set: + +- `ToolDefinition` gets a new `visibility: Literal["model", "app"]` list field (default `["model", "app"]` per spec). +- Outbound MCP clients advertise `io.modelcontextprotocol/ui` unconditionally — no per-server opt-in. Servers that don't understand the capability ignore it. +- Iframes persist for the lifetime of the conversation; teardown happens on conversation reset, on explicit user dismiss, or on tab close. No per-turn teardown. +- Default display mode is `inline`; fullscreen and PiP supported in PR #4. +- Per-frame nonce, generated client-side, used to authenticate every postMessage exchange (origin will be `"null"` in srcdoc inner iframes; nonce is the real check). +- Theming: `hostCapabilities.theme` exposes `light` | `dark` at initialize; `ui/notifications/host-context-changed` pushes updates when the user toggles theme. + +## Risks and unknowns + +- **CSP / `frame-ancestors` interplay.** The outer `mcp-sandbox` origin needs `frame-ancestors` permitting `ai.client`; the inner iframe needs CSP composed from `_meta.ui.csp`. We don't have prior art for nested CSP in our stack — expect 0.5–1 day of CSP debugging on PR #1. +- **`tools/call` proxy when the SSE stream is idle.** PR #5's "inject synthesized event into a closed SSE stream" needs a small event broker. If we punt it, app-initiated tool calls work but the chat thread misses them until the user opens a new turn. Acceptable for a v1; flag as known limitation if we ship without the broker. +- **Spec drift.** `specification/draft/apps.mdx` may have moved since 2026-01-26. Diff before PR #1 lands; if there's material movement, adjust PRs #2–#4 accordingly. +- **AgentCore Gateway pass-through of `_meta`.** Confirm `_meta.ui.resourceUri` survives Gateway's MCP proxying — if Gateway strips unknown `_meta` keys, PR #2 needs Gateway-side work too. Verify in PR #2's integration test. +- **Strands `agent.state` schema.** Our `TurnBasedSessionManager` doesn't currently round-trip `agent.state` through long-term memory. PR #6 may need a small adjacent change to ensure `mcp_apps.context.*` survives turn boundaries. + +## Definition of done + +- All seven PRs land on `develop` behind `MCP_APPS_HOST_ENABLED=false`; PR #7 flips it on. +- One example MCP App from `ext-apps/examples` runs end-to-end in dev. +- Every MUST in the compliance checklist has a corresponding test (unit or integration). +- The dogfood scenario in PR #7 exercises: resource fetch, iframe render, `tool-input` push, app-initiated `tools/call`, `ui/update-model-context` mutating the next turn, `ui/open-link` consent prompt. +- CLAUDE.md SSE event table updated with `ui_resource` and `ui_consent_required` rows. +- A runbook entry describes how to register a new MCP-Apps-capable MCP server (one section in the docs, no separate doc). + +## Timeline + +3–4 weeks across calendar, depending on review cadence: + +| PR | Effort | Notes | +|---|---|---| +| #0 renderer registry | 0.5d | low-risk refactor | +| #1 sandbox CDK | 1–1.5d | CDK + CORS skill + DNS + cert | +| #2 backend MCP capabilities | 1d | + Gateway pass-through verification | +| #3 backend SSE event | 1d | | +| #4 frontend iframe + bridge | 2–3d | postMessage protocol surface is wide | +| #5 tools/call proxying | 2d | + event broker for idle streams (or punt as known limit) | +| #6 message/context/consent | 2d | reuses oauth-consent pattern | +| #7 dogfood + flag flip | 0.5–1d | | + +Total: ~10–12 engineering days, sequenced; parallelization possible after PR #2 lands (frontend can race backend on #3–#4). diff --git a/docs/kaizen/scoping/mcp-sandbox-dynamic-csp.md b/docs/kaizen/scoping/mcp-sandbox-dynamic-csp.md new file mode 100644 index 00000000..5e67ff5d --- /dev/null +++ b/docs/kaizen/scoping/mcp-sandbox-dynamic-csp.md @@ -0,0 +1,167 @@ +# Scoping — MCP Sandbox Dynamic Per-Resource CSP + +> Status: Shipping — feature/mcp-sandbox-dynamic-csp +> Owner: Phil Merrell +> Source: dogfood gotcha #3 in [[project-mcp-apps-pr-progress]] (Option 3 of the host-renderer CSP fix); follow-up to #353 +> Spec read: draft `specification/draft/apps.mdx` lines 283–296; reference implementation `modelcontextprotocol/ext-apps/examples/basic-host/serve.ts` + +## TL;DR — **Ship** + +PR #353 shipped Options 1+2 (broad static outer CSP + `document.write()` mount). That works for the 22/25 reference servers that don't declare `_meta.ui.csp`, but **including our PR #7 dogfood App, Excalidraw**, three real Apps declare external domains the static CSP can't honor — they fail at runtime trying to fetch declared CDN scripts / tiles / fonts / soundfonts under our `connect-src 'self'`. Excalidraw's `create_view` is the canonical case: its server declares `resourceDomains: ['https://esm.sh']` + `connectDomains: ['https://esm.sh']` (see `excalidraw/excalidraw-mcp/src/server.ts`), and the dogfood console shows a wall of blocked esm.sh font / script / stylesheet loads. The spec's draft `apps.mdx` line 283 makes this a **host MUST**: "Host MUST construct CSP headers based on declared domains." We're not currently violating "MUST NOT allow undeclared domains" (we have no externals in our CSP at all), but we're failing the contract Apps rely on. Implementation: a CloudFront Function on viewer-response reading `?csp=` matching the upstream `examples/basic-host/serve.ts` `buildCspHeader` — ~50–100 LoC across `infrastructure/assets/mcp-sandbox/csp-function.js`, `mcp-sandbox-stack.ts`, frontend `proxy-url.ts`, plus tests. Cache stays simple (CFN runs on viewer-response including cache hits; one cached `proxy.html` body, dynamic header per request). + +## Apps that need it + +Empirical scan of `modelcontextprotocol/ext-apps/examples/*-server/server.ts` and the Excalidraw MCP server for `_meta.ui.csp` declarations. Four servers declare external domains: + +### Excalidraw `create_view` (our dogfood) + +```typescript +// excalidraw/excalidraw-mcp/src/server.ts +const cspMeta = { + ui: { + csp: { + resourceDomains: ['https://esm.sh'], + connectDomains: ['https://esm.sh'], + }, + }, +}; +``` + +The view's HTML pulls React 19, ReactDOM, Excalidraw 0.18, and the font/CSS bundle from `esm.sh`. On broad static CSP every one of those loads is blocked (`script-src` / `style-src` / `font-src` allow only `'self' blob: data:` — no `esm.sh`). The dogfood demo is visibly broken until this lands. + +### map-server (CesiumJS globe + OSM tiles) + +```typescript +const cspMeta = { + ui: { + csp: { + connectDomains: [ + "https://*.openstreetmap.org", // OSM tiles + Nominatim geocoding + "https://cesium.com", + "https://*.cesium.com", + ], + resourceDomains: [ + "https://*.openstreetmap.org", // OSM map tiles + "https://cesium.com", + "https://*.cesium.com", + ], + }, + }, +}; +``` + +Hard fail on broad static — Cesium needs the tile servers + CDN both for `connect` (XHR for tile bytes / geocoding) and `resource` (script-src for ion-loaded JS modules). Our `connect-src 'self'` blocks every tile request the moment the globe initialises. + +### pdf-server (PDF.js standard fonts) + +```typescript +csp: { + // pdf.js loads the Standard-14 fonts TWO ways: + // - fetch()s the .ttf bytes → connect-src + // - creates FontFace('name', 'url(...)') → font-src + // resourceDomains maps to font-src; we need both. + connectDomains: [STANDARD_FONT_ORIGIN], + resourceDomains: [STANDARD_FONT_ORIGIN], +}, +``` + +`STANDARD_FONT_ORIGIN` resolves to the pdf.js CDN host. PDF body renders but every glyph that requires a Standard-14 font (Helvetica, Times, Courier, Symbol, ZapfDingbats) falls back to a substitute or renders as a box — a visible quality regression, not a hard fail. + +### sheet-music-server (audio soundfonts) + +```typescript +csp: { + // Allow loading soundfonts for audio playback + connectDomains: ["https://paulrosen.github.io"], +}, +``` + +Visual sheet-music rendering works on broad static (abcjs is bundled). Only the "play audio" button silently fails — soundfont fetches hit `connect-src 'self'` block. + +## Apps that don't need it + +22 of 25 reference servers declare no `_meta.ui.csp` at all. These work today on our broad static CSP because they: + +- Bundle everything (no external CDN fetches). +- Use only same-origin postMessage to the host (no external network). +- Use only `permissions` (mic/camera/clipboard) without external resource needs — covered by our `_meta.ui.permissions` plumbing, not CSP. + +Concrete list: `basic-server-*` (preact/react/solid/svelte/vanillajs/vue), budget-allocator-server, scenario-modeler-server, cohort-heatmap-server, customer-segmentation-server, integration-server, transcript-server, debug-server, qr-server, say-server, shadertoy-server, system-monitor-server, threejs-server, video-resource-server, wiki-explorer-server. **All five of the "rich UI" candidates the scoping doc considered for PR #7 dogfood** (budget-allocator, scenario-modeler, threejs, shadertoy, transcript) are in this set — none of them are blocked. + +Note: shadertoy / threejs being in this set is non-obvious — they're WebGL-heavy and you'd expect external asset CDNs, but in the reference repo they ship fully bundled. + +## Cost vs. benefit + +### Security gain — small, bordering on theatre + +The threat model: "untrusted App HTML escapes its CSP and exfiltrates / phishes from the user." Our current static CSP has: + +- `connect-src 'self'` — App cannot make any external network request from inside the iframe. +- `frame-src 'none'` — App cannot frame anything else. +- `base-uri 'none'`, `form-action 'none'`, `object-src 'none'` — no base / form / plugin injection. + +The remaining attack surface is `'unsafe-inline' 'unsafe-eval' blob: data:` on scripts/styles. But: + +1. The inner App iframe is **already cross-origin sandboxed** to the SPA (null origin under `sandbox` attribute). Even if an attacker fully owns the App's JS execution, they can't reach SPA cookies, localStorage, or DOM. +2. The outer `proxy.html` ships **zero inline content** — every byte that runs is `proxy.js` loaded from same-origin (the dedicated mcp-sandbox CloudFront). `'unsafe-inline'`/`'unsafe-eval'` on the outer document can't be exploited unless an attacker can already inject into a static CloudFront asset, which is a much bigger compromise. +3. Going dynamic would *narrow* `connect-src` and `script-src` to per-App declared domains. But for the 22/25 Apps without declared CSP, we'd use the spec's restrictive default (`connect-src 'none'`, `script-src 'self' 'unsafe-inline'` — *no* `'unsafe-eval' blob:`), which would **break** many of the bundled-but-eval-needing Apps we currently render fine. The reference implementation acknowledges this by baking `'unsafe-eval' blob: data:` into its default too. + +So dynamic CSP buys us: a tighter `connect-src` for the 3 Apps that actually declare it. That's a marginal defense-in-depth gain stacked behind the existing cross-origin sandbox boundary. + +### Spec-compliance gain — real but not violated today + +Draft `apps.mdx` line 283: **"Host MUST construct CSP headers based on declared domains."** +Line 295: "No Loosening: Host MAY further restrict but MUST NOT allow undeclared domains." + +We don't violate "MUST NOT allow undeclared domains" — we have no external domains in our CSP at all. We *do* violate "MUST construct CSP headers based on declared domains" in the sense that we ignore declared `connectDomains`/`resourceDomains`. The user-visible consequence is that map-server / pdf-server / sheet-music-server can't fully function on us — they DECLARED what they need, we DIDN'T honor the declaration, the App fails. That's not "leaky security," it's "host doesn't implement the contract the App relied on." + +If someone is grading us on spec compliance (an external review, an audit, an MCP showcase), this gap is visible. If we're shipping internally, no one notices until we onboard a CSP-declaring App. + +### Implementation options compared + +| Option | Code | Deploy time | Runtime cost | Cache impact | On-call | +|---|---|---|---|---|---| +| **A. CloudFront Function (viewer-request → -response)** | ~30 LoC JS, no async, sanitize `?csp=`, emit header | Standard CFN deploy (~5 min) | $0.10/M invocations — pennies | proxy.html cache key adds `?csp`; hit rate drops to ~0 but origin is S3 (fast). proxy.js unaffected | Low — sync function, no cold start, no env vars | +| **B. Lambda@Edge (viewer-response)** | ~50 LoC Node, full SDK, easier to test | Slower deploy (~10 min replication) | $0.60/M + duration; <$1/month at our traffic | same as A | Medium — Lambda@Edge logs land in *viewer* region CloudWatch, harder to follow; rollback is slower | +| **C. Replace CloudFront+S3 with API Gateway + Lambda** | ~150 LoC + CDK rewrite | New stack | Higher | Lose CloudFront edge cache for proxy.js too | High — bigger surface | +| **D. Origin Lambda behind CloudFront** | Lambda + CFN integration | Standard | Higher than A/B | proxy.js still cacheable; proxy.html per-request | Medium | + +Plus, for any option, frontend side: `mcp-app-frame.component.ts` already has `csp` from the `ui_resource` SSE event — it would build `${proxyOrigin}/proxy.html?csp=${encodeURIComponent(JSON.stringify(csp))}` before assigning `iframe.src`. ~10 LoC change, no new SSE event. + +**Recommended option if/when we ship: A (CloudFront Function).** It fits the constraint set (sync, no I/O, sanitize + concat into a header), is cheaper and lower-latency than Lambda@Edge, and has the simpler operational story. The sanitizer from `serve.ts` (`/[;\r\n'" ]/.test(d)` reject) is straightforwardly portable to the CFN JS runtime. + +The `ResponseHeadersPolicy` in `infrastructure/lib/mcp-sandbox-stack.ts` would need to drop its static `Content-Security-Policy` (the dynamic header would conflict with the policy's "override: true" semantics). Other security headers (HSTS, Referrer-Policy, X-Content-Type-Options) stay in the policy. `frame-ancestors` becomes part of the dynamic CSP since it's the security-critical bit — though it could also stay in a separate static `Content-Security-Policy` header alongside the dynamic one (CSPs combine via intersection). + +### Cache implications + +Today: `CacheQueryStringBehavior.none()` — every request to `proxy.html` returns the same cached body. Switch to `CacheQueryStringBehavior.allowList(['csp'])` and each unique `?csp=` value becomes a separate cache entry. With ~hundreds of distinct Apps in any deployed env, hit rate on `proxy.html` drops from ~100% to ~0%. proxy.html is ~2 KB, S3 origin response is sub-10ms — the cost is invisible at our traffic. `proxy.js` cache is untouched (no query param on its fetch). + +One real concern: cache *explosion* if Apps generate per-call unique `?csp=` query strings (e.g. dynamic per-conversation CSP). The 25 reference Apps all use static `_meta.ui.csp` at resource-declaration time, so in practice the cardinality is bounded by the number of distinct UI resources, not the number of conversations. + +## Trigger — what would change the recommendation + +**Ship if any of these happen:** + +1. We onboard map-server, pdf-server, or sheet-music-server (or any App declaring non-`'self'` `connectDomains` / `resourceDomains` / `frameDomains` / `baseUriDomains`). The CSP work goes in *that* PR — same author, fresh context, no need to reload prior state. **Most likely trigger: CesiumJS map-server when we want a "wow" demo.** +2. The spec MUST tightens further (e.g. "Host MUST reject UI resources that declare CSP the host doesn't honor"). Skim the draft on each kaizen-research pass — currently line 283 is the relevant MUST; nothing has been added that makes it a *rejection* requirement yet. +3. An external review / showcase / partner asks for SEP-1865 compliance attestation. The "declared domains not honored" gap is visible to anyone who reads the spec. +4. We onboard an App that needs nested iframes (`frameDomains`) — our static `frame-src 'none'` blocks all nested framing absolutely. Reference Apps that fit this profile: none today, but anything embedding YouTube / a Tableau viz / a third-party widget would need it. + +**Don't ship for:** + +- "Defense-in-depth feels nice." The cross-origin sandbox is the real boundary. CSP tightening is icing. +- "The reference does it, so should we." The reference is a demo host; we're a product. Match capability when we have a user-facing reason. +- "It's in the scoping doc as a risk." The original scoping doc (`docs/kaizen/scoping/mcp-apps-host-renderer.md`) called out the CSP/`frame-ancestors` interplay as a 0.5–1d debug; we paid that debt. The dynamic-per-resource piece was always a follow-up. + +## Files that would change if we ship + +For reference (not implementation): + +- `infrastructure/lib/mcp-sandbox-stack.ts` — new CloudFront Function resource, drop static CSP from `ResponseHeadersPolicy`, update `CachePolicy` to include `?csp` in cache key on the `proxy.html` path behavior. +- `infrastructure/lib/mcp-sandbox-function.js` (new) — the CFN handler: read `?csp=`, parse JSON, sanitize domains, build CSP string (mirror `buildCspHeader` from `examples/basic-host/serve.ts`), set response header. +- `infrastructure/test/mcp-sandbox-stack.test.ts` — unit tests for sanitization (the `/[;\r\n'" ]/` reject rule is security-critical — every CSP-injection attack hides in domain entries with embedded `'`/`;`/space). +- `frontend/ai.client/src/app/.../mcp-app-frame.component.ts` — build `?csp=` query before setting `iframe.src`; the bridge already receives `csp` on the `ui_resource` event. +- `frontend/ai.client/src/app/.../mcp-app-frame.component.spec.ts` — unit test the query-string encoding. +- No backend changes (the `ui_resource` SSE event already carries `csp`). + +Total: 1 new file (CFN handler), edits to 4 files, ~80–100 LoC + tests. 1–2 days, mostly testing the cache invalidation + redeploy behavior end-to-end. diff --git a/frontend/ai.client/angular.json b/frontend/ai.client/angular.json index fe26333a..e7490d83 100644 --- a/frontend/ai.client/angular.json +++ b/frontend/ai.client/angular.json @@ -41,7 +41,13 @@ "scripts": [ "node_modules/prismjs/prism.js", "node_modules/prismjs/components/prism-csharp.min.js", + "node_modules/prismjs/components/prism-javascript.min.js", + "node_modules/prismjs/components/prism-typescript.min.js", + "node_modules/prismjs/components/prism-python.min.js", + "node_modules/prismjs/components/prism-sql.min.js", "node_modules/prismjs/components/prism-css.min.js", + "node_modules/prismjs/components/prism-json.min.js", + "node_modules/prismjs/components/prism-markdown.min.js", "node_modules/mermaid/dist/mermaid.min.js", "node_modules/katex/dist/katex.min.js", "node_modules/katex/dist/contrib/auto-render.min.js", diff --git a/frontend/ai.client/e2e/auth-admin.setup.ts b/frontend/ai.client/e2e/auth-admin.setup.ts index 4d8c48c5..95a4f371 100644 --- a/frontend/ai.client/e2e/auth-admin.setup.ts +++ b/frontend/ai.client/e2e/auth-admin.setup.ts @@ -17,6 +17,8 @@ async function cognitoLogin( ) { await page.goto('/auth/login'); await page.getByRole('button', { name: 'Sign in with Cognito' }).click(); + + // Wait for Cognito managed login page await page.getByRole('textbox', { name: 'Username' }).waitFor({ timeout: 15_000 }); await page.getByRole('textbox', { name: 'Username' }).fill(username); await page.getByRole('textbox', { name: 'Password' }).fill(password); @@ -31,7 +33,36 @@ async function cognitoLogin( ); } - await page.waitForURL('**/', { timeout: 30_000 }); + // Wait for the browser to leave Cognito and return to our app. + // After Cognito submit, the redirect chain is: + // Cognito → /api/auth/callback → BFF token exchange → 302 to / + // If the BFF callback fails, it redirects to /?auth_error=... or /auth/login + // If cookies land on the wrong domain (ALB instead of CloudFront), the + // APP_INITIALIZER gets 401 and redirects back to /auth/login. + + // Track the callback to diagnose cookie-domain issues + let callbackResponseUrl = ''; + page.on('response', async (response) => { + if (response.url().includes('/auth/callback')) { + callbackResponseUrl = response.url(); + } + }); + + try { + await page.waitForURL('**/', { timeout: 45_000 }); + } catch { + const finalUrl = page.url(); + const cookies = await page.context().cookies(); + const bffCookies = cookies.filter(c => c.name.startsWith('__Host-bff')); + const cookieDetails = bffCookies.map(c => `${c.name}(domain=${c.domain},path=${c.path},secure=${c.secure})`).join('; '); + throw new Error( + `OAuth redirect chain failed. Final URL: ${finalUrl} | ` + + `Callback response URL: ${callbackResponseUrl || 'NEVER HIT'} | ` + + `BFF cookies: ${cookieDetails || 'NONE'} | ` + + `All cookie domains: ${[...new Set(cookies.map(c => c.domain))].join(', ')}`, + ); + } + await expect(page.locator('textarea#user-message')).toBeVisible({ timeout: 10_000 }); await page.context().storageState({ path: storageStatePath }); } diff --git a/frontend/ai.client/e2e/auth-user.setup.ts b/frontend/ai.client/e2e/auth-user.setup.ts index 584d6c6f..d9cfb0f7 100644 --- a/frontend/ai.client/e2e/auth-user.setup.ts +++ b/frontend/ai.client/e2e/auth-user.setup.ts @@ -17,6 +17,8 @@ async function cognitoLogin( ) { await page.goto('/auth/login'); await page.getByRole('button', { name: 'Sign in with Cognito' }).click(); + + // Wait for Cognito managed login page await page.getByRole('textbox', { name: 'Username' }).waitFor({ timeout: 15_000 }); await page.getByRole('textbox', { name: 'Username' }).fill(username); await page.getByRole('textbox', { name: 'Password' }).fill(password); @@ -31,7 +33,51 @@ async function cognitoLogin( ); } - await page.waitForURL('**/', { timeout: 30_000 }); + // Wait for the browser to leave Cognito and return to our app. + // After Cognito submit, the redirect chain is: + // Cognito → /api/auth/callback → BFF token exchange → 302 to / + // If the BFF callback fails, it redirects to /?auth_error=... or /auth/login + + // Intercept the /auth/session request to see what's happening + let sessionResponseStatus = 0; + let sessionResponseBody = ''; + let sessionRequestCookies = ''; + // Track the callback redirect to diagnose cookie-domain issues + let callbackResponseUrl = ''; + let callbackSetCookies: string[] = []; + page.on('response', async (response) => { + if (response.url().includes('/auth/session')) { + sessionResponseStatus = response.status(); + sessionRequestCookies = response.request().headers()['cookie'] || 'NO COOKIE HEADER'; + try { sessionResponseBody = await response.text(); } catch { sessionResponseBody = ''; } + } + // Capture the callback response to see where cookies are being set + if (response.url().includes('/auth/callback')) { + callbackResponseUrl = response.url(); + const headers = response.headers(); + // Collect all set-cookie headers (may be multiple) + const setCookie = headers['set-cookie'] || ''; + if (setCookie) callbackSetCookies.push(setCookie); + } + }); + + try { + await page.waitForURL('**/', { timeout: 45_000 }); + } catch { + const finalUrl = page.url(); + const cookies = await page.context().cookies(); + const bffCookies = cookies.filter(c => c.name.startsWith('__Host-bff')); + const cookieDetails = bffCookies.map(c => `${c.name}(domain=${c.domain},path=${c.path},secure=${c.secure})`).join('; '); + throw new Error( + `OAuth redirect chain failed. Final URL: ${finalUrl} | ` + + `Callback response URL: ${callbackResponseUrl || 'NEVER HIT'} | ` + + `Session response: ${sessionResponseStatus} ${sessionResponseBody.substring(0, 100)} | ` + + `Cookie header sent: ${sessionRequestCookies.substring(0, 150)} | ` + + `BFF cookies in jar: ${cookieDetails || 'NONE'} | ` + + `All cookie domains: ${[...new Set(cookies.map(c => c.domain))].join(', ')}`, + ); + } + await expect(page.locator('textarea#user-message')).toBeVisible({ timeout: 10_000 }); await page.context().storageState({ path: storageStatePath }); } diff --git a/frontend/ai.client/e2e/home-page/chat.user.spec.ts b/frontend/ai.client/e2e/home-page/chat.user.spec.ts index 29e3b765..70f94466 100644 --- a/frontend/ai.client/e2e/home-page/chat.user.spec.ts +++ b/frontend/ai.client/e2e/home-page/chat.user.spec.ts @@ -31,8 +31,8 @@ async function sendMessageAndWaitForResponse( await page.getByRole('button', { name: 'Submit message' }).click(); const assistantMessage = page.locator('app-assistant-message').last(); - await expect(assistantMessage).toBeVisible({ timeout: 150_000 }); - await expect(page.locator('app-pulsating-loader')).toBeHidden({ timeout: 250_000 }); + await expect(assistantMessage).toBeVisible({ timeout: 300_000 }); + await expect(page.locator('app-pulsating-loader')).toBeHidden({ timeout: 300_000 }); return (await assistantMessage.innerText()).trim(); } @@ -41,6 +41,7 @@ async function sendMessageAndWaitForResponse( test.describe('Chat (user)', () => { test.describe.serial('Chat lifecycle with Claude Haiku 4.5', () => { test('should select Haiku, send a message, and receive a response', async ({ page }) => { + test.setTimeout(60_000); // 60s for this test await page.goto('/'); await expect(page.locator('textarea#user-message')).toBeVisible({ timeout: 15_000 }); @@ -58,6 +59,7 @@ test.describe('Chat (user)', () => { }); test('should send a second message in the same session', async ({ page }) => { + test.setTimeout(60_000); // 60s for this test await page.goto('/'); await expect(page.locator('textarea#user-message')).toBeVisible({ timeout: 15_000 }); diff --git a/frontend/ai.client/e2e/manage-sessions.user.spec.ts b/frontend/ai.client/e2e/manage-sessions.user.spec.ts index 9f25bc01..3d9d20b4 100644 --- a/frontend/ai.client/e2e/manage-sessions.user.spec.ts +++ b/frontend/ai.client/e2e/manage-sessions.user.spec.ts @@ -64,12 +64,18 @@ test.describe('Manage Sessions Page (user)', () => { test('should show the delete selected button disabled when nothing is selected', async ({ page }) => { await page.goto('/manage-sessions'); - await expect(page.getByText('Loading conversations...')).toBeHidden({ timeout: 15_000 }); - const deleteButton = page.getByRole('button', { name: /Delete Selected/i }); - const hasButton = (await deleteButton.count()) > 0; - test.skip(!hasButton, 'No sessions available — delete button not rendered'); + // Wait for the page to fully render before checking anything + await expect( + page.getByRole('heading', { name: 'Manage Conversations' }), + ).toBeVisible({ timeout: 15_000 }); + + // Wait for loading to finish + await expect(page.getByText('Loading conversations...')).toBeHidden({ timeout: 30_000 }); + // The Delete Selected button is always rendered (not conditional on sessions existing) + const deleteButton = page.getByRole('button', { name: /Delete Selected/i }); + await expect(deleteButton).toBeVisible({ timeout: 5_000 }); await expect(deleteButton).toBeDisabled(); }); }); diff --git a/frontend/ai.client/package-lock.json b/frontend/ai.client/package-lock.json index fb94ae19..fdff2c3f 100644 --- a/frontend/ai.client/package-lock.json +++ b/frontend/ai.client/package-lock.json @@ -1,12 +1,12 @@ { "name": "ai.client", - "version": "1.0.0-beta.24", + "version": "1.0.0-beta.28", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ai.client", - "version": "1.0.0-beta.24", + "version": "1.0.0-beta.28", "dependencies": { "@angular/cdk": "21.2.9", "@angular/common": "21.2.11", diff --git a/frontend/ai.client/package.json b/frontend/ai.client/package.json index 27283c25..a48c2245 100644 --- a/frontend/ai.client/package.json +++ b/frontend/ai.client/package.json @@ -1,6 +1,6 @@ { "name": "ai.client", - "version": "1.0.0-beta.24", + "version": "1.0.0-beta.28", "scripts": { "ng": "ng", "start": "ng serve", diff --git a/frontend/ai.client/playwright.ci.config.ts b/frontend/ai.client/playwright.ci.config.ts index e38dce1a..083f3989 100644 --- a/frontend/ai.client/playwright.ci.config.ts +++ b/frontend/ai.client/playwright.ci.config.ts @@ -41,10 +41,12 @@ export default defineConfig({ { name: 'admin-setup', testMatch: /auth-admin\.setup\.ts/, + timeout: 60_000, }, { name: 'user-setup', testMatch: /auth-user\.setup\.ts/, + timeout: 60_000, }, // --- Unauthenticated tests (no login needed) --- diff --git a/frontend/ai.client/src/app/admin/admin.layout.ts b/frontend/ai.client/src/app/admin/admin.layout.ts new file mode 100644 index 00000000..0920bcef --- /dev/null +++ b/frontend/ai.client/src/app/admin/admin.layout.ts @@ -0,0 +1,174 @@ +import { + Component, + ChangeDetectionStrategy, + inject, +} from '@angular/core'; +import { Router, RouterLink, RouterLinkActive, RouterOutlet } from '@angular/router'; +import { NgIcon, provideIcons } from '@ng-icons/core'; +import { + heroArrowLeft, + heroShieldCheck, + heroCurrencyDollar, + heroScale, + heroAcademicCap, + heroPencilSquare, + heroWrenchScrewdriver, + heroLink, + heroUsers, + heroKey, + heroFingerPrint, + heroBars3, +} from '@ng-icons/heroicons/outline'; + +interface NavItem { + label: string; + icon: string; + route: string; +} + +interface NavGroup { + label: string; + items: NavItem[]; +} + +@Component({ + selector: 'app-admin-layout', + changeDetection: ChangeDetectionStrategy.OnPush, + imports: [RouterLink, RouterLinkActive, RouterOutlet, NgIcon], + providers: [ + provideIcons({ + heroArrowLeft, + heroShieldCheck, + heroCurrencyDollar, + heroScale, + heroAcademicCap, + heroPencilSquare, + heroWrenchScrewdriver, + heroLink, + heroUsers, + heroKey, + heroFingerPrint, + heroBars3, + }), + ], + host: { class: 'block' }, + template: ` +
+ +
+
+ + + + +
+
+ +

Admin

+
+
+
+ +
+
+ + + + +
+ +
+
+
+
+ `, +}) +export class AdminLayout { + private router = inject(Router); + + readonly navGroups: NavGroup[] = [ + { + label: 'Usage & Spend', + items: [ + { label: 'Cost Analytics', icon: 'heroCurrencyDollar', route: '/admin/costs' }, + { label: 'Quotas', icon: 'heroScale', route: '/admin/quota' }, + { label: 'Fine-Tuning', icon: 'heroAcademicCap', route: '/admin/fine-tuning' }, + ], + }, + { + label: 'AI Configuration', + items: [ + { label: 'Models', icon: 'heroPencilSquare', route: '/admin/manage-models' }, + { label: 'Tools', icon: 'heroWrenchScrewdriver', route: '/admin/tools' }, + { label: 'Connectors', icon: 'heroLink', route: '/admin/connectors' }, + ], + }, + { + label: 'Identity & Access', + items: [ + { label: 'Users', icon: 'heroUsers', route: '/admin/users' }, + { label: 'Roles', icon: 'heroKey', route: '/admin/roles' }, + { label: 'Auth Providers', icon: 'heroFingerPrint', route: '/admin/auth-providers' }, + ], + }, + { + label: 'Customization', + items: [ + { label: 'User Menu Links', icon: 'heroBars3', route: '/admin/manage-user-menu-links' }, + ], + }, + ]; + + onMobileNavChange(event: Event): void { + const select = event.target as HTMLSelectElement; + this.router.navigateByUrl(select.value); + } +} diff --git a/frontend/ai.client/src/app/admin/admin.page.css b/frontend/ai.client/src/app/admin/admin.page.css deleted file mode 100644 index 6d1fe4e2..00000000 --- a/frontend/ai.client/src/app/admin/admin.page.css +++ /dev/null @@ -1 +0,0 @@ -/* Admin landing page styles */ diff --git a/frontend/ai.client/src/app/admin/admin.page.html b/frontend/ai.client/src/app/admin/admin.page.html deleted file mode 100644 index 517a0bcf..00000000 --- a/frontend/ai.client/src/app/admin/admin.page.html +++ /dev/null @@ -1,79 +0,0 @@ -
-
- -
-

Admin Dashboard

-

- Manage AI models, quotas, and system configuration -

-
- - -
- @for (feature of features; track feature.route; let i = $index) { - - -
-
- -
-
- - -
-

- {{ feature.title }} -

-

- {{ feature.description }} -

-
- - -
- Open - - - -
- - } -
- - -
-

About Admin Features

-
-

- Model Management: Configure which AI models are available to users. Control access by role and set pricing information. -

-

- Quota Management: Comprehensive quota system with tiered limits, role-based assignments, email domain matching, temporary overrides, and detailed monitoring. -

-
-
- - - -
-
diff --git a/frontend/ai.client/src/app/admin/admin.page.ts b/frontend/ai.client/src/app/admin/admin.page.ts deleted file mode 100644 index c5b4166b..00000000 --- a/frontend/ai.client/src/app/admin/admin.page.ts +++ /dev/null @@ -1,192 +0,0 @@ -import { Component, ChangeDetectionStrategy } from '@angular/core'; -import { RouterLink } from '@angular/router'; -import { NgIcon, provideIcons } from '@ng-icons/core'; -import { - heroCpuChip, - heroPencilSquare, - heroScale, - heroChartBar, - heroClipboardDocumentList, - heroMagnifyingGlass, - heroCalendar, - heroSparkles, - heroCurrencyDollar, - heroUsers, - heroShieldCheck, - heroWrenchScrewdriver, - heroLink, - heroFingerPrint, - heroAcademicCap, -} from '@ng-icons/heroicons/outline'; - -interface AdminFeature { - title: string; - description: string; - icon: string; - route: string; -} - -@Component({ - selector: 'app-admin-page', - imports: [RouterLink, NgIcon], - providers: [ - provideIcons({ - heroCpuChip, - heroPencilSquare, - heroScale, - heroChartBar, - heroClipboardDocumentList, - heroMagnifyingGlass, - heroCalendar, - heroSparkles, - heroCurrencyDollar, - heroUsers, - heroShieldCheck, - heroWrenchScrewdriver, - heroLink, - heroFingerPrint, - heroAcademicCap, - }) - ], - templateUrl: './admin.page.html', - styleUrl: './admin.page.css', - changeDetection: ChangeDetectionStrategy.OnPush, -}) -export class AdminPage { - readonly features: AdminFeature[] = [ - { - title: 'Cost Analytics', - description: 'View system-wide usage metrics, top users by cost, model breakdowns, and cost trends. Export reports for analysis.', - icon: 'heroCurrencyDollar', - route: '/admin/costs', - }, - { - title: 'Manage Models', - description: 'Configure and manage AI models available to users. Control model access by role, set pricing, and enable/disable models.', - icon: 'heroPencilSquare', - route: '/admin/manage-models', - }, - { - title: 'Tool Catalog', - description: 'Manage the tool catalog, configure role-based access, and sync tools from the registry. Control which tools are available to users.', - icon: 'heroWrenchScrewdriver', - route: '/admin/tools', - }, - // { - // title: 'Bedrock Models', - // description: 'Browse and explore AWS Bedrock foundation models. View model capabilities, pricing, and add models to your managed collection.', - // icon: 'heroCpuChip', - // route: '/admin/bedrock/models', - // }, - // { - // title: 'Gemini Models', - // description: 'Browse and explore Google Gemini AI models. View model specifications, features, and add models to your managed collection.', - // icon: 'heroSparkles', - // route: '/admin/gemini/models', - // }, - // { - // title: 'OpenAI Models', - // description: 'Browse and explore OpenAI models including GPT-4 and other offerings. View capabilities and add models to your managed collection.', - // icon: 'heroCpuChip', - // route: '/admin/openai/models', - // }, - - { - title: 'User Lookup', - description: 'Search and browse users to view their profile, costs, and quota status. Manage user-specific overrides and assignments.', - icon: 'heroUsers', - route: '/admin/users', - }, - { - title: 'Role Management', - description: 'Create and manage application roles with tool and model permissions. Configure JWT mappings and role inheritance.', - icon: 'heroShieldCheck', - route: '/admin/roles', - }, - { - title: 'Auth Providers', - description: 'Configure OIDC authentication providers for user login. Manage issuer URLs, client credentials, claim mappings, and login page appearance.', - icon: 'heroFingerPrint', - route: '/admin/auth-providers', - }, - { - title: 'Connectors', - description: 'Configure third-party OAuth integrations that users can connect for MCP tool authentication. Manage Google, Microsoft, GitHub, and custom connectors.', - icon: 'heroLink', - route: '/admin/connectors', - }, - { - title: 'Fine-Tuning Access', - description: 'Manage which users can access fine-tuning. Grant or revoke access, set monthly compute hour quotas, and monitor usage.', - icon: 'heroAcademicCap', - route: '/admin/fine-tuning', - }, - { - title: 'Fine-Tuning Costs', - description: 'View per-user GPU compute costs, hours used, and job counts for fine-tuning. Drill into monthly breakdowns.', - icon: 'heroChartBar', - route: '/admin/fine-tuning/costs', - }, - { - title: 'Quota Tiers', - description: 'Create and manage quota tiers with cost limits and soft limit configurations. Define monthly/daily limits and warning thresholds.', - icon: 'heroScale', - route: '/admin/quota/tiers', - }, - { - title: 'Quota Assignments', - description: 'Assign quota tiers to users, roles, or email domains. Control priority and manage default tier assignments.', - icon: 'heroClipboardDocumentList', - route: '/admin/quota/assignments', - }, - { - title: 'Quota Overrides', - description: 'Create temporary quota exceptions for individual users. Set custom limits or unlimited access with expiration dates.', - icon: 'heroCalendar', - route: '/admin/quota/overrides', - }, - { - title: 'Quota Inspector', - description: 'Debug and inspect quota resolution for individual users. View resolved quotas, current usage, and recent blocks.', - icon: 'heroMagnifyingGlass', - route: '/admin/quota/inspector', - }, - { - title: 'Quota Events', - description: 'Monitor quota enforcement events including warnings, blocks, resets, and override applications. Export event data to CSV.', - icon: 'heroChartBar', - route: '/admin/quota/events', - }, - - ]; - - getIconBackgroundClasses(index: number): string { - const backgrounds = [ - 'bg-purple-100 dark:bg-purple-900/30', - 'bg-blue-100 dark:bg-blue-900/30', - 'bg-green-100 dark:bg-green-900/30', - 'bg-amber-100 dark:bg-amber-900/30', - 'bg-pink-100 dark:bg-pink-900/30', - 'bg-indigo-100 dark:bg-indigo-900/30', - 'bg-teal-100 dark:bg-teal-900/30', - 'bg-rose-100 dark:bg-rose-900/30', - 'bg-emerald-100 dark:bg-emerald-900/30', - ]; - return backgrounds[index % backgrounds.length]; - } - - getIconColorClasses(index: number): string { - const colors = [ - 'text-purple-600 dark:text-purple-400', - 'text-blue-600 dark:text-blue-400', - 'text-green-600 dark:text-green-400', - 'text-amber-600 dark:text-amber-400', - 'text-pink-600 dark:text-pink-400', - 'text-indigo-600 dark:text-indigo-400', - 'text-teal-600 dark:text-teal-400', - 'text-rose-600 dark:text-rose-400', - 'text-emerald-600 dark:text-emerald-400', - ]; - return colors[index % colors.length]; - } -} diff --git a/frontend/ai.client/src/app/admin/admin.routes.ts b/frontend/ai.client/src/app/admin/admin.routes.ts new file mode 100644 index 00000000..49f1ed09 --- /dev/null +++ b/frontend/ai.client/src/app/admin/admin.routes.ts @@ -0,0 +1,139 @@ +import { Routes } from '@angular/router'; +import { FineTuningLayout } from './fine-tuning-access/fine-tuning.layout'; + +export const adminRoutes: Routes = [ + { + path: '', + redirectTo: 'costs', + pathMatch: 'full', + }, + { + path: 'costs', + loadComponent: () => import('./costs/admin-costs.page').then(m => m.AdminCostsPage), + }, + { + path: 'quota', + loadChildren: () => import('./quota-tiers/quota-routing.module').then(m => m.quotaRoutes), + }, + { + path: 'fine-tuning', + component: FineTuningLayout, + children: [ + { + path: '', + loadComponent: () => import('./fine-tuning-access/fine-tuning-access.page').then(m => m.FineTuningAccessPage), + }, + { + path: 'costs', + loadComponent: () => import('./fine-tuning-costs/fine-tuning-costs.page').then(m => m.FineTuningCostsPage), + }, + ], + }, + { + path: 'manage-models', + loadComponent: () => import('./manage-models/manage-models.page').then(m => m.ManageModelsPage), + }, + { + path: 'manage-models/new', + loadComponent: () => import('./manage-models/model-form.page').then(m => m.ModelFormPage), + }, + { + path: 'manage-models/edit/:id', + loadComponent: () => import('./manage-models/model-form.page').then(m => m.ModelFormPage), + }, + { + path: 'bedrock/models', + loadComponent: () => import('./bedrock-models/bedrock-models.page').then(m => m.BedrockModelsPage), + }, + { + path: 'gemini/models', + loadComponent: () => import('./gemini-models/gemini-models.page').then(m => m.GeminiModelsPage), + }, + { + path: 'openai/models', + loadComponent: () => import('./openai-models/openai-models.page').then(m => m.OpenAIModelsPage), + }, + { + path: 'tools', + loadComponent: () => import('./tools/pages/tool-list.page').then(m => m.ToolListPage), + }, + { + path: 'tools/new', + loadComponent: () => import('./tools/pages/tool-form.page').then(m => m.ToolFormPage), + }, + { + path: 'tools/edit/:toolId', + loadComponent: () => import('./tools/pages/tool-form.page').then(m => m.ToolFormPage), + }, + { + path: 'connectors', + loadComponent: () => import('./connectors/pages/connector-list.page').then(m => m.ConnectorListPage), + }, + { + path: 'connectors/new', + loadComponent: () => import('./connectors/pages/connector-form.page').then(m => m.ConnectorFormPage), + }, + { + path: 'connectors/edit/:providerId', + loadComponent: () => import('./connectors/pages/connector-form.page').then(m => m.ConnectorFormPage), + }, + { + path: 'oauth-providers', + redirectTo: 'connectors', + pathMatch: 'full', + }, + { + path: 'oauth-providers/new', + redirectTo: 'connectors/new', + pathMatch: 'full', + }, + { + path: 'oauth-providers/edit/:providerId', + redirectTo: 'connectors/edit/:providerId', + pathMatch: 'full', + }, + { + path: 'users', + loadComponent: () => import('./users/pages/user-list/user-list.page').then(m => m.UserListPage), + }, + { + path: 'users/:userId', + loadComponent: () => import('./users/pages/user-detail/user-detail.page').then(m => m.UserDetailPage), + }, + { + path: 'roles', + loadComponent: () => import('./roles/pages/role-list.page').then(m => m.RoleListPage), + }, + { + path: 'roles/new', + loadComponent: () => import('./roles/pages/role-form.page').then(m => m.RoleFormPage), + }, + { + path: 'roles/edit/:id', + loadComponent: () => import('./roles/pages/role-form.page').then(m => m.RoleFormPage), + }, + { + path: 'auth-providers', + loadComponent: () => import('./auth-providers/pages/provider-list.page').then(m => m.AuthProviderListPage), + }, + { + path: 'auth-providers/new', + loadComponent: () => import('./auth-providers/pages/provider-form.page').then(m => m.AuthProviderFormPage), + }, + { + path: 'auth-providers/edit/:providerId', + loadComponent: () => import('./auth-providers/pages/provider-form.page').then(m => m.AuthProviderFormPage), + }, + { + path: 'manage-user-menu-links', + loadComponent: () => import('./manage-user-menu-links/manage-user-menu-links.page').then(m => m.ManageUserMenuLinksPage), + }, + { + path: 'manage-user-menu-links/new', + loadComponent: () => import('./manage-user-menu-links/user-menu-link-form.page').then(m => m.UserMenuLinkFormPage), + }, + { + path: 'manage-user-menu-links/edit/:id', + loadComponent: () => import('./manage-user-menu-links/user-menu-link-form.page').then(m => m.UserMenuLinkFormPage), + }, +]; diff --git a/frontend/ai.client/src/app/admin/auth-providers/pages/provider-form.page.ts b/frontend/ai.client/src/app/admin/auth-providers/pages/provider-form.page.ts index c2468ae9..654cc338 100644 --- a/frontend/ai.client/src/app/admin/auth-providers/pages/provider-form.page.ts +++ b/frontend/ai.client/src/app/admin/auth-providers/pages/provider-form.page.ts @@ -69,8 +69,7 @@ interface ProviderFormGroup { class: 'block', }, template: ` -
-
+
`, }) diff --git a/frontend/ai.client/src/app/admin/auth-providers/pages/provider-list.page.ts b/frontend/ai.client/src/app/admin/auth-providers/pages/provider-list.page.ts index 2472f7b5..2fc47b63 100644 --- a/frontend/ai.client/src/app/admin/auth-providers/pages/provider-list.page.ts +++ b/frontend/ai.client/src/app/admin/auth-providers/pages/provider-list.page.ts @@ -45,15 +45,6 @@ import { AuthProvider } from '../models/auth-provider.model'; class: 'block p-6', }, template: ` - - - - Back to Admin - -

Authentication Providers

@@ -118,7 +109,7 @@ import { AuthProvider } from '../models/auth-provider.model';

Loading providers... diff --git a/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.html b/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.html index 608cc9d9..995e7713 100644 --- a/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.html +++ b/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.html @@ -1,149 +1,119 @@

-
+
- + -
-

Bedrock Foundation Models

-

- View and filter available AWS Bedrock foundation models +

+

Bedrock Foundation Models

+

+ Browse available AWS Bedrock foundation models and add them to your managed list.

- -
-

Filters

- -
- -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
- - -
+ +
+
+
- -
+ + + + + + + + + + + + + + + + + @if (hasActiveFilters()) { - @if (hasActiveFilters()) { - - } -
+ }
@@ -155,157 +125,125 @@

Filters @if (error()) { -
+

Error loading models

{{ error() }}

} - @if (!isLoading() && !error()) { -
-

- Showing {{ models().length }} model{{ models().length !== 1 ? 's' : '' }} -

-
+ +

+ {{ models().length }} model{{ models().length !== 1 ? 's' : '' }} +

@if (models().length === 0) { -
-

+

+

No models found matching the current filters.

} @else { -
+
    @for (model of models(); track model.modelId) { -
    - -
    -
    -

    +
  • + +
    + + +
    + {{ model.modelName }} -

    • -

    + +

    {{ model.modelId }}

    - + + -
    - - -
    - -
    -

    Input Modalities

    -
    - @for (modality of model.inputModalities; track modality) { - - {{ modality }} - - } - @if (model.inputModalities.length === 0) { - None - } -
    -
    - -
    -

    Output Modalities

    -
    - @for (modality of model.outputModalities; track modality) { - - {{ modality }} - - } - @if (model.outputModalities.length === 0) { - None - } -
    -
    - - -
    -

    Inference Types

    -
    - @for (type of model.inferenceTypesSupported; track type) { - - {{ type }} - - } - @if (model.inferenceTypesSupported.length === 0) { - None - } -
    -
    - - -
    -

    Customizations

    -
    - @for (customization of model.customizationsSupported; track customization) { - - {{ customization }} - - } - @if (model.customizationsSupported.length === 0) { - None - } -
    -
    -
    - - -
    -
    -
    - @if (model.responseStreamingSupported) { - - - - Streaming supported - } @else { - - - - No streaming - } -
    - - @if (model.modelLifecycle) { -
    - Status: - {{ model.modelLifecycle }} -
    - } -
    - - @if (isModelAdded(model.modelId)) { -
    - - - + +
    + } @else { }
    -
    + + + @if (isExpanded(model.modelId)) { +
    +
    +
    +
    Input modalities
    +
    + {{ model.inputModalities.length > 0 ? model.inputModalities.join(', ') : 'None' }} +
    +
    +
    +
    Output modalities
    +
    + {{ model.outputModalities.length > 0 ? model.outputModalities.join(', ') : 'None' }} +
    +
    +
    +
    Streaming
    +
    + {{ model.responseStreamingSupported ? 'Supported' : 'Not supported' }} +
    +
    +
    +
    Inference types
    +
    + {{ model.inferenceTypesSupported.length > 0 ? model.inferenceTypesSupported.join(', ') : 'None' }} +
    +
    +
    +
    Customizations
    +
    + {{ model.customizationsSupported.length > 0 ? model.customizationsSupported.join(', ') : 'None' }} +
    +
    + @if (model.modelLifecycle) { +
    +
    Lifecycle
    +
    {{ model.modelLifecycle }}
    +
    + } +
    +
    + } + } -
+ } }
diff --git a/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.ts b/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.ts index 6eec47d7..039c51e3 100644 --- a/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.ts +++ b/frontend/ai.client/src/app/admin/bedrock-models/bedrock-models.page.ts @@ -2,7 +2,13 @@ import { Component, ChangeDetectionStrategy, inject, signal, computed } from '@a import { Router, RouterLink } from '@angular/router'; import { FormsModule } from '@angular/forms'; import { NgIcon, provideIcons } from '@ng-icons/core'; -import { heroArrowLeft } from '@ng-icons/heroicons/outline'; +import { + heroArrowLeft, + heroPlus, + heroMagnifyingGlass, + heroChevronDown, +} from '@ng-icons/heroicons/outline'; +import { heroCheckCircleSolid } from '@ng-icons/heroicons/solid'; import { BedrockModelsService } from './services/bedrock-models.service'; import { FoundationModelSummary } from './models/bedrock-model.model'; import { ManagedModelsService } from '../manage-models/services/managed-models.service'; @@ -11,7 +17,15 @@ import { ThinkingDotsComponent } from '../../components/thinking-dots.component' @Component({ selector: 'app-bedrock-models-page', imports: [FormsModule, ThinkingDotsComponent, RouterLink, NgIcon], - providers: [provideIcons({ heroArrowLeft })], + providers: [ + provideIcons({ + heroArrowLeft, + heroPlus, + heroMagnifyingGlass, + heroChevronDown, + heroCheckCircleSolid, + }), + ], templateUrl: './bedrock-models.page.html', styleUrl: './bedrock-models.page.css', changeDetection: ChangeDetectionStrategy.OnPush, @@ -29,6 +43,9 @@ export class BedrockModelsPage { maxResultsFilter = signal(undefined); searchQuery = signal(''); + // Row detail expansion state (set of model ids currently expanded) + private expandedIds = signal>(new Set()); + // Access the models resource from the service readonly modelsResource = this.bedrockModelsService.modelsResource; @@ -125,6 +142,22 @@ export class BedrockModelsPage { ); }); + isExpanded(modelId: string): boolean { + return this.expandedIds().has(modelId); + } + + toggleExpand(modelId: string): void { + this.expandedIds.update(current => { + const next = new Set(current); + if (next.has(modelId)) { + next.delete(modelId); + } else { + next.add(modelId); + } + return next; + }); + } + /** * Check if a model has already been added to the managed models list */ diff --git a/frontend/ai.client/src/app/admin/connectors/pages/connector-form.page.ts b/frontend/ai.client/src/app/admin/connectors/pages/connector-form.page.ts index b43df68c..35adbbc6 100644 --- a/frontend/ai.client/src/app/admin/connectors/pages/connector-form.page.ts +++ b/frontend/ai.client/src/app/admin/connectors/pages/connector-form.page.ts @@ -103,8 +103,7 @@ const ICON_ACCEPTED_MIME_TYPES = [ ], host: { class: 'block' }, template: ` -
-
+
@@ -572,7 +571,6 @@ const ICON_ACCEPTED_MIME_TYPES = [
} -
`, }) diff --git a/frontend/ai.client/src/app/admin/connectors/pages/connector-list.page.ts b/frontend/ai.client/src/app/admin/connectors/pages/connector-list.page.ts index 720b9e0f..eb9db52a 100644 --- a/frontend/ai.client/src/app/admin/connectors/pages/connector-list.page.ts +++ b/frontend/ai.client/src/app/admin/connectors/pages/connector-list.page.ts @@ -58,17 +58,7 @@ import { class: 'block', }, template: ` -
-
- - - - Back to Admin - - +
@@ -149,7 +139,7 @@ import {

Loading connectors... @@ -364,7 +354,6 @@ import {

} -
`, }) diff --git a/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts b/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts index 106ebd48..069a0998 100644 --- a/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts +++ b/frontend/ai.client/src/app/admin/costs/admin-costs.page.ts @@ -39,18 +39,7 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component' providers: [provideIcons({ heroArrowLeft, heroArrowDownTray })], changeDetection: ChangeDetectionStrategy.OnPush, template: ` -
- -
- - - - Back to Admin - - +
@@ -83,7 +72,7 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component'

Loading dashboard data... @@ -128,7 +117,7 @@ import { ModelBreakdownComponent } from './components/model-breakdown.component'

} @else { -
+
} -
`, }) diff --git a/frontend/ai.client/src/app/admin/costs/components/system-summary-card.component.ts b/frontend/ai.client/src/app/admin/costs/components/system-summary-card.component.ts index a724f5cb..b25ba1b3 100644 --- a/frontend/ai.client/src/app/admin/costs/components/system-summary-card.component.ts +++ b/frontend/ai.client/src/app/admin/costs/components/system-summary-card.component.ts @@ -46,52 +46,50 @@ export type SummaryCardIcon =
-
-
-

- {{ title() }} -

-

- {{ value() }} -

- - @if (trend() !== null && trend() !== undefined) { -
- @if (trend()! > 0) { - - - +{{ trend() | number : '1.1-1' }}% - - } @else if (trend()! < 0) { - - - {{ trend() | number : '1.1-1' }}% - - } @else { - - No change - - } - - vs last period - -
- } -
- +
+

+ {{ title() }} +

- +
+ +

+ {{ value() }} +

+ + @if (trend() !== null && trend() !== undefined) { +
+ @if (trend()! > 0) { + + + +{{ trend() | number : '1.1-1' }}% + + } @else if (trend()! < 0) { + + + {{ trend() | number : '1.1-1' }}% + + } @else { + + No change + + } + + vs last period + +
+ }
`, }) diff --git a/frontend/ai.client/src/app/admin/costs/components/top-users-table.component.ts b/frontend/ai.client/src/app/admin/costs/components/top-users-table.component.ts index aca037d8..d801cb2a 100644 --- a/frontend/ai.client/src/app/admin/costs/components/top-users-table.component.ts +++ b/frontend/ai.client/src/app/admin/costs/components/top-users-table.component.ts @@ -303,7 +303,7 @@ type SortDirection = 'asc' | 'desc'; >
Loading more users...
diff --git a/frontend/ai.client/src/app/admin/fine-tuning-access/fine-tuning-access.page.html b/frontend/ai.client/src/app/admin/fine-tuning-access/fine-tuning-access.page.html index 010a0fc0..1d052338 100644 --- a/frontend/ai.client/src/app/admin/fine-tuning-access/fine-tuning-access.page.html +++ b/frontend/ai.client/src/app/admin/fine-tuning-access/fine-tuning-access.page.html @@ -1,10 +1,4 @@
- - - - Back to Admin - -
@@ -94,7 +88,7 @@

Grant @if (state.loading() && state.grantCount() === 0) {
-
+
} diff --git a/frontend/ai.client/src/app/admin/fine-tuning-access/fine-tuning.layout.ts b/frontend/ai.client/src/app/admin/fine-tuning-access/fine-tuning.layout.ts new file mode 100644 index 00000000..0d366a6e --- /dev/null +++ b/frontend/ai.client/src/app/admin/fine-tuning-access/fine-tuning.layout.ts @@ -0,0 +1,48 @@ +import { Component, ChangeDetectionStrategy } from '@angular/core'; +import { RouterLink, RouterLinkActive, RouterOutlet } from '@angular/router'; + +interface FineTuningTab { + label: string; + route: string; + exact: boolean; +} + +@Component({ + selector: 'app-fine-tuning-layout', + changeDetection: ChangeDetectionStrategy.OnPush, + imports: [RouterLink, RouterLinkActive, RouterOutlet], + host: { class: 'block' }, + template: ` +
+

Fine-Tuning

+

+ Manage who can access fine-tuning and review the resulting compute spend. +

+
+ +
+ +
+ + + `, +}) +export class FineTuningLayout { + readonly tabs: FineTuningTab[] = [ + { label: 'Access', route: '.', exact: true }, + { label: 'Costs', route: 'costs', exact: false }, + ]; +} diff --git a/frontend/ai.client/src/app/admin/fine-tuning-costs/fine-tuning-costs.page.html b/frontend/ai.client/src/app/admin/fine-tuning-costs/fine-tuning-costs.page.html index dc634d59..de655b87 100644 --- a/frontend/ai.client/src/app/admin/fine-tuning-costs/fine-tuning-costs.page.html +++ b/frontend/ai.client/src/app/admin/fine-tuning-costs/fine-tuning-costs.page.html @@ -1,14 +1,5 @@
- - - - Back to Admin - -
@@ -48,7 +39,7 @@

Loading cost data diff --git a/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.html b/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.html index d77e21d5..39f9251e 100644 --- a/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.html +++ b/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.html @@ -1,77 +1,67 @@
-
+
- + -
-

Google Gemini Models

-

- View and manage available Google Gemini models +

+

Google Gemini Models

+

+ Browse available Google Gemini models and add them to your managed list.

- -
-

Filters

- -
- -
- - -
- - -
- - -
+ +
+
+
- -
+ + + + + @if (hasActiveFilters()) { - @if (hasActiveFilters()) { - - } -
+ }
@@ -83,148 +73,133 @@

Filters @if (error()) { -
+

Error loading models

{{ error() }}

} - @if (!isLoading() && !error()) { -
-

- Showing {{ models().length }} model{{ models().length !== 1 ? 's' : '' }} -

-
+ +

+ {{ models().length }} model{{ models().length !== 1 ? 's' : '' }} +

@if (models().length === 0) { -
-

- No models found. -

+
+

No models found.

} @else { -
+
    @for (model of models(); track model.name) { -
    - -
    -
    -

    - {{ model.displayName }} -

    -

    - {{ model.name }} -

    - @if (model.description) { -

    - {{ model.description }} -

    - } -
    - - Google - -
    +
  • + +
    + - -
    - -
    -

    Token Limits

    -
    - @if (model.inputTokenLimit) { -
    - Input: - {{ model.inputTokenLimit | number }} -
    - } - @if (model.outputTokenLimit) { -
    - Output: - {{ model.outputTokenLimit | number }} -
    - } - @if (!model.inputTokenLimit && !model.outputTokenLimit) { - Not specified +
    +
    + + {{ model.displayName }} + + @if (model.thinking) { + + Thinking + }
    +

    + {{ model.name }} +

    - - @if (model.temperature !== null || model.topP !== null || model.topK !== null) { -
    -

    Parameters

    -
    - @if (model.temperature !== null) { -
    - Temperature: - {{ model.temperature }} -
    - } - @if (model.topP !== null) { -
    - Top-P: - {{ model.topP }} -
    - } - @if (model.topK !== null) { -
    - Top-K: - {{ model.topK }} -
    - } -
    -
    - } - - - @if (model.version) { -
    -

    Version

    -

    {{ model.version }}

    -
    - } -
    - - -
    -
    - - @if (model.thinking) { -
    - - - - Thinking model -
    - } -
    + - @if (isModelAdded(model.name)) { -
    - - - + +
    + } @else { }
    -
    + + + @if (isExpanded(model.name)) { +
    + @if (model.description) { +

    {{ model.description }}

    + } +
    +
    +
    Token limits
    +
    + @if (model.inputTokenLimit || model.outputTokenLimit) { + {{ (model.inputTokenLimit || 0) | number }} in + · + {{ (model.outputTokenLimit || 0) | number }} out + } @else { + Not specified + } +
    +
    + + @if (model.temperature !== undefined || model.topP !== undefined || model.topK !== undefined) { +
    +
    Parameters
    +
    + @if (model.temperature !== undefined) { +
    Temperature: {{ model.temperature }}
    + } + @if (model.topP !== undefined) { +
    Top-P: {{ model.topP }}
    + } + @if (model.topK !== undefined) { +
    Top-K: {{ model.topK }}
    + } +
    +
    + } + + @if (model.version) { +
    +
    Version
    +
    {{ model.version }}
    +
    + } +
    +
    + } +
    • } -
    +
} }
diff --git a/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.ts b/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.ts index 2a9eeb50..caca6ae6 100644 --- a/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.ts +++ b/frontend/ai.client/src/app/admin/gemini-models/gemini-models.page.ts @@ -3,7 +3,13 @@ import { Router, RouterLink } from '@angular/router'; import { FormsModule } from '@angular/forms'; import { DecimalPipe } from '@angular/common'; import { NgIcon, provideIcons } from '@ng-icons/core'; -import { heroArrowLeft } from '@ng-icons/heroicons/outline'; +import { + heroArrowLeft, + heroPlus, + heroMagnifyingGlass, + heroChevronDown, +} from '@ng-icons/heroicons/outline'; +import { heroCheckCircleSolid } from '@ng-icons/heroicons/solid'; import { GeminiModelsService } from './services/gemini-models.service'; import { GeminiModelSummary } from './models/gemini-model.model'; import { ManagedModelsService } from '../manage-models/services/managed-models.service'; @@ -12,7 +18,15 @@ import { ThinkingDotsComponent } from '../../components/thinking-dots.component' @Component({ selector: 'app-gemini-models-page', imports: [FormsModule, ThinkingDotsComponent, DecimalPipe, RouterLink, NgIcon], - providers: [provideIcons({ heroArrowLeft })], + providers: [ + provideIcons({ + heroArrowLeft, + heroPlus, + heroMagnifyingGlass, + heroChevronDown, + heroCheckCircleSolid, + }), + ], templateUrl: './gemini-models.page.html', styleUrl: './gemini-models.page.css', changeDetection: ChangeDetectionStrategy.OnPush, @@ -26,6 +40,9 @@ export class GeminiModelsPage { maxResultsFilter = signal(undefined); searchQuery = signal(''); + // Row detail expansion state (set of model names currently expanded) + private expandedIds = signal>(new Set()); + // Access the models resource from the service readonly modelsResource = this.geminiModelsService.modelsResource; @@ -82,6 +99,22 @@ export class GeminiModelsPage { return !!(this.maxResultsFilter() || this.searchQuery()); }); + isExpanded(modelName: string): boolean { + return this.expandedIds().has(modelName); + } + + toggleExpand(modelName: string): void { + this.expandedIds.update(current => { + const next = new Set(current); + if (next.has(modelName)) { + next.delete(modelName); + } else { + next.add(modelName); + } + return next; + }); + } + /** * Check if a model has already been added to the managed models list */ diff --git a/frontend/ai.client/src/app/admin/manage-models/manage-models.page.html b/frontend/ai.client/src/app/admin/manage-models/manage-models.page.html index 49c5651c..b16b08bb 100644 --- a/frontend/ai.client/src/app/admin/manage-models/manage-models.page.html +++ b/frontend/ai.client/src/app/admin/manage-models/manage-models.page.html @@ -1,243 +1,244 @@
-
- - - - Back to Admin - - +
-
+
-

Manage Models

-

- View and manage AI models available to users. +

Manage Models

+

+ Enable, configure, and remove the models available to users.

- - - +
- -
-

Search & Filters

- -
- -
- - -
+ +
+
+
- -
- - -
+ + - -
- - -
-
+ + - @if (hasActiveFilters()) { -
- -
+ }
- -
-

- Showing {{ filteredModels().length }} model{{ filteredModels().length !== 1 ? 's' : '' }} + +

+

+ {{ filteredModels().length }} model{{ filteredModels().length !== 1 ? 's' : '' }}

-
- - Browse Bedrock Models → - - - Browse Gemini Models → - - - Browse OpenAI Models → - +
+ Add from catalog: + Bedrock + Gemini + OpenAI
@if (filteredModels().length === 0) { -
-

+

+

No models found matching the current filters.

} @else { -
+
    @for (model of filteredModels(); track model.id) { -
    - -
    -
    -
    -

    +
  • + +
    + + + + +
    +
    + {{ model.modelName }} -

    • - @if (model.enabled) { - - Enabled - - } @else { - - Disabled - + + @if (model.isDefault) { + }
    -

    +

    {{ model.modelId }}

    -
    - - {{ model.provider }} - - - {{ model.providerName }} - -
    -
    - -
    - -
    -

    Allowed Roles

    -
    - @if (model.allowedAppRoles && model.allowedAppRoles.length > 0) { - @for (roleId of model.allowedAppRoles; track roleId) { - - {{ getRoleDisplayName(roleId) }} - - } - } @else { - No roles assigned - } -
    -
    + + - -
    -

    Pricing (per 1M tokens)

    -
    -

    - Input: ${{ model.inputPricePerMillionTokens.toFixed(2) }} -

    -

    - Output: ${{ model.outputPricePerMillionTokens.toFixed(2) }} -

    -
    + +
    + +
    - -
    -

    Modalities

    -
    -

    - Input: {{ model.inputModalities.join(', ') }} -

    -

    - Output: {{ model.outputModalities.join(', ') }} -

    -
    + +
    + + +
    - -
    - + @if (isExpanded(model.id)) { +
    - - - - Edit - - -
    -
    +
    +
    +
    + Pricing / 1M tokens +
    +
    + ${{ model.inputPricePerMillionTokens.toFixed(2) }} in + · + ${{ model.outputPricePerMillionTokens.toFixed(2) }} out +
    +
    + +
    +
    + Modalities +
    +
    + {{ model.inputModalities.join(', ') }} + → + {{ model.outputModalities.join(', ') }} +
    +
    + +
    +
    + Allowed roles +
    +
    + @if (model.allowedAppRoles && model.allowedAppRoles.length > 0) { + @for (roleId of model.allowedAppRoles; track roleId) { + + {{ getRoleDisplayName(roleId) }} + + } + } @else { + No roles assigned + } +
    +
    +
    +
    + } + } -
    +
}
diff --git a/frontend/ai.client/src/app/admin/manage-models/manage-models.page.ts b/frontend/ai.client/src/app/admin/manage-models/manage-models.page.ts index e67191d6..e46ca05d 100644 --- a/frontend/ai.client/src/app/admin/manage-models/manage-models.page.ts +++ b/frontend/ai.client/src/app/admin/manage-models/manage-models.page.ts @@ -2,14 +2,31 @@ import { Component, ChangeDetectionStrategy, signal, computed, inject } from '@a import { RouterLink } from '@angular/router'; import { FormsModule } from '@angular/forms'; import { NgIcon, provideIcons } from '@ng-icons/core'; -import { heroArrowLeft } from '@ng-icons/heroicons/outline'; +import { + heroPlus, + heroMagnifyingGlass, + heroChevronDown, + heroPencilSquare, + heroTrash, +} from '@ng-icons/heroicons/outline'; +import { heroStarSolid } from '@ng-icons/heroicons/solid'; import { ManagedModelsService } from './services/managed-models.service'; import { AppRolesService } from '../roles/services/app-roles.service'; +import type { ManagedModel } from './models/managed-model.model'; @Component({ selector: 'app-manage-models-page', imports: [RouterLink, FormsModule, NgIcon], - providers: [provideIcons({ heroArrowLeft })], + providers: [ + provideIcons({ + heroPlus, + heroMagnifyingGlass, + heroChevronDown, + heroPencilSquare, + heroTrash, + heroStarSolid, + }), + ], templateUrl: './manage-models.page.html', styleUrl: './manage-models.page.css', changeDetection: ChangeDetectionStrategy.OnPush, @@ -23,12 +40,17 @@ export class ManageModelsPage { providerFilter = signal(''); enabledFilter = signal(''); - // Get models from service - private mockModels = computed(() => this.managedModelsService.getManagedModels()); + // Row detail expansion state (set of model ids currently expanded) + private expandedIds = signal>(new Set()); + + // Models with an in-flight enable/disable request + private togglingIds = signal>(new Set()); + + private allModels = computed(() => this.managedModelsService.getManagedModels()); // Filtered models based on search and filters readonly filteredModels = computed(() => { - let models = this.mockModels(); + let models = this.allModels(); const query = this.searchQuery().toLowerCase(); const provider = this.providerFilter(); const enabled = this.enabledFilter(); @@ -56,7 +78,7 @@ export class ManageModelsPage { // Available providers for filter dropdown readonly availableProviders = computed(() => { - const providers = new Set(this.mockModels().map(m => m.providerName)); + const providers = new Set(this.allModels().map(m => m.providerName)); return Array.from(providers).sort(); }); @@ -74,6 +96,48 @@ export class ManageModelsPage { this.enabledFilter.set(''); } + isExpanded(modelId: string): boolean { + return this.expandedIds().has(modelId); + } + + toggleExpand(modelId: string): void { + this.expandedIds.update(current => { + const next = new Set(current); + if (next.has(modelId)) { + next.delete(modelId); + } else { + next.add(modelId); + } + return next; + }); + } + + isToggling(modelId: string): boolean { + return this.togglingIds().has(modelId); + } + + /** + * Flip a model's enabled state in place via a partial update. + */ + async toggleEnabled(model: ManagedModel): Promise { + if (this.isToggling(model.id)) { + return; + } + this.togglingIds.update(current => new Set(current).add(model.id)); + try { + await this.managedModelsService.updateModel(model.id, { enabled: !model.enabled }); + } catch (error) { + console.error('Error updating model status:', error); + alert('Failed to update model status. Please try again.'); + } finally { + this.togglingIds.update(current => { + const next = new Set(current); + next.delete(model.id); + return next; + }); + } + } + /** * Delete a model */ diff --git a/frontend/ai.client/src/app/admin/manage-models/model-form.page.html b/frontend/ai.client/src/app/admin/manage-models/model-form.page.html index 88748bcb..f5ec0a4c 100644 --- a/frontend/ai.client/src/app/admin/manage-models/model-form.page.html +++ b/frontend/ai.client/src/app/admin/manage-models/model-form.page.html @@ -552,6 +552,37 @@

Inference Para Default to enabled

+ } @else if (meta.kind === 'select') { +
+ + Levels this model supports + +
+ @for (lvl of meta.options ?? []; track lvl) { + + } +
+
+
+ + +
}