feat: add Serper search fallback + search cache + concurrency limiter

[andyzengmath]

Problem

When multiple subagents run web searches in parallel (e.g. via the Workflow tool or Agent tool), DuckDuckGo aggressively rate-limits requests, causing cascading failures:

⚠ DDG Lite error: fetch failed
⚠ DDG Instant error: fetch failed
⚠ All search providers failed

This is a known issue — DDG was never designed for programmatic use at agent scale.

Solution

Three changes to proxy.mjs:

1. Serper.dev as fallback search provider

• Adds Serper.dev (Google SERP API) as a fallback provider via SERPER_API_KEY env var
• Free tier: 2,500 queries with no credit card
• Positioned after Exa/Parallel MCP (free) and Brave, before DDG — only burns credits when free providers fail

2. Search result cache (5-min TTL)

• Deduplicates identical queries across parallel subagents
• Logs 📋 Search cache hit: for visibility
• Caches even empty results to avoid re-hitting a failing provider

3. Concurrency semaphore (max 2 concurrent searches)

• Prevents more than 2 search requests hitting external APIs simultaneously
• Excess searches queue automatically (logs ⏳ Search queued:)
• Eliminates the DDG rate-limit storm that causes cascading failures

Updated Provider Chain

1. Exa / Parallel MCP   (free, no quota — tried first)
2. Brave Search API      (if BRAVE_API_KEY set)
3. Serper.dev            (if SERPER_API_KEY set — NEW)
4. DuckDuckGo Lite       (now protected by concurrency semaphore)
5. DuckDuckGo Instant    (last resort)

New Environment Variable

Variable	Description
SERPER_API_KEY	Serper.dev API key for Google SERP fallback (free 2,500 queries at serper.dev)

Testing

Verified Serper returns results:

🔍 Web search: "University of Pennsylvania Mathematics alumni venture capital"
✓ Serper returned 5 results

Verified cache + concurrency with 4 parallel searches (2 unique, 2 duplicates):

✅ FETCHED (3 results): Penn math alumni venture capital
📋 CACHE HIT: Penn math alumni venture capital
⏳ QUEUED (2/2 active): Cambridge Churchill College founders
✅ FETCHED (3 results): Cambridge Churchill College founders

Only 2 API calls instead of 4 — cache saved 50%.

Depends On

This PR builds on #3 (MCP providers). The provider chain assumes Exa/Parallel MCP are available as the primary free providers.

🤖 Generated with Claude Code