diff --git a/claudedocs/handoff-agent-setup-onboarding.md b/claudedocs/handoff-agent-setup-onboarding.md index 8325aa7..0ae0565 100644 --- a/claudedocs/handoff-agent-setup-onboarding.md +++ b/claudedocs/handoff-agent-setup-onboarding.md @@ -244,6 +244,59 @@ dashboard change by someone with zone access. `via: measurement` (ruled out as ours: the same `nginx.conf` returns 200 locally to the blocked UA, and there is no `_headers`/`_redirects`/`wrangler.*` anywhere in the docs repo). +### βœ… RESOLVED β€” `developer.civitai.com` 403 to `Python-urllib` + +πŸ”΄ **The block below titled *"returns 403 to `Python-urllib` β€” IN FLIGHT, delegated"* +is RETIRED. Do NOT run its "Next probe", and do NOT look in `nginx.conf`.** Its +leading hypothesis β€” *"a UA denylist in the docs repo's own `nginx.conf`"*, tagged +`via: assumed` β€” is **REFUTED**. The block is left intact above only because a +corrected reading is worth more than a deleted one. + +**Root cause: Cloudflare Browser Integrity Check (BIC)**, a zone-level setting on +the `civitai.com` zone. Never the origin, never the docs repo. + +- **Symptom + exact repro (now fixed β€” this reproduces nothing today):** + ```bash + curl -s -o /dev/null -w '%{http_code}\n' -A 'Python-urllib/3.12' \ + https://developer.civitai.com/agent-setup/prompt.md # was 403, now 200 + ``` +- **Observed (with values), measured 2026-09-09 β†’ 2026-09-11:** + - Cloudflare firewall events (GraphQL `firewallEventsAdaptive`, zone + `civitai.com`, host `developer.civitai.com`): + `action=block source=bic ruleId=bic` on `/agent-setup/prompt.md` for + `Python-urllib/3.11`, `Python-urllib/3.12`, `Python-urllib`, `libwww-perl/6.0`. + - Block body was **17 bytes**, `content-type: text/plain`: `error code: 1010` + β€” Cloudflare's BIC error code. + - The 403 carried **none** of the origin's headers (no `etag`, no + `x-content-type-options`, no `cf-cache-status`), while the 200 carried all + three. The request never reached nginx. + - Zone settings: `browser_check = "on"` (zone-wide, `editable: true`), + `security_level = "essentially_off"`. BIC is independent of security level. + - πŸ”΄ **The denylist is exact-string and case-sensitive, and mostly porous:** + `Python-urllib/3.11` β†’ 403 but `python-urllib/3.11` β†’ **200**; + `MyAgent (urllib inside)` β†’ 200; `` β†’ **200**; `Wget/1.21`, + `Go-http-client/1.1`, `node-fetch`, `axios`, `okhttp`, `Mozilla/5.0` β†’ all 200. + Only `Python-urllib*` and `libwww-perl*` were blocked. Cloudflare publishes no + list, no versioning and no changelog for it. +- **Ruled out:** that it is in the docs repo's `nginx.conf` β€” the 403 carries no + origin headers at all and Cloudflare's own events attribute it to `bic`. + `via: measurement` Β· That a WAF **managed** ruleset did it β€” events name `bic`, + not a ruleset id, and BIC is a zone product a managed-rules skip cannot reach. + `via: measurement` Β· That a **custom** firewall rule did it β€” all 14 read; none + matches `urllib`/`libwww`, and the host-scoped ones target `civitai.com` and + `image.civitai.com` only. `via: command` Β· That **AI Crawl Control** could fix + it β€” it manages known, self-identifying crawlers; `Python-urllib` is neither + recognised nor verifiable. `via: doc` + (https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) +- **Resolution:** a **Configuration Rule** on the `civitai.com` zone: + `(http.host eq "developer.civitai.com")` β†’ `action: set_config`, + `action_parameters: {"bic": false}`. `browser_check` stays **`on`** zone-wide, + so `civitai.com` and `image.civitai.com` keep BIC. +- **Verified (content, not just status):** `Python-urllib/3.12` now receives + byte-identical content to a known-good UA on `/`, `/index.html`, `/llms.txt`, + `/apps/` and `/agent-setup/prompt.md` β€” `prompt.md` 6,949 B and `llms.txt` + 13,066 B, `cmp` clean both, `content-type: text/markdown`, origin `etag` present. + ## Next steps (ranked) πŸ”΄ **Ranks 1, 2, 3, 5 and 7 are DONE β€” numbering is preserved deliberately** so any @@ -462,6 +515,44 @@ live `claim-work` slug keeps pointing at the item it was taken for. paths. The better a session follows the delegate-and-isolate defaults, the blinder that window is. +- πŸ”΄ **A Page Rule target without a trailing `*` matches ONE path.** The first + attempt at the fix above landed as a Page Rule targeting + `developer.civitai.com/` (operator `matches`). Result: `/` β†’ 200 while + `/index.html`, `/llms.txt`, `/apps/` and `/agent-setup/prompt.md` all stayed + 403. It reads as "the rule didn't work"; it worked, on exactly one path. Needs + `developer.civitai.com/*`. +- πŸ”΄ **Prefer a Configuration Rule over a Page Rule for this class.** Page Rules + are **(deprecated)**; Cloudflare's BIC page says *"To use this feature on + specific hostnamesβ€”instead of across your entire zoneβ€”use a configuration + rule"*, and Configuration Rules take **precedence over** Page Rules, so keeping + both makes the interaction ambiguous. + (https://developers.cloudflare.com/waf/tools/browser-integrity-check/) +- πŸ”΄ **`PUT /zones/{z}/rulesets/{id}` REPLACES the entire rules array β€” and the + official docs only demonstrate `PUT`.** This zone has 5 other Configuration + Rules, two of them active `security_level` overrides; following Cloudflare's + own example verbatim would have silently deleted all five. **Append with + `POST /zones/{z}/rulesets/{id}/rules`.** +- πŸ”΄ **A Configuration Rule DOES override the zone-level `browser_check`, and no + Cloudflare doc says so.** Searched: the settings page states a precedence rule + only for Disable RUM. Confirmed **empirically instead** β€” `browser_check` reads + `"on"` while the exempted host serves 200 to a UA that setting blocks. If this + ever regresses, re-probe rather than re-reading the docs. +- **`set_config` is non-terminating, so within `http_config_settings` LAST MATCH + WINS.** Position matters if another rule ever sets `bic` for an overlapping host. +- **The `action` field is mandatory and is `set_config`** β€” a Configuration Rule + body carrying only `expression` + `action_parameters` 400s. + (https://developers.cloudflare.com/rules/configuration-rules/create-api/) +- **Reading Cloudflare state needs four separate token scopes**, and they fail + differently: Zone Settings read (`9109 Unauthorized`), Config Rules read and + Page Rules read (`request is not authorized`). πŸ”΄ **A `pagerules` call without + the scope returns a body whose `.result` is null β€” read as a count it yields a + confident `0` for a zone that has 13.** Check `.success` before any count. +- **Related, not done:** Cloudflare's **Markdown for Agents** + (`"content_converter": true`) sits on the same Configuration Rules surface and + serves the same "agents fetch our docs" goal; Cloudflare runs it on their own + docs. Worth its own decision. + (https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) + ## How to verify The nightly's fix β€” the ONLY form that actually exercises it: