Skip to content

feat(renderer): reach the solver tier on hosts that block datacenter addresses - #569

Merged
us merged 2 commits into
mainfrom
fix/cloak-after-egress
Sep 19, 2026
Merged

us merged 2 commits into
mainfrom
fix/cloak-after-egress

Conversation

@us

@us us commented Sep 19, 2026

Copy link
Copy Markdown
Owner

What

Some origins answer a datacenter address with a plain block page and only serve
their challenge once the request arrives from a residential one. The decision to
route to the solver tier was taken from the ladder's body alone, before the
residential retry ran, so on these hosts it was always no: the retry then
uncovered the challenge, detected it, and threw the verdict away. The tier sat
configured and idle while every such request returned a block.

The residential retry's own verdict now opens that route.

Changes

  • renderer.cloak_after_egress (default off) lets the solver tier fire on a
    challenge the residential retry uncovered.
  • The retry's breaker verdict counts a challenge as the site blocking us rather
    than the tier misbehaving, but only for the shape that is routed onward, so an
    unrelated vendor wall still trips the breaker.
  • The solver runs on a budget sized from renderer.cloak_timeout_ms whenever
    the remaining request time cannot fit a whole attempt, on both routes. The
    sidecar refuses an attempt it cannot finish, so a partial budget wasted the
    slot.
  • renderer.cloak_pool_size makes the tier's concurrency configurable instead
    of a fixed 4. This is the box-wide throughput ceiling for the tier.
  • A counter for how often the route becomes eligible, incremented whether or not
    the flag is on, so the arrival rate is measurable before enabling it.
  • One log line per solver result with the host and which route fired.
  • The current Cloudflare block-page template is classified as cloudflare
    rather than generic_block: the previous arm keyed on a marker the template
    no longer contains. Response label only; no tier reads it.

Behaviour with the flag off

Unchanged, except the vendor label on Cloudflare block pages. The error code
stays the same.

Deploy caveats

  • Default off. Set CRW_RENDERER__CLOAK_AFTER_EGRESS to enable, and only
    alongside a sidecar that can serve it.
  • CRW_RENDERER__CLOAK_POOL_SIZE must be raised together with the sidecar's own
    concurrency and memory, or the engine hands out permits nothing can serve.
  • Enabling it adds a solver attempt to requests that previously returned a block
    immediately, so those get slower before they get answers.

Verification

  • cargo test -p crw-renderer --features cdp,cloak --lib: 1005 passed
  • cargo test -p crw-crawl --lib: 464 passed
  • cargo test -p crw-core --lib: 513 passed
  • cargo clippy -p crw-renderer -p crw-crawl -p crw-core --all-targets -- -D warnings: clean
  • cargo fmt --check: clean

New tests cover the route firing on a real captured block page, staying out on a
non-Cloudflare wall, both budget cases, the breaker verdict following the flag,
and the pool default. The block-page fixture is a real capture with the host, ray
ids and origin address scrubbed.

Not yet verified end to end against a live wall; that needs the paired sidecar
change and a staging run.

us added 2 commits September 19, 2026 19:45
The block-page arm keyed on `<span class="cf-error-code">`, which the template
Cloudflare serves today does not contain: a live capture has zero occurrences.
The page therefore fell through to `generic_block`, so a host that answers
datacenter addresses with it looked like an anonymous wall rather than a known
vendor, and the routing learner (which counts `blockVendor = 'cloudflare'`)
could never flag it.

Keyed on the two structural markers the current template does carry, required
together so prose mentioning one cannot trip it ahead of the markdown guard.
Only the response's vendor label changes; nothing in the renderer reads it, so
tier selection is untouched.

The fixture is a real capture with the host, ray ids and origin address
scrubbed.
…addresses

Some origins answer a datacenter address with a plain block page and only serve
their challenge once the request arrives from a residential one. The decision to
route to the solver tier was taken from the ladder's body alone, before the
residential retry ran, so on these hosts it was always no: the retry then
uncovered the challenge, detected it, and threw the verdict away. The tier sat
configured and idle while every such request returned a block.

The residential retry's own verdict now opens that route, behind
`renderer.cloak_after_egress` (default off, so this ships inert).

Alongside it:

- the retry's breaker verdict counts a challenge as the site blocking us rather
  than the tier misbehaving, but only for the shape that is actually routed
  onward, so an unrelated vendor wall still trips the breaker and stops a crawl
  from paying for a residential render on every page;
- the solver runs on a budget sized from `renderer.cloak_timeout_ms` whenever
  the request's remaining time cannot fit a whole attempt, on both routes. The
  sidecar refuses an attempt it cannot finish, so handing it a partial budget
  wasted the slot;
- `renderer.cloak_pool_size` makes the tier's concurrency configurable instead
  of a fixed 4. It is the box-wide throughput ceiling for this tier, so a batch
  on a walled host is bounded by it long before any per-plan limit. Raise it
  together with the sidecar's own concurrency and memory;
- a counter for how often the route becomes eligible, incremented whether or not
  the flag is on, so the arrival rate is measurable before enabling it;
- one log line per solver result carrying the host and which route fired. The
  existing counter cannot say which, and the previous line was behind a
  diagnostic flag that is off in production.

Tests cover the route firing on a real captured block page, staying out on a
non-Cloudflare wall, both budget cases, the breaker verdict following the flag,
and the pool default.
@us
us merged commit 63762e7 into main Sep 19, 2026
11 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant