feat(renderer): reach the solver tier on hosts that block datacenter addresses - #569
Merged
Merged
Conversation
The block-page arm keyed on `<span class="cf-error-code">`, which the template Cloudflare serves today does not contain: a live capture has zero occurrences. The page therefore fell through to `generic_block`, so a host that answers datacenter addresses with it looked like an anonymous wall rather than a known vendor, and the routing learner (which counts `blockVendor = 'cloudflare'`) could never flag it. Keyed on the two structural markers the current template does carry, required together so prose mentioning one cannot trip it ahead of the markdown guard. Only the response's vendor label changes; nothing in the renderer reads it, so tier selection is untouched. The fixture is a real capture with the host, ray ids and origin address scrubbed.
…addresses Some origins answer a datacenter address with a plain block page and only serve their challenge once the request arrives from a residential one. The decision to route to the solver tier was taken from the ladder's body alone, before the residential retry ran, so on these hosts it was always no: the retry then uncovered the challenge, detected it, and threw the verdict away. The tier sat configured and idle while every such request returned a block. The residential retry's own verdict now opens that route, behind `renderer.cloak_after_egress` (default off, so this ships inert). Alongside it: - the retry's breaker verdict counts a challenge as the site blocking us rather than the tier misbehaving, but only for the shape that is actually routed onward, so an unrelated vendor wall still trips the breaker and stops a crawl from paying for a residential render on every page; - the solver runs on a budget sized from `renderer.cloak_timeout_ms` whenever the request's remaining time cannot fit a whole attempt, on both routes. The sidecar refuses an attempt it cannot finish, so handing it a partial budget wasted the slot; - `renderer.cloak_pool_size` makes the tier's concurrency configurable instead of a fixed 4. It is the box-wide throughput ceiling for this tier, so a batch on a walled host is bounded by it long before any per-plan limit. Raise it together with the sidecar's own concurrency and memory; - a counter for how often the route becomes eligible, incremented whether or not the flag is on, so the arrival rate is measurable before enabling it; - one log line per solver result carrying the host and which route fired. The existing counter cannot say which, and the previous line was behind a diagnostic flag that is off in production. Tests cover the route firing on a real captured block page, staying out on a non-Cloudflare wall, both budget cases, the breaker verdict following the flag, and the pool default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Some origins answer a datacenter address with a plain block page and only serve
their challenge once the request arrives from a residential one. The decision to
route to the solver tier was taken from the ladder's body alone, before the
residential retry ran, so on these hosts it was always no: the retry then
uncovered the challenge, detected it, and threw the verdict away. The tier sat
configured and idle while every such request returned a block.
The residential retry's own verdict now opens that route.
Changes
renderer.cloak_after_egress(default off) lets the solver tier fire on achallenge the residential retry uncovered.
than the tier misbehaving, but only for the shape that is routed onward, so an
unrelated vendor wall still trips the breaker.
renderer.cloak_timeout_mswheneverthe remaining request time cannot fit a whole attempt, on both routes. The
sidecar refuses an attempt it cannot finish, so a partial budget wasted the
slot.
renderer.cloak_pool_sizemakes the tier's concurrency configurable insteadof a fixed 4. This is the box-wide throughput ceiling for the tier.
the flag is on, so the arrival rate is measurable before enabling it.
cloudflarerather than
generic_block: the previous arm keyed on a marker the templateno longer contains. Response label only; no tier reads it.
Behaviour with the flag off
Unchanged, except the vendor label on Cloudflare block pages. The error code
stays the same.
Deploy caveats
CRW_RENDERER__CLOAK_AFTER_EGRESSto enable, and onlyalongside a sidecar that can serve it.
CRW_RENDERER__CLOAK_POOL_SIZEmust be raised together with the sidecar's ownconcurrency and memory, or the engine hands out permits nothing can serve.
immediately, so those get slower before they get answers.
Verification
cargo test -p crw-renderer --features cdp,cloak --lib: 1005 passedcargo test -p crw-crawl --lib: 464 passedcargo test -p crw-core --lib: 513 passedcargo clippy -p crw-renderer -p crw-crawl -p crw-core --all-targets -- -D warnings: cleancargo fmt --check: cleanNew tests cover the route firing on a real captured block page, staying out on a
non-Cloudflare wall, both budget cases, the breaker verdict following the flag,
and the pool default. The block-page fixture is a real capture with the host, ray
ids and origin address scrubbed.
Not yet verified end to end against a live wall; that needs the paired sidecar
change and a staging run.