fix(v2): match Firecrawl's success semantics, final URL and error envelope

[behramcelen]

A target that answers 404 / 403 / 429 / 500 is not a failed scrape. Firecrawl returns it as success: true with the code in metadata.statusCode. We returned success: false — and did it inconsistently, because the old gate (ScrapeData::http_error) only fires under ERROR_PAGE_MAX_TEXT (200 bytes), so the same site failed on a terse 404 and succeeded on a chatty one.

The rule

scrapeURL/index.ts, "Success factors":

const isGoodStatusCode = (s >= 200 && s < 300) || s === 304;
const hasRequiredOutput = isParsedImage || isLongEnough || !isGoodStatusCode;

!isGoodStatusCode is an OR term, so a bad status is sufficient to accept the result — even with an empty body. controllers/v2/scrape.ts then never reads metadata.statusCode at all.

In Firecrawl, success:false means their infrastructure failed — DNS, timeout, every engine dead, a policy block. It never means "the target said 404."

Captured from api.firecrawl.dev for every one of 401/403/404/429/500/503:

HTTP 200  {"success":true,"data":{"metadata":{"statusCode":404,"error":"Not Found",...}}}

What changed — /v2 only

#	Was	Now
1	4xx/5xx → success:false	success:true + metadata.statusCode + metadata.error
2	metadata.url aliased to sourceURL	the real post-redirect URL
3	no metadata.error	the bare reason phrase, exactly as captured
4	errorCode only	both code (Firecrawl's taxonomy) and errorCode (ours)
5	timeout 504, binary 422	408 / 500
6	200+empty → 200 {success:false}	500 SCRAPE_ALL_ENGINES_FAILED

#2 is a plain bug. adapters.rs set url: m.source_url.clone() — the same string as sourceURL — so a v2 caller had no way to learn where a redirect landed. FetchResult.final_url already existed and was already computed; it just never reached the wire.

#4 matters more than it looks. Both official SDKs read code. Every error code we emit today is invisible to firecrawl-py / firecrawl-js error handling. We now emit both keys — errorCode stays because the SaaS resolves it for RequestLog (upstreamErrorCode in api-handler.ts).

/v1 is untouched. http_error() is still what the native surface, crawl and batch consult; only this compat surface stopped asking.

Deliberately NOT matched

A vendor wall served with HTTP 200 still fails for us. Firecrawl hands back the Cloudflare challenge shell as the page's content, and we have already paid for that — see is_cdn_origin_error in crw-crawl:

sacg.me behind a dead origin was returned as success: true with "The initial connection between Cloudflare's network and the origin web server timed out" as its markdown, billed, and counted as a completed crawl page — for one customer, on the same source, since June.

On a 4xx/5xx the wall does not fail — the status already explains the page, which is the case this PR is about. Pinned by vendor_wall_on_a_200_still_fails and vendor_wall_on_a_403_is_a_document.

Two more rows are recorded and not matched: the live API 500s on a 3-line valid CSV (SCRAPE_RETRY_LIMIT (document_antibot)) and on recoverable malformed HTML (SCRAPE_ALL_ENGINES_FAILED), both of which we parse fine. Matching their contract is not the same as copying their extraction failures.

Billing

Unaffected. The SaaS refunds an engine 5xx and an envelope success:false alike (api-handler.ts:904 and :954), so a 4xx/5xx page still costs the same, and now costs the same on v1 and v2.

Two consequences worth a decision, neither blocking: RequestLog.outcome for a 404 moves from upstream_failed to success, so any error-rate metric shifts meaning; and the once-per-user onboarding mail flips from "first-error" to "first-crawl-success" for a user whose first call is a 404.

Verification

New conformance axis, conformance/mock_parity.py, driven at mock.fastcrw.com and diffed against responses captured from the real api.firecrawl.dev for the identical URLs:

before:  7/23 match Firecrawl v2
after:  21/21 match Firecrawl v2      (7 rows recorded, not asserted)

FIRECRAWL_API_KEY=fc-... ./run.sh capture mock   # refresh ground truth
CRW_URL=http://localhost:3000 ./run.sh parity    # diff against it

The existing golden corpus could not have caught any of this. All 11 fixtures target pages that return 200 and never redirect, so sourceURL == url holds in every one of them by accident and no error path is exercised at all. That is exactly how #2 shipped.

Plus 11 unit tests (v2_verdict, routes::v2::error) that run in CI with no network. The error tests drive the real into_response and read the real body — mutation-checked by deleting the body.code line, which fails four of them.

Two gaps found against a local stack WITH browsers, not fixed here

Recorded in conformance/FIRECRAWL-DIFF.md §4b/§5b, both measured:

Thin pages. /js/csr, /js/hydrate and /js/fetch all come back from Firecrawl as success:true with the fixture sentinel in the markdown (43, 23 and 17 chars). We answer 500 — chrome renders them correctly and structural_failure then discards the render as minimal_text on small page. Firecrawl's bar is trim().length > 0; ours is a heuristic built to catch JS shells, and loosening it is how a Cloudflare interstitial gets billed as content. A trade, not an oversight — but the whole /js/* group is affected.

/v2/scrape and /v2/crawl now disagree about the same URL. This change never touched state.rs, so crawl still turns http_error() into a block:

/v2/scrape  /status/404  ->  success:true, 1 credit,  HTTP tier only
/v2/crawl   /status/404  ->  blocked:1,    0 credits, escalated chrome + lightpanda

That is the "not billed on one surface and refunded on the other" split one level down, and the SaaS reads it through page-billing.ts, so it costs real money. state.rs is shared with /v1/crawl, which is why it stayed out of a v2-scoped change. It should be the next change.

Test status

cargo test --workspace --no-fail-fast matches origin/main: 8 pre-existing PDF-extraction failures, plus http_only::…invalid_user_agent which appears only under full parallel load and passes in isolation. Nothing in crw-renderer is touched by this PR.

Follow-ups

• Distinguish resolver failures from refused connections in the renderer's error chain, so /v2 can answer Firecrawl's 200-DNS vs 500-site split (§4b)
• crawl/batch status omits createdAt / completedAt / duration / warning
• an invalid crawl job id hits axum's Path<Uuid> rejection → plain-text 400, unparseable by any SDK
• /v2/map, /v2/search, /v2/extract error paths undiffed on both sides

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[behramcelen]

I have read the CLA Document and I hereby sign the CLA