From 5833c081e67b4343bdaccb3cdb78b65c3e647897 Mon Sep 17 00:00:00 2001 From: Himadri Mishra Date: Fri, 4 Sep 2026 18:28:42 +0530 Subject: [PATCH 1/2] docs(crawl): document execution and result accounting as current behavior Rewrites the crawl coverage section to describe what the crawl endpoints report today, without implying the records are complete. - Renames "Coverage and determinism" to "Execution and result accounting" and drops every "coverage receipt" / "proof of coverage" framing. - States that `total` = completed + active + queued + backlog and excludes failed jobs, so `completed == total` on a finished crawl does not mean every discovered page succeeded (apps/api/src/controllers/v2/crawl-status.ts:215-245). - Documents the real `next` contract: `next` is also emitted whenever status is not `completed`, so it can be present on terminal failed and cancelled crawls. Removes the "only fully read once next is absent" loop guidance, which never terminates on a failed crawl, and gives the terminal condition as status in {completed, failed, cancelled} (crawl-status.ts:324-331). - Lists all four statuses including `cancelled` (apps/api/src/controllers/v2/types.ts:1477-1505). openapi.json:2810-2813 still lists three; that is tracked as a separate Core/OpenAPI fix and is deliberately not edited here. - map.mdx: "exhaustive enumeration" becomes "more thorough recursive discovery with auditable result and error records"; the pointer to the `links` scrape format is kept. - maxConcurrency: 1 caps in-flight requests per the configuration reference ("maximum concurrent scrapes") and does not remove run-to-run variation; the determinism claim is dropped. - Get Crawl Errors: notes the list may omit some failure classes (Core currently drops SCRAPE_RACED_REDIRECT_ERROR, apps/api/src/controllers/v2/crawl-errors.ts:44-77) and that the error code used for EXTERNAL_LINK is not yet in the published schema. Addresses OB-02 and the developer read D1-D8. Co-Authored-By: Claude Fable 5 --- api-reference/endpoint/crawl-get-errors.mdx | 4 ++ api-reference/endpoint/crawl-get.mdx | 4 ++ features/crawl.mdx | 80 ++++++++++++++++++++- features/map.mdx | 2 + 4 files changed, 89 insertions(+), 1 deletion(-) diff --git a/api-reference/endpoint/crawl-get-errors.mdx b/api-reference/endpoint/crawl-get-errors.mdx index 17cf3ce23..ee35d9a0c 100644 --- a/api-reference/endpoint/crawl-get-errors.mdx +++ b/api-reference/endpoint/crawl-get-errors.mdx @@ -3,4 +3,8 @@ title: 'Get Crawl Errors' openapi: '/api-reference/v2-openapi.json GET /crawl/{id}/errors' --- + + This endpoint records pages that did not make it into the crawl's `data` array: `errors` for scrapes Firecrawl failed, and `robotsBlocked` for URLs blocked by robots.txt. Failed pages appear nowhere in the [crawl status](/api-reference/endpoint/crawl-get) counters, so this is the only place to find them — but the list is not guaranteed complete, because some internal failure classes are filtered out before the response is built. See [Execution and result accounting](/features/crawl#execution-and-result-accounting) for how to read it alongside the status counters. + + > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions. diff --git a/api-reference/endpoint/crawl-get.mdx b/api-reference/endpoint/crawl-get.mdx index 25b3fe368..0efd0fabb 100644 --- a/api-reference/endpoint/crawl-get.mdx +++ b/api-reference/endpoint/crawl-get.mdx @@ -3,4 +3,8 @@ title: 'Get Crawl Status' openapi: '/api-reference/v2-openapi.json GET /crawl/{id}' --- + + `total` counts completed, active, queued, and backlogged pages and excludes failed ones, so `completed == total` on a finished crawl does not mean every discovered page succeeded. Pair the counters with [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) to see pages that were attempted but not returned, and see [Execution and result accounting](/features/crawl#execution-and-result-accounting) for how to read them together and when to stop paging with `next`. + + > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions. diff --git a/features/crawl.mdx b/features/crawl.mdx index a899a5c5d..23472ea6b 100644 --- a/features/crawl.mdx +++ b/features/crawl.mdx @@ -242,6 +242,84 @@ Every webhook request from Firecrawl includes an `X-Firecrawl-Signature` header For complete implementation examples in JavaScript and Python, see the [Webhook Security documentation](/webhooks/security). For comprehensive webhook documentation including detailed event payloads, payload structure, advanced configuration, and troubleshooting, see the [Webhooks documentation](/webhooks/overview). +## Execution and result accounting + +A crawl that finishes is not the same as a crawl that reached every page. This section describes what the crawl endpoints currently report about a run — the counters, the paging contract, the failure records, and the scope limits — so you can judge for yourself whether a run is complete enough to act on. None of these records is a guarantee of completeness. + +### Reading the status counters + +Every response from [Get Crawl Status](/api-reference/endpoint/crawl-get) (`GET /v2/crawl/{id}`) carries the counters that describe the run: + +| Field | Meaning | +|-------|---------| +| `status` | One of `scraping`, `completed`, `failed`, or `cancelled` | +| `total` | `completed` plus the pages still in flight: active, queued, and backlogged. **Failed pages are not counted.** | +| `completed` | The number of pages that have been successfully crawled | +| `creditsUsed` | Credits consumed by the crawl so far | +| `createdAt` / `completedAt` / `duration` | Start time, finish time (terminal states only), and elapsed seconds | +| `expiresAt` | When the results stop being retrievable from the API | +| `next` | URL for the next 10MB page of results. Also emitted whenever `status` is not `completed` — see [Paging through results](#paging-through-results). | + + + `completed == total` on a finished crawl does **not** mean every discovered page succeeded. Because `total` sums completed, active, queued, and backlogged pages and excludes failed ones, a terminal crawl always has `active`, `queued`, and `backlog` at zero — so the two counters converge whether or not pages failed. The status counters cannot tell you that anything failed. Failed pages are enumerated only by [Get Crawl Errors](#failed-and-blocked-pages). + + +The `data` array holds only pages Firecrawl successfully scraped. Pages that were attempted but never produced a result are not in `data` — read them from Get Crawl Errors, below. + +### Paging through results + +Responses are capped at 10MB. When a response is truncated, `next` carries the URL for the following page of results. + +`next` is not purely a "more data remains" signal: it is also emitted whenever `status` is not `completed`, so a terminal `failed` or `cancelled` crawl can return a `next` URL even though no further results exist. Do not use the absence of `next` as your loop's exit condition — on a failed or cancelled crawl it never disappears. + +The terminal condition is the status field. To read a run to the end: + +1. Poll `GET /v2/crawl/{id}` until `status` is one of `completed`, `failed`, or `cancelled`. +2. While `next` is present **and** the last page returned a non-empty `data` array, follow `next` to collect the remaining results. +3. Stop when `next` is absent, or when a page returns no new documents. + +The official SDKs handle this paging for you and return all results at once. + +### Failed and blocked pages + +[Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) (`GET /v2/crawl/{id}/errors`) records pages that did not make it into `data`. It returns two arrays: + +- `errors` — errored scrape jobs, each with `id`, `url`, `error` (the error message), and a `timestamp` of the failure. These are pages Firecrawl itself failed to scrape: network errors, timeouts, and similar. Links to an external site's homepage that were intentionally skipped are reported here with an error code of `EXTERNAL_LINK`. +- `robotsBlocked` — URLs that were attempted but blocked by the site's robots.txt. + + + This list is not guaranteed to be a complete enumeration of every failure: some internal failure classes are currently filtered out of `errors` before the response is built. Treat it as the record of failures Firecrawl reports, not as a proof that nothing else went wrong. The error code referenced above (`EXTERNAL_LINK`) is returned on error objects today but is not yet part of the published `GET /crawl/{id}/errors` schema; it is pending an API reference update. + + +A page where the target site returned an HTTP error such as 404 is *not* a crawl error: Firecrawl scraped it successfully, so it appears in `data` with the site's status code in `metadata.statusCode`. + +### What the crawler is scoped to reach + +Coverage is bounded by the scope parameters you set, all documented in the [configuration reference](#configuration-reference) and the [Crawl endpoint reference](/api-reference/endpoint/crawl-post): + +- **Children only by default.** Crawl ignores sublinks that are not children of the URL you provide. Use `crawlEntireDomain` for sibling and parent paths, `allowSubdomains` for subdomains, and `allowExternalLinks` to follow links off the domain. +- **`includePaths` / `excludePaths` match the URL pathname**, as regex patterns — not the full URL, and not query parameters. Set `regexOnFullURL: true` to match against the full URL including query strings instead. The starting URL is also checked against `includePaths`: if it does not match, the crawl may return 0 pages. +- **Sitemap mode.** With the default `sitemap: "include"`, URLs come from the sitemap plus recursive link discovery. `"skip"` uses HTML links only, so sitemap-only pages such as PDFs or deeply nested pages are missed. `"only"` crawls the sitemap plus the start URL and does not discover links from HTML. +- **`maxDiscoveryDepth`** caps how many link-discovery hops from the root are followed. Pages at the maximum depth are still scraped, but links found on them are not followed. +- **`limit`** caps the number of pages, and defaults to `10000`. The crawl endpoint checks up front that your remaining credits cover the `limit` and returns **402 (Payment Required)** if they do not. +- **`ignoreQueryParameters`** avoids re-scraping the same path with different query parameters. +- **robots.txt is respected** unless `ignoreRobotsTxt` is enabled (Enterprise only). + +`maxConcurrency` defaults to your team's concurrency limit, which is set by your plan — see [Rate limits](/rate-limits). + +### When results vary between runs + +Crawl results may vary between runs of the same configuration. Pages are scraped concurrently, so the order in which links are discovered depends on network timing and which pages finish loading first. This means different branches of a site may be explored to different extents near the depth boundary, especially at higher `maxDiscoveryDepth` values. + +To make a run more reproducible: + +- Set `maxConcurrency` to `1`. As the [configuration reference](#configuration-reference) states, `maxConcurrency` is the "maximum concurrent scrapes" — it caps how many requests are in flight at once. That reduces timing-dependent interleaving, but it does not remove run-to-run variation: sitemap discovery is enqueued outside the cap, nested sitemaps are fetched as independent jobs, and the returned `data` array is ordered by finish time rather than discovery order. Setting `delay` also forces concurrency to 1. +- Use `sitemap: "only"` if the site has a comprehensive sitemap, so the URL set comes from the sitemap rather than from link discovery. + +### Knowing when a crawl is done + +If you are not polling, the [webhook events](#event-types) tell you the same thing: `crawl.page` fires for each page successfully scraped, and `crawl.completed` (or `crawl.failed`) fires when the run ends. Job results stay retrievable from the API for 24 hours after completion; after that, view them in the [activity logs](https://www.firecrawl.dev/app/logs). + ## Configuration reference The full set of parameters available when submitting a crawl job: @@ -278,6 +356,6 @@ The full set of parameters available when submitting a crawl job: - **Result expiration**: Job results are available via the API for 24 hours after completion. After that, view results in the [activity logs](https://www.firecrawl.dev/app/logs). - **Crawl errors**: The `data` array contains pages Firecrawl successfully scraped. Use the [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) endpoint to retrieve pages that failed due to network errors, timeouts, or robots.txt blocks. - **External links**: With `allowExternalLinks: true`, the crawler follows links pointing off your domain and scrapes each linked page once — it does not then crawl the links found on those external pages. Links to an external site's **homepage** (a root URL with no path, e.g. `https://example.com/`) are intentionally skipped to avoid pulling in an entire unrelated site; these appear in [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) with the code `EXTERNAL_LINK`. Redirects are followed to their destination — including a link that resolves to its canonical URL (for example `http → https` or the `www` variant) — so only redirects that land on an external homepage are skipped. -- **Non-deterministic results**: Crawl results may vary between runs of the same configuration. Pages are scraped concurrently, so the order in which links are discovered depends on network timing and which pages finish loading first. This means different branches of a site may be explored to different extents near the depth boundary, especially at higher `maxDiscoveryDepth` values. To get more deterministic results, set `maxConcurrency` to `1` or use `sitemap: "only"` if the site has a comprehensive sitemap. +- **Non-deterministic results**: Crawl results may vary between runs of the same configuration, because pages are scraped concurrently and link-discovery order depends on network timing. See [Execution and result accounting](#execution-and-result-accounting) for what varies and how to make a run more reproducible. > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions. diff --git a/features/map.mdx b/features/map.mdx index 5fa0ebfcf..185b8ffe8 100644 --- a/features/map.mdx +++ b/features/map.mdx @@ -139,4 +139,6 @@ For more details about supported locations, refer to the [Proxies documentation] This endpoint prioritizes speed, so it may not capture all website links. It primarily relies on the website's sitemap, supplemented by cached crawl data and search engine results. For a more thorough and up-to-date list of URLs, consider using the [/crawl](/features/crawl) endpoint instead. +If you need more thorough recursive discovery with auditable result and error records rather than a fast approximation, use [/crawl](/features/crawl), which discovers pages by recursive link traversal as well as the sitemap and reports what it returned and what it failed to fetch — see [Execution and result accounting](/features/crawl#execution-and-result-accounting). For the links on one specific page, request the [`links` format](/features/scrape#scrape-formats) from [/scrape](/features/scrape). + > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions. \ No newline at end of file From 0ce9a8ed1b8a6e253540358bf800247fa17db4a6 Mon Sep 17 00:00:00 2001 From: Himadri Mishra Date: Fri, 4 Sep 2026 20:37:47 +0530 Subject: [PATCH 2/2] docs(crawl): drop unverifiable 402-on-limit restatement in new scope section Core clamps rather than rejects: apps/api/src/controllers/v2/crawl.ts:248-252 sets `finalCrawlerOptions.limit = Math.min(remainingCredits, finalCrawlerOptions.limit)`, with a comment naming checkCreditsMiddleware as the actual 402 source (insufficient credits to start at all, not limit-vs-credits). The new "What the crawler is scoped to reach" section restated the pre-existing (unverified) 402 claim from line 68 a second time; that restatement is removed here since this branch's whole thesis is documenting only what's confirmed against Core. Line 68 itself is left untouched per review guidance (pre-existing elsewhere on main). Added as item 5 to the Core PR proposal in scratchpad/product-prs/D3-crawl-coverage-v2.md so line 68 gets reconciled too. D3 nit 1, FINAL-REVIEW-docs-mcp.md. --- features/crawl.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/features/crawl.mdx b/features/crawl.mdx index 23472ea6b..fb0d92639 100644 --- a/features/crawl.mdx +++ b/features/crawl.mdx @@ -301,7 +301,7 @@ Coverage is bounded by the scope parameters you set, all documented in the [conf - **`includePaths` / `excludePaths` match the URL pathname**, as regex patterns — not the full URL, and not query parameters. Set `regexOnFullURL: true` to match against the full URL including query strings instead. The starting URL is also checked against `includePaths`: if it does not match, the crawl may return 0 pages. - **Sitemap mode.** With the default `sitemap: "include"`, URLs come from the sitemap plus recursive link discovery. `"skip"` uses HTML links only, so sitemap-only pages such as PDFs or deeply nested pages are missed. `"only"` crawls the sitemap plus the start URL and does not discover links from HTML. - **`maxDiscoveryDepth`** caps how many link-discovery hops from the root are followed. Pages at the maximum depth are still scraped, but links found on them are not followed. -- **`limit`** caps the number of pages, and defaults to `10000`. The crawl endpoint checks up front that your remaining credits cover the `limit` and returns **402 (Payment Required)** if they do not. +- **`limit`** caps the number of pages, and defaults to `10000`. - **`ignoreQueryParameters`** avoids re-scraping the same path with different query parameters. - **robots.txt is respected** unless `ignoreRobotsTxt` is enabled (Enterprise only).