Document crawl execution and result accounting as current behavior

api-reference/endpoint/crawl-get-errors.mdx

@@ -3,4 +3,8 @@ title: 'Get Crawl Errors'
 openapi: '/api-reference/v2-openapi.json GET /crawl/{id}/errors'
 ---
 
+<Info>
+  This endpoint records pages that did not make it into the crawl's `data` array: `errors` for scrapes Firecrawl failed, and `robotsBlocked` for URLs blocked by robots.txt. Failed pages appear nowhere in the [crawl status](/api-reference/endpoint/crawl-get) counters, so this is the only place to find them — but the list is not guaranteed complete, because some internal failure classes are filtered out before the response is built. See [Execution and result accounting](/features/crawl#execution-and-result-accounting) for how to read it alongside the status counters.
+</Info>
+
 > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.

api-reference/endpoint/crawl-get.mdx

@@ -3,4 +3,8 @@ title: 'Get Crawl Status'
 openapi: '/api-reference/v2-openapi.json GET /crawl/{id}'
 ---
 
+<Info>
+  `total` counts completed, active, queued, and backlogged pages and excludes failed ones, so `completed == total` on a finished crawl does not mean every discovered page succeeded. Pair the counters with [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) to see pages that were attempted but not returned, and see [Execution and result accounting](/features/crawl#execution-and-result-accounting) for how to read them together and when to stop paging with `next`.
+</Info>
+
 > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.

features/crawl.mdx

@@ -242,6 +242,84 @@ Every webhook request from Firecrawl includes an `X-Firecrawl-Signature` header
 
 For complete implementation examples in JavaScript and Python, see the [Webhook Security documentation](/webhooks/security). For comprehensive webhook documentation including detailed event payloads, payload structure, advanced configuration, and troubleshooting, see the [Webhooks documentation](/webhooks/overview).
 
+## Execution and result accounting
+
+A crawl that finishes is not the same as a crawl that reached every page. This section describes what the crawl endpoints currently report about a run — the counters, the paging contract, the failure records, and the scope limits — so you can judge for yourself whether a run is complete enough to act on. None of these records is a guarantee of completeness.
+
+### Reading the status counters
+
+Every response from [Get Crawl Status](/api-reference/endpoint/crawl-get) (`GET /v2/crawl/{id}`) carries the counters that describe the run:
+
+| Field | Meaning |
+|-------|---------|
+| `status` | One of `scraping`, `completed`, `failed`, or `cancelled` |
+| `total` | `completed` plus the pages still in flight: active, queued, and backlogged. **Failed pages are not counted.** |
+| `completed` | The number of pages that have been successfully crawled |
+| `creditsUsed` | Credits consumed by the crawl so far |
+| `createdAt` / `completedAt` / `duration` | Start time, finish time (terminal states only), and elapsed seconds |
+| `expiresAt` | When the results stop being retrievable from the API |
+| `next` | URL for the next 10MB page of results. Also emitted whenever `status` is not `completed` — see [Paging through results](#paging-through-results). |
+
+<Warning>
+  `completed == total` on a finished crawl does **not** mean every discovered page succeeded. Because `total` sums completed, active, queued, and backlogged pages and excludes failed ones, a terminal crawl always has `active`, `queued`, and `backlog` at zero — so the two counters converge whether or not pages failed. The status counters cannot tell you that anything failed. Failed pages are enumerated only by [Get Crawl Errors](#failed-and-blocked-pages).
+</Warning>
+
+The `data` array holds only pages Firecrawl successfully scraped. Pages that were attempted but never produced a result are not in `data` — read them from Get Crawl Errors, below.
+
+### Paging through results
+
+Responses are capped at 10MB. When a response is truncated, `next` carries the URL for the following page of results.
+
+`next` is not purely a "more data remains" signal: it is also emitted whenever `status` is not `completed`, so a terminal `failed` or `cancelled` crawl can return a `next` URL even though no further results exist. Do not use the absence of `next` as your loop's exit condition — on a failed or cancelled crawl it never disappears.
+
+The terminal condition is the status field. To read a run to the end:
+
+1. Poll `GET /v2/crawl/{id}` until `status` is one of `completed`, `failed`, or `cancelled`.
+2. While `next` is present **and** the last page returned a non-empty `data` array, follow `next` to collect the remaining results.
+3. Stop when `next` is absent, or when a page returns no new documents.
+
+The official SDKs handle this paging for you and return all results at once.
+
+### Failed and blocked pages
+
+[Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) (`GET /v2/crawl/{id}/errors`) records pages that did not make it into `data`. It returns two arrays:
+
+- `errors` — errored scrape jobs, each with `id`, `url`, `error` (the error message), and a `timestamp` of the failure. These are pages Firecrawl itself failed to scrape: network errors, timeouts, and similar. Links to an external site's homepage that were intentionally skipped are reported here with an error code of `EXTERNAL_LINK`.
+- `robotsBlocked` — URLs that were attempted but blocked by the site's robots.txt.
+
+<Note>
+  This list is not guaranteed to be a complete enumeration of every failure: some internal failure classes are currently filtered out of `errors` before the response is built. Treat it as the record of failures Firecrawl reports, not as a proof that nothing else went wrong. The error code referenced above (`EXTERNAL_LINK`) is returned on error objects today but is not yet part of the published `GET /crawl/{id}/errors` schema; it is pending an API reference update.
+</Note>
+
+A page where the target site returned an HTTP error such as 404 is *not* a crawl error: Firecrawl scraped it successfully, so it appears in `data` with the site's status code in `metadata.statusCode`.
+
+### What the crawler is scoped to reach
+
+Coverage is bounded by the scope parameters you set, all documented in the [configuration reference](#configuration-reference) and the [Crawl endpoint reference](/api-reference/endpoint/crawl-post):
+
+- **Children only by default.** Crawl ignores sublinks that are not children of the URL you provide. Use `crawlEntireDomain` for sibling and parent paths, `allowSubdomains` for subdomains, and `allowExternalLinks` to follow links off the domain.
+- **`includePaths` / `excludePaths` match the URL pathname**, as regex patterns — not the full URL, and not query parameters. Set `regexOnFullURL: true` to match against the full URL including query strings instead. The starting URL is also checked against `includePaths`: if it does not match, the crawl may return 0 pages.
+- **Sitemap mode.** With the default `sitemap: "include"`, URLs come from the sitemap plus recursive link discovery. `"skip"` uses HTML links only, so sitemap-only pages such as PDFs or deeply nested pages are missed. `"only"` crawls the sitemap plus the start URL and does not discover links from HTML.
+- **`maxDiscoveryDepth`** caps how many link-discovery hops from the root are followed. Pages at the maximum depth are still scraped, but links found on them are not followed.
+- **`limit`** caps the number of pages, and defaults to `10000`.
+- **`ignoreQueryParameters`** avoids re-scraping the same path with different query parameters.
+- **robots.txt is respected** unless `ignoreRobotsTxt` is enabled (Enterprise only).
+
+`maxConcurrency` defaults to your team's concurrency limit, which is set by your plan — see [Rate limits](/rate-limits).
+
+### When results vary between runs
+
+Crawl results may vary between runs of the same configuration. Pages are scraped concurrently, so the order in which links are discovered depends on network timing and which pages finish loading first. This means different branches of a site may be explored to different extents near the depth boundary, especially at higher `maxDiscoveryDepth` values.
+
+To make a run more reproducible:
+
+- Set `maxConcurrency` to `1`. As the [configuration reference](#configuration-reference) states, `maxConcurrency` is the "maximum concurrent scrapes" — it caps how many requests are in flight at once. That reduces timing-dependent interleaving, but it does not remove run-to-run variation: sitemap discovery is enqueued outside the cap, nested sitemaps are fetched as independent jobs, and the returned `data` array is ordered by finish time rather than discovery order. Setting `delay` also forces concurrency to 1.
+- Use `sitemap: "only"` if the site has a comprehensive sitemap, so the URL set comes from the sitemap rather than from link discovery.
+
+### Knowing when a crawl is done
+
+If you are not polling, the [webhook events](#event-types) tell you the same thing: `crawl.page` fires for each page successfully scraped, and `crawl.completed` (or `crawl.failed`) fires when the run ends. Job results stay retrievable from the API for 24 hours after completion; after that, view them in the [activity logs](https://www.firecrawl.dev/app/logs).
+
 ## Configuration reference
 
 The full set of parameters available when submitting a crawl job:
@@ -278,6 +356,6 @@ The full set of parameters available when submitting a crawl job:
 - **Result expiration**: Job results are available via the API for 24 hours after completion. After that, view results in the [activity logs](https://www.firecrawl.dev/app/logs).
 - **Crawl errors**: The `data` array contains pages Firecrawl successfully scraped. Use the [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) endpoint to retrieve pages that failed due to network errors, timeouts, or robots.txt blocks.
 - <a id="external-links"></a>**External links**: With `allowExternalLinks: true`, the crawler follows links pointing off your domain and scrapes each linked page once — it does not then crawl the links found on those external pages. Links to an external site's **homepage** (a root URL with no path, e.g. `https://example.com/`) are intentionally skipped to avoid pulling in an entire unrelated site; these appear in [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) with the code `EXTERNAL_LINK`. Redirects are followed to their destination — including a link that resolves to its canonical URL (for example `http → https` or the `www` variant) — so only redirects that land on an external homepage are skipped.
-- **Non-deterministic results**: Crawl results may vary between runs of the same configuration. Pages are scraped concurrently, so the order in which links are discovered depends on network timing and which pages finish loading first. This means different branches of a site may be explored to different extents near the depth boundary, especially at higher `maxDiscoveryDepth` values. To get more deterministic results, set `maxConcurrency` to `1` or use `sitemap: "only"` if the site has a comprehensive sitemap.
+- **Non-deterministic results**: Crawl results may vary between runs of the same configuration, because pages are scraped concurrently and link-discovery order depends on network timing. See [Execution and result accounting](#execution-and-result-accounting) for what varies and how to make a run more reproducible.
 
 > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.

features/map.mdx

@@ -139,4 +139,6 @@ For more details about supported locations, refer to the [Proxies documentation]
 
 This endpoint prioritizes speed, so it may not capture all website links. It primarily relies on the website's sitemap, supplemented by cached crawl data and search engine results. For a more thorough and up-to-date list of URLs, consider using the [/crawl](/features/crawl) endpoint instead.
 
+If you need more thorough recursive discovery with auditable result and error records rather than a fast approximation, use [/crawl](/features/crawl), which discovers pages by recursive link traversal as well as the sitemap and reports what it returned and what it failed to fetch — see [Execution and result accounting](/features/crawl#execution-and-result-accounting). For the links on one specific page, request the [`links` format](/features/scrape#scrape-formats) from [/scrape](/features/scrape).
+
 > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.