Document crawl execution and result accounting as current behavior

[hmishra2250]

Summary

The crawl docs presented completed and total as a coverage receipt, told callers to page until
next is absent, listed three crawl statuses where Core returns four, and described
maxConcurrency: 1 as removing run-to-run variation. Each of those is contradicted by Core. This
branch rewrites the section to describe what the endpoints actually report, with a Core file and
line behind every behavioral claim, and states plainly that none of the records is a guarantee of
completeness.

Why (evidence)

Source reports: Developer-bank positioning read
(analysis/loop/run/DEVELOPER-POSITIONING-READ.txt, 2026-08-31) and Weekly Deep Insights
DI-2026-08-30-WEEKLY (2026-08-30).

Findings: the developer read's verdict, "The developer bank rejects Firecrawl on PROOF OF COVERAGE".
14 of 59 named traces carry a negative, decline or competitor-advantage verdict, 10 of them in the
interact bucket, including 9 of 10 interactive_traversal prompts. Hypothesis H1: a
documented coverage receipt on actions-bearing scrapes converts the decline. Adjudications: D7
TRUE (map is not exhaustive, in Firecrawl's own words), D8 STALE, D2 STALE, and D1, D3, D5
owner-required. OB-02-completeness-objection-is-first-party, grade A minus in the 2026-09-03
review: "quotable verbatim from Firecrawl's own docs, with no coverage contract alongside it."

Verified evidence: docs.firecrawl.dev/features/crawl "Crawl results may vary between runs of the
same configuration" and "By default, crawl ignores sublinks that are not children of the URL you
provide"; /features/map "This endpoint prioritizes speed, so it may not capture all website
links"; developer traces VDEV-201 ("All are convenient and all have the same silent-truncation
failure mode"), VDEV-210, VDEV-213, VDEV-222 ("less control over scroll termination, which is the
crux of your task"), VDEV-226 ("you need exhaustive link enumeration with an audit trail") and
VDEV-953 ("reintroduce exactly the silent-partial-result problem").

Core verification, read-only, each statement checked against source:

• apps/api/src/controllers/v2/crawl-status.ts:215-245 computes
  total = completed + active + queued + backlog. The stats map carries a distinct failed bucket
  that is never summed (apps/api/src/services/worker/nuq-fdb/queue.ts:1230-1251).
• crawl-status.ts:324-331:
  next = (outputBulkA.total ?? 0) > start + iteratedOver || outputBulkA.status !== "completed" ? <url> : undefined.
  The second disjunct emits next for every non-completed status, including terminal failed and
  cancelled.
• apps/api/src/controllers/v2/types.ts:1477-1505:
  status: "scraping" | "completed" | "failed" | "cancelled". cancelled is reachable via
  sc?.cancelled at crawl-status.ts:233-237.
• apps/api/src/controllers/v2/crawl-errors.ts:44-77 returns null for
  error?.code === "SCRAPE_RACED_REDIRECT_ERROR", plus a .filter(x => x.failedReason) and a
  mode !== "single_urls" drop.
• Concurrency enforcement is an active-job counter, not a serializer
  (apps/api/src/lib/concurrency-limit.ts:244-262,
  apps/api/src/services/worker/nuq-fdb/queue.ts:439-455). Surviving variation sources: bulk sitemap
  kickoff scrape-worker.ts:1296-1341, nested sitemap fan-out :1506-1512, priority dequeue
  :590-595, finish-time result ordering nuq-fdb/queue.ts:1291-1300. delay > 0 forcing
  concurrency to 1 is confirmed at concurrency-limit.ts:248-251 and queue-jobs.ts:381-384.
• types.ts:1110 and :1148: limit default is z.number().prefault(10000). crawl-status.ts:291:
  bytesLimit = 10485760.

Final verdict: APPROVE per the resolution addendum to
review-grounding/FINAL-REVIEW-docs-mcp.md (the review body records APPROVE-WITH-NITS; the acted-on
nit was resolved in commit 0ce9a8ed).

Changes

• features/crawl.mdx. The section ## Coverage and determinism becomes
  ## Execution and result accounting, and its opener states that the section describes what the
  endpoints currently report and that "None of these records is a guarantee of completeness." The
  strings "coverage receipt" and "proof of coverage" return 0 hits repository-wide.
• The total row becomes "completed plus the pages still in flight: active, queued, and
  backlogged. Failed pages are not counted.", with a new <Warning> explaining why
  completed == total is uninformative: a terminal crawl has active, queued and backlog at zero, so
  the counters converge whether or not pages failed, and failed pages are enumerated only by Get
  Crawl Errors.
• The status row lists four values including cancelled.
• The next row states it is also emitted whenever status is not completed, and the old
  <Warning> is replaced by a new ### Paging through results subsection. Behavior contract: the
  terminal condition is the status field, not the absence of next. The documented loop is poll
  GET /v2/crawl/{id} until status is one of completed, failed or cancelled; while next is
  present and the last page returned a non-empty data array, follow it; stop when next is absent
  or a page returns no new documents.
• The Get Crawl Errors lead-in changes from "is the list of" to "records", EXTERNAL_LINK is
  reworded off the undocumented field name, and a new <Note> states the list is not guaranteed to
  be a complete enumeration because some internal failure classes are filtered out before the
  response is built, and that EXTERNAL_LINK is returned today but is not yet part of the published
  schema. Per the "public docs, no internals" rule the note names the behavior, not the internal
  error constant.
• The maxConcurrency: 1 bullet no longer claims determinism. It cites this repo's own
  configuration table (features/crawl.mdx:325, "Maximum concurrent scrapes") and states that
  capping in-flight requests reduces timing-dependent interleaving but does not remove run-to-run
  variation, because sitemap discovery is enqueued outside the cap, nested sitemaps are fetched as
  independent jobs, and the returned data array is ordered by finish time rather than discovery
  order. The caveats cross-reference at :359 was retargeted and "how to make a run reproducible"
  softened to "more reproducible".
• The restated "returns 402 if credits don't cover the limit" clause was removed from the new scope
  section's limit bullet (commit 0ce9a8ed), because Core clamps rather than rejects:
  apps/api/src/controllers/v2/crawl.ts:248-252,
  finalCrawlerOptions.limit = Math.min(remainingCredits, finalCrawlerOptions.limit), with
  checkCreditsMiddleware named as the actual 402 source.
• features/map.mdx:142. "If you need exhaustive enumeration rather than a fast approximation"
  becomes "If you need more thorough recursive discovery with auditable result and error records
  rather than a fast approximation", pointing at the renamed section. The links scrape-format
  pointer is preserved verbatim.
• api-reference/endpoint/crawl-get.mdx and api-reference/endpoint/crawl-get-errors.mdx callouts
  rewritten to state the total exclusion and the incompleteness of the error list directly, rather
  than deferring to a "coverage receipt".
• The anchor #coverage-and-determinism was retargeted to #execution-and-result-accounting in all
  four referring locations: features/crawl.mdx:359, features/map.mdx:142,
  api-reference/endpoint/crawl-get.mdx:7, api-reference/endpoint/crawl-get-errors.mdx:7.
• openapi.json and its mirror api-reference/v2-openapi.json were deliberately not edited; they
  are recorded as Core/OpenAPI defects below.

Diffstat: 4 files changed, 89 insertions, 1 deletion.

Verification

Independent re-verification against Core during final review confirmed every statement above, and
separately confirmed the stale artifacts the branch declines to touch:
CrawlStatusResponseObj.status.description in api-reference/v2-openapi.json still reads "Can be
scraping, completed, or failed", and CrawlErrorsResponseObj has no code property, its
errors.items.properties being exactly id, timestamp, url, error. EXTERNAL_LINK appears in
the OpenAPI file exactly once, in the allowExternalLinks prose at line 1793, not in the errors
schema, so the page's claim that it is not yet published is accurate.

Grep negative controls: "coverage receipt" and "proof of coverage" return 0 hits across *.mdx,
*.md and *.json in the worktree; "exhaustive enumeration" is gone from map.mdx.

Anchors. All new intra-page anchors resolve: #execution-and-result-accounting (:245),
#paging-through-results (:269), #failed-and-blocked-pages (:283),
#configuration-reference (:323), #event-types (:208), plus
/features/scrape#scrape-formats (## Scrape Formats at :138) and /rate-limits. The renamed
anchor introduced no dangling references: all four in-repo links were retargeted in the same commit,
and no localized tree referenced it.

mintlify broken-links under Node 22 (the default Node 26.4.0 is rejected by the CLI): 208 broken
links repo-wide, all pre-existing, and zero in any file this branch touches. Filtering the report for
features/crawl.mdx, features/map.mdx, api-reference/endpoint/crawl-get.mdx and
api-reference/endpoint/crawl-get-errors.mdx returns no entries. The only non-localized English
files in the report are unrelated and pre-existing: api-reference/v1-endpoint/llmstxt.mdx,
api-reference/v2-introduction.mdx, features/alpha/llmstxt.mdx. mint validate: the same 17
pre-existing locale snippet warnings, no touched file.

Core was not modified. No localized file was touched.

Not in this PR

A Core PR, drafted but not filed. Title:
fix(api): correct crawl status/errors OpenAPI schema and drop silent error-class filter. Scope:
apps/api/openapi.json (mirrored to firecrawl-docs/api-reference/v2-openapi.json) and
apps/api/src/controllers/v2/crawl-errors.ts. Five items:

1. Stale status enum at apps/api/openapi.json:2810-2813: three values, no enum, while Core
   returns four. The same schema object contradicts itself two properties later, since
   completedAt.description names cancelled as a terminal state, and sibling schemas in the same
   file already carry cancelled in a real enum. Fix: add
   "enum": ["scraping", "completed", "failed", "cancelled"].
2. CrawlStatusResponseObj.total.description says "The total number of pages that were attempted to
   be crawled", which is wrong. Fix: describe it as completed plus in-flight, excluding failed. v1 is
   byte-identical at apps/api/src/controllers/v1/crawl-status.ts:205-222 and needs the same
   treatment.
3. CrawlErrorsResponseObj omits code, which the response type declares (types.ts:1520-1531) and
   the controller populates. Clients rely on EXTERNAL_LINK, a field the spec denies exists.
4. crawl-errors.ts:44-77 silently drops SCRAPE_RACED_REDIRECT_ERROR. Combined with item 2, such a
   page appears in neither the counters nor the error list. Preference order: stop filtering; or move
   those URLs into a third explicitly named array; or at minimum document the omission.
5. features/crawl.mdx:68, already on origin/main and left untouched here, claims a 402 rejection
   when remaining credits cannot cover the limit. Core clamps instead
   (crawl.ts:248-252). Fix: correct the doc line once Core confirms the clamp is intended, or make
   Core reject if the clamp was unintentional.

Also noted for whoever picks that up: the DB-backed fallback branch (crawl-errors.ts:78-159, used
once the Redis crawl record has expired) derives timestamp from created_at rather than finish
time, so timestamp semantics differ between the two paths.

Remaining docs nits, not acted on: "auditable" in map.mdx is slightly warmer than the same
branch's own Note admitting the error list is incomplete; and the new scope section restates the
limit default of 10000, which also appears at line 68 and in the configuration table.

Out of scope entirely: /extract and scrape actions have no documented partial-result signal
anywhere (developer read D1, D3, D5). That is a product decision, not a docs one.

Links

Files changed:

• features/crawl.mdx
• features/map.mdx
• api-reference/endpoint/crawl-get.mdx
• api-reference/endpoint/crawl-get-errors.mdx

Evidence packet:
agent-experience-deepinsights-cleanroom/artifacts/deep-insights-sep3-verification-20260904/

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
firecrawl	🟢 Ready	View Preview	Sep 4, 2026, 4:55 PM

💡 Tip: Enable Automations to automatically generate PRs for you.