From 641b61578f572fd33c1054ef61ca4b2388d08519 Mon Sep 17 00:00:00 2001 From: Himadri Mishra Date: Fri, 4 Sep 2026 18:29:59 +0530 Subject: [PATCH 1/3] docs(search,extract): add measured-performance table and credit-semantics link MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deep Insights Sep-3 verification (F6, F1): docs.firecrawl.dev is the owned host agents actually reach, yet /features/search carried no measured performance while competitor pages cite numbers about Firecrawl search. Add a dated "Measured Performance" table with one independently-verified third-party figure (AIMultiple Agentic Search Benchmark, Agent Score 14.58 / 4.30 of 5), framed as ties within CI and pointing to firecrawl.dev/benchmarks for first-party numbers. Two other figures from the evidence packet (an "openbenchmarks" 70.3 ± 1.5 / 2nd-of-11 claim and DevDex 63.1% Recall@10) could not be attributed to a matching public source or are first-party, not independent, and were excluded rather than guessed — see report for detail. Also link the extract page's "each credit is worth 15 tokens" statement to /billing and to the in-docs 1-credit-per-page statement on /features/scrape, so credit semantics aren't a dead-end statement (F1). --- features/extract.mdx | 2 +- features/search.mdx | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/features/extract.mdx b/features/extract.mdx index 4676e8a79..c4dac1f6e 100644 --- a/features/extract.mdx +++ b/features/extract.mdx @@ -218,7 +218,7 @@ curl -X POST https://api.firecrawl.dev/v2/extract \ ## Billing and Usage Tracking -We've simplified billing so that Extract now uses credits, just like all of the other endpoints. Each credit is worth 15 tokens. +We've simplified billing so that Extract now uses credits, just like all of the other endpoints. Each credit is worth 15 tokens. See [Billing](/billing) for how credits map to plan pricing; scrape and crawl are billed at 1 credit per page ([Scrape cost](/features/scrape#scraping-a-url-with-firecrawl)). You can monitor Extract usage via the [dashboard](https://www.firecrawl.dev/app/extract). diff --git a/features/search.mdx b/features/search.mdx index a88ef371b..4c6460e6b 100644 --- a/features/search.mdx +++ b/features/search.mdx @@ -611,6 +611,18 @@ For more details about the scraping options, refer to the [Scrape Feature docume > Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions. +## Measured Performance + +Independent evaluators have benchmarked Firecrawl Search against other search APIs. The figures below are pulled directly from each evaluator's published results as of the date shown. + +| Metric | Value | Source | Date | n / ± | +| --- | --- | --- | --- | --- | +| Agent Score (Mean Relevant × Quality) | 14.58, rank 2 of 8 (Mean Relevant 4.30/5, Quality 3.39/5) | [AIMultiple — Agentic Search Benchmark](https://aimultiple.com/agentic-search) | 2026 | 95% CI 13.12–15.98, n=100 queries (10,000 bootstrap resamples) | + + +These are third-party measurements, each with its own harness, task set, and methodology — they are not directly comparable to one another or to Firecrawl's own benchmarks. AIMultiple's results place Firecrawl within the top statistical tier alongside Brave, Exa, and Parallel Search Pro (the ~0.3-point gaps between them fall inside the reported confidence intervals and are ties, not a ranking). Firecrawl's own first-party benchmark pages, including the open-source Developer Index evaluation, live at [firecrawl.dev/benchmarks](https://www.firecrawl.dev/benchmarks). + + ## Search feedback When a search result is useful or misses important content, submit feedback with `POST /v2/search/{jobId}/feedback`. The first feedback submission for a search job can refund 1 credit, subject to team limits, and helps improve Firecrawl search quality. See [Search Feedback](/api-reference/endpoint/search-feedback). From 28c6a31e91578fca9c3cc9d861960935c876167f Mon Sep 17 00:00:00 2001 From: Himadri Mishra Date: Fri, 4 Sep 2026 19:54:44 +0530 Subject: [PATCH 2/3] docs(search,extract): fix measured-performance date, tier framing, and credit link (F6/F1 Deep Insights Sep-3) Grounding review (Deep Insights Sep-3, D1-D3 report) found three defects in the prior commit: - The Date column read "2026", but AIMultiple's own Limitations section states the benchmark "reflects December 2025 snapshot only"; May 25, 2026 is only the article's update/retrieval date. Change to "Dec 2025 (source updated May 25, 2026)" and disclose that the 100 queries are AI/LLM-domain queries drawn from the source's own organic search traffic, carrying the source's own non-generalizability caveat. - The Note said the top-tier gaps "are ties, not a ranking", directly contradicting the table's own "rank 2 of 8" two lines above, and understated the tier span as "~0.3-point gaps" (actual span Brave 14.89 to Parallel Pro 14.21 = 0.68). Replaced with the source's own framing: point-estimate rank 2 of 8, no statistically significant gap under paired-bootstrap testing versus Brave, Exa, or Parallel Search Pro. - The prior commit linked "each credit is worth 15 tokens" to /billing and the scrape 1-credit-per-page statement only on the second (Billing and Usage Tracking) occurrence in features/extract.mdx. The first occurrence a reader hits, the top-of-page callout, was left unlinked. Added the same link there. mintlify broken-links and mintlify validate (Node 22) both pass with only pre-existing, unrelated warnings (missing zh/es/fr/ja/pt-BR snippet files); neither touches features/search.mdx or features/extract.mdx. Local only, no push. --- features/extract.mdx | 2 +- features/search.mdx | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/features/extract.mdx b/features/extract.mdx index c4dac1f6e..399645f51 100644 --- a/features/extract.mdx +++ b/features/extract.mdx @@ -38,7 +38,7 @@ import ExtractWithoutURLsCURL from "/snippets/v2/extract/without-urls/curl.mdx"; The `/extract` endpoint simplifies collecting structured data from any number of URLs or entire domains. Provide a list of URLs, optionally with wildcards (e.g., `example.com/*`), and a prompt or schema describing the information you want. Firecrawl handles the details of crawling, parsing, and collating large or small datasets. -We've simplified billing so that Extract now uses credits, just like all of the other endpoints. Each credit is worth 15 tokens. +We've simplified billing so that Extract now uses credits, just like all of the other endpoints. Each credit is worth 15 tokens. See [Billing](/billing) for how credits map to plan pricing; scrape and crawl are billed at 1 credit per page ([Scrape cost](/features/scrape#scraping-a-url-with-firecrawl)). ## Using `/extract` diff --git a/features/search.mdx b/features/search.mdx index 4c6460e6b..0c03d1908 100644 --- a/features/search.mdx +++ b/features/search.mdx @@ -617,10 +617,10 @@ Independent evaluators have benchmarked Firecrawl Search against other search AP | Metric | Value | Source | Date | n / ± | | --- | --- | --- | --- | --- | -| Agent Score (Mean Relevant × Quality) | 14.58, rank 2 of 8 (Mean Relevant 4.30/5, Quality 3.39/5) | [AIMultiple — Agentic Search Benchmark](https://aimultiple.com/agentic-search) | 2026 | 95% CI 13.12–15.98, n=100 queries (10,000 bootstrap resamples) | +| Agent Score (Mean Relevant × Quality) | 14.58, rank 2 of 8 (Mean Relevant 4.30/5, Quality 3.39/5) | [AIMultiple — Agentic Search Benchmark](https://aimultiple.com/agentic-search) | Dec 2025 (source updated May 25, 2026) | 95% CI 13.12–15.98, n=100 AI/LLM-domain queries (10,000 bootstrap resamples); the source notes results don't generalize to other domains | -These are third-party measurements, each with its own harness, task set, and methodology — they are not directly comparable to one another or to Firecrawl's own benchmarks. AIMultiple's results place Firecrawl within the top statistical tier alongside Brave, Exa, and Parallel Search Pro (the ~0.3-point gaps between them fall inside the reported confidence intervals and are ties, not a ranking). Firecrawl's own first-party benchmark pages, including the open-source Developer Index evaluation, live at [firecrawl.dev/benchmarks](https://www.firecrawl.dev/benchmarks). +These are third-party measurements, each with its own harness, task set, and methodology — they are not directly comparable to one another or to Firecrawl's own benchmarks. Point-estimate rank 2 of 8; paired-bootstrap testing found no statistically significant gap versus Brave, Exa, or Parallel Search Pro. Firecrawl's own first-party benchmark pages, including the open-source Developer Index evaluation, live at [firecrawl.dev/benchmarks](https://www.firecrawl.dev/benchmarks). ## Search feedback From 3c66fc32374341c719960c9602b2a0c149cf42a2 Mon Sep 17 00:00:00 2001 From: Himadri Mishra Date: Fri, 4 Sep 2026 20:35:29 +0530 Subject: [PATCH 3/3] docs(search): note the AIMultiple benchmark queries are the source's own traffic The 100 queries aren't an independent sample; AIMultiple drew them from its own site's organic search traffic in the AI/LLM domain. Per the archived source (review-grounding/aimultiple.md:422): "Source: Top 500 queries from AIMultiple.com organic search traffic (Dec 2024 to Jan 2025)" -- narrowed to the 100 used in the benchmark. Making that self-selection legible on the page, not just in the commit message, per the D1 nit in FINAL-REVIEW-docs-mcp.md. --- features/search.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/features/search.mdx b/features/search.mdx index 0c03d1908..6be990d80 100644 --- a/features/search.mdx +++ b/features/search.mdx @@ -617,7 +617,7 @@ Independent evaluators have benchmarked Firecrawl Search against other search AP | Metric | Value | Source | Date | n / ± | | --- | --- | --- | --- | --- | -| Agent Score (Mean Relevant × Quality) | 14.58, rank 2 of 8 (Mean Relevant 4.30/5, Quality 3.39/5) | [AIMultiple — Agentic Search Benchmark](https://aimultiple.com/agentic-search) | Dec 2025 (source updated May 25, 2026) | 95% CI 13.12–15.98, n=100 AI/LLM-domain queries (10,000 bootstrap resamples); the source notes results don't generalize to other domains | +| Agent Score (Mean Relevant × Quality) | 14.58, rank 2 of 8 (Mean Relevant 4.30/5, Quality 3.39/5) | [AIMultiple — Agentic Search Benchmark](https://aimultiple.com/agentic-search) | Dec 2025 (source updated May 25, 2026) | 95% CI 13.12–15.98, n=100 AI/LLM-domain queries drawn from AIMultiple's own organic search traffic (10,000 bootstrap resamples); the source notes results don't generalize to other domains | These are third-party measurements, each with its own harness, task set, and methodology — they are not directly comparable to one another or to Firecrawl's own benchmarks. Point-estimate rank 2 of 8; paired-bootstrap testing found no statistically significant gap versus Brave, Exa, or Parallel Search Pro. Firecrawl's own first-party benchmark pages, including the open-source Developer Index evaluation, live at [firecrawl.dev/benchmarks](https://www.firecrawl.dev/benchmarks).