From 25d22436c632b797ef9f30cc6b785757b89690f0 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 3 Sep 2026 13:15:16 +0000 Subject: [PATCH] Add agent quickstart docs for all SDK languages Create canonical agent quickstart files for Node.js, Python, Rust, Java, and Elixir covering search, scrape, and interact endpoints. Each file includes install, auth, examples, and exhaustive parameter tables generated from the latest SDK source code. - Node.js: firecrawl v4.38.0 - Python: firecrawl-py v4.22.1 - Rust: firecrawl crate v2.18.0 - Java: firecrawl-java v1.17.0 - Elixir: firecrawl v1.11.0 Also adds Agent Quickstarts navigation group to docs.json under the Build with AI tab. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_01Dfgvg6Khaj6ASeD1M5WEVj --- agent-quickstart/elixir.mdx | 190 +++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 218 +++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 191 +++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 193 +++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 221 ++++++++++++++++++++++++++++++++++++ docs.json | 10 ++ 6 files changed, 1023 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..58c488bf2 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,190 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Elixir quickstart for agents. Generated from SDK source (`:firecrawl` **v1.11.0**) and the v2 OpenAPI spec. The Elixir client is auto-generated from the OpenAPI spec; function names and parameter keys reflect this. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.11"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass api_key per call: +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + [url: "https://example.com"], + api_key: "fc-your-api-key" +) +``` + +All functions accept an optional trailing `opts` keyword list supporting `:api_key` and `:base_url` overrides. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs code execution in a scrape-bound browser session. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query string. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` → `{:ok, map()} | {:error, Firecrawl.Error.t()}` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + sources: [:web, :news], + limit: 10, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) + +# Results in res["data"]["web"], res["data"]["news"], res["data"]["images"] +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` (required) | Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `list` | Sources: `:web`, `:news`, `:images` (atoms or strings). | +| `categories` | `list` | Filter results: `:developer`, `:research`, `:pdf` (atoms or strings). | +| `include_domains` | `list(string)` | Restrict results to these domains. | +| `exclude_domains` | `list(string)` | Exclude results from these domains. | +| `limit` | `integer` | Max results. | +| `tbs` | `string` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `location` | `string` | Location string for localized results. | +| `country` | `string` | ISO 3166-1 alpha-2 country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | Drop URLs that cannot be scraped. | +| `timeout` | `integer` | Request timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `scrape_options` | `keyword` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list(string)` | Enterprise options: `["zdr"]` for Zero Data Retention, `["anon"]` for anonymized search. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` → `{:ok, map()} | {:error, Firecrawl.Error.t()}` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000 +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` (required) | The URL to scrape. | +| `formats` | `list` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. Maps: `%{type: "json", prompt: ..., schema: ...}`, `%{type: "screenshot", fullPage: true}`, `%{type: "changeTracking", modes: ["git-diff"]}`. | +| `headers` | `map` | Custom HTTP headers. | +| `include_tags` | `list(string)` | Only include content from these HTML tags. | +| `exclude_tags` | `list(string)` | Exclude content from these HTML tags. | +| `only_main_content` | `boolean` | Strip nav, footer, and boilerplate. | +| `timeout` | `integer` | Timeout in milliseconds. Min 1000, default 60000, max 300000. | +| `wait_for` | `integer` | Wait for page to render (milliseconds). | +| `mobile` | `boolean` | Use a mobile viewport. | +| `parsers` | `list` | File parsers: `%{type: "pdf", mode: "auto", maxPages: 5}`. | +| `actions` | `list(map)` | Pre-scrape browser actions: `wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `keyword` | `[country: "US", languages: ["en-US"]]` for geo-aware scraping. | +| `skip_tls_verification` | `boolean` | Skip TLS verification. | +| `remove_base64_images` | `boolean` | Drop base64 images from markdown. | +| `block_ads` | `boolean` | Block ads and cookie popups. | +| `proxy` | `atom` | `:basic`, `:enhanced`, `:auto`. | +| `max_age` | `integer` | Accept cached data up to this age (milliseconds). | +| `min_age` | `integer` | Accept cached data only if at least this old (milliseconds). | +| `store_in_cache` | `boolean` | Cache the result. | +| `lockdown` | `boolean` | Serve only previously cached results; never make outbound requests. | +| `redact_pii` | `boolean` | Redact personally identifiable information. | +| `profile` | `keyword` | Persistent browser profile: `[name: "...", save_changes: true]`. | +| `audit_metadata` | `keyword` | User attribution for SIEM logging: `[username: "..."]`. | +| `zero_data_retention` | `boolean` | Enable zero data retention for this scrape. | + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. The Elixir SDK supports code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` → `{:ok, map()} | {:error, Firecrawl.Error.t()}` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = get_in(scrape_res, ["data", "metadata", "scrapeId"]) + +{:ok, result} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) + +# Stop the session when done +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `string` (required, path) | Scrape job ID from scrape response metadata. | +| `code` | `string` (required) | Code to execute in the browser session. | +| `language` | `atom` | Runtime: `:python`, `:node`, `:bash`. | +| `timeout` | `integer` | Execution timeout in seconds. | + +`Firecrawl.stop_interactive_scrape_browser_session(job_id)` ends the browser session. + +## Notes + +- The Elixir client is auto-generated from the OpenAPI spec; function names follow the spec operation IDs. +- Each function has a bang (`!`) variant that raises on error: e.g. `search_and_scrape!/2`, `scrape_and_extract_from_url!/2`. +- This SDK exposes **code-based interactions only** — there is no `prompt` parameter (unlike Node.js, Python, and Rust SDKs). +- No deprecated aliases exist in the Elixir SDK. +- The proxy parameter uses atoms (`:basic`, `:enhanced`, `:auto`) rather than strings. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..661f3cc09 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,218 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Java quickstart for agents. Generated from SDK source (`firecrawl-java` **v1.17.0**) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.17.0 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.17.0") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment (reads FIRECRAWL_API_KEY env var / firecrawl.apiKey system property): +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs code execution in a scrape-bound browser session. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query string. + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.SearchData; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +Results are grouped: `getWeb()`, `getNews()`, `getImages()` — each returns `List>` (may be null). + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `List` | Sources: `"web"`, `"news"`, `"images"`. | +| `options.categories` | `List` | Filter results: `"github"`, `"research"`, `"pdf"`. | +| `options.includeDomains` | `List` | Restrict results to these domains. | +| `options.excludeDomains` | `List` | Exclude results from these domains. | +| `options.limit` | `Integer` | Max results. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location string for localized results. | +| `options.ignoreInvalidURLs` | `Boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `Integer` | Request timeout in milliseconds. | +| `options.highlights` | `Boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | The URL to scrape. | +| `options.formats` | `List` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. Objects: `JsonFormat.builder().prompt(...).schema(...).build()`, or `Map.of("type", "screenshot", "fullPage", true)`. | +| `options.headers` | `Map` | Custom HTTP headers. | +| `options.includeTags` | `List` | Only include content from these HTML tags. | +| `options.excludeTags` | `List` | Exclude content from these HTML tags. | +| `options.onlyMainContent` | `Boolean` | Strip nav, footer, and boilerplate. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.waitFor` | `Integer` | Wait for page to render (milliseconds). | +| `options.mobile` | `Boolean` | Use a mobile viewport. | +| `options.parsers` | `List` | File parsers: `"pdf"` or `Map.of("type", "pdf", "maxPages", 5)`. | +| `options.actions` | `List>` | Pre-scrape browser actions: `wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`, `pdf`. | +| `options.location` | `LocationConfig` | `LocationConfig.builder().country("US").languages(List.of("en-US")).build()`. | +| `options.skipTlsVerification` | `Boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `Boolean` | Drop base64 images from markdown. | +| `options.blockAds` | `Boolean` | Block ads and cookie popups. | +| `options.proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. | +| `options.maxAge` | `Long` | Accept cached data up to this age (milliseconds). | +| `options.storeInCache` | `Boolean` | Cache the result. | +| `options.lockdown` | `Boolean` | Serve only previously cached results; never make outbound requests. | +| `options.redactPII` | `Boolean` | Redact personally identifiable information. | +| `options.auditMetadata` | `AuditMetadata` | User attribution for SIEM logging. Has field `username`. | + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. The Java SDK supports code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +- `client.interact(jobId, code)` — uses default language `"node"` +- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), or null for API default (30s) +- `client.interact(jobId, code, language, timeout, origin)` — with optional origin tag + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); +System.out.println(result.getStdout()); + +// Stop the session when done +client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | Scrape job ID from `document.getMetadata().get("scrapeId")`. | +| `code` | `String` | Code to execute in the browser session. | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Defaults to `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). Null uses API default (30s). | +| `origin` | `String` | Optional origin label for request attribution. | + +`client.stopInteractiveBrowser(jobId)` ends the browser session. Returns `BrowserDeleteResponse` with `isSuccess()`, `getSessionDurationMs()`, `getCreditsBilled()`, `getError()`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- The Java SDK exposes **code-based interactions only** — there is no `prompt` parameter on `interact` (unlike Node.js, Python, and Rust SDKs). +- All methods have async variants (e.g. `scrapeAsync`, `searchAsync`, `interactAsync`) returning `CompletableFuture`. +- Uses camelCase for all parameter names (Java convention). + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..427be390f --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,191 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Node.js quickstart for agents. Generated from SDK source (`firecrawl` **v4.38.0**) and the v2 OpenAPI spec. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL env var +}); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query string. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +Results are grouped by source: `results.web`, `results.news`, `results.images`. Do not access `results.data`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `("web" \| "news" \| "images")[]` | Which result sources to include. | +| `options.categories` | `("github" \| "developer" \| "research" \| "pdf")[]` | Filter web results by category. | +| `options.includeDomains` | `string[]` | Restrict results to these domains. Mutually exclusive with `excludeDomains`. | +| `options.excludeDomains` | `string[]` | Exclude results from these domains. Mutually exclusive with `includeDomains`. | +| `options.limit` | `number` | Max number of results. | +| `options.tbs` | `string` | Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). | +| `options.location` | `string` | Location string for localized results. | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `number` | Request timeout in milliseconds. | +| `options.highlights` | `boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `options.enterprise` | `("default" \| "anon" \| "zdr")[]` | Enterprise search options. `"zdr"` for Zero Data Retention. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, + waitFor: 1000, +}); +console.log(doc.markdown); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | The URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. Objects: `{ type: "json", prompt?, schema? }`, `{ type: "question", question }`, `{ type: "highlights", query }`, `{ type: "screenshot", fullPage?, quality?, viewport? }`, `{ type: "changeTracking", modes, schema?, prompt?, tag? }`, `{ type: "attributes", selectors: [{ selector, attribute }] }`. | +| `options.headers` | `Record` | Custom HTTP headers. | +| `options.includeTags` | `string[]` | Only include content from these HTML tags. | +| `options.excludeTags` | `string[]` | Exclude content from these HTML tags. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, and boilerplate. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.waitFor` | `number` | Wait for page to render (milliseconds). | +| `options.mobile` | `boolean` | Use a mobile viewport. | +| `options.parsers` | `(string \| PDFParser)[]` | File parsers. `"pdf"` or `{ type: "pdf", mode?: "fast" \| "auto" \| "ocr", maxPages?, pages?, blocks?, pageMarkers? }`. | +| `options.actions` | `ActionOption[]` | Pre-scrape browser actions: `wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`, `pdf`. | +| `options.location` | `{ country?: string, languages?: string[] }` | Geo/language-aware scraping. | +| `options.skipTlsVerification` | `boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown. | +| `options.fastMode` | `boolean` | Faster scrapes with reduced fidelity. | +| `options.blockAds` | `boolean` | Block ads and cookie popups. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy mode or custom URL. | +| `options.maxAge` | `number` | Accept cached data up to this age (milliseconds). Set to `0` to bypass index reuse. | +| `options.minAge` | `number` | Accept cached data only if at least this old (milliseconds). | +| `options.storeInCache` | `boolean` | Cache the result. | +| `options.lockdown` | `boolean` | Serve only previously cached results; never make outbound requests. | +| `options.profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser profile. | +| `options.redactPII` | `boolean \| RedactPIIOptions` | Redact personally identifiable information. | +| `options.auditMetadata` | `{ username: string }` | User attribution for SIEM logging. | + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language browser instructions. Requires a `scrapeId` from a prior scrape response. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId"); + +// Natural-language interaction +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +// Code-based interaction +const codeResult = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); + +// Stop the session when done +await client.stopInteraction(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to execute in the browser session. At least one of `code` or `prompt` required. | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. At least one of `code` or `prompt` required. | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Defaults to `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +`client.stopInteraction(jobId)` ends the browser session. Returns `{ success, sessionDurationMs?, creditsBilled?, error? }`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`; `scrapeUrl` → `scrape`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas in `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. +- `"json"` as a plain string in `formats` is rejected — use `{ type: "json", prompt?, schema? }`. +- The package declares **Node.js >= 22** in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..b05b92927 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,193 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Python quickstart for agents. Generated from SDK source (`firecrawl-py` **v4.22.1**) and the v2 OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +An async client is available as `AsyncFirecrawl`. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrape_id` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query string. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web", "news"], + limit=10, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +Results are grouped by source: `results.web`, `results.news`, `results.images`. Do not access `results.data`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `list[str \| Source]` | Which result sources to include: `"web"`, `"news"`, `"images"`. | +| `categories` | `list[str \| Category]` | Filter results: `"github"`, `"developer"`, `"research"`, `"pdf"`. | +| `include_domains` | `list[str]` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. Mutually exclusive with `include_domains`. | +| `limit` | `int` | Max results. Defaults to `5`. | +| `tbs` | `str` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `location` | `str` | Location string for localized results. | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `timeout` | `int` | Request timeout in milliseconds. Defaults to `300000`. | +| `highlights` | `bool` | Generate query-relevant highlights. Defaults to `True`. | +| `scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list[str]` | Enterprise options: `["zdr"]` for Zero Data Retention, `["anon"]` for anonymized search. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, +) +print(doc.markdown) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | The URL to scrape. | +| `formats` | `list` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"audio"`, `"video"`. Objects: `{"type": "json", "prompt": ..., "schema": ...}`, `{"type": "question", "question": ...}`, `{"type": "highlights", "query": ...}`, `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}`, `{"type": "changeTracking", "modes": [...], "tag": ...}`, `{"type": "attributes", "selectors": [{"selector": ..., "attribute": ...}]}`. | +| `headers` | `dict[str, str]` | Custom HTTP headers. | +| `include_tags` | `list[str]` | Only include content from these HTML tags. | +| `exclude_tags` | `list[str]` | Exclude content from these HTML tags. | +| `only_main_content` | `bool` | Strip nav, footer, and boilerplate. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait for page to render (milliseconds). | +| `mobile` | `bool` | Use a mobile viewport. | +| `parsers` | `list` | File parsers. `"pdf"` or `PDFParser(mode="fast" \| "auto" \| "ocr", max_pages=...)`. | +| `actions` | `list` | Pre-scrape browser actions: `WaitAction`, `ClickAction`, `WriteAction`, `PressAction`, `ScrollAction`, `ScreenshotAction`, `ScrapeAction`, `ExecuteJavascriptAction`, `PDFAction`. | +| `location` | `Location` | `Location(country="US", languages=["en-US"])` for geo-aware scraping. | +| `skip_tls_verification` | `bool` | Skip TLS verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Accept cached data up to this age (milliseconds). Set to `0` to bypass index reuse. | +| `store_in_cache` | `bool` | Cache the result. | +| `lockdown` | `bool` | Serve only previously cached results; never make outbound requests. | +| `profile` | `dict` | Persistent browser profile: `{"name": "...", "saveChanges": True}`. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. Has field `username`. | + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language browser instructions. Requires a `scrape_id` from a prior scrape response. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only. At least one of `code` or `prompt` must be provided. + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id") + +# Natural-language interaction +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") + +# Code-based interaction +result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) + +# Stop the session when done +client.stop_interaction(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | Scrape job ID from `document.metadata.scrape_id`. | +| `code` | `str` | Code to execute in the browser session. At least one of `code` or `prompt` required. | +| `prompt` | `str` | Natural-language instruction for the browser agent (keyword-only). At least one of `code` or `prompt` required. | +| `language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Defaults to `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. | + +`client.stop_interaction(job_id)` ends the browser session. Returns `BrowserDeleteResponse` with `success`, optional `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `FirecrawlApp` is a deprecated alias for `Firecrawl`; `AsyncFirecrawlApp` is a deprecated alias for `AsyncFirecrawl`. +- Format strings accept both camelCase (`"rawHtml"`) and snake_case (`"raw_html"`). +- `"json"` as a plain string in `formats` is allowed in Python (unlike Node.js), but an object form `{"type": "json", "prompt": ...}` is preferred for extraction. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..cc240e7b3 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,221 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Rust quickstart for agents. Generated from SDK source (`firecrawl` crate **v2.18.0**) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query string. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; + +// Results in results.data.web, results.data.news, results.data.images +``` + +A convenience helper `client.search_and_scrape(query, limit)` returns `Vec` directly. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `Vec` | Sources: `Web`, `News`, `Images`. | +| `options.categories` | `Vec` | Filter results: `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Vec` | Restrict results to these domains. | +| `options.exclude_domains` | `Vec` | Exclude results from these domains. | +| `options.limit` | `u32` | Max results. Default: 5, max: 20. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location string for localized results. | +| `options.ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `options.timeout` | `u32` | Request timeout in milliseconds. | +| `options.highlights` | `bool` | Generate query-relevant highlights. Defaults to `true`. | +| `options.scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | The URL to scrape. | +| `options.formats` | `Vec` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. | +| `options.headers` | `HashMap` | Custom HTTP headers. | +| `options.include_tags` | `Vec` | Only include content from these HTML tags. | +| `options.exclude_tags` | `Vec` | Exclude content from these HTML tags. | +| `options.only_main_content` | `bool` | Strip nav, footer, and boilerplate. | +| `options.timeout` | `u32` | Timeout in milliseconds. | +| `options.wait_for` | `u32` | Wait for page to render (milliseconds). | +| `options.mobile` | `bool` | Use a mobile viewport. | +| `options.parsers` | `Vec` | File parsers. `ParserConfig::Simple("pdf")` or `ParserConfig::Pdf { parser_type, mode?, max_pages?, pages?, blocks?, page_markers? }`. | +| `options.actions` | `Vec` | Pre-scrape browser actions: `Wait`, `Click`, `Write`, `Press`, `Scroll`, `Screenshot`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `options.location` | `LocationConfig` | `{ country, languages }` for geo-aware scraping. | +| `options.skip_tls_verification` | `bool` | Skip TLS verification. | +| `options.remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `options.fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `options.block_ads` | `bool` | Block ads and cookie popups. | +| `options.proxy` | `ProxyType` | `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `u32` | Accept cached data up to this age (milliseconds). | +| `options.min_age` | `u32` | Accept cached data only if at least this old (milliseconds). | +| `options.store_in_cache` | `bool` | Cache the result. | +| `options.lockdown` | `bool` | Serve only previously cached results; never make outbound requests. | +| `options.redact_pii` | `bool` | Redact personally identifiable information. | +| `options.audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. Has field `username`. | +| `options.profile` | `ProfileConfig` | Persistent browser profile: `{ name, save_changes? }`. | +| `options.json_options` | `JsonOptions` | JSON extraction: `{ schema?, system_prompt?, prompt?, check_prompt_injection? }`. | +| `options.screenshot_options` | `ScreenshotOptions` | Screenshot config: `{ full_page?, quality?, viewport? }`. | +| `options.change_tracking_options` | `ChangeTrackingOptions` | Change tracking: `{ modes? (GitDiff, Json), schema?, prompt?, tag? }`. | +| `options.attribute_selectors` | `Vec` | Attribute extraction: `{ selector, attribute }`. | + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language browser instructions. Requires a scrape job ID from a prior scrape. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +At least one of `code` or `prompt` must be provided, or the SDK returns `FirecrawlError::Misuse`. + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, ScrapeExecuteLanguage, Format}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.scrape_id.as_deref()) + .ok_or("Missing scrape_id")?; + +// Natural-language interaction +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +// Code-based interaction +let result = client + .interact(job_id, ScrapeExecuteOptions { + code: Some("console.log(await page.title());".to_string()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(60), + ..Default::default() + }) + .await?; + +// Stop the session when done +client.stop_interaction(job_id).await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | Scrape job ID from `document.metadata.scrape_id`. | +| `options.code` | `Option` | Code to execute. At least one of `code` or `prompt` required. | +| `options.prompt` | `Option` | Natural-language browser instruction. At least one of `code` or `prompt` required. | +| `options.language` | `ScrapeExecuteLanguage` | Runtime: `Python`, `Node`, `Bash`. Defaults to `Node`. | +| `options.timeout` | `u32` | Execution timeout in seconds. | + +`client.stop_interaction(job_id)` ends the browser session. Returns `ScrapeBrowserDeleteResponse` with `success`, optional `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- `ScrapeOptions` uses dedicated sub-structs (`json_options`, `screenshot_options`, `change_tracking_options`) for advanced format configuration. +- `search_and_scrape(query, limit)` is a convenience helper returning `Vec` from web results. +- All types export at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). +- Rust fields are `snake_case` but serialize to `camelCase` on the wire via serde. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index a9d0cccef..40b954f46 100755 --- a/docs.json +++ b/docs.json @@ -666,6 +666,16 @@ "agents/fire-1-extract" ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "Agentic Debugging", "pages": [