From d33d688a3bbdf02295ec1b93151816710b82100a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 1 Sep 2026 13:22:04 +0000 Subject: [PATCH] docs: add agent quickstart files for Node.js, Python, Rust, Java, Elixir Add one canonical quickstart per SDK language in agent-quickstart/, covering search, scrape, and interact endpoints with confirmed parameters, examples, and language-specific notes. Generated from SDK source code and the v2 OpenAPI spec. Co-Authored-By: Claude --- agent-quickstart/elixir.mdx | 179 ++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 210 +++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 177 +++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 176 +++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 214 ++++++++++++++++++++++++++++++++++++ 5 files changed, 956 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..84911b459 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,179 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from the `firecrawl` Elixir SDK source and the v2 OpenAPI spec. Function names match the auto-generated module in `lib/firecrawl.ex`. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.11"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass api_key per call: +{:ok, res} = Firecrawl.search_and_scrape( + [query: "firecrawl webhooks"], + api_key: "fc-your-api-key" +) +``` + +Every function accepts `base_url` in opts (defaults to `"https://api.firecrawl.dev/v2"`). Additional opts are passed through to `Req`. + +## When To Use What + +- **search**: use when you start with a query and need discovery. Returns relevant pages you can then scrape or interact with. +- **scrape**: use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **interact**: use when the page needs clicks, form fills, or other browser actions after a scrape has created a session. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query. Constrain results to a site with `site:` in the query string (e.g. `site:docs.firecrawl.dev webhooks`). + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape(query: "site:docs.firecrawl.dev webhook retries") +``` + +Every function has a bang variant (`search_and_scrape!`) that raises on error. + +### Parameters + +Keyword list `params`: + +| Parameter | Type | Required | Description | +|---|---|---|---| +| `query` | `string` | **yes** | Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `list` | no | Which sources to search: `:web`, `:news`, `:images` (atoms or strings). | +| `categories` | `list` | no | Filter by category: `:developer`, `:research`, `:pdf` (atoms or strings). | +| `include_domains` | `list(string)` | no | Restrict results to these domains. | +| `exclude_domains` | `list(string)` | no | Exclude results from these domains. | +| `limit` | `integer` | no | Maximum number of results. | +| `tbs` | `string` | no | Time-based filter (e.g. `"qdr:d"` for past day). | +| `location` | `string` | no | Localized results. | +| `country` | `string` | no | ISO 3166-1 alpha-2 country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | no | Drop URLs that cannot be scraped. | +| `timeout` | `integer` | no | Request timeout in milliseconds. | +| `highlights` | `boolean` | no | Generate query-relevant highlights. | +| `scrape_options` | `keyword list` | no | Scrape each search result. See Scrape parameters. | +| `enterprise` | `list(string)` | no | Enterprise controls: `"zdr"` (zero data retention), `"anon"` (anonymized). | + +## Scrape + +### Why use it + +Retrieve structured content from a URL in one or more formats: markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"], + only_main_content: true +) +``` + +### Parameters + +Keyword list `params`: + +| Parameter | Type | Required | Description | +|---|---|---|---| +| `url` | `string` | **yes** | The URL to scrape. | +| `formats` | `list` | no | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. Maps: `%{type: "json", prompt: ..., schema: ...}`, `%{type: "screenshot", fullPage: true}`, `%{type: "changeTracking", modes: ["git-diff"]}`, `%{type: "attributes", selectors: [...]}`. | +| `headers` | `map` | no | Custom HTTP headers sent with the request. | +| `include_tags` | `list(string)` | no | Only include content from these HTML tags. | +| `exclude_tags` | `list(string)` | no | Exclude content from these HTML tags. | +| `only_main_content` | `boolean` | no | Strip nav, footer, and other boilerplate. | +| `timeout` | `integer` | no | Request timeout in milliseconds. | +| `wait_for` | `integer` | no | Wait for the page to render (milliseconds). | +| `mobile` | `boolean` | no | Emulate a mobile viewport. | +| `parsers` | `list` | no | File parsing controls: `"pdf"` or `%{type: "pdf", mode: "fast" \| "auto" \| "ocr", maxPages: n}`. | +| `actions` | `list(map)` | no | Browser actions before scraping: `%{type: "click", selector: ...}`, `%{type: "wait", milliseconds: ...}`, `%{type: "write", text: ...}`, `%{type: "press", key: ...}`, `%{type: "scroll", direction: "up" \| "down"}`, `%{type: "scrape"}`, `%{type: "executeJavascript", script: ...}`, `%{type: "pdf"}`. | +| `location` | `keyword list` | no | Geo targeting: `[country: "US", languages: ["en-US"]]`. | +| `skip_tls_verification` | `boolean` | no | Skip TLS certificate verification. | +| `remove_base64_images` | `boolean` | no | Drop base64 images from markdown output. | +| `block_ads` | `boolean` | no | Block ads and cookie popups. | +| `proxy` | `:basic \| :enhanced \| :auto` | no | Proxy mode. | +| `max_age` | `integer` | no | Use cached content if younger than this (milliseconds). | +| `min_age` | `integer` | no | Use cached content only if at least this old (milliseconds). | +| `store_in_cache` | `boolean` | no | Cache the scrape result. | +| `lockdown` | `boolean` | no | Serve only cached results; no outbound requests. | +| `redact_pii` | `boolean` | no | Redact personally identifiable information. | +| `audit_metadata` | `keyword list` | no | User attribution for SIEM logging: `[username: "..."]`. | +| `profile` | `keyword list` | no | Persistent browser profile: `[name: "...", save_changes: true]`. | +| `zero_data_retention` | `boolean` | no | End-to-end zero data retention. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape. Use for code execution in the browser session. Requires a scrape job ID from a prior scrape response. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = get_in(scrape_res.body, ["data", "metadata", "scrapeId"]) + +{:ok, result} = Firecrawl.interact_with_scrape_browser_session(job_id, + code: "console.log(await page.title());", + language: :node +) +``` + +To stop the session: `Firecrawl.stop_interactive_scrape_browser_session(job_id)` + +### Parameters + +| Parameter | Type | Required | Description | +|---|---|---|---| +| `job_id` | `string` | **yes** | Scrape job ID (path parameter). | +| `code` | `string` | **yes** | Code to execute in the browser session. | +| `language` | `:python \| :node \| :bash` | no | Runtime for code execution. | +| `timeout` | `integer` | no | Execution timeout in seconds. | + +## Notes + +- Function names are **auto-generated from the OpenAPI spec** and follow a verbose pattern: `scrape_and_extract_from_url`, `search_and_scrape`, `interact_with_scrape_browser_session`. Use them exactly as named. +- Every function has a **bang variant** (e.g. `search_and_scrape!`) that raises `Firecrawl.Error` on failure instead of returning `{:error, ...}`. +- Parameter names in the keyword list use **snake_case** (e.g. `only_main_content`, `scrape_options`), which the SDK converts to camelCase JSON keys before sending. +- The Elixir SDK's `interact_with_scrape_browser_session` only exposes the `code` parameter, not `prompt`. For natural-language browser instructions, use the HTTP API directly or another SDK. +- No deprecated aliases exist — the SDK is auto-generated from the OpenAPI spec. +- Proxy values use atoms (`:basic`, `:enhanced`, `:auto`) rather than strings. +- All functions accept `api_key` and `base_url` in the trailing `opts` keyword list. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..dbf4e8c34 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,210 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from the `firecrawl-java` SDK source and the v2 OpenAPI spec. Method names and parameter types match the `FirecrawlClient` public API. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.17.0 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.17.0") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment: FirecrawlClient.fromEnv() +``` + +Builder options: `apiKey` (falls back to `FIRECRAWL_API_KEY` env or `firecrawl.apiKey` system property), `apiUrl` (defaults to `https://api.firecrawl.dev`), `timeoutMs` (default `300000`), `maxRetries` (default `3`), `backoffFactor` (default `0.5`), `asyncExecutor` (default `ForkJoinPool.commonPool()`), `httpClient` (custom OkHttp instance). + +## When To Use What + +- **search**: use when you start with a query and need discovery. Returns relevant pages you can then scrape or interact with. +- **scrape**: use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **interact**: use when the page needs clicks, form fills, or other browser actions after a scrape has created a session. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query. Constrain results to a site with `site:` in the query string (e.g. `site:docs.firecrawl.dev webhooks`). + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries"); +List> web = results.getWeb(); +``` + +**Important:** `search()` returns `SearchData`. Access results via `getWeb()`, `getNews()`, `getImages()` — each returns `List>` (may be null). + +### Parameters + +`SearchOptions` built via `SearchOptions.builder()`: + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | Search query (first positional argument). Use `site:example.com` to scope. | +| `sources` | `List` | Which sources to search: `"web"`, `"news"`, `"images"`. | +| `categories` | `List` | Filter by category: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Restrict results to these domains. | +| `excludeDomains` | `List` | Exclude results from these domains. | +| `limit` | `Integer` | Maximum number of results. | +| `tbs` | `String` | Time-based filter (e.g. `"qdr:d"` for past day). | +| `location` | `String` | Localized results. | +| `ignoreInvalidURLs` | `Boolean` | Drop URLs that cannot be scraped. | +| `timeout` | `Integer` | Request timeout in milliseconds. | +| `highlights` | `Boolean` | Generate query-relevant highlights. Defaults to `true` server-side. | +| `scrapeOptions` | `ScrapeOptions` | Scrape each search result. See Scrape parameters. | +| `integration` | `String` | Integration identifier for server-side tracking. | + +## Scrape + +### Why use it + +Retrieve structured content from a URL in one or more formats: markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() +); + +System.out.println(doc.getMarkdown()); +``` + +Async variant: `client.scrapeAsync(url, options)` returns `CompletableFuture`. + +### Parameters + +`ScrapeOptions` built via `ScrapeOptions.builder()`: + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | The URL to scrape (first positional argument). | +| `formats` | `List` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Objects: `JsonFormat`, `QuestionFormat`, `HighlightsFormat`. | +| `headers` | `Map` | Custom HTTP headers sent with the request. | +| `includeTags` | `List` | Only include content from these HTML tags. | +| `excludeTags` | `List` | Exclude content from these HTML tags. | +| `onlyMainContent` | `Boolean` | Strip nav, footer, and other boilerplate. | +| `timeout` | `Integer` | Request timeout in milliseconds. | +| `waitFor` | `Integer` | Wait for the page to render (milliseconds). | +| `mobile` | `Boolean` | Emulate a mobile viewport. | +| `parsers` | `List` | File parsing controls (e.g. `"pdf"` or `PdfParser` with `maxPages`). | +| `actions` | `List>` | Browser actions before scraping. | +| `location` | `LocationConfig` | Geo targeting with `country` and `languages`. | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Drop base64 images from markdown output. | +| `blockAds` | `Boolean` | Block ads and cookie popups. | +| `proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. | +| `maxAge` | `Long` | Use cached content if younger than this (milliseconds). | +| `storeInCache` | `Boolean` | Cache the scrape result. | +| `lockdown` | `Boolean` | Serve only cached results; no outbound requests. | +| `redactPII` | `Boolean` | Redact personally identifiable information. | +| `auditMetadata` | `AuditMetadata` | User attribution for SIEM logging (has `username` field). | +| `integration` | `String` | Integration identifier for server-side tracking. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape (via `metadata.scrapeId`). Use for code execution in the browser session. The Java SDK exposes `code`-based interaction. + +### Preferred SDK method + +- `client.interact(jobId, code)` → `BrowserExecuteResponse` +- `client.interact(jobId, code, language, timeout)` → `BrowserExecuteResponse` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build() +); + +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact(jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); +``` + +To stop the session: `client.stopInteractiveBrowser(jobId)` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | Scrape job ID from `document.metadata.scrapeId`. | +| `code` | `String` | Code to execute in the browser session (e.g. Playwright `page` usage). | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Defaults to `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1-300). | + +Async variants: `client.interactAsync(...)` returns `CompletableFuture`. + +## Notes + +- All parameter names use **camelCase** (e.g. `onlyMainContent`, `scrapeOptions`, `ignoreInvalidURLs`). +- Options use the **builder pattern**: `ScrapeOptions.builder().formats(...).build()`. +- The Java SDK `interact` method exposes the `code` parameter directly. For natural-language prompts, use the HTTP API directly or another SDK. +- Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser`. Always use the modern names. +- Every sync method has an `Async` variant returning `CompletableFuture`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..64be05f09 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,177 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from the `firecrawl` JS/TS SDK source and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public surface. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, +}); +``` + +The constructor also accepts a plain string: `new Firecrawl("fc-...")`. Optional fields: `apiUrl` (defaults to `FIRECRAWL_API_URL` env or `https://api.firecrawl.dev`), `timeoutMs`, `maxRetries`, `backoffFactor`. + +## When To Use What + +- **search**: use when you start with a query and need discovery. Returns relevant pages you can then scrape or interact with. +- **scrape**: use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **interact**: use when the page needs clicks, form fills, or other browser actions after a scrape has created a session. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query. Constrain results to a site with `site:` in the query string (e.g. `site:docs.firecrawl.dev webhooks`). + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries"); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Important:** `search()` does not return `{ data: [...] }`. Access results via `results.web`, `results.news`, or `results.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `("web" \| "news" \| "images")[]` | Which sources to search. | +| `options.categories` | `("github" \| "research" \| "pdf" \| "developer")[]` | Filter results by category. | +| `options.includeDomains` | `string[]` | Restrict results to these domains. Cannot combine with `excludeDomains`. | +| `options.excludeDomains` | `string[]` | Exclude results from these domains. Cannot combine with `includeDomains`. | +| `options.limit` | `number` | Maximum number of results. | +| `options.tbs` | `string` | Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). | +| `options.location` | `string` | Localized results (e.g. `"San Francisco,California,United States"`). | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `number` | Request timeout in milliseconds. | +| `options.highlights` | `boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result with these options. See Scrape parameters. | +| `options.enterprise` | `("default" \| "anon" \| "zdr")[]` | Enterprise zero-data-retention or anonymized search. | +| `options.integration` | `string` | Integration identifier for server-side tracking. | + +## Scrape + +### Why use it + +Retrieve structured content from a URL in one or more formats: markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { + formats: ["markdown"], + onlyMainContent: true, +}); + +console.log(doc.markdown); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | The URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. Objects: `{ type: "json", prompt?, schema? }`, `{ type: "question", question }`, `{ type: "highlights", query }`, `{ type: "screenshot", fullPage?, quality?, viewport? }`, `{ type: "changeTracking", modes, schema?, prompt?, tag? }`, `{ type: "attributes", selectors: [{ selector, attribute }] }`. Note: plain string `"json"` is rejected by the SDK — use the object form. | +| `options.headers` | `Record` | Custom HTTP headers sent with the request. | +| `options.includeTags` | `string[]` | Only include content from these HTML tags. | +| `options.excludeTags` | `string[]` | Exclude content from these HTML tags. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, and other boilerplate. | +| `options.timeout` | `number` | Request timeout in milliseconds. | +| `options.waitFor` | `number` | Wait for the page to render (milliseconds). | +| `options.mobile` | `boolean` | Emulate a mobile viewport. | +| `options.parsers` | `(string \| { type: "pdf", mode?, maxPages? })[]` | File parsing controls. Use `"pdf"` or the object form for PDF options. | +| `options.actions` | `ActionOption[]` | Browser actions to run before scraping. Types: `wait` (milliseconds or selector), `click` (selector), `write` (text), `press` (key), `scroll` (direction: up/down), `screenshot`, `scrape`, `executeJavascript` (script), `pdf` (format, landscape, scale). | +| `options.location` | `{ country?: string, languages?: string[] }` | Geo and language targeting. | +| `options.skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown output. | +| `options.fastMode` | `boolean` | Faster scrapes with reduced fidelity. | +| `options.blockAds` | `boolean` | Block ads and cookie popups. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy mode or custom proxy URL. | +| `options.maxAge` | `number` | Use cached content if younger than this (milliseconds). `0` to bypass cache. | +| `options.minAge` | `number` | Use cached content only if at least this old (milliseconds). | +| `options.storeInCache` | `boolean` | Cache the scrape result. | +| `options.lockdown` | `boolean` | Serve only cached results; no outbound requests. | +| `options.redactPII` | `boolean \| RedactPIIOptions` | Redact personally identifiable information. | +| `options.auditMetadata` | `{ username: string }` | User attribution for SIEM logging. | +| `options.profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser profile shared across scrapes and interactions. | +| `options.autoResume` | `boolean` | SDK-only. Auto-resume when server signals continued processing (e.g. large PDFs). Defaults to `true`. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape (via `metadata.scrapeId`). Use for clicks, form fills, navigation, code execution, or natural-language browser instructions. Prefer `interact` over scrape-time `actions` for multi-step flows. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { + formats: ["markdown"], +}); + +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +console.log(result.output); +``` + +To stop the session: `client.stopInteraction(jobId)` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to execute in the browser session (e.g. Playwright `page` usage). One of `code` or `prompt` is required. | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. One of `code` or `prompt` is required. | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Defaults to `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +## Notes + +- All parameter names use **camelCase** (e.g. `onlyMainContent`, `scrapeOptions`, `ignoreInvalidURLs`). +- The SDK types file header states: "Public types for Firecrawl JS/TS SDK v2 (camelCase only)". +- Deprecated aliases exist for V1 compatibility: `scrapeUrl` → `scrape`, `scrapeExecute` → `interact`, `stopInteractiveBrowser`/`deleteScrapeBrowser` → `stopInteraction`, `crawlUrl` → `crawl`, `mapUrl` → `map`, and others. Always use the modern names. +- The `Firecrawl` default export extends `FirecrawlClient` (V2) and adds `.v1` for V1 backward compat. Use the top-level client. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..63bc4771b --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,176 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from the `firecrawl-py` SDK source and the v2 OpenAPI spec. Method names and parameters match the v2 client in `firecrawl/v2/client.py`. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +``` + +Optional constructor parameters: `api_url` (defaults to `https://api.firecrawl.dev`), `timeout` (seconds), `max_retries` (default `3`), `backoff_factor` (default `0.5`). An async variant is available: `AsyncFirecrawl`. + +## When To Use What + +- **search**: use when you start with a query and need discovery. Returns relevant pages you can then scrape or interact with. +- **scrape**: use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **interact**: use when the page needs clicks, form fills, or other browser actions after a scrape has created a session. Requires a `scrape_id` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query. Constrain results to a site with `site:` in the query string (e.g. `site:docs.firecrawl.dev webhooks`). + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search("site:docs.firecrawl.dev webhook retries") + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Important:** `search()` does not return `{ data: [...] }`. Access results via `results.web`, `results.news`, or `results.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `list[str \| Source]` | Which sources to search: `"web"`, `"news"`, `"images"`. | +| `categories` | `list[str \| Category]` | Filter by category: `"github"`, `"research"`, `"pdf"`, `"developer"`. | +| `include_domains` | `list[str]` | Restrict results to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. Cannot combine with `include_domains`. | +| `limit` | `int` | Maximum number of results. | +| `tbs` | `str` | Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). | +| `location` | `str` | Localized results (e.g. `"San Francisco,California,United States"`). | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `timeout` | `int` | Request timeout in milliseconds. | +| `highlights` | `bool` | Generate query-relevant highlights. | +| `scrape_options` | `ScrapeOptions` | Scrape each search result with these options. See Scrape parameters. | +| `enterprise` | `list[str]` | Enterprise controls: `"zdr"` (zero data retention), `"anon"` (anonymized). | +| `integration` | `str` | Integration identifier for server-side tracking. | + +## Scrape + +### Why use it + +Retrieve structured content from a URL in one or more formats: markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com", + formats=["markdown"], + only_main_content=True, +) + +print(doc.markdown) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | The URL to scrape. | +| `formats` | `list[FormatOption]` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"` / `"raw_html"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` / `"change_tracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. Objects: `{"type": "json", "prompt": ..., "schema": ...}`, `{"type": "question", "question": ...}`, `{"type": "highlights", "query": ...}`, `{"type": "screenshot", "fullPage": ..., "quality": ..., "viewport": ...}`, `{"type": "changeTracking", "modes": [...], "tag": ...}`, `{"type": "attributes", "selectors": [{"selector": ..., "attribute": ...}]}`. | +| `headers` | `dict[str, str]` | Custom HTTP headers sent with the request. | +| `include_tags` | `list[str]` | Only include content from these HTML tags. | +| `exclude_tags` | `list[str]` | Exclude content from these HTML tags. | +| `only_main_content` | `bool` | Strip nav, footer, and other boilerplate. | +| `timeout` | `int` | Request timeout in milliseconds. | +| `wait_for` | `int` | Wait for the page to render (milliseconds). | +| `mobile` | `bool` | Emulate a mobile viewport. | +| `parsers` | `list[str \| PDFParser]` | File parsing controls. Use `"pdf"` or `{"type": "pdf", "mode": "fast" \| "auto" \| "ocr", "maxPages": n}`. | +| `actions` | `list[Action]` | Browser actions to run before scraping. Types: `wait` (milliseconds or selector), `click` (selector), `write` (text), `press` (key), `scroll` (direction: up/down), `screenshot`, `scrape`, `executeJavascript` (script), `pdf` (format, landscape, scale). | +| `location` | `Location` | Geo and language targeting with `country` and `languages` fields. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown output. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Use cached content if younger than this (milliseconds). | +| `store_in_cache` | `bool` | Cache the scrape result. | +| `lockdown` | `bool` | Serve only cached results; no outbound requests. | +| `threat_protection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. | +| `profile` | `dict` | Persistent browser profile with `name` and optional `saveChanges`. | +| `auto_resume` | `bool` | SDK-only. Auto-resume when server signals continued processing. | +| `integration` | `str` | Integration identifier for server-side tracking. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape (via `metadata.scrape_id`). Use for clicks, form fills, navigation, code execution, or natural-language browser instructions. Prefer `interact` over scrape-time `actions` for multi-step flows. + +### Preferred SDK method + +`client.interact(job_id, code=None, **options)` → `BrowserExecuteResponse` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) + +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise ValueError("Missing scrape_id") + +result = client.interact( + job_id, + prompt="Click the pricing tab and summarize the plans.", +) + +print(result.output) +``` + +To stop the session: `client.stop_interaction(job_id)` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | Scrape job ID from `document.metadata.scrape_id`. | +| `code` | `str` | Code to execute in the browser session. One of `code` or `prompt` is required. | +| `prompt` | `str` | Natural-language instruction for the browser agent. One of `code` or `prompt` is required. | +| `language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Defaults to `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. | + +## Notes + +- All parameter names use **snake_case** (e.g. `only_main_content`, `scrape_options`, `ignore_invalid_urls`). +- The types module accepts both snake_case and camelCase in model construction (via Pydantic aliases), but method signatures are exclusively snake_case. +- Deprecated aliases: `scrape_url` → `scrape`, `scrape_execute` → `interact`, `stop_interactive_browser`/`delete_scrape_browser` → `stop_interaction`, `crawl_url` → `crawl`, `map_url` → `map`. Always use the modern names. +- `FirecrawlApp` is a class alias for `Firecrawl`; `AsyncFirecrawlApp` aliases `AsyncFirecrawl`. Use the short names. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..1ea08f235 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,214 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from the `firecrawl` Rust SDK source and the v2 OpenAPI spec. Method names and struct fields match the SDK public surface. + +## Install + +```bash +cargo add firecrawl +``` + +Crate: **`firecrawl`** on crates.io. + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +All public types are re-exported from `firecrawl::` (e.g. `firecrawl::Client`, `firecrawl::ScrapeOptions`). + +## When To Use What + +- **search**: use when you start with a query and need discovery. Returns relevant pages you can then scrape or interact with. +- **scrape**: use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **interact**: use when the page needs clicks, form fills, or other browser actions after a scrape has created a session. Requires a `scrape_id` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query. Constrain results to a site with `site:` in the query string (e.g. `site:docs.firecrawl.dev webhooks`). + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", None) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or ::Document + println!("{:?}", item); + } +} +``` + +### Parameters + +`SearchOptions` struct fields (all `Option`, use `..Default::default()` for defaults): + +| Field | Type | Description | +|---|---|---| +| `sources` | `Vec` | Which sources to search: `Web`, `News`, `Images`. | +| `categories` | `Vec` | Filter by category: `Github`, `Research`, `Pdf`. | +| `include_domains` | `Vec` | Restrict results to these domains. | +| `exclude_domains` | `Vec` | Exclude results from these domains. | +| `limit` | `u32` | Maximum number of results. | +| `tbs` | `String` | Time-based filter (e.g. `"qdr:d"` for past day). | +| `location` | `String` | Localized results. | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `timeout` | `u32` | Request timeout in milliseconds. | +| `highlights` | `bool` | Generate query-relevant highlights. Defaults to `true`. | +| `scrape_options` | `ScrapeOptions` | Scrape each search result with these options. See Scrape parameters. | +| `integration` | `String` | Integration identifier for server-side tracking. | + +### Return type + +`SearchResponse` contains `success: bool`, `data: SearchData`, `warning: Option`. `SearchData` has `web: Option>`, `news: Option>`, `images: Option>`. + +## Scrape + +### Why use it + +Retrieve structured content from a URL in one or more formats: markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }) + .await?; + +println!("{}", doc.markdown.unwrap_or_default()); +``` + +A convenience method is also available: `client.scrape_with_schema(url, schema, prompt)` for JSON extraction with a schema. + +### Parameters + +`ScrapeOptions` struct fields (all `Option`, use `..Default::default()` for defaults): + +| Field | Type | Description | +|---|---|---| +| `formats` | `Vec` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. Also `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. | +| `headers` | `HashMap` | Custom HTTP headers sent with the request. | +| `include_tags` | `Vec` | Only include content from these HTML tags. | +| `exclude_tags` | `Vec` | Exclude content from these HTML tags. | +| `only_main_content` | `bool` | Strip nav, footer, and other boilerplate. | +| `timeout` | `u32` | Request timeout in milliseconds. | +| `wait_for` | `u32` | Wait for the page to render (milliseconds). | +| `mobile` | `bool` | Emulate a mobile viewport. | +| `parsers` | `Vec` | File parsing controls. Use `ParserConfig::Simple("pdf".into())` or `ParserConfig::Pdf { ... }`. | +| `actions` | `Vec` | Browser actions before scraping: `Wait`, `Click`, `Write`, `Press`, `Scroll`, `Screenshot`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `location` | `LocationConfig` | Geo targeting with `country` and `languages` fields. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown output. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `ProxyType` | Proxy mode: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `max_age` | `u32` | Use cached content if younger than this (seconds). | +| `min_age` | `u32` | Use cached content only if at least this old (seconds). | +| `store_in_cache` | `bool` | Cache the scrape result. | +| `lockdown` | `bool` | Serve only cached results; no outbound requests. | +| `redact_pii` | `bool` | Redact personally identifiable information. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. | +| `profile` | `ProfileConfig` | Persistent browser profile with `name` and optional `save_changes`. | +| `json_options` | `JsonOptions` | JSON extraction options: `schema`, `prompt`, `system_prompt`, `check_prompt_injection`. | +| `screenshot_options` | `ScreenshotOptions` | Screenshot config: `full_page`, `quality`, `viewport`. | +| `change_tracking_options` | `ChangeTrackingOptions` | Change tracking: `modes` (GitDiff, Json), `schema`, `prompt`, `tag`. | +| `attribute_selectors` | `Vec` | Attribute extraction: each has `selector` and `attribute`. | +| `integration` | `String` | Integration identifier for server-side tracking. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape. Use for clicks, form fills, navigation, code execution, or natural-language browser instructions. Prefer `interact` over scrape-time `actions` for multi-step flows. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.scrape_id.as_ref()) + .expect("Missing scrapeId"); + +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +println!("{}", result.output.unwrap_or_default()); +``` + +To stop the session: `client.stop_interaction(job_id).await?` + +### Parameters + +`ScrapeExecuteOptions` struct fields (all `Option`, use `..Default::default()` for defaults): + +| Field | Type | Description | +|---|---|---| +| `code` | `String` | Code to execute in the browser session. One of `code` or `prompt` is required. | +| `prompt` | `String` | Natural-language instruction for the browser agent. One of `code` or `prompt` is required. | +| `language` | `ScrapeExecuteLanguage` | Runtime: `Python`, `Node`, `Bash`. Defaults to `Node`. | +| `timeout` | `u32` | Execution timeout in seconds. | + +## Notes + +- All struct fields use **snake_case** (e.g. `only_main_content`, `scrape_options`, `ignore_invalid_urls`). +- Use `..Default::default()` to fill unneeded fields in option structs. +- The `options` parameter in `scrape()` and `search()` accepts `None`, a bare struct, or `Some(struct)` via `impl Into>`. +- Deprecated aliases: `scrape_execute` → `interact`, `stop_interactive_browser`/`delete_scrape_browser` → `stop_interaction`. Always use the modern names. +- All methods are `async` and return `Result`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/src/v2/client.rs` +- `firecrawl/apps/rust-sdk/src/v2/search.rs` +- `firecrawl/apps/rust-sdk/src/v2/scrape.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`