From 63cbd27902ff80d5dc58f7fb026415c12bbdaadc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 2 Sep 2026 13:18:34 +0000 Subject: [PATCH] Add agent quickstart docs for all SDK languages Create canonical agent quickstart files for Node.js, Python, Rust, Java, and Elixir covering search, scrape, and interact endpoints. Each file is generated from current SDK source and the v2 OpenAPI spec. SDK versions referenced: - Node.js (firecrawl): 4.38.0 - Python (firecrawl-py): 4.41.0 - Rust (firecrawl): 2.18.0 - Java (firecrawl-java): 1.17.0 - Elixir (firecrawl): 1.11.0 Navigation updated in docs.json under Build with AI tab. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_0142BEBtdpGgzuHXGCsf1RvA --- agent-quickstart/elixir.mdx | 198 ++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 219 +++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 206 +++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 206 +++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 233 ++++++++++++++++++++++++++++++++++++ docs.json | 12 +- 6 files changed, 1073 insertions(+), 1 deletion(-) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..5854553b0 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,198 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`:firecrawl` **v1.11.0**) and the v2 OpenAPI spec. Function names and parameter keys match the auto-generated SDK module. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.11"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass api_key per call: +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + [url: "https://example.com"], + api_key: "fc-your-api-key" +) +``` + +There is no client struct to instantiate. Auth is handled per-request via application config or the trailing `opts` keyword list. + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs code execution in a post-scrape browser session. Requires a scrape job ID. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` → `{:ok, %Req.Response{}} | {:error, exception}` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + sources: [:web], + limit: 5, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) + +web_results = res.body["data"]["web"] +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` (required) | Search query. Use `site:example.com` to limit to a domain. | +| `sources` | `list` | Which sources: `:web`, `:news`, `:images` (atoms or strings). | +| `categories` | `list` | Filter by category: `:developer`, `:research`, `:pdf` (atoms or strings). | +| `include_domains` | `list[string]` | Only include these domains. | +| `exclude_domains` | `list[string]` | Exclude these domains. | +| `limit` | `integer` | Max results to return. | +| `tbs` | `string` | Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). | +| `location` | `string` | Location for localized results. | +| `country` | `string` | ISO 3166-1 alpha-2 country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | Drop unscrappable URLs. | +| `highlights` | `boolean` | Return query-relevant text highlights. Server default: `true`. | +| `timeout` | `integer` | Request timeout in milliseconds. | +| `scrape_options` | `keyword list` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list[string]` | Enterprise options: `"zdr"`, `"anon"`. | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` → `{:ok, %Req.Response{}} | {:error, exception}` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true +) + +doc = res.body["data"] +IO.puts(doc["markdown"]) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` (required) | URL to scrape. | +| `formats` | `list` | Output formats: strings (`"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`) or maps (`%{type: "json", prompt: "..."}`, etc.). | +| `headers` | `map` | Custom request headers. | +| `include_tags` | `list[string]` | Include only these HTML tags. | +| `exclude_tags` | `list[string]` | Exclude these HTML tags. | +| `only_main_content` | `boolean` | Strip nav, footer, boilerplate. | +| `timeout` | `integer` | Timeout in milliseconds. | +| `wait_for` | `integer` | Wait for page to render (milliseconds). | +| `mobile` | `boolean` | Use a mobile viewport. | +| `parsers` | `list` | File parsing controls (e.g. `%{type: "pdf", mode: "auto", maxPages: 5}`). | +| `actions` | `list[map]` | Pre-scrape browser actions. | +| `location` | `keyword list` | Geo/language-aware scraping: `country:`, `languages:`. | +| `skip_tls_verification` | `boolean` | Skip TLS verification. | +| `remove_base64_images` | `boolean` | Drop base64 images from markdown. | +| `block_ads` | `boolean` | Block ads and cookie popups. | +| `proxy` | `:basic \| :enhanced \| :auto` | Proxy mode. | +| `max_age` | `integer` | Use cached data up to this age (milliseconds). | +| `min_age` | `integer` | Cache-only mode with minimum age (milliseconds). | +| `store_in_cache` | `boolean` | Cache the result. | +| `lockdown` | `boolean` | Serve only cached results. | +| `profile` | `keyword list` | Persistent browser profile: `name:`, `save_changes:`. | +| `zero_data_retention` | `boolean` | Enable zero data retention. | + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. Use for Playwright-style page manipulation after a scrape creates a session. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` → `{:ok, %Req.Response{}} | {:error, exception}` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = scrape_res.body["data"]["metadata"]["scrapeId"] + +# Code-based interaction +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) + +IO.puts(res.body["stdout"]) + +# Clean up +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `string` (positional) | Scrape job ID from response metadata. | +| `code` | `string` (required) | Code to execute in the browser session. | +| `language` | `:python \| :node \| :bash` | Runtime for code execution. | +| `timeout` | `integer` | Execution timeout in seconds. | + +The Elixir SDK exposes **code-based interactions only**. There is no `prompt` parameter (the SDK is auto-generated from the OpenAPI spec which lists `code` only). + +**Stop session:** `Firecrawl.stop_interactive_scrape_browser_session(job_id)` ends the browser session. + +## Notes + +- The Elixir SDK is auto-generated from the OpenAPI spec. Function names and parameter keys match the generated code. +- Every function has a bang (`!`) variant that raises on error: e.g. `search_and_scrape!`, `scrape_and_extract_from_url!`. +- Parameters are passed as snake_case keyword lists; the SDK converts them to camelCase JSON for the API. +- Enum values (proxy, language) are passed as atoms: `:basic`, `:node`, etc. +- Nested objects (location, scrape_options, profile) are passed as keyword lists. +- All functions return `{:ok, %Req.Response{}}` or `{:error, exception}`. The response body is the decoded JSON map. +- No SDK-level defaults are set; all defaults come from the server. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..b06c87bfe --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,219 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-java` **v1.17.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.17.0 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.17.0") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs code execution in a post-scrape browser session. Requires a scrape job ID. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web")) + .limit(5) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `List` | Which sources: `"web"`, `"news"`, `"images"`. | +| `options.categories` | `List` | Filter by category: `"developer"`, `"research"`, `"pdf"`, `"github"`. | +| `options.includeDomains` | `List` | Only include these domains. | +| `options.excludeDomains` | `List` | Exclude these domains. | +| `options.limit` | `Integer` | Max results to return. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location for localized results. | +| `options.ignoreInvalidURLs` | `Boolean` | Drop unscrappable URLs. | +| `options.highlights` | `Boolean` | Return query-relevant text highlights. Server default: `true`. | +| `options.timeout` | `Integer` | Request timeout in milliseconds. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | URL to scrape. | +| `options.formats` | `List` | Output formats: strings (`"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`) or typed objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). | +| `options.headers` | `Map` | Custom request headers. | +| `options.includeTags` | `List` | Include only these HTML tags. | +| `options.excludeTags` | `List` | Exclude these HTML tags. | +| `options.onlyMainContent` | `Boolean` | Strip nav, footer, boilerplate. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.waitFor` | `Integer` | Wait for page to render (milliseconds). | +| `options.mobile` | `Boolean` | Use a mobile viewport. | +| `options.parsers` | `List` | File parsing controls (e.g. `"pdf"` or `Map.of("type", "pdf", "maxPages", 5)`). | +| `options.actions` | `List>` | Pre-scrape browser actions. | +| `options.location` | `LocationConfig` | Geo/language-aware scraping via builder: `.country("US").languages(List.of("en-US"))`. | +| `options.skipTlsVerification` | `Boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `Boolean` | Drop base64 images from markdown. | +| `options.blockAds` | `Boolean` | Block ads and cookie popups. | +| `options.proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `options.maxAge` | `Long` | Use cached data up to this age (milliseconds). | +| `options.storeInCache` | `Boolean` | Cache the result. | +| `options.lockdown` | `Boolean` | Serve only cached results. | + +**Format helper types:** +- `JsonFormat.builder().prompt("...").schema(Map.of(...)).build()` — JSON extraction. +- `QuestionFormat.builder().question("...").build()` — question-answer extraction. +- `HighlightsFormat.builder().query("...").build()` — relevant source-text extraction. + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. Use for Playwright-style page manipulation after a scrape creates a session. + +### Preferred SDK method + +`client.interact(jobId, code)` or `client.interact(jobId, code, language, timeout)` → `BrowserExecuteResponse` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; +import com.firecrawl.models.BrowserDeleteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); +String jobId = (String) doc.getMetadata().get("scrapeId"); + +// Code-based interaction +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); + +// Clean up +BrowserDeleteResponse stopped = client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | Scrape job ID from `document.getMetadata().get("scrapeId")`. | +| `code` | `String` | Code to execute in the browser session. | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1-300). `null` uses API default (30s). | + +The Java SDK exposes **code-based interactions only**. There is no `prompt` parameter (unlike the JS and Python SDKs). + +**Stop session:** `client.stopInteractiveBrowser(jobId)` ends the browser session. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- Every sync method has an async variant returning `CompletableFuture` (e.g. `scrapeAsync`, `searchAsync`, `interactAsync`). +- All option classes use the builder pattern: `ScrapeOptions.builder()...build()`. +- `ScrapeOptions` supports `toBuilder()` for cloning and modifying. +- The client supports a keyless free tier (rate-limited per IP) when no API key is provided. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..74ea87183 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,206 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` **v4.38.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL env var +}); +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web"], + limit: 5, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `("web" \| "news" \| "images")[]` | Which source indexes to search. | +| `options.categories` | `("developer" \| "research" \| "pdf" \| "github")[]` | Filter results by category. | +| `options.includeDomains` | `string[]` | Only include these domains. Cannot combine with `excludeDomains`. | +| `options.excludeDomains` | `string[]` | Exclude these domains. Cannot combine with `includeDomains`. | +| `options.limit` | `number` | Max results to return. | +| `options.tbs` | `string` | Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). | +| `options.location` | `string` | Localized results (e.g. `"San Francisco,California,United States"`). | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped. | +| `options.highlights` | `boolean` | Return query-relevant text highlights. Defaults to `true` server-side. | +| `options.timeout` | `number` | Request timeout in milliseconds. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `options.enterprise` | `("default" \| "anon" \| "zdr")[]` | Enterprise search controls. | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats (see below). | +| `options.headers` | `Record` | Custom request headers. | +| `options.includeTags` | `string[]` | Include only these HTML tags. | +| `options.excludeTags` | `string[]` | Exclude these HTML tags. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, boilerplate. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.waitFor` | `number` | Wait for page to render (milliseconds). | +| `options.mobile` | `boolean` | Use a mobile viewport. | +| `options.parsers` | `("pdf" \| PDFParser)[]` | File parsing controls (e.g. `{ type: "pdf", mode: "auto", maxPages: 5 }`). | +| `options.actions` | `ActionOption[]` | Pre-scrape browser actions (click, wait, write, press, scroll, scrape, executeJavascript, screenshot, pdf). | +| `options.location` | `{ country?: string; languages?: string[] }` | Geo/language-aware scraping. | +| `options.skipTlsVerification` | `boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown. | +| `options.fastMode` | `boolean` | Faster scrapes with reduced fidelity. | +| `options.blockAds` | `boolean` | Block ads and cookie popups. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto"` | Proxy mode. | +| `options.maxAge` | `number` | Use cached data up to this age (milliseconds). | +| `options.storeInCache` | `boolean` | Cache the result. | +| `options.lockdown` | `boolean` | Serve only cached results; never make outbound request. | +| `options.profile` | `{ name: string; saveChanges?: boolean }` | Persistent browser profile across scrapes and interactions. | + +**Format options:** + +String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. + +Object formats: +- `{ type: "json", prompt?: string, schema?: object }` — at least one of `prompt` or `schema` required. +- `{ type: "question", question: string }` — question-answer extraction. +- `{ type: "highlights", query: string }` — relevant source-text extraction. +- `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }` +- `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }` — `modes` required. +- `{ type: "attributes", selectors: { selector: string, attribute: string }[] }` + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language instructions after a scrape creates a session. Requires `scrapeId` from `document.metadata.scrapeId`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId"); + +// Natural-language interaction +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +// Or code-based interaction +const codeResult = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); + +// Clean up +await client.stopInteraction(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to run in the browser session. | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Default: `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided. + +**Stop session:** `client.stopInteraction(jobId)` ends the browser session. Returns `{ success, sessionDurationMs?, creditsBilled?, error? }`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` / `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed in `formats` (for `json` or `changeTracking`) are auto-converted to JSON Schema by the SDK, and the return type narrows accordingly. +- The SDK auto-resumes scrapes for large documents (PDFs) that outlive the request window (up to 5 retries / 20 minutes). +- The package declares Node.js >= 22 in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..7c11f1711 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,206 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-py` **v4.41.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the v2 client. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python >= 3.8. + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrape_id` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web"], + limit=5, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | Search query. Use `site:example.com` to limit to a domain. | +| `sources` | `list[str]` | Which source indexes to search: `"web"`, `"news"`, `"images"`. | +| `categories` | `list[str]` | Filter by category: `"developer"`, `"research"`, `"pdf"`, `"github"`. | +| `include_domains` | `list[str]` | Only include these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude these domains. Cannot combine with `include_domains`. | +| `limit` | `int` | Max results. Default: `5` (SDK model default). | +| `tbs` | `str` | Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). | +| `location` | `str` | Localized results (plain string, not a `Location` object). | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `highlights` | `bool` | Return query-relevant text highlights. Defaults to `true` server-side. | +| `timeout` | `int` | Request timeout in milliseconds. Default: `300000`. | +| `scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list[str]` | Enterprise options: `"zdr"` (zero data retention), `"anon"` (anonymized). | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | URL to scrape. | +| `formats` | `list` | Output formats (see below). | +| `headers` | `dict[str, str]` | Custom request headers. | +| `include_tags` | `list[str]` | Include only these HTML tags. | +| `exclude_tags` | `list[str]` | Exclude these HTML tags. | +| `only_main_content` | `bool` | Strip nav, footer, boilerplate. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait for page to render (milliseconds). | +| `mobile` | `bool` | Use a mobile viewport. | +| `parsers` | `list` | File parsing controls (e.g. `{"type": "pdf", "mode": "auto", "max_pages": 5}`). | +| `actions` | `list[dict]` | Pre-scrape browser actions (click, wait, write, press, scroll, scrape, executeJavascript, screenshot, pdf). | +| `location` | `Location` | Geo/language-aware scraping. `Location(country="US", languages=["en-US"])`. | +| `skip_tls_verification` | `bool` | Skip TLS verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Use cached data up to this age (milliseconds). | +| `store_in_cache` | `bool` | Cache the result. | +| `lockdown` | `bool` | Serve only cached results; never make outbound request. | +| `profile` | `dict` | Persistent browser profile: `{"name": "my-session", "saveChanges": True}`. | + +**Format options:** + +String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"audio"`, `"video"`. + +Object formats: +- `{"type": "json", "prompt": "...", "schema": {...}}` — at least one of `prompt` or `schema` required. +- `{"type": "question", "question": "..."}` — question-answer extraction. +- `{"type": "highlights", "query": "..."}` — relevant source-text extraction. +- `{"type": "screenshot", "full_page": True, "quality": 80, "viewport": {"width": 1280, "height": 720}}` +- `{"type": "changeTracking", "modes": ["git-diff"], "tag": "..."}` — `modes` required. +- `{"type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}]}` + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language instructions after a scrape creates a session. Requires `scrape_id` from `document.metadata.scrape_id`. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id") + +# Natural-language interaction +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") + +# Or code-based interaction +code_result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) + +# Clean up +client.stop_interaction(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | Scrape job ID from `document.metadata.scrape_id`. | +| `code` | `str` | Code to run in the browser session (optional if `prompt` is set). | +| `prompt` | `str` | Natural-language instruction for the browser agent (keyword-only; optional if `code` is set). | +| `language` | `str` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided. + +**Stop session:** `client.stop_interaction(job_id)` ends the browser session. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `FirecrawlApp` is a direct alias for `Firecrawl`. +- `search()` location is a plain `str`, not a `Location` object (unlike `scrape()`). +- `SearchRequest` model defaults: `limit=5`, `timeout=300000`. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..5fecbcac7 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,233 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` crate **v2.18.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +cargo add firecrawl +``` + +Requires an async runtime (Tokio). + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web]), + limit: Some(5), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; + +if let Some(web) = results.data.web { + for item in web { + // item is SearchResultOrDocument::WebResult(...) or ::Document(...) + } +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `Vec` | Which sources: `Web`, `News`, `Images`. | +| `options.categories` | `Vec` | Filter by category: `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Vec` | Only include these domains. | +| `options.exclude_domains` | `Vec` | Exclude these domains. | +| `options.limit` | `u32` | Max results. Doc comment says default 5, max 20. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location for localized results. | +| `options.ignore_invalid_urls` | `bool` | Drop unscrappable URLs. | +| `options.highlights` | `bool` | Return query-relevant highlights. Defaults to `true`. | +| `options.timeout` | `u32` | Request timeout in milliseconds. | +| `options.scrape_options` | `ScrapeOptions` | Scrape each result (see Scrape parameters). | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | URL to scrape. | +| `options.formats` | `Vec` | Output formats (see below). | +| `options.headers` | `HashMap` | Custom request headers. | +| `options.include_tags` | `Vec` | Include only these HTML tags. | +| `options.exclude_tags` | `Vec` | Exclude these HTML tags. | +| `options.only_main_content` | `bool` | Strip nav, footer, boilerplate. | +| `options.timeout` | `u32` | Timeout in milliseconds. | +| `options.wait_for` | `u32` | Wait for page to render (milliseconds). | +| `options.mobile` | `bool` | Use a mobile viewport. | +| `options.parsers` | `Vec` | File parsing controls (e.g. `ParserConfig::Pdf { ... }`). | +| `options.actions` | `Vec` | Pre-scrape browser actions (Click, Wait, Write, Press, Scroll, Scrape, ExecuteJavascript, Screenshot, Pdf). | +| `options.location` | `LocationConfig` | Geo/language-aware scraping: `country`, `languages`. | +| `options.skip_tls_verification` | `bool` | Skip TLS verification. | +| `options.remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `options.fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `options.block_ads` | `bool` | Block ads and cookie popups. | +| `options.proxy` | `ProxyType` | Proxy mode: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `u32` | Use cached data up to this age (milliseconds). | +| `options.store_in_cache` | `bool` | Cache the result. | +| `options.lockdown` | `bool` | Serve only cached results. | +| `options.profile` | `ProfileConfig` | Persistent browser profile: `name`, `save_changes`. | +| `options.json_options` | `JsonOptions` | JSON extraction config: `schema`, `prompt`, `system_prompt`. | +| `options.screenshot_options` | `ScreenshotOptions` | Screenshot config: `full_page`, `quality`, `viewport`. | +| `options.change_tracking_options` | `ChangeTrackingOptions` | Change tracking: `modes` (`GitDiff`/`Json`), `schema`, `prompt`, `tag`. | +| `options.attribute_selectors` | `Vec` | Attribute extraction: `selector`, `attribute`. | + +**Format enum values:** + +Simple: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. + +Object variants: `Question(QuestionFormat { question })`, `Highlights(HighlightsFormat { query })`. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for code execution or natural-language instructions after a scrape creates a session. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +// Natural-language interaction +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +// Or code-based interaction +let code_result = client + .interact(job_id, ScrapeExecuteOptions { + code: Some("console.log(await page.title());".to_string()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(60), + ..Default::default() + }) + .await?; + +// Clean up +client.stop_interaction(job_id).await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | Scrape job ID from document metadata. | +| `options.code` | `Option` | Code to run in the browser session. | +| `options.prompt` | `Option` | Natural-language instruction for the browser agent. | +| `options.language` | `ScrapeExecuteLanguage` | Runtime: `Python`, `Node`, `Bash`. Default: `Node`. | +| `options.timeout` | `u32` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be non-empty (SDK returns `FirecrawlError::Misuse` otherwise). + +**Stop session:** `client.stop_interaction(job_id)` ends the browser session. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- All option structs derive `Default`; use struct-update syntax: `ScrapeOptions { formats: Some(vec![...]), ..Default::default() }`. +- The `options` parameter on `scrape` and `search` accepts `impl Into>`, so you can pass `None` or a bare struct. +- All methods are `async` and require a Tokio runtime. +- `search_and_scrape(query, limit)` is a convenience helper that returns `Vec`. +- All serialization uses camelCase for the API wire format. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index a9d0cccef..bc86f6516 100755 --- a/docs.json +++ b/docs.json @@ -611,6 +611,16 @@ } ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "AI Tools", "pages": [ @@ -5352,4 +5362,4 @@ } }, "theme": "aspen" -} \ No newline at end of file +}