diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..5e8378fb2 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,164 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`:firecrawl` Hex package) and the v2 OpenAPI spec. Function names and parameters match the SDK public API. The Elixir client is OpenAPI-generated — function names and parameter keys come from the spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.10"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape([query: "example"], api_key: "fc-your-api-key") +``` + +All functions accept `api_key` and `base_url` (default `"https://api.firecrawl.dev/v2"`) as trailing `opts`. A nil/empty key falls back to the keyless free tier. + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` → `{:ok, Req.Response.t()} | {:error, Exception.t()}` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape(query: "site:docs.firecrawl.dev webhook retries") +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | **Required.** Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `list` | Sources: `:web`, `:news`, `:images` (atoms or strings). | +| `categories` | `list` | Filter: `:developer`, `:research`, `:pdf`. | +| `include_domains` | `list` of strings | Restrict results to these domains. | +| `exclude_domains` | `list` of strings | Exclude results from these domains. | +| `limit` | `integer` | Max number of results. | +| `tbs` | `string` | Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). | +| `location` | `string` | Localized search results. | +| `country` | `string` | ISO 3166-1 alpha-2 country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | Drop URLs that cannot be scraped. | +| `timeout` | `integer` | Request timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. Default: `true`. | +| `scrape_options` | `keyword list` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list` of strings | Enterprise: `"zdr"` for zero data retention, `"anon"` for anonymized. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` → `{:ok, Req.Response.t()} | {:error, Exception.t()}` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://docs.firecrawl.dev", + formats: ["markdown"] +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | **Required.** URL to scrape. | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. For JSON use `%{type: "json", prompt: "..."}`. For questions use `%{type: "question", question: "..."}`. For highlights use `%{type: "highlights", query: "..."}`. | +| `headers` | `map` | Custom HTTP headers. | +| `include_tags` | `list` of strings | HTML tags to include. | +| `exclude_tags` | `list` of strings | HTML tags to exclude. | +| `only_main_content` | `boolean` | Strip nav, footer, and boilerplate. | +| `timeout` | `integer` | Timeout in milliseconds. | +| `wait_for` | `integer` | Wait for page render (milliseconds). | +| `mobile` | `boolean` | Mobile viewport. | +| `parsers` | `list` | File parsers: `"pdf"` or `%{type: "pdf", mode: "auto", maxPages: 5}`. | +| `actions` | `list` of maps | Pre-scrape actions: `wait`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `keyword list` | Geo config: `country:`, `languages:`. | +| `skip_tls_verification` | `boolean` | Skip TLS verification. | +| `remove_base64_images` | `boolean` | Drop base64 images from markdown. | +| `block_ads` | `boolean` | Block ads and cookie popups. | +| `proxy` | atom or string | Proxy: `:basic`, `:enhanced`, `:auto`. | +| `max_age` | `integer` | Max age of cached content in ms. | +| `min_age` | `integer` | Min age of cached content in ms. Set to `1` to accept any cached data. | +| `store_in_cache` | `boolean` | Cache the result. | +| `profile` | `keyword list` | Persistent browser profile: `name:`, `save_changes:`. | +| `zero_data_retention` | `boolean` | Enable zero data retention. | + +## Interact + +### Why use it + +Execute code in the browser session tied to a prior scrape. The Elixir SDK supports code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` → `{:ok, Req.Response.t()} | {:error, Exception.t()}` + +### Example + +```elixir +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + "", + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) + +# When done: +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session("") +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `string` | **Required.** Scrape job ID (first positional argument). | +| `code` | `string` | **Required.** Code to execute in the browser session. | +| `language` | atom or string | Runtime: `:python`, `:node`, `:bash`. | +| `timeout` | `integer` | Execution timeout in seconds. | +| `origin` | `string` | Optional origin label for telemetry. | + +Stop the session with `Firecrawl.stop_interactive_scrape_browser_session(job_id)`. + +## Notes + +- Every function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- The Elixir client is OpenAPI-generated; function names come from the spec (e.g. `search_and_scrape`, `scrape_and_extract_from_url`). +- This SDK exposes code-based interactions only — no `prompt` parameter on `interact_with_scrape_browser_session` (unlike Node.js, Python, and Rust SDKs). +- Uses snake_case parameter keys. Maps use camelCase keys for format objects (e.g. `%{type: "json", prompt: "..."}`). +- No deprecated aliases exist — function names have been stable since the OpenAPI codegen. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..757d77e3d --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,193 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-java`) and the v2 OpenAPI spec. Method names and parameters match the SDK public API. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.16.0 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.16.0") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); +``` + +Builder options: `apiKey` (falls back to `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property), `apiUrl` (default `"https://api.firecrawl.dev"`), `timeoutMs` (default `300000`), `maxRetries` (default `3`), `backoffFactor` (default `0.5`). Shortcut: `FirecrawlClient.fromEnv()`. + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries"); +List> web = results.getWeb(); +``` + +Results are in `results.getWeb()`, `results.getNews()`, `results.getImages()`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `List` | Sources: `"web"`, `"news"`, `"images"` (strings or typed maps). | +| `options.categories` | `List` | Filter: `"github"`, `"research"`, `"pdf"`. | +| `options.includeDomains` | `List` | Restrict results to these domains. | +| `options.excludeDomains` | `List` | Exclude results from these domains. | +| `options.limit` | `Integer` | Max number of results. | +| `options.tbs` | `String` | Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). | +| `options.location` | `String` | Localized search results. | +| `options.ignoreInvalidURLs` | `Boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `Integer` | Request timeout in milliseconds. | +| `options.highlights` | `Boolean` | Generate query-relevant highlights. Default: `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `options.integration` | `String` | Integration identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; + +Document doc = client.scrape( + "https://docs.firecrawl.dev", + ScrapeOptions.builder().formats(List.of("markdown")).build() +); +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | URL to scrape. | +| `options.formats` | `List` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. For JSON use `JsonFormat.builder().prompt("...").build()`. For questions use `QuestionFormat`. For highlights use `HighlightsFormat`. | +| `options.headers` | `Map` | Custom HTTP headers. | +| `options.includeTags` | `List` | HTML tags to include. | +| `options.excludeTags` | `List` | HTML tags to exclude. | +| `options.onlyMainContent` | `Boolean` | Strip nav, footer, and boilerplate. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.waitFor` | `Integer` | Wait for page render (milliseconds). | +| `options.mobile` | `Boolean` | Mobile viewport. | +| `options.parsers` | `List` | File parsers: `"pdf"` or `PdfParser` with `maxPages`. | +| `options.actions` | `List>` | Pre-scrape actions: `wait`, `click`, `write`, `press`, `scroll`, `scrape`, `screenshot`, `executeJavascript`, `pdf`. | +| `options.location` | `LocationConfig` | Geo config: `country`, `languages`. | +| `options.skipTlsVerification` | `Boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `Boolean` | Drop base64 images from markdown. | +| `options.blockAds` | `Boolean` | Block ads and cookie popups. | +| `options.proxy` | `String` | Proxy: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. | +| `options.maxAge` | `Long` | Max age of cached content in ms. | +| `options.storeInCache` | `Boolean` | Cache the result. | + +## Interact + +### Why use it + +Execute code in the browser session tied to a prior scrape. The Java SDK supports code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`client.interact(jobId, code)` or `client.interact(jobId, code, language, timeout)` → `BrowserExecuteResponse` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); + +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); +System.out.println(result.getStdout()); + +// When done: +client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | Scrape job ID from `doc.getMetadata().get("scrapeId")`. | +| `code` | `String` | Code to execute in the browser session. | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). `null` uses API default (30s). | +| `origin` | `String` | Optional origin label (5th overload parameter). | + +Stop the session with `client.stopInteractiveBrowser(jobId)`. + +All methods have `*Async` variants returning `CompletableFuture`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- The Java SDK exposes code-based interactions only — no `prompt` parameter on `interact` (unlike Node.js, Python, and Rust SDKs). +- Uses camelCase parameter names matching the API wire format. +- `SearchData` results are accessed via `getWeb()`, `getNews()`, `getImages()`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..389520515 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,171 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` JS SDK) and the v2 OpenAPI spec. Method names and parameters match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, +}); +``` + +Constructor options: `apiKey` (falls back to `FIRECRAWL_API_KEY` env var), `apiUrl` (falls back to `FIRECRAWL_API_URL` or `https://api.firecrawl.dev`), `timeoutMs`, `maxRetries`, `backoffFactor`. + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries"); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Important:** `search()` does not return `{ data: [...] }`. Results are in `results.web`, `results.news`, `results.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `Array<"web" \| "news" \| "images">` | Which search sources to query. | +| `options.categories` | `Array<"developer" \| "research" \| "pdf">` | Filter results by category. | +| `options.includeDomains` | `string[]` | Restrict results to these domains. | +| `options.excludeDomains` | `string[]` | Exclude results from these domains. Cannot combine with `includeDomains`. | +| `options.limit` | `number` | Max number of results. | +| `options.tbs` | `string` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `string` | Localized search results. | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `number` | Request timeout in milliseconds. | +| `options.highlights` | `boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `options.enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise options: `"zdr"` for zero data retention, `"anon"` for anonymized. | +| `options.integration` | `string` | Integration identifier. | +| `options.origin` | `string` | Origin identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://docs.firecrawl.dev", { + formats: ["markdown"], +}); +console.log(doc.markdown); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. For JSON extraction use `{ type: "json", prompt?: string, schema?: object }`. For questions use `{ type: "question", question: string }`. For highlights use `{ type: "highlights", query: string }`. | +| `options.headers` | `Record` | Custom HTTP headers. | +| `options.includeTags` | `string[]` | HTML tags to include. | +| `options.excludeTags` | `string[]` | HTML tags to exclude. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, and boilerplate. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.waitFor` | `number` | Wait for the page to render (milliseconds). | +| `options.mobile` | `boolean` | Use a mobile viewport. | +| `options.parsers` | `Array` | File parsers. PDF parser: `{ type: "pdf", mode?: "fast" \| "auto" \| "ocr", maxPages?: number }`. | +| `options.actions` | `ActionOption[]` | Pre-scrape browser actions: `wait`, `click`, `write`, `press`, `scroll`, `scrape`, `screenshot`, `executeJavascript`, `pdf`. | +| `options.location` | `{ country?: string, languages?: string[] }` | Geo/language-aware scraping. | +| `options.skipTlsVerification` | `boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown. | +| `options.fastMode` | `boolean` | Faster scrapes, reduced fidelity. | +| `options.blockAds` | `boolean` | Block ads and cookie popups. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy control. | +| `options.maxAge` | `number` | Max age of cached content in ms. Set to `0` to bypass cache. | +| `options.storeInCache` | `boolean` | Cache the result. | +| `options.profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser profile. | +| `options.autoResume` | `boolean` | SDK-only. Auto-resume large documents. Set `false` to surface timeout immediately. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape. Use for clicks, form fills, code execution, or natural-language instructions. Requires a `scrapeId` from `document.metadata.scrapeId`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); +console.log(result.output); + +// When done: +await client.stopInteraction(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to execute in the browser session. At least one of `code` or `prompt` required. | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. At least one of `code` or `prompt` required. | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Defaults to `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +Stop the session with `client.stopInteraction(jobId)`. + +## Notes + +- Deprecated aliases: `scrapeUrl` → `scrape`, `scrapeExecute` → `interact`, `stopInteractiveBrowser` / `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client. V1 remains under `client.v1`. +- Plain string `"json"` in `formats` is rejected by the SDK — use `{ type: "json", prompt: "..." }`. +- `search()` returns `SearchData` with `.web`, `.news`, `.images` — not `.data`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..d193c7a01 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,167 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-py`) and the v2 OpenAPI spec. Method names and parameters match the SDK public API. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +``` + +Constructor options: `api_key` (falls back to `FIRECRAWL_API_KEY` env var), `api_url` (default `"https://api.firecrawl.dev"`), `timeout` (seconds), `max_retries` (default `3`), `backoff_factor` (default `0.5`). An async client is available as `AsyncFirecrawl`. + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrape_id` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search("site:docs.firecrawl.dev webhook retries") + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Important:** `search()` does not return `{ data: [...] }`. Results are in `results.web`, `results.news`, `results.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `list` | Sources to query: `"web"`, `"news"`, `"images"`. | +| `categories` | `list` | Filter by category: `"developer"`, `"research"`, `"pdf"`. | +| `include_domains` | `list[str]` | Restrict results to these domains. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. Mutually exclusive with `include_domains`. | +| `limit` | `int` | Max results. Default: `5`. | +| `tbs` | `str` | Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). | +| `location` | `str` | Localized search results. | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `timeout` | `int` | Request timeout in milliseconds. Default: `300000`. | +| `highlights` | `bool` | Generate query-relevant highlights. Default: `True`. | +| `scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list[str]` | Enterprise options: `"zdr"` for zero data retention, `"anon"` for anonymized. | +| `integration` | `str` | Integration identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape("https://docs.firecrawl.dev", formats=["markdown"]) +print(doc.markdown) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | URL to scrape. | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"` / `"raw_html"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` / `"change_tracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. For JSON use `{"type": "json", "prompt": "..."}`. For questions use `{"type": "question", "question": "..."}`. For highlights use `{"type": "highlights", "query": "..."}`. | +| `headers` | `dict` | Custom HTTP headers. | +| `include_tags` | `list[str]` | HTML tags to include. | +| `exclude_tags` | `list[str]` | HTML tags to exclude. | +| `only_main_content` | `bool` | Strip nav, footer, and boilerplate. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait for the page to render (milliseconds). | +| `mobile` | `bool` | Use a mobile viewport. | +| `parsers` | `list` | File parsers. PDF: `{"type": "pdf", "mode": "fast" \| "auto" \| "ocr", "max_pages": int}`. | +| `actions` | `list` | Pre-scrape browser actions: `wait`, `click`, `write`, `press`, `scroll`, `scrape`, `screenshot`, `executeJavascript`, `pdf`. | +| `location` | `Location` | Geo/language-aware scraping: `{"country": "US", "languages": ["en-US"]}`. | +| `skip_tls_verification` | `bool` | Skip TLS verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `fast_mode` | `bool` | Faster scrapes, reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Max age of cached content in ms. Set to `0` to bypass cache. | +| `store_in_cache` | `bool` | Cache the result. | +| `profile` | `dict` | Persistent browser profile: `{"name": "...", "saveChanges": True}`. | +| `auto_resume` | `bool` | Auto-resume large documents. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape. Use for clicks, form fills, code execution, or natural-language instructions. Requires a `scrape_id` from `doc.metadata.scrape_id`. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only. At least one of `code` or `prompt` must be non-empty. + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +print(result.output) + +# When done: +client.stop_interaction(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | Scrape job ID from `doc.metadata.scrape_id`. | +| `code` | `str` | Code to execute in the browser session. At least one of `code` or `prompt` required. | +| `prompt` | `str` | Natural-language instruction for the browser agent (keyword-only). At least one of `code` or `prompt` required. | +| `language` | `str` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. | + +Stop the session with `client.stop_interaction(job_id)`. + +## Notes + +- Deprecated aliases: `scrape_url` → `scrape`, `scrape_execute` → `interact`, `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- The top-level `Firecrawl` client exposes v2 methods directly. V1 remains under `client.v1`. +- `search()` returns `SearchData` with `.web`, `.news`, `.images` — not `.data`. +- Python uses snake_case parameter names (`only_main_content`, `wait_for`, etc.). + +## Source Of Truth + +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..7fb4389d5 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,206 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` crate) and the v2 OpenAPI spec. Method names and parameters match the SDK public API. + +## Install + +```bash +cargo add firecrawl +``` + +Crate: `firecrawl` on crates.io. + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +`Client::new(api_key)` connects to `https://api.firecrawl.dev`. `Client::new_selfhosted(api_url, api_key)` connects to a self-hosted instance. An empty key falls back to the keyless free tier. + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::Client; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", None) + .await?; + +if let Some(web) = results.data.web { + for item in web { + println!("{:?}", item); + } +} +``` + +Results are in `results.data.web`, `results.data.news`, `results.data.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `Vec` | Sources: `Web`, `News`, `Images`. | +| `options.categories` | `Vec` | Filter: `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Vec` | Restrict results to these domains. | +| `options.exclude_domains` | `Vec` | Exclude results from these domains. | +| `options.limit` | `u32` | Max results. Default: `5`, max: `20`. | +| `options.tbs` | `String` | Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). | +| `options.location` | `String` | Localized search results. | +| `options.ignore_invalid_urls` | `bool` | Drop invalid URLs. | +| `options.timeout` | `u32` | Request timeout in milliseconds. | +| `options.highlights` | `bool` | Generate query-relevant highlights. Default: `true`. | +| `options.scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `options.integration` | `String` | Integration identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let doc = client + .scrape("https://docs.firecrawl.dev", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +println!("{:?}", doc.markdown); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | URL to scrape. | +| `options.formats` | `Vec` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. Also `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. | +| `options.headers` | `HashMap` | Custom HTTP headers. | +| `options.include_tags` | `Vec` | HTML tags to include. | +| `options.exclude_tags` | `Vec` | HTML tags to exclude. | +| `options.only_main_content` | `bool` | Strip nav, footer, and boilerplate. | +| `options.timeout` | `u32` | Timeout in milliseconds. | +| `options.wait_for` | `u32` | Wait for page render (milliseconds). | +| `options.mobile` | `bool` | Mobile viewport. | +| `options.parsers` | `Vec` | File parsers. PDF: `ParserConfig::Pdf { parser_type, max_pages, pages, blocks, page_markers }`. | +| `options.actions` | `Vec` | Pre-scrape actions: `Wait`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `Screenshot`, `ExecuteJavascript`, `Pdf`. | +| `options.location` | `LocationConfig` | Geo config: `country`, `languages`. | +| `options.skip_tls_verification` | `bool` | Skip TLS verification. | +| `options.remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `options.fast_mode` | `bool` | Faster scrapes, reduced fidelity. | +| `options.block_ads` | `bool` | Block ads and cookie popups. | +| `options.proxy` | `ProxyType` | Proxy: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `u32` | Max age of cached content (milliseconds). | +| `options.min_age` | `u32` | Min age of cached content (milliseconds). | +| `options.store_in_cache` | `bool` | Cache the result. | +| `options.profile` | `ProfileConfig` | Persistent browser profile: `name`, `save_changes`. | +| `options.json_options` | `JsonOptions` | JSON extraction: `schema`, `system_prompt`, `prompt`. | +| `options.screenshot_options` | `ScreenshotOptions` | Screenshot: `full_page`, `quality`, `viewport`. | +| `options.change_tracking_options` | `ChangeTrackingOptions` | Change tracking: `modes` (`GitDiff`, `Json`), `schema`, `prompt`, `tag`. | +| `options.attribute_selectors` | `Vec` | Attribute extraction: `selector`, `attribute`. | + +## Interact + +### Why use it + +Control the browser session tied to a prior scrape. Use for clicks, form fills, code execution, or natural-language instructions. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +At least one of `code` or `prompt` must be non-empty; otherwise returns `FirecrawlError::Misuse`. + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.scrape_id.as_ref()) + .expect("Missing scrapeId"); + +let result = client + .interact( + job_id, + ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }, + ) + .await?; + +// When done: +client.stop_interaction(job_id).await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | Scrape job ID from `document.metadata.scrape_id`. | +| `options.code` | `Option` | Code to execute. At least one of `code` or `prompt` required. | +| `options.prompt` | `Option` | Natural-language instruction. At least one of `code` or `prompt` required. | +| `options.language` | `ScrapeExecuteLanguage` | Runtime: `Python`, `Node`, `Bash`. Default: `Node`. | +| `options.timeout` | `u32` | Execution timeout in seconds. | + +Stop the session with `client.stop_interaction(job_id)`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`, `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- All types are exported at crate root: `use firecrawl::Client`. +- `ScrapeOptions` has dedicated sub-option structs: `json_options`, `screenshot_options`, `change_tracking_options`. +- `search_and_scrape(query, limit)` is a convenience that calls `search` with default scrape options and returns `Vec`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index 5ce628dac..9f0524599 100755 --- a/docs.json +++ b/docs.json @@ -132,6 +132,16 @@ }, "sdks/cli", "ai-onboarding", + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, "advanced-scraping-guide", { "group": "Plans and Billing",