diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..5854553b0 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,198 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`:firecrawl` **v1.11.0**) and the v2 OpenAPI spec. Function names and parameter keys match the auto-generated SDK module. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.11"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass api_key per call: +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + [url: "https://example.com"], + api_key: "fc-your-api-key" +) +``` + +There is no client struct to instantiate. Auth is handled per-request via application config or the trailing `opts` keyword list. + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs code execution in a post-scrape browser session. Requires a scrape job ID. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` → `{:ok, %Req.Response{}} | {:error, exception}` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + sources: [:web], + limit: 5, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) + +web_results = res.body["data"]["web"] +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` (required) | Search query. Use `site:example.com` to limit to a domain. | +| `sources` | `list` | Which sources: `:web`, `:news`, `:images` (atoms or strings). | +| `categories` | `list` | Filter by category: `:developer`, `:research`, `:pdf` (atoms or strings). | +| `include_domains` | `list[string]` | Only include these domains. | +| `exclude_domains` | `list[string]` | Exclude these domains. | +| `limit` | `integer` | Max results to return. | +| `tbs` | `string` | Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). | +| `location` | `string` | Location for localized results. | +| `country` | `string` | ISO 3166-1 alpha-2 country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | Drop unscrappable URLs. | +| `highlights` | `boolean` | Return query-relevant text highlights. Server default: `true`. | +| `timeout` | `integer` | Request timeout in milliseconds. | +| `scrape_options` | `keyword list` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list[string]` | Enterprise options: `"zdr"`, `"anon"`. | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` → `{:ok, %Req.Response{}} | {:error, exception}` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true +) + +doc = res.body["data"] +IO.puts(doc["markdown"]) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` (required) | URL to scrape. | +| `formats` | `list` | Output formats: strings (`"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`) or maps (`%{type: "json", prompt: "..."}`, etc.). | +| `headers` | `map` | Custom request headers. | +| `include_tags` | `list[string]` | Include only these HTML tags. | +| `exclude_tags` | `list[string]` | Exclude these HTML tags. | +| `only_main_content` | `boolean` | Strip nav, footer, boilerplate. | +| `timeout` | `integer` | Timeout in milliseconds. | +| `wait_for` | `integer` | Wait for page to render (milliseconds). | +| `mobile` | `boolean` | Use a mobile viewport. | +| `parsers` | `list` | File parsing controls (e.g. `%{type: "pdf", mode: "auto", maxPages: 5}`). | +| `actions` | `list[map]` | Pre-scrape browser actions. | +| `location` | `keyword list` | Geo/language-aware scraping: `country:`, `languages:`. | +| `skip_tls_verification` | `boolean` | Skip TLS verification. | +| `remove_base64_images` | `boolean` | Drop base64 images from markdown. | +| `block_ads` | `boolean` | Block ads and cookie popups. | +| `proxy` | `:basic \| :enhanced \| :auto` | Proxy mode. | +| `max_age` | `integer` | Use cached data up to this age (milliseconds). | +| `min_age` | `integer` | Cache-only mode with minimum age (milliseconds). | +| `store_in_cache` | `boolean` | Cache the result. | +| `lockdown` | `boolean` | Serve only cached results. | +| `profile` | `keyword list` | Persistent browser profile: `name:`, `save_changes:`. | +| `zero_data_retention` | `boolean` | Enable zero data retention. | + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. Use for Playwright-style page manipulation after a scrape creates a session. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` → `{:ok, %Req.Response{}} | {:error, exception}` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = scrape_res.body["data"]["metadata"]["scrapeId"] + +# Code-based interaction +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) + +IO.puts(res.body["stdout"]) + +# Clean up +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `string` (positional) | Scrape job ID from response metadata. | +| `code` | `string` (required) | Code to execute in the browser session. | +| `language` | `:python \| :node \| :bash` | Runtime for code execution. | +| `timeout` | `integer` | Execution timeout in seconds. | + +The Elixir SDK exposes **code-based interactions only**. There is no `prompt` parameter (the SDK is auto-generated from the OpenAPI spec which lists `code` only). + +**Stop session:** `Firecrawl.stop_interactive_scrape_browser_session(job_id)` ends the browser session. + +## Notes + +- The Elixir SDK is auto-generated from the OpenAPI spec. Function names and parameter keys match the generated code. +- Every function has a bang (`!`) variant that raises on error: e.g. `search_and_scrape!`, `scrape_and_extract_from_url!`. +- Parameters are passed as snake_case keyword lists; the SDK converts them to camelCase JSON for the API. +- Enum values (proxy, language) are passed as atoms: `:basic`, `:node`, etc. +- Nested objects (location, scrape_options, profile) are passed as keyword lists. +- All functions return `{:ok, %Req.Response{}}` or `{:error, exception}`. The response body is the decoded JSON map. +- No SDK-level defaults are set; all defaults come from the server. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..b06c87bfe --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,219 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-java` **v1.17.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.17.0 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.17.0") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs code execution in a post-scrape browser session. Requires a scrape job ID. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web")) + .limit(5) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `List` | Which sources: `"web"`, `"news"`, `"images"`. | +| `options.categories` | `List` | Filter by category: `"developer"`, `"research"`, `"pdf"`, `"github"`. | +| `options.includeDomains` | `List` | Only include these domains. | +| `options.excludeDomains` | `List` | Exclude these domains. | +| `options.limit` | `Integer` | Max results to return. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location for localized results. | +| `options.ignoreInvalidURLs` | `Boolean` | Drop unscrappable URLs. | +| `options.highlights` | `Boolean` | Return query-relevant text highlights. Server default: `true`. | +| `options.timeout` | `Integer` | Request timeout in milliseconds. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | URL to scrape. | +| `options.formats` | `List` | Output formats: strings (`"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`) or typed objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). | +| `options.headers` | `Map` | Custom request headers. | +| `options.includeTags` | `List` | Include only these HTML tags. | +| `options.excludeTags` | `List` | Exclude these HTML tags. | +| `options.onlyMainContent` | `Boolean` | Strip nav, footer, boilerplate. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.waitFor` | `Integer` | Wait for page to render (milliseconds). | +| `options.mobile` | `Boolean` | Use a mobile viewport. | +| `options.parsers` | `List` | File parsing controls (e.g. `"pdf"` or `Map.of("type", "pdf", "maxPages", 5)`). | +| `options.actions` | `List>` | Pre-scrape browser actions. | +| `options.location` | `LocationConfig` | Geo/language-aware scraping via builder: `.country("US").languages(List.of("en-US"))`. | +| `options.skipTlsVerification` | `Boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `Boolean` | Drop base64 images from markdown. | +| `options.blockAds` | `Boolean` | Block ads and cookie popups. | +| `options.proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `options.maxAge` | `Long` | Use cached data up to this age (milliseconds). | +| `options.storeInCache` | `Boolean` | Cache the result. | +| `options.lockdown` | `Boolean` | Serve only cached results. | + +**Format helper types:** +- `JsonFormat.builder().prompt("...").schema(Map.of(...)).build()` — JSON extraction. +- `QuestionFormat.builder().question("...").build()` — question-answer extraction. +- `HighlightsFormat.builder().query("...").build()` — relevant source-text extraction. + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. Use for Playwright-style page manipulation after a scrape creates a session. + +### Preferred SDK method + +`client.interact(jobId, code)` or `client.interact(jobId, code, language, timeout)` → `BrowserExecuteResponse` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; +import com.firecrawl.models.BrowserDeleteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); +String jobId = (String) doc.getMetadata().get("scrapeId"); + +// Code-based interaction +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); + +// Clean up +BrowserDeleteResponse stopped = client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | Scrape job ID from `document.getMetadata().get("scrapeId")`. | +| `code` | `String` | Code to execute in the browser session. | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1-300). `null` uses API default (30s). | + +The Java SDK exposes **code-based interactions only**. There is no `prompt` parameter (unlike the JS and Python SDKs). + +**Stop session:** `client.stopInteractiveBrowser(jobId)` ends the browser session. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- Every sync method has an async variant returning `CompletableFuture` (e.g. `scrapeAsync`, `searchAsync`, `interactAsync`). +- All option classes use the builder pattern: `ScrapeOptions.builder()...build()`. +- `ScrapeOptions` supports `toBuilder()` for cloning and modifying. +- The client supports a keyless free tier (rate-limited per IP) when no API key is provided. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..74ea87183 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,206 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` **v4.38.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL env var +}); +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web"], + limit: 5, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `("web" \| "news" \| "images")[]` | Which source indexes to search. | +| `options.categories` | `("developer" \| "research" \| "pdf" \| "github")[]` | Filter results by category. | +| `options.includeDomains` | `string[]` | Only include these domains. Cannot combine with `excludeDomains`. | +| `options.excludeDomains` | `string[]` | Exclude these domains. Cannot combine with `includeDomains`. | +| `options.limit` | `number` | Max results to return. | +| `options.tbs` | `string` | Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). | +| `options.location` | `string` | Localized results (e.g. `"San Francisco,California,United States"`). | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped. | +| `options.highlights` | `boolean` | Return query-relevant text highlights. Defaults to `true` server-side. | +| `options.timeout` | `number` | Request timeout in milliseconds. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `options.enterprise` | `("default" \| "anon" \| "zdr")[]` | Enterprise search controls. | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats (see below). | +| `options.headers` | `Record` | Custom request headers. | +| `options.includeTags` | `string[]` | Include only these HTML tags. | +| `options.excludeTags` | `string[]` | Exclude these HTML tags. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, boilerplate. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.waitFor` | `number` | Wait for page to render (milliseconds). | +| `options.mobile` | `boolean` | Use a mobile viewport. | +| `options.parsers` | `("pdf" \| PDFParser)[]` | File parsing controls (e.g. `{ type: "pdf", mode: "auto", maxPages: 5 }`). | +| `options.actions` | `ActionOption[]` | Pre-scrape browser actions (click, wait, write, press, scroll, scrape, executeJavascript, screenshot, pdf). | +| `options.location` | `{ country?: string; languages?: string[] }` | Geo/language-aware scraping. | +| `options.skipTlsVerification` | `boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown. | +| `options.fastMode` | `boolean` | Faster scrapes with reduced fidelity. | +| `options.blockAds` | `boolean` | Block ads and cookie popups. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto"` | Proxy mode. | +| `options.maxAge` | `number` | Use cached data up to this age (milliseconds). | +| `options.storeInCache` | `boolean` | Cache the result. | +| `options.lockdown` | `boolean` | Serve only cached results; never make outbound request. | +| `options.profile` | `{ name: string; saveChanges?: boolean }` | Persistent browser profile across scrapes and interactions. | + +**Format options:** + +String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. + +Object formats: +- `{ type: "json", prompt?: string, schema?: object }` — at least one of `prompt` or `schema` required. +- `{ type: "question", question: string }` — question-answer extraction. +- `{ type: "highlights", query: string }` — relevant source-text extraction. +- `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }` +- `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }` — `modes` required. +- `{ type: "attributes", selectors: { selector: string, attribute: string }[] }` + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language instructions after a scrape creates a session. Requires `scrapeId` from `document.metadata.scrapeId`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId"); + +// Natural-language interaction +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +// Or code-based interaction +const codeResult = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); + +// Clean up +await client.stopInteraction(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to run in the browser session. | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Default: `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided. + +**Stop session:** `client.stopInteraction(jobId)` ends the browser session. Returns `{ success, sessionDurationMs?, creditsBilled?, error? }`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` / `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed in `formats` (for `json` or `changeTracking`) are auto-converted to JSON Schema by the SDK, and the return type narrows accordingly. +- The SDK auto-resumes scrapes for large documents (PDFs) that outlive the request window (up to 5 retries / 20 minutes). +- The package declares Node.js >= 22 in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..7c11f1711 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,206 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-py` **v4.41.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the v2 client. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python >= 3.8. + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrape_id` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web"], + limit=5, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | Search query. Use `site:example.com` to limit to a domain. | +| `sources` | `list[str]` | Which source indexes to search: `"web"`, `"news"`, `"images"`. | +| `categories` | `list[str]` | Filter by category: `"developer"`, `"research"`, `"pdf"`, `"github"`. | +| `include_domains` | `list[str]` | Only include these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude these domains. Cannot combine with `include_domains`. | +| `limit` | `int` | Max results. Default: `5` (SDK model default). | +| `tbs` | `str` | Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). | +| `location` | `str` | Localized results (plain string, not a `Location` object). | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `highlights` | `bool` | Return query-relevant text highlights. Defaults to `true` server-side. | +| `timeout` | `int` | Request timeout in milliseconds. Default: `300000`. | +| `scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list[str]` | Enterprise options: `"zdr"` (zero data retention), `"anon"` (anonymized). | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | URL to scrape. | +| `formats` | `list` | Output formats (see below). | +| `headers` | `dict[str, str]` | Custom request headers. | +| `include_tags` | `list[str]` | Include only these HTML tags. | +| `exclude_tags` | `list[str]` | Exclude these HTML tags. | +| `only_main_content` | `bool` | Strip nav, footer, boilerplate. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait for page to render (milliseconds). | +| `mobile` | `bool` | Use a mobile viewport. | +| `parsers` | `list` | File parsing controls (e.g. `{"type": "pdf", "mode": "auto", "max_pages": 5}`). | +| `actions` | `list[dict]` | Pre-scrape browser actions (click, wait, write, press, scroll, scrape, executeJavascript, screenshot, pdf). | +| `location` | `Location` | Geo/language-aware scraping. `Location(country="US", languages=["en-US"])`. | +| `skip_tls_verification` | `bool` | Skip TLS verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Use cached data up to this age (milliseconds). | +| `store_in_cache` | `bool` | Cache the result. | +| `lockdown` | `bool` | Serve only cached results; never make outbound request. | +| `profile` | `dict` | Persistent browser profile: `{"name": "my-session", "saveChanges": True}`. | + +**Format options:** + +String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"audio"`, `"video"`. + +Object formats: +- `{"type": "json", "prompt": "...", "schema": {...}}` — at least one of `prompt` or `schema` required. +- `{"type": "question", "question": "..."}` — question-answer extraction. +- `{"type": "highlights", "query": "..."}` — relevant source-text extraction. +- `{"type": "screenshot", "full_page": True, "quality": 80, "viewport": {"width": 1280, "height": 720}}` +- `{"type": "changeTracking", "modes": ["git-diff"], "tag": "..."}` — `modes` required. +- `{"type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}]}` + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language instructions after a scrape creates a session. Requires `scrape_id` from `document.metadata.scrape_id`. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id") + +# Natural-language interaction +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") + +# Or code-based interaction +code_result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) + +# Clean up +client.stop_interaction(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | Scrape job ID from `document.metadata.scrape_id`. | +| `code` | `str` | Code to run in the browser session (optional if `prompt` is set). | +| `prompt` | `str` | Natural-language instruction for the browser agent (keyword-only; optional if `code` is set). | +| `language` | `str` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided. + +**Stop session:** `client.stop_interaction(job_id)` ends the browser session. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `FirecrawlApp` is a direct alias for `Firecrawl`. +- `search()` location is a plain `str`, not a `Location` object (unlike `scrape()`). +- `SearchRequest` model defaults: `limit=5`, `timeout=300000`. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..5fecbcac7 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,233 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` crate **v2.18.0**) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +cargo add firecrawl +``` + +Requires an async runtime (Tokio). + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +## When To Use What + +- **`search`**: use when you start with a query and need discovery. +- **`scrape`**: use when you already have a URL and want page content. +- **`interact`**: use when the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web]), + limit: Some(5), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; + +if let Some(web) = results.data.web { + for item in web { + // item is SearchResultOrDocument::WebResult(...) or ::Document(...) + } +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `Vec` | Which sources: `Web`, `News`, `Images`. | +| `options.categories` | `Vec` | Filter by category: `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Vec` | Only include these domains. | +| `options.exclude_domains` | `Vec` | Exclude these domains. | +| `options.limit` | `u32` | Max results. Doc comment says default 5, max 20. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location for localized results. | +| `options.ignore_invalid_urls` | `bool` | Drop unscrappable URLs. | +| `options.highlights` | `bool` | Return query-relevant highlights. Defaults to `true`. | +| `options.timeout` | `u32` | Request timeout in milliseconds. | +| `options.scrape_options` | `ScrapeOptions` | Scrape each result (see Scrape parameters). | + +## Scrape + +### Why use it + +Fetch structured content from a URL in one or more formats. Use when you already have the URL. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | URL to scrape. | +| `options.formats` | `Vec` | Output formats (see below). | +| `options.headers` | `HashMap` | Custom request headers. | +| `options.include_tags` | `Vec` | Include only these HTML tags. | +| `options.exclude_tags` | `Vec` | Exclude these HTML tags. | +| `options.only_main_content` | `bool` | Strip nav, footer, boilerplate. | +| `options.timeout` | `u32` | Timeout in milliseconds. | +| `options.wait_for` | `u32` | Wait for page to render (milliseconds). | +| `options.mobile` | `bool` | Use a mobile viewport. | +| `options.parsers` | `Vec` | File parsing controls (e.g. `ParserConfig::Pdf { ... }`). | +| `options.actions` | `Vec` | Pre-scrape browser actions (Click, Wait, Write, Press, Scroll, Scrape, ExecuteJavascript, Screenshot, Pdf). | +| `options.location` | `LocationConfig` | Geo/language-aware scraping: `country`, `languages`. | +| `options.skip_tls_verification` | `bool` | Skip TLS verification. | +| `options.remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `options.fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `options.block_ads` | `bool` | Block ads and cookie popups. | +| `options.proxy` | `ProxyType` | Proxy mode: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `u32` | Use cached data up to this age (milliseconds). | +| `options.store_in_cache` | `bool` | Cache the result. | +| `options.lockdown` | `bool` | Serve only cached results. | +| `options.profile` | `ProfileConfig` | Persistent browser profile: `name`, `save_changes`. | +| `options.json_options` | `JsonOptions` | JSON extraction config: `schema`, `prompt`, `system_prompt`. | +| `options.screenshot_options` | `ScreenshotOptions` | Screenshot config: `full_page`, `quality`, `viewport`. | +| `options.change_tracking_options` | `ChangeTrackingOptions` | Change tracking: `modes` (`GitDiff`/`Json`), `schema`, `prompt`, `tag`. | +| `options.attribute_selectors` | `Vec` | Attribute extraction: `selector`, `attribute`. | + +**Format enum values:** + +Simple: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. + +Object variants: `Question(QuestionFormat { question })`, `Highlights(HighlightsFormat { query })`. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for code execution or natural-language instructions after a scrape creates a session. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +// Natural-language interaction +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +// Or code-based interaction +let code_result = client + .interact(job_id, ScrapeExecuteOptions { + code: Some("console.log(await page.title());".to_string()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(60), + ..Default::default() + }) + .await?; + +// Clean up +client.stop_interaction(job_id).await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | Scrape job ID from document metadata. | +| `options.code` | `Option` | Code to run in the browser session. | +| `options.prompt` | `Option` | Natural-language instruction for the browser agent. | +| `options.language` | `ScrapeExecuteLanguage` | Runtime: `Python`, `Node`, `Bash`. Default: `Node`. | +| `options.timeout` | `u32` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be non-empty (SDK returns `FirecrawlError::Misuse` otherwise). + +**Stop session:** `client.stop_interaction(job_id)` ends the browser session. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- All option structs derive `Default`; use struct-update syntax: `ScrapeOptions { formats: Some(vec![...]), ..Default::default() }`. +- The `options` parameter on `scrape` and `search` accepts `impl Into>`, so you can pass `None` or a bare struct. +- All methods are `async` and require a Tokio runtime. +- `search_and_scrape(query, limit)` is a convenience helper that returns `Vec`. +- All serialization uses camelCase for the API wire format. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index a9d0cccef..bc86f6516 100755 --- a/docs.json +++ b/docs.json @@ -611,6 +611,16 @@ } ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "AI Tools", "pages": [ @@ -5352,4 +5362,4 @@ } }, "theme": "aspen" -} \ No newline at end of file +}