From 9c9a0883e432bb89cc33833635ee5b1e399238b8 Mon Sep 17 00:00:00 2001 From: neil Date: Fri, 4 Sep 2026 12:24:07 +0800 Subject: [PATCH 1/4] feat(http): delegate container browser rendering to gateway --- CONTEXT.md | 12 +- Dockerfile | 18 +- README.md | 28 +- docs/adr/0001-containerized-rest-api.md | 6 + .../0003-http-browser-rendering-gateway.md | 41 +++ docs/http-service.md | 20 +- packages/web-core/src/browser-gateway.ts | 258 +++++++++++++++ packages/web-core/src/fetch.ts | 32 +- packages/web-core/src/index.ts | 31 +- .../web-core/test/browser-gateway.test.ts | 301 ++++++++++++++++++ packages/web/src/http.ts | 28 +- packages/web/src/program.ts | 1 - packages/web/test/http.test.ts | 55 ++++ 13 files changed, 788 insertions(+), 43 deletions(-) create mode 100644 docs/adr/0003-http-browser-rendering-gateway.md create mode 100644 packages/web-core/src/browser-gateway.ts create mode 100644 packages/web-core/test/browser-gateway.test.ts diff --git a/CONTEXT.md b/CONTEXT.md index da0a783..e5585f2 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -32,8 +32,10 @@ _Avoid_: Bridge URL parameter **Page Rendering**: Fetch and Links use `render: "http"` by default or explicit `render: "browser"`. Browser rendering requires `waitMs` from 0 through 30,000 and is never selected -automatically. The operator-installed `agent-browser` runtime is an implementation -and setup detail, not an adapter-facing request value. +automatically. The containerized HTTP Service delegates browser requests to the +server-local Browser Rendering Gateway; CLI, MCP, Pi, and DSH retain the +operator-installed `agent-browser` capability. Both are implementation and setup +details, not adapter-facing request values. _Avoid_: Backend-specific renderer labels, automatic fallback **Page Navigation**: @@ -57,8 +59,10 @@ The Hono-based `/api/v1/web` JSON API shipped by `web serve` and the GHCR image. uses server-local credentials, Bridge Route, and optional DeepSeek provider configuration; clients do not select providers or submit a generic Bridge command. Its page-reading routes use -the same `render: "http" | "browser"`, `mode`, and `section_id` contract; the -browser executable name appears only in operator setup. +the same `render: "http" | "browser"`, `mode`, and `section_id` contract. Browser +rendering is delegated to the +server-local Browser Rendering Gateway configured by the operator; the browser +executable name appears only in gateway-side setup. _Avoid_: Remote MCP, public service **Gateway Web API prefix**: diff --git a/Dockerfile b/Dockerfile index a78f982..80542fa 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,5 +1,6 @@ # Guion Web's supported HTTP-service image. The build stage bundles the web -# executable; the runtime stage adds the optional rendered-fetch capability. +# executable; browser rendering is delegated to the configured in-cluster +# Browser Rendering Gateway rather than installed in this image. FROM node:24-bookworm-slim AS build WORKDIR /workspace @@ -13,22 +14,13 @@ RUN pnpm install --frozen-lockfile COPY . . RUN pnpm --filter @guionai/web run build -FROM node:24-bookworm +FROM node:24-bookworm-slim ENV NODE_ENV=production -# Chrome for Testing has no Linux ARM64 distribution. Debian's Chromium works -# on both released container architectures, and agent-browser documents this -# executable override for containers. -ENV AGENT_BROWSER_EXECUTABLE_PATH=/usr/bin/chromium WORKDIR /app -# Rendering is deliberately explicit at the HTTP contract. The executable and -# browser runtime are image capabilities, while credentials remain env-only. -ARG AGENT_BROWSER_VERSION=0.36.0 -RUN apt-get update \ - && apt-get install --yes --no-install-recommends chromium \ - && rm -rf /var/lib/apt/lists/* \ - && npm install --global agent-browser@${AGENT_BROWSER_VERSION} +# `BROWSER_GATEWAY_URL` is optional at startup; browser requests fail +# explicitly until the operator points the service at the internal gateway. COPY --from=build /workspace/packages/web/dist ./dist diff --git a/README.md b/README.md index e625041..ab2363f 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,8 @@ export DEEPSEEK_API_KEY="..." export CONTEXT7_API_KEY="..." # optional complete Bridge route for `web serve` export KEPOS_BRIDGE_ENDPOINT="http://127.0.0.1:8787/codex/web-search" +# optional Browser Rendering Gateway origin for `web serve` browser requests +export BROWSER_GATEWAY_URL="http://browser-gateway" # HTTP/Pi: select DeepSeek server-side (HTTP clients still send {"query":"..."}) export WEB_SEARCH_PROVIDER="deepseek" ``` @@ -70,6 +72,7 @@ web serve --host 0.0.0.0 --port 8787 docker run --rm -p 8787:8787 \ -e EXA_API_KEY="$EXA_API_KEY" \ -e KEPOS_BRIDGE_ENDPOINT="http://host.docker.internal:17480/codex/web-search" \ + -e BROWSER_GATEWAY_URL="http://host.docker.internal:8788" \ ghcr.io/guionai/web:v0.1.0 ``` @@ -101,10 +104,11 @@ cancellation is reported as 499. Fetch and Links use HTTP rendering when `render` is omitted (or set to `"http"`). Browser rendering is explicit and requires both `render: "browser"` and an integer `waitMs` from 0 through 30,000; HTTP rendering never silently -switches backends. The container installs the `agent-browser` executable with -Debian Chromium (including Linux ARM64, where Chrome for Testing has no build), -while credentials and Bridge configuration remain server-local environment -variables. +switches backends. The `web serve` HTTP Service sends browser requests to the +server-local Browser Rendering Gateway configured by `BROWSER_GATEWAY_URL`. +The GHCR image contains no Chromium or `agent-browser`; an absent, unreachable, +overloaded, or failed gateway returns an explicit browser-render failure while +ordinary HTTP rendering remains available. This is a Personal Web Service: a single-trust-boundary deployment for its operator and agents. It is not hardened for public or multi-tenant exposure; @@ -226,9 +230,11 @@ from the original page DOM. `web fetch` has two renderers. `http` (the default) uses Node `fetch`, `linkedom`, and Defuddle for HTML-to-Markdown extraction from static, SSR, and pre-rendered -pages. `browser` renders client-side pages through the separately installed -host browser capability. HTTP rendering is used by default; choose browser -explicitly when needed. The implementation never falls back automatically: +pages. `browser` renders client-side pages through the host capability: `web +serve` delegates to its configured Browser Rendering Gateway, while CLI, MCP, +Pi, and DSH use the separately installed `agent-browser` capability. HTTP +rendering is used by default; choose browser explicitly when needed. The +implementation never falls back automatically: ```bash web fetch https://example.com/app --render=browser --wait=2000 @@ -249,7 +255,8 @@ or `links` requests must not provide `--wait`. The same `render: "browser"` and `javascript_rendering_may_be_required` hint with the 2,000 ms suggestion; the agent decides whether to retry with a longer wait or abandon the page. -Rendering is an optional host capability. If you choose to use it, install +Direct rendering is an optional host capability for CLI, MCP, Pi, and DSH. If +you choose to use it, install [agent-browser](https://github.com/vercel-labs/agent-browser) separately on the host: @@ -323,8 +330,9 @@ Three independent, non-fail-fast protected `npm` Environment matrix cells then publish one package each through npm Trusted Publishing with provenance. The synchronized version selects npm's `latest` tag for stable SemVer and `beta` for a prerelease. A matching immutable-tagged image is published to -`ghcr.io/guionai/web:` with the `web serve` entrypoint and the -`agent-browser` runtime. After all three npm cells and the image job succeed, +`ghcr.io/guionai/web:` with the `web serve` entrypoint. The image delegates +explicit browser rendering to the configured internal Browser Rendering +Gateway and contains no browser executable. After all three npm cells and the image job succeed, the workflow creates the GitHub release with generated notes, source archives, and the build-generated `openapi.yaml` asset. The asset is generated from the same Hono route schemas as the image and package; it is not checked in or diff --git a/docs/adr/0001-containerized-rest-api.md b/docs/adr/0001-containerized-rest-api.md index fd9effa..dc59249 100644 --- a/docs/adr/0001-containerized-rest-api.md +++ b/docs/adr/0001-containerized-rest-api.md @@ -4,4 +4,10 @@ Guion Web will add a self-hosted, single-user HTTP service in a portable contain ## Consequences +The HTTP Service's explicit browser mode is implemented by the server-local +Browser Rendering Gateway; the image does not install or launch a browser. +The shared Core still gives CLI, MCP, Pi, and DSH adapters their direct host +browser capability. See [ADR 0003](0003-http-browser-rendering-gateway.md) for +the boundary and failure behavior. + The service's provider credentials, Bridge Route, and optional `WEB_SEARCH_PROVIDER=deepseek` selection are server-local configuration; clients cannot choose a provider or supply a Bridge Route per request. With no provider selection, search tries Kepos Bridge and uses Exa only when the Bridge is operationally unavailable; it does not fall back for cancellation, malformed client input, or an empty result set. A failed Bridge attempt is retried through Exa exactly once, and a successful empty Bridge response is returned unchanged. When DeepSeek is selected, the service calls only DeepSeek and never falls back. DeepSeek performs one auxiliary model call internally and returns the same normalized result contract; its Messages/tool protocol is not exposed to HTTP callers. Typed Bridge operations are intentionally not exposed: Exa has no equivalent official weather, sports, or time API, and its premium finance integration is not contract-compatible. Fetch and Links use `render: "http"` by default or explicit `render: "browser"` with a required `waitMs`; the operator-installed executable remains an implementation detail. Fetch navigation input uses `mode: "auto"` by default, with explicit `"full"` or `"tree"` modes; a non-empty `section_id` with omitted or `"auto"` mode retrieves a section, while full/tree reject it. Results report `"auto"`, `"full"`, `"tree"`, or `"section"` as appropriate and include a `truncated` flag for content cut by the Core limit. `openapi.yaml` is generated from the release build rather than manually maintained or independently versioned; the standalone [HTTP service reference](../http-service.md) is its human-readable companion. Public or multi-tenant deployment hardening is deliberately deferred in `.scratch/defered/public-http-service-security.md`. diff --git a/docs/adr/0003-http-browser-rendering-gateway.md b/docs/adr/0003-http-browser-rendering-gateway.md new file mode 100644 index 0000000..b8477b1 --- /dev/null +++ b/docs/adr/0003-http-browser-rendering-gateway.md @@ -0,0 +1,41 @@ +# Delegate HTTP browser rendering to the Browser Rendering Gateway + +## Status + +Accepted + +## Context + +The GHCR image previously installed Chromium and `agent-browser` so each +explicit `render: "browser"` request could start a private browser process. +The apps-dev deployment already operates a persistent Browser Rendering +Gateway with the proxy, anti-bot, and capacity controls needed for browser +work. Maintaining a second browser runtime in the HTTP-service image adds +startup, memory, and lifecycle cost. + +## Decision + +Only the containerized HTTP Service delegates explicit browser rendering to +the server-local Browser Rendering Gateway. It sends `POST /api/render` with +`{ "url", "waitMs" }` and receives raw rendered `{ "html", "url" }`. The +existing Web Core then performs target validation, extraction, navigation, and +link handling, so Fetch and Links keep their public contracts. `waitMs` remains +caller-visible and is required from 0 through 30,000; browser rendering never +falls back to HTTP. + +`BROWSER_GATEWAY_URL` is server-local configuration. Missing configuration, +gateway overload, transport failure, timeout, or malformed output becomes an +explicit `render_*` capability failure. HTTP rendering remains independent. + +CLI, MCP, Pi, and DSH continue to use the shared Web Core's direct +`agent-browser` capability. The GHCR image therefore contains neither +Chromium nor `agent-browser`; the gateway owns that runtime and its deployment +boundary. + +## Consequences + +The HTTP-service image is smaller and does not launch a local browser. Browser +requests require the in-cluster gateway to be reachable, and operators must +configure `BROWSER_GATEWAY_URL` before using them. Gateway deployment, +capacity, authentication, and rollout remain outside this repository and are +owned by the separate browser-gateway service. diff --git a/docs/http-service.md b/docs/http-service.md index 988a11d..392316a 100644 --- a/docs/http-service.md +++ b/docs/http-service.md @@ -69,10 +69,13 @@ or `"tree"`. A non-empty `section_id` may be supplied with omitted mode or `"tree"`. HTTP rendering fetches the page with Node HTTP, linkedom, and Defuddle. -Browser rendering invokes the operator-installed `agent-browser` executable -through the isolated renderer implementation; the executable name is not a -public request value. Browser rendering is never selected automatically, and -the service does not fall back between renderers. +Browser rendering delegates the raw DOM request to the internal Browser +Rendering Gateway; the gateway's browser executable is not a public request +value. Configure its origin with the server-local `BROWSER_GATEWAY_URL` +environment variable. Browser rendering is never selected automatically, and +the service does not fall back between renderers. If the URL is missing or the +gateway rejects, times out, or returns an invalid response, the operation +fails explicitly with a `render_*` capability error. The shared module owns the 5,000-character automatic-tree policy. An `"auto"` request for an unsectioned document longer than that threshold with navigable @@ -155,9 +158,12 @@ response bodies. ## Configuration and OpenAPI -Credentials, the optional `KEPOS_BRIDGE_ENDPOINT`, and the optional -server-local `WEB_SEARCH_PROVIDER=deepseek` selection are environment -variables. They are not accepted in request bodies. DeepSeek performs one +Credentials, the optional `KEPOS_BRIDGE_ENDPOINT`, the optional +`BROWSER_GATEWAY_URL`, and the optional server-local +`WEB_SEARCH_PROVIDER=deepseek` selection are environment variables. The +gateway URL is a base URL; the service calls its `POST /api/render` raw-render +operation with `{ "url", "waitMs" }`. They are not accepted in request bodies. +DeepSeek performs one auxiliary model call per search; callers receive only normalized results and do not need to know the Messages/tool wire protocol. The route schemas in [`packages/web/src/http.ts`](../packages/web/src/http.ts) are the diff --git a/packages/web-core/src/browser-gateway.ts b/packages/web-core/src/browser-gateway.ts new file mode 100644 index 0000000..f117cd2 --- /dev/null +++ b/packages/web-core/src/browser-gateway.ts @@ -0,0 +1,258 @@ +import { + boundedRequest, + isOperationAborted, + isRequestTimeout, + isResponseBodyLimit, + OperationAbortedError, + readResponseText, + throwIfAborted, +} from "./request.js"; + +/** The raw-render operation exposed by the in-cluster Browser Rendering Gateway. */ +export const BROWSER_GATEWAY_RENDER_PATH = "/api/render" as const; + +/** Leave a small transport margin below the gateway's browser budget. */ +export const BROWSER_GATEWAY_TIMEOUT_MS = 50_000 as const; +export const BROWSER_GATEWAY_MAX_RESPONSE_BYTES = 10 * 1024 * 1024; + +export type BrowserGatewayPage = { + html: string; + /** The post-redirect URL reported by the gateway. */ + url: string; +}; + +export type BrowserGatewayRequest = { + url: string; + waitMs: number; + signal?: AbortSignal; +}; + +/** Test-owned transport seam for a gateway request without a live service. */ +export type BrowserGatewayTransport = ( + request: BrowserGatewayRequest, +) => Promise; + +export type BrowserGatewayOptions = { + /** Server-local gateway origin; browser requests fail explicitly when absent. */ + baseUrl?: string; + /** Test-owned transport that replaces the HTTP request. */ + transport?: BrowserGatewayTransport; + /** Test-owned HTTP implementation for the gateway request. */ + fetch?: typeof globalThis.fetch; + /** Bounded gateway request timeout override for tests. */ + timeoutMs?: number; +}; + +export type BrowserGatewayFailure = + | "unavailable" + | "timed_out" + | "output_too_large" + | "invalid_output" + | "failed"; + +/** A bounded, normalized failure from the gateway boundary. */ +export class BrowserGatewayError extends Error { + constructor(readonly kind: BrowserGatewayFailure) { + super(`browser gateway ${kind}`); + this.name = "BrowserGatewayError"; + } +} + +/** + * Requests raw rendered HTML from the gateway and validates only its stable + * response contract. Markdown extraction and link handling remain in fetch.ts. + */ +export async function renderThroughBrowserGateway( + options: BrowserGatewayOptions, + request: BrowserGatewayRequest, +): Promise { + throwIfAborted(request.signal); + if ( + !Number.isInteger(request.waitMs) || + request.waitMs < 0 || + request.waitMs > 30_000 + ) + throw new BrowserGatewayError("failed"); + + if (options.transport !== undefined) { + try { + const page = await awaitWithAbort( + options.transport(request), + request.signal, + ); + throwIfAborted(request.signal); + return validateGatewayPage(page); + } catch (error) { + throw normalizeGatewayError(error, request.signal); + } + } + + const endpoint = gatewayRenderEndpoint(options.baseUrl); + if (endpoint === undefined) throw new BrowserGatewayError("unavailable"); + const timeoutMs = options.timeoutMs ?? BROWSER_GATEWAY_TIMEOUT_MS; + if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) + throw new BrowserGatewayError("unavailable"); + + let body: string; + try { + body = JSON.stringify({ url: request.url, waitMs: request.waitMs }); + } catch { + throw new BrowserGatewayError("failed"); + } + + try { + return await boundedRequest( + options.fetch, + endpoint, + { + method: "POST", + headers: { + accept: "application/json", + "content-type": "application/json", + }, + body, + }, + { + callerSignal: request.signal, + timeoutMs, + timeoutMessage: `browser gateway request timed out after ${timeoutMs / 1000} seconds`, + }, + async (response, signal) => { + if (!response.ok) throw new BrowserGatewayHTTPError(response.status); + let decoded: unknown; + try { + decoded = JSON.parse( + await readResponseText( + response, + BROWSER_GATEWAY_MAX_RESPONSE_BYTES, + signal, + ), + ); + } catch (error) { + if ( + isOperationAborted(error) || + isRequestTimeout(error) || + isResponseBodyLimit(error) + ) + throw error; + throw new BrowserGatewayError("invalid_output"); + } + throwIfAborted(request.signal); + return validateGatewayPage(decoded); + }, + ); + } catch (error) { + throw normalizeGatewayError(error, request.signal); + } +} + +function gatewayRenderEndpoint( + baseUrl: string | undefined, +): string | undefined { + if ( + typeof baseUrl !== "string" || + baseUrl.length === 0 || + baseUrl.trim() !== baseUrl + ) + return undefined; + try { + const base = new URL(baseUrl); + if ( + (base.protocol !== "http:" && base.protocol !== "https:") || + base.username !== "" || + base.password !== "" || + base.search !== "" || + base.hash !== "" + ) + return undefined; + return new URL(BROWSER_GATEWAY_RENDER_PATH, base).href; + } catch { + return undefined; + } +} + +function validateGatewayPage(value: unknown): BrowserGatewayPage { + if ( + !value || + typeof value !== "object" || + Array.isArray(value) || + typeof (value as { html?: unknown }).html !== "string" || + typeof (value as { url?: unknown }).url !== "string" + ) + throw new BrowserGatewayError("invalid_output"); + + const html = (value as { html: string }).html; + const url = (value as { url: string }).url; + if ( + new TextEncoder().encode(html).byteLength > + BROWSER_GATEWAY_MAX_RESPONSE_BYTES + ) + throw new BrowserGatewayError("output_too_large"); + try { + const parsed = new URL(url); + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") + throw new Error("unsupported rendered URL protocol"); + } catch { + throw new BrowserGatewayError("invalid_output"); + } + return { html, url }; +} + +class BrowserGatewayHTTPError extends Error { + constructor(readonly status: number) { + super(`browser gateway HTTP ${status}`); + this.name = "BrowserGatewayHTTPError"; + } +} + +function normalizeGatewayError( + error: unknown, + signal: AbortSignal | undefined, +): Error { + if (signal?.aborted || isOperationAborted(error) || isAbortError(error)) + return new OperationAbortedError(); + if (error instanceof BrowserGatewayError) return error; + if (isRequestTimeout(error)) return new BrowserGatewayError("timed_out"); + if (isResponseBodyLimit(error)) + return new BrowserGatewayError("output_too_large"); + if (error instanceof BrowserGatewayHTTPError) { + if (error.status === 408 || error.status === 504) + return new BrowserGatewayError("timed_out"); + if (error.status === 429 || error.status === 500 || error.status === 503) + return new BrowserGatewayError("unavailable"); + if (error.status >= 500) return new BrowserGatewayError("failed"); + return new BrowserGatewayError("failed"); + } + // Network failures are intentionally opaque to callers and explicit about + // the unavailable gateway boundary rather than falling back to HTTP. + return new BrowserGatewayError("unavailable"); +} + +function isAbortError(error: unknown): boolean { + return ( + (error instanceof DOMException && error.name === "AbortError") || + (error instanceof Error && error.name === "AbortError") + ); +} + +function awaitWithAbort( + work: Promise, + signal: AbortSignal | undefined, +): Promise { + throwIfAborted(signal); + if (!signal) return work; + return new Promise((resolve, reject) => { + const abort = () => reject(new OperationAbortedError()); + signal.addEventListener("abort", abort, { once: true }); + work.then( + (value) => { + signal.removeEventListener("abort", abort); + resolve(value); + }, + (error: unknown) => { + signal.removeEventListener("abort", abort); + reject(error); + }, + ); + }); +} diff --git a/packages/web-core/src/fetch.ts b/packages/web-core/src/fetch.ts index f5ad4f5..e1c5b95 100644 --- a/packages/web-core/src/fetch.ts +++ b/packages/web-core/src/fetch.ts @@ -9,6 +9,11 @@ import { spawn } from "node:child_process"; import { Defuddle } from "defuddle/node"; import { parseHTML } from "linkedom"; +import { + BrowserGatewayError, + renderThroughBrowserGateway, + type BrowserGatewayOptions, +} from "./browser-gateway.js"; import { FETCH_MODES, renderMarkdown, type FetchMode } from "./markdown.js"; import { boundedRequest, @@ -116,6 +121,11 @@ export interface FetchOptions { removeWorkDirectory?: (path: string) => Promise; /** Test-only override for the renderer cleanup allowance. */ rendererCleanupTimeoutMs?: number; + /** + * Server-local raw-render gateway configuration. When present, browser + * rendering uses this boundary instead of launching a local browser. + */ + browserGateway?: BrowserGatewayOptions; } /** Fetches static HTML or explicit browser-rendered DOM as established Markdown navigation modes. */ @@ -390,7 +400,7 @@ async function renderPage( options, target, ); - return await extractHTML(page.html, url, callerSignal, false); + return await extractHTML(page.html, page.url, callerSignal, false); } catch (error) { throw rendererFailure(error); } @@ -405,6 +415,17 @@ async function renderPageHTML( ): Promise { const renderTarget = target ?? (await validateRenderTarget(url, callerSignal, options)); + if (options?.browserGateway !== undefined) { + try { + return await renderThroughBrowserGateway(options.browserGateway, { + url, + waitMs, + signal: callerSignal, + }); + } catch (error) { + throw rendererFailure(error); + } + } const workDirectory = await mkdtemp(join(tmpdir(), "guionai-web-render-")); const configPath = join(workDirectory, "agent-browser.json"); const session = randomUUID(); @@ -625,7 +646,12 @@ async function extractHTML( } return ensureTrailingNewline(content); } catch (error) { - if (error instanceof FetchCapabilityError) throw error; + if ( + error instanceof FetchCapabilityError || + isOperationAborted(error) || + isRequestTimeout(error) + ) + throw error; throw new Error(`defuddle parse failed: ${errorMessage(error)}`); } } @@ -884,6 +910,8 @@ function parseSuccessEnvelope(stdout: string): Record { function rendererFailure(error: unknown): Error { if (isOperationAborted(error)) return error as Error; if (error instanceof FetchCapabilityError) return error; + if (error instanceof BrowserGatewayError) + return new FetchCapabilityError(`render_${error.kind}`); if (!(error instanceof RendererCommandError)) return new FetchCapabilityError("render_failed"); if (error.kind === "aborted") return new OperationAbortedError(); diff --git a/packages/web-core/src/index.ts b/packages/web-core/src/index.ts index f910076..c5f4335 100644 --- a/packages/web-core/src/index.ts +++ b/packages/web-core/src/index.ts @@ -15,6 +15,7 @@ import { type LinksInput, type LinksResult, } from "./fetch.js"; +import type { BrowserGatewayOptions } from "./browser-gateway.js"; import { sgraphSearch, type SGraphInput, type SGraphResult } from "./sgraph.js"; import { callKeposBridge, @@ -30,6 +31,19 @@ import { throwIfAborted, } from "./request.js"; +export { + BROWSER_GATEWAY_MAX_RESPONSE_BYTES, + BROWSER_GATEWAY_RENDER_PATH, + BROWSER_GATEWAY_TIMEOUT_MS, + BrowserGatewayError, + renderThroughBrowserGateway, + type BrowserGatewayFailure, + type BrowserGatewayOptions, + type BrowserGatewayPage, + type BrowserGatewayRequest, + type BrowserGatewayTransport, +} from "./browser-gateway.js"; + export { OperationAbortedError, RequestTimeoutError, @@ -166,13 +180,24 @@ export type WebOperations = { sgraphSearch(input: SGraphInput): Promise; }; +export type WebOperationsOptions = { + /** Optional server-local browser gateway used by HTTP-service operations. */ + browserGateway?: BrowserGatewayOptions; +}; + /** Creates the default in-process implementation shared by every host adapter. */ -export function createWebOperations(): WebOperations { +export function createWebOperations( + options: WebOperationsOptions = {}, +): WebOperations { + const fetchOptions = + options.browserGateway === undefined + ? undefined + : { browserGateway: options.browserGateway }; return { search, keposBridge: callKeposBridge, - fetch: fetchWebPage, - links: fetchWebLinks, + fetch: (input, signal) => fetchWebPage(input, signal, fetchOptions), + links: (input, signal) => fetchWebLinks(input, signal, fetchOptions), docsResolve, docsFetch, sgraphSearch, diff --git a/packages/web-core/test/browser-gateway.test.ts b/packages/web-core/test/browser-gateway.test.ts new file mode 100644 index 0000000..03ccb2b --- /dev/null +++ b/packages/web-core/test/browser-gateway.test.ts @@ -0,0 +1,301 @@ +import { describe, expect, it, vi } from "vitest"; + +import { + BROWSER_GATEWAY_RENDER_PATH, + BROWSER_GATEWAY_TIMEOUT_MS, + fetchWebLinks, + fetchWebPage, + OperationAbortedError, + type BrowserGatewayPage, +} from "../src/index.js"; + +const publicTarget = { resolveHost: async () => ["93.184.216.34"] }; + +function renderedPage( + overrides: Partial = {}, +): BrowserGatewayPage { + return { + html: "

Rendered page

Gateway content.

", + url: "https://render.test/final/page", + ...overrides, + }; +} + +describe("Browser Rendering Gateway fetch seam", () => { + it("keeps Fetch extraction and Links navigation on gateway HTML and final URL", async () => { + const transport = vi.fn(async (request) => { + expect(request.url).toBe("https://render.test/start"); + expect(request.waitMs).toBe(1_250); + return renderedPage({ + html: '

Rendered page

Gateway content.

Next', + }); + }); + const options = { + ...publicTarget, + browserGateway: { transport }, + }; + + await expect( + fetchWebPage( + { + url: "https://render.test/start", + render: "browser", + waitMs: 1_250, + mode: "full", + }, + undefined, + options, + ), + ).resolves.toEqual({ + url: "https://render.test/start", + mode: "full", + content: "Gateway content.\n", + truncated: false, + }); + + await expect( + fetchWebLinks( + { + url: "https://render.test/start", + render: "browser", + waitMs: 1_250, + }, + undefined, + options, + ), + ).resolves.toEqual({ + url: "https://render.test/start", + links: [{ text: "Next", url: "https://render.test/docs/next" }], + truncated: false, + }); + expect(transport).toHaveBeenCalledTimes(2); + }); + + it("posts the bounded raw-render contract to the configured gateway", async () => { + let receivedURL = ""; + let receivedInit: RequestInit | undefined; + const result = await fetchWebPage( + { + url: "https://render.test/page", + render: "browser", + waitMs: 0, + mode: "full", + }, + undefined, + { + ...publicTarget, + browserGateway: { + baseUrl: "http://browser-gateway", + fetch: async (url, init) => { + receivedURL = String(url); + receivedInit = init; + return Response.json(renderedPage()); + }, + }, + }, + ); + + expect(receivedURL).toBe( + `http://browser-gateway${BROWSER_GATEWAY_RENDER_PATH}`, + ); + expect(receivedInit?.method).toBe("POST"); + expect(new Headers(receivedInit?.headers).get("accept")).toBe( + "application/json", + ); + expect(JSON.parse(String(receivedInit?.body))).toEqual({ + url: "https://render.test/page", + waitMs: 0, + }); + expect(result.content).toBe("Gateway content.\n"); + }); + + it("fails explicitly when the gateway is missing, overloaded, unreachable, or malformed", async () => { + await expect( + fetchWebPage( + { url: "https://render.test/page", render: "browser", waitMs: 0 }, + undefined, + { ...publicTarget, browserGateway: {} }, + ), + ).rejects.toMatchObject({ code: "render_unavailable" }); + + for (const status of [429, 503]) { + await expect( + fetchWebPage( + { url: "https://render.test/page", render: "browser", waitMs: 0 }, + undefined, + { + ...publicTarget, + browserGateway: { + baseUrl: "http://browser-gateway", + fetch: async () => new Response("busy", { status }), + }, + }, + ), + ).rejects.toMatchObject({ code: "render_unavailable" }); + } + + await expect( + fetchWebPage( + { url: "https://render.test/page", render: "browser", waitMs: 0 }, + undefined, + { + ...publicTarget, + browserGateway: { + baseUrl: "http://browser-gateway", + fetch: async () => new Response("upstream", { status: 502 }), + }, + }, + ), + ).rejects.toMatchObject({ code: "render_failed" }); + + await expect( + fetchWebPage( + { url: "https://render.test/page", render: "browser", waitMs: 0 }, + undefined, + { + ...publicTarget, + browserGateway: { + baseUrl: "http://browser-gateway", + fetch: async () => new Response("not-json"), + }, + }, + ), + ).rejects.toMatchObject({ code: "render_invalid_output" }); + + await expect( + fetchWebPage( + { url: "https://render.test/page", render: "browser", waitMs: 0 }, + undefined, + { + ...publicTarget, + browserGateway: { + baseUrl: "http://browser-gateway", + fetch: async () => Response.json({ html: "missing url" }), + }, + }, + ), + ).rejects.toMatchObject({ code: "render_invalid_output" }); + + await expect( + fetchWebPage( + { url: "https://render.test/page", render: "browser", waitMs: 0 }, + undefined, + { + ...publicTarget, + browserGateway: { + baseUrl: "http://browser-gateway", + fetch: async () => { + throw new Error("connection refused"); + }, + }, + }, + ), + ).rejects.toMatchObject({ code: "render_unavailable" }); + }); + + it("preserves caller cancellation through gateway transport", async () => { + const controller = new AbortController(); + let sawSignal = false; + let started = false; + const pending = fetchWebPage( + { url: "https://render.test/page", render: "browser", waitMs: 0 }, + controller.signal, + { + ...publicTarget, + browserGateway: { + transport: async ({ signal }) => { + started = true; + return new Promise((_resolve, reject) => { + signal?.addEventListener( + "abort", + () => { + sawSignal = true; + reject(new OperationAbortedError()); + }, + { once: true }, + ); + }); + }, + }, + }, + ); + for (let attempt = 0; attempt < 100 && !started; attempt++) + await new Promise((resolve) => setTimeout(resolve, 1)); + controller.abort(); + await expect(pending).rejects.toMatchObject({ + name: "OperationAbortedError", + }); + expect(sawSignal).toBe(true); + }); + + it("aborts the gateway HTTP request when the caller cancels", async () => { + const controller = new AbortController(); + let sawAbort = false; + const pending = fetchWebPage( + { url: "https://93.184.216.34/page", render: "browser", waitMs: 0 }, + controller.signal, + { + browserGateway: { + baseUrl: "http://browser-gateway", + fetch: async (_url, init) => + new Promise((_resolve, reject) => { + init?.signal?.addEventListener( + "abort", + () => { + sawAbort = true; + reject(new DOMException("aborted", "AbortError")); + }, + { once: true }, + ); + }), + }, + }, + ); + await new Promise((resolve) => setTimeout(resolve, 0)); + controller.abort(); + await expect(pending).rejects.toMatchObject({ + name: "OperationAbortedError", + }); + expect(sawAbort).toBe(true); + }); + + it("bounds a pending gateway request", async () => { + vi.useFakeTimers(); + let sawAbort = false; + const pending = fetchWebPage( + { url: "https://93.184.216.34/page", render: "browser", waitMs: 0 }, + undefined, + { + browserGateway: { + baseUrl: "http://browser-gateway", + timeoutMs: 50, + fetch: async (_url, init) => + new Promise((_resolve, reject) => { + init?.signal?.addEventListener( + "abort", + () => { + sawAbort = true; + reject(new DOMException("aborted", "AbortError")); + }, + { once: true }, + ); + }), + }, + }, + ); + try { + const outcome = pending.then( + () => undefined, + (error: unknown) => error, + ); + await vi.advanceTimersByTimeAsync(50); + await expect(outcome).resolves.toMatchObject({ + code: "render_timed_out", + }); + expect(sawAbort).toBe(true); + expect(BROWSER_GATEWAY_TIMEOUT_MS).toBe(50_000); + } finally { + vi.useRealTimers(); + } + }); +}); diff --git a/packages/web/src/http.ts b/packages/web/src/http.ts index 85afa17..72b6b98 100644 --- a/packages/web/src/http.ts +++ b/packages/web/src/http.ts @@ -10,6 +10,7 @@ import { throwIfAborted, validateKeposBridgeEndpoint, type SearchResponse, + type BrowserGatewayTransport, type WebCredentials, type WebOperations, } from "@guionai/web-core"; @@ -24,6 +25,12 @@ export type HttpServiceDependencies = { operations?: WebOperations; credentials?: WebCredentials | (() => WebCredentials); keposBridgeEndpoint?: string; + /** Server-local origin for the internal Browser Rendering Gateway. */ + browserGatewayUrl?: string; + /** Test-owned raw-render transport; production uses the configured origin. */ + browserGatewayTransport?: BrowserGatewayTransport; + /** Test-owned HTTP implementation for the gateway request. */ + browserGatewayFetch?: typeof globalThis.fetch; /** Injectable environment for startup/configuration tests. */ environment?: NodeJS.ProcessEnv; /** Used by build-time OpenAPI generation to skip runtime credential checks. */ @@ -34,6 +41,7 @@ export type HttpServiceState = { operations: WebOperations; credentials: WebCredentials; keposBridgeEndpoint: string; + browserGatewayUrl?: string; /** Server-local override; undefined keeps the Bridge-to-Exa default. */ searchProvider?: "deepseek"; }; @@ -203,7 +211,7 @@ const fetchRoute = createRoute({ operationId: "fetch", summary: "Fetch a web page", description: - 'Fetch through HTTP by default with auto navigation; use mode "full" or "tree" for explicit navigation. Supply section_id with omitted mode or mode "auto" to retrieve a section. Browser rendering requires render=browser and waitMs.', + 'Fetch through HTTP by default with auto navigation; use mode "full" or "tree" for explicit navigation. Supply section_id with omitted mode or mode "auto" to retrieve a section. Browser rendering requires render=browser and waitMs and uses the server-local Browser Rendering Gateway.', request: jsonRequest(FetchRequestSchema), responses: { 200: jsonResponse(FetchResponseSchema, "Fetched page."), @@ -217,7 +225,7 @@ const linksRoute = createRoute({ operationId: "links", summary: "List page links", description: - "List HTTP(S) anchors using HTTP rendering by default or explicit browser rendering.", + "List HTTP(S) anchors using HTTP rendering by default or explicit browser rendering through the server-local Browser Rendering Gateway.", request: jsonRequest(LinksRequestSchema), responses: { 200: jsonResponse(LinksResponseSchema, "Page links."), @@ -348,10 +356,24 @@ export function resolveHttpServiceState( dependencies.keposBridgeEndpoint ?? environment.KEPOS_BRIDGE_ENDPOINT ?? DEFAULT_KEPOS_BRIDGE_ENDPOINT; + const browserGatewayUrl = + dependencies.browserGatewayUrl ?? environment.BROWSER_GATEWAY_URL; + const browserGateway = { + ...(browserGatewayUrl === undefined ? {} : { baseUrl: browserGatewayUrl }), + ...(dependencies.browserGatewayTransport === undefined + ? {} + : { transport: dependencies.browserGatewayTransport }), + ...(dependencies.browserGatewayFetch === undefined + ? {} + : { fetch: dependencies.browserGatewayFetch }), + }; return { - operations: dependencies.operations ?? webCoreModule.createWebOperations(), + operations: + dependencies.operations ?? + webCoreModule.createWebOperations({ browserGateway }), credentials, keposBridgeEndpoint: validateKeposBridgeEndpoint(endpoint), + ...(browserGatewayUrl === undefined ? {} : { browserGatewayUrl }), ...(searchProvider === undefined ? {} : { searchProvider }), }; } diff --git a/packages/web/src/program.ts b/packages/web/src/program.ts index 423e84c..649f018 100644 --- a/packages/web/src/program.ts +++ b/packages/web/src/program.ts @@ -56,7 +56,6 @@ function createServeCommand(dependencies: ProgramDependencies): Command { .action((options: { host: string; port: number }) => { startHttpServer( { - operations: dependencies.operations, credentials: dependencies.credentials, }, { diff --git a/packages/web/test/http.test.ts b/packages/web/test/http.test.ts index a06c94d..79f51c2 100644 --- a/packages/web/test/http.test.ts +++ b/packages/web/test/http.test.ts @@ -325,6 +325,61 @@ describe("personal HTTP service", () => { ); }); + it("delegates default HTTP-service browser rendering to the gateway transport", async () => { + const transport = vi.fn(async ({ url, waitMs }) => ({ + url: "https://93.184.216.34/final", + html: `

Gateway ${url} waited ${waitMs}.

`, + })); + const app = createHttpApp({ + credentials: { exaApiKey: "exa-secret" }, + browserGatewayTransport: transport, + }); + + const fetched = await json(app, "/api/v1/web/fetch", { + url: "https://93.184.216.34/page", + render: "browser", + waitMs: 125, + mode: "full", + }); + expect(fetched.response.status).toBe(200); + expect(fetched.body).toEqual({ + url: "https://93.184.216.34/page", + mode: "full", + content: "Gateway https://93.184.216.34/page waited 125.\n", + truncated: false, + }); + + const linked = await json(app, "/api/v1/web/links", { + url: "https://93.184.216.34/page", + render: "browser", + waitMs: 0, + }); + expect(linked.response.status).toBe(200); + expect(transport).toHaveBeenCalledTimes(2); + expect(transport).toHaveBeenNthCalledWith( + 2, + expect.objectContaining({ + url: "https://93.184.216.34/page", + waitMs: 0, + }), + ); + }); + + it("keeps browser rendering explicit when the HTTP-service gateway is absent", async () => { + const app = createHttpApp({ credentials: { exaApiKey: "exa-secret" } }); + const result = await json(app, "/api/v1/web/fetch", { + url: "https://93.184.216.34/page", + render: "browser", + waitMs: 0, + mode: "full", + }); + expect(result.response.status).toBe(502); + expect(result.body).toEqual({ + code: "render_unavailable", + message: "fetch requires an explicit capability retry", + }); + }); + it("rejects invalid search and fetch requests before an operation", async () => { const ops = operations(); const app = createHttpApp({ ...dependencies(), operations: ops }); From 84abdd73b0b55031518ff857e945784424917148 Mon Sep 17 00:00:00 2001 From: neil Date: Fri, 4 Sep 2026 12:24:49 +0800 Subject: [PATCH 2/4] chore(http): add implementation report --- .../implementation-report.md | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 .scratch/browser-gateway-renderer/implementation-report.md diff --git a/.scratch/browser-gateway-renderer/implementation-report.md b/.scratch/browser-gateway-renderer/implementation-report.md new file mode 100644 index 0000000..1ee38e1 --- /dev/null +++ b/.scratch/browser-gateway-renderer/implementation-report.md @@ -0,0 +1,81 @@ +# Browser gateway renderer implementation report + +## Result + +The whole `browser-gateway-renderer` spec and its only ticket are implemented +on the `browser-gateway-renderer` branch. The containerized HTTP Service now +delegates explicit browser rendering to the server-local Browser Rendering +Gateway (`POST /api/render`), while Web Core continues to own HTML extraction, +navigation, links, and content limits. CLI, MCP, Pi, and DSH keep the existing +direct `agent-browser` renderer. + +## Acceptance criteria + +- [x] Configured gateway Fetch and Links requests send `{ url, waitMs }`, use + returned raw DOM and final URL, preserve navigation/link response contracts, + and accept `waitMs` from 0 through 30,000. +- [x] Missing, overloaded, unreachable, timed-out, malformed, oversized, or + failed gateway work is translated to explicit `render_*` capability errors; + cancellation propagates and HTTP rendering remains independent. +- [x] The production image uses the slim Node runtime without Chromium or + `agent-browser`; direct browser users outside the HTTP image retain their + existing capability. +- [x] README, HTTP-service/operator documentation, glossary, Dockerfile + comments, release documentation, and ADRs describe the gateway boundary and + `BROWSER_GATEWAY_URL` configuration. +- [x] Tests cover the fetch transport seam, HTTP routes, failure/cancellation + behavior, and the image contract without a live browser, cluster, + credentials, or production service. + +## Verification + +All checks completed successfully from the final implementation: + +- `pnpm typecheck` +- `pnpm build` +- `pnpm test` — 20 files, 152 tests passed +- `pnpm test:release` +- `pnpm test:pack` — web, Pi, and DSH package smoke checks passed +- `pnpm format:check` +- `git diff --check` +- `docker build --tag guionai-web:browser-gateway-test .` +- Test-owned gateway plus image smoke request — HTTP Fetch returned rendered + content and the gateway log confirmed `/api/render` with `{ url, waitMs }`. +- Runtime binary probe — `no-browser-binaries` for `agent-browser`, Chromium, + and Google Chrome. + +## Fixed-point LOC accounting + +The fixed point is `ed74871`. Generated files and lockfiles are excluded. +Actual additions and deletions are: + +| Category | Additions | Deletions | +| --- | ---: | ---: | +| Product code | 341 | 9 | +| Tests | 356 | 0 | +| Configuration and docs | 91 | 34 | +| **Total** | **788** | **43** | + +The total (831 changed lines) is within the spec estimate of 570–960. Product +code and test additions are near the upper half because the gateway boundary +includes bounded transport, response validation, cancellation, timeout/error +translation, and HTTP plus package-level seams; no compatibility layer or +unfinished infrastructure was added. + +## Commits and scope + +- `9c9a088 feat(http): delegate container browser rendering to gateway` +- Changed implementation paths: `packages/web-core/src/`, + `packages/web-core/test/`, `packages/web/src/http.ts`, + `packages/web/src/program.ts`, `Dockerfile`, `README.md`, `CONTEXT.md`, + `docs/http-service.md`, and `docs/adr/`. + +The required report is intentionally kept under `.scratch` and is excluded +from the implementation LOC table above. + +## Remaining operational boundary + +The separately deployed Browser Rendering Gateway must be reachable and +configured through `BROWSER_GATEWAY_URL`; its deployment, capacity, proxy, +authentication, and rollout remain outside this repository. Code review and +deployment are excluded from this implementation task. From aec662e8cf340985f19354c2d6dbe2a3840a81cf Mon Sep 17 00:00:00 2001 From: neil Date: Fri, 4 Sep 2026 12:40:20 +0800 Subject: [PATCH 3/4] fix(http): keep gateway rendering image-only --- .github/workflows/ci.yaml | 3 + .github/workflows/release.yaml | 5 +- CONTEXT.md | 9 +- Dockerfile | 5 +- README.md | 11 +- .../0003-http-browser-rendering-gateway.md | 19 +- docs/http-service.md | 19 +- package.json | 1 + packages/web/src/http.ts | 12 +- packages/web/src/program.ts | 1 + packages/web/test/http.test.ts | 43 +++- scripts/test-image-contract.mjs | 193 ++++++++++++++++++ 12 files changed, 291 insertions(+), 30 deletions(-) create mode 100644 scripts/test-image-contract.mjs diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 680ce35..a282155 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -46,3 +46,6 @@ jobs: - name: Packed installation smoke tests run: pnpm test:pack + + - name: Container image browser-gateway contract + run: pnpm test:image diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 172eb0b..ee1fea7 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -56,6 +56,9 @@ jobs: - name: Packed installation smoke tests run: pnpm test:pack + - name: Container image browser-gateway contract + run: pnpm test:image + - name: Upload built package artifacts uses: actions/upload-artifact@v4 with: @@ -90,7 +93,7 @@ jobs: echo '| Packages | `@guionai/web, @guionai/pi-web, @guionai/dsh-web` |' printf '| Synchronized version | `%s` |\n' "$version" printf '| npm dist-tag | `%s` |\n' "$dist_tag" - echo "| Checks | format, typecheck, build, tests, release version invariants, packed smoke |" + echo "| Checks | format, typecheck, build, tests, release version invariants, packed smoke, image contract |" } >> "$GITHUB_STEP_SUMMARY" publish: diff --git a/CONTEXT.md b/CONTEXT.md index e5585f2..7c22753 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -59,10 +59,11 @@ The Hono-based `/api/v1/web` JSON API shipped by `web serve` and the GHCR image. uses server-local credentials, Bridge Route, and optional DeepSeek provider configuration; clients do not select providers or submit a generic Bridge command. Its page-reading routes use -the same `render: "http" | "browser"`, `mode`, and `section_id` contract. Browser -rendering is delegated to the -server-local Browser Rendering Gateway configured by the operator; the browser -executable name appears only in gateway-side setup. +the same `render: "http" | "browser"`, `mode`, and `section_id` contract. The +GHCR image sets `GUIONAI_HTTP_IMAGE=1` and delegates browser rendering to the +server-local Browser Rendering Gateway configured by the operator; a local/npm +server with direct operations retains its direct browser capability. The +browser executable name appears only in gateway-side setup for the image. _Avoid_: Remote MCP, public service **Gateway Web API prefix**: diff --git a/Dockerfile b/Dockerfile index 80542fa..757f090 100644 --- a/Dockerfile +++ b/Dockerfile @@ -17,9 +17,12 @@ RUN pnpm --filter @guionai/web run build FROM node:24-bookworm-slim ENV NODE_ENV=production +# The image entrypoint opts into the gateway-only browser path. Local npm +# installs leave this marker unset and keep their supplied direct operations. +ENV GUIONAI_HTTP_IMAGE=1 WORKDIR /app -# `BROWSER_GATEWAY_URL` is optional at startup; browser requests fail +# `BROWSER_GATEWAY_URL` is optional at startup; image browser requests fail # explicitly until the operator points the service at the internal gateway. COPY --from=build /workspace/packages/web/dist ./dist diff --git a/README.md b/README.md index ab2363f..8be0eed 100644 --- a/README.md +++ b/README.md @@ -104,7 +104,8 @@ cancellation is reported as 499. Fetch and Links use HTTP rendering when `render` is omitted (or set to `"http"`). Browser rendering is explicit and requires both `render: "browser"` and an integer `waitMs` from 0 through 30,000; HTTP rendering never silently -switches backends. The `web serve` HTTP Service sends browser requests to the +switches backends. A local/npm `web serve` keeps its supplied direct operations. +The GHCR image sets `GUIONAI_HTTP_IMAGE=1` and sends browser requests to the server-local Browser Rendering Gateway configured by `BROWSER_GATEWAY_URL`. The GHCR image contains no Chromium or `agent-browser`; an absent, unreachable, overloaded, or failed gateway returns an explicit browser-render failure while @@ -312,18 +313,22 @@ pnpm build pnpm test pnpm test:release pnpm test:pack +pnpm test:image ``` `test:release` uses disposable manifests to exercise tag-version synchronization. `test:pack` runs each public package's packed installation or -host-loading contract in test-owned temporary directories. +host-loading contract in test-owned temporary directories. `test:image` builds a +test-owned disposable Docker image, runs it against a fake `/api/render` gateway, +and verifies the image has no browser executable. ## Releases A `v` tag is the release source of truth for all three public packages: `@guionai/web`, `@guionai/pi-web`, and `@guionai/dsh-web`. The release preflight synchronizes its checkout manifests from that tag, then completes formatting, -typechecking, build, tests, release-version checks, and packed smoke tests +typechecking, build, tests, release-version checks, packed smoke tests, and the +Docker image contract before any publication begins. Three independent, non-fail-fast protected `npm` Environment matrix cells then diff --git a/docs/adr/0003-http-browser-rendering-gateway.md b/docs/adr/0003-http-browser-rendering-gateway.md index b8477b1..2db97d3 100644 --- a/docs/adr/0003-http-browser-rendering-gateway.md +++ b/docs/adr/0003-http-browser-rendering-gateway.md @@ -15,17 +15,20 @@ startup, memory, and lifecycle cost. ## Decision -Only the containerized HTTP Service delegates explicit browser rendering to -the server-local Browser Rendering Gateway. It sends `POST /api/render` with +Only the containerized HTTP Service, explicitly marked with +`GUIONAI_HTTP_IMAGE=1`, delegates explicit browser rendering to the server-local +Browser Rendering Gateway. It sends `POST /api/render` with `{ "url", "waitMs" }` and receives raw rendered `{ "html", "url" }`. The existing Web Core then performs target validation, extraction, navigation, and link handling, so Fetch and Links keep their public contracts. `waitMs` remains caller-visible and is required from 0 through 30,000; browser rendering never falls back to HTTP. -`BROWSER_GATEWAY_URL` is server-local configuration. Missing configuration, -gateway overload, transport failure, timeout, or malformed output becomes an -explicit `render_*` capability failure. HTTP rendering remains independent. +`BROWSER_GATEWAY_URL` is image-only server-local configuration. Missing +configuration, gateway overload, transport failure, timeout, or malformed output +becomes an explicit `render_*` capability failure. HTTP rendering remains +independent. Local/npm `web serve` instances leave the image marker unset and +retain their supplied direct operations. CLI, MCP, Pi, and DSH continue to use the shared Web Core's direct `agent-browser` capability. The GHCR image therefore contains neither @@ -34,8 +37,8 @@ boundary. ## Consequences -The HTTP-service image is smaller and does not launch a local browser. Browser -requests require the in-cluster gateway to be reachable, and operators must -configure `BROWSER_GATEWAY_URL` before using them. Gateway deployment, +The HTTP-service image is smaller and does not launch a local browser. Image +browser requests require the in-cluster gateway to be reachable, and operators +must configure `BROWSER_GATEWAY_URL` before using them. Gateway deployment, capacity, authentication, and rollout remain outside this repository and are owned by the separate browser-gateway service. diff --git a/docs/http-service.md b/docs/http-service.md index 392316a..6c78eb5 100644 --- a/docs/http-service.md +++ b/docs/http-service.md @@ -72,10 +72,13 @@ HTTP rendering fetches the page with Node HTTP, linkedom, and Defuddle. Browser rendering delegates the raw DOM request to the internal Browser Rendering Gateway; the gateway's browser executable is not a public request value. Configure its origin with the server-local `BROWSER_GATEWAY_URL` -environment variable. Browser rendering is never selected automatically, and -the service does not fall back between renderers. If the URL is missing or the -gateway rejects, times out, or returns an invalid response, the operation -fails explicitly with a `render_*` capability error. +environment variable when running the GHCR image. A local/npm `web serve` with +supplied direct operations keeps its direct browser capability; the image sets +`GUIONAI_HTTP_IMAGE=1` to select the gateway-only path. Browser rendering is +never selected automatically, and the service does not fall back between +renderers. If gateway configuration is missing or the gateway rejects, times +out, or returns an invalid response, the image operation fails explicitly with +a `render_*` capability error. The shared module owns the 5,000-character automatic-tree policy. An `"auto"` request for an unsectioned document longer than that threshold with navigable @@ -158,11 +161,13 @@ response bodies. ## Configuration and OpenAPI -Credentials, the optional `KEPOS_BRIDGE_ENDPOINT`, the optional +Credentials, the optional `KEPOS_BRIDGE_ENDPOINT`, the optional image-only `BROWSER_GATEWAY_URL`, and the optional server-local `WEB_SEARCH_PROVIDER=deepseek` selection are environment variables. The -gateway URL is a base URL; the service calls its `POST /api/render` raw-render -operation with `{ "url", "waitMs" }`. They are not accepted in request bodies. +image sets `GUIONAI_HTTP_IMAGE=1`; local/npm servers leave that marker unset and +retain their direct operations. The gateway URL is a base URL; image mode calls +its `POST /api/render` raw-render operation with `{ "url", "waitMs" }`. They +are not accepted in request bodies. DeepSeek performs one auxiliary model call per search; callers receive only normalized results and do not need to know the Messages/tool wire protocol. The route diff --git a/package.json b/package.json index 238cfb2..41fd5d4 100644 --- a/package.json +++ b/package.json @@ -11,6 +11,7 @@ "build": "pnpm -r --filter './packages/*' run build", "typecheck": "tsc -p tsconfig.json --noEmit", "test": "vitest run", + "test:image": "node scripts/test-image-contract.mjs", "test:release": "node scripts/test-release-invariants.mjs", "test:pack": "pnpm --filter @guionai/web run test:pack && pnpm --filter @guionai/pi-web run test:pack && pnpm --filter @guionai/dsh-web run test:pack", "format": "prettier --write README.md package.json pnpm-workspace.yaml tsconfig.json \"packages/**/*.{ts,tsx,js,mjs,cjs,json,yml,yaml}\" \"scripts/**/*.{mjs,ts}\" \".github/**/*.{yml,yaml}\"", diff --git a/packages/web/src/http.ts b/packages/web/src/http.ts index 72b6b98..b5331b3 100644 --- a/packages/web/src/http.ts +++ b/packages/web/src/http.ts @@ -25,6 +25,8 @@ export type HttpServiceDependencies = { operations?: WebOperations; credentials?: WebCredentials | (() => WebCredentials); keposBridgeEndpoint?: string; + /** Selects the browserless image renderer instead of supplied operations. */ + imageMode?: boolean; /** Server-local origin for the internal Browser Rendering Gateway. */ browserGatewayUrl?: string; /** Test-owned raw-render transport; production uses the configured origin. */ @@ -41,7 +43,6 @@ export type HttpServiceState = { operations: WebOperations; credentials: WebCredentials; keposBridgeEndpoint: string; - browserGatewayUrl?: string; /** Server-local override; undefined keeps the Bridge-to-Exa default. */ searchProvider?: "deepseek"; }; @@ -356,6 +357,8 @@ export function resolveHttpServiceState( dependencies.keposBridgeEndpoint ?? environment.KEPOS_BRIDGE_ENDPOINT ?? DEFAULT_KEPOS_BRIDGE_ENDPOINT; + const imageMode = + dependencies.imageMode ?? environment.GUIONAI_HTTP_IMAGE === "1"; const browserGatewayUrl = dependencies.browserGatewayUrl ?? environment.BROWSER_GATEWAY_URL; const browserGateway = { @@ -368,12 +371,11 @@ export function resolveHttpServiceState( : { fetch: dependencies.browserGatewayFetch }), }; return { - operations: - dependencies.operations ?? - webCoreModule.createWebOperations({ browserGateway }), + operations: imageMode + ? webCoreModule.createWebOperations({ browserGateway }) + : (dependencies.operations ?? webCoreModule.createWebOperations()), credentials, keposBridgeEndpoint: validateKeposBridgeEndpoint(endpoint), - ...(browserGatewayUrl === undefined ? {} : { browserGatewayUrl }), ...(searchProvider === undefined ? {} : { searchProvider }), }; } diff --git a/packages/web/src/program.ts b/packages/web/src/program.ts index 649f018..423e84c 100644 --- a/packages/web/src/program.ts +++ b/packages/web/src/program.ts @@ -56,6 +56,7 @@ function createServeCommand(dependencies: ProgramDependencies): Command { .action((options: { host: string; port: number }) => { startHttpServer( { + operations: dependencies.operations, credentials: dependencies.credentials, }, { diff --git a/packages/web/test/http.test.ts b/packages/web/test/http.test.ts index 79f51c2..56609b3 100644 --- a/packages/web/test/http.test.ts +++ b/packages/web/test/http.test.ts @@ -332,6 +332,7 @@ describe("personal HTTP service", () => { })); const app = createHttpApp({ credentials: { exaApiKey: "exa-secret" }, + imageMode: true, browserGatewayTransport: transport, }); @@ -366,7 +367,10 @@ describe("personal HTTP service", () => { }); it("keeps browser rendering explicit when the HTTP-service gateway is absent", async () => { - const app = createHttpApp({ credentials: { exaApiKey: "exa-secret" } }); + const app = createHttpApp({ + credentials: { exaApiKey: "exa-secret" }, + imageMode: true, + }); const result = await json(app, "/api/v1/web/fetch", { url: "https://93.184.216.34/page", render: "browser", @@ -380,6 +384,43 @@ describe("personal HTTP service", () => { }); }); + it("keeps supplied direct browser operations for a normal server", async () => { + const direct = operations({ + fetch: vi.fn(async (input) => ({ + url: input.url, + mode: "full" as const, + content: "Direct browser fixture.\n", + truncated: false, + })), + }); + const app = createHttpApp({ + operations: direct, + credentials: { exaApiKey: "exa-secret" }, + environment: {}, + }); + const result = await json(app, "/api/v1/web/fetch", { + url: "https://93.184.216.34/page", + render: "browser", + waitMs: 0, + mode: "full", + }); + expect(result.response.status).toBe(200); + expect(result.body).toEqual({ + url: "https://93.184.216.34/page", + mode: "full", + content: "Direct browser fixture.\n", + truncated: false, + }); + expect(direct.fetch).toHaveBeenCalledWith( + expect.objectContaining({ + url: "https://93.184.216.34/page", + render: "browser", + waitMs: 0, + }), + expect.any(AbortSignal), + ); + }); + it("rejects invalid search and fetch requests before an operation", async () => { const ops = operations(); const app = createHttpApp({ ...dependencies(), operations: ops }); diff --git a/scripts/test-image-contract.mjs b/scripts/test-image-contract.mjs new file mode 100644 index 0000000..9a5340f --- /dev/null +++ b/scripts/test-image-contract.mjs @@ -0,0 +1,193 @@ +import { execFile } from "node:child_process"; +import { createServer } from "node:http"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify } from "node:util"; + +const execFileAsync = promisify(execFile); +const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const image = `guionai-web:image-contract-${process.pid}-${Date.now()}`; +const gatewayRequests = []; +let containerID; +let gatewayServer; +let gatewayListening = false; + +async function docker(args) { + return execFileAsync("docker", args, { + cwd: repositoryRoot, + maxBuffer: 16 * 1024 * 1024, + }); +} + +async function ignoreDocker(args) { + try { + await docker(args); + } catch { + // Cleanup is best effort after the test-owned resources are identified. + } +} + +function closeServer(server) { + return new Promise((resolve, reject) => { + server.close((error) => (error ? reject(error) : resolve())); + }); +} + +async function findFreePort() { + const probe = createServer(); + await new Promise((resolve, reject) => { + probe.once("error", reject); + probe.listen(0, "127.0.0.1", resolve); + }); + const address = probe.address(); + if (!address || typeof address === "string") { + await closeServer(probe); + throw new Error("could not reserve an HTTP service port"); + } + const port = address.port; + await closeServer(probe); + return port; +} + +async function waitForService(port) { + let lastError; + for (let attempt = 0; attempt < 60; attempt += 1) { + try { + const response = await fetch(`http://127.0.0.1:${port}/`); + const status = response.status; + await response.arrayBuffer(); + if (status === 404) return; + lastError = new Error(`readiness returned HTTP ${status}`); + } catch (error) { + lastError = error; + } + await new Promise((resolve) => setTimeout(resolve, 250)); + } + throw new Error(`HTTP image did not become ready: ${String(lastError)}`); +} + +try { + gatewayServer = createServer((request, response) => { + if (request.method !== "POST" || request.url !== "/api/render") { + response.statusCode = 404; + response.end(); + return; + } + let body = ""; + request.setEncoding("utf8"); + request.on("data", (chunk) => { + body += chunk; + }); + request.on("end", () => { + let payload; + try { + payload = JSON.parse(body); + } catch { + response.statusCode = 400; + response.end("invalid JSON"); + return; + } + gatewayRequests.push(payload); + response.writeHead(200, { "content-type": "application/json" }); + response.end( + JSON.stringify({ + html: "

Docker gateway fixture.

", + url: "https://93.184.216.34/final", + }), + ); + }); + }); + await new Promise((resolve, reject) => { + gatewayServer.once("error", reject); + gatewayServer.listen(0, "127.0.0.1", resolve); + }); + const address = gatewayServer.address(); + if (!address || typeof address === "string") + throw new Error("fake gateway did not expose a TCP port"); + gatewayListening = true; + const servicePort = await findFreePort(); + + await docker(["build", "--tag", image, "."]); + const started = await docker([ + "run", + "--detach", + "--rm", + "--network", + "host", + "-e", + "EXA_API_KEY=image-contract", + "-e", + "GUIONAI_HTTP_IMAGE=1", + "-e", + `BROWSER_GATEWAY_URL=http://127.0.0.1:${address.port}`, + image, + "--port", + String(servicePort), + ]); + containerID = started.stdout.trim(); + if (!containerID) throw new Error("docker run did not return a container ID"); + + await waitForService(servicePort); + const response = await fetch( + `http://127.0.0.1:${servicePort}/api/v1/web/fetch`, + { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + url: "https://93.184.216.34/page", + render: "browser", + waitMs: 125, + mode: "full", + }), + signal: AbortSignal.timeout(10_000), + }, + ); + const result = await response.json(); + if ( + response.status !== 200 || + JSON.stringify(result) !== + JSON.stringify({ + url: "https://93.184.216.34/page", + mode: "full", + content: "Docker gateway fixture.\n", + truncated: false, + }) + ) + throw new Error( + `image browser Fetch contract failed: HTTP ${response.status} ${JSON.stringify(result)}`, + ); + if ( + gatewayRequests.length !== 1 || + gatewayRequests[0].url !== "https://93.184.216.34/page" || + gatewayRequests[0].waitMs !== 125 + ) + throw new Error( + `fake gateway received an unexpected request: ${JSON.stringify(gatewayRequests)}`, + ); + + await docker([ + "run", + "--rm", + "--entrypoint", + "sh", + image, + "-c", + 'for binary in agent-browser chromium chromium-browser google-chrome google-chrome-stable; do if command -v "$binary" >/dev/null 2>&1; then echo "$binary is installed" >&2; exit 1; fi; done', + ]); + console.log("Docker image browser-gateway contract passed"); +} catch (error) { + if (containerID) { + try { + const logs = await docker(["logs", containerID]); + if (logs.stdout || logs.stderr) + console.error(`${logs.stdout ?? ""}${logs.stderr ?? ""}`); + } catch { + // Preserve the original test failure when logs are unavailable. + } + } + throw error; +} finally { + if (containerID) await ignoreDocker(["rm", "--force", containerID]); + await ignoreDocker(["rmi", "--force", image]); + if (gatewayServer && gatewayListening) await closeServer(gatewayServer); +} From 00bebbc58188ebade1c7a8f4f765e9295400db7e Mon Sep 17 00:00:00 2001 From: neil Date: Fri, 4 Sep 2026 12:41:18 +0800 Subject: [PATCH 4/4] chore(http): update review implementation report --- .../implementation-report.md | 35 ++++++++++++------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/.scratch/browser-gateway-renderer/implementation-report.md b/.scratch/browser-gateway-renderer/implementation-report.md index 1ee38e1..ff0a819 100644 --- a/.scratch/browser-gateway-renderer/implementation-report.md +++ b/.scratch/browser-gateway-renderer/implementation-report.md @@ -7,13 +7,18 @@ on the `browser-gateway-renderer` branch. The containerized HTTP Service now delegates explicit browser rendering to the server-local Browser Rendering Gateway (`POST /api/render`), while Web Core continues to own HTML extraction, navigation, links, and content limits. CLI, MCP, Pi, and DSH keep the existing -direct `agent-browser` renderer. +direct `agent-browser` renderer. Review fixes make the boundary image-only: +`GUIONAI_HTTP_IMAGE=1` is set by the Docker runtime, while a normal npm/local +`web serve` preserves its supplied direct operations. ## Acceptance criteria - [x] Configured gateway Fetch and Links requests send `{ url, waitMs }`, use returned raw DOM and final URL, preserve navigation/link response contracts, and accept `waitMs` from 0 through 30,000. +- [x] Gateway delegation is limited to the explicitly marked image path; + normal npm/local servers with direct operations retain direct browser + rendering. - [x] Missing, overloaded, unreachable, timed-out, malformed, oversized, or failed gateway work is translated to explicit `render_*` capability errors; cancellation propagates and HTTP rendering remains independent. @@ -33,9 +38,11 @@ All checks completed successfully from the final implementation: - `pnpm typecheck` - `pnpm build` -- `pnpm test` — 20 files, 152 tests passed +- `pnpm test` — 20 files, 153 tests passed - `pnpm test:release` - `pnpm test:pack` — web, Pi, and DSH package smoke checks passed +- `pnpm test:image` — builds a disposable image, runs a fake `/api/render`, + verifies browser Fetch, and probes browser binaries are absent - `pnpm format:check` - `git diff --check` - `docker build --tag guionai-web:browser-gateway-test .` @@ -51,24 +58,26 @@ Actual additions and deletions are: | Category | Additions | Deletions | | --- | ---: | ---: | -| Product code | 341 | 9 | -| Tests | 356 | 0 | -| Configuration and docs | 91 | 34 | -| **Total** | **788** | **43** | - -The total (831 changed lines) is within the spec estimate of 570–960. Product -code and test additions are near the upper half because the gateway boundary -includes bounded transport, response validation, cancellation, timeout/error -translation, and HTTP plus package-level seams; no compatibility layer or +| Product code | 343 | 8 | +| Tests | 590 | 0 | +| Configuration and docs | 118 | 37 | +| **Total** | **1,051** | **45** | + +The total (1,096 changed lines) exceeds the original 570–960 estimate because +the review required a repeatable 193-line Docker contract harness, explicit +image-mode selection, CI/release invocation, and corresponding documentation. +The added paths remain test-owned and bounded; no compatibility layer or unfinished infrastructure was added. ## Commits and scope - `9c9a088 feat(http): delegate container browser rendering to gateway` +- `aec662e fix(http): keep gateway rendering image-only` - Changed implementation paths: `packages/web-core/src/`, `packages/web-core/test/`, `packages/web/src/http.ts`, - `packages/web/src/program.ts`, `Dockerfile`, `README.md`, `CONTEXT.md`, - `docs/http-service.md`, and `docs/adr/`. + `packages/web/src/program.ts`, `packages/web/test/http.test.ts`, + `scripts/test-image-contract.mjs`, `Dockerfile`, `README.md`, `CONTEXT.md`, + `docs/http-service.md`, `docs/adr/`, `package.json`, and CI/release workflows. The required report is intentionally kept under `.scratch` and is excluded from the implementation LOC table above.