Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,14 @@ _Avoid_: Backend-specific renderer labels, automatic fallback
**Page Navigation**:
The shared page-reading module owns its fixed 5,000-character policy. A long,
unsectioned request with navigable headings returns a navigation tree; a
headingless long document uses the normal bounded response. `full: true`
returns complete Markdown, and a tree's `section_id` retrieves one section.
`full: true` and `section_id` are mutually exclusive.
_Avoid_: Caller-selected tree thresholds, public `tree` controls
headingless long document uses the normal bounded automatic response. The
request `mode` is `auto` by default and may be `full` or `tree`; `section_id`
is allowed with omitted or `auto` mode and retrieves one section. Full and tree
reject `section_id`, and input mode `section` is removed. Ordinary automatic
document results report `mode: "auto"`; full, tree, and section results report
their corresponding modes. Every Fetch result includes `truncated`, which is
true only when automatic content was cut by the content-length limit.
_Avoid_: Caller-selected tree thresholds, legacy navigation booleans

**Release Contract**:
The versioned public distribution of Guion Web: its npm packages, GHCR container image, and the generated `openapi.yaml` attached to the matching GitHub Release.
Expand All @@ -53,6 +57,6 @@ The Hono-based `/v1` JSON API shipped by `web serve` and the GHCR image. It
uses server-local credentials, Bridge Route, and optional DeepSeek provider
configuration; clients do not select providers or submit a generic Bridge
command. Its page-reading routes use
the same `render: "http" | "browser"`, `full`, and `section_id` contract; the
the same `render: "http" | "browser"`, `mode`, and `section_id` contract; the
browser executable name appears only in operator setup.
_Avoid_: Remote MCP, public service
34 changes: 28 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ are generated into `openapi.yaml` from the same route definitions:
| Route | Request | Purpose |
| ------------ | --------------------------------------------------------- | --------------------------------------------------------------- |
| `/v1/search` | `{ "query": "..." }` | Server-selected search: Bridge→Exa by default, or DeepSeek only |
| `/v1/fetch` | `{ "url", "section_id?", "full?", "render?", "waitMs?" }` | Fetch Markdown |
| `/v1/fetch` | `{ "url", "mode?", "section_id?", "render?", "waitMs?" }` | Fetch Markdown |
| `/v1/links` | `{ "url", "limit?", "render?", "waitMs?" }` | List page HTTP(S) links |

The complete human-readable contract is in the [HTTP service reference](docs/http-service.md).
Expand Down Expand Up @@ -122,19 +122,28 @@ web search --provider deepseek -- "Node AbortSignal"
web search --provider kepos-bridge -- "Node AbortSignal"
web fetch https://example.com/article
web fetch https://example.com/article --section introduction
web fetch https://example.com/article --mode auto --section introduction
web fetch https://example.com/article --mode tree
web fetch https://example.com/article --mode full
web links https://example.com/article --limit 50
web docs resolve react
web docs fetch /facebook/react --topic hooks --tokens 2000
web sgraph --count 10 -- "repo:^github\\.com/nodejs/node$ AbortSignal"
```

Use `--` before a search or Sourcegraph query that begins with a hyphen. `fetch`
supports `--full` and `--section`; long extracted documents with navigable
supports `--mode auto|full|tree`; omitted mode means `auto`. `--section` may be
used with omitted mode or `--mode auto` to retrieve a section, and is rejected
with `--mode full` or `--mode tree`. Long extracted documents with navigable
headings automatically return a heading tree so a later request can retrieve a
stable `section_id`. A headingless long document uses the normal bounded
response. `--full` returns the complete extracted Markdown, and `--full` cannot
be combined with `--section`. `links` lists up to 100 unique HTTP(S) anchors from the original
page DOM.
stable `section_id`. Ordinary automatic document results report `mode: "auto"`;
heading-tree, explicit full-document, and section results report `"tree"`,
`"full"`, and `"section"` respectively. A headingless long document uses the
normal bounded automatic response. `truncated` is true only when that response
is cut by the content-length limit. `mode: "full"` returns the complete
extracted Markdown, while `mode: "tree"` always returns the heading-tree
representation, including the explicit no-headings result. `links` lists up to
100 unique HTTP(S) anchors from the original page DOM.

## MCP

Expand All @@ -153,6 +162,11 @@ The server exposes six read-only tools: `search`, `fetch`, `links`, `docs_resolv
messages; diagnostics go to stderr. For a client-rendered page, explicitly call
`fetch` or `links` with `render: "browser"` and an integer `waitMs`; this optional
retry requires a host-installed executable and never happens automatically.
The `fetch` tool accepts input `mode: "auto" | "full" | "tree"` (default
`"auto"`). Pass a returned `section_id` with omitted mode or `mode: "auto"`
to retrieve that section; `mode: "full"` and `mode: "tree"` reject
`section_id`. Results include `mode: "auto" | "full" | "tree" | "section"`
and `truncated`, which is true only when content was cut by the length limit.

## Pi

Expand All @@ -167,6 +181,10 @@ the bundled core in-process. Pi and TypeBox are peer dependencies supplied by
the host; no CLI executable or MCP configuration is required. `web_fetch` uses
HTTP rendering by default and can explicitly use `render: "browser"` with
an integer `waitMs` when its host provides that optional executable.
Its navigation input is `mode: "auto" | "full" | "tree"` (default `"auto"`);
`section_id` with omitted/`"auto"` mode retrieves a section, while full/tree
reject it. Results report `mode: "auto" | "full" | "tree" | "section"` and a
`truncated` flag that only indicates content cut by the length limit.
`web_links` uses the same explicit rendering contract and lists HTTP(S) anchors
from the original page DOM.

Expand Down Expand Up @@ -196,6 +214,10 @@ also run in-process. The host DSH packages and React are peers supplied by DSH.
`web_fetch` uses HTTP rendering by default and can explicitly use
`render: "browser"` with an integer `waitMs` on a host that supplies the
optional executable.
Its navigation input uses the same `mode` and `section_id` contract as the
other adapters: input mode is `auto|full|tree` (default `auto`), and omitted or
`auto` mode plus `section_id` retrieves a section. Results report
`auto|full|tree|section` and `truncated`.
`web_links` uses the same explicit rendering contract and lists HTTP(S) anchors
from the original page DOM.

Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0001-containerized-rest-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ Guion Web will add a self-hosted, single-user HTTP service in a portable contain

## Consequences

The service's provider credentials, Bridge Route, and optional `WEB_SEARCH_PROVIDER=deepseek` selection are server-local configuration; clients cannot choose a provider or supply a Bridge Route per request. With no provider selection, search tries Kepos Bridge and uses Exa only when the Bridge is operationally unavailable; it does not fall back for cancellation, malformed client input, or an empty result set. A failed Bridge attempt is retried through Exa exactly once, and a successful empty Bridge response is returned unchanged. When DeepSeek is selected, the service calls only DeepSeek and never falls back. DeepSeek performs one auxiliary model call internally and returns the same normalized result contract; its Messages/tool protocol is not exposed to HTTP callers. Typed Bridge operations are intentionally not exposed: Exa has no equivalent official weather, sports, or time API, and its premium finance integration is not contract-compatible. Fetch and Links use `render: "http"` by default or explicit `render: "browser"` with a required `waitMs`; the operator-installed executable remains an implementation detail. The shared page-reading module owns automatic navigation trees, complete `full` extraction, and `section_id` continuation. `openapi.yaml` is generated from the release build rather than manually maintained or independently versioned; the standalone [HTTP service reference](../http-service.md) is its human-readable companion. Public or multi-tenant deployment hardening is deliberately deferred in `.scratch/defered/public-http-service-security.md`.
The service's provider credentials, Bridge Route, and optional `WEB_SEARCH_PROVIDER=deepseek` selection are server-local configuration; clients cannot choose a provider or supply a Bridge Route per request. With no provider selection, search tries Kepos Bridge and uses Exa only when the Bridge is operationally unavailable; it does not fall back for cancellation, malformed client input, or an empty result set. A failed Bridge attempt is retried through Exa exactly once, and a successful empty Bridge response is returned unchanged. When DeepSeek is selected, the service calls only DeepSeek and never falls back. DeepSeek performs one auxiliary model call internally and returns the same normalized result contract; its Messages/tool protocol is not exposed to HTTP callers. Typed Bridge operations are intentionally not exposed: Exa has no equivalent official weather, sports, or time API, and its premium finance integration is not contract-compatible. Fetch and Links use `render: "http"` by default or explicit `render: "browser"` with a required `waitMs`; the operator-installed executable remains an implementation detail. Fetch navigation input uses `mode: "auto"` by default, with explicit `"full"` or `"tree"` modes; a non-empty `section_id` with omitted or `"auto"` mode retrieves a section, while full/tree reject it. Results report `"auto"`, `"full"`, `"tree"`, or `"section"` as appropriate and include a `truncated` flag for content cut by the Core limit. `openapi.yaml` is generated from the release build rather than manually maintained or independently versioned; the standalone [HTTP service reference](../http-service.md) is its human-readable companion. Public or multi-tenant deployment hardening is deliberately deferred in `.scratch/defered/public-http-service-security.md`.
32 changes: 19 additions & 13 deletions docs/http-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,40 +55,46 @@ Request:
{
"url": "https://example.test/article",
"render": "http",
"full": false,
"section_id": "7i"
}
```

`url` must be an absolute `http:` or `https:` URL. `render` is optional and
must be exactly `"http"` or `"browser"`; omission selects `"http"`. `waitMs`
is an integer from 0 through 30,000, required with `render: "browser"` and
forbidden with `render: "http"` (or when `render` is omitted). `full` is an
optional boolean. `section_id` is an optional non-empty string returned in a
navigation tree. `full: true` and `section_id` cannot be sent together.
forbidden with `render: "http"` (or when `render` is omitted). `mode` is
optional and defaults to `"auto"`; it must be one of `"auto"`, `"full"`,
or `"tree"`. A non-empty `section_id` may be supplied with omitted mode or
`mode: "auto"` to retrieve that section; it is rejected with `"full"` or
`"tree"`.

HTTP rendering fetches the page with Node HTTP, linkedom, and Defuddle.
Browser rendering invokes the operator-installed `agent-browser` executable
through the isolated renderer implementation; the executable name is not a
public request value. Browser rendering is never selected automatically, and
the service does not fall back between renderers.

The shared module owns the 5,000-character automatic-tree policy. A non-full,
unsectioned document longer than that threshold with navigable headings returns
`mode: "tree"` with stable section IDs. Use one of those IDs in a subsequent
request to retrieve a section. A long document without headings uses the normal
bounded `mode: "full"` response because it has no section to navigate. The
`full: true` option returns the complete extracted Markdown without the Core
content limit. A short result uses `mode: "full"`; a section request uses
`mode: "section"`.
The shared module owns the 5,000-character automatic-tree policy. An `"auto"`
request for an unsectioned document longer than that threshold with navigable
headings returns `mode: "tree"` with stable section IDs. Use one of those IDs
in a subsequent request with omitted mode or `mode: "auto"` to retrieve a
section. A short automatic response and a headingless long document report
`mode: "auto"`; the latter remains bounded by the Core content limit.
`mode: "full"` returns the complete extracted Markdown without that limit and
reports `mode: "full"`. `mode: "tree"` always returns the heading-tree
representation, including the explicit no-headings result, and reports
`mode: "tree"`. Section requests report `mode: "section"`. Every response
includes `truncated`, which is true only when ordinary automatic content was
cut by the Core content-length limit.

Response `200`:

```json
{
"url": "https://example.test/article",
"mode": "full",
"content": "# Article\n...\n"
"content": "# Article\n...\n",
"truncated": false
}
```

Expand Down
19 changes: 16 additions & 3 deletions packages/dsh-web/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,22 @@ It uses the same HTTP default and explicit `render: "browser"` / required
`waitMs` contract as `web_fetch`.

Long `web_fetch` documents with navigable headings return a navigation tree
automatically. A headingless long document uses the normal bounded response.
Use `full: true` for complete Markdown or pass a returned `section_id` to
continue with one section; those fields are mutually exclusive.
automatically when `mode: "auto"` (the default). A headingless long document
uses the normal bounded response. Set `mode: "full"` for complete Markdown,
`mode: "tree"` to force the heading tree, or supply a returned `section_id`
with omitted mode or `mode: "auto"` to continue with one section. Input mode
`"section"` is not supported, and `section_id` is rejected with `"full"` or
`"tree"`. Ordinary automatic document results report `mode: "auto"`; tree,
full, and section results report `"tree"`, `"full"`, and `"section"`.
Every result includes `truncated`, which is true only when automatic content
was cut by the content-length limit.

For example, request a tree and then continue with one returned section:

```json
{ "url": "https://example.test/article", "mode": "tree" }
{ "url": "https://example.test/article", "section_id": "7i" }
```

Rendered requests are bounded and constrained to the requested hostname,
`*.<requested-hostname>` (the target and its subdomains), and this fixed common
Expand Down
11 changes: 8 additions & 3 deletions packages/dsh-web/src/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -312,10 +312,15 @@ function formatWait(waitMs: number): string {
}

function fetchResultMode(args: Record<string, unknown>): string {
if (typeof args.section_id === "string" && args.section_id !== "")
if (
(args.mode === undefined || args.mode === "auto") &&
typeof args.section_id === "string" &&
args.section_id !== ""
)
return `Section: ${args.section_id}`;
if (args.full === true) return "Full document";
return "Adaptive document";
if (args.mode === "full") return "Full document";
if (args.mode === "tree") return "Heading tree";
return "Automatic navigation";
}

function excerpt(output: string): string {
Expand Down
41 changes: 26 additions & 15 deletions packages/dsh-web/src/tools.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import {
createWebOperations,
DEFAULT_LINK_LIMIT,
FETCH_MODES,
normalizeDocsToolInput,
formatSize,
MAX_LINK_LIMIT,
Expand All @@ -10,6 +11,7 @@ import {
type DocsResolveResult,
type DocsToolInput,
type FetchInput,
type FetchMode,
type FetchResult,
type LinksInput,
type LinksResult,
Expand Down Expand Up @@ -42,13 +44,16 @@ const fetchParameters = {
required: true,
description: "HTTP or HTTPS URL to fetch",
},
section_id: {
mode: {
type: "string",
description: "Optional heading section ID to return",
enum: [...FETCH_MODES],
default: "auto",
description: "Navigation mode: auto (default), full, or tree",
},
full: {
type: "boolean",
description: "Return full content without automatic tree mode",
section_id: {
type: "string",
description:
"Heading section ID; retrieves a section with omitted/auto mode",
},
render: {
type: "string",
Expand Down Expand Up @@ -201,18 +206,19 @@ const fetchOutput = {
url: { type: "string", required: true },
mode: {
type: "string",
enum: ["full", "tree", "section"],
enum: ["auto", "full", "tree", "section"],
required: true,
},
content: { type: "string", required: true },
truncated: { type: "boolean", required: true },
},
} as const,
render: (_args: unknown, value: FetchResult) => [
{
type: "text" as const,
text: boundedToolText(
value.content,
"Use web_fetch with full: true or a returned section_id to navigate the document.",
'Use web_fetch with mode: "full" for the complete document, or section_id with the default/auto mode to navigate to a section.',
),
},
],
Expand Down Expand Up @@ -382,26 +388,31 @@ function normalizeFetch(input: unknown): FetchInput {
if (!isRecord(input)) throw new Error("web_fetch input must be an object");
rejectUnknownFields(
input,
["url", "section_id", "full", "render", "waitMs"],
["url", "mode", "section_id", "render", "waitMs"],
"web_fetch",
);
const url = requireString(input, "url");
const mode = input.mode;
if (mode !== undefined && !FETCH_MODES.includes(mode as FetchMode))
throw new Error('mode must be one of "auto", "full", or "tree"');
const selectedMode = mode as FetchMode | undefined;
const sectionID = input.section_id;
if (
sectionID !== undefined &&
(typeof sectionID !== "string" || sectionID.trim().length === 0)
)
throw new Error("section_id must be a non-empty string");
const full = input.full;
if (full !== undefined && typeof full !== "boolean")
throw new Error("full must be a boolean");
if (full === true && sectionID !== undefined)
throw new Error("full and section_id cannot be used together");
if (
sectionID !== undefined &&
selectedMode !== undefined &&
selectedMode !== "auto"
)
throw new Error('section_id is only valid with mode "auto"');
const renderOptions = validateRenderOptions(input);
return {
url,
...(selectedMode === undefined ? {} : { mode: selectedMode }),
...(sectionID === undefined ? {} : { section_id: sectionID }),
...(full === undefined ? {} : { full }),
...renderOptions,
};
}
Expand Down Expand Up @@ -489,7 +500,7 @@ function webFetchTool(
defineTool({
name: "web_fetch",
description:
"Use HTTP rendering for static, SSR, and pre-rendered pages. For client-rendered or SPA pages, set render: browser with required waitMs when the host provides browser capability; there is no automatic fallback.",
"Use HTTP rendering for static, SSR, and pre-rendered pages. mode selects auto, full, or tree navigation; section_id with omitted/auto mode retrieves a section. For client-rendered or SPA pages, set render: browser with required waitMs when the host provides browser capability; there is no automatic fallback.",
parameters: fetchParameters,
output: fetchOutput,
isConcurrencySafe: () => true,
Expand Down
4 changes: 2 additions & 2 deletions packages/dsh-web/test/artifact.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ describe("DSH 0.1.2-alpha.3 packed package contract", () => {
throw new Error("packed DSH artifact did not register web_links");
process.env.PATH = `${browser.bin}:${originalPath ?? ""}`;
const direct = await fetchTool.execute(
{ url: "https://93.184.216.34/direct", full: true },
{ url: "https://93.184.216.34/direct", mode: "full" },
{ signal: new AbortController().signal },
);
expect(direct.content).toBe("Packed DSH browserless fixture.\n");
Expand All @@ -313,7 +313,7 @@ describe("DSH 0.1.2-alpha.3 packed package contract", () => {
url: "https://93.184.216.34/rendered",
render: "browser",
waitMs: 0,
full: true,
mode: "full",
},
{ signal: new AbortController().signal },
);
Expand Down
Loading
Loading