feat(inference): add browser point suggestions to the editor

.github/path-filters.yml

@@ -99,6 +99,18 @@ docs:
 # `python` -- and this file says outright that a path may appear in several.
 browser-models:
   - "frontend/browser-inference/**"
+  - "frontend/app/e2e/browserSuggestion.spec.ts"
+  # Everything that suite actually exercises. Naming only the spec would have been
+  # the usual trap: `browserSuggestion.spec.ts` is the one place a real ONNX model
+  # runs in a real browser, and it self-skips in the `frontend` job (no artifacts, no
+  # `VISIONSET_REQUIRE_BROWSER_MODELS`), so a change to the composition root or to the
+  # panel it drives would have been "covered" by a job that ran nothing.
+  - "frontend/app/src/data/browserInference/**"
+  - "frontend/app/vite.config.ts"
+  - "frontend/app/playwright.config.ts"
+  - "frontend/ui-core/src/inference/**"
+  - "frontend/ui-core/src/annotator/SuggestPanel.tsx"
+  - "frontend/ui-core/src/annotator/AnnotationPage.tsx"
   - "scripts/browser_models/**"
   - "tests/browser_models/**"
   - "pyproject.toml"

.github/workflows/ci.yml

@@ -1092,6 +1092,14 @@ jobs:
         if: needs.changes.result != 'success' || needs.changes.outputs['browser-models'] == 'true'
         run: pnpm --filter @visionset/browser-inference test:browser
 
+      - name: Install the app's Playwright browsers
+        if: needs.changes.result != 'success' || needs.changes.outputs['browser-models'] == 'true'
+        run: pnpm --filter @visionset/app exec playwright install chromium
+
+      - name: Run the browser suggestion e2e suite, with the model present
+        if: needs.changes.result != 'success' || needs.changes.outputs['browser-models'] == 'true'
+        run: pnpm --filter @visionset/app e2e browserSuggestion
+
       # The step that makes this job's green check mean something. Everything above answers
       # "did what ran succeed?"; a suite that skips itself, or a spec file that stopped being
       # collected at all, answers that with a cheerful yes. This reads the JSON report the

docs/content/architecture/frontend/ui-core.md

@@ -74,16 +74,22 @@ concrete adapter, and it discovers none either.
 
 Absence is the ordinary case, so the hook answers `null` and never throws - the rule stated
 twice above, for a third capability. **A host that offers browser inference composes the
-runtime at the host boundary; a host that does not offer it supplies nothing.** No host in this
-repository supplies one, so every build behaves as it did before the port existed.
-
-The port is two members wide on purpose, and it is an *execution* contract rather than a
-catalog: finding and obtaining a model is a question about things that cannot answer yet, and
-this is not where it is answered. Nor is the narrowness a promise that growth is free - the
-interface is published, so adding a **required** member to it would break every host
-implementing the older one, and
+runtime at the host boundary; a host that does not offer it supplies nothing.** `frontend/app`
+now supplies one: `data/browserInference/BrowserInferenceRuntime.ts` composes a real
+`VisionSetBrowserInferenceRuntime` over `@visionset/browser-inference` and passes it through
+`OssSession.tsx` unconditionally. A host that omitted it would still behave exactly as it did
+before the port existed.
+
+The port is four members wide, and it is an *execution* contract rather than a catalog: the two
+required members - `listTargets` and `executorFor` - are only about asking a model that can
+already answer. The two added since are optional, which is what let them arrive without
+breaking a host implementing the older interface: `listAcquisitions?()`, which names models this
+device could run once their bytes are fetched, and `setActiveAsset?()`, which tells the runtime
+which asset is on screen so an executor's staleness checks have something to check against. That
+is exactly how
 [browser inference is host-injected](../decisions/browser-inference-is-host-injected.md) says
-how it is grown instead.
+the interface is grown - the published shape means a new **required** member would break every
+host implementing the older one, so growth is additive and optional.
 
 > An `InferenceConnection` is a model the VisionSet server has. A browser inference runtime is
 > something this browser can do. They are not two spellings of one idea, and neither is
@@ -108,9 +114,10 @@ reference for one asset from reading pixels of the next asset either through a r
 through the detached one a host's real asset-switch (unmount the old canvas, mount a fresh one)
 leaves behind.
 
-This is resource plumbing only. It neither selects a browser model nor exposes an inference
-target — no capability in this phase lets a host ask for local inference at all; composing pixels
-with a browser runtime remains a later host decision.
+This is resource plumbing only: it neither selects a browser model nor exposes an inference
+target. Composing the two is the host's decision, and `frontend/app` now makes it — the lease
+reaches the browser runtime through the port's `setActiveAsset`, and the executor behind
+`executorFor` is what reads `readRgb`.
 
 **What is proved, and what is not claimed.** `e2e/assetPixels.spec.ts` drives a real Chromium
 against a tiny runtime-generated image and confirms, in that browser, on that image: one content
@@ -126,10 +133,18 @@ rounding loss on partially-transparent pixels, EXIF-orientation auto-rotation (a
 image's EXIF orientation when decoding to canvas; the server's direct-bytes decode path does not),
 and AdobeRGB ICC-profile color management (a browser color-manages a tagged profile toward sRGB on
 decode; the server's path does not). None of these are exact figures worth repeating here — they
-are a known limitation, not a benchmark — and no consumer of this lease has yet needed to reconcile
-them, since nothing in this phase reads pixels for inference. A future phase that feeds this lease's
-`readRgb` output to a model must account for these divergences before treating browser-decoded
-pixels as equivalent to the server's own decode of the same asset.
+are a known limitation, not a benchmark.
+
+**This is now a live limitation, not a future one.** The browser suggestion executor feeds this
+lease's `readRgb` output straight to a model, so the pixels "This device" segments are the
+browser's decode and the pixels the server segments are Pillow's. **Server and "This device" can
+therefore return different masks for the same asset and the same click**, and for an asset
+carrying EXIF orientation or an AdobeRGB profile the difference can be large rather than
+marginal — an auto-rotated decode is not a variation on the same picture. Nothing reconciles
+the two, and nothing in the editor tells a person which decode answered. Treat a browser
+suggestion as this browser's answer about this browser's decode; it is not a claim about what
+the server would have said. Closing the gap means changing a decode, not adding a tolerance,
+and neither side has been changed here.
 
 ## Asking for a suggestion is not sending one
 

docs/content/ui.md

@@ -599,9 +599,10 @@ object list is how a lane is selected, which is a real affordance rather than a
 
 The sparkles button - hotkey `S` - arms the **suggest tool**: click the thing you
 want and a segmentation model proposes its shape, which you can then adjust
-before accepting. It runs through a model
-connection (`docs/content/inference.md`), and the server side of it is
-`POST /inference/suggest`.
+before accepting. It answers from one of two places - a model connection on the
+server, or a model running in this browser - and the panel carries that choice.
+Through a connection it is `POST /inference/suggest`
+(`docs/content/inference.md`).
 
 **It runs through a connection that can answer a click**, which is a narrower set
 than "the ones that are ready": only those declaring `point_suggest`. A workspace
@@ -619,6 +620,25 @@ there is no control at all, only a line naming what is answering. The picker
 appears on the idle card alone: changing which model answers while a proposal is
 on screen would leave a shape nothing on the card explains.
 
+**Server, or this device.** The panel's two tabs are where a click goes. **Server**
+is the connection described above. **This device** runs a smaller segmentation model
+in this browser, which answers without the picture or the click leaving the machine
+and without a connection being configured at all - useful where the workspace has no
+model that can answer a click, or where the server is slow to reach.
+
+It has to be downloaded first, and the panel says so: about 41 MB, on an explicit
+press, never on its own. **The download does not survive a page reload.** It is held
+for the session only, so reopening the editor tomorrow - or reloading today - means
+downloading it again before "This device" can answer. Until then the tab shows the
+Download button rather than inviting a click that would do nothing, and the choice
+falls back to Server.
+
+**Alt-click needs Server.** The model running here takes only points that are *on*
+the object; a point marking something that is not part of it is refused rather than
+answered with something that is not an exclusion. Switch to Server to refine that
+way. Everything else - the first click, refining with more points, the tolerance,
+`↵` and `Esc` - works the same on both.
+
 The gesture:
 
 | Press | What it does |