Add SAM2/SAM3 browser segmentation and stereo auto-population - #1672
Merged
Merged
Conversation
…l points or a tighter box
With auto compute on the other camera, the mapped shape gets the same mask and/or points pass once the transfer succeeds; a derived head/tail follows the source camera's direction.
Lets the VIAME service keep a line-prompted mask in scale with the line.
A box warped to the other camera whose mask overlaps it by less than half its union takes the mask's bounds instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The service's polygons list, with holes, is stored as keyed polygons on the detection; a refinement drops the components it no longer has. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… is on A confirmed mask on a brand-new detection now takes the same keypoint pass as a drawn box, on the source camera and on the stereo copy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With auto-segmentation on, points inside the source mask are warped instead of the box corners, the other camera is segmented from them, and its box is that mask's bounds. Corner warping remains the fallback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…moves it The pass runs after each click's prediction on both cameras, replacing only a line it derived itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The other camera's mask comes from the service, which seeds from inside the source mask and refuses an out-of-scale result, instead of DIVE warping prompts and predicting itself. Click segmentation sends every source polygon and draws every returned one.
… dev/point-seg-multipolygon
# Conflicts: # client/dive-common/apispec.ts # client/dive-common/components/EditorMenu.vue # client/dive-common/components/TrackSettingsPanel.vue # client/dive-common/recipes/segmentationpointclick.spec.ts # client/dive-common/store/settings.ts # client/dive-common/use/autoPopulate.spec.ts # client/dive-common/use/autoPopulate.ts # client/dive-common/use/useModeManager.spec.ts # client/dive-common/use/useModeManager.ts # client/platform/desktop/backend/native/interactive.ts # client/platform/desktop/backend/native/segmentation.ts # client/platform/desktop/frontend/autoPopulate.spec.ts # client/platform/desktop/frontend/autoPopulate.ts # client/platform/desktop/frontend/components/ViewerLoader.vue
ImageData's data, width and height are prototype getters, so spreading it produced an object without pixels; Firefox then failed the first SAM click.
When the higher quality model fails to load or to run on this browser, the setting drops to template matching so it matches what runs, and the error says so instead of asking the user to switch.
GeoJS ends editing on its own (right-click, completion) and leaves the box behind in disabled mode; skipping disable() there kept it as a ghost on every frame with a hover highlight.
A failed load was cached for the session and surfaced later as a bare 'could not be loaded'; now the load is retried on the next warp and the caller's error carries the underlying reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DIVE Web now supports point-prompt segmentation with SAM2.1 Tiny (default) or SAM3, selected in Track Settings. The regular Segment controls replace the embedding-only prototype: positive/negative clicks produce editable polygons, preserve disconnected components and holes, and support confirm/reset/cancel. The selected ONNX model runs in the browser, loads on demand, and caches two camera-frame embeddings. Hardware WebGPU is preferred, with WASM fallback; software GPU adapters use WASM directly.
SAM2 and SAM3 now share a bundled 2 KB ONNX post-processing graph for best-mask selection, bilinear resize, unpadding and binary thresholding. Only the winning candidate is upscaled. DIVE receives a binary mask for polygon editing, while the graph and its weight-free exporter can also be used by desktop ONNX hosts. The exporter includes reference validation; the desktop service has not been switched to this new graph.
New boxes and lines can auto-populate masks and head/tail geometry using the shared annotation application code. In calibrated stereo mode, segmentation prompts and interior mask samples transfer through the existing correspondence model, followed by SAM inference on the other camera. Generated lines and mask bounds finish before the stereo length is refreshed, including subsequent multi-polygon refinements. Manual counterpart edits are preserved, and cancelled or stale work cannot replace newer geometry. Image and video frames are captured without annotation overlays.
Stereo transfer now follows desktop's Fast-FoundationStereo post-processing defaults: cached dense disparity, foreground-biased 90th-percentile sampling over a 7×7 neighbourhood at original-image pixel spacing, and an 11-sample straight-line disparity fit with up to three outliers and a 10-pixel residual. Failed fits retain endpoint matching; curves keep individually mapped vertices. Dense disparity is independent of NCC search limits.
Sparse stereo sampling now runs in a bundled 6 KB ONNX graph with a weight-free exporter (
plugins/onnx/export_stereo_sampler.pyin VIAME). Interpolation at native-image spacing, finite/positive filtering, clipped-neighbourhood counts and 90th-percentile selection are inside the graph. It gathers only 49 neighbours per point from cached dense disparity. Straight-line transfer batches all eleven samples into one graph call, including its endpoint fallback. The old JavaScript sampler is retained only as a test oracle; native desktop adoption of the graph is a separate change.Mask transfer uses desktop's deep-interior/farthest-point sampling (five seeds overall, at least two per component), per-component median-offset filtering, and original labelled-click fallback if no interior matches survive. The other-camera mask must be within a 2.5× area ratio, accounting for holes. Browser rectification, model exports, and polygon rasterization can still differ from native results.
This branch incorporates the existing multi-polygon/auto-populate dependencies from #1952 and #1941 and is updated to current main. It has not been merged into viame/main. Native desktop inference continues to use VIAME; its auto-populate geometry application is now shared with the web provider.
Validation:
The browser path supports image sequences and loaded video frames; tiled large-image datasets remain unsupported. The SAM3 option is its point/box tracker model, not text-prompt search. Model files download from Hugging Face; image pixels remain local. SAM3 is substantially heavier than SAM2, and hardware-GPU performance still needs validation.
The helper exporter sources have moved to VIAME’s
plugins/onnx/in VIAME/VIAME#303, using snake_case names and the ONNX plugin installation list. This PR removes the DIVE copies and retains both bundled ONNX assets. Regeneration passes all 30 reference cases and changes only model producer/documentation metadata, with computational graphs unchanged.