feat: handle Nexus resumable-download prompt in the Wabbajack flow#20
Open
sndnz wants to merge 14 commits into
Open
feat: handle Nexus resumable-download prompt in the Wabbajack flow#20sndnz wants to merge 14 commits into
sndnz wants to merge 14 commits into
Conversation
… gray images Extend TemplateCandidate to carry template keypoints, grayscale image and dimensions; build the per-button-type candidate list once at init instead of rebuilding it on every detect() call. Drops the now-unused descriptor fields from ButtonAssets. No behavior change.
Replace the keypoint-centroid click point with a geometrically verified fit: estimateAffinePartial2D over template->scene keypoint pairs, gated on inlier count, inlier ratio, plausible scale (0.5-2.0) and rotation (<=5 deg). The click point is the template center projected through the transform (sub-pixel accurate: mean fixture error drops 4.5px -> 0.1px). Confidence now blends inlier count and inlier ratio. DetectionResult gains scale/inliers/method fields; num_matches reports inliers.
When the SIFT path finds nothing (small or low-feature templates, sub-1.0 DPI scales that starve keypoint detection), fall back to grayscale TM_CCOEFF_NORMED over a geometric scale ladder (0.60-2.00, ~9% steps), accepting peaks >= 0.80. Debug frames now draw the detection box at the recovered template scale.
A geometrically coherent fit can still be a shared text fragment inside a different button (the 'Slow download' template matches the 'download' word in 'Download manually'). Verify every accepted fit with NCC between the predicted scene region and the template at the recovered scale: fragment matches score ~0.5, true matches >=0.8, threshold 0.60. Also lower the fallback TM threshold to 0.75 -- downscaled thin-text templates bottom out around 0.78 from resampling blur while cross-template peaks stay below 0.5.
…d sub-1.0 scales - scales 0.75/0.85 added (exercises the NCC fallback on SIFT-starved templates), tolerances tightened to 10px clean / 15px degraded (projected centers are sub-pixel; the old 0.5*max(w,h)+15 hid drift) - cross-template negatives per scale-1.0 frame, including the documented CLICK->VORTEX false-positive case previously omitted because main fails it - Gaussian-noise (seeded) and JPEG-70 degraded variants, clutter scenes with two distractor buttons - WABBAJACK and WEBSITE templates depict the same 'Slow download' button at different sizes, so negatives never pair them Manifest grows 19 -> 68 cases. Scores: this branch 1.000 (48/48, 0 FP, mean error 0.4px), main 0.717 on the same suite (42/48, 9 FP, 4.3px).
utils/dpi.py declares per-monitor DPI awareness at startup (v2 -> shcore -> system fallback chain) so win32 reports/accepts the same physical pixels mss captures; without it clicks are off by the display scale factor on any monitor scaled != 100%. services/geometry.py adds pure, win32-free layout math: MonitorFrame (per-monitor image<->virtual conversion, primary = origin invariant), VirtualDesktop (frame lookup, click clamping with dead-zone refusal, win32-vs-mss layout agreement check that catches failed DPI awareness) and compute_search_bbox (clips window rects to the frame and applies a fixed fractional inset). Drops the unused Monitor.is_negative.
Replace random cursor/window-title mocks with a configurable MockDisplayState (monitor layout, window rects, cursor, recorded events) so coordinate tests can assert exact behaviour. utils/platform gains a send_input_mouse shim: real ctypes SendInput on Windows (raising on blocked input so callers can fall back to mouse_event), state-recording mock elsewhere.
ScreenCapture now grabs each monitor separately, pairing every image with its MonitorFrame geometry, so coordinate conversion uses the correct per-monitor offset (exact on negative origins, mixed DPI, portrait and non-contiguous layouts). Deletes the single-offset img<->monitor helpers and the L1-nearest win32<->mss matching heuristic; mss geometry is authoritative and win32 rects are only cross-checked for DPI sanity. mss_factory injection + FakeMSS enable layout tests on any platform.
- ClickController takes the monitor layout, clamps click targets onto the nearest monitor (refusing dead-zone targets) and sends button events via SendInput with a mouse_event fallback; click() reports success so the scanner only counts real clicks - WindowManager places windows by geometry (primary = origin invariant, Vortex on the first secondary) instead of assuming EnumDisplayMonitors ordering; get_all_monitors ensures DPI awareness first and sorts primary first - Scanner iterates per-monitor frames: Vortex search runs only on the frame hosting the window (via compute_search_bbox, replacing the window-position-dependent 'fac' padding formula), web/dialog handlers sweep every frame, detections convert to virtual-desktop coordinates with the owning frame's offset - main() declares DPI awareness before anything touches win32/mss Adds scanner-level coordinate integration tests over a monitor left of the primary.
…line Both Windows fixture tools now declare DPI awareness first and work per monitor frame: ground truth is frame-relative (fx*width, fy*height) -- trivially exact since capture is per-frame -- and case meta records the frame geometry, full virtual layout, DPI awareness level and orientation. Drops the win32<->mss nearest-bounds heuristic copy. README's DPI compatibility-override workaround is obsolete (the app sets per-monitor awareness itself) and is replaced by a note on the logged awareness level.
Fixture families now cover: scale sweep 0.65-2.0 at varied frame positions, five degradations (noise/jpeg/brightness/contrast/blur), cursor occlusion, feature-rich backgrounds (synthetic UI panels, block noise), clutter, multi-instance frames (nearest ground-truth point wins via new 'points' support), a mega frame with all templates, and absent guards: empty, busy button-free scenes, cross-template negatives, and lookalike buttons whose text differs. Detector hardening driven by the new adversarial scenes: - NCC fallback ladder capped at 1.1x (it exists to rescue SIFT-starved DOWN-scales; wide ladders let generic rect+text UI widgets impersonate small templates at inflated scales) - layered fallback acceptance: intensity peaks >=0.90 pass; 0.80-0.90 band peaks must also match the template's blurred-gradient signature (>=0.80) at the peak -- band FPs score <=0.71, true positives >=0.91 - SIFT scale gate widened to 0.4-2.2 so 2.0x buttons stay on the SIFT path detection_report.py grows per-variant breakdown, per-detect latency and a --fail-under gate; synthetic PNGs (15MB) are no longer committed -- both pytest and the report regenerate them deterministically when absent. Scores on the 188-case suite: this branch 1.000 (120/120, 0 FP across 68 negatives, 0.3px mean error), main 0.665 (97/120, 20 FP).
…32 services - ButtonDetector: transform validation windows (scale/rotation bounds, inclusive edges), projected-center math, bbox crop offset round-trip, method/inliers/scale reporting on both paths, legacy-template mode, detect_multiple, hallucination guard - Scanner state machine with a stubbed detector: full vortex->web->dialog cycle, web retry-limit fallback to vortex search, missing-window wait (detector never consulted), legacy popup precedence, wabbajack-mode reset semantics, website-before-wabbajack ordering - WindowManager against mock win32: vortex to first secondary, single monitor no-op, missing window no-op, title-substring placement, primary first in get_all_monitors regardless of enumeration order, off-Windows guard - ClickController: SendInput-blocked fallback to mouse_event, double click - DebugRecorder: annotated frame output, scaled box near frame edge - Fixture generator invariants: deterministic records + byte-stable images, ground-truth points inside bounds, >=5 variant families per type, equivalence rule enforced on cross-template negatives, background keypoint-count contracts, position pool never clips the widest template 275 tests total (was 106 before this pass, 20 before the PR).
New detection-benchmark job (ubuntu + windows) regenerates the deterministic fixture suite, runs detection_report.py --fail-under 0.999 (any single miss or false positive among the 188 cases drops the score below the bar) and publishes the per-type/per-variant breakdown to the job summary.
Nexus beta resumable-downloads interposes a Standard download vs Resumable download modal for files over 500MB. The resumable option streams via the File System API save dialog, which Wabbajack and browser download watchers cannot intercept, stalling automation. The scanner now detects the modal and clicks Standard download first in the non-Vortex (browser/Wabbajack) path to clear it. Driven by an optional StandardDownloadButton.png template so it degrades gracefully when the asset is absent. Adds the STANDARD_DOWNLOAD button type through the detector, scanner, validation, synthetic fixtures and tests; regenerates the detection manifest (score holds at 1.000). Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds handling for Nexus Mods' beta resumable-downloads prompt in the non-Vortex (browser / Wabbajack) flow.
For files over 500MB, Nexus now interrupts a manual download with a modal offering Standard download vs Resumable download. The resumable option streams via the browser File System API (a save-file dialog) that Wabbajack and browser download watchers cannot intercept, which stalls automation. The scanner now detects this modal and clicks Standard download first — ahead of the Website/Wabbajack button probes — to clear it and keep the normal download flow working.
Changes
ButtonType.STANDARD_DOWNLOADwired through the detector (optionalStandardDownloadButton.pngtemplate), the scanner state machine,validate.py, the synthetic fixture generator, and the test-harness gating.*New.png). This PR ships a capturedassets/StandardDownloadButton.png.standard_download).Testing
python validate.py→ passespytest→ 282 passed, 26 skippedruff check .andruff format --check .→ cleanpython detection_report.py --fail-under 0.999→ score 1.000 (standard_download19/19 present detected, 0 false positives)Warp conversation
Co-Authored-By: Oz oz-agent@warp.dev