Add typed UI geometry context by nmetulev · Pull Request #844 · microsoft/winappCli

Nikola Metulev (nmetulev) · 2026-09-15T00:27:41Z

Summary

add target-window DPI context to ui inspect --json windows and ui status --json
add a scrubbed typed element to ui get-property --json while preserving elementId and string-valued properties
document the canonical JSON envelopes, physical-pixel geometry semantics, and type field

Validation

targeted UI/DPI tests: 605 passed, 1 opt-in interactive test skipped
live ui status --json against a per-monitor-aware window: 120 DPI / 1.25 scale
live compatibility check against a process with no top-level window: succeeds with hwnd: 0 and omitted DPI fields
scripts/build-cli.ps1 -SkipTests: passed, including NativeAOT publish, npm package, NuGet package, and MSIX packaging
scripts/validate-plugin-package.ps1: passed
full scripts/build-cli.ps1: compilation/npm/analyzer phases passed; test phase had three unrelated environment-sensitive failures in existing crash-dump/recording tests

Mixed-monitor placement could not be exercised on the available desktop; distinct per-window DPI/awareness values are covered by command tests.

Fixes #820

Expose per-window DPI context in JSON output and add typed element metadata to get-property while preserving existing fields. Fixes #820 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟢 Approval recommended

The implementation is additive, handles invalid DPI reads explicitly, preserves compatibility, and includes focused coverage and documentation.

Pull request overview

Adds typed geometry and DPI context to UI automation JSON while preserving compatibility.

Changes:

Adds per-window DPI metadata to inspect and status.
Adds a scrubbed typed element to get-property.
Updates tests and JSON documentation.

File summaries

File	Description
`WinApp.UIAutomation/Models/UiElement.cs`	Clarifies physical-pixel geometry.
`WinApp.Cli/NativeMethods.txt`	Enables DPI APIs.
`Helpers/WindowDpiContextProvider.cs`	Reads and validates window DPI context.
`Helpers/IWindowDpiContextProvider.cs`	Defines the DPI provider contract.
`Helpers/UiJsonContext.cs`	Extends JSON models.
`Helpers/HostBuilderExtensions.cs`	Registers the DPI provider.
`Commands/UiStatusCommand.cs`	Adds target DPI metadata.
`Commands/UiInspectCommand.cs`	Adds per-window DPI metadata.
`Commands/UiGetPropertyCommand.cs`	Adds the typed element projection.
`WindowDpiContextProviderTests.cs`	Tests mapping and failures.
`UiCommandTests.SimpleVerbs.cs`	Tests status DPI errors.
`UiCommandTests.Inspect.cs`	Tests inspect DPI errors.
`UiCommandTests.cs`	Tests new JSON contracts.
`FakeWindowDpiContextProvider.cs`	Supports command testing.
`SKILL.md`	Documents the updated envelopes.
`references/ui-json-envelope.md`	Provides canonical JSON examples.
`docs/ui-automation.md`	Documents user-facing semantics.

Review details

Files reviewed: 17/17 changed files
Comments generated: 0
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-15T00:52:26Z

Build Metrics Report

Binary Sizes

Artifact	Baseline	Current	Delta
CLI (ARM64)	48.44 MB	48.49 MB	📈 +51.6 KB (+0.10%)
CLI (x64)	48.29 MB	48.33 MB	📈 +49.1 KB (+0.10%)
MSIX (ARM64)	19.96 MB	19.98 MB	📈 +18.4 KB (+0.09%)
MSIX (x64)	21.16 MB	21.17 MB	📈 +17.0 KB (+0.08%)
NPM Package	41.53 MB	41.57 MB	📈 +45.6 KB (+0.11%)
NuGet Package	41.66 MB	41.71 MB	📈 +45.5 KB (+0.11%)

Test Results

✅ 6841 passed, 37 skipped out of 6878 tests in 891.3s (+22 tests, -184.5s vs. baseline)

Test Coverage

✅ 85.9% line coverage, 79.9% branch coverage · ✅ no change vs. baseline

CLI Startup Time

71ms median (x64, winapp --version) · ✅ +7ms vs. baseline

Try This Build

Installs the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing.

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 844

Switching between builds often?

Put the tool on your PATH once:

& ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPath

Then this build is just:

winapp-pr 844

Run winapp-pr with no arguments to pick from a list of open PRs.

Updated 2026-09-17 00:33:48 UTC · commit 330da32 · workflow run

Separate nested top-level HWND trees without duplication, preserve selector safety, and recover PID-only ancestor window context. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Zach Teutsch (zateutsch) · 2026-09-15T21:42:56Z

PR Review — typed UI geometry context

Decision: changes required. The feature is well-scoped, on-mission, documented, and green (builds clean with 0 warnings, 68 targeted tests pass). One real regression blocks it: ui inspect --json now hard-fails for a window whose HWND is 0 — a state the rest of the code (and this same PR's ui status path) treats as normal.

🔴 Must fix — `ui inspect --json` crashes on a window with no HWND

What is wrong: The new DPI loop calls windowDpiContextProvider.GetForWindow(windowInfo.Hwnd) for every window unconditionally. BuildWindows can emit a window with Hwnd == 0 (it uses el.WindowHandle ?? uiTarget.WindowHandle / ?? 0, and Flush() deliberately keeps a zero-HWND window when it has element roots). WindowDpiContextProvider.GetForWindow(0) throws InvalidOperationException("...HWND is zero"), so the whole inspect fails with exit 1. ui status avoids this with a uiTarget.WindowHandle != 0 guard — inspect has no such guard.
Show me: winapp ui inspect -a <process resolved before it has a top-level window, or a windowless UIA root> --json → exits 1 with {"error":{...,"message":"Cannot read the target window DPI context because its HWND is zero."}}; expected: the element tree still serializes, with DPI fields omitted for that window (exactly what ui status --json does for the same state).
Why it matters: Optional DPI metadata turns the command's primary result (the UI tree) into a hard failure. Existing automation on the PID/windowless path loses inspect --json entirely. The added Status_Json_ProcessWithoutWindow test proves the team already considers HWND 0 a supported, non-error state.
Smallest fix: Mirror ui status: skip DPI enrichment when windowInfo.Hwnd == 0 and make the inspect window's DPI fields nullable/omitted for that case. Add an inspect test with a zero-HWND window — the current FakeWindowDpiContextProvider returns a value for HWND 0 instead of throwing like the real provider, so it masks this path.
Location: src/winapp-CLI/WinApp.Cli/Commands/UiInspectCommand.cs (the DPI enrichment loop, ~L147-153)

🟡 Non-blocking — `get-property` typed `element` includes `value` regardless of `--property`

What is wrong: The new typed element serializes the full UiElement. UiElementScrubber.Strip clears only Id/ParentSelector/WindowHandle, never Value (editable-field text from UIA ValuePattern). So get-property <field> --property IsEnabled --json scopes the string properties map to IsEnabled but still emits the field's text in element.value.
Show me: winapp ui get-property <edit-selector> -a <app> --property IsEnabled --json → output includes element.value: "<field text>"; a caller scoping to one benign property wouldn't expect the field content.
Why it matters: Places arbitrary app field text into JSON a script may log. Bounded, and consistent with shipped ui search/ui wait-for (which already emit value) — so this is a conscious-decision call, not a clear defect.
Smallest fix: In get-property, clear element.Value unless --property is Value or omitted; or accept it as intended and note it in the reference doc.
Location: src/winapp-CLI/WinApp.Cli/Commands/UiGetPropertyCommand.cs (~L97-106), src/winapp-CLI/WinApp.Cli/Helpers/UiElementScrubber.cs (Strip, ~L44-48)

What was exercised

dotnet build WinApp.Cli.csproj -c Debug (branch worktree, cold restore) — succeeded, 0 warnings / 0 errors.
dotnet test filtered to DPI/status/get-property — 68 passed, 0 failed (offline; no network needed).
Static trace of BuildWindows → inspect DPI loop → WindowDpiContextProvider.GetForWindow — confirmed the zero-HWND crash chain; the real provider's throw-on-0 is proven by GetForWindow_ZeroHwnd_ThrowsExplicitError.
Not exercised: a live windowless/zero-HWND inspect target — hard to construct deterministically. The finding rests on the code asymmetry with ui status plus the real provider's unit-proven throw, not a live repro, so the frequency in the wild is unquantified while the failure path itself is certain.

Necessity and ship-surfaces reviews returned clean (docs are consistent and complete). A separate different-model-family cross-check confirmed all findings and added none. A low "centralize the 4-field DPI copy into one helper" suggestion was dropped as no user impact.

_{🤖 Generated with multi-dimensional PR review}

Zach Teutsch (zateutsch)

One must fix surfaced above, it seems like the trigger circumstances would be pretty rare (an inspect before a window loads) but since it would cause a crash it might be worth addressing.

Up to your judgement.

Copilot

🟢 Approval recommended

The solution builds cleanly, targeted suites pass, and live status, inspect, and get-property JSON behave as documented.

Review details

Files reviewed: 20/20 changed files
Comments generated: 0 new
Review effort level: Balanced

Address the Code Quality review without changing promotion behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Scoped inspection of an owned window can query DPI using the process’s main HWND and emit incorrect geometry context.

Get a fresh assessment by requesting another Copilot review.

Review details

Files reviewed: 20/20 changed files
Comments generated: 1
Review effort level: Balanced

Resolve scoped inspect geometry against the selected element's top-level HWND. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Scoped WinUI elements without their own HWND can still receive DPI context from the wrong window.

Get a fresh assessment by requesting another Copilot review.

Review details

Files reviewed: 20/20 changed files
Comments generated: 1
Review effort level: Balanced

Walk UIA ancestors to derive the owned top-level window for scoped inspect and capture targeting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟢 Approval recommended

The implementation matches the requested contract, builds cleanly, and all targeted tests and plugin validation passed.

Review details

Files reviewed: 22/22 changed files
Comments generated: 0 new
Review effort level: Balanced

Preserve owned-window action HWNDs and surface secondary DPI failures without discarding the inspect tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Inspect can retain a stale nonzero session HWND after successfully recovering a different live UIA root, causing incorrect or failed DPI output.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/winapp-CLI/WinApp.UIAutomation/Services/UiAutomationService.cs:176

What is wrong: a nonzero session HWND is kept even when GetRootElement rejected that handle and recovered a different live window through its PID fallback. Show me: a target whose stored window closes between resolution and inspect causes GetRootElementCore to fall back successfully to the process's current UIA root, but this branch skips resolving that root because the stale handle is nonzero; JSON then reads DPI from the stale HWND and fails (or reports another window if the handle was reused). Why it matters: ui inspect -a <pid> --json can fail or emit the wrong DPI context even though the existing UIA recovery found a valid window. Smallest fix: derive mainHwnd from the resolved root first and use uiTarget.WindowHandle only when the root exposes no HWND; add a stale-nonzero-handle regression alongside the PID-only recovery test.

Files reviewed: 22/22 changed files
Comments generated: 0 new
Review effort level: Balanced

Use the live UIA root handle when a stored session window is stale. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟢 Approval recommended

The previously identified HWND-resolution defects are fixed with focused real-provider and command-level regression coverage.

Review details

Files reviewed: 22/22 changed files
Comments generated: 0 new
Review effort level: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> # Conflicts: # src/winapp-CLI/WinApp.Cli/Helpers/HostBuilderExtensions.cs

Copilot

🟢 Approval recommended

The prior HWND-resolution issues are addressed with focused real-provider regressions, and no unresolved defects were found.

Review details

Files reviewed: 22/22 changed files
Comments generated: 0 new
Review effort level: Balanced

## Description Read `FontWeight`, `FontName`, `FontSize`, `ForegroundColor`, `IsItalic`, and `StrikethroughStyle` across the element's whole TextPattern document. Uniform values are invariant strings; UIA's mixed and reserved-not-supported COM tokens become `Mixed` and `NotSupported`, while elements without TextPattern return `Unavailable`. Provider failures retain the existing error contract. Unknown case-sensitive property names now fail with `invalid_arguments` before querying the app. Omitting `--property` includes all six attributes. The published `elementId` and string-valued `properties` JSON envelope and existing geometry formats are unchanged. ## Usage Example ```powershell winapp ui get-property Document -a myapp --property FontWeight --json # {"elementId":"Document","properties":{"FontWeight":"700"}} winapp ui get-property Document -a myapp --json ``` ## Related Issue Fixes #822. Independent of #841, #842, and #844. No query predicates, explicit actions, selection/caret/range APIs, batch mode, or persistence. ## Type of Change - New feature - Test update - Documentation ## Checklist - [x] New unit, CLI, and real-provider tests - [x] Tested locally on Windows: 26 formatting/property library tests and 28 CLI/property/public-API tests pass - [x] `scripts/build-cli.ps1 -SkipTests` completes NativeAOT x64/arm64 publishing, npm/NuGet/MSIX packaging, and generated docs/schema - [x] Canonical UI automation docs and shipped skill/reference updated ## Additional Notes Real-app coverage reads native RichEdit uniform and mixed formatting independently of selection, a Button without TextPattern, and an HWND-based provider returning UIAutomationCore's actual reserved-not-supported token through the repository COM projection. Background fixtures assert they never activate. Unit tests cover invariant conversion, actual COM sentinel identity, unknown names, and provider-error propagation; CLI tests pin string serialization and scrubbed errors. Pre-human-review gate is pending automated review and CI; this PR is not being merged by the implementation session. --------- Co-authored-by: Nikola Metulev <711864+nmetulev@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Zach Teutsch <88554871+zateutsch@users.noreply.github.com>

Nikola Metulev (nmetulev) and others added 2 commits September 14, 2026 17:26

Add typed UI geometry context

bbdd65a

Expose per-window DPI context in JSON output and add typed element metadata to get-property while preserving existing fields. Fixes #820 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Remove trailing blank lines

ed9e107

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings September 15, 2026 00:27

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 15, 2026 00:28 View session

Copilot AI reviewed Sep 15, 2026

View reviewed changes

This was referenced Sep 15, 2026

Add deterministic explicit UI invoke actions #846

Open

Expose whole-document TextPattern formatting in get-property #847

Merged

Nikola Metulev (nmetulev) added pr-review-done and removed pr-review-done labels Sep 15, 2026

Preserve per-window inspect context

2581568

Separate nested top-level HWND trees without duplication, preserve selector safety, and recover PID-only ancestor window context. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Nikola Metulev (nmetulev) requested a balanced review from Copilot September 15, 2026 21:35

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 15, 2026 21:36 View session

github-code-quality Bot found potential problems Sep 15, 2026

View reviewed changes

Comment thread src/winapp-CLI/WinApp.UIAutomation/Services/UiAutomationService.cs Fixed

Merge branch 'main' into nmetulev-typed-ui-geometry-context

7f3cfbf

Zach Teutsch (zateutsch) reviewed Sep 15, 2026

View reviewed changes

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Nikola Metulev (nmetulev) and others added 2 commits September 15, 2026 15:21

Use explicit window filtering

59d414f

Address the Code Quality review without changing promotion behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge updated PR branch

150a5a2

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Nikola Metulev (nmetulev) requested a balanced review from Copilot September 15, 2026 22:22

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 15, 2026 22:23 View session

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Comment thread src/winapp-CLI/WinApp.UIAutomation/Services/UiAutomationService.cs

Use scoped element window context

215f34e

Resolve scoped inspect geometry against the selected element's top-level HWND. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Nikola Metulev (nmetulev) requested a balanced review from Copilot September 15, 2026 22:59

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 15, 2026 23:00 View session

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Comment thread src/winapp-CLI/WinApp.UIAutomation/Services/UiAutomationService.cs

Resolve windowless scoped element HWNDs

1cd0bc6

Walk UIA ancestors to derive the owned top-level window for scoped inspect and capture targeting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Nikola Metulev (nmetulev) requested a balanced review from Copilot September 16, 2026 00:07

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 16, 2026 00:08 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Nikola Metulev (nmetulev) added the pr-review-done label Sep 16, 2026

Harden multi-window inspect context

04a9e7e

Preserve owned-window action HWNDs and surface secondary DPI failures without discarding the inspect tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Nikola Metulev (nmetulev) removed the pr-review-done label Sep 16, 2026

Nikola Metulev (nmetulev) requested a balanced review from Copilot September 16, 2026 01:35

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 16, 2026 01:36 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Prefer recovered inspect root HWND

1cb7971

Use the live UIA root handle when a stored session window is stale. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Nikola Metulev (nmetulev) requested a balanced review from Copilot September 16, 2026 02:26

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 16, 2026 02:27 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Nikola Metulev (nmetulev) added the pr-review-done label Sep 16, 2026

Merge branch 'main' into nmetulev-typed-ui-geometry-context

177b274

Zach Teutsch (zateutsch) approved these changes Sep 16, 2026

View reviewed changes

Nikola Metulev (nmetulev) removed the pr-review-done label Sep 16, 2026

Merge latest main into PR branch

330da32

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> # Conflicts: # src/winapp-CLI/WinApp.Cli/Helpers/HostBuilderExtensions.cs

Nikola Metulev (nmetulev) requested a balanced review from Copilot September 16, 2026 21:55

Copilot started reviewing on behalf of Nikola Metulev (nmetulev) September 16, 2026 21:55 View session

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Zach Teutsch (zateutsch) self-requested a review September 17, 2026 00:17

Zach Teutsch (zateutsch) approved these changes Sep 17, 2026

View reviewed changes

Nikola Metulev (nmetulev) added the pr-review-done label Sep 17, 2026

Nikola Metulev (nmetulev) merged commit 6ad7061 into main Sep 17, 2026
36 of 37 checks passed

Nikola Metulev (nmetulev) deleted the nmetulev-typed-ui-geometry-context branch September 17, 2026 00:57

Navigation Menu

Add typed UI geometry context - #844

Add typed UI geometry context#844
Nikola Metulev (nmetulev) merged 12 commits into
mainfrom
nmetulev-typed-ui-geometry-context

Nikola Metulev (nmetulev) commented Sep 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Sep 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Zach Teutsch (zateutsch) commented Sep 15, 2026

Uh oh!

Zach Teutsch (zateutsch) left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Nikola Metulev (nmetulev) commented Sep 15, 2026

Summary

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

github-actions Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Metrics Report

Binary Sizes

Test Results

Test Coverage

CLI Startup Time

Try This Build

Uh oh!

Uh oh!

Zach Teutsch (zateutsch) commented Sep 15, 2026

PR Review — typed UI geometry context

🔴 Must fix — ui inspect --json crashes on a window with no HWND

🟡 Non-blocking — get-property typed element includes value regardless of --property

What was exercised

Uh oh!

Zach Teutsch (zateutsch) left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (1)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Sep 15, 2026 •

edited

Loading

🔴 Must fix — `ui inspect --json` crashes on a window with no HWND

🟡 Non-blocking — `get-property` typed `element` includes `value` regardless of `--property`