[Feature]: UI automation on physical devices via CoreDevice HID (pymobiledevice3)

[n0ah37]

Summary

The device workflow can build, install, launch, stop and test on a physical device, but has no input or capture tools — ui-automation is simulator-only by design ("UI automation and accessibility testing tools for iOS simulators"). That leaves agent-driven verification on real hardware out of reach, even though the device itself exposes everything needed.

I'd like to propose adding device-scoped tap / swipe / type / button and screenshot to the device workflow, and I'm happy to write the PR if there's appetite for it. Checking first because the design questions below are yours to answer, not mine.

The capability is already on the device

devicectl device info details against an iPhone 17 Pro on iOS 27.0 lists these CoreDevice features as available:

HID Digitizer       com.apple.coredevice.feature.remote.hid.digitizer
HID Keyboard        com.apple.coredevice.feature.remote.hid.keyboard
HID Button          com.apple.coredevice.feature.remote.hid.button
HID Scroll          com.apple.coredevice.feature.remote.hid.scroll
Universal HID       com.apple.coredevice.feature.remote.universalhid
View Device Screen  com.apple.coredevice.feature.viewdevicescreen

Touch, keyboard, buttons, scroll and a screen stream. On iOS 26+ these sit behind a consent toggle at Settings › Developer › UI Automation › Enable UI Automation, which pairs with "Paired Mac computers can remotely view and control this iPhone" — this is the same plumbing Xcode 27's Device Hub uses for its remote-control view.

Apple ships no CLI verb for any of it. devicectl in the Xcode 27 beta has appResize capture copy info install notification orientation pairings pasteboard process profile reboot rename settings simulate sysdiagnose uninstall — no input subcommand. Device Hub is the only first-party client, and it's a GUI with no scripting interface.

But it is reachable without one

pymobiledevice3 implements the iOS 17+ RemoteXPC tunnel and exposes the service directly:

pymobiledevice3 developer core-device universal-hid-service tap -- 32768 32768

Coordinates are normalised UInt16 — (0,0) top-left, (65535,65535) bottom-right — with drag, swipe and move alongside tap, and batched gestures readable from stdin or a script file.

devicectl device capture screenshot already works against a physical device today, so pairing the two gives a complete see-and-touch loop with no WebDriverAgent, no vendored Appium and no on-device runner to sign — which I think is the reason this has looked bigger than it is.

Design questions I'd want your call on before writing anything

1. The Python dependency. This is a Node project and pymobiledevice3 is a Python CLI. Detect at runtime and degrade gracefully when absent, or something stricter?
2. Coordinates. Simulator tools take points; the device service takes normalised UInt16. Convert inside the tool so the API matches ui-automation, or expose the normalised space and let callers deal with it? Converting needs a reliable source for the device's point size.
3. Placement. Extend device, or a separate device-ui-automation workflow so the Python requirement is opt-in?
4. No accessibility tree. There's no device equivalent of snapshot_ui, so device automation would be screenshot-driven only. Worth stating in the tool descriptions so agents don't reach for a tree that isn't there.

Not verified on my side

• Whether the tunnel needs root in all transports (it does in some) and how that interacts with an MCP server's process.
• Whether the tunnel coexists with Xcode or Device Hub holding the same device.

Both would need answering in the PR rather than assumed.