feat(ios): expose clipboard get/set in the iOS action space - #2956
Open
zubeyralmaho wants to merge 2 commits into
Open
feat(ios): expose clipboard get/set in the iOS action space#2956zubeyralmaho wants to merge 2 commits into
zubeyralmaho wants to merge 2 commits into
Conversation
Add IOSWebDriverClient.getPasteboard()/.setPasteboard(), the low-level POST .../wda/getPasteboard and .../wda/setPasteboard calls (base64 content, contentType: "plaintext"), and IOSDevice.getClipboardText()/ .setClipboardText() as the public, ergonomic wrapper. There was previously no way to read a value that an app only exposes through a native "Copy" action (e.g. a share sheet's "Copy Link" for a generated invite/meeting link) -- WDA's own GET /source also cannot help there, since it 500s while a native share sheet (UIActivityViewController) is presented and its accessibility tree cannot be walked. The pasteboard is WDA-native and system-wide, so it works regardless. The request shape (POST + JSON body, not GET + query string) is cross-checked against the unmerged web-infra-dev#2628, which shipped passing unit tests against this exact endpoint contract. Test plan: - npx nx test ios (151 passed) - npx nx build ios
Wire IOSDevice.getClipboardText()/.setClipboardText() into actionSpace() as the IOSGetClipboard and IOSSetClipboard actions, following the existing IOSHomeButton / IOSAppSwitcher pattern for iOS-specific actions. Without this the pasteboard methods are reachable only from hand-written code: the planner has no tool for them, so a prompt like "copy the invite link and read it back" cannot work. IOSGetClipboard is what makes a value that exists only behind a native "Copy" action (e.g. a share sheet's "Copy Link") usable inside a plan, since such a value is rendered nowhere on screen and cannot be reached by locating or extraction. Both actions are registered locally rather than as canonical cross-platform actions in @midscene/core, matching how the other iOS-specific actions are declared. interfaceAlias also exposes them as agent.getClipboardText() and agent.setClipboardText(). Test plan: - npx nx test ios (163 passed) - npx nx build ios - pnpm run lint
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Registers two iOS-specific actions in
IOSDevice.actionSpace():IOSGetClipboard— no params, returns the pasteboard text(alias
agent.getClipboardText())IOSSetClipboard—{ text }, writes plain text to the pasteboard(alias
agent.setClipboardText())Why
#2954 adds the device-level pasteboard methods, but nothing wires them into the
action space — so they are reachable only from hand-written code. The planner has
no tool for them, which means a prompt like "copy the invite link and read it
back" cannot work today.
That is the case the capability exists for. A value behind a native "Copy" action
— a share sheet's "Copy Link", for instance — is rendered nowhere on screen, so it
cannot be located or extracted; and WDA's
GET /sourcereturns 500 while a sharesheet (
UIActivityViewController) is presented, so the accessibility tree is nohelp either. Reading the pasteboard is the only path, and it is only useful to an
agent if the agent can actually choose it.
Design note
Both actions are registered locally in
packages/ios/src/device.ts, following theexisting
IOSHomeButton/IOSAppSwitcherpattern for iOS-specific actions,rather than as canonical cross-platform actions in
@midscene/core.A canonical
ClipboardGet/ClipboardSetpair would be defensible — threebackends already exist in the repo (iOS via WDA here, Android via
dumpsys,desktop via
clipboardyinpackages/computer/src/device.ts, currently used onlyinternally for
typeViaClipboard). But that touches core's action registry and isa naming/schema decision for maintainers, so it is deliberately left out of this
PR. Happy to open an issue for it if there is interest.
Validation
npx nx test ios— 13 files, 163 tests passed (2 new, covering dispatch throughactionSpace()into the WDA client for both directions)npx nx build ios— successpnpm run lint— clean