feat(android): add clipboard access and expose it in the action space - #2959
Open
zubeyralmaho wants to merge 2 commits into
Open
feat(android): add clipboard access and expose it in the action space#2959zubeyralmaho wants to merge 2 commits into
zubeyralmaho wants to merge 2 commits into
Conversation
Add AndroidDevice.getClipboardText(), parsed from `dumpsys clipboard`
(there is no public `cmd clipboard get` on stock Android). Several known
dumpsys phrasings are tried (they differ across Android
versions/OEMs -- e.g. `ClipData { text/plain "..." }` vs `ClipData.Item
{ T:"..." }`); returns an empty string if the clipboard is genuinely
empty or not text.
Same motivation as the iOS pasteboard support in this PR: a value an app
only exposes through a native "Copy" action in a share sheet (e.g. "Copy
Link" for a generated invite/meeting link) has no other on-screen
representation to read.
Write support is intentionally left out here: there is no reliable,
version-stable way to set the Android clipboard from a plain ADB shell
command without a helper APK or a fragile `service call` binder
invocation, unlike iOS's WDA-native setPasteboard.
Test plan:
- npx nx test android (327 passed)
- npx nx build android
… space Add AndroidDevice.setClipboardText() via yadb's `-writeClipboard`, and register both directions in actionSpace() as AndroidGetClipboard / AndroidSetClipboard. Writing has no `dumpsys` equivalent and stock Android exposes no public `cmd clipboard set`, so the write path goes through yadb -- already required here for IME input and forced screenshots -- which drives the platform IClipboard binder directly. Text is escaped with the same escapeForShell() helper used for yadb keyboard input, so quotes and newlines survive transport. Registering the actions is what makes the capability reachable by the planner. Without it the clipboard is available only to hand-written code, and a prompt like "copy the invite link and read it back" cannot work -- which is the case the capability exists for, since a value behind a native "Copy" action is rendered nowhere on screen and cannot be located or extracted. Both actions are registered locally in createPlatformActions, matching how RunAdbShell / Launch / Terminate are declared, rather than as canonical cross-platform actions in @midscene/core. interfaceAlias also exposes them as agent.getClipboardText() and agent.setClipboardText(). Test plan: - npx nx test android (365 passed) - npx nx build android - pnpm run lint
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
System clipboard access for Android, in both directions, reachable by the planner:
AndroidDevice.getClipboardText()— reads viadumpsys clipboardAndroidDevice.setClipboardText()— writes via yadb's-writeClipboardAndroidGetClipboard/AndroidSetClipboardregistered inactionSpace()(aliases
agent.getClipboardText()/agent.setClipboardText())Why
A value that an app exposes only through a native "Copy" action — a share sheet's
"Copy Link", for instance — is rendered nowhere on screen, so it cannot be located
or extracted. Reading the clipboard is the only path to it.
Registering the actions is what makes that usable: without them the capability is
reachable only from hand-written code, and a prompt like "copy the invite link and
read it back" cannot work.
How
Writing has no
dumpsysequivalent, and stock Android exposes no publiccmd clipboard set. It does, however, turn out that the yadb build alreadyvendored in
packages/android/bin/yadb(v1.1.1) supports-readClipboard,-writeClipboardand-pasteClipboard, driving the platformIClipboardbinderdirectly. So the write path reuses the same
app_processinvocation already usedfor IME input and forced screenshots — no new dependency — and escapes its argument
with the existing
escapeForShell()helper, so quotes and newlines survivetransport.
Both actions are registered locally in
createPlatformActions, matching howRunAdbShell/Launch/Terminateare declared, rather than as canonicalcross-platform actions in
@midscene/core. A canonicalClipboardGet/ClipboardSetpair would be defensible — iOS can back it via WDA's pasteboardendpoints and desktop already drives
clipboardyinpackages/computer/src/device.ts— but that touches core's action registry and isa naming/schema decision for maintainers, so it is deliberately out of scope here.
Happy to open an issue for it if there is interest.
Follow-up worth considering
The read path parses
dumpsys clipboardoutput, whose wording differs acrossAndroid versions and OEMs (
ClipData { text/plain "..." }vsClipData.Item { T:"..." }), and returns''when nothing matches — whichconflates "clipboard is empty" with "output was not parseable". Since the vendored
yadb also offers
-readClipboardthrough the same binder used for writing, movingthe read path over would remove the regex heuristics and make that distinction
possible. Left out of this PR to keep the change reviewable.
Validation
npx nx test android— 18 files, 365 tests passed (7 new, covering the yadbcommand construction, its shell escaping, and dispatch through
actionSpace()in both directions)
npx nx build android— successpnpm run lint— cleanNote on scope of that coverage: the unit tests mock
adb, so they verify that thecorrect command is constructed and dispatched, not yadb's on-device behaviour. The
-writeClipboardpath has not been exercised against a physical device in thischange.