Skip to content

feat(android): add clipboard access and expose it in the action space - #2959

Open
zubeyralmaho wants to merge 2 commits into
web-infra-dev:mainfrom
zubeyralmaho:feat/android-clipboard
Open

feat(android): add clipboard access and expose it in the action space#2959
zubeyralmaho wants to merge 2 commits into
web-infra-dev:mainfrom
zubeyralmaho:feat/android-clipboard

Conversation

@zubeyralmaho

Copy link
Copy Markdown

What

System clipboard access for Android, in both directions, reachable by the planner:

  • AndroidDevice.getClipboardText() — reads via dumpsys clipboard
  • AndroidDevice.setClipboardText() — writes via yadb's -writeClipboard
  • AndroidGetClipboard / AndroidSetClipboard registered in actionSpace()
    (aliases agent.getClipboardText() / agent.setClipboardText())

Why

A value that an app exposes only through a native "Copy" action — a share sheet's
"Copy Link", for instance — is rendered nowhere on screen, so it cannot be located
or extracted. Reading the clipboard is the only path to it.

Registering the actions is what makes that usable: without them the capability is
reachable only from hand-written code, and a prompt like "copy the invite link and
read it back"
cannot work.

How

Writing has no dumpsys equivalent, and stock Android exposes no public
cmd clipboard set. It does, however, turn out that the yadb build already
vendored in packages/android/bin/yadb (v1.1.1) supports -readClipboard,
-writeClipboard and -pasteClipboard, driving the platform IClipboard binder
directly. So the write path reuses the same app_process invocation already used
for IME input and forced screenshots — no new dependency — and escapes its argument
with the existing escapeForShell() helper, so quotes and newlines survive
transport.

Both actions are registered locally in createPlatformActions, matching how
RunAdbShell / Launch / Terminate are declared, rather than as canonical
cross-platform actions in @midscene/core. A canonical ClipboardGet /
ClipboardSet pair would be defensible — iOS can back it via WDA's pasteboard
endpoints and desktop already drives clipboardy in
packages/computer/src/device.ts — but that touches core's action registry and is
a naming/schema decision for maintainers, so it is deliberately out of scope here.
Happy to open an issue for it if there is interest.

Follow-up worth considering

The read path parses dumpsys clipboard output, whose wording differs across
Android versions and OEMs (ClipData { text/plain "..." } vs
ClipData.Item { T:"..." }), and returns '' when nothing matches — which
conflates "clipboard is empty" with "output was not parseable". Since the vendored
yadb also offers -readClipboard through the same binder used for writing, moving
the read path over would remove the regex heuristics and make that distinction
possible. Left out of this PR to keep the change reviewable.

Validation

  • npx nx test android — 18 files, 365 tests passed (7 new, covering the yadb
    command construction, its shell escaping, and dispatch through actionSpace()
    in both directions)
  • npx nx build android — success
  • pnpm run lint — clean

Note on scope of that coverage: the unit tests mock adb, so they verify that the
correct command is constructed and dispatched, not yadb's on-device behaviour. The
-writeClipboard path has not been exercised against a physical device in this
change.

Add AndroidDevice.getClipboardText(), parsed from `dumpsys clipboard`
(there is no public `cmd clipboard get` on stock Android). Several known
dumpsys phrasings are tried (they differ across Android
versions/OEMs -- e.g. `ClipData { text/plain "..." }` vs `ClipData.Item
{ T:"..." }`); returns an empty string if the clipboard is genuinely
empty or not text.

Same motivation as the iOS pasteboard support in this PR: a value an app
only exposes through a native "Copy" action in a share sheet (e.g. "Copy
Link" for a generated invite/meeting link) has no other on-screen
representation to read.

Write support is intentionally left out here: there is no reliable,
version-stable way to set the Android clipboard from a plain ADB shell
command without a helper APK or a fragile `service call` binder
invocation, unlike iOS's WDA-native setPasteboard.

Test plan:
- npx nx test android (327 passed)
- npx nx build android
… space

Add AndroidDevice.setClipboardText() via yadb's `-writeClipboard`, and register
both directions in actionSpace() as AndroidGetClipboard / AndroidSetClipboard.

Writing has no `dumpsys` equivalent and stock Android exposes no public
`cmd clipboard set`, so the write path goes through yadb -- already required
here for IME input and forced screenshots -- which drives the platform
IClipboard binder directly. Text is escaped with the same escapeForShell()
helper used for yadb keyboard input, so quotes and newlines survive transport.

Registering the actions is what makes the capability reachable by the planner.
Without it the clipboard is available only to hand-written code, and a prompt
like "copy the invite link and read it back" cannot work -- which is the case
the capability exists for, since a value behind a native "Copy" action is
rendered nowhere on screen and cannot be located or extracted.

Both actions are registered locally in createPlatformActions, matching how
RunAdbShell / Launch / Terminate are declared, rather than as canonical
cross-platform actions in @midscene/core. interfaceAlias also exposes them as
agent.getClipboardText() and agent.setClipboardText().

Test plan:
- npx nx test android (365 passed)
- npx nx build android
- pnpm run lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant