Skip to content

feat(doubao): add edit-image command for text-to-image and image editing - #2447

Open
RusianHu wants to merge 8 commits into
jackwener:mainfrom
RusianHu:feat/doubao-edit-image
Open

RusianHu wants to merge 8 commits into
jackwener:mainfrom
RusianHu:feat/doubao-edit-image

Conversation

@RusianHu

@RusianHu RusianHu commented Aug 31, 2026

Copy link
Copy Markdown

Description

Adds opencli doubao edit-image: one command covering Doubao web image generation (text-to-image) and editing (image + prompt), each in a fresh conversation, waiting for generation and downloading every candidate at full resolution (2048px).

# text-to-image (Doubao returns a 4-image grid, all downloaded)
opencli doubao edit-image "画一张图:一只橘猫在月光下的屋顶弹钢琴"

# image edit
opencli doubao edit-image "把这张人设图的背景改成樱花粉色渐变,人物保持完全不变" --image C:/images/persona.png

Implementation notes (UI_SELECTOR, all anchors verified against the live logged-in doubao.com DOM on 2026-08-31):

  • Upload via CDP setFileInputFiles on the hidden input[type=file] (accept includes png/jpeg/jpg/webp).
  • Composer anchors cover both current generations: 2026-07 Semi textarea and 2026-08 tiptap ProseMirror.
  • Submit: the send control is button#flow-end-msg-send (re-verified 2026-09-01). Submission is verified by the composer clearing — the URL flips to /chat/local_<id> and the user bubble renders optimistically even when the submit event was swallowed, so neither is a success signal on its own.
  • Result extraction: img[src*="rc_gen_image"] inside the message list; the signed CDN URL downloads fine outside the page. One asset is served as several ~tplv templates on random CDN hosts (ds_wm_* 384px thumbnail, i_pre_wm_* 2048px+ variant), so rows are deduped by asset pathname with the hi-res template preferred.
  • The poll loop recovers by re-entering the conversation URL if the owned-session tab is recycled mid-wait.
  • Expected failures are typed (ArgumentError / CommandExecutionError / TimeoutError), including Doubao-side generation-failure text.
  • The internal SSE generation endpoint is deliberately not used (no external contract); every UI anchor above is observable and tested.

Independently mergeable — only imports pre-existing utils.js exports; no dependency on #2133 (though they touch the same composer generation).

Update 2026-09-01 (live re-verification + hardening)

Re-testing against the logged-in doubao.com UI on 2026-09-01 surfaced three issues, addressed in the two new commits:

  1. The synthetic-Enter submit regressed. Doubao ships UI changes fast: the send control is button#flow-end-msg-send again, and a synthetic KeyboardEvent('Enter') no longer reaches the submit handler. Worse, a swallowed submit still flips the URL to /chat/local_<id> and renders the user bubble optimistically — the original URL-based success check then polls a conversation the server never saw until timeout. edit-image now submits through clickDoubaoSendButton() (shared with ask/send, see below) and treats composer-clear as the real-sent signal, re-clicking with the other click flavor when text is still stuck.
  2. No single click flavor is reliable. An in-page synthetic .click() is intermittently swallowed, and a trusted CDP coordinate click lands in the void when the bridge window is hidden (innerWidth/innerHeight collapse to 0 — same class of issue as fix(chatgpt): adapt to 2026-08 chatgpt.com UI and zero-size bridge viewports #2436). clickDoubaoSendButton() in utils.js surfaces a collapsed viewport via Page.bringToFront (raw CDP), prefers the JS click (immune to window visibility), and keeps the trusted CDP click as the other flavor; sendDoubaoMessage() gains the same verify-and-escalate treatment (button click → native Enter → native-flavor button click) with a diagnostic error if every flavor is swallowed. Live doubao ask round-trips verified after the fix (3+4 → 7, 9×9 → 81).
  3. Result dedup leaked duplicates and random sizes. The same generated asset is served under several ~tplv templates on different CDN hosts (p3/p11/p26), which defeated the full-URL dedup — runs returned 6 rows for 4 images, randomly 384px or 2048px. Rows are now deduped by asset pathname with the i_pre_wm_* hi-res template preferred, and the poll waits for the key set to stabilize so the hi-res variant (which lands a beat after the 384px thumbnail) is not missed. Two live text-to-image runs each returned 4 unique 2048×2048 images.

Tests updated for the new submit flow (57/57 doubao tests green); opencli validate doubao PASS.

Related issue: none

Update 2026-09-13 (upload menu fix for the 2026-09 composer)

Another live drift, caught by re-running both modes against the logged-in doubao.com UI on 2026-09-13:

  1. Edit mode broke at upload. The composer file input is now a Radix-menu input rendered with class="hidden" (display:none) until the composer "+" trigger opens its menu. The extension's setFileInput works by clicking the input and intercepting Page.fileChooserOpened, and a display:none input never opens a chooser — every --image run failed with Page.fileChooserOpened not received within 5s. Text-to-image (no --image) was and remains unaffected.
  2. Fix (f641697): edit-image.js gains a best-effort openDoubaoComposerMenu() step before setFileInput: it natively clicks the "+" trigger via a trusted CDP click (page.nativeClick — synthetic pointerdown/click sequences are ignored by the page) and waits until the file input drops its hidden class, then hands off to the unchanged setFileInput flow. No-op on older DOMs where the input is already visible; existing mocks and assertions are unaffected (the menu probe returns null on the test page mock and the step is skipped).
  3. Live verification (2026-09-13, logged-in doubao.com): text-to-image returned 4/4 images downloaded at full resolution; edit mode (image upload + edit prompt) completed a full round-trip — upload, submit (composer cleared, URL flipped to /chat/local_<id>), generation poll, and download of the edited image.

Raw original download (291d9b0, same day)

While integrating, one more property of the watermark surfaced: it is applied by the Imagex CDN template, not baked into pixels. The generation SSE stream carries every variant per image, including image.image_ori_raw.url on the unmarked image_raw template with its own server-side signature — the DOM lk3s signature is bound to its template, so rewriting the suffix on a watermarked URL fails with 403 (verified).

edit-image now wraps page JSON.parse while the run's patches stream in, collects image_ori_raw URLs keyed by asset path, and downloads the unmarked raw original by default; --watermark opts back into the watermarked CDN variants. Falls back to the watermarked URL when the hook misses an asset, so behavior degrades gracefully. Existing mocks are unaffected (evaluate returns undefined in tests → the hook is a no-op).

Live 2026-09-13: text-to-image 4/4 unmarked full-resolution (1728x2304, was 384-2048 compressed watermarked); edit mode round-tripped unmarked at full resolution.

⚠ The watermarked variant carries the AI-generated content label required by Chinese regulation (AI content labeling measures, effective 2025-09-01). The --watermark flag exists so users can make an informed choice; label-removal for redistribution is the user's responsibility.

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 🌐 New site adapter
  • 📝 Documentation
  • ♻️ Refactor
  • 🔧 CI / build / tooling

Checklist

  • I ran the checks relevant to this PR
  • I updated tests or docs if needed
  • I included output or screenshots when useful

Documentation (if adding/modifying an adapter)

  • Added doc page under docs/adapters/ (if new adapter) — N/A, existing adapter
  • Updated docs/adapters/index.md table
  • Updated sidebar in docs/.vitepress/config.mts (if new adapter) — N/A, existing adapter
  • Updated README.md / README.zh-CN.md when command discoverability changed — N/A (README lists adapters, not commands; discoverability is via opencli list)
  • Used positional args for the command's primary subject unless a named flag is clearly better (<prompt> positional; --image/--out/--timeout named flags)
  • Normalized expected adapter failures to CliError subclasses instead of raw Error

Checks run locally

npx tsc --noEmit                → clean
npx vitest run clis/doubao      → 47 passed (32 existing + 15 new)
npm run check:typed-error-lint  → OK, no new violations
npm run check:silent-column-drop→ OK, no new violations
npm run build                   → manifest 1367 entries
node dist/src/main.js validate doubao → PASS, 12 commands, 0 errors 0 warnings

Screenshots / Output

Sample run (2026-08-31, live):

$ opencli doubao edit-image "画一张图:黄昏的京都小巷,一位穿和服的女孩撑着油纸伞走过,水彩插画风格" --out ./results
- Index: 1
  ConversationId: '38439595492180994'
  SavedTo: ./results/doubao_edit_38439595492180994_1.png
- Index: 2
  ConversationId: '38439595492180994'
  SavedTo: ./results/doubao_edit_38439595492180994_2.png
- Index: 3
  ConversationId: '38439595492180994'
  SavedTo: ./results/doubao_edit_38439595492180994_3.png
- Index: 4
  ConversationId: '38439595492180994'
  SavedTo: ./results/doubao_edit_38439595492180994_4.png

Edit mode output has the same shape (single result when Doubao returns one candidate).

Generate images from a prompt (text-to-image) or edit an uploaded image
with a prompt (image edit) in a new Doubao conversation, wait for
generation, and download every generated candidate at full resolution.

- UI_SELECTOR strategy, all anchors verified against the live logged-in
  doubao.com DOM (2026-08-31): hidden input[type=file] upload via CDP
  setFileInputFiles, tiptap ProseMirror / Semi textarea composer,
  synthetic KeyboardEvent('Enter') submit (the 2026-08 composer's send
  control is a div.send-btn-wrapper with no button semantics, and CDP
  key events do not reach the tiptap keymap), rc_gen_image result
  extraction with signed-CDN download outside the page.
- Poll loop recovers by re-entering the conversation URL when the
  owned-session tab gets recycled mid-wait.
- Expected failures surface as CliError subclasses (ArgumentError /
  CommandExecutionError / TimeoutError).
- 15 new tests (JSDOM script coverage + mocked command flows); docs
  updated (docs/adapters/browser/doubao.md, docs/adapters/index.md).
…bmit check

Fresh conversations now sometimes get /chat/local_<digits> URLs instead
of numeric-only ids (observed live 2026-08-31 23:5x); the submit
success check only matched digits and falsely reported the prompt as
unsubmitted.
Bursts of automated image requests can trip Doubao risk control, which
injects a captcha iframe; image submissions then silently never
generate. Run detectDoubaoVerificationScript on the submit-failure and
timeout paths so the command reports the challenge immediately instead
of burning the whole timeout window. Export the shared detector from
utils.js for reuse.
The mocked page.wait resolved instantly, so the timeout-path test spun
the poll loop hot for the whole real-time deadline and exhausted the
worker heap. Advance fake system time from the wait mock instead, and
dispatch remaining evaluate mocks by script content.
@RusianHu

Copy link
Copy Markdown
Author

Two live findings from continued testing tonight (both handled in follow-up commits):

  1. local_ prefixed conversation ids — fresh conversations now sometimes get /chat/local_<digits> URLs instead of numeric-only ids; the submit success check initially only matched digits and falsely reported the prompt as unsubmitted (b7fd947). Regression test included.
  2. Risk control / captcha — bursts of automated image requests can trip Doubao's risk control: the message is accepted into the conversation but generation silently never starts, and a captcha iframe appears. The timeout/submit-failure paths now run detectDoubaoVerificationScript (exported from utils.js for reuse) and fail fast with the challenge signal instead of burning the timeout window (6cdba86).

Current checks: clis/doubao vitest 49/49, tsc --noEmit clean, both lint gates green.

Live evidence (2026-09-01, logged-in doubao.com): a submit can be
swallowed silently — the URL still flips to /chat/local_<id> and the
user bubble renders optimistically, but the server never receives the
message, so 'ask' waits for a reply that never comes and 'doubao ask'
times out. Two additional live findings shape the fix:

- The send control is button#flow-end-msg-send again; an in-page
  synthetic .click() on it is intermittently swallowed, and a trusted
  CDP coordinate click lands in the void when the bridge window is
  hidden (innerWidth/innerHeight collapse to 0).
- The composer only clears on a real send, which makes composer-clear
  the reliable post-submit signal.

clickDoubaoSendButton() surfaces a collapsed viewport via
Page.bringToFront (raw CDP), prefers the JS click (immune to window
visibility), and keeps the trusted CDP click as the other flavor.
sendDoubaoMessage() now verifies the composer cleared and escalates:
button click -> native Enter -> button click with the native flavor,
throwing a diagnostic error when every flavor is swallowed. Verified
live: doubao ask round-trips (3+4 -> 7, 9x9 -> 81) after the fix.
…s variants

The PR's original submit path dispatched a synthetic
KeyboardEvent('Enter') on the composer. Live re-verification on
2026-09-01 showed that mechanism regressed: Enter no longer reaches the
submit handler, the send control is button#flow-end-msg-send again, and
the swallowed submit still flips the URL to /chat/local_<id> with an
optimistic user bubble — the old success check then polled a dead
conversation until timeout.

- Submit through clickDoubaoSendButton (verified composer-clear),
  with the in-page script (button click, synthetic Enter fallback) as
  the second flavor and a native-flavor re-click as the third.
- Result extraction: one generated asset is served as several ~tplv
  templates on random CDN hosts (p3/p11/p26) — ds_wm_* is a 384px
  thumbnail, i_pre_wm_* the 2048px+ variant. Dedup by asset pathname
  and prefer the hi-res template instead of returning duplicate rows
  with a random size; wait for the key set to stabilize so the hi-res
  variant (landing a beat later) is not missed.

Verified live on 2026-09-01: two text-to-image runs return 4 unique
2048x2048 images each; opencli validate doubao PASS.
@RusianHu

RusianHu commented Sep 1, 2026

Copy link
Copy Markdown
Author

Updated the branch with 2026-09-01 live re-verification (details in the PR description's Update 2026-09-01 section):

  • The 2026-08-31 synthetic-Enter submit regressed against the current UI — the send control is button#flow-end-msg-send again, and a swallowed submit still flips the URL to /chat/local_<id> with an optimistic user bubble, so the URL-based success check polled a conversation the server never saw. Submit now goes through a verified send-button click with composer-clear as the real-sent signal, escalating across click flavors (JS click → native Enter → trusted CDP click) — this also hardens ask/send, which failed the same way in live testing today.
  • Collapsed-viewport (0x0 bridge window) coordinate clicks are surfaced via Page.bringToFront before clicking — same failure class as fix(chatgpt): adapt to 2026-08 chatgpt.com UI and zero-size bridge viewports #2436 fixed for chatgpt.
  • Result extraction now dedupes the multi-host ~tplv variants of one asset and prefers the i_pre_wm_* 2048px+ template (live runs previously returned 6 rows for 4 images with random 384px/2048px sizes).

Live evidence (all logged-in, 2026-09-01): two text-to-image runs → 4 unique 2048×2048 images each; doubao ask 3+4 → 7 and 9×9 → 81 round-trips; 57/57 doubao tests, opencli validate doubao PASS.

The 2026-09 doubao.com composer renders input[type=file] inside a Radix
menu with class="hidden" until the composer "+" trigger opens. A
display:none input cannot open a file chooser, so the extension's
setFileInput (click + Page.fileChooserOpened interception) timed out
after 5s on every --image run. Best-effort openDoubaoComposerMenu()
natively clicks the "+" trigger (CDP trusted click; synthetic events
are ignored by the page) and waits for the input to become visible
before handing off to the existing setFileInput flow.

Verified live against logged-in doubao.com on 2026-09-13: edit mode
round-trip succeeds (upload -> prompt -> submit -> generated image
downloaded); text-to-image mode unaffected (4/4 images).
…stream by default

Doubao watermarks are applied by the Imagex CDN template, not baked into
pixels. The generation SSE stream carries every variant per image,
including image.image_ori_raw.url on the unmarked image_raw template
with its own server-side signature (the DOM lk3s signature is bound to
its template, so rewriting the suffix on a watermarked URL fails with
403).

edit-image now wraps page JSON.parse while the run's patches stream in,
collecting image_ori_raw URLs keyed by asset path, and swaps each
downloaded URL for the raw original by default. --watermark opts back
into the watermarked CDN variants. Falls back to the watermarked URL
when the hook missed the asset, so behavior degrades gracefully.

Note: the watermarked variant carries the AI-generated content label
required by Chinese regulation (AI content labeling measures,
effective 2025-09); the flag exists so downstream users can make an
informed choice.

Verified live against logged-in doubao.com on 2026-09-13: text-to-image
returned 4/4 unmarked full-resolution images (1728x2304); edit mode
(upload + edit prompt) also round-tripped unmarked at full resolution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant