feat(core): load element XPath maps in aiAct

[quanruzhuoxiu]

Summary

• add loadElementXpaths to aiAct for loading name-to-XPath maps from YAML
• pass the map to planning, match the selected semantic locator name, and resolve it through the supplied XPath before execution
• prefer an exact mapped XPath over accidental planner coordinates, including model outputs shaped as { description, rect, center }
• preserve fallback AI locate for unmapped or non-DOM targets, task-cache isolation, extra-action expansion, blank per-call context, and YAML replay behavior
• add reproducible web benchmarks for a form, a dense action table, and five complex page archetypes

API

elements:
  First name input: //*[@id="first-name"]
  Last name input: //*[@id="last-name"]

await agent.aiAct('Fill the profile form', {
  loadElementXpaths: ['/path/to/element-xpaths.yaml'],
});

Cross-scene small-model evidence

The final benchmark uses qwen3-vl-8b-instruct, retry 0, deepThink: false, cacheable: false, and identical pages/prompts between arms. It covers four DOM-backed hard-page archetypes plus a Canvas boundary case:

• repeated grouped table controls
• duplicated version text in an open dropdown and results table
• 32 dense icon-only card tools
• overlapping dialog, drawer, and toast close controls
• Canvas-only connection controls absent from the XPath map

The formal matrix contains 18 targets × 3 rounds × 2 arms = 108 physical runs. All 108 produced valid comparison evidence.

DOM-backed targets	Baseline	XPath Map
Strict success	33/48 (68.75%)	48/48 (100%)
Exact XPath hits	0/48	48/48
Target AI Locate usages	0	0
Mean total tokens	34,386	15,984
Mean LLM time	12,149 ms	5,910 ms
Mean wall time	14,092 ms	6,787 ms

Paired outcomes were 15 baseline-failure → Map-success, 33 both-success, 0 reverse regressions, and 0 both-failure (two-sided exact McNemar p = 0.000061). Every mapped success recorded hitBy.from = User expected path, the exact expected XPath, and one exact DOM event.

The six Canvas Map runs recorded xpathHits = 0 and aiLocateUsages = 1; they are retained as a negative boundary and are not attributed to XPath.

Validation

• pnpm run lint
• pnpm --filter @midscene/core exec vitest --run tests/unit-test/element-xpaths.test.ts (5 passed)
• pnpm exec nx test @midscene/core (127 files; 1387 passed, 8 skipped)
• pnpm exec nx build @midscene/core
• fixture DOM validation: 57/57 YAML XPaths resolve to exactly one element
• AI_TEST_TYPE=web ... vitest --run tests/ai/web/puppeteer/element-xpath-complex-benchmark.test.ts (108 independent model runs)

The AI benchmarks are gated and skip unless their explicit benchmark variant is selected.

Experiment document

• XPath Map × small-model final experiment

Published evidence

• Goofy cross-scene evidence site (108 runs, five test scenes, ten representative Midscene reports, and full evidence JSON)
• Feishu final experiment document

[cloudflare-workers-and-pages]

Deploying midscene with    Cloudflare Pages

Latest commit:	9f0ebf3
Status:	✅  Deploy successful!
Preview URL:	https://f4b65cc4.midscene.pages.dev
Branch Preview URL:	https://feat-ai-act-load-element-xpa.midscene.pages.dev

View logs