aboutsummaryrefslogtreecommitdiff
path: root/src/domains/ref-resolve.ts
AgeCommit message (Collapse)Author
2026-06-29feat: address elements by stable refs across re-renders (#8)Aman Kumar
Interaction tools could only target elements by raw viewport coordinates (browser_click) or model-guessed CSS selectors (browser_fill/focus/ select_option/upload). Neither survives a re-render: on React/SPA forms, every save or edit reflows the page, so captured @(x,y) coordinates go stale and guessed selectors miss — leaving the agent to fall back to hand-rolled execute_js DOM walks and unnoticed failed saves. Add a stable ref system, mirroring Playwright MCP / browser-use: - snapshot assigns each interactive element a ref (e1, e2, …), shown as [eN] in the outline, and publishes a per-tab ref → backendNodeId map - interaction tools accept an optional `ref` that re-resolves the element at call time (via DOM.resolveNode / getBoxModel), so it stays valid across re-renders; coordinates/selectors remain as fallbacks - mutating tools append a compact diff of changed / newly-appeared (*) / removed interactive elements, so silent failed saves surface immediately - stale refs return an actionable "re-snapshot" error instead of a miss Ref map is replaced per snapshot (bounded memory); fill keeps the native _valueTracker write path so React controlled inputs still update. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>