aboutsummaryrefslogtreecommitdiff
path: root/src/domains/click.ts
diff options
context:
space:
mode:
authoramankumarsingh77 <amankumarsingh7702@gmal.com>2026-05-06 00:06:49 +0530
committeramankumarsingh77 <amankumarsingh7702@gmal.com>2026-05-06 00:06:49 +0530
commit589cd294931c01b3748779635d17a13dd240af19 (patch)
treeb635f9fa443cd120de905660fe0d2665184e9a8c /src/domains/click.ts
parent20ceaa1c6ade4950ed7d16caa5269ad2f4805db4 (diff)
release: v0.4.0 — accessibility snapshot, network capture, snapshot-first prompts
Adds two new tools and pivots the agent's default workflow away from screenshots toward structured page data. browser_snapshot — CDP accessibility tree (roles, names, states, hierarchy) with click coordinates @(x,y) attached to every interactive element via DOM.getBoxModel (parallel, 1.5s aggregate budget). Pass directly to browser_click — no screenshot round-trip. Optional includeScreenshot:true for visual confirmation. format:"json" returns a slim structure with full box per node. browser_network_requests — replaces the deprecated browser_get_network_log placeholder. Filters by urlPattern (substring or /regex/), method, status, resource type, recency. Optional response-body capture under a 5s aggregate budget with 50KB per-body cap. Buffer is page-scoped (cleared on tab switch), 500 records/tab, with a bufferOverflowed flag on drain. CDP plumbing: in-process aggregator in src/cdp/network-buffer.ts wired into the existing single-consumer event loop. Accessibility domain enabled alongside Page/DOM/Runtime/Network on every attach. Ctrl+O expand/collapse on three tools' renderResult — browser_snapshot, browser_network_requests, browser_execute_js. Compact-by-default views with keyHint("app.tools.expand") that respects user keybinding remaps. Tool prompts pivoted snapshot-first: browser_screenshot is documented as LAST RESORT for visual verification only; browser_snapshot is the default for understanding pages; browser_execute_js is the default for surgical reads. browser_click and browser_open_urls guidance follows. SKILL.md collapsed from ~342 to ~53 lines — keeping only what isn't already in tool prompts: the decision tree, connection rules, browser_run_script daemon bindings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diffstat (limited to 'src/domains/click.ts')
-rw-r--r--src/domains/click.ts11
1 files changed, 6 insertions, 5 deletions
diff --git a/src/domains/click.ts b/src/domains/click.ts
index c269dd8..9946664 100644
--- a/src/domains/click.ts
+++ b/src/domains/click.ts
@@ -41,14 +41,15 @@ export const clickTool = defineBrowserTool({
name: "browser_click",
label: "Browser Click",
description:
- "Click at viewport CSS-pixel coordinates. Compositor-level click works through iframes, shadow DOM, and cross-origin content. Use browser_screenshot first to find coordinates.",
+ "Click at viewport CSS-pixel coordinates. Compositor-level click works through iframes, shadow DOM, and cross-origin content. Get coordinates from browser_snapshot's @(x,y) hints — do not screenshot to find click targets.",
promptSnippet: "Click at pixel coordinates (x, y) on the page",
promptGuidelines: [
- "Use browser_click for all clicks. Coordinates are viewport CSS pixels (not device pixels).",
- "Capture a browser_screenshot BEFORE clicking to find the right coordinates.",
- "Capture another browser_screenshot AFTER clicking to verify the action worked.",
+ "Get (x, y) from browser_snapshot — the outline shows '@(x,y)' for every interactive element. NO screenshot needed.",
+ "If the target isn't an interactive role in the snapshot, fall back to browser_execute_js with `el.getBoundingClientRect()` — still NO screenshot.",
+ "Coordinates are viewport CSS pixels (not device pixels).",
+ "After clicking, verify with browser_snapshot (or browser_execute_js for a specific value). Only screenshot if you need to confirm visual rendering.",
"Compositor-level clicks pass through iframes, shadow DOM, and cross-origin content — no selector needed.",
- "If a click doesn't register, try setting BH_DEBUG_CLICKS=1 to get annotated screenshots showing exact click positions.",
+ "If a click doesn't register, set BH_DEBUG_CLICKS=1 to get annotated screenshots showing exact click positions (debug only).",
"For React/Vue components that don't respond to clicks, try browser_dispatch_key to send DOM-level events.",
],
parameters: ClickArgs,