aboutsummaryrefslogtreecommitdiff
path: root/skills
diff options
context:
space:
mode:
authoramankumarsingh77 <amankumarsingh7702@gmal.com>2026-05-06 00:06:49 +0530
committeramankumarsingh77 <amankumarsingh7702@gmal.com>2026-05-06 00:06:49 +0530
commit589cd294931c01b3748779635d17a13dd240af19 (patch)
treeb635f9fa443cd120de905660fe0d2665184e9a8c /skills
parent20ceaa1c6ade4950ed7d16caa5269ad2f4805db4 (diff)
release: v0.4.0 — accessibility snapshot, network capture, snapshot-first prompts
Adds two new tools and pivots the agent's default workflow away from screenshots toward structured page data. browser_snapshot — CDP accessibility tree (roles, names, states, hierarchy) with click coordinates @(x,y) attached to every interactive element via DOM.getBoxModel (parallel, 1.5s aggregate budget). Pass directly to browser_click — no screenshot round-trip. Optional includeScreenshot:true for visual confirmation. format:"json" returns a slim structure with full box per node. browser_network_requests — replaces the deprecated browser_get_network_log placeholder. Filters by urlPattern (substring or /regex/), method, status, resource type, recency. Optional response-body capture under a 5s aggregate budget with 50KB per-body cap. Buffer is page-scoped (cleared on tab switch), 500 records/tab, with a bufferOverflowed flag on drain. CDP plumbing: in-process aggregator in src/cdp/network-buffer.ts wired into the existing single-consumer event loop. Accessibility domain enabled alongside Page/DOM/Runtime/Network on every attach. Ctrl+O expand/collapse on three tools' renderResult — browser_snapshot, browser_network_requests, browser_execute_js. Compact-by-default views with keyHint("app.tools.expand") that respects user keybinding remaps. Tool prompts pivoted snapshot-first: browser_screenshot is documented as LAST RESORT for visual verification only; browser_snapshot is the default for understanding pages; browser_execute_js is the default for surgical reads. browser_click and browser_open_urls guidance follows. SKILL.md collapsed from ~342 to ~53 lines — keeping only what isn't already in tool prompts: the decision tree, connection rules, browser_run_script daemon bindings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Diffstat (limited to 'skills')
-rw-r--r--skills/pi-browser-harness/SKILL.md282
1 files changed, 32 insertions, 250 deletions
diff --git a/skills/pi-browser-harness/SKILL.md b/skills/pi-browser-harness/SKILL.md
index 2116d4c..7de9800 100644
--- a/skills/pi-browser-harness/SKILL.md
+++ b/skills/pi-browser-harness/SKILL.md
@@ -1,271 +1,53 @@
---
name: pi-browser-harness
-description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome.
+description: Direct browser control via CDP. Use when the user wants to automate, scrape, test, or interact with web pages. Connects to the user's already-running Chrome. Default to browser_snapshot for understanding pages and browser_execute_js for surgical reads — browser_screenshot is for visual verification only.
---
# pi-browser-harness
-Direct browser control via CDP. Gives pi agents full control of a real Chrome browser.
-Browser tools coexist with all standard pi tools (read, bash, edit, write, etc.).
+Direct browser control of the user's running Chrome via CDP.
-## Setup
+## Tool hierarchy
-Run `/browser-setup` to install browser-harness and connect to Chrome. Chrome must be running
-with remote debugging enabled. Start Chrome with:
-
-```
---remote-debugging-port=9222
-```
-
-Check status with `/browser-status`. Restart the daemon with `/browser-reload-daemon`.
-
-## Core Workflow
-
-```
-browser_screenshot() → browser_click(x, y) → browser_screenshot()
```
+What do you need to know?
-1. **Screenshot first** — see what's on screen before any interaction
-2. **Coordinate click** — read the target pixel from the screenshot, click at (x, y)
-3. **Screenshot again** — verify the action worked
-
-Compositor-level clicks (`Input.dispatchMouseEvent`) work through iframes, shadow DOM,
-and cross-origin content. No selectors needed.
-
-## Parallelization
-
-**Always parallelize when possible.** The harness automatically serializes mutation tools so they never race on shared state. Emit independent tools in the same turn for better performance.
-
-### What can run in parallel
-
-- **Observation tools** (`browser_screenshot`, `browser_page_info`, `browser_execute_js`, `browser_list_tabs`, `browser_current_tab`, `browser_http_get`, `browser_print_to_pdf`) can run in parallel with each other and with mutation tools.
-
-- **Opening result pages:** Use `browser_open_urls` to open multiple URLs in parallel tabs. After opening, use `browser_list_tabs` to see the new tabs and `browser_switch_tab` to visit each one.
-
-- **API calls:** `browser_http_get` calls are independent of browser state — fire multiple GETs in parallel.
-
-- **Read + browser tools:** Use `read` and `browser_*` tools in the same turn — reading files or previous results is independent of browser actions.
-
-### What must be sequential (harness enforces this)
-
-- **Mutation tools** (`click`, `type`, `scroll`, `navigate`, `switch_tab`, `press_key`, `drag_and_drop`, `upload_file`, `go_back`, `go_forward`, `reload`, `handle_dialog`, `viewport_resize`, `wait_for_load`, `download`) are automatically queued through a mutex. Even if you emit them in parallel, they execute serially in FIFO order. You do not need to worry about race conditions.
-
-- **Tab switching then acting:** `browser_switch_tab` must complete before acting on the switched-to tab. This is enforced automatically.
-
-### Pattern: Parallel research across tabs
-
-```
-# Step 1: Open all result pages at once
-browser_open_urls(["url1", "url2", "url3"])
-
-# Step 2: Wait for them to load, then extract in parallel
-browser_list_tabs() # get targetIds
-
-# Step 3: Extract data from each tab (parallel)
-browser_switch_tab(targetId1) → browser_screenshot() # These are sequential per tab
-browser_switch_tab(targetId2) → browser_execute_js(...) # but you batch switch+act
-```
-
-### Pattern: Parallel observation + mutation
-
-```
-# Fire observation tools while a mutation is in progress
-browser_click(x, y) # mutation (queued)
-browser_screenshot() # observation (runs immediately in parallel)
-browser_page_info() # observation (runs immediately in parallel)
-browser_http_get("https://api.example.com") # external (runs immediately in parallel)
-```
-
-## Available Tools
-
-### Navigation
-- `browser_navigate` — navigate to a URL (creates new tab on first call, reuses current tab after)
-- `browser_new_tab` — open a new tab, optionally navigate (use for first navigation)
-- `browser_open_urls` — open multiple URLs in parallel tabs (use after search to open result links)
-- `browser_go_back` / `browser_go_forward` / `browser_reload` — history navigation
-- `browser_page_info` — get URL, title, viewport, scroll position, or dialog info
-- `browser_list_tabs` / `browser_current_tab` / `browser_switch_tab` — tab management
-
-### Interaction
-- `browser_click` — click at viewport CSS-pixel coordinates
-- `browser_type` — type text into focused element
-- `browser_press_key` — press a key (Enter, Tab, Escape, arrows, etc.)
-- `browser_scroll` — scroll the page at coordinates
-
-### Visual
-- `browser_screenshot` — capture PNG or JPEG screenshot (rendered inline in the TUI)
-
-### Data Extraction
-- `browser_execute_js` — execute JavaScript and return result
-- `browser_http_get` — direct HTTP GET outside browser (10-50x faster for APIs)
-
-### Utility
-- `browser_wait` — wait N seconds
-- `browser_wait_for_load` — wait for document.readyState === 'complete'
-- `browser_handle_dialog` — accept or dismiss JS dialogs (alert/confirm/prompt)
-
-### Extending
-- `browser_run_script` — execute a temporary script file with daemon access
-
-## Pattern Reference
-
-### Navigation
-```
-browser_new_tab("https://example.com") → browser_wait_for_load() → browser_screenshot()
-```
-
-### Form Filling
-```
-browser_screenshot() → find input coordinates
-→ browser_click(x, y) → browser_type("text")
-→ browser_press_key("Tab") → browser_screenshot()
-```
-
-### Data Extraction
-```
-browser_execute_js("document.querySelector('.price').innerText")
-// or for APIs directly:
-browser_http_get("https://api.example.com/data")
-```
-
-### Scrolling
-```
-browser_screenshot() → browser_scroll({ deltaY: 500 }) → browser_screenshot()
-```
-Note: deltaY follows W3C wheel-event convention: positive=down, negative=up. Default deltaY=300 scrolls down.
-
-### Research (search engine + browser_open_urls)
-```
-browser_new_tab("https://google.com/search?q=query") → search engine results
-browser_open_urls({urls: ["url1", "url2"]}) → open result pages in parallel tabs
-browser_list_tabs() → see all open tabs with targetIds
-browser_switch_tab({targetId: "..."}) → switch to a specific tab
-browser_wait_for_load() → wait for the page to render
-browser_screenshot() → visually inspect the page
-browser_execute_js("document.querySelector('.content').innerText") → extract text
-```
-
-## Temporary Scripts
-
-When the built-in tools aren't enough, write a temporary script to disk and execute it
-with `browser_run_script`. Scripts run in the harness process with direct access to
-the browser daemon and full Node.js APIs — no dynamic tool registration needed.
-
-```
- ● agent: needs to scrape 47 paginated search result pages
+ ├─ Page structure / what's clickable / labels?
+ │ → browser_snapshot (DEFAULT — AX tree with @(x,y) per interactive element)
│
- ● write("/tmp/scrape-pages.js", "...script using daemon.cdp() and daemon.evaluateJS()...")
+ ├─ A specific element's value / attribute / coords?
+ │ → browser_execute_js (e.g. el.innerText, el.getBoundingClientRect())
│
- ● browser_run_script("/tmp/scrape-pages.js", { urls: [...] })
+ ├─ Network behavior on the current page?
+ │ → browser_network_requests
│
- ✓ script executes and returns results
-```
-
-The script is written to disk, so it's auditable and re-runnable.
-
-**When to use scripts:**
-- You're about to do the same 3+ step sequence more than once
-- You need a capability the built-in tools don't provide (e.g., bulk CSV export, structured extraction, pagination)
-- You discover a domain-specific pattern that should be reusable
-- The alternative is a fragile multi-turn loop of screenshots + clicks
-
-**What NOT to do:**
-- Don't write scripts for one-off actions — use the built-in browser_* tools directly.
-- Don't try to call browser_* tools from within a script — sequence them as separate tool calls.
-
-**Script bindings:**
-- `params` — the arguments passed to browser_run_script
-- `daemon` — browser daemon with these methods:
- - `daemon.evaluateJs(expression)` — run JS in the current page
- - `daemon.pageInfo()` — get current page info or dialog
- - `daemon.listTabs()` — returns `{ success, data: [{ targetId, url, title }] }`
- - `daemon.switchTab(targetId)` — switch to a tab by full targetId
- - `daemon.newTab(url?)` — open and switch to a new tab
- - `daemon.current()` — returns `{ targetId, url?, title? }`
- - `daemon.session(targetId)` — returns a session object for raw CDP:
- - `session.call(method, params?)` — CDP command on the page target
- - `session.callOnTarget(method, params, sessionId)` — CDP on a specific session
- - `session.callBrowser(method, params?)` — CDP on the browser target
- - `session.takeDialog()` — get/dismiss pending dialog
-- `require` — Node.js require() (use 'fs', 'path', 'crypto', etc.)
-- `signal` — AbortSignal for cancellation
-- `onUpdate` — progress callback: onUpdate({ content: [{ type: 'text', text: '...' }] })
-- `ctx` — ExtensionContext (ctx.cwd, ctx.signal, etc.)
-- `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout`
-
-**Example — a script that extracts structured data:**
-```javascript
-// /tmp/extract-products.js
-const info = await daemon.pageInfo();
-if (info && "dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message);
-const data = await daemon.evaluateJs(`
- JSON.stringify(Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({
- title: el.querySelector('${params.titleSelector}')?.textContent?.trim(),
- price: el.querySelector('${params.priceSelector}')?.textContent?.trim(),
- })))
-`);
-return { content: [{ type: 'text', text: data }], details: { raw: JSON.parse(data) } };
+ └─ Visual rendering (layout / colors / chart drew correctly)?
+ → browser_screenshot (LAST RESORT — pixels only)
```
-## What NOT To Do
-
-- **Don't launch your own browser** — you're connected to the user's real Chrome
-- **Don't type credentials** — if you hit an auth wall, stop and ask the user
-- **Don't assume page state** — screenshot to confirm
-- **Don't write pixel coordinates into documentation** — describe how to locate targets (selectors, text, aria-label)
-- **Don't use browser_navigate for first navigation** — use browser_new_tab to explicitly preserve the user's active tab (browser_navigate auto-creates a new tab on first call, but explicit is clearer)
-- **Don't ignore dialogs** — if browser_page_info returns a dialog, handle it first
+Pass `@(x,y)` from `browser_snapshot` straight to `browser_click`. No screenshot round-trip.
-## Dialogs
+## Connection
-JS dialogs (alert, confirm, prompt, beforeunload) freeze the page's JS thread.
-If browser_page_info returns a dialog, handle it with browser_handle_dialog
-before any other browser action.
+You're attached to the user's real Chrome — never launch your own. If auth is required, stop and ask the user. If `browser_page_info` returns a dialog, handle it first with `browser_handle_dialog`.
-```
-// Check page_info first — if it shows a dialog:
-browser_handle_dialog({ action: "accept" }) // accept/confirm
-browser_handle_dialog({ action: "dismiss" }) // dismiss/cancel
-browser_handle_dialog({ action: "accept", promptText: "hello" }) // prompt with text
-```
-
-## SPAs (React, Vue, etc.)
+## Temporary scripts
-`browser_wait_for_load()` only checks `document.readyState === "complete"`.
-Single-page apps often paint content after this. After wait_for_load, use
-`browser_execute_js` to check for specific elements:
+When a workflow repeats 3+ times or needs Node.js APIs, write a script to disk and run it with `browser_run_script`. Scripts get a `daemon` binding for direct CDP access — much faster than chaining tool calls.
-```
-browser_execute_js("!!document.querySelector('.loaded-content')")
-```
+**Bindings inside a script:**
-## Keyboard Modifiers
-
-`browser_press_key` supports modifier bitfield:
-- 1 = Alt, 2 = Ctrl, 4 = Meta/Cmd, 8 = Shift
-- Combine: Ctrl+Shift = 2|8 = 10, Cmd+Shift = 4|8 = 12
-
-## Troubleshooting
-
-- **Daemon not starting**: Run `/browser-setup` to re-run the guided setup
-- **"DevToolsActivePort not found"**: Open chrome://inspect/#remote-debugging, tick checkbox, click Allow
-- **Stale session**: Run `/browser-reload-daemon` to restart the daemon
-- **Dialog blocking**: Run browser_page_info to detect it, then browser_handle_dialog
-- **Status check**: Run `/browser-status` to see daemon health and current page
-- **Screenshot maxDim not working**: Install sharp (`npm install sharp`) for auto-resize support
-- **Google navigation fails**: Google's anti-bot detection may reject CDP navigation; use `browser_http_get` to fetch search results instead
-
-## Architecture
-
-```
-pi agent → pi-browser-harness (TypeScript)
- │ CDP WebSocket
- ▼
- Chrome
-
-Temporary scripts:
- write("/tmp/script.js") → browser_run_script("/tmp/script.js")
- │ │
- ▼ ▼
- script on disk (auditable) executed in harness process with daemon access
-```
+- `params` — args passed to `browser_run_script`
+- `daemon`:
+ - `daemon.evaluateJs(expression)` — run JS in the current page
+ - `daemon.pageInfo()` — `{ url, title, ... }` or `{ dialog }`
+ - `daemon.listTabs()` / `daemon.switchTab(targetId)` / `daemon.newTab(url?)` / `daemon.current()`
+ - `daemon.session(targetId)` for raw CDP: `session.call`, `session.callOnTarget`, `session.callBrowser`, `session.takeDialog`
+- `require`, `fetch`, `JSON`, `Buffer`, `console`, `setTimeout`, `clearTimeout`
+- `signal` — AbortSignal
+- `onUpdate({ content: [{ type: 'text', text }] })` — progress callback
+- `ctx` — `ExtensionContext`
+
+**Don't:**
+- Use scripts for one-off actions — call `browser_*` tools directly.
+- Call `browser_*` tools from inside a script — sequence them as separate tool calls outside.