From 0fc1ece6fa4a2f24e87eb4a66eb828a9928e30f7 Mon Sep 17 00:00:00 2001 From: amankumarsingh77 Date: Fri, 24 Jul 2026 22:47:33 +0530 Subject: docs: rewrite README for accuracy and readability The README taught the coordinate-first interaction model that the harness replaced with stable [eN] refs, and documented 22 of the 38 registered tools. Restructured so each section answers one question, and rewrote the content against the actual tool registry. - Teach the ref-first model: snapshot yields [eN] handles that re-resolve at action time, with coordinates as the fallback rather than the default. - Document the 16 missing tools: the forms path (fill/select_option/focus), console, wait_for, dispatch_key, drag_and_drop, upload_file, download, print_to_pdf, viewport_resize, close_tab, and setup. - Rebuild the comparison table as prose cells grouped by dimension. Drops the unverifiable "unknown" rows and the duplicated click/dispatch claims, and adds a provenance line so competitor claims stay correctable. - Remove stale guidance: /browser-reload-daemon as the fix for a Chrome restart (auto-reconnect landed in 0.6.0), and the unbenchmarked "10-50x faster" claim on http_get. - Add sections for concurrency and tab isolation, flags and environment variables (BU_CDP_WS, BU_CDP_PORTS, BH_SCRIPT_DIR), and a troubleshooting table keyed to the errors the harness actually emits. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 420 +++++++++++++++++++++++++++++++++----------------------------- 1 file changed, 221 insertions(+), 199 deletions(-) diff --git a/README.md b/README.md index 86e191e..e0fce44 100644 --- a/README.md +++ b/README.md @@ -2,347 +2,369 @@ ![pi-browser-harness](https://raw.githubusercontent.com/amankumarsingh77/pi-browser-harness/main/assets/hero.png) -Full browser control for pi agents in **your real Chrome** — your sessions, your cookies, your tabs. Drives navigation, structured page reads, network capture, clicks, typing, screenshots, and arbitrary scripts via CDP. +**Browser control for [pi](https://github.com/badlogic/pi-mono) agents, in the Chrome you're already using.** + +Your profile, your logins, your cookies. The agent reads pages as an accessibility tree with stable +element handles, fills forms that React won't revert, captures network traffic and console errors, +searches the web, and drops to raw CDP when the tools run out. 38 tools over a single CDP connection — +no MCP hop, no second browser, no extra API keys. --- -## Why pi-browser-harness? - -| Capability | pi-browser-harness | Playwright MCP | Stagehand | Puppeteer MCP | -|---|:---:|:---:|:---:|:---:| -| **Drives your real Chrome** (logged-in sessions preserved) | ✅ | ❌ launches its own browser | ❌ | ❌ | -| **Coordinate clicks** that work through iframes, shadow DOM, cross-origin | ✅ | ❌ selector-based | ❌ selector-based | ❌ selector-based | -| **Inline TUI screenshot rendering** (Kitty/iTerm2/Ghostty/WezTerm) | ✅ | ❌ | ❌ | ❌ | -| **Accessibility-tree snapshot with click coords `@(x,y)` per element** | ✅ | ✅ tree only, no coords | partial | ❌ | -| **Network request capture** with filters + body capture | ✅ | ✅ post-hoc list | ❌ | ❌ | -| **Parallel tool execution** with automatic mutation serialization | ✅ | ❌ | ❌ | ❌ | -| **Temporary scripts** with daemon + full Node.js | ✅ | ❌ | ❌ | ❌ | -| **Direct HTTP GET** outside the browser (10–50× faster for APIs) | ✅ | ❌ | ❌ | ❌ | -| **Tab ownership isolation** — never touches the user's other tabs | ✅ | N/A | N/A | N/A | -| **Pi-native** — no MCP/JSON-RPC overhead, no extra LLM API keys | ✅ | ❌ MCP roundtrip | ❌ external LLM | ❌ MCP roundtrip | -| **TypeScript strict mode**, zero `any`, all CDP casts documented | ✅ | unknown | unknown | unknown | -| Ctrl+O expand/collapse on tool output | ✅ | ❌ | ❌ | ❌ | -| Compositor-level dispatch (works on every site, no flakey waits) | ✅ | ❌ | ❌ | ❌ | - -If you live in pi and you want an agent driving the same Chrome you're already signed into, this is the only one that fits. +## How it compares + +| | **pi-browser-harness** | Playwright MCP | Puppeteer MCP | Stagehand | +|---|---|---|---|---| +| **Browser** | Your running Chrome — real profile, live sessions | Launches a clean instance | Launches a clean instance | Launches a clean instance | +| **Element targeting** | AX-tree refs (`[eN]`) that re-resolve at action time; coordinate and selector fallbacks | Selectors / AX locators | CSS selectors | LLM-inferred selectors | +| **Input dispatch** | Compositor-level CDP input — works through iframes, shadow DOM, cross-origin frames | DOM-level | DOM-level | DOM-level | +| **Framework-safe writes** | `browser_fill` writes via the native setter and fires bubbling events, so controlled inputs keep the value | Native fill | Manual | Native fill | +| **Transport** | Native pi extension, in-process | MCP / JSON-RPC round trip | MCP / JSON-RPC round trip | In-process + its own LLM calls | +| **Extra credentials** | None | None | None | Its own model API key | +| **Concurrency** | Reads run in parallel; mutations auto-serialize through a shared mutex | Sequential | Sequential | Sequential | +| **Isolation** | Dedicated Chrome window per session; refuses tabs it didn't open | N/A (own browser) | N/A | N/A | +| **Terminal output** | Inline screenshots and `Ctrl+O`-expandable tool results | Plain text | Plain text | Plain text | +| **Escape hatch** | `browser_run_script` — raw CDP + full Node in one call | — | — | — | + +Reflects each project's default configuration. Corrections welcome via PR. + +**The short version:** if you live in pi and want an agent driving the same Chrome you're already +signed into, this is the one that fits. --- -## Quick Start +## Install ```bash -# 1. Install pi install npm:pi-browser-harness - -# 2. Enable Chrome remote debugging -# Open chrome://inspect/#remote-debugging in Chrome, -# tick "Discover network targets", click Allow. -# Or launch Chrome with --remote-debugging-port=9222 - -# 3. Connect -/browser-setup ``` -### Requirements +Then enable remote debugging in your browser — either open `chrome://inspect/#remote-debugging` +(or `brave://inspect`, `edge://inspect`), tick **Discover network targets**, and click **Allow** — +or relaunch the browser with `--remote-debugging-port=9222`. + +Finally, run `/browser-setup` in pi to connect. It's a one-time step — the connection survives across +pi sessions and reconnects on its own when Chrome restarts. -- pi (latest) -- Node.js ≥ 22 -- Chrome / Chromium / Edge +**Requires:** pi (latest) · Node.js ≥ 22 · Chrome, Chromium, Brave, or Edge --- -## Tool hierarchy — read this first +## The mental model + +Interaction is **ref-first**. `browser_snapshot` returns the accessibility tree, and every +interactive element carries a stable ref (`[eN]`) plus click coordinates `@(x,y)`: ``` -What do you need to know? - - ├─ Page structure / what's clickable / labels? - │ → browser_snapshot (DEFAULT — AX tree with @(x,y) per interactive element) - │ - ├─ A specific element's value / attribute / coords? - │ → browser_execute_js (e.g. el.innerText, el.getBoundingClientRect()) - │ - ├─ Network behavior on the current page? - │ → browser_network_requests - │ - └─ Visual rendering (layout / colors / chart drew correctly)? - → browser_screenshot (LAST RESORT — pixels only) +[e7] textbox "Email" @(412,288) +[e9] combobox "Country" @(412,344) +[e12] button "Create account" @(456,412) ``` -Pass `@(x,y)` from `browser_snapshot` straight to `browser_click`. **No screenshot round-trip needed** to find click targets — the snapshot already has them. +Pass the ref — not the coordinates — to `browser_click`, `browser_fill`, `browser_select_option`, +`browser_focus`, `browser_upload_file`, or `browser_dispatch_key`. Refs are keyed to element +identity, so they re-resolve position at action time and survive the re-renders that SPA forms +trigger on every keystroke. Coordinates go stale the moment the layout reflows. + +Three things follow from that: + +- **Never screenshot to find a click target.** The snapshot already has refs and coordinates. +- **Mutating calls append a "Page changes" diff.** Read it to confirm the step landed — a panel + closed, a new `[eN]` appeared — instead of taking a screenshot to check. +- **`"ref is stale"` means the page changed.** Re-snapshot for fresh refs; don't retry blindly. + +### Which tool answers your question + +| You need to know… | Reach for | +|---|---| +| What's on the page, what's clickable | `browser_snapshot` — the default | +| One element's text, value, or geometry | `browser_execute_js` — one round trip | +| An article's readable content | `browser_read_page` | +| Why nothing happened after an action | `browser_console` | +| What the page requested over the network | `browser_network_requests` | +| Whether something *rendered* correctly | `browser_screenshot` — last resort, pixels only | --- ## Tools -### Page inspection (use these by default) +### Reading the page + +| Tool | Purpose | +|---|---| +| `browser_snapshot` | **Default.** Accessibility tree with `[eN]` refs and `@(x,y)` per interactive element. Optional `includeScreenshot`. | +| `browser_execute_js` | Surgical DOM reads — text, attributes, `getBoundingClientRect()`. Cheapest precise read. | +| `browser_page_info` | URL, title, viewport, scroll position — or the pending JS dialog, if one is blocking. | +| `browser_console` | JS errors and console output. Pass `sinceSeq` from the previous `nextCursor` to see only what an action caused. | +| `browser_screenshot` | PNG/JPEG capture. For verifying visual rendering only. | + +### Web research | Tool | Purpose | -|------|---------| -| `browser_snapshot` | **Default for page inspection.** Returns the CDP accessibility tree with click coords `@(x,y)` for every interactive element. Optional `includeScreenshot:true`. | -| `browser_execute_js` | Surgical DOM reads — element text, attributes, `getBoundingClientRect()`. Cheapest, most precise. | -| `browser_network_requests` | List recent network requests on the current tab. Filter by url/method/status/type/recency; optional response-body capture. | -| `browser_page_info` | URL, title, viewport, scroll position, or pending dialog. | -| `browser_http_get` | Direct HTTP GET outside the browser — 10-50× faster for APIs. | +|---|---| +| `browser_web_search` | Query → ranked `{title, url, snippet, rank}`, by scraping a real Google SERP in your Chrome. No API key. Links only. | +| `browser_read_page` | Reader mode. A `url` (opened and closed in its own tab) or an owned `targetId` → clean article text, boilerplate stripped. | -### Research & reading +Both run in isolated tabs and never disturb your current page. + +### Interacting | Tool | Purpose | -|------|---------| -| `browser_web_search` | Search the web — scrapes a Google results page in the real Chrome (no API key), returns ranked `{title, url, snippet, rank}`. **Links only** — follow up with `browser_read_page`. Runs in its own isolated tab. | -| `browser_read_page` | Reader mode — a `url` (opened + closed in an isolated tab) or an owned `targetId` → clean main-article text with nav/ads/boilerplate stripped. Use over snapshot/execute_js when you want an article's content. | +|---|---| +| `browser_click` | Click by `ref` (preferred) or viewport `x`/`y`. Compositor-level, so iframes and shadow DOM work. | +| `browser_fill` | Set an input, textarea, or contenteditable. Fires bubbling `input`/`change`, returns the value written. | +| `browser_select_option` | Choose in a native `