# pi-browser-harness > **Full browser control for pi agents.** Navigate, click, type, screenshot, and extract data from a real Chrome browser — all through CDP, all from your coding agent.

Version License Pi Package Node

**pi-browser-harness** gives pi agents full control of a real Chrome browser. No headless mode, no emulation — your agent sees what you see, clicks where you'd click, and reads pages exactly as they render. Browser tools coexist seamlessly with all standard pi tools (`read`, `bash`, `edit`, `write`). --- ## Why pi-browser-harness? | | | |---|---| | 🖱️ **Real browser, real pixels** | Connects to your running Chrome via CDP. Compositor-level clicks work through iframes, shadow DOM, and cross-origin content. | | 🧩 **Seamless pi integration** | `browser_*` tools appear alongside `read`, `bash`, `edit` — no context switching, no separate daemon management. | | 🔧 **Self-extending harness** | The agent writes its own tools mid-task. Need a paginated scraper? Bulk CSV exporter? Register it at runtime, callable next turn. | | ⚡ **Fast data extraction** | `browser_http_get` bypasses the browser entirely for APIs — 10-50x faster than DOM scraping. `browser_execute_js` for rendered data. | | 🛡️ **Dialog-aware** | Auto-detects JS dialogs (alert, confirm, prompt, beforeunload) before they block your session. | | 📸 **Visual feedback** | Screenshots before and after every action. The agent can see exactly what it's doing. | --- ## Quick Start ```bash # 1. Install the package pi install npm:pi-browser-harness # 2. Start Chrome with remote debugging # macOS: /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 & # Linux: google-chrome --remote-debugging-port=9222 & # 3. Run the guided setup inside pi /browser-setup ``` **That's it.** Your pi agent now has 20 browser control tools. Try: ``` > navigate to https://github.com and take a screenshot ``` --- ## Features ### 🧭 Navigation & Tabs | Tool | What it does | |------|-------------| | `browser_navigate` | Navigate to a URL (safe: creates new tab on first call) | | `browser_new_tab` | Open a new tab, optionally navigate | | `browser_open_urls` | Open multiple URLs in parallel tabs (great after web search) | | `browser_go_back` / `browser_go_forward` | History navigation | | `browser_reload` | Reload the current page | | `browser_page_info` | Get URL, title, viewport, scroll position, or detect JS dialogs | | `browser_list_tabs` / `browser_current_tab` | Tab discovery and identification | | `browser_switch_tab` | Switch to a specific tab by targetId | ### 🖱️ Interaction | Tool | What it does | |------|-------------| | `browser_click` | Click at viewport CSS-pixel coordinates (compositor-level) | | `browser_type` | Type text into the focused element | | `browser_press_key` | Press any key with modifier support (Ctrl, Alt, Meta, Shift) | | `browser_scroll` | Scroll the page by delta pixels | | `browser_handle_dialog` | Accept or dismiss JS dialogs | ### 📸 Visual & Data | Tool | What it does | |------|-------------| | `browser_screenshot` | Capture a PNG screenshot of the current page | | `browser_execute_js` | Execute JavaScript and return the result | | `browser_http_get` | Direct HTTP GET (bypasses browser — 10-50x faster for APIs) | | `browser_wait` | Wait N seconds | | `browser_wait_for_load` | Wait for `document.readyState === 'complete'` | ### 🔧 Self-Extending Harness | Tool | What it does | |------|-------------| | `list_dynamic_tools` | See what's already been registered | | `register_tool` | Inject a new tool (available immediately, next turn) | | `remove_tool` | Retire a tool that's no longer needed | > **Philosophy:** "The agent writes what's missing, mid-task." When the built-in tools aren't enough, the agent extends the harness itself — no reload, no restart. --- ## Installation ### Prerequisites | Requirement | Version | Notes | |------------|---------|-------| | **pi** | Latest | The pi coding agent CLI | | **Node.js** | ≥ 18 | Runtime for the extension | | **Chrome / Chromium / Edge** | Any recent | Must support CDP (Chrome DevTools Protocol) | | **browser-harness daemon** | Latest | Installed automatically by `/browser-setup` | | **uv** (optional) | Latest | Python package manager; used for daemon installation | | **git** (optional) | Any | Fallback installation method | ### Installing the package ```bash # From npm (recommended) pi install npm:pi-browser-harness # From git pi install git:github.com/amankumarsingh77/pi-browser-harness # Locally (for development) pi install ./pi-browser-harness ``` ### Installing the browser-harness daemon The daemon (`daemon.py`) is what connects pi-browser-harness to Chrome. Run the guided setup: ``` /browser-setup ``` This will: 1. Check for an existing browser-harness installation 2. Auto-install via `uv tool install browser-harness` or `git clone` 3. Verify Chrome is running with remote debugging 4. Start the daemon and test the connection **Manual installation:** ```bash # Option A: uv (recommended) uv tool install browser-harness # Option B: git clone git clone https://github.com/browser-use/browser-harness ~/Developer/browser-harness cd ~/Developer/browser-harness && uv sync ``` ### Enabling Chrome Remote Debugging Start Chrome with the `--remote-debugging-port` flag: ```bash # macOS /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ --remote-debugging-port=9222 # Linux google-chrome --remote-debugging-port=9222 # Windows "C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 ``` > **Edge / Chromium**: Works identically — just replace the binary path. --- ## Usage ### Core Workflow The fundamental pattern: **screenshot → interact → screenshot → verify.** ``` browser_screenshot() → find the target → browser_click(x, y) → browser_screenshot() ``` Every interaction is verified visually. Never assume an action worked. ### Navigation ```bash # First navigation: creates a new tab (preserves user's active tab) browser_navigate("https://github.com/browser-use/browser-harness") browser_wait_for_load() browser_screenshot() ``` ### Form Filling ```bash browser_screenshot() # See the form browser_click(400, 200) # Click the search input browser_type("pi browser harness") # Type the query browser_press_key("Enter") # Submit browser_wait_for_load() browser_screenshot() # Verify results ``` ### Data Extraction ```bash # Fast: Direct HTTP for APIs browser_http_get("https://api.github.com/repos/browser-use/browser-harness") # DOM scraping: Execute JS in the rendered page browser_execute_js("document.querySelector('.price').innerText") # Structured: Extract arrays of data browser_execute_js(` JSON.stringify( Array.from(document.querySelectorAll('.result')).map(el => ({ title: el.querySelector('h3')?.textContent, link: el.querySelector('a')?.href, })) ) `) ``` ### Research Workflow (Search + Tabs) ```bash # Step 1: Search browser_navigate("https://google.com/search?q=chrome+devtools+protocol+reference") # Step 2: Open result links in parallel tabs browser_open_urls({ urls: [ "https://chromedevtools.github.io/devtools-protocol/", "https://developer.chrome.com/docs/devtools", ] }) # Step 3: List all tabs to get target IDs browser_list_tabs() # Step 4: Switch to the tab you want browser_switch_tab({ targetId: "ABC123..." }) # Step 5: Inspect browser_wait_for_load() browser_screenshot() browser_execute_js("document.querySelector('.main-content').innerText") ``` ### SPAs (React, Vue, etc.) `browser_wait_for_load()` only checks `document.readyState === 'complete'`. SPAs often render content after this. Check for specific elements: ```bash browser_navigate("https://my-spa.example.com") browser_wait_for_load() # Poll for the element to appear browser_execute_js("!!document.querySelector('.loaded-content')") ``` ### Keyboard Modifiers `browser_press_key` supports modifier bitfield combinations: | Combo | Bitfield | Example | |-------|----------|---------| | Ctrl+C | 2 | `browser_press_key("c", { modifiers: 2 })` | | Cmd+V | 4 | `browser_press_key("v", { modifiers: 4 })` | | Ctrl+Shift+T | 10 (2\|8) | `browser_press_key("T", { modifiers: 10 })` | | Cmd+Shift+N | 12 (4\|8) | `browser_press_key("N", { modifiers: 12 })` | --- ## Self-Extending Harness When the built-in tools aren't enough, the agent writes its own: ``` ● agent: needs to scrape 47 paginated search result pages │ ● calls list_dynamic_tools() → empty │ ● calls register_tool({ │ name: "scrape_paginated", │ implementation: "..." │ }) │ ✓ next turn: scrape_paginated() is available and callable ``` ### Tool Lifecycle 1. **`list_dynamic_tools`** — Check what's already registered 2. **`register_tool`** — Inject a new tool with custom JavaScript 3. **`remove_tool`** — Retire a tool you no longer need ### Dynamic Tool Bindings Tools receive these bindings at runtime: | Binding | Description | |---------|-------------| | `params` | Tool arguments (as passed by the caller) | | `daemon` | Browser daemon — `daemon.cdp()`, `daemon.evaluateJS()`, `daemon.getPageInfo()` | | `require` | Node.js `require()` for builtins and installed packages | | `signal` | `AbortSignal` for cancellation | | `onUpdate` | Progress callback: `onUpdate({ content: [{ type: 'text', text: '...' }] })` | | `ctx` | `ExtensionContext` with `cwd`, `sessionManager`, `ui`, `signal` | | `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout` | Standard globals | ### Example: Structured Data Extractor ```javascript // register_tool implementation: const info = await daemon.getPageInfo(); if ("dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message); const data = await daemon.evaluateJS(` JSON.stringify( Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({ title: el.querySelector('${params.titleSelector}')?.textContent?.trim(), price: el.querySelector('${params.priceSelector}')?.textContent?.trim(), link: el.querySelector('a')?.href, })) ) `); const parsed = JSON.parse(data); return { content: [{ type: 'text', text: `Extracted ${parsed.length} items:\n\n${JSON.stringify(parsed, null, 2)}` }], details: { items: parsed, count: parsed.length } }; ``` --- ## Architecture ``` ┌─────────────────────────────────────────────────┐ │ pi agent │ │ │ │ │ ├─ browser_navigate() │ │ ├─ browser_screenshot() │ │ ├─ browser_click(x, y) │ │ ├─ browser_execute_js(...) │ │ ├─ list_dynamic_tools() │ │ ├─ register_tool(...) │ │ └─ remove_tool(...) │ │ │ │ pi-browser-harness (TypeScript extension) │ └────────────────────┬────────────────────────────┘ │ Unix socket │ /tmp/bu-.sock ▼ ┌────────────────┐ │ daemon.py │ │ (Python) │ └───────┬────────┘ │ CDP WebSocket │ ws://localhost:9222 ▼ ┌────────────────┐ │ Chrome / Edge │ │ (user's real │ │ browser) │ └────────────────┘ ``` ### Data Flow 1. **pi agent** calls a `browser_*` tool (e.g., `browser_screenshot()`) 2. **pi-browser-harness extension** translates the call into a CDP command 3. **JSON-line request** sent over Unix socket to `daemon.py` 4. **daemon.py** relays the command over CDP WebSocket to Chrome 5. **Chrome** executes the command and returns the result 6. Result flows back through the chain to the agent ### Session Persistence - Tab history, daemon namespace, and preferences persist across session reloads and branch navigation - State is stored via `pi.appendEntry()` in the session's JSONL file - On `session_start`, state is restored by scanning the branch for the last `browser-harness-state` entry ### Key Design Decisions - **One socket connection per request** — stateless, matches the daemon's internal `_send` pattern - **Session auto-recovery** — stale CDP sessions are automatically re-attached - **Compositor-level clicks** — `Input.dispatchMouseEvent` works through iframes, shadow DOM, and cross-origin content without selectors - **Daemon auto-restart** — if the daemon socket dies mid-request, it's restarted and the request is retried once --- ## Configuration ### CLI Flags | Flag | Type | Default | Description | |------|------|---------|-------------| | `--browser-namespace` | `string` | auto-generated | Daemon namespace (`BU_NAME`). Use to run multiple instances. | | `--browser-debug-clicks` | `boolean` | `false` | Enable debug click overlay (saves annotated screenshots to `/tmp`). | ### Commands | Command | Description | |---------|-------------| | `/browser-setup` | Guided setup wizard — installs browser-harness, connects to Chrome | | `/browser-status` | Show daemon health, current page URL/title/viewport | | `/browser-reload-daemon` | Restart the browser daemon | --- ## Troubleshooting | Symptom | Likely Cause | Fix | |---------|-------------|-----| | Daemon not starting | Chrome remote debugging not enabled | Start Chrome with `--remote-debugging-port=9222` | | "DevToolsActivePort not found" | Chrome didn't accept the debugging connection | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow | | Daemon starts but no tools work | Stale or dead CDP session | Run `/browser-reload-daemon` to restart | | Dialog blocking page | JS `alert`/`confirm`/`prompt` is open | Run `browser_page_info()` to detect it, then `browser_handle_dialog()` | | Page not fully loaded | SPA with async rendering | After `browser_wait_for_load()`, use `browser_execute_js()` to check for specific elements | | `uv` not found | Python package manager not installed | Install via `curl -LsSf https://astral.sh/uv/install.sh \| sh` or use the git fallback | | Socket permission denied | Stale socket file from previous run | `rm /tmp/bu-*.sock /tmp/bu-*.pid` and restart | | Windows: `ps` not found | `setup.ts` uses Unix commands | Windows detection is best-effort; see manual installation steps above | --- ## Security > **Warning:** This extension gives your pi agent full control over your real Chrome browser. It can see your open tabs, read page content, submit forms, and interact with authenticated sessions. - **No sandboxing** — the agent has access to whatever Chrome has access to - **Credentials** — if the agent hits an auth wall, it should stop and ask you. Never store credentials in agent prompts. - **Dynamic tools** — `register_tool` evaluates user-supplied JavaScript in the pi process with full `require` access. Review any dynamic tool implementations you didn't write yourself. - **Data handling** — Screenshots are saved to `/tmp`. HTTP responses and JS evaluation results may contain sensitive page data. - **Session persistence** — Tab history and daemon state are stored in pi's session file (`.jsonl`). These contain page URLs and titles. --- ## Contributing We welcome contributions! Here's how to get started: ```bash # Clone the repository git clone https://github.com/amankumarsingh77/pi-browser-harness cd pi-browser-harness # Install dependencies npm install # Type-check npm run typecheck # Test locally pi install ./ # Install from local path ``` ### Development Guidelines - **TypeScript strict mode** — all code must pass `tsc --noEmit` - **Error handling** — tools must catch errors and return `isError: true` with a descriptive message - **Tool names** — use `snake_case` for consistency with pi conventions - **Parameters** — use `typebox` schemas for all tool parameters - **Prompt guidelines** — every tool must include `promptSnippet` and `promptGuidelines` for LLM context ### Project Structure ``` pi-browser-harness/ ├── src/ │ ├── index.ts # Extension entry point, lifecycle, flags, commands │ ├── daemon.ts # BrowserDaemon: spawns daemon.py, CDP transport │ ├── tools.ts # All browser_* tool registrations │ ├── dynamic-tools.ts # Self-extending harness (register_tool, etc.) │ ├── protocol.ts # Unix socket protocol types and low-level transport │ ├── prompt.ts # System prompt injection │ ├── renderers.ts # Custom TUI renderers │ ├── setup.ts # /browser-setup wizard │ └── state.ts # Session persistence ├── skills/ │ └── pi-browser-harness/ │ └── SKILL.md # User-facing skill documentation ├── package.json ├── tsconfig.json ├── LICENSE ├── CHANGELOG.md └── README.md ``` --- ## Roadmap - [ ] Windows-native setup wizard (remove `ps`/`grep` dependency) - [ ] iframe-target evaluation support in `browser_execute_js` - [ ] Remote browser support (connect to Chrome running on another machine) - [ ] Screenshot annotations (click markers, element highlights) - [ ] Performance benchmarks and CI test suite - [ ] Configurable screenshot output directory - [ ] Recording/playback of interaction sequences - [ ] Better audio/video support for media-heavy pages --- ## License MIT — see [LICENSE](LICENSE) for details. ---

Built with ❤️ for the pi ecosystem. Makes coding agents see the web.