diff options
| author | amankumarsingh77 <amankumarsingh7702@gmal.com> | 2026-04-29 01:38:17 +0530 |
|---|---|---|
| committer | amankumarsingh77 <amankumarsingh7702@gmal.com> | 2026-04-29 01:38:17 +0530 |
| commit | 63b16957587fe20566129ffc5191f66f24a70872 (patch) | |
| tree | 8a40fa38e1ae5b4309b1e38d7a3779321f15db4e /README.md | |
| parent | 5317b32976edce3ab8e2f11d7fb4ff3f16dcac6f (diff) | |
Replace socket daemon with direct CDP WebSocket connection
Drops the Unix socket subprocess in favor of talking CDP directly over
WebSocket from the extension. Removes the protocol's send/health helpers,
simplifies setup to just verifying Chrome's remote debugging port, and
tightens types in dynamic-tools. Bumps Node engine to >=22 and adds the
ws dependency.
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 507 |
1 files changed, 102 insertions, 405 deletions
@@ -1,513 +1,210 @@ # pi-browser-harness -> **Full browser control for pi agents.** Navigate, click, type, screenshot, and extract data from a real Chrome browser — all through CDP, all from your coding agent. - -<p align="center"> - <img alt="Version" src="https://img.shields.io/npm/v/@pi-browser-harness?style=flat-square&color=2563eb"> - <img alt="License" src="https://img.shields.io/badge/license-MIT-blue?style=flat-square"> - <img alt="Pi Package" src="https://img.shields.io/badge/pi-package-2563eb?style=flat-square"> - <img alt="Node" src="https://img.shields.io/badge/node-%3E%3D18-339933?style=flat-square&logo=node.js&logoColor=white"> -</p> - -**pi-browser-harness** gives pi agents full control of a real Chrome browser. No headless mode, no emulation — your agent sees what you see, clicks where you'd click, and reads pages exactly as they render. Browser tools coexist seamlessly with all standard pi tools (`read`, `bash`, `edit`, `write`). - ---- - -## Why pi-browser-harness? - -| | | -|---|---| -| 🖱️ **Real browser, real pixels** | Connects to your running Chrome via CDP. Compositor-level clicks work through iframes, shadow DOM, and cross-origin content. | -| 🧩 **Seamless pi integration** | `browser_*` tools appear alongside `read`, `bash`, `edit` — no context switching, no separate daemon management. | -| 🔧 **Self-extending harness** | The agent writes its own tools mid-task. Need a paginated scraper? Bulk CSV exporter? Register it at runtime, callable next turn. | -| ⚡ **Fast data extraction** | `browser_http_get` bypasses the browser entirely for APIs — 10-50x faster than DOM scraping. `browser_execute_js` for rendered data. | -| 🛡️ **Dialog-aware** | Auto-detects JS dialogs (alert, confirm, prompt, beforeunload) before they block your session. | -| 📸 **Visual feedback** | Screenshots before and after every action. The agent can see exactly what it's doing. | +Full browser control for pi agents. Navigate, click, type, screenshot, and extract data from your real Chrome — driven by your coding agent. --- ## Quick Start ```bash -# 1. Install the package -pi install npm:pi-browser-harness - -# 2. Start Chrome with remote debugging -# macOS: -/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 & +# 1. Install (from this repo, until published to npm) +pi install git:github.com/amankumarsingh77/pi-browser-harness -# Linux: -google-chrome --remote-debugging-port=9222 & +# 2. Enable Chrome remote debugging +# Open chrome://inspect/#remote-debugging in Chrome, +# tick the "Discover network targets" checkbox, click Allow. +# No Chrome relaunch needed. -# 3. Run the guided setup inside pi +# 3. Connect pi to Chrome /browser-setup ``` -**That's it.** Your pi agent now has 20 browser control tools. Try: +That's it. Try: ``` > navigate to https://github.com and take a screenshot ``` +### Requirements + +- pi (latest) +- Node.js ≥ 22 +- Chrome / Chromium / Edge (any recent version) + +> If `chrome://inspect/#remote-debugging` isn't available, you can launch Chrome with `--remote-debugging-port=9222` instead. The harness will find it either way. + --- -## Features +## What you can do -### 🧭 Navigation & Tabs +### Navigation & tabs | Tool | What it does | -|------|-------------| -| `browser_navigate` | Navigate to a URL (safe: creates new tab on first call) | +|------|--------------| +| `browser_navigate` | Go to a URL (reuses the attached tab) | | `browser_new_tab` | Open a new tab, optionally navigate | -| `browser_open_urls` | Open multiple URLs in parallel tabs (great after web search) | +| `browser_open_urls` | Open many URLs in parallel tabs | | `browser_go_back` / `browser_go_forward` | History navigation | | `browser_reload` | Reload the current page | -| `browser_page_info` | Get URL, title, viewport, scroll position, or detect JS dialogs | -| `browser_list_tabs` / `browser_current_tab` | Tab discovery and identification | -| `browser_switch_tab` | Switch to a specific tab by targetId | +| `browser_list_tabs` / `browser_current_tab` | Discover tabs | +| `browser_switch_tab` | Switch the active tab | +| `browser_page_info` | URL, title, viewport, scroll, dialog state | -### 🖱️ Interaction +### Interaction | Tool | What it does | -|------|-------------| -| `browser_click` | Click at viewport CSS-pixel coordinates (compositor-level) | -| `browser_type` | Type text into the focused element | -| `browser_press_key` | Press any key with modifier support (Ctrl, Alt, Meta, Shift) | -| `browser_scroll` | Scroll the page by delta pixels | -| `browser_handle_dialog` | Accept or dismiss JS dialogs | +|------|--------------| +| `browser_click` | Click at viewport coordinates (works through iframes / shadow DOM) | +| `browser_type` | Type into the focused element | +| `browser_press_key` | Send a key with optional modifiers | +| `browser_scroll` | Scroll by delta pixels | +| `browser_handle_dialog` | Accept or dismiss `alert` / `confirm` / `prompt` | -### 📸 Visual & Data +### Visual & data | Tool | What it does | -|------|-------------| -| `browser_screenshot` | Capture a PNG screenshot of the current page | -| `browser_execute_js` | Execute JavaScript and return the result | -| `browser_http_get` | Direct HTTP GET (bypasses browser — 10-50x faster for APIs) | -| `browser_wait` | Wait N seconds | -| `browser_wait_for_load` | Wait for `document.readyState === 'complete'` | +|------|--------------| +| `browser_screenshot` | Capture the current page as PNG | +| `browser_execute_js` | Run JS in the page and return the result | +| `browser_http_get` | Direct HTTP GET — bypasses the browser, much faster for APIs | +| `browser_wait` / `browser_wait_for_load` | Sleep, or wait for `readyState === 'complete'` | -### 🔧 Self-Extending Harness +### Extending the harness at runtime | Tool | What it does | -|------|-------------| -| `list_dynamic_tools` | See what's already been registered | -| `register_tool` | Inject a new tool (available immediately, next turn) | -| `remove_tool` | Retire a tool that's no longer needed | - -> **Philosophy:** "The agent writes what's missing, mid-task." When the built-in tools aren't enough, the agent extends the harness itself — no reload, no restart. - ---- - -## Installation - -### Prerequisites - -| Requirement | Version | Notes | -|------------|---------|-------| -| **pi** | Latest | The pi coding agent CLI | -| **Node.js** | ≥ 18 | Runtime for the extension | -| **Chrome / Chromium / Edge** | Any recent | Must support CDP (Chrome DevTools Protocol) | -| **browser-harness daemon** | Latest | Installed automatically by `/browser-setup` | -| **uv** (optional) | Latest | Python package manager; used for daemon installation | -| **git** (optional) | Any | Fallback installation method | - -### Installing the package - -```bash -# From npm (recommended) -pi install npm:pi-browser-harness - -# From git -pi install git:github.com/amankumarsingh77/pi-browser-harness - -# Locally (for development) -pi install ./pi-browser-harness -``` - -### Installing the browser-harness daemon - -The daemon (`daemon.py`) is what connects pi-browser-harness to Chrome. Run the guided setup: - -``` -/browser-setup -``` - -This will: -1. Check for an existing browser-harness installation -2. Auto-install via `uv tool install browser-harness` or `git clone` -3. Verify Chrome is running with remote debugging -4. Start the daemon and test the connection - -**Manual installation:** - -```bash -# Option A: uv (recommended) -uv tool install browser-harness - -# Option B: git clone -git clone https://github.com/browser-use/browser-harness ~/Developer/browser-harness -cd ~/Developer/browser-harness && uv sync -``` - -### Enabling Chrome Remote Debugging - -Start Chrome with the `--remote-debugging-port` flag: - -```bash -# macOS -/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ - --remote-debugging-port=9222 - -# Linux -google-chrome --remote-debugging-port=9222 - -# Windows -"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 -``` - -> **Edge / Chromium**: Works identically — just replace the binary path. +|------|--------------| +| `list_dynamic_tools` | List tools registered this session | +| `register_tool` | Add a new tool from JS — callable next turn | +| `remove_tool` | Retire a registered tool | --- ## Usage -### Core Workflow +The core pattern is **screenshot → act → screenshot → verify.** Don't assume an action worked — look. -The fundamental pattern: **screenshot → interact → screenshot → verify.** +### Navigate and screenshot ``` -browser_screenshot() → find the target → browser_click(x, y) → browser_screenshot() -``` - -Every interaction is verified visually. Never assume an action worked. - -### Navigation - -```bash -# First navigation: creates a new tab (preserves user's active tab) browser_navigate("https://github.com/browser-use/browser-harness") browser_wait_for_load() browser_screenshot() ``` -### Form Filling +### Fill a form -```bash -browser_screenshot() # See the form -browser_click(400, 200) # Click the search input -browser_type("pi browser harness") # Type the query -browser_press_key("Enter") # Submit +``` +browser_screenshot() # see the form +browser_click(400, 200) # click the search input +browser_type("pi browser harness") # type +browser_press_key("Enter") # submit browser_wait_for_load() -browser_screenshot() # Verify results +browser_screenshot() # verify ``` -### Data Extraction +### Extract data -```bash -# Fast: Direct HTTP for APIs +```js +// Fast: hit the API directly, no browser browser_http_get("https://api.github.com/repos/browser-use/browser-harness") -# DOM scraping: Execute JS in the rendered page +// DOM scraping for rendered content browser_execute_js("document.querySelector('.price').innerText") -# Structured: Extract arrays of data +// Structured arrays browser_execute_js(` JSON.stringify( Array.from(document.querySelectorAll('.result')).map(el => ({ title: el.querySelector('h3')?.textContent, - link: el.querySelector('a')?.href, + link: el.querySelector('a')?.href, })) ) `) ``` -### Research Workflow (Search + Tabs) - -```bash -# Step 1: Search -browser_navigate("https://google.com/search?q=chrome+devtools+protocol+reference") +### Open multiple results in parallel -# Step 2: Open result links in parallel tabs +``` +browser_navigate("https://google.com/search?q=chrome+devtools+protocol") browser_open_urls({ urls: [ "https://chromedevtools.github.io/devtools-protocol/", "https://developer.chrome.com/docs/devtools", -] }) - -# Step 3: List all tabs to get target IDs +]}) browser_list_tabs() - -# Step 4: Switch to the tab you want -browser_switch_tab({ targetId: "ABC123..." }) - -# Step 5: Inspect -browser_wait_for_load() -browser_screenshot() -browser_execute_js("document.querySelector('.main-content').innerText") +browser_switch_tab({ targetId: "..." }) ``` -### SPAs (React, Vue, etc.) +### SPAs -`browser_wait_for_load()` only checks `document.readyState === 'complete'`. SPAs often render content after this. Check for specific elements: +`browser_wait_for_load` only checks `readyState`. For React/Vue/etc., poll for the actual element you want: -```bash -browser_navigate("https://my-spa.example.com") -browser_wait_for_load() - -# Poll for the element to appear +```js browser_execute_js("!!document.querySelector('.loaded-content')") ``` -### Keyboard Modifiers - -`browser_press_key` supports modifier bitfield combinations: +### Keyboard modifiers -| Combo | Bitfield | Example | -|-------|----------|---------| -| Ctrl+C | 2 | `browser_press_key("c", { modifiers: 2 })` | -| Cmd+V | 4 | `browser_press_key("v", { modifiers: 4 })` | -| Ctrl+Shift+T | 10 (2\|8) | `browser_press_key("T", { modifiers: 10 })` | -| Cmd+Shift+N | 12 (4\|8) | `browser_press_key("N", { modifiers: 12 })` | - ---- - -## Self-Extending Harness - -When the built-in tools aren't enough, the agent writes its own: +`browser_press_key` takes a modifier bitfield: `Alt=1`, `Ctrl=2`, `Meta/Cmd=4`, `Shift=8`. Combine with `|`. ``` - ● agent: needs to scrape 47 paginated search result pages - │ - ● calls list_dynamic_tools() → empty - │ - ● calls register_tool({ - │ name: "scrape_paginated", - │ implementation: "..." - │ }) - │ - ✓ next turn: scrape_paginated() is available and callable -``` - -### Tool Lifecycle - -1. **`list_dynamic_tools`** — Check what's already registered -2. **`register_tool`** — Inject a new tool with custom JavaScript -3. **`remove_tool`** — Retire a tool you no longer need - -### Dynamic Tool Bindings - -Tools receive these bindings at runtime: - -| Binding | Description | -|---------|-------------| -| `params` | Tool arguments (as passed by the caller) | -| `daemon` | Browser daemon — `daemon.cdp()`, `daemon.evaluateJS()`, `daemon.getPageInfo()` | -| `require` | Node.js `require()` for builtins and installed packages | -| `signal` | `AbortSignal` for cancellation | -| `onUpdate` | Progress callback: `onUpdate({ content: [{ type: 'text', text: '...' }] })` | -| `ctx` | `ExtensionContext` with `cwd`, `sessionManager`, `ui`, `signal` | -| `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout` | Standard globals | - -### Example: Structured Data Extractor - -```javascript -// register_tool implementation: -const info = await daemon.getPageInfo(); -if ("dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message); - -const data = await daemon.evaluateJS(` - JSON.stringify( - Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({ - title: el.querySelector('${params.titleSelector}')?.textContent?.trim(), - price: el.querySelector('${params.priceSelector}')?.textContent?.trim(), - link: el.querySelector('a')?.href, - })) - ) -`); - -const parsed = JSON.parse(data); -return { - content: [{ type: 'text', text: `Extracted ${parsed.length} items:\n\n${JSON.stringify(parsed, null, 2)}` }], - details: { items: parsed, count: parsed.length } -}; +browser_press_key("c", { modifiers: 2 }) // Ctrl+C +browser_press_key("v", { modifiers: 4 }) // Cmd+V +browser_press_key("T", { modifiers: 10 }) // Ctrl+Shift+T ``` ---- +### Writing your own tool mid-task -## Architecture +When the built-ins aren't enough, `register_tool` injects a new one — available on the next turn. +```js +register_tool({ + name: "scrape_results", + implementation: ` + const data = await daemon.evaluateJS(\` + JSON.stringify( + Array.from(document.querySelectorAll('${params.selector}')) + .map(el => el.textContent.trim()) + ) + \`); + return { + content: [{ type: "text", text: data }], + details: { items: JSON.parse(data) }, + }; + `, +}); ``` -┌─────────────────────────────────────────────────┐ -│ pi agent │ -│ │ │ -│ ├─ browser_navigate() │ -│ ├─ browser_screenshot() │ -│ ├─ browser_click(x, y) │ -│ ├─ browser_execute_js(...) │ -│ ├─ list_dynamic_tools() │ -│ ├─ register_tool(...) │ -│ └─ remove_tool(...) │ -│ │ -│ pi-browser-harness (TypeScript extension) │ -└────────────────────┬────────────────────────────┘ - │ Unix socket - │ /tmp/bu-<namespace>.sock - ▼ - ┌────────────────┐ - │ daemon.py │ - │ (Python) │ - └───────┬────────┘ - │ CDP WebSocket - │ ws://localhost:9222 - ▼ - ┌────────────────┐ - │ Chrome / Edge │ - │ (user's real │ - │ browser) │ - └────────────────┘ -``` - -### Data Flow - -1. **pi agent** calls a `browser_*` tool (e.g., `browser_screenshot()`) -2. **pi-browser-harness extension** translates the call into a CDP command -3. **JSON-line request** sent over Unix socket to `daemon.py` -4. **daemon.py** relays the command over CDP WebSocket to Chrome -5. **Chrome** executes the command and returns the result -6. Result flows back through the chain to the agent - -### Session Persistence - -- Tab history, daemon namespace, and preferences persist across session reloads and branch navigation -- State is stored via `pi.appendEntry()` in the session's JSONL file -- On `session_start`, state is restored by scanning the branch for the last `browser-harness-state` entry -### Key Design Decisions - -- **One socket connection per request** — stateless, matches the daemon's internal `_send` pattern -- **Session auto-recovery** — stale CDP sessions are automatically re-attached -- **Compositor-level clicks** — `Input.dispatchMouseEvent` works through iframes, shadow DOM, and cross-origin content without selectors -- **Daemon auto-restart** — if the daemon socket dies mid-request, it's restarted and the request is retried once +Tool implementations receive `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, plus the standard JS globals. --- -## Configuration - -### CLI Flags - -| Flag | Type | Default | Description | -|------|------|---------|-------------| -| `--browser-namespace` | `string` | auto-generated | Daemon namespace (`BU_NAME`). Use to run multiple instances. | -| `--browser-debug-clicks` | `boolean` | `false` | Enable debug click overlay (saves annotated screenshots to `/tmp`). | - -### Commands +## Commands | Command | Description | |---------|-------------| -| `/browser-setup` | Guided setup wizard — installs browser-harness, connects to Chrome | -| `/browser-status` | Show daemon health, current page URL/title/viewport | -| `/browser-reload-daemon` | Restart the browser daemon | +| `/browser-setup` | Connect pi to Chrome (run once) | +| `/browser-status` | Show daemon health and current page | +| `/browser-reload-daemon` | Restart the connection | --- ## Troubleshooting -| Symptom | Likely Cause | Fix | -|---------|-------------|-----| -| Daemon not starting | Chrome remote debugging not enabled | Start Chrome with `--remote-debugging-port=9222` | -| "DevToolsActivePort not found" | Chrome didn't accept the debugging connection | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow | -| Daemon starts but no tools work | Stale or dead CDP session | Run `/browser-reload-daemon` to restart | -| Dialog blocking page | JS `alert`/`confirm`/`prompt` is open | Run `browser_page_info()` to detect it, then `browser_handle_dialog()` | -| Page not fully loaded | SPA with async rendering | After `browser_wait_for_load()`, use `browser_execute_js()` to check for specific elements | -| `uv` not found | Python package manager not installed | Install via `curl -LsSf https://astral.sh/uv/install.sh \| sh` or use the git fallback | -| Socket permission denied | Stale socket file from previous run | `rm /tmp/bu-*.sock /tmp/bu-*.pid` and restart | -| Windows: `ps` not found | `setup.ts` uses Unix commands | Windows detection is best-effort; see manual installation steps above | +| Symptom | Fix | +|---------|-----| +| `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow, then retry. | +| Connection fails after Chrome restart | Run `/browser-reload-daemon`. | +| Page seems loaded but content is missing | SPA — `browser_wait_for_load` isn't enough. Poll with `browser_execute_js` for the element you need. | +| A JS dialog is blocking actions | `browser_page_info` will report the dialog. Use `browser_handle_dialog` to accept or dismiss. | --- ## Security -> **Warning:** This extension gives your pi agent full control over your real Chrome browser. It can see your open tabs, read page content, submit forms, and interact with authenticated sessions. - -- **No sandboxing** — the agent has access to whatever Chrome has access to -- **Credentials** — if the agent hits an auth wall, it should stop and ask you. Never store credentials in agent prompts. -- **Dynamic tools** — `register_tool` evaluates user-supplied JavaScript in the pi process with full `require` access. Review any dynamic tool implementations you didn't write yourself. -- **Data handling** — Screenshots are saved to `/tmp`. HTTP responses and JS evaluation results may contain sensitive page data. -- **Session persistence** — Tab history and daemon state are stored in pi's session file (`.jsonl`). These contain page URLs and titles. - ---- - -## Contributing - -We welcome contributions! Here's how to get started: - -```bash -# Clone the repository -git clone https://github.com/amankumarsingh77/pi-browser-harness -cd pi-browser-harness - -# Install dependencies -npm install - -# Type-check -npm run typecheck - -# Test locally -pi install ./ # Install from local path -``` - -### Development Guidelines - -- **TypeScript strict mode** — all code must pass `tsc --noEmit` -- **Error handling** — tools must catch errors and return `isError: true` with a descriptive message -- **Tool names** — use `snake_case` for consistency with pi conventions -- **Parameters** — use `typebox` schemas for all tool parameters -- **Prompt guidelines** — every tool must include `promptSnippet` and `promptGuidelines` for LLM context - -### Project Structure - -``` -pi-browser-harness/ -├── src/ -│ ├── index.ts # Extension entry point, lifecycle, flags, commands -│ ├── daemon.ts # BrowserDaemon: spawns daemon.py, CDP transport -│ ├── tools.ts # All browser_* tool registrations -│ ├── dynamic-tools.ts # Self-extending harness (register_tool, etc.) -│ ├── protocol.ts # Unix socket protocol types and low-level transport -│ ├── prompt.ts # System prompt injection -│ ├── renderers.ts # Custom TUI renderers -│ ├── setup.ts # /browser-setup wizard -│ └── state.ts # Session persistence -├── skills/ -│ └── pi-browser-harness/ -│ └── SKILL.md # User-facing skill documentation -├── package.json -├── tsconfig.json -├── LICENSE -├── CHANGELOG.md -└── README.md -``` - ---- - -## Roadmap - -- [ ] Windows-native setup wizard (remove `ps`/`grep` dependency) -- [ ] iframe-target evaluation support in `browser_execute_js` -- [ ] Remote browser support (connect to Chrome running on another machine) -- [ ] Screenshot annotations (click markers, element highlights) -- [ ] Performance benchmarks and CI test suite -- [ ] Configurable screenshot output directory -- [ ] Recording/playback of interaction sequences -- [ ] Better audio/video support for media-heavy pages +This extension drives your real Chrome. The agent can see open tabs, read page content, submit forms, and act inside authenticated sessions. `register_tool` evaluates JavaScript in the pi process with full `require` access — review any dynamic tool you didn't write yourself. --- ## License -MIT — see [LICENSE](LICENSE) for details. - ---- - -<p align="center"> - <sub>Built with ❤️ for the pi ecosystem. Makes coding agents see the web.</sub> -</p> +MIT — see [LICENSE](LICENSE). |
