aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authoramankumarsingh77 <amankumarsingh7702@gmal.com>2026-04-29 01:38:17 +0530
committeramankumarsingh77 <amankumarsingh7702@gmal.com>2026-04-29 01:38:17 +0530
commit63b16957587fe20566129ffc5191f66f24a70872 (patch)
tree8a40fa38e1ae5b4309b1e38d7a3779321f15db4e /README.md
parent5317b32976edce3ab8e2f11d7fb4ff3f16dcac6f (diff)
Replace socket daemon with direct CDP WebSocket connection
Drops the Unix socket subprocess in favor of talking CDP directly over WebSocket from the extension. Removes the protocol's send/health helpers, simplifies setup to just verifying Chrome's remote debugging port, and tightens types in dynamic-tools. Bumps Node engine to >=22 and adds the ws dependency.
Diffstat (limited to 'README.md')
-rw-r--r--README.md507
1 files changed, 102 insertions, 405 deletions
diff --git a/README.md b/README.md
index 7708de0..2e97bc3 100644
--- a/README.md
+++ b/README.md
@@ -1,513 +1,210 @@
# pi-browser-harness
-> **Full browser control for pi agents.** Navigate, click, type, screenshot, and extract data from a real Chrome browser — all through CDP, all from your coding agent.
-
-<p align="center">
- <img alt="Version" src="https://img.shields.io/npm/v/@pi-browser-harness?style=flat-square&color=2563eb">
- <img alt="License" src="https://img.shields.io/badge/license-MIT-blue?style=flat-square">
- <img alt="Pi Package" src="https://img.shields.io/badge/pi-package-2563eb?style=flat-square">
- <img alt="Node" src="https://img.shields.io/badge/node-%3E%3D18-339933?style=flat-square&logo=node.js&logoColor=white">
-</p>
-
-**pi-browser-harness** gives pi agents full control of a real Chrome browser. No headless mode, no emulation — your agent sees what you see, clicks where you'd click, and reads pages exactly as they render. Browser tools coexist seamlessly with all standard pi tools (`read`, `bash`, `edit`, `write`).
-
----
-
-## Why pi-browser-harness?
-
-| | |
-|---|---|
-| 🖱️ **Real browser, real pixels** | Connects to your running Chrome via CDP. Compositor-level clicks work through iframes, shadow DOM, and cross-origin content. |
-| 🧩 **Seamless pi integration** | `browser_*` tools appear alongside `read`, `bash`, `edit` — no context switching, no separate daemon management. |
-| 🔧 **Self-extending harness** | The agent writes its own tools mid-task. Need a paginated scraper? Bulk CSV exporter? Register it at runtime, callable next turn. |
-| ⚡ **Fast data extraction** | `browser_http_get` bypasses the browser entirely for APIs — 10-50x faster than DOM scraping. `browser_execute_js` for rendered data. |
-| 🛡️ **Dialog-aware** | Auto-detects JS dialogs (alert, confirm, prompt, beforeunload) before they block your session. |
-| 📸 **Visual feedback** | Screenshots before and after every action. The agent can see exactly what it's doing. |
+Full browser control for pi agents. Navigate, click, type, screenshot, and extract data from your real Chrome — driven by your coding agent.
---
## Quick Start
```bash
-# 1. Install the package
-pi install npm:pi-browser-harness
-
-# 2. Start Chrome with remote debugging
-# macOS:
-/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 &
+# 1. Install (from this repo, until published to npm)
+pi install git:github.com/amankumarsingh77/pi-browser-harness
-# Linux:
-google-chrome --remote-debugging-port=9222 &
+# 2. Enable Chrome remote debugging
+# Open chrome://inspect/#remote-debugging in Chrome,
+# tick the "Discover network targets" checkbox, click Allow.
+# No Chrome relaunch needed.
-# 3. Run the guided setup inside pi
+# 3. Connect pi to Chrome
/browser-setup
```
-**That's it.** Your pi agent now has 20 browser control tools. Try:
+That's it. Try:
```
> navigate to https://github.com and take a screenshot
```
+### Requirements
+
+- pi (latest)
+- Node.js ≥ 22
+- Chrome / Chromium / Edge (any recent version)
+
+> If `chrome://inspect/#remote-debugging` isn't available, you can launch Chrome with `--remote-debugging-port=9222` instead. The harness will find it either way.
+
---
-## Features
+## What you can do
-### 🧭 Navigation & Tabs
+### Navigation & tabs
| Tool | What it does |
-|------|-------------|
-| `browser_navigate` | Navigate to a URL (safe: creates new tab on first call) |
+|------|--------------|
+| `browser_navigate` | Go to a URL (reuses the attached tab) |
| `browser_new_tab` | Open a new tab, optionally navigate |
-| `browser_open_urls` | Open multiple URLs in parallel tabs (great after web search) |
+| `browser_open_urls` | Open many URLs in parallel tabs |
| `browser_go_back` / `browser_go_forward` | History navigation |
| `browser_reload` | Reload the current page |
-| `browser_page_info` | Get URL, title, viewport, scroll position, or detect JS dialogs |
-| `browser_list_tabs` / `browser_current_tab` | Tab discovery and identification |
-| `browser_switch_tab` | Switch to a specific tab by targetId |
+| `browser_list_tabs` / `browser_current_tab` | Discover tabs |
+| `browser_switch_tab` | Switch the active tab |
+| `browser_page_info` | URL, title, viewport, scroll, dialog state |
-### 🖱️ Interaction
+### Interaction
| Tool | What it does |
-|------|-------------|
-| `browser_click` | Click at viewport CSS-pixel coordinates (compositor-level) |
-| `browser_type` | Type text into the focused element |
-| `browser_press_key` | Press any key with modifier support (Ctrl, Alt, Meta, Shift) |
-| `browser_scroll` | Scroll the page by delta pixels |
-| `browser_handle_dialog` | Accept or dismiss JS dialogs |
+|------|--------------|
+| `browser_click` | Click at viewport coordinates (works through iframes / shadow DOM) |
+| `browser_type` | Type into the focused element |
+| `browser_press_key` | Send a key with optional modifiers |
+| `browser_scroll` | Scroll by delta pixels |
+| `browser_handle_dialog` | Accept or dismiss `alert` / `confirm` / `prompt` |
-### 📸 Visual & Data
+### Visual & data
| Tool | What it does |
-|------|-------------|
-| `browser_screenshot` | Capture a PNG screenshot of the current page |
-| `browser_execute_js` | Execute JavaScript and return the result |
-| `browser_http_get` | Direct HTTP GET (bypasses browser — 10-50x faster for APIs) |
-| `browser_wait` | Wait N seconds |
-| `browser_wait_for_load` | Wait for `document.readyState === 'complete'` |
+|------|--------------|
+| `browser_screenshot` | Capture the current page as PNG |
+| `browser_execute_js` | Run JS in the page and return the result |
+| `browser_http_get` | Direct HTTP GET — bypasses the browser, much faster for APIs |
+| `browser_wait` / `browser_wait_for_load` | Sleep, or wait for `readyState === 'complete'` |
-### 🔧 Self-Extending Harness
+### Extending the harness at runtime
| Tool | What it does |
-|------|-------------|
-| `list_dynamic_tools` | See what's already been registered |
-| `register_tool` | Inject a new tool (available immediately, next turn) |
-| `remove_tool` | Retire a tool that's no longer needed |
-
-> **Philosophy:** "The agent writes what's missing, mid-task." When the built-in tools aren't enough, the agent extends the harness itself — no reload, no restart.
-
----
-
-## Installation
-
-### Prerequisites
-
-| Requirement | Version | Notes |
-|------------|---------|-------|
-| **pi** | Latest | The pi coding agent CLI |
-| **Node.js** | ≥ 18 | Runtime for the extension |
-| **Chrome / Chromium / Edge** | Any recent | Must support CDP (Chrome DevTools Protocol) |
-| **browser-harness daemon** | Latest | Installed automatically by `/browser-setup` |
-| **uv** (optional) | Latest | Python package manager; used for daemon installation |
-| **git** (optional) | Any | Fallback installation method |
-
-### Installing the package
-
-```bash
-# From npm (recommended)
-pi install npm:pi-browser-harness
-
-# From git
-pi install git:github.com/amankumarsingh77/pi-browser-harness
-
-# Locally (for development)
-pi install ./pi-browser-harness
-```
-
-### Installing the browser-harness daemon
-
-The daemon (`daemon.py`) is what connects pi-browser-harness to Chrome. Run the guided setup:
-
-```
-/browser-setup
-```
-
-This will:
-1. Check for an existing browser-harness installation
-2. Auto-install via `uv tool install browser-harness` or `git clone`
-3. Verify Chrome is running with remote debugging
-4. Start the daemon and test the connection
-
-**Manual installation:**
-
-```bash
-# Option A: uv (recommended)
-uv tool install browser-harness
-
-# Option B: git clone
-git clone https://github.com/browser-use/browser-harness ~/Developer/browser-harness
-cd ~/Developer/browser-harness && uv sync
-```
-
-### Enabling Chrome Remote Debugging
-
-Start Chrome with the `--remote-debugging-port` flag:
-
-```bash
-# macOS
-/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
- --remote-debugging-port=9222
-
-# Linux
-google-chrome --remote-debugging-port=9222
-
-# Windows
-"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222
-```
-
-> **Edge / Chromium**: Works identically — just replace the binary path.
+|------|--------------|
+| `list_dynamic_tools` | List tools registered this session |
+| `register_tool` | Add a new tool from JS — callable next turn |
+| `remove_tool` | Retire a registered tool |
---
## Usage
-### Core Workflow
+The core pattern is **screenshot → act → screenshot → verify.** Don't assume an action worked — look.
-The fundamental pattern: **screenshot → interact → screenshot → verify.**
+### Navigate and screenshot
```
-browser_screenshot() → find the target → browser_click(x, y) → browser_screenshot()
-```
-
-Every interaction is verified visually. Never assume an action worked.
-
-### Navigation
-
-```bash
-# First navigation: creates a new tab (preserves user's active tab)
browser_navigate("https://github.com/browser-use/browser-harness")
browser_wait_for_load()
browser_screenshot()
```
-### Form Filling
+### Fill a form
-```bash
-browser_screenshot() # See the form
-browser_click(400, 200) # Click the search input
-browser_type("pi browser harness") # Type the query
-browser_press_key("Enter") # Submit
+```
+browser_screenshot() # see the form
+browser_click(400, 200) # click the search input
+browser_type("pi browser harness") # type
+browser_press_key("Enter") # submit
browser_wait_for_load()
-browser_screenshot() # Verify results
+browser_screenshot() # verify
```
-### Data Extraction
+### Extract data
-```bash
-# Fast: Direct HTTP for APIs
+```js
+// Fast: hit the API directly, no browser
browser_http_get("https://api.github.com/repos/browser-use/browser-harness")
-# DOM scraping: Execute JS in the rendered page
+// DOM scraping for rendered content
browser_execute_js("document.querySelector('.price').innerText")
-# Structured: Extract arrays of data
+// Structured arrays
browser_execute_js(`
JSON.stringify(
Array.from(document.querySelectorAll('.result')).map(el => ({
title: el.querySelector('h3')?.textContent,
- link: el.querySelector('a')?.href,
+ link: el.querySelector('a')?.href,
}))
)
`)
```
-### Research Workflow (Search + Tabs)
-
-```bash
-# Step 1: Search
-browser_navigate("https://google.com/search?q=chrome+devtools+protocol+reference")
+### Open multiple results in parallel
-# Step 2: Open result links in parallel tabs
+```
+browser_navigate("https://google.com/search?q=chrome+devtools+protocol")
browser_open_urls({ urls: [
"https://chromedevtools.github.io/devtools-protocol/",
"https://developer.chrome.com/docs/devtools",
-] })
-
-# Step 3: List all tabs to get target IDs
+]})
browser_list_tabs()
-
-# Step 4: Switch to the tab you want
-browser_switch_tab({ targetId: "ABC123..." })
-
-# Step 5: Inspect
-browser_wait_for_load()
-browser_screenshot()
-browser_execute_js("document.querySelector('.main-content').innerText")
+browser_switch_tab({ targetId: "..." })
```
-### SPAs (React, Vue, etc.)
+### SPAs
-`browser_wait_for_load()` only checks `document.readyState === 'complete'`. SPAs often render content after this. Check for specific elements:
+`browser_wait_for_load` only checks `readyState`. For React/Vue/etc., poll for the actual element you want:
-```bash
-browser_navigate("https://my-spa.example.com")
-browser_wait_for_load()
-
-# Poll for the element to appear
+```js
browser_execute_js("!!document.querySelector('.loaded-content')")
```
-### Keyboard Modifiers
-
-`browser_press_key` supports modifier bitfield combinations:
+### Keyboard modifiers
-| Combo | Bitfield | Example |
-|-------|----------|---------|
-| Ctrl+C | 2 | `browser_press_key("c", { modifiers: 2 })` |
-| Cmd+V | 4 | `browser_press_key("v", { modifiers: 4 })` |
-| Ctrl+Shift+T | 10 (2\|8) | `browser_press_key("T", { modifiers: 10 })` |
-| Cmd+Shift+N | 12 (4\|8) | `browser_press_key("N", { modifiers: 12 })` |
-
----
-
-## Self-Extending Harness
-
-When the built-in tools aren't enough, the agent writes its own:
+`browser_press_key` takes a modifier bitfield: `Alt=1`, `Ctrl=2`, `Meta/Cmd=4`, `Shift=8`. Combine with `|`.
```
- ● agent: needs to scrape 47 paginated search result pages
- │
- ● calls list_dynamic_tools() → empty
- │
- ● calls register_tool({
- │ name: "scrape_paginated",
- │ implementation: "..."
- │ })
- │
- ✓ next turn: scrape_paginated() is available and callable
-```
-
-### Tool Lifecycle
-
-1. **`list_dynamic_tools`** — Check what's already registered
-2. **`register_tool`** — Inject a new tool with custom JavaScript
-3. **`remove_tool`** — Retire a tool you no longer need
-
-### Dynamic Tool Bindings
-
-Tools receive these bindings at runtime:
-
-| Binding | Description |
-|---------|-------------|
-| `params` | Tool arguments (as passed by the caller) |
-| `daemon` | Browser daemon — `daemon.cdp()`, `daemon.evaluateJS()`, `daemon.getPageInfo()` |
-| `require` | Node.js `require()` for builtins and installed packages |
-| `signal` | `AbortSignal` for cancellation |
-| `onUpdate` | Progress callback: `onUpdate({ content: [{ type: 'text', text: '...' }] })` |
-| `ctx` | `ExtensionContext` with `cwd`, `sessionManager`, `ui`, `signal` |
-| `console`, `fetch`, `JSON`, `Buffer`, `setTimeout`, `clearTimeout` | Standard globals |
-
-### Example: Structured Data Extractor
-
-```javascript
-// register_tool implementation:
-const info = await daemon.getPageInfo();
-if ("dialog" in info) throw new Error("Dialog is blocking: " + info.dialog.message);
-
-const data = await daemon.evaluateJS(`
- JSON.stringify(
- Array.from(document.querySelectorAll('${params.rowSelector}')).map(el => ({
- title: el.querySelector('${params.titleSelector}')?.textContent?.trim(),
- price: el.querySelector('${params.priceSelector}')?.textContent?.trim(),
- link: el.querySelector('a')?.href,
- }))
- )
-`);
-
-const parsed = JSON.parse(data);
-return {
- content: [{ type: 'text', text: `Extracted ${parsed.length} items:\n\n${JSON.stringify(parsed, null, 2)}` }],
- details: { items: parsed, count: parsed.length }
-};
+browser_press_key("c", { modifiers: 2 }) // Ctrl+C
+browser_press_key("v", { modifiers: 4 }) // Cmd+V
+browser_press_key("T", { modifiers: 10 }) // Ctrl+Shift+T
```
----
+### Writing your own tool mid-task
-## Architecture
+When the built-ins aren't enough, `register_tool` injects a new one — available on the next turn.
+```js
+register_tool({
+ name: "scrape_results",
+ implementation: `
+ const data = await daemon.evaluateJS(\`
+ JSON.stringify(
+ Array.from(document.querySelectorAll('${params.selector}'))
+ .map(el => el.textContent.trim())
+ )
+ \`);
+ return {
+ content: [{ type: "text", text: data }],
+ details: { items: JSON.parse(data) },
+ };
+ `,
+});
```
-┌─────────────────────────────────────────────────┐
-│ pi agent │
-│ │ │
-│ ├─ browser_navigate() │
-│ ├─ browser_screenshot() │
-│ ├─ browser_click(x, y) │
-│ ├─ browser_execute_js(...) │
-│ ├─ list_dynamic_tools() │
-│ ├─ register_tool(...) │
-│ └─ remove_tool(...) │
-│ │
-│ pi-browser-harness (TypeScript extension) │
-└────────────────────┬────────────────────────────┘
- │ Unix socket
- │ /tmp/bu-<namespace>.sock
- ▼
- ┌────────────────┐
- │ daemon.py │
- │ (Python) │
- └───────┬────────┘
- │ CDP WebSocket
- │ ws://localhost:9222
- ▼
- ┌────────────────┐
- │ Chrome / Edge │
- │ (user's real │
- │ browser) │
- └────────────────┘
-```
-
-### Data Flow
-
-1. **pi agent** calls a `browser_*` tool (e.g., `browser_screenshot()`)
-2. **pi-browser-harness extension** translates the call into a CDP command
-3. **JSON-line request** sent over Unix socket to `daemon.py`
-4. **daemon.py** relays the command over CDP WebSocket to Chrome
-5. **Chrome** executes the command and returns the result
-6. Result flows back through the chain to the agent
-
-### Session Persistence
-
-- Tab history, daemon namespace, and preferences persist across session reloads and branch navigation
-- State is stored via `pi.appendEntry()` in the session's JSONL file
-- On `session_start`, state is restored by scanning the branch for the last `browser-harness-state` entry
-### Key Design Decisions
-
-- **One socket connection per request** — stateless, matches the daemon's internal `_send` pattern
-- **Session auto-recovery** — stale CDP sessions are automatically re-attached
-- **Compositor-level clicks** — `Input.dispatchMouseEvent` works through iframes, shadow DOM, and cross-origin content without selectors
-- **Daemon auto-restart** — if the daemon socket dies mid-request, it's restarted and the request is retried once
+Tool implementations receive `params`, `daemon`, `require`, `signal`, `onUpdate`, `ctx`, plus the standard JS globals.
---
-## Configuration
-
-### CLI Flags
-
-| Flag | Type | Default | Description |
-|------|------|---------|-------------|
-| `--browser-namespace` | `string` | auto-generated | Daemon namespace (`BU_NAME`). Use to run multiple instances. |
-| `--browser-debug-clicks` | `boolean` | `false` | Enable debug click overlay (saves annotated screenshots to `/tmp`). |
-
-### Commands
+## Commands
| Command | Description |
|---------|-------------|
-| `/browser-setup` | Guided setup wizard — installs browser-harness, connects to Chrome |
-| `/browser-status` | Show daemon health, current page URL/title/viewport |
-| `/browser-reload-daemon` | Restart the browser daemon |
+| `/browser-setup` | Connect pi to Chrome (run once) |
+| `/browser-status` | Show daemon health and current page |
+| `/browser-reload-daemon` | Restart the connection |
---
## Troubleshooting
-| Symptom | Likely Cause | Fix |
-|---------|-------------|-----|
-| Daemon not starting | Chrome remote debugging not enabled | Start Chrome with `--remote-debugging-port=9222` |
-| "DevToolsActivePort not found" | Chrome didn't accept the debugging connection | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow |
-| Daemon starts but no tools work | Stale or dead CDP session | Run `/browser-reload-daemon` to restart |
-| Dialog blocking page | JS `alert`/`confirm`/`prompt` is open | Run `browser_page_info()` to detect it, then `browser_handle_dialog()` |
-| Page not fully loaded | SPA with async rendering | After `browser_wait_for_load()`, use `browser_execute_js()` to check for specific elements |
-| `uv` not found | Python package manager not installed | Install via `curl -LsSf https://astral.sh/uv/install.sh \| sh` or use the git fallback |
-| Socket permission denied | Stale socket file from previous run | `rm /tmp/bu-*.sock /tmp/bu-*.pid` and restart |
-| Windows: `ps` not found | `setup.ts` uses Unix commands | Windows detection is best-effort; see manual installation steps above |
+| Symptom | Fix |
+|---------|-----|
+| `DevToolsActivePort not found` | Open `chrome://inspect/#remote-debugging`, tick the checkbox, click Allow, then retry. |
+| Connection fails after Chrome restart | Run `/browser-reload-daemon`. |
+| Page seems loaded but content is missing | SPA — `browser_wait_for_load` isn't enough. Poll with `browser_execute_js` for the element you need. |
+| A JS dialog is blocking actions | `browser_page_info` will report the dialog. Use `browser_handle_dialog` to accept or dismiss. |
---
## Security
-> **Warning:** This extension gives your pi agent full control over your real Chrome browser. It can see your open tabs, read page content, submit forms, and interact with authenticated sessions.
-
-- **No sandboxing** — the agent has access to whatever Chrome has access to
-- **Credentials** — if the agent hits an auth wall, it should stop and ask you. Never store credentials in agent prompts.
-- **Dynamic tools** — `register_tool` evaluates user-supplied JavaScript in the pi process with full `require` access. Review any dynamic tool implementations you didn't write yourself.
-- **Data handling** — Screenshots are saved to `/tmp`. HTTP responses and JS evaluation results may contain sensitive page data.
-- **Session persistence** — Tab history and daemon state are stored in pi's session file (`.jsonl`). These contain page URLs and titles.
-
----
-
-## Contributing
-
-We welcome contributions! Here's how to get started:
-
-```bash
-# Clone the repository
-git clone https://github.com/amankumarsingh77/pi-browser-harness
-cd pi-browser-harness
-
-# Install dependencies
-npm install
-
-# Type-check
-npm run typecheck
-
-# Test locally
-pi install ./ # Install from local path
-```
-
-### Development Guidelines
-
-- **TypeScript strict mode** — all code must pass `tsc --noEmit`
-- **Error handling** — tools must catch errors and return `isError: true` with a descriptive message
-- **Tool names** — use `snake_case` for consistency with pi conventions
-- **Parameters** — use `typebox` schemas for all tool parameters
-- **Prompt guidelines** — every tool must include `promptSnippet` and `promptGuidelines` for LLM context
-
-### Project Structure
-
-```
-pi-browser-harness/
-├── src/
-│ ├── index.ts # Extension entry point, lifecycle, flags, commands
-│ ├── daemon.ts # BrowserDaemon: spawns daemon.py, CDP transport
-│ ├── tools.ts # All browser_* tool registrations
-│ ├── dynamic-tools.ts # Self-extending harness (register_tool, etc.)
-│ ├── protocol.ts # Unix socket protocol types and low-level transport
-│ ├── prompt.ts # System prompt injection
-│ ├── renderers.ts # Custom TUI renderers
-│ ├── setup.ts # /browser-setup wizard
-│ └── state.ts # Session persistence
-├── skills/
-│ └── pi-browser-harness/
-│ └── SKILL.md # User-facing skill documentation
-├── package.json
-├── tsconfig.json
-├── LICENSE
-├── CHANGELOG.md
-└── README.md
-```
-
----
-
-## Roadmap
-
-- [ ] Windows-native setup wizard (remove `ps`/`grep` dependency)
-- [ ] iframe-target evaluation support in `browser_execute_js`
-- [ ] Remote browser support (connect to Chrome running on another machine)
-- [ ] Screenshot annotations (click markers, element highlights)
-- [ ] Performance benchmarks and CI test suite
-- [ ] Configurable screenshot output directory
-- [ ] Recording/playback of interaction sequences
-- [ ] Better audio/video support for media-heavy pages
+This extension drives your real Chrome. The agent can see open tabs, read page content, submit forms, and act inside authenticated sessions. `register_tool` evaluates JavaScript in the pi process with full `require` access — review any dynamic tool you didn't write yourself.
---
## License
-MIT — see [LICENSE](LICENSE) for details.
-
----
-
-<p align="center">
- <sub>Built with ❤️ for the pi ecosystem. Makes coding agents see the web.</sub>
-</p>
+MIT — see [LICENSE](LICENSE).