path: root/README.md

# pi-chrome

[![npm version](https://img.shields.io/npm/v/pi-chrome.svg)](https://www.npmjs.com/package/pi-chrome)
[![npm downloads](https://img.shields.io/npm/dm/pi-chrome.svg)](https://www.npmjs.com/package/pi-chrome)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)

> **The fastest way to give a [Pi](https://pi.dev) agent your real Chrome.**
> No CDP. No throwaway profile. No re-login. Watch it work — or run silent.

```text
You:    "Find my open GitHub PR tab, summarize review state, and screenshot the failing CI."
Agent:  chrome_tab(list) → chrome_snapshot(uid:…) → chrome_screenshot(...)
        ✓ 3 reviewers, 1 change requested, CI red on iOS. Saved → .pi/chrome-screenshots/ci.png
You:    [keeps coding — agent never asked you to log in]
```

`pi-chrome` ships **20+ browser tools** for Pi agents, backed by a small MIT-licensed Chrome extension that runs inside the Chrome profile **you already use** — including every site you're already signed into.

---

## Why pi-chrome vs. everything else

> Short version: **pi-chrome is primitives — "Playwright for the Chrome you're already signed into."** Not an agent loop. Plug it under any agent framework (Browser Use, Stagehand, LangGraph) or call its tools directly from a Pi agent. See [docs/COMPARISON.md](./docs/COMPARISON.md) for the full three-axis landscape (drivers, agents, cloud providers).

|                                | **pi-chrome**                     | Playwright / Puppeteer        | CDP-based agents              | Selenium / WebDriver          |
| ------------------------------ | --------------------------------- | ----------------------------- | ----------------------------- | ----------------------------- |
| Uses your real signed-in Chrome | ✅ yes (extension in your profile) | ❌ throwaway profile            | ⚠️ requires `--remote-debug` | ❌ throwaway profile            |
| Re-login required               | **Never**                         | Every run                     | Sometimes                     | Every run                     |
| Watch agent work, live          | ✅ default; toggle quiet          | ❌ headless or new window      | ⚠️ debugger banner always     | ❌ new window                  |
| Works on strict-CSP pages       | ✅ `new Function` MAIN-world      | ✅                             | ✅                             | ✅                             |
| Real browser-trusted clicks     | ✅ opt-in (`chrome clicks on`)    | ✅                             | ✅                             | ✅                             |
| Multi-session safe              | ✅ shared local bridge            | ❌ port collisions             | ❌                             | ❌                             |
| Network/console capture         | ✅ built-in                       | ✅                             | ✅                             | ⚠️ via extensions             |
| Honest result envelopes¹       | ✅                                 | ⚠️                            | ❌                             | ❌                             |
| Built-in benchmark suite²      | ✅ 38 primitives + 4 long-horizon  | n/a                           | n/a                           | n/a                           |

¹ Every action returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy`, and `valueMatches` so the agent knows when a click didn't take effect — instead of looping blindly.
² See [`test-suite/`](./test-suite) — 38 primitive challenges plus 4 hermetic BrowserGym-style tasks. Scoring is expected-outcome-by-mode (`synthetic` / `trusted` / `manual`), not raw PASS count. Pages grade any browser-control tool on trusted clicks, pointer humanization, keyboard fidelity, drag/drop, clipboard, Shadow DOM, iframes, file uploads, network capture, and fingerprint leaks.

---

## What an agent gets

**20 tools**, grouped by job. Every one runs against your already-open tabs.

| Category        | Tools                                                                                          |
| --------------- | ---------------------------------------------------------------------------------------------- |
| **Tabs**        | `chrome_tab` (list/new/activate/close/version), `chrome_launch`                                |
| **Inspect**     | `chrome_snapshot` (uids + selectors + text + viewport), `chrome_screenshot`, `chrome_evaluate` |
| **Navigate**    | `chrome_navigate` (with optional `initScript` at `document_start`), `chrome_wait_for`          |
| **Interact**    | `chrome_click`, `chrome_type`, `chrome_fill`, `chrome_key`, `chrome_hover`                     |
| **Gesture**     | `chrome_drag` (HTML5 DataTransfer), `chrome_scroll` (wheel + momentum), `chrome_tap` (touch)   |
| **Files**       | `chrome_upload_file` (no native picker; works with React/Vue/Angular file inputs)              |
| **Observe**     | `chrome_list_console_messages`, `chrome_list_network_requests`, `chrome_get_network_request` (with response body) |

Each tool is documented inline in Pi — agents see the parameters and the gotchas (synthetic vs. trusted, autoplay gates, file picker limits) without trial-and-error.

---

## 60-second install

```bash
pi install npm:pi-chrome
```

Then in Pi:

```text
/chrome onboard
```

On macOS this opens `chrome://extensions`, reveals the bundled `browser-extension/` folder in Finder, and copies its path to your clipboard. In Chrome: **Developer mode** → **Load unpacked** → paste the path. Done.

Verify:

```text
/chrome doctor
```

```text
Performing Chrome bridge health check
pi-chrome v<version>
• Local bridge: mode=server, url=http://127.0.0.1:17318
✓ Companion Chrome extension responding (ID: <chrome-extension-id>, ext v<version>)
```

---

## Try this in 30 seconds after install

```text
Use chrome_tab list to find my GitHub notifications tab.
chrome_snapshot it, then write a 5-bullet triage:
which PRs need my review today, sorted by staleness.
Do not click anything yet.
```

You'll watch the agent jump to your GitHub tab and read the page — using **your** session, **your** filters, **your** orgs.

---

## Killer recipes (copy-paste into Pi)

Each one assumes tabs you already have open + accounts you're already signed into.

**PR triage**
> Use chrome_tab list to find my GitHub notifications tab, snapshot it, summarize PRs needing my review.

**Linear standup**
> Open my Linear current cycle in the active tab, snapshot it, write a 5-bullet standup.

**Bug repro with evidence**
> Open the staging app I'm already signed into, reproduce \<bug>, save a screenshot of each step under `./repro/`.

**Form auto-fill (no submit)**
> Open the vendor portal, fill the new-vendor form from this JSON, stop before submit.

**Admin cross-check**
> Across my Stripe / Postmark / our admin tabs, find any user where state disagrees.

**Local dev visual diff**
> Snapshot `localhost:3000` and the staging URL of the same page; tell me what's visually different.

**Auth-only data pull**
> Open my analytics dashboard tab and `chrome_evaluate` to extract today's KPIs from page state.

**Network forensics**
> Reproduce the checkout bug, then use `chrome_list_network_requests` to find the failing call and dump its response body.

**File upload through React**
> Open the photo uploader, `chrome_upload_file` with `./fixtures/sample.png`, confirm preview rendered.

---

## Architecture

```
  ┌──────────────────────┐                         ┌──────────────────────────┐
  │  Pi agent (terminal) │  ─── http://127.0.0.1:17318 ─→  │ Chrome extension     │
  │  chrome_* tools      │                         │ (your real profile)      │
  └──────────┬───────────┘                         └─────────┬────────────────┘
             │ same machine                                  │
             ▼                                               ▼
   Other Pi sessions                              Tabs you already have open
   share the same bridge                          (signed in to GitHub,
   automatically                                   Linear, Stripe, etc.)
```

Multiple Pi sessions (planner / worker / audit) can all drive the same Chrome at once. The first session opens the local bridge; later sessions detect it and pipe their commands through.

---

## Click & input modes

`pi-chrome` can drive Chrome two ways:

- **Quiet** — synthetic DOM events. Fast, no UI banners. Drives React/Vue/Angular state. Won't satisfy autoplay, clipboard, file picker, fullscreen, or user-activation gates.
- **Trusted** — `chrome.debugger` / CDP under the hood. Indistinguishable from a person clicking. Shows Chrome's *"Pi Chrome Connector started debugging this browser"* banner while active.

```text
/chrome clicks auto     # default: quiet, upgrade to trusted only when needed
/chrome clicks off      # always quiet, never banner
/chrome clicks on       # always trusted, banner stays up
/chrome clicks status
```

Per-call `trusted: true / false` on any input tool wins over the global mode.

---

## Background / watch modes

By default, every `chrome_*` call focuses Chrome and activates the target tab so you can **watch the agent work** — invaluable for demos, debugging, and first-time confidence.

```text
/chrome quiet          # toggle for the whole session
/chrome quiet on       # explicit
/chrome quiet off      # explicit
```

Per-call `background: true` wins over the session toggle.

---

## Honest results

Most browser-automation libraries return `void` or a generic ack. `pi-chrome` returns a structured envelope on every interaction:

```text
chrome_click(occluded-button) →
  "Clicked el-3 — pageMutated=false; occluded by <div#overlay>"
```

```text
chrome_type(react-input, "hello") →
  "Typed into el-7 — valueMatches=true; pageMutated=true"
```

This is why agents using pi-chrome don't get stuck in retry loops on broken sites. They get the **reason** the action didn't land and can fix course in one turn.

---

## Diagnostics

- `/chrome doctor` — single command: connectivity, extension version, bridge owner, version drift, MAIN-world helper injection, `chrome_evaluate("1+1") === 2`, fingerprint flags.
- `/chrome onboard` — guided first-time setup.
- `/chrome quiet status`, `/chrome clicks status` — current modes.

If the loaded Chrome extension is older than `pi-chrome` on disk, `/chrome doctor` tells you to reload it from `chrome://extensions`.

---

## Composes with

- **[pi-qq](https://www.npmjs.com/package/pi-qq)** — `/qq summarize what the active GitHub tab shows` without polluting the main transcript.
- **[pi-bar](https://www.npmjs.com/package/pi-bar)** — when the agent scrapes large pages, watch the context-usage segment turn yellow → red as a signal to `/qq` for a recap.
- **PR demo skills** — screenshots write to `.pi/chrome-screenshots/` so you can attach them to PR descriptions or demo bundles.

---

## Why an unpacked Chrome extension?

`pi-chrome` cannot ship through the Chrome Web Store — a Web Store extension cannot talk to a local bridge controlled by another tool on the same machine. So it ships as a small MIT-licensed unpacked extension in [`extensions/chrome-profile-bridge/browser-extension/`](./extensions/chrome-profile-bridge/browser-extension). **Read the source before loading.** `/chrome doctor` reports the extension version and warns when it drifts from your installed `pi-chrome`.

---

## Security model

The companion extension runs in the Chrome profile where you install it and has broad tab/scripting permissions. Only install it from a package source you trust.

The Pi side listens on `127.0.0.1:17318` by default. Override before starting Pi:

```bash
PI_CHROME_BRIDGE_PORT=17319 pi
```

There is no network exposure; the bridge binds to loopback only.

---

## Built-in benchmark suite

[`test-suite/`](./test-suite) is a benchmark for **any** browser-control agent (not just pi-chrome). It includes **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks**.

Scoring is **expected-outcome-by-mode**, not raw PASS count: each challenge has an expected verdict per mode (`synthetic`, `trusted`, `manual`) and a tool grades itself by whether its actual outcome matches the expected one. This avoids false equivalence between modes — a synthetic-events tool isn't supposed to satisfy a clipboard user-activation gate; matching that expectation is the pass.

Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events` and a manifest entry with expected results per mode.

```bash
cd test-suite && python3 -m http.server 8765
# open http://127.0.0.1:8765/ in the Chrome window pi-chrome controls
```

Categories: `trusted-input`, `pointer-humanization`, `keyboard`, `activation-gates`, `scroll`, `drag-drop`, `clipboard`, `native-controls`, `frameworks`, `editing`, `dom-complexity`, `frames`, `files`, `observability`, `fingerprint`, `agent-safety`.

If you build a competing tool, please open a PR with your scores. We benchmark in public.

---

## Roadmap signals

`pi-chrome` is actively shipped. Things on the near roadmap:

- More observability tools (DOM mutation streams, performance traces)
- First-class iframe + Shadow-DOM uid stability across snapshots
- Web Push & service worker introspection
- Recorder mode that emits agent prompts from your own clicks

If you want one of those next, open an issue.

---

## Contributing

PRs welcome. The bar:

1. Add a benchmark page in `test-suite/` that fails before your change and passes after.
2. Keep `chrome_*` tool results honest — surface `pageMutated`, `valueMatches`, `defaultPrevented`, etc.
3. Don't break the "no re-login" guarantee. Anything that requires a fresh profile is out of scope.

---

## License

MIT. See [LICENSE](./LICENSE).