pi-chrome: snapshot pre-migration state

3 files changed, 411 insertions, 0 deletions

diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md
new file mode 100644
index 0000000..4f51291
--- /dev/null
+++ b/docs/COMPARISON.md
@@ -0,0 +1,166 @@
+# pi-chrome vs. the rest of the browser-automation landscape
+
+This is the honest "which tool when" page. The browser-automation space has three different layers and people often compare across them — let's not do that.
+
+We benchmark in public — see [`../test-suite/`](../test-suite). Where exact scores matter (WebVoyager, WorkArena++, BrowseComp, Mind2Web 2), check the live leaderboards; they shift monthly.
+
+---
+
+## TL;DR
+
+| You are…                                                       | Use…                            |
+| -------------------------------------------------------------- | ------------------------------- |
+| A Pi agent operator who wants the agent to use **your real Chrome** (logged-in tabs, cookies, extensions) | **pi-chrome**                   |
+| Building a Pi/LLM agent and want low-level browser primitives  | **pi-chrome**                   |
+| Writing deterministic end-to-end tests in CI                   | Playwright / Cypress            |
+| Building a hosted scraping/agent fleet on isolated profiles    | Playwright / Puppeteer + Browserbase / Steel |
+| Want a turnkey "natural-language agent" with built-in loop     | Browser Use / Stagehand / Skyvern |
+| Want a hosted, vendor-managed agent                            | OpenAI Operator / Project Mariner / Surfer |
+| Debugging your own app from your editor without leaving your real session | **pi-chrome**                   |
+
+`pi-chrome` is **primitives**, not an opinionated agent loop. Think of it as **"Playwright for the Chrome you're already signed into"** — and pluggable under any agent framework above the line.
+
+---
+
+## The three axes you should compare on
+
+| Axis                              | Examples                                                              | What it gives you             |
+| --------------------------------- | --------------------------------------------------------------------- | ----------------------------- |
+| **1. Driver / transport**         | Playwright, Puppeteer, Selenium, CDP raw, `puppeteer-stealth`, **pi-chrome** | low-level tools (click, type, navigate) |
+| **2. Agent framework**            | Browser Use, Stagehand, Skyvern, Magnitude, Alumnium, LangGraph-with-Playwright, Operator, Mariner, Surfer | LLM loop, planning, NL API    |
+| **3. Cloud browser provider**     | Browserbase, Steel.dev, Hyperbrowser, Anchor, Browserless             | managed Chromes, sessions, quotas |
+
+`pi-chrome` is **axis 1**. You can wrap it with any axis-2 agent framework, or run it directly from a Pi agent's `chrome_*` tool calls. It explicitly does NOT compete with axis 3 — it runs locally inside *your* Chrome.
+
+---
+
+## Axis 1 — drivers (where pi-chrome lives)
+
+| Tool                              | Transport                                  | Profile                            | Trusted events       | Banner when controlling            | Default detectable as bot |
+| --------------------------------- | ------------------------------------------ | ---------------------------------- | -------------------- | ----------------------------------- | ------------------------- |
+| Playwright                        | CDP (own driver)                           | throwaway by default               | always               | always ("controlled by test software") | yes (webdriver flag, automation flags) |
+| Puppeteer                         | CDP                                        | throwaway by default               | always               | always                              | yes                       |
+| Selenium                          | WebDriver / BiDi                           | throwaway                          | partial (BiDi improves) | always                          | most detectable           |
+| puppeteer-stealth / playwright-extra | CDP + patches                           | throwaway                          | always               | always                              | medium (patches flags)    |
+| Raw CDP                           | direct devtools protocol                   | either (needs `--remote-debugging-port`) | always           | always                              | yes                       |
+| **pi-chrome**                     | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **opt-in** (`/chrome clicks on` or `trusted: true`) | **only when trusted mode is active** | **synthetic mode bypasses common detection signals**¹ |
+
+¹ pi-chrome synthetic mode dispatches DOM events with `isTrusted=false` — most sites don't check; some anti-bot defenses do. The [`test-suite/`](../test-suite) grades both modes against common detection signals. Trusted mode uses `chrome.debugger` and shows Chrome's banner like every other CDP-based tool.
+
+### What makes pi-chrome different on this axis
+
+1. **Profile attach, not driver launch.** Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it *is* your Chrome.
+2. **Synthetic-first, trusted-on-demand.** Two-tier event model — agents pick the right tradeoff per call. Competitors are all-trusted (CDP) and always show the banner. pi-chrome avoids it by default; you opt in when a site needs it (autoplay gate, clipboard, file picker).
+3. **Extension bridge transport.** No `--remote-debugging-port`, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage.
+4. **Honest result envelopes.** Every action returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy`, `valueMatches`. Competitors return `void` or generic acks; agents loop blindly on broken clicks.
+5. **Multi-session shared bridge.** Planner + worker + audit Pi sessions all drive the same Chrome concurrently.
+6. **Stable element uids.** `chrome_snapshot` returns deterministic uids you can pass to subsequent actions — similar to BrowserGym's `bid`, but built into the snapshot tool itself.
+
+---
+
+## Axis 2 — agent frameworks (built on top of axis 1)
+
+These wrap a driver with an LLM loop. They are **higher-level than pi-chrome** and **complementary**, not competitors.
+
+| Framework                | Driver underneath              | Approach                                                                                      | Open source     |
+| ------------------------ | ------------------------------ | --------------------------------------------------------------------------------------------- | --------------- |
+| **Browser Use**          | Playwright                     | DOM + a11y tree → LLM → action JSON. Open-source leader; widely cited on WebVoyager.          | MIT (Python)    |
+| **Stagehand** (Browserbase) | Playwright                  | Natural-language `.act()` / `.observe()` / `.extract()`; deterministic + AI mix.              | MIT (TypeScript)|
+| **Skyvern**              | Playwright + own DOM model     | Vision-first + DOM; YAML workflows for form/workflow automation.                              | AGPL (Python)   |
+| **Magnitude**            | Playwright                     | NL test authoring; QA-focused.                                                                | open            |
+| **Alumnium**             | Selenium / Playwright          | Test-author NL → agent. QA-focused.                                                           | open            |
+| **LangGraph / AutoGen + Playwright** | Playwright             | Generic agent graph + browser tools.                                                          | open            |
+| **OpenAI Operator**      | proprietary                    | OpenAI's own VLM + browser; ChatGPT-integrated.                                               | closed, hosted  |
+| **Project Mariner** (Google) | proprietary Chrome integration | Google's own VLM Chrome experiment.                                                       | closed          |
+| **Surfer 2 / Surfer-H** (H Company) | proprietary             | Hosted proprietary agent stack.                                                               | closed, hosted  |
+| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; trusted events at OS level.                                       | closed (API)    |
+
+**Why pi-chrome is not on this list:** it's intentionally **not an agent**. There's no LLM loop, no `.act("click the blue button")`. Pi handles the loop; pi-chrome provides the primitives. This means:
+
+- You can use pi-chrome under Browser Use, Stagehand, LangGraph, or any other agent framework (see [Interop](#interop)).
+- You don't pay for an opinion you don't want.
+- Agent improvements compound across all your workflows because the primitives stay stable.
+
+---
+
+## Axis 3 — cloud browser providers (orthogonal)
+
+| Provider          | What it sells                                            |
+| ----------------- | -------------------------------------------------------- |
+| **Browserbase**   | Managed browsers; pairs with Stagehand.                  |
+| **Steel.dev**     | Managed browsers + public agent leaderboards.            |
+| **Hyperbrowser**  | Managed browsers, session APIs.                          |
+| **Anchor Browser**| Managed browsers.                                        |
+| **Browserless**   | Managed Chrome, scraping focus.                          |
+
+**pi-chrome doesn't compete here.** It runs locally in *your* Chrome. The right framing if someone asks: *"no cloud cost, no session-handoff, no rate limits — and the agent runs against your real logged-in state."* If you need fleets of isolated Chromes in CI, you want one of these.
+
+---
+
+## Interop
+
+`pi-chrome` exposes tools that any Pi agent can call. If you want to use it from outside Pi:
+
+1. The local bridge speaks HTTP JSON-RPC at `127.0.0.1:17318` (default). The API is internal but stable across patch versions.
+2. Tool surface mirrors Playwright closely (click/type/navigate/snapshot/screenshot/evaluate/wait_for) so adapter code is short.
+3. Honest envelopes (`pageMutated`, `valueMatches`, `occludedBy`) let agent harnesses skip retry/heal logic.
+
+If you want a first-class pi-chrome adapter for Browser Use / Stagehand / LangGraph, file an issue with your use case.
+
+---
+
+## "But Playwright has `storageState` / Puppeteer has user-data-dir"
+
+Yes — you can export cookies and replay them, or point Playwright at your existing profile directory. In practice for agent workflows that breaks down fast:
+
+1. **OAuth + SSO** providers (Okta, Google, GitHub) often pin sessions to TLS fingerprints, device IDs, and browser-extension state that doesn't survive replay or a parallel Chrome instance.
+2. **MFA** tokens expire mid-run.
+3. **Internal admin tools** hard-pin to your real device.
+4. **Pointing Playwright at your real `user-data-dir`** requires closing your normal Chrome (Chrome won't share the profile lock). pi-chrome doesn't fight you for the profile because it lives *inside* it.
+5. **Watching the agent work** in your real window is a different UX than a hidden parallel Chrome. Demos, pair-driving, and confidence-building all want axis-1-with-attach.
+
+---
+
+## "Is this safer than CDP?"
+
+Different security boundary, not strictly safer.
+
+- **CDP-based tools** require `chrome --remote-debugging-port=...`. That port is unauthenticated and exposes the whole browser to any local process. Easy to misconfigure.
+- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only. **Only install the bundled extension if you trust the source you got the npm package from.**
+
+If your threat model excludes extensions with broad permissions, neither approach is a fit — you want a sandboxed CI runner.
+
+---
+
+## Public benchmarks worth knowing (for axis 2 / axis 3 comparison)
+
+Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted-input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, file uploads, network observability, fingerprint leaks, and agent-safety honeypots. Scoring is **expected-outcome-by-mode** (not raw PASS count): each challenge has expected verdicts per mode (`synthetic` / `trusted` / `manual`) and a tool grades itself by whether its actual outcome matches expectations. That's **driver-level** grading.
+
+For **agent-level** comparison (axis 2), the public benchmarks worth citing:
+
+| Benchmark        | What it measures                                  | Notes                                            |
+| ---------------- | ------------------------------------------------- | ------------------------------------------------ |
+| **WebArena** (CMU) | Hermetic, programmatic graders                  | Gold standard for reproducibility.               |
+| **WorkArena++** (ServiceNow) | Enterprise SaaS workflows             | Hardest realistic benchmark; <5% frontier.        |
+| **BrowseComp** (OpenAI) | Hard info-retrieval                        | Not saturated.                                    |
+| **Mind2Web 2** (NeurIPS '25) | Long-horizon, rubric-tree judge       | New, well-designed.                              |
+| **WebChoreArena** | Tedious cross-page workflows                     | Reflects real ops work.                          |
+| **WebVoyager**   | Live web tasks                                    | **Saturated** — 90%+ scores common; cite only as smoke test. |
+| **VisualWebArena** | Multimodal                                      |                                                  |
+| **MiniWoB++**    | Classic unit-task suite                           |                                                  |
+| **BrowserGym + AgentLab** | Research harness covering the above     | The de-facto research API; pi-chrome's snapshot uid is comparable to BrowserGym's `bid`. |
+
+Cite live leaderboards rather than hard-coded numbers; agent scores shift monthly.
+
+---
+
+## Reproducing pi-chrome's driver-level claims
+
+Run [`../test-suite/`](../test-suite) against any browser-control tool. Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events`, so any tool (Playwright, Puppeteer, Selenium, Stagehand, pi-chrome) can grade itself deterministically.
+
+```bash
+cd test-suite && python3 -m http.server 8765
+# open http://127.0.0.1:8765/ in the Chrome window the tool controls
+```
+
+If you build a competing tool, please open a PR with your scores.
diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md
new file mode 100644
index 0000000..905d278
--- /dev/null
+++ b/docs/EXAMPLES.md
@@ -0,0 +1,166 @@
+# pi-chrome examples
+
+Real, useful agent prompts. Drop any of these into Pi after running `/chrome onboard`. Each one uses Chrome tabs and accounts you already have.
+
+## Daily workflow
+
+### PR triage
+
+```text
+Use chrome_tab list to find my GitHub notifications tab.
+chrome_snapshot it. Group PRs by:
+  - awaiting my review
+  - blocked on me (changes requested back)
+  - mergeable (approved + green CI)
+Output a 5-bullet ranked triage. Do not click anything.
+```
+
+### Linear standup
+
+```text
+Open my Linear current cycle in the active tab.
+chrome_snapshot, then write yesterday/today/blockers
+in the exact format my standup channel uses.
+```
+
+### Slack catch-up
+
+```text
+For each unread channel in my Slack tab, chrome_snapshot,
+extract the top 3 messages that mention me or my team,
+and summarize what I missed in <100 words total.
+```
+
+## Debugging
+
+### Reproduce a customer bug with evidence
+
+```text
+1. chrome_navigate to https://staging.acme.com/orders/<id>
+2. chrome_snapshot
+3. Click "Refund" with chrome_click
+4. Use chrome_list_network_requests to capture the API call
+5. chrome_get_network_request on the failing one — give me the response body
+6. chrome_screenshot the error state to ./repro/refund-bug.png
+```
+
+### Visual diff local vs staging
+
+```text
+chrome_screenshot http://localhost:3000/pricing → ./diff/local.png
+chrome_screenshot https://staging.acme.com/pricing → ./diff/staging.png
+Then read both, describe layout differences in plain English.
+```
+
+### Console + network forensics
+
+```text
+Reproduce the checkout bug on the active tab.
+After the failure:
+  - chrome_list_console_messages
+  - chrome_list_network_requests
+Cross-reference the timestamps and tell me what broke first.
+```
+
+## Admin / ops
+
+### Multi-tab cross-check
+
+```text
+I have Stripe, Postmark, and our internal admin open in 3 tabs.
+For user <id>, chrome_snapshot each tab in turn and find
+any field where state disagrees. Output a 3-column table.
+```
+
+### Bulk gentle action (safe form-fill, no submit)
+
+```text
+Open our vendor portal "Add Vendor" form.
+For each row in ./vendors.csv:
+  - chrome_fill the form
+  - chrome_screenshot it
+  - STOP before submit
+  - chrome_evaluate "history.back()" to return to the list
+I will review screenshots and submit manually.
+```
+
+### Auth-only data pull
+
+```text
+My analytics dashboard is open and the cookie auth would die in headless mode.
+chrome_evaluate to read window.__APP_STATE__.dashboardData
+and dump today's KPIs as JSON.
+```
+
+## Demos / PRs
+
+### Capture screenshots for a PR description
+
+```text
+On localhost:3000/feature-x:
+  - empty state → ./pr/01-empty.png
+  - filled state → ./pr/02-filled.png
+  - error state (delete the API key from devtools first) → ./pr/03-error.png
+Save each with chrome_screenshot. Output a markdown block I can paste into the PR.
+```
+
+### Record a guided demo flow
+
+```text
+On my staging app:
+1. Walk the new-onboarding flow start to finish
+2. After each chrome_click or chrome_navigate, chrome_screenshot
+3. Save numbered PNGs under ./demo/
+4. Write narration captions for each step
+```
+
+## Forms with frameworks
+
+### React controlled inputs
+
+```text
+chrome_fill (not chrome_type) for React inputs — it uses the
+framework-aware native value setter so the form's state actually updates.
+After each fill, the result envelope's valueMatches=true confirms the
+component re-rendered with the new value.
+```
+
+### File upload without the native picker
+
+```text
+chrome_upload_file paths=[./fixtures/avatar.png] selector="input[type=file]"
+# No native file picker opens. Works with React/Vue/Angular controlled inputs.
+```
+
+### Drag-to-reorder lists
+
+```text
+chrome_drag fromUid=row-3 toUid=row-1
+# Fires real HTML5 dragstart/dragover/drop with a shared DataTransfer.
+```
+
+## Multi-session patterns
+
+`pi-chrome` shares one bridge across all Pi sessions on the same machine. Useful patterns:
+
+### Planner + Worker
+
+- **Planner session** stays high level: "find the bug, decide the fix."
+- **Worker session** runs the actual `chrome_*` tools.
+- Both see the same Chrome state because they're both pointing at your real profile.
+
+### Watcher
+
+A third Pi session can run `chrome_snapshot` periodically in `background: true` mode and post summaries via `pi-qq` — handy for long-running flows.
+
+## When to prefer trusted clicks
+
+Pass `trusted: true` on `chrome_click` (or run `/chrome clicks on`) when:
+
+- the click should open a file picker
+- the click should write to the clipboard or read it
+- the click should start an audio/video play
+- the click should request fullscreen / push permission
+- the page is wrapped in a strict user-activation guard (some paywalls / login flows)
+
+Everything else is faster and quieter without it.
diff --git a/docs/FAQ.md b/docs/FAQ.md
new file mode 100644
index 0000000..2839f8f
--- /dev/null
+++ b/docs/FAQ.md
@@ -0,0 +1,79 @@
+# pi-chrome FAQ
+
+## Does this work with Brave / Arc / Edge / Vivaldi?
+
+Yes. Any Chromium-based browser that supports unpacked extensions and the `chrome.debugger` API will work. The extension is named "Pi Chrome Connector" but the source is browser-agnostic. Firefox / WebKit are out of scope (different extension models).
+
+## Will it slow my browser down?
+
+The companion extension is idle when no Pi command is in flight. It uses Manifest V3 service worker activation, so it wakes for a request and goes back to sleep. No content script is injected globally.
+
+## Does it work in Chrome incognito?
+
+By default no — extensions need explicit "Allow in incognito" permission. Toggle it on `chrome://extensions` if you want pi-chrome to see incognito tabs. We don't recommend it for sensitive work.
+
+## Will sites detect that I'm automating?
+
+For **synthetic** (quiet) input: yes, technically. `event.isTrusted` is `false`. Most sites don't check; some anti-bot defenses do.
+
+For **trusted** (CDP) input: events are `isTrusted=true`, pointer paths are humanized, key cadence has variance. Most fingerprint-based detectors don't fire. Some specifically check for the `chrome.debugger` API attached and will show the "Chrome is being debugged" banner. That banner is the visible cost of trusted mode.
+
+The [`test-suite/`](../test-suite) grades both modes against common detection signals.
+
+## Why do I see a banner saying "Pi Chrome Connector started debugging this browser"?
+
+That's Chrome's built-in warning when an extension uses `chrome.debugger`. pi-chrome uses it only in trusted-input mode. If you don't want to see it, run `/chrome clicks off` and accept that some sign-in flows / file pickers / clipboard ops won't work.
+
+## Can a malicious page escape and access my other tabs?
+
+No — pages cannot directly talk to extensions. Commands flow agent → local bridge (`127.0.0.1:17318`) → extension → tab. The bridge binds to loopback only. The risk surface is **other local processes** that could connect to `127.0.0.1:17318` and impersonate Pi. If that's in your threat model, run pi-chrome on a separate user account.
+
+## Can multiple Pi sessions use it at once?
+
+Yes. The first session opens the local bridge; later sessions detect it and pipe their commands through the same bridge. Planner + worker + audit can all drive the same Chrome concurrently.
+
+## Why can't this be on the Chrome Web Store?
+
+Web Store extensions cannot communicate with a local process bridge controlled by another tool — Google's policy. pi-chrome must ship as an unpacked extension you load yourself. The upside: you can read the source. The downside: each Chrome update may prompt you to re-confirm.
+
+## What happens when I update pi-chrome?
+
+`/chrome doctor` will warn you if the loaded extension is older than the installed `pi-chrome`. Reload it from `chrome://extensions` to pick up the new version. Trusted-input mode in particular requires re-approving the `debugger` permission once.
+
+## What's the install footprint?
+
+- Pi side: one extension that registers ~20 tools and a few slash commands.
+- Chrome side: one unpacked extension, ~5000 LOC of plain JavaScript, no dependencies.
+
+## Can I script it without Pi?
+
+The Pi-facing tools are thin wrappers around an HTTP bridge at `127.0.0.1:17318`. You could call it directly from any process, but the API is internal and may change. If you need a stable scripting interface, file an issue and we'll consider stabilizing.
+
+## Does `chrome_evaluate` work on strict-CSP pages?
+
+Yes. The handler compiles with `new Function(...)` in the MAIN world, which works under `script-src 'self'` without `'unsafe-eval'`. Multi-statement bodies are supported via a statement-mode fallback. Exceptions are surfaced to the agent.
+
+## Why does my click return `pageMutated=false`?
+
+Either:
+- The element was occluded (look for `occludedBy: <selector>` in the envelope).
+- The click handler called `event.preventDefault()` and the page intentionally ignored it.
+- The site rejects synthetic events. Try `trusted: true` or `/chrome clicks on`.
+
+The result envelope tells you which one. **Don't blind-retry.**
+
+## Why does `chrome_type` return `valueMatches=false`?
+
+The editor rejected the synthetic input. Common culprits: contenteditable rich-text editors, native date pickers, masked-input libraries. Try `chrome_fill` (uses framework-aware native setters) or `trusted: true`.
+
+## How do I attach a file to a React file input?
+
+`chrome_upload_file` — populates `input.files` via a real `DataTransfer` and fires `input` + `change` events. It does **not** open the native file picker (no synthetic event can; that's a user-activation gate). Works with React/Vue/Angular controlled inputs.
+
+## Can it record videos?
+
+Not yet. Screenshots only. Video recording is on the roadmap.
+
+## How do I file a good bug report?
+
+Include `/chrome doctor` output, the exact tool call, and the result envelope. If the page is public, link to it; if private, distill it into a benchmark page under `test-suite/challenges/`. See [CONTRIBUTING.md](../CONTRIBUTING.md).