pi-chrome: snapshot pre-migration state

1 files changed, 166 insertions, 0 deletions

diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md
new file mode 100644
index 0000000..4f51291
--- /dev/null
+++ b/docs/COMPARISON.md
@@ -0,0 +1,166 @@
+# pi-chrome vs. the rest of the browser-automation landscape
+
+This is the honest "which tool when" page. The browser-automation space has three different layers and people often compare across them — let's not do that.
+
+We benchmark in public — see [`../test-suite/`](../test-suite). Where exact scores matter (WebVoyager, WorkArena++, BrowseComp, Mind2Web 2), check the live leaderboards; they shift monthly.
+
+---
+
+## TL;DR
+
+| You are…                                                       | Use…                            |
+| -------------------------------------------------------------- | ------------------------------- |
+| A Pi agent operator who wants the agent to use **your real Chrome** (logged-in tabs, cookies, extensions) | **pi-chrome**                   |
+| Building a Pi/LLM agent and want low-level browser primitives  | **pi-chrome**                   |
+| Writing deterministic end-to-end tests in CI                   | Playwright / Cypress            |
+| Building a hosted scraping/agent fleet on isolated profiles    | Playwright / Puppeteer + Browserbase / Steel |
+| Want a turnkey "natural-language agent" with built-in loop     | Browser Use / Stagehand / Skyvern |
+| Want a hosted, vendor-managed agent                            | OpenAI Operator / Project Mariner / Surfer |
+| Debugging your own app from your editor without leaving your real session | **pi-chrome**                   |
+
+`pi-chrome` is **primitives**, not an opinionated agent loop. Think of it as **"Playwright for the Chrome you're already signed into"** — and pluggable under any agent framework above the line.
+
+---
+
+## The three axes you should compare on
+
+| Axis                              | Examples                                                              | What it gives you             |
+| --------------------------------- | --------------------------------------------------------------------- | ----------------------------- |
+| **1. Driver / transport**         | Playwright, Puppeteer, Selenium, CDP raw, `puppeteer-stealth`, **pi-chrome** | low-level tools (click, type, navigate) |
+| **2. Agent framework**            | Browser Use, Stagehand, Skyvern, Magnitude, Alumnium, LangGraph-with-Playwright, Operator, Mariner, Surfer | LLM loop, planning, NL API    |
+| **3. Cloud browser provider**     | Browserbase, Steel.dev, Hyperbrowser, Anchor, Browserless             | managed Chromes, sessions, quotas |
+
+`pi-chrome` is **axis 1**. You can wrap it with any axis-2 agent framework, or run it directly from a Pi agent's `chrome_*` tool calls. It explicitly does NOT compete with axis 3 — it runs locally inside *your* Chrome.
+
+---
+
+## Axis 1 — drivers (where pi-chrome lives)
+
+| Tool                              | Transport                                  | Profile                            | Trusted events       | Banner when controlling            | Default detectable as bot |
+| --------------------------------- | ------------------------------------------ | ---------------------------------- | -------------------- | ----------------------------------- | ------------------------- |
+| Playwright                        | CDP (own driver)                           | throwaway by default               | always               | always ("controlled by test software") | yes (webdriver flag, automation flags) |
+| Puppeteer                         | CDP                                        | throwaway by default               | always               | always                              | yes                       |
+| Selenium                          | WebDriver / BiDi                           | throwaway                          | partial (BiDi improves) | always                          | most detectable           |
+| puppeteer-stealth / playwright-extra | CDP + patches                           | throwaway                          | always               | always                              | medium (patches flags)    |
+| Raw CDP                           | direct devtools protocol                   | either (needs `--remote-debugging-port`) | always           | always                              | yes                       |
+| **pi-chrome**                     | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **opt-in** (`/chrome clicks on` or `trusted: true`) | **only when trusted mode is active** | **synthetic mode bypasses common detection signals**¹ |
+
+¹ pi-chrome synthetic mode dispatches DOM events with `isTrusted=false` — most sites don't check; some anti-bot defenses do. The [`test-suite/`](../test-suite) grades both modes against common detection signals. Trusted mode uses `chrome.debugger` and shows Chrome's banner like every other CDP-based tool.
+
+### What makes pi-chrome different on this axis
+
+1. **Profile attach, not driver launch.** Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it *is* your Chrome.
+2. **Synthetic-first, trusted-on-demand.** Two-tier event model — agents pick the right tradeoff per call. Competitors are all-trusted (CDP) and always show the banner. pi-chrome avoids it by default; you opt in when a site needs it (autoplay gate, clipboard, file picker).
+3. **Extension bridge transport.** No `--remote-debugging-port`, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage.
+4. **Honest result envelopes.** Every action returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy`, `valueMatches`. Competitors return `void` or generic acks; agents loop blindly on broken clicks.
+5. **Multi-session shared bridge.** Planner + worker + audit Pi sessions all drive the same Chrome concurrently.
+6. **Stable element uids.** `chrome_snapshot` returns deterministic uids you can pass to subsequent actions — similar to BrowserGym's `bid`, but built into the snapshot tool itself.
+
+---
+
+## Axis 2 — agent frameworks (built on top of axis 1)
+
+These wrap a driver with an LLM loop. They are **higher-level than pi-chrome** and **complementary**, not competitors.
+
+| Framework                | Driver underneath              | Approach                                                                                      | Open source     |
+| ------------------------ | ------------------------------ | --------------------------------------------------------------------------------------------- | --------------- |
+| **Browser Use**          | Playwright                     | DOM + a11y tree → LLM → action JSON. Open-source leader; widely cited on WebVoyager.          | MIT (Python)    |
+| **Stagehand** (Browserbase) | Playwright                  | Natural-language `.act()` / `.observe()` / `.extract()`; deterministic + AI mix.              | MIT (TypeScript)|
+| **Skyvern**              | Playwright + own DOM model     | Vision-first + DOM; YAML workflows for form/workflow automation.                              | AGPL (Python)   |
+| **Magnitude**            | Playwright                     | NL test authoring; QA-focused.                                                                | open            |
+| **Alumnium**             | Selenium / Playwright          | Test-author NL → agent. QA-focused.                                                           | open            |
+| **LangGraph / AutoGen + Playwright** | Playwright             | Generic agent graph + browser tools.                                                          | open            |
+| **OpenAI Operator**      | proprietary                    | OpenAI's own VLM + browser; ChatGPT-integrated.                                               | closed, hosted  |
+| **Project Mariner** (Google) | proprietary Chrome integration | Google's own VLM Chrome experiment.                                                       | closed          |
+| **Surfer 2 / Surfer-H** (H Company) | proprietary             | Hosted proprietary agent stack.                                                               | closed, hosted  |
+| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; trusted events at OS level.                                       | closed (API)    |
+
+**Why pi-chrome is not on this list:** it's intentionally **not an agent**. There's no LLM loop, no `.act("click the blue button")`. Pi handles the loop; pi-chrome provides the primitives. This means:
+
+- You can use pi-chrome under Browser Use, Stagehand, LangGraph, or any other agent framework (see [Interop](#interop)).
+- You don't pay for an opinion you don't want.
+- Agent improvements compound across all your workflows because the primitives stay stable.
+
+---
+
+## Axis 3 — cloud browser providers (orthogonal)
+
+| Provider          | What it sells                                            |
+| ----------------- | -------------------------------------------------------- |
+| **Browserbase**   | Managed browsers; pairs with Stagehand.                  |
+| **Steel.dev**     | Managed browsers + public agent leaderboards.            |
+| **Hyperbrowser**  | Managed browsers, session APIs.                          |
+| **Anchor Browser**| Managed browsers.                                        |
+| **Browserless**   | Managed Chrome, scraping focus.                          |
+
+**pi-chrome doesn't compete here.** It runs locally in *your* Chrome. The right framing if someone asks: *"no cloud cost, no session-handoff, no rate limits — and the agent runs against your real logged-in state."* If you need fleets of isolated Chromes in CI, you want one of these.
+
+---
+
+## Interop
+
+`pi-chrome` exposes tools that any Pi agent can call. If you want to use it from outside Pi:
+
+1. The local bridge speaks HTTP JSON-RPC at `127.0.0.1:17318` (default). The API is internal but stable across patch versions.
+2. Tool surface mirrors Playwright closely (click/type/navigate/snapshot/screenshot/evaluate/wait_for) so adapter code is short.
+3. Honest envelopes (`pageMutated`, `valueMatches`, `occludedBy`) let agent harnesses skip retry/heal logic.
+
+If you want a first-class pi-chrome adapter for Browser Use / Stagehand / LangGraph, file an issue with your use case.
+
+---
+
+## "But Playwright has `storageState` / Puppeteer has user-data-dir"
+
+Yes — you can export cookies and replay them, or point Playwright at your existing profile directory. In practice for agent workflows that breaks down fast:
+
+1. **OAuth + SSO** providers (Okta, Google, GitHub) often pin sessions to TLS fingerprints, device IDs, and browser-extension state that doesn't survive replay or a parallel Chrome instance.
+2. **MFA** tokens expire mid-run.
+3. **Internal admin tools** hard-pin to your real device.
+4. **Pointing Playwright at your real `user-data-dir`** requires closing your normal Chrome (Chrome won't share the profile lock). pi-chrome doesn't fight you for the profile because it lives *inside* it.
+5. **Watching the agent work** in your real window is a different UX than a hidden parallel Chrome. Demos, pair-driving, and confidence-building all want axis-1-with-attach.
+
+---
+
+## "Is this safer than CDP?"
+
+Different security boundary, not strictly safer.
+
+- **CDP-based tools** require `chrome --remote-debugging-port=...`. That port is unauthenticated and exposes the whole browser to any local process. Easy to misconfigure.
+- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only. **Only install the bundled extension if you trust the source you got the npm package from.**
+
+If your threat model excludes extensions with broad permissions, neither approach is a fit — you want a sandboxed CI runner.
+
+---
+
+## Public benchmarks worth knowing (for axis 2 / axis 3 comparison)
+
+Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted-input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, file uploads, network observability, fingerprint leaks, and agent-safety honeypots. Scoring is **expected-outcome-by-mode** (not raw PASS count): each challenge has expected verdicts per mode (`synthetic` / `trusted` / `manual`) and a tool grades itself by whether its actual outcome matches expectations. That's **driver-level** grading.
+
+For **agent-level** comparison (axis 2), the public benchmarks worth citing:
+
+| Benchmark        | What it measures                                  | Notes                                            |
+| ---------------- | ------------------------------------------------- | ------------------------------------------------ |
+| **WebArena** (CMU) | Hermetic, programmatic graders                  | Gold standard for reproducibility.               |
+| **WorkArena++** (ServiceNow) | Enterprise SaaS workflows             | Hardest realistic benchmark; <5% frontier.        |
+| **BrowseComp** (OpenAI) | Hard info-retrieval                        | Not saturated.                                    |
+| **Mind2Web 2** (NeurIPS '25) | Long-horizon, rubric-tree judge       | New, well-designed.                              |
+| **WebChoreArena** | Tedious cross-page workflows                     | Reflects real ops work.                          |
+| **WebVoyager**   | Live web tasks                                    | **Saturated** — 90%+ scores common; cite only as smoke test. |
+| **VisualWebArena** | Multimodal                                      |                                                  |
+| **MiniWoB++**    | Classic unit-task suite                           |                                                  |
+| **BrowserGym + AgentLab** | Research harness covering the above     | The de-facto research API; pi-chrome's snapshot uid is comparable to BrowserGym's `bid`. |
+
+Cite live leaderboards rather than hard-coded numbers; agent scores shift monthly.
+
+---
+
+## Reproducing pi-chrome's driver-level claims
+
+Run [`../test-suite/`](../test-suite) against any browser-control tool. Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events`, so any tool (Playwright, Puppeteer, Selenium, Stagehand, pi-chrome) can grade itself deterministically.
+
+```bash
+cd test-suite && python3 -m http.server 8765
+# open http://127.0.0.1:8765/ in the Chrome window the tool controls
+```
+
+If you build a competing tool, please open a PR with your scores.