diff options
| author | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-14 22:09:07 -0400 |
|---|---|---|
| committer | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-14 22:09:07 -0400 |
| commit | a13b6e4b51daf687e0cb16ef8fbc7040fde415a9 (patch) | |
| tree | 56309e6cf89c98b428fa173bd785ef97bab5b9be /docs | |
| parent | 815fd19c72f7a5a57f87eb0debe50194588e0202 (diff) | |
Restore terminal Chrome authorization
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/COMPARISON.md | 14 | ||||
| -rw-r--r-- | docs/EXAMPLES.md | 22 | ||||
| -rw-r--r-- | docs/FAQ.md | 24 |
3 files changed, 30 insertions, 30 deletions
diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md index 4f51291..d30cb6a 100644 --- a/docs/COMPARISON.md +++ b/docs/COMPARISON.md @@ -36,21 +36,21 @@ We benchmark in public — see [`../test-suite/`](../test-suite). Where exact sc ## Axis 1 — drivers (where pi-chrome lives) -| Tool | Transport | Profile | Trusted events | Banner when controlling | Default detectable as bot | +| Tool | Transport | Profile | Browser input | Banner when controlling | Default detectable as bot | | --------------------------------- | ------------------------------------------ | ---------------------------------- | -------------------- | ----------------------------------- | ------------------------- | | Playwright | CDP (own driver) | throwaway by default | always | always ("controlled by test software") | yes (webdriver flag, automation flags) | | Puppeteer | CDP | throwaway by default | always | always | yes | | Selenium | WebDriver / BiDi | throwaway | partial (BiDi improves) | always | most detectable | | puppeteer-stealth / playwright-extra | CDP + patches | throwaway | always | always | medium (patches flags) | | Raw CDP | direct devtools protocol | either (needs `--remote-debugging-port`) | always | always | yes | -| **pi-chrome** | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **opt-in** (`/chrome clicks on` or `trusted: true`) | **only when trusted mode is active** | **synthetic mode bypasses common detection signals**¹ | +| **pi-chrome** | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **always for input tools** | **while Chrome input is attached** | **low (real profile + Chrome input)¹** | -¹ pi-chrome synthetic mode dispatches DOM events with `isTrusted=false` — most sites don't check; some anti-bot defenses do. The [`test-suite/`](../test-suite) grades both modes against common detection signals. Trusted mode uses `chrome.debugger` and shows Chrome's banner like every other CDP-based tool. +¹ pi-chrome uses `chrome.debugger` for browser input and shows Chrome's banner like other CDP-based tools. The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals. ### What makes pi-chrome different on this axis 1. **Profile attach, not driver launch.** Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it *is* your Chrome. -2. **Synthetic-first, trusted-on-demand.** Two-tier event model — agents pick the right tradeoff per call. Competitors are all-trusted (CDP) and always show the banner. pi-chrome avoids it by default; you opt in when a site needs it (autoplay gate, clipboard, file picker). +2. **Chrome input against your real profile.** Interactive tools use CDP input for reliability while still controlling the Chrome profile you already use. 3. **Extension bridge transport.** No `--remote-debugging-port`, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage. 4. **Honest result envelopes.** Every action returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy`, `valueMatches`. Competitors return `void` or generic acks; agents loop blindly on broken clicks. 5. **Multi-session shared bridge.** Planner + worker + audit Pi sessions all drive the same Chrome concurrently. @@ -73,7 +73,7 @@ These wrap a driver with an LLM loop. They are **higher-level than pi-chrome** a | **OpenAI Operator** | proprietary | OpenAI's own VLM + browser; ChatGPT-integrated. | closed, hosted | | **Project Mariner** (Google) | proprietary Chrome integration | Google's own VLM Chrome experiment. | closed | | **Surfer 2 / Surfer-H** (H Company) | proprietary | Hosted proprietary agent stack. | closed, hosted | -| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; trusted events at OS level. | closed (API) | +| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; OS-level events. | closed (API) | **Why pi-chrome is not on this list:** it's intentionally **not an agent**. There's no LLM loop, no `.act("click the blue button")`. Pi handles the loop; pi-chrome provides the primitives. This means: @@ -126,7 +126,7 @@ Yes — you can export cookies and replay them, or point Playwright at your exis Different security boundary, not strictly safer. - **CDP-based tools** require `chrome --remote-debugging-port=...`. That port is unauthenticated and exposes the whole browser to any local process. Easy to misconfigure. -- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only. **Only install the bundled extension if you trust the source you got the npm package from.** +- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only, rejects browser-origin command requests, and keeps chrome_* tools locked until `/chrome authorize` is run in the current Pi session. **Only install the bundled extension if you trust the source you got the npm package from.** If your threat model excludes extensions with broad permissions, neither approach is a fit — you want a sandboxed CI runner. @@ -134,7 +134,7 @@ If your threat model excludes extensions with broad permissions, neither approac ## Public benchmarks worth knowing (for axis 2 / axis 3 comparison) -Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted-input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, file uploads, network observability, fingerprint leaks, and agent-safety honeypots. Scoring is **expected-outcome-by-mode** (not raw PASS count): each challenge has expected verdicts per mode (`synthetic` / `trusted` / `manual`) and a tool grades itself by whether its actual outcome matches expectations. That's **driver-level** grading. +Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering real input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, file uploads, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count. That's **driver-level** grading. For **agent-level** comparison (axis 2), the public benchmarks worth citing: diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md index 905d278..a5cccf4 100644 --- a/docs/EXAMPLES.md +++ b/docs/EXAMPLES.md @@ -1,6 +1,6 @@ # pi-chrome examples -Real, useful agent prompts. Drop any of these into Pi after running `/chrome onboard`. Each one uses Chrome tabs and accounts you already have. +Real, useful agent prompts. Drop any of these into Pi after running `/chrome onboard`, then `/chrome authorize`. Each one uses Chrome tabs and accounts you already have. ## Daily workflow @@ -129,14 +129,14 @@ component re-rendered with the new value. ```text chrome_upload_file paths=[./fixtures/avatar.png] selector="input[type=file]" -# No native file picker opens. Works with React/Vue/Angular controlled inputs. +# Uses Chrome file-input control. No native file picker opens. ``` ### Drag-to-reorder lists ```text chrome_drag fromUid=row-3 toUid=row-1 -# Fires real HTML5 dragstart/dragover/drop with a shared DataTransfer. +# Uses Chrome pointer drag through its input layer. ``` ## Multi-session patterns @@ -153,14 +153,14 @@ chrome_drag fromUid=row-3 toUid=row-1 A third Pi session can run `chrome_snapshot` periodically in `background: true` mode and post summaries via `pi-qq` — handy for long-running flows. -## When to prefer trusted clicks +## Chrome input -Pass `trusted: true` on `chrome_click` (or run `/chrome clicks on`) when: +Interactive tools use Chrome's real input layer by default: clicks, typing, fill, keys, hover, drag, scroll, and touch. This is reliable for: -- the click should open a file picker -- the click should write to the clipboard or read it -- the click should start an audio/video play -- the click should request fullscreen / push permission -- the page is wrapped in a strict user-activation guard (some paywalls / login flows) +- sign-in flows +- guarded buttons +- audio/video controls +- fullscreen / permission prompts +- pages with strict CSP or user-activation checks -Everything else is faster and quieter without it. +Chrome may show its debugger banner while pi-chrome is attached. diff --git a/docs/FAQ.md b/docs/FAQ.md index 2839f8f..33902d3 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -14,23 +14,23 @@ By default no — extensions need explicit "Allow in incognito" permission. Togg ## Will sites detect that I'm automating? -For **synthetic** (quiet) input: yes, technically. `event.isTrusted` is `false`. Most sites don't check; some anti-bot defenses do. +Interactive controls use Chrome's real input layer via CDP: pointer paths are humanized, key cadence has variance, and normal user-activation gates are satisfied. Some detectors check for the `chrome.debugger` API attached and Chrome will show the "Chrome is being debugged" banner. -For **trusted** (CDP) input: events are `isTrusted=true`, pointer paths are humanized, key cadence has variance. Most fingerprint-based detectors don't fire. Some specifically check for the `chrome.debugger` API attached and will show the "Chrome is being debugged" banner. That banner is the visible cost of trusted mode. - -The [`test-suite/`](../test-suite) grades both modes against common detection signals. +The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals. ## Why do I see a banner saying "Pi Chrome Connector started debugging this browser"? -That's Chrome's built-in warning when an extension uses `chrome.debugger`. pi-chrome uses it only in trusted-input mode. If you don't want to see it, run `/chrome clicks off` and accept that some sign-in flows / file pickers / clipboard ops won't work. +That's Chrome's built-in warning when an extension uses `chrome.debugger`. pi-chrome uses Chrome's input layer for interactive controls, so the banner appears while attached. ## Can a malicious page escape and access my other tabs? -No — pages cannot directly talk to extensions. Commands flow agent → local bridge (`127.0.0.1:17318`) → extension → tab. The bridge binds to loopback only. The risk surface is **other local processes** that could connect to `127.0.0.1:17318` and impersonate Pi. If that's in your threat model, run pi-chrome on a separate user account. +No — pages cannot directly talk to extensions. Commands flow agent → local bridge (`127.0.0.1:17318`) → extension → tab. The bridge binds to loopback only and rejects browser-origin command requests, so ordinary web pages cannot use CORS to drive it. + +Chrome control is also locked per Pi session until you run `/chrome authorize`; `/chrome revoke` locks it again. The remaining risk surface is **other local processes running as you** that can connect to loopback and imitate Pi. If that's in your threat model, run pi-chrome in a separate OS user account. ## Can multiple Pi sessions use it at once? -Yes. The first session opens the local bridge; later sessions detect it and pipe their commands through the same bridge. Planner + worker + audit can all drive the same Chrome concurrently. +Yes. The first session opens the local bridge; later sessions detect it and pipe their commands through the same bridge. Each Pi session must be authorized with `/chrome authorize` before its chrome_* tools work. ## Why can't this be on the Chrome Web Store? @@ -38,7 +38,7 @@ Web Store extensions cannot communicate with a local process bridge controlled b ## What happens when I update pi-chrome? -`/chrome doctor` will warn you if the loaded extension is older than the installed `pi-chrome`. Reload it from `chrome://extensions` to pick up the new version. Trusted-input mode in particular requires re-approving the `debugger` permission once. +`/chrome doctor` will warn you if the loaded extension is older than the installed `pi-chrome`. Reload it from `chrome://extensions` to pick up the new version. Updates that add Chrome permissions may require re-approval once. ## What's the install footprint? @@ -51,24 +51,24 @@ The Pi-facing tools are thin wrappers around an HTTP bridge at `127.0.0.1:17318` ## Does `chrome_evaluate` work on strict-CSP pages? -Yes. The handler compiles with `new Function(...)` in the MAIN world, which works under `script-src 'self'` without `'unsafe-eval'`. Multi-statement bodies are supported via a statement-mode fallback. Exceptions are surfaced to the agent. +Not always. `chrome_evaluate` and `chrome_snapshot` run in the page's MAIN world through the Function constructor, so pages whose CSP blocks `'unsafe-eval'` can reject them. `chrome_screenshot`, `chrome_navigate`, tab tools, and real Chrome input still work because they use extension/browser APIs rather than page JavaScript. ## Why does my click return `pageMutated=false`? Either: - The element was occluded (look for `occludedBy: <selector>` in the envelope). - The click handler called `event.preventDefault()` and the page intentionally ignored it. -- The site rejects synthetic events. Try `trusted: true` or `/chrome clicks on`. +- The target changed after your snapshot; take a fresh snapshot or screenshot. The result envelope tells you which one. **Don't blind-retry.** ## Why does `chrome_type` return `valueMatches=false`? -The editor rejected the synthetic input. Common culprits: contenteditable rich-text editors, native date pickers, masked-input libraries. Try `chrome_fill` (uses framework-aware native setters) or `trusted: true`. +The field rejected or transformed the typed value. Common culprits: contenteditable rich-text editors, native date pickers, masked-input libraries, or masks. Try `chrome_fill`, then verify with `includeSnapshot=true`. ## How do I attach a file to a React file input? -`chrome_upload_file` — populates `input.files` via a real `DataTransfer` and fires `input` + `change` events. It does **not** open the native file picker (no synthetic event can; that's a user-activation gate). Works with React/Vue/Angular controlled inputs. +`chrome_upload_file` — uses Chrome DevTools file-input control and fires `input` + `change` events. It does **not** open the native file picker. Works with React/Vue/Angular controlled inputs. ## Can it record videos? |
