diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 284 |
1 files changed, 71 insertions, 213 deletions
@@ -1,301 +1,159 @@ # pi-chrome -> **Let [Pi](https://pi.dev) use your existing signed-in Chrome profile after explicit authorization.** +> Let [Pi](https://pi.dev) use your existing signed-in Chrome profile after explicit authorization. -**MIT · 0 runtime deps · loopback-only bridge (`127.0.0.1:17318`) · inspect [`extensions/chrome-profile-bridge/browser-extension/`](./extensions/chrome-profile-bridge/browser-extension) before loading.** Verify connectivity in one command: `/chrome doctor`. +**MIT · 0 runtime deps · loopback-only bridge (`127.0.0.1:17318`) · inspectable unpacked Chrome extension.** Review [`extensions/chrome-profile-bridge/browser-extension/`](./extensions/chrome-profile-bridge/browser-extension) before loading. Verify setup with `/chrome doctor`. ```text -You: "Find my open GitHub PR tab, summarize review state, and screenshot the failing CI." +You: "Find my open GitHub PR tab, summarize review state, and screenshot failing CI." Agent: chrome_tab(list) → chrome_snapshot(uid:…) → chrome_screenshot(...) ✓ 3 reviewers, 1 change requested, CI red on iOS. Saved → .pi/chrome-screenshots/ci.png You: [keeps coding — agent never asked you to log in] ``` -`pi-chrome` is backed by a small MIT-licensed Chrome extension that runs inside the Chrome profile **you already use** — including every site you're already signed into. Agents can inspect or control Chrome only after you run `/chrome authorize` in the current Pi session. +`pi-chrome` runs through a small Chrome extension inside the Chrome profile **you already use** — including sites where you're already signed in. Agents can inspect or control Chrome only after you run `/chrome authorize` in current Pi session. --- -## 60-second install instruction - -To install pi-chrome, run the following command: +## Install ```bash pi install npm:pi-chrome ``` - -Then in Pi, run the next command, which will: - -1. Reveal the bundled browser-extension folder in Finder, and copy the folder path to your clipboard. -2. Pop open the chrome://extensions webpage in Chrome. - -In the Chrome Extensions page it opened, **YOU WILL NEED TO**: - -1. Turn on **developer mode** (top right). -2. Click the **load unpacked** button (top left). -3. Use **Cmd + Shift + G** (Mac) or **Ctrl + L** (Windows/Linux) to open the folder path field. -4. **Cmd + V** (Mac) or **Ctrl + V** (Windows/Linux) to paste the copied path and press Enter. -5. You're done with the chrome extensions page, and you can continue with the rest of the installation commands +In Pi: ```text /chrome onboard ``` -Reload Pi so the newly installed package is actually loaded: - -```text -/reload -``` - - -Verify the chrome connection: +This opens `chrome://extensions` and copies bundled extension path. In Chrome Extensions: -```text -/chrome doctor -``` -In the output, you just need to make sure the following line is present (It's okay if the other ones are still not checked): +1. Turn on **Developer mode**. +2. Click **Load unpacked**. +3. Open path field with **Cmd+Shift+G** on macOS or **Ctrl+L** on Windows/Linux. +4. Paste copied path. +5. Press Enter. - ✓ Chrome is connected (companion extension v0.15.36, responded in 11ms). +Reload Pi so installed package loads: -Lastly, authorize the current session by running: ```text -/chrome authorize +/reload ``` -Run the following once more, and you should see all the lines checked: +Check bridge: ```text /chrome doctor ``` ---- - -## Try this in 30 seconds after install - -```text -Look at my GitHub notifications tab and triage which PRs -need my review today, sorted by staleness. -Don't click anything yet — just read and summarize. -``` - -You'll watch the agent jump to your GitHub tab and read the page — using **your** session, **your** filters, **your** orgs. - ---- - -## Killer recipes (copy-paste into Pi) - -Each recipe assumes the relevant tab is already open in the Chrome you control. - -**PR triage** - -```text -You: "Look at my GitHub notifications tab and summarize the PRs needing my review today, sorted by staleness." -Agent: chrome_tab(list) → chrome_snapshot(uid:el-notifications) → chrome_evaluate(...) - ✓ 7 PRs waiting on you. 2 stale >3d (storage-rewrite, billing-v2). - 1 just turned CI-green (api-keys-prune). Full sorted list below. -You: [pastes the list straight into Linear] -``` -**Linear standup** +You should see: ```text -You: "Open my Linear current cycle and write a 5-bullet standup from it." -Agent: chrome_tab(activate, urlIncludes:"linear.app") → chrome_snapshot(uid:el-cycle) → chrome_evaluate(...) - ✓ 5 in-progress, 2 blocked. Standup draft: - • Shipped: bridge auto-recover. • In flight: input reliability path. - • Blocked: vendor portal CSP (waiting on infra). - • Next: benchmark v2. • Risk: none today. -You: [drops it into #standup] +✓ Chrome is connected (...) ``` -**Bug repro with evidence** +Authorize current session: ```text -You: "Reproduce the checkout 500 on staging. Save a screenshot at each step under ./repro/." -Agent: chrome_navigate(staging) → chrome_click(uid:el-add-to-cart) → chrome_screenshot(./repro/01-cart.png) - → chrome_click(uid:el-checkout) → chrome_list_network_requests() → chrome_screenshot(./repro/02-500.png) - ✓ POST /api/checkout → 500. Response body saved → ./repro/checkout-500.json - 3 screenshots in ./repro/. Trigger: missing tax_id when cart contains digital goods. -You: [files the ticket with the folder attached] +/chrome authorize +/chrome doctor ``` -<details> -<summary><strong>More recipes</strong> (form auto-fill, admin cross-check, visual diff, auth-only data pull, network forensics, file upload)</summary> - -**Form auto-fill (no submit)** -> Open the vendor portal, fill the new-vendor form from this JSON, stop before submit. - -**Admin cross-check** -> Across my Stripe / Postmark / our admin tabs, find any user where state disagrees. - -**Local dev visual diff** -> Snapshot `localhost:3000` and the staging URL of the same page; tell me what's visually different. - -**Auth-only data pull** -> Open my analytics dashboard tab and pull today's KPIs from the page. - -**Network forensics** -> Reproduce the checkout bug, find the failing API call, and dump its response body. - -**File upload through React** -> Open the photo uploader, upload `./fixtures/sample.png`, confirm the preview renders. - -</details> +Second doctor run should show all checks passing. --- -## Verifiable actions +## Quick start -Input tools return structured details such as the coordinates used, target tag, uploaded paths, key pressed, or scroll distance. For click/type/fill/key calls, pass `includeSnapshot: true` to get a fresh page snapshot in the same result: +After install + authorize, try: ```text -chrome_click(uid:"el-3", includeSnapshot:true) → - result: { input:"chrome", x:412, y:238, tag:"BUTTON" } - snapshot: { title, url, text, elements:[...] } +Look at my GitHub notifications tab and triage which PRs +need my review today, sorted by staleness. +Don't click anything yet — just read and summarize. ``` -Agents can verify page state immediately instead of blindly retrying. +More examples: [`docs/EXAMPLES.md`](./docs/EXAMPLES.md). --- -## What an agent gets - -**21 tools**, grouped by job. Every one runs against your already-open tabs. - -| Category | Tools | -| --------------- | ---------------------------------------------------------------------------------------------- | -| **Tabs** | `chrome_tab` (list/new/activate/close/version), `chrome_launch` | -| **Inspect** | `chrome_snapshot` (concise observation: layout, actions, forms, page map, query matches, diff + stable uids), `chrome_find` (query → ranked uids), `chrome_inspect` (deep single-element context), `chrome_screenshot`, `chrome_evaluate` | -| **Navigate** | `chrome_navigate` (with optional `initScript` at `document_start`), `chrome_wait_for` | -| **Interact** | `chrome_click`, `chrome_type`, `chrome_fill`, `chrome_key`, `chrome_hover` | -| **Gesture** | `chrome_drag` (Chrome pointer drag), `chrome_scroll` (wheel + momentum), `chrome_tap` (touch) | -| **Files** | `chrome_upload_file` (Chrome file-input control; no native picker) | -| **Observe** | `chrome_list_console_messages`, `chrome_list_network_requests`, `chrome_get_network_request` (with response body) | +## What it can do -Each tool is documented inline in Pi — agents see the parameters and gotchas (Chrome input, CSP limits, file upload behavior) without trial-and-error. +Pi gets browser tools for: -### Tab/window isolation +- **Tabs** — list, open, activate, close, launch Chrome. +- **Inspect** — page snapshots, element search/inspection, screenshots, JS evaluation. +- **Navigate/wait** — open URLs, wait for page state. +- **Interact** — click, type, fill, key, hover, drag, scroll, tap. +- **Files** — upload files through `<input type=file>` without native picker. +- **Observe** — console logs, network requests, response bodies. -pi-chrome never overwrites the tab you're currently looking at. The first time a chrome_* action runs without an explicit target, pi-chrome opens a **dedicated automation window** (falling back to a dedicated tab if a separate window can't be created) and reuses it for the rest of the session. Your existing tabs and windows are left untouched. To point pi-chrome at a specific tab you already have open, pass `targetId`/`urlIncludes`/`titleIncludes`. +Input tools return structured verification data. Pass `includeSnapshot: true` on click/type/fill/key calls to get fresh page state in same result. -Details: - -- **Per session.** Each Pi session owns its *own* automation window, so concurrent sessions (which all share one companion extension) never fight over a tab. -- **Survives `/reload` and Chrome service-worker restarts.** Ownership is tracked by id and mirrored to `chrome.storage.session`, so the window is reused rather than orphaned. -- **Cleanup is safe.** The dedicated target is closed when you revoke Chrome control (`/chrome revoke`) and on real session end — never on `/reload`. Cleanup only ever closes the calling session's own automation window/tab; user tabs/windows and other sessions' targets are never closed. Cleanup is fire-and-forget so it never blocks `/quit`, `/reload`, or session end. -- **Management actions are guarded.** `chrome_tab` `activate`/`close`/`group`/`ungroup` with no explicit target act on the session's automation tab if it exists, otherwise they error instead of touching your active tab. - -### Known limits vs. human Chrome use - -pi-chrome is strongest on web-page workflows exposed through DOM, screenshots, tabs, and Chrome input. It is not a full human/OS substitute. Current limitations include native Chrome/OS surfaces (print/save dialogs, permission bubbles, password-manager prompts), cross-origin iframe DOM access, rich multitouch/pinch/stylus gestures, visual CAPTCHA/bot challenges, hardware-backed auth (passkeys/security keys/biometrics), and arbitrary OS app interaction. For strict-CSP pages, use screenshot + coordinate input when `chrome_snapshot`/`chrome_evaluate` are blocked. +Tool parameters and gotchas are documented inline in Pi. --- -## Click & input behavior - -`pi-chrome` drives interactive controls through Chrome's real input layer: clicks, typing, fill, keys, hover, drag, scroll, and touch. Under the hood it uses `chrome.debugger` / CDP, so input satisfies normal user-activation gates. Chrome may show the *"Pi Chrome Connector started debugging this browser"* banner while attached. - -### Authorization +## Safety model -Chrome control is locked by default. Before any agent can use `chrome_*` tools, explicitly authorize the current Pi session from the terminal with `/chrome authorize`. +Chrome control is locked by default. Authorize per Pi session: ```text -/chrome authorize # default: authorize for 15 minutes -/chrome authorize 30m # authorize for 30 minutes -/chrome authorize 45 # custom minutes -/chrome authorize indefinite # authorize until revoked or Pi exits +/chrome authorize # 15 minutes +/chrome authorize 30m # custom duration +/chrome authorize indefinite /chrome revoke # lock again -/chrome status # shows connection + auth + background -``` - -This protects your signed-in Chrome profile from accidental agent use. The loopback bridge also rejects browser-origin command requests so arbitrary web pages cannot call into `127.0.0.1:17318` through CORS. - -### Run in background / watch modes - -By default, every `chrome_*` call runs in the **background** — Chrome won't pop up or steal focus. Switch to foreground when you want to **watch the agent work** (demos, debugging, first-time confidence). - -```text -/chrome background # toggle for the whole session -/chrome background on # run in background (default) -/chrome background off # bring Chrome forward so you can watch +/chrome status ``` -Per-call `background: false` wins over the session setting (and `background: true` forces background). - -### Diagnostics +Safety properties: -- `/chrome doctor` — single command: connectivity, extension version, bridge owner, version drift, MAIN-world helper injection, `chrome_evaluate("1+1") === 2`, fingerprint flags. -- `/chrome onboard` — guided first-time setup. -- `/chrome status` — current connection, authorization, and background state. -- `/chrome background status` — current watch/background setting. +- Extension runs in your real Chrome profile and has broad tab/scripting permissions. Install only from trusted package source. +- Pi side binds to `127.0.0.1:17318` only; no default network exposure. +- Bridge rejects browser-origin command requests, so ordinary web pages cannot drive it through CORS. +- Each Pi session gets its own automation target; user tabs/windows are not closed by cleanup. +- `/chrome revoke` closes only calling session's automation target. -If the loaded Chrome extension is older than `pi-chrome` on disk, `/chrome doctor` tells you to reload it from `chrome://extensions`. +Security details: [`SECURITY.md`](./SECURITY.md). Architecture details: [`docs/ARCHITECTURE.md`](./docs/ARCHITECTURE.md). --- -## Architecture +## Commands ```text - +----------------------+ +--------------------------+ - | Pi agent (terminal) | -- 127.0.0.1:17318 ->| Chrome extension | - | chrome_* tools | | (your real profile) | - +-----------+----------+ +-------------+------------+ - | same machine | - v v - Other Pi sessions Tabs you already have open - share the same bridge (signed in to GitHub, - automatically Linear, Stripe, etc.) -``` - -Multiple Pi sessions (planner / worker / audit) can all drive the same Chrome at once. The first session opens the local bridge; later sessions detect it and pipe their commands through. - ---- - -## Built-in benchmark suite - -[`test-suite/`](./test-suite) is a benchmark for **any** browser-control agent (not just pi-chrome). It includes **42 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks**. - -Scoring tracks expected outcomes per challenge rather than raw PASS count, so tools are judged against their declared browser-control capability. Unit challenges are split into gate buckets: - -- `core` — expected release blockers for normal trusted-mode browser control. -- `conditional` — capability/environment gated (clipboard, touch, dialogs, native UI, etc.). -- `quality` — adversarial humanization/fingerprint signals; report trends, don't block general release by default. - -Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events` and a manifest entry with expected results per mode. - -```bash -cd test-suite && python3 -m http.server 8765 -# open http://127.0.0.1:8765/ in the Chrome window pi-chrome controls +/chrome onboard # guided setup +/chrome doctor # connectivity + version + eval checks +/chrome status # connection + auth + background state +/chrome authorize [duration] +/chrome revoke +/chrome background on # default: don't steal focus +/chrome background off # foreground/watch mode +/chrome background status ``` -Categories include: `trusted-input`, `pointer-humanization`, `keyboard`, `focus-keyboard`, `activation-gates`, `scroll`, `drag-drop`, `clipboard`, `native-controls`, `frameworks`, `editing`, `dom-complexity`, `frames`, `files`, `observability`, `csp`, `lazy-loading`, `dialogs`, `popups`, `spa-routing`, `fingerprint`, and `agent-safety`. - -If you build a competing tool, please open a PR with your scores. We benchmark in public. +If loaded extension is older than installed `pi-chrome`, `/chrome doctor` tells you to reload it from `chrome://extensions`. --- -## Security model & why unpacked - -**Unpacked on purpose.** pi-chrome ships as an inspectable, MIT-licensed extension folder you load once with Developer Mode, so the local bridge and browser permissions are easy to audit and update without a Web Store release cycle. Every line is yours to read in [`extensions/chrome-profile-bridge/browser-extension/`](./extensions/chrome-profile-bridge/browser-extension). `/chrome doctor` reports the loaded extension version and warns when it drifts from your installed `pi-chrome`. +## Limits -The companion extension runs in the Chrome profile where you install it and has broad tab/scripting permissions. Only install it from a package source you trust. Even after install, `chrome_*` tools stay locked until you run `/chrome authorize` in Pi. Use `/chrome revoke` to lock them again. +`pi-chrome` works best on web-page workflows exposed through DOM, screenshots, tabs, network, console, and Chrome input. It is not full OS automation. -The Pi side listens on `127.0.0.1:17318` and rejects browser-origin command requests; ordinary web pages cannot use CORS to drive the bridge. The bundled Chrome extension currently polls that default port, so custom bridge ports are not supported without editing the extension source and reloading it. +Current limits include native Chrome/OS surfaces, print/save dialogs, permission bubbles, password-manager prompts, cross-origin iframe DOM access, CAPTCHA/bot challenges, passkeys/security keys/biometrics, rich multitouch/pinch/stylus gestures, and arbitrary desktop apps. -There is no network exposure in the default configuration; the bridge binds to loopback only. +For strict-CSP pages, use screenshots + coordinate input when snapshot/evaluate paths are blocked. --- -## Roadmap signals - -`pi-chrome` is actively shipped. Things on the near roadmap: - -- More observability tools (DOM mutation streams, performance traces) -- First-class cross-origin iframe + Shadow-DOM uid stability across snapshots -- Native-browser surface coverage where extension APIs allow it (downloads, permissions, context menus) -- Web Push & service worker introspection -- Recorder mode that emits agent prompts from your own clicks +## Docs -If you want one of those next, open an issue. +- Examples: [`docs/EXAMPLES.md`](./docs/EXAMPLES.md) +- FAQ: [`docs/FAQ.md`](./docs/FAQ.md) +- Comparison: [`docs/COMPARISON.md`](./docs/COMPARISON.md) +- Security: [`SECURITY.md`](./SECURITY.md) +- Benchmark suite: [`test-suite/README.md`](./test-suite/README.md) +- Architecture: [`docs/ARCHITECTURE.md`](./docs/ARCHITECTURE.md) --- |