diff options
| author | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-14 22:09:07 -0400 |
|---|---|---|
| committer | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-14 22:09:07 -0400 |
| commit | a13b6e4b51daf687e0cb16ef8fbc7040fde415a9 (patch) | |
| tree | 56309e6cf89c98b428fa173bd785ef97bab5b9be | |
| parent | 815fd19c72f7a5a57f87eb0debe50194588e0202 (diff) | |
Restore terminal Chrome authorization
| -rw-r--r-- | CHANGELOG.md | 54 | ||||
| -rw-r--r-- | README.md | 105 | ||||
| -rw-r--r-- | SECURITY.md | 16 | ||||
| -rw-r--r-- | docs/COMPARISON.md | 14 | ||||
| -rw-r--r-- | docs/EXAMPLES.md | 22 | ||||
| -rw-r--r-- | docs/FAQ.md | 24 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/browser-extension/manifest.json | 2 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/browser-extension/service_worker.js | 230 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/index.ts | 504 | ||||
| -rw-r--r-- | package.json | 7 |
10 files changed, 467 insertions, 511 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index a782e0a..72af148 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,60 @@ All notable user-facing changes to `pi-chrome`. +## 0.15.15 — 2026-05-14 + +- **Terminal authorization restored.** `/chrome authorize` is back to terminal-based confirmation. Removed the browser-side Chrome consent page and companion-extension consent polling. + +## 0.15.14 — 2026-05-14 + +- **Clearer consent wait state.** After the Chrome approval page opens, Pi now says “Approve or deny the Chrome approval page to continue” instead of looking stuck at the launch step. + +## 0.15.13 — 2026-05-14 + +- **Fix Chrome-side consent hang.** `/chrome authorize` now launches the browser consent page as a short command, then polls for the decision. This avoids holding one long extension command open while the user reads/clicks the page, which could leave Pi stuck at “Opening Chrome approval page…”. + +## 0.15.12 — 2026-05-14 + +- **Docs accuracy.** Clarified that the bundled Chrome extension currently polls `127.0.0.1:17318`; custom bridge ports are not supported without editing/reloading the extension source. Also softened the unpacked-extension rationale to avoid overstating Web Store limitations and fixed stale strict-CSP guidance for `chrome_evaluate`. + +## 0.15.11 — 2026-05-14 + +- **README cleanup.** Removed the Playwright/CDP/Selenium comparison table and low-signal Composes with / Contributing sections from the package page because they are noisy and easy to drift. + +## 0.15.10 — 2026-05-14 + +- **Browser-side Chrome consent.** `/chrome authorize` now opens a Pi Chrome Connector approval page inside Chrome showing duration, workspace, process id, and extension/package versions. Chrome control unlocks only after the user approves there; denying, closing the tab, or timeout leaves control locked. +- **README cleanup.** Removed npm/download/license shield badges from the package page because they are noisy and easy to drift. + +## 0.15.9 — 2026-05-14 + +- **Tighter `/chrome` menu.** Removed the redundant “Connection status” item from the interactive `/chrome` menu because connection/auth/background are already shown in the menu header. `/chrome status` remains available as a slash command. + +## 0.15.8 — 2026-05-14 + +- **Simpler `/chrome` submenus.** Authorize menu now offers 15 minutes, 30 minutes, indefinite, and custom minutes only. Background menu now offers only foreground/background. Esc from a submenu returns to the main `/chrome` menu. + +## 0.15.7 — 2026-05-14 + +- **Grouped `/chrome` menu.** Bare `/chrome` now opens a status dashboard with grouped actions: authorize, lock, status, doctor, background/watch mode, and onboard. Authorize/background open submenus instead of showing one flat command list. + +## 0.15.6 — 2026-05-14 + +- **Bare `/chrome` is now a command menu.** Running `/chrome` shows interactive options for every `/chrome ...` command, including authorize/revoke/status/doctor/onboard/background variants. + +## 0.15.5 — 2026-05-14 + +- **Chrome control authorization.** `chrome_*` tools are locked until the user runs `/chrome authorize` in the current Pi session. Grants can be one command, 15 minutes, 1 hour, or the session; `/chrome revoke` locks control again and `/chrome status` shows auth state. +- **Browser-origin bridge hardening.** The loopback bridge no longer sends wildcard CORS headers. `/command` accepts only local-process requests, while extension polling/result endpoints reject non-extension browser origins, blocking ordinary web pages from driving or draining the bridge through `127.0.0.1`. + +## 0.15.4 — 2026-05-14 + +- **Quiet renamed to background.** Public UX now uses `/chrome background [on|off|toggle|status]` and docs/status say “run in background” instead of “quiet mode”. + +## 0.15.3 — 2026-05-14 + +- **Chrome real input only.** Public trusted/synthetic mode controls were removed. Interactive tools now always use Chrome's real input layer; `/chrome clicks` and public `trusted` parameters are gone. + ## 0.15.2 — 2026-05-13 - **Recipe prompts rewritten in user-language.** Earlier recipes leaked tool names into the `You:` prompts ("Use `chrome_tab list` to find my GitHub notifications tab…"), implying users need to know the tool catalog before they can ask anything. Prompts now read as natural intent; the agent trace below each one still shows the `chrome_*` primitives the agent picked. Affects the 30-second try-this block, all 3 hero recipes (PR triage / Linear standup / Bug repro), and 3 of the 6 collapsed recipes (auth-only data pull, network forensics, file upload). @@ -12,10 +12,6 @@ Agent: chrome_tab(list) → chrome_snapshot(uid:…) → chrome_screenshot(...) You: [keeps coding — agent never asked you to log in] ``` -[](https://www.npmjs.com/package/pi-chrome) -[](https://www.npmjs.com/package/pi-chrome) -[](./LICENSE) - `pi-chrome` ships **20+ browser tools** for Pi agents, backed by a small MIT-licensed Chrome extension that runs inside the Chrome profile **you already use** — including every site you're already signed into. --- @@ -34,10 +30,11 @@ Then in Pi: On macOS this opens `chrome://extensions`, reveals the bundled `browser-extension/` folder in Finder, and copies its path to your clipboard. In Chrome: **Developer mode** → **Load unpacked** → paste the path. Done. -Verify: +Verify, then authorize current Pi session in Pi: ```text /chrome doctor +/chrome authorize ``` ```text @@ -81,7 +78,7 @@ You: [pastes the list straight into Linear] You: "Open my Linear current cycle and write a 5-bullet standup from it." Agent: chrome_tab(activate, urlIncludes:"linear.app") → chrome_snapshot(uid:el-cycle) → chrome_evaluate(...) ✓ 5 in-progress, 2 blocked. Standup draft: - • Shipped: bridge auto-recover. • In flight: trusted-mode retry path. + • Shipped: bridge auto-recover. • In flight: input reliability path. • Blocked: vendor portal CSP (waiting on infra). • Next: benchmark v2. • Risk: none today. You: [drops it into #standup] @@ -123,28 +120,6 @@ You: [files the ticket with the folder attached] --- -## Why pi-chrome vs. Playwright / CDP / Selenium - -> Short version: **pi-chrome is primitives — "Playwright for the Chrome you're already signed into."** Not an agent loop. Plug it under any agent framework (Browser Use, Stagehand, LangGraph) or call its tools directly from a Pi agent. See [docs/COMPARISON.md](./docs/COMPARISON.md) for the full three-axis landscape (drivers, agents, cloud providers). - -| | **pi-chrome** | Playwright / Puppeteer | CDP-based agents | Selenium / WebDriver | -| ------------------------------ | --------------------------------- | ----------------------------- | ----------------------------- | ----------------------------- | -| **Time from `pi install` → first useful action on your real account** | ~60s (load unpacked, `/chrome doctor`) | hours (script login, store creds, debug headless) | 30+ min (`--remote-debug` setup, attach) | hours (driver + login script) | -| **Survives MFA / SSO without code** | ✅ already logged in | ❌ | ⚠️ if you re-auth | ❌ | -| Uses your real signed-in Chrome | ✅ extension in your profile | ❌ throwaway profile | ⚠️ requires `--remote-debug` | ❌ throwaway profile | -| Re-login required | **Never** | Every run | Sometimes | Every run | -| **Multiple agents drive the same Chrome at once** | ✅ shared bridge | ❌ port collisions | ❌ | ❌ | -| Watch agent work, live | ✅ default; toggle quiet | ❌ headless or new window | ⚠️ debugger banner always | ❌ new window | -| Real browser-trusted clicks | ✅ opt-in (`chrome clicks on`) | ✅ | ✅ | ✅ | -| Network/console capture | ✅ built-in | ✅ | ✅ | ⚠️ via extensions | -| **Honest result envelopes¹** | ✅ | ⚠️ | ❌ | ❌ | -| Self-graded by built-in benchmark² | ✅ 38 primitives + 4 long-horizon | n/a | n/a | n/a | - -¹ Every action returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy`, and `valueMatches` so the agent knows when a click didn't take effect — instead of looping blindly. -² [`test-suite/`](./test-suite) is mode-aware: a synthetic-events tool is *expected* to fail clipboard. If you build a competing tool, send a PR with your scores. We benchmark in public. - ---- - ## Honest results Most browser-automation libraries return `void` or a generic ack. `pi-chrome` returns a structured envelope on every interaction: @@ -173,47 +148,51 @@ This is why agents using pi-chrome don't get stuck in retry loops on broken site | **Inspect** | `chrome_snapshot` (uids + selectors + text + viewport), `chrome_screenshot`, `chrome_evaluate` | | **Navigate** | `chrome_navigate` (with optional `initScript` at `document_start`), `chrome_wait_for` | | **Interact** | `chrome_click`, `chrome_type`, `chrome_fill`, `chrome_key`, `chrome_hover` | -| **Gesture** | `chrome_drag` (HTML5 DataTransfer), `chrome_scroll` (wheel + momentum), `chrome_tap` (touch) | -| **Files** | `chrome_upload_file` (no native picker; works with React/Vue/Angular file inputs) | +| **Gesture** | `chrome_drag` (Chrome pointer drag), `chrome_scroll` (wheel + momentum), `chrome_tap` (touch) | +| **Files** | `chrome_upload_file` (Chrome file-input control; no native picker) | | **Observe** | `chrome_list_console_messages`, `chrome_list_network_requests`, `chrome_get_network_request` (with response body) | -Each tool is documented inline in Pi — agents see the parameters and the gotchas (synthetic vs. trusted, autoplay gates, file picker limits) without trial-and-error. +Each tool is documented inline in Pi — agents see the parameters and gotchas (Chrome input, CSP limits, file upload behavior) without trial-and-error. --- -## Click & input modes +## Click & input behavior + +`pi-chrome` drives interactive controls through Chrome's real input layer: clicks, typing, fill, keys, hover, drag, scroll, and touch. Under the hood it uses `chrome.debugger` / CDP, so input satisfies normal user-activation gates. Chrome may show the *"Pi Chrome Connector started debugging this browser"* banner while attached. -`pi-chrome` can drive Chrome two ways: +### Authorization -- **Quiet** — synthetic DOM events. Fast, no UI banners. Drives React/Vue/Angular state. Won't satisfy autoplay, clipboard, file picker, fullscreen, or user-activation gates. -- **Trusted** — `chrome.debugger` / CDP under the hood. Indistinguishable from a person clicking. Shows Chrome's *"Pi Chrome Connector started debugging this browser"* banner while active. +Chrome control is locked by default. Before any agent can use `chrome_*` tools, explicitly authorize the current Pi session from the terminal with `/chrome authorize`. ```text -/chrome clicks auto # default: quiet, upgrade to trusted only when needed -/chrome clicks off # always quiet, never banner -/chrome clicks on # always trusted, banner stays up -/chrome clicks status +/chrome authorize # default: authorize for 15 minutes +/chrome authorize 30m # authorize for 30 minutes +/chrome authorize 45 # custom minutes +/chrome authorize indefinite # authorize until revoked or Pi exits +/chrome revoke # lock again +/chrome status # shows connection + auth + background ``` -Per-call `trusted: true / false` on any input tool wins over the global mode. +This protects your signed-in Chrome profile from accidental agent use. The loopback bridge also rejects browser-origin command requests so arbitrary web pages cannot call into `127.0.0.1:17318` through CORS. -### Background / watch modes +### Run in background / watch modes By default, every `chrome_*` call focuses Chrome and activates the target tab so you can **watch the agent work** — invaluable for demos, debugging, and first-time confidence. ```text -/chrome quiet # toggle for the whole session -/chrome quiet on # explicit -/chrome quiet off # explicit +/chrome background # toggle for the whole session +/chrome background on # run in background +/chrome background off # bring Chrome forward so you can watch ``` -Per-call `background: true` wins over the session toggle. +Per-call `background: true` wins over the session setting. ### Diagnostics - `/chrome doctor` — single command: connectivity, extension version, bridge owner, version drift, MAIN-world helper injection, `chrome_evaluate("1+1") === 2`, fingerprint flags. - `/chrome onboard` — guided first-time setup. -- `/chrome quiet status`, `/chrome clicks status` — current modes. +- `/chrome status` — current connection, authorization, and background state. +- `/chrome background status` — current watch/background setting. If the loaded Chrome extension is older than `pi-chrome` on disk, `/chrome doctor` tells you to reload it from `chrome://extensions`. @@ -241,7 +220,7 @@ Multiple Pi sessions (planner / worker / audit) can all drive the same Chrome at [`test-suite/`](./test-suite) is a benchmark for **any** browser-control agent (not just pi-chrome). It includes **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks**. -Scoring is **expected-outcome-by-mode**, not raw PASS count: each challenge has an expected verdict per mode (`synthetic`, `trusted`, `manual`) and a tool grades itself by whether its actual outcome matches the expected one. This avoids false equivalence between modes — a synthetic-events tool isn't supposed to satisfy a clipboard user-activation gate; matching that expectation is the pass. +Scoring tracks expected outcomes per challenge rather than raw PASS count, so tools are judged against their declared browser-control capability. Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events` and a manifest entry with expected results per mode. @@ -250,7 +229,7 @@ cd test-suite && python3 -m http.server 8765 # open http://127.0.0.1:8765/ in the Chrome window pi-chrome controls ``` -Categories: `trusted-input`, `pointer-humanization`, `keyboard`, `activation-gates`, `scroll`, `drag-drop`, `clipboard`, `native-controls`, `frameworks`, `editing`, `dom-complexity`, `frames`, `files`, `observability`, `fingerprint`, `agent-safety`. +Categories: `real-input`, `pointer-humanization`, `keyboard`, `activation-gates`, `scroll`, `drag-drop`, `clipboard`, `native-controls`, `frameworks`, `editing`, `dom-complexity`, `frames`, `files`, `observability`, `fingerprint`, `agent-safety`. If you build a competing tool, please open a PR with your scores. We benchmark in public. @@ -258,25 +237,13 @@ If you build a competing tool, please open a PR with your scores. We benchmark i ## Security model & why unpacked -**Unpacked on purpose.** A Web Store extension cannot talk to a local bridge controlled by another tool on the same machine — so pi-chrome ships its bridge as an inspectable, MIT-licensed folder you load once with Developer Mode. Every line is yours to read in [`extensions/chrome-profile-bridge/browser-extension/`](./extensions/chrome-profile-bridge/browser-extension). `/chrome doctor` reports the loaded extension version and warns when it drifts from your installed `pi-chrome`. - -The companion extension runs in the Chrome profile where you install it and has broad tab/scripting permissions. Only install it from a package source you trust. - -The Pi side listens on `127.0.0.1:17318` by default. Override before starting Pi: +**Unpacked on purpose.** pi-chrome ships as an inspectable, MIT-licensed extension folder you load once with Developer Mode, so the local bridge and browser permissions are easy to audit and update without a Web Store release cycle. Every line is yours to read in [`extensions/chrome-profile-bridge/browser-extension/`](./extensions/chrome-profile-bridge/browser-extension). `/chrome doctor` reports the loaded extension version and warns when it drifts from your installed `pi-chrome`. -```bash -PI_CHROME_BRIDGE_PORT=17319 pi -``` - -There is no network exposure; the bridge binds to loopback only. - ---- +The companion extension runs in the Chrome profile where you install it and has broad tab/scripting permissions. Only install it from a package source you trust. Even after install, `chrome_*` tools stay locked until you run `/chrome authorize` in Pi. Use `/chrome revoke` to lock them again. -## Composes with +The Pi side listens on `127.0.0.1:17318` and rejects browser-origin command requests; ordinary web pages cannot use CORS to drive the bridge. The bundled Chrome extension currently polls that default port, so custom bridge ports are not supported without editing the extension source and reloading it. -- **[pi-qq](https://www.npmjs.com/package/pi-qq)** — `/qq summarize what the active GitHub tab shows` without polluting the main transcript. -- **[pi-bar](https://www.npmjs.com/package/pi-bar)** — when the agent scrapes large pages, watch the context-usage segment turn yellow → red as a signal to `/qq` for a recap. -- **PR demo skills** — screenshots write to `.pi/chrome-screenshots/` so you can attach them to PR descriptions or demo bundles. +There is no network exposure in the default configuration; the bridge binds to loopback only. --- @@ -293,16 +260,6 @@ If you want one of those next, open an issue. --- -## Contributing - -PRs welcome. The bar: - -1. Add a benchmark page in `test-suite/` that fails before your change and passes after. -2. Keep `chrome_*` tool results honest — surface `pageMutated`, `valueMatches`, `defaultPrevented`, etc. -3. Don't break the "no re-login" guarantee. Anything that requires a fresh profile is out of scope. - ---- - ## License MIT. See [LICENSE](./LICENSE). diff --git a/SECURITY.md b/SECURITY.md index afa4e15..ffc2c06 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,13 +9,14 @@ Open a GitHub issue prefixed with `[security]` at https://github.com/tianrendong `pi-chrome` is a developer tool you install knowingly. It is **not** designed to defend against: - Hostile pages running in your Chrome trying to detect or escape automation. (Standard browser security boundaries still apply, but a hostile page that already runs in your tab can do anything that page can already do.) -- Other processes on your local machine. The bridge binds to `127.0.0.1:17318` (loopback only) but **does not authenticate** local callers. Any process running as your user can issue commands. If your threat model includes hostile local processes, run pi-chrome on a separate user account. +- Other processes on your local machine. The bridge binds to `127.0.0.1:17318` (loopback only) and chrome_* tools require `/chrome authorize` inside Pi, but the bridge does not authenticate arbitrary non-browser local callers. If your threat model includes hostile local processes running as you, run pi-chrome on a separate user account. `pi-chrome` **is** designed to: - Never exfiltrate page state to the network. All communication is loopback (`127.0.0.1`). - Surface every action with an honest result envelope so the agent can't silently do the wrong thing. -- Require explicit opt-in for trusted-input mode (`/chrome clicks on` or `trusted: true`), which uses `chrome.debugger` and shows Chrome's banner. +- Keep Chrome control locked until the user explicitly runs `/chrome authorize` in the current Pi session. +- Reject browser-origin command requests to the loopback bridge so ordinary web pages cannot use CORS to drive Chrome. ## The companion extension @@ -24,14 +25,13 @@ The Chrome extension under `extensions/chrome-profile-bridge/browser-extension/` ## Defaults - Loopback bridge only. No remote port. No telemetry. -- Synthetic events first; trusted CDP only when explicitly enabled. -- Quiet mode optional; tab/window focus is observable (the user can see Pi acting). +- Chrome real input layer for interactive controls. +- Chrome control locked by default; `/chrome authorize` unlocks current Pi session after terminal confirmation, `/chrome revoke` locks it again. +- Run-in-background optional; tab/window focus is observable by default (the user can see Pi acting). -## Override the port +## Custom ports -```bash -PI_CHROME_BRIDGE_PORT=17319 pi -``` +The bundled Chrome extension currently polls `127.0.0.1:17318`. Custom bridge ports are not supported without editing the extension source and reloading it. ## Supported versions diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md index 4f51291..d30cb6a 100644 --- a/docs/COMPARISON.md +++ b/docs/COMPARISON.md @@ -36,21 +36,21 @@ We benchmark in public — see [`../test-suite/`](../test-suite). Where exact sc ## Axis 1 — drivers (where pi-chrome lives) -| Tool | Transport | Profile | Trusted events | Banner when controlling | Default detectable as bot | +| Tool | Transport | Profile | Browser input | Banner when controlling | Default detectable as bot | | --------------------------------- | ------------------------------------------ | ---------------------------------- | -------------------- | ----------------------------------- | ------------------------- | | Playwright | CDP (own driver) | throwaway by default | always | always ("controlled by test software") | yes (webdriver flag, automation flags) | | Puppeteer | CDP | throwaway by default | always | always | yes | | Selenium | WebDriver / BiDi | throwaway | partial (BiDi improves) | always | most detectable | | puppeteer-stealth / playwright-extra | CDP + patches | throwaway | always | always | medium (patches flags) | | Raw CDP | direct devtools protocol | either (needs `--remote-debugging-port`) | always | always | yes | -| **pi-chrome** | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **opt-in** (`/chrome clicks on` or `trusted: true`) | **only when trusted mode is active** | **synthetic mode bypasses common detection signals**¹ | +| **pi-chrome** | **Chrome extension bridge → local loopback** | **your real Chrome profile, signed-in cookies, extensions, history** | **always for input tools** | **while Chrome input is attached** | **low (real profile + Chrome input)¹** | -¹ pi-chrome synthetic mode dispatches DOM events with `isTrusted=false` — most sites don't check; some anti-bot defenses do. The [`test-suite/`](../test-suite) grades both modes against common detection signals. Trusted mode uses `chrome.debugger` and shows Chrome's banner like every other CDP-based tool. +¹ pi-chrome uses `chrome.debugger` for browser input and shows Chrome's banner like other CDP-based tools. The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals. ### What makes pi-chrome different on this axis 1. **Profile attach, not driver launch.** Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it *is* your Chrome. -2. **Synthetic-first, trusted-on-demand.** Two-tier event model — agents pick the right tradeoff per call. Competitors are all-trusted (CDP) and always show the banner. pi-chrome avoids it by default; you opt in when a site needs it (autoplay gate, clipboard, file picker). +2. **Chrome input against your real profile.** Interactive tools use CDP input for reliability while still controlling the Chrome profile you already use. 3. **Extension bridge transport.** No `--remote-debugging-port`, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage. 4. **Honest result envelopes.** Every action returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy`, `valueMatches`. Competitors return `void` or generic acks; agents loop blindly on broken clicks. 5. **Multi-session shared bridge.** Planner + worker + audit Pi sessions all drive the same Chrome concurrently. @@ -73,7 +73,7 @@ These wrap a driver with an LLM loop. They are **higher-level than pi-chrome** a | **OpenAI Operator** | proprietary | OpenAI's own VLM + browser; ChatGPT-integrated. | closed, hosted | | **Project Mariner** (Google) | proprietary Chrome integration | Google's own VLM Chrome experiment. | closed | | **Surfer 2 / Surfer-H** (H Company) | proprietary | Hosted proprietary agent stack. | closed, hosted | -| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; trusted events at OS level. | closed (API) | +| **Anthropic Computer Use** | OS-level screenshots + mouse/keyboard | Broader than browser; OS-level events. | closed (API) | **Why pi-chrome is not on this list:** it's intentionally **not an agent**. There's no LLM loop, no `.act("click the blue button")`. Pi handles the loop; pi-chrome provides the primitives. This means: @@ -126,7 +126,7 @@ Yes — you can export cookies and replay them, or point Playwright at your exis Different security boundary, not strictly safer. - **CDP-based tools** require `chrome --remote-debugging-port=...`. That port is unauthenticated and exposes the whole browser to any local process. Easy to misconfigure. -- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only. **Only install the bundled extension if you trust the source you got the npm package from.** +- **pi-chrome** runs through an extension you install yourself with broad permissions (tabs, scripting, debugger, webNavigation). The bridge listens on `127.0.0.1:17318` loopback only, rejects browser-origin command requests, and keeps chrome_* tools locked until `/chrome authorize` is run in the current Pi session. **Only install the bundled extension if you trust the source you got the npm package from.** If your threat model excludes extensions with broad permissions, neither approach is a fit — you want a sandboxed CI runner. @@ -134,7 +134,7 @@ If your threat model excludes extensions with broad permissions, neither approac ## Public benchmarks worth knowing (for axis 2 / axis 3 comparison) -Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted-input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, file uploads, network observability, fingerprint leaks, and agent-safety honeypots. Scoring is **expected-outcome-by-mode** (not raw PASS count): each challenge has expected verdicts per mode (`synthetic` / `trusted` / `manual`) and a tool grades itself by whether its actual outcome matches expectations. That's **driver-level** grading. +Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering real input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, file uploads, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count. That's **driver-level** grading. For **agent-level** comparison (axis 2), the public benchmarks worth citing: diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md index 905d278..a5cccf4 100644 --- a/docs/EXAMPLES.md +++ b/docs/EXAMPLES.md @@ -1,6 +1,6 @@ # pi-chrome examples -Real, useful agent prompts. Drop any of these into Pi after running `/chrome onboard`. Each one uses Chrome tabs and accounts you already have. +Real, useful agent prompts. Drop any of these into Pi after running `/chrome onboard`, then `/chrome authorize`. Each one uses Chrome tabs and accounts you already have. ## Daily workflow @@ -129,14 +129,14 @@ component re-rendered with the new value. ```text chrome_upload_file paths=[./fixtures/avatar.png] selector="input[type=file]" -# No native file picker opens. Works with React/Vue/Angular controlled inputs. +# Uses Chrome file-input control. No native file picker opens. ``` ### Drag-to-reorder lists ```text chrome_drag fromUid=row-3 toUid=row-1 -# Fires real HTML5 dragstart/dragover/drop with a shared DataTransfer. +# Uses Chrome pointer drag through its input layer. ``` ## Multi-session patterns @@ -153,14 +153,14 @@ chrome_drag fromUid=row-3 toUid=row-1 A third Pi session can run `chrome_snapshot` periodically in `background: true` mode and post summaries via `pi-qq` — handy for long-running flows. -## When to prefer trusted clicks +## Chrome input -Pass `trusted: true` on `chrome_click` (or run `/chrome clicks on`) when: +Interactive tools use Chrome's real input layer by default: clicks, typing, fill, keys, hover, drag, scroll, and touch. This is reliable for: -- the click should open a file picker -- the click should write to the clipboard or read it -- the click should start an audio/video play -- the click should request fullscreen / push permission -- the page is wrapped in a strict user-activation guard (some paywalls / login flows) +- sign-in flows +- guarded buttons +- audio/video controls +- fullscreen / permission prompts +- pages with strict CSP or user-activation checks -Everything else is faster and quieter without it. +Chrome may show its debugger banner while pi-chrome is attached. diff --git a/docs/FAQ.md b/docs/FAQ.md index 2839f8f..33902d3 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -14,23 +14,23 @@ By default no — extensions need explicit "Allow in incognito" permission. Togg ## Will sites detect that I'm automating? -For **synthetic** (quiet) input: yes, technically. `event.isTrusted` is `false`. Most sites don't check; some anti-bot defenses do. +Interactive controls use Chrome's real input layer via CDP: pointer paths are humanized, key cadence has variance, and normal user-activation gates are satisfied. Some detectors check for the `chrome.debugger` API attached and Chrome will show the "Chrome is being debugged" banner. -For **trusted** (CDP) input: events are `isTrusted=true`, pointer paths are humanized, key cadence has variance. Most fingerprint-based detectors don't fire. Some specifically check for the `chrome.debugger` API attached and will show the "Chrome is being debugged" banner. That banner is the visible cost of trusted mode. - -The [`test-suite/`](../test-suite) grades both modes against common detection signals. +The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals. ## Why do I see a banner saying "Pi Chrome Connector started debugging this browser"? -That's Chrome's built-in warning when an extension uses `chrome.debugger`. pi-chrome uses it only in trusted-input mode. If you don't want to see it, run `/chrome clicks off` and accept that some sign-in flows / file pickers / clipboard ops won't work. +That's Chrome's built-in warning when an extension uses `chrome.debugger`. pi-chrome uses Chrome's input layer for interactive controls, so the banner appears while attached. ## Can a malicious page escape and access my other tabs? -No — pages cannot directly talk to extensions. Commands flow agent → local bridge (`127.0.0.1:17318`) → extension → tab. The bridge binds to loopback only. The risk surface is **other local processes** that could connect to `127.0.0.1:17318` and impersonate Pi. If that's in your threat model, run pi-chrome on a separate user account. +No — pages cannot directly talk to extensions. Commands flow agent → local bridge (`127.0.0.1:17318`) → extension → tab. The bridge binds to loopback only and rejects browser-origin command requests, so ordinary web pages cannot use CORS to drive it. + +Chrome control is also locked per Pi session until you run `/chrome authorize`; `/chrome revoke` locks it again. The remaining risk surface is **other local processes running as you** that can connect to loopback and imitate Pi. If that's in your threat model, run pi-chrome in a separate OS user account. ## Can multiple Pi sessions use it at once? -Yes. The first session opens the local bridge; later sessions detect it and pipe their commands through the same bridge. Planner + worker + audit can all drive the same Chrome concurrently. +Yes. The first session opens the local bridge; later sessions detect it and pipe their commands through the same bridge. Each Pi session must be authorized with `/chrome authorize` before its chrome_* tools work. ## Why can't this be on the Chrome Web Store? @@ -38,7 +38,7 @@ Web Store extensions cannot communicate with a local process bridge controlled b ## What happens when I update pi-chrome? -`/chrome doctor` will warn you if the loaded extension is older than the installed `pi-chrome`. Reload it from `chrome://extensions` to pick up the new version. Trusted-input mode in particular requires re-approving the `debugger` permission once. +`/chrome doctor` will warn you if the loaded extension is older than the installed `pi-chrome`. Reload it from `chrome://extensions` to pick up the new version. Updates that add Chrome permissions may require re-approval once. ## What's the install footprint? @@ -51,24 +51,24 @@ The Pi-facing tools are thin wrappers around an HTTP bridge at `127.0.0.1:17318` ## Does `chrome_evaluate` work on strict-CSP pages? -Yes. The handler compiles with `new Function(...)` in the MAIN world, which works under `script-src 'self'` without `'unsafe-eval'`. Multi-statement bodies are supported via a statement-mode fallback. Exceptions are surfaced to the agent. +Not always. `chrome_evaluate` and `chrome_snapshot` run in the page's MAIN world through the Function constructor, so pages whose CSP blocks `'unsafe-eval'` can reject them. `chrome_screenshot`, `chrome_navigate`, tab tools, and real Chrome input still work because they use extension/browser APIs rather than page JavaScript. ## Why does my click return `pageMutated=false`? Either: - The element was occluded (look for `occludedBy: <selector>` in the envelope). - The click handler called `event.preventDefault()` and the page intentionally ignored it. -- The site rejects synthetic events. Try `trusted: true` or `/chrome clicks on`. +- The target changed after your snapshot; take a fresh snapshot or screenshot. The result envelope tells you which one. **Don't blind-retry.** ## Why does `chrome_type` return `valueMatches=false`? -The editor rejected the synthetic input. Common culprits: contenteditable rich-text editors, native date pickers, masked-input libraries. Try `chrome_fill` (uses framework-aware native setters) or `trusted: true`. +The field rejected or transformed the typed value. Common culprits: contenteditable rich-text editors, native date pickers, masked-input libraries, or masks. Try `chrome_fill`, then verify with `includeSnapshot=true`. ## How do I attach a file to a React file input? -`chrome_upload_file` — populates `input.files` via a real `DataTransfer` and fires `input` + `change` events. It does **not** open the native file picker (no synthetic event can; that's a user-activation gate). Works with React/Vue/Angular controlled inputs. +`chrome_upload_file` — uses Chrome DevTools file-input control and fires `input` + `change` events. It does **not** open the native file picker. Works with React/Vue/Angular controlled inputs. ## Can it record videos? diff --git a/extensions/chrome-profile-bridge/browser-extension/manifest.json b/extensions/chrome-profile-bridge/browser-extension/manifest.json index d998d19..4e0d8ac 100644 --- a/extensions/chrome-profile-bridge/browser-extension/manifest.json +++ b/extensions/chrome-profile-bridge/browser-extension/manifest.json @@ -1,7 +1,7 @@ { "manifest_version": 3, "name": "Pi Chrome Connector", - "version": "0.15.1", + "version": "0.15.15", "description": "Lets Pi control tabs in Chrome via a local connector at 127.0.0.1.", "permissions": [ "tabs", diff --git a/extensions/chrome-profile-bridge/browser-extension/service_worker.js b/extensions/chrome-profile-bridge/browser-extension/service_worker.js index 36e5b20..08d712b 100644 --- a/extensions/chrome-profile-bridge/browser-extension/service_worker.js +++ b/extensions/chrome-profile-bridge/browser-extension/service_worker.js @@ -3,64 +3,23 @@ const CLIENT_NAME = `Pi Chrome Connector ${chrome.runtime.id}`; const POLL_ERROR_BACKOFF_MS = 2000; let polling = false; -// =================== Trusted-input (CDP) layer =================== -// Tracks which tabs we have attached chrome.debugger to, plus session-level mode. +// =================== Chrome input (CDP) layer =================== +// Tracks which tabs we have attached chrome.debugger to. const attachedTabs = new Map(); // tabId -> { detachAt: number, pointer: {x,y} } -let TRUSTED_MODE = "auto"; // "off" | "on" | "auto" (default: smart retry only) -const TRUSTED_IDLE_DETACH_MS = 15_000; +const INPUT_IDLE_DETACH_MS = 15_000; const CDP_VERSION = "1.3"; function sleep(ms) { return new Promise((r) => setTimeout(r, ms)); } function rng(min, max) { return min + Math.random() * (max - min); } -async function wantsTrusted(params) { - if (params && params.trusted === false) return false; - if (params && params.trusted === true) return true; - return TRUSTED_MODE === "on"; -} - -function setTrustedMode(mode) { - const next = String(mode || "").toLowerCase(); - if (!["off", "on", "auto"].includes(next)) throw new Error(`bad trusted mode: ${next}`); - TRUSTED_MODE = next; - if (next === "off") void detachAll(); - return { mode: TRUSTED_MODE }; -} - -function trustedStatus() { +function inputStatus() { return { - mode: TRUSTED_MODE, attachedTabs: Array.from(attachedTabs.keys()), permissionGranted: typeof chrome !== "undefined" && !!chrome.debugger, }; } -// Auto-upgrade: if synthetic result carries suggestTrusted=true, the bridge mode is "auto" -// (default) or "on", and the caller didn't explicitly opt out, retry once with trusted CDP -// path. Surfaces both results so callers can see what happened. -async function maybeUpgradeToTrusted(kind, params, syntheticResult, trustedFn) { - if (!syntheticResult || !syntheticResult.suggestTrusted) return syntheticResult; - if (params && params.trusted === false) return syntheticResult; - if (TRUSTED_MODE === "off") return syntheticResult; - if (!chrome.debugger) return syntheticResult; - try { - const trustedResult = await trustedFn(); - return { - ...trustedResult, - autoRetried: true, - autoRetryReason: syntheticResult.suggestReason || `${kind} produced no mutation`, - syntheticAttempt: { pageMutated: syntheticResult.pageMutated, suggestReason: syntheticResult.suggestReason }, - }; - } catch (error) { - return { - ...syntheticResult, - autoRetryAttempted: true, - autoRetryError: error?.message || String(error), - }; - } -} - -// Last few attach failures, kept for /chrome doctor + trusted.debug diagnostics. +// Last few attach failures, kept for diagnostics. const attachDebugLog = []; function recordAttachEvent(entry) { attachDebugLog.push({ ...entry, t: Date.now() }); @@ -71,7 +30,7 @@ async function attachDebugger(tabId) { if (!chrome.debugger) throw new Error("chrome.debugger API unavailable; reload the extension to grant the new permission"); if (attachedTabs.has(tabId)) { const entry = attachedTabs.get(tabId); - entry.detachAt = Date.now() + TRUSTED_IDLE_DETACH_MS; + entry.detachAt = Date.now() + INPUT_IDLE_DETACH_MS; return entry; } // Before each attach, force-detach any stale CDP target this extension owns on the tab. @@ -123,19 +82,18 @@ async function attachDebugger(tabId) { } recordAttachEvent({ kind: "attached", tabId }); // Seed pointer in a plausible "just left the address bar" location. - const entry = { detachAt: Date.now() + TRUSTED_IDLE_DETACH_MS, pointer: { x: 120 + Math.random() * 200, y: 80 + Math.random() * 120 } }; + const entry = { detachAt: Date.now() + INPUT_IDLE_DETACH_MS, pointer: { x: 120 + Math.random() * 200, y: 80 + Math.random() * 120 } }; attachedTabs.set(tabId, entry); return entry; } -async function trustedDebug(params) { +async function inputDebug(params) { const tab = params?.targetId ? await chrome.tabs.get(Number(params.targetId)).catch(() => null) : null; let targets = []; try { targets = await new Promise((resolve) => chrome.debugger.getTargets((t) => resolve(t || []))); } catch {} return { extensionVersion: chrome.runtime.getManifest().version, extensionId: chrome.runtime.id, - trustedMode: TRUSTED_MODE, attachedTabs: Array.from(attachedTabs.keys()), requestedTab: tab ? { id: tab.id, url: tab.url, status: tab.status, title: tab.title } : null, cdpTargets: targets, @@ -158,7 +116,7 @@ if (chrome.debugger && chrome.debugger.onDetach) { chrome.debugger.onDetach.addListener(({ tabId }, reason) => { if (tabId !== undefined) attachedTabs.delete(tabId); if (reason === "canceled_by_user") { - console.warn(`[pi-chrome] debugger canceled by user on tab ${tabId}; trusted mode will reattach on next call`); + console.warn(`[pi-chrome] debugger canceled by user on tab ${tabId}; Chrome input will reattach on next call`); } }); } @@ -166,7 +124,7 @@ if (chrome.debugger && chrome.debugger.onDetach) { setInterval(() => { const now = Date.now(); for (const [tabId, entry] of attachedTabs) { - if (entry.detachAt && entry.detachAt < now && TRUSTED_MODE !== "on") { + if (entry.detachAt && entry.detachAt < now) { void detachDebugger(tabId); } } @@ -245,7 +203,7 @@ async function cdp(tabId, method, params) { const id = extractForeignExtId(after) || extractForeignExtId(before) || "unknown"; throw new Error( `Another Chrome extension (${id}) has an input overlay on this page (e.g. a password manager / autofill popup). \n` + - `pi-chrome tried to dismiss it with Escape but it reappeared. Disable that extension on this page, focus the field via Tab instead of clicking, or run /chrome quiet off so the agent uses synthetic input here.`, + `pi-chrome tried to dismiss it with Escape but it reappeared. Disable that extension on this page, close its popup, or focus the field via Tab instead of clicking.`, ); } throw retryErr; @@ -254,7 +212,7 @@ async function cdp(tabId, method, params) { if (!isStale) throw error; attachedTabs.delete(tabId); await chrome.debugger.attach({ tabId }, CDP_VERSION).catch(() => undefined); - attachedTabs.set(tabId, { detachAt: Date.now() + TRUSTED_IDLE_DETACH_MS, pointer: { x: 120 + Math.random() * 200, y: 80 + Math.random() * 120 } }); + attachedTabs.set(tabId, { detachAt: Date.now() + INPUT_IDLE_DETACH_MS, pointer: { x: 120 + Math.random() * 200, y: 80 + Math.random() * 120 } }); return cdpRaw(tabId, method, params); } } @@ -280,7 +238,7 @@ async function resolveTargetInTab(tabId, params) { args: [params.selector ?? null, params.uid ?? null, params.x ?? null, params.y ?? null], }); const v = results?.[0]?.result; - if (!v || !v.found) throw new Error("Could not resolve target element for trusted action"); + if (!v || !v.found) throw new Error("Could not resolve target element for Chrome input"); return v; } @@ -378,7 +336,7 @@ async function cdpTypeChar(tabId, ch) { await sleep(rng(35, 130)); } -async function trustedClick(params) { +async function chromeInputClick(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); @@ -390,7 +348,7 @@ async function trustedClick(params) { await cdp(tab.id, "Input.dispatchMouseEvent", { type: "mouseReleased", x: point.x, y: point.y, button: "left", buttons: 0, clickCount: 1, pointerType: "mouse" }); // Reset :focus-visible if the click landed on a focusable element. CDP-driven pointer // focus can leave :focus-visible=true in Chromium, which trips heuristics that expect - // pointer focus to suppress the focus ring (synthetic clicks naturally land on false). + // Reset focus styling after pointer click when possible. if (params.selector || params.uid) { await chrome.scripting.executeScript({ target: { tabId: tab.id, frameIds: [0] }, @@ -407,10 +365,10 @@ async function trustedClick(params) { args: [params.selector ?? null, params.uid ?? null], }).catch(() => undefined); } - return { trusted: true, x: point.x, y: point.y, tag: resolved.tag }; + return { input: "chrome", x: point.x, y: point.y, tag: resolved.tag }; } -async function trustedHover(params) { +async function chromeInputHover(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); @@ -418,15 +376,15 @@ async function trustedHover(params) { const point = resolved.rect ? pickInsideRect(resolved.rect) : { x: resolved.x, y: resolved.y }; await cdpMoveTo(tab.id, point.x, point.y); await sleep(rng(80, 220)); - return { trusted: true, x: point.x, y: point.y, tag: resolved.tag }; + return { input: "chrome", x: point.x, y: point.y, tag: resolved.tag }; } -async function trustedKey(params) { +async function chromeInputKey(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); const key = String(params.key || ""); - if (!key) throw new Error("trusted.key: missing key"); + if (!key) throw new Error("chrome.key: missing key"); const mods = params.modifiers || {}; const modBits = cdpModifiersFor(mods); // Press modifiers in standard order, then key, then release in reverse. @@ -456,10 +414,10 @@ async function trustedKey(params) { await sleep(rng(5, 18)); await cdp(tab.id, "Input.dispatchKeyEvent", { type: "keyUp", key: m.key, code: m.code, windowsVirtualKeyCode: m.vk, modifiers: 0 }); } - return { trusted: true, key: info.key, modifiers: mods }; + return { input: "chrome", key: info.key, modifiers: mods }; } -async function trustedType(params) { +async function chromeInputType(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); @@ -477,16 +435,16 @@ async function trustedType(params) { for (const ch of Array.from(text)) await cdpTypeChar(tab.id, ch); if (params.pressEnter) { await cdpTypeChar(tab.id, "\r").catch(() => undefined); - await trustedKey({ ...params, key: "Enter" }); + await chromeInputKey({ ...params, key: "Enter" }); } - return { trusted: true, length: text.length }; + return { input: "chrome", length: text.length }; } -async function trustedFill(params) { +async function chromeInputFill(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); - if (!(params.selector || params.uid)) throw new Error("trusted.fill: selector or uid required"); + if (!(params.selector || params.uid)) throw new Error("chrome.fill: selector or uid required"); const resolved = await resolveTargetInTab(tab.id, params); const point = resolved.rect ? pickInsideRect(resolved.rect) : { x: resolved.x, y: resolved.y }; await cdpMoveTo(tab.id, point.x, point.y); @@ -503,11 +461,11 @@ async function trustedFill(params) { await sleep(rng(20, 60)); const text = String(params.text || ""); for (const ch of Array.from(text)) await cdpTypeChar(tab.id, ch); - if (params.submit) await trustedKey({ ...params, key: "Enter" }); - return { trusted: true, length: text.length }; + if (params.submit) await chromeInputKey({ ...params, key: "Enter" }); + return { input: "chrome", length: text.length }; } -async function trustedScroll(params) { +async function chromeInputScroll(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); @@ -554,26 +512,26 @@ async function trustedScroll(params) { // Sleep one+ frame so IntersectionObserver / rAF samples can run between events. await sleep(rng(22, 48)); } - return { trusted: true, deltaX: totalX, deltaY: totalY, steps: n }; + return { input: "chrome", deltaX: totalX, deltaY: totalY, steps: n }; } -async function trustedTap(params) { +async function chromeInputTap(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); const resolved = (params.selector || params.uid || (typeof params.x === "number" && typeof params.y === "number")) ? await resolveTargetInTab(tab.id, params) : null; - if (!resolved || !resolved.found) throw new Error("trusted.tap: target not found"); + if (!resolved || !resolved.found) throw new Error("chrome.tap: target not found"); const point = resolved.rect ? pickInsideRect(resolved.rect) : { x: resolved.x, y: resolved.y }; const tp = { x: point.x, y: point.y, radiusX: 8, radiusY: 8, rotationAngle: 0, force: 0.5, id: 1 }; await cdp(tab.id, "Input.dispatchTouchEvent", { type: "touchStart", touchPoints: [tp] }); await sleep(rng(40, 110)); await cdp(tab.id, "Input.dispatchTouchEvent", { type: "touchEnd", touchPoints: [] }); - return { trusted: true, x: point.x, y: point.y, tag: resolved.tag }; + return { input: "chrome", x: point.x, y: point.y, tag: resolved.tag }; } -async function trustedDrag(params) { +async function chromeInputDrag(params) { const tab = await getTabByParams(params); if (params.foreground) await bringToFront(tab); await attachDebugger(tab.id); @@ -595,7 +553,40 @@ async function trustedDrag(params) { await sleep(rng(10, 26)); } await cdp(tab.id, "Input.dispatchMouseEvent", { type: "mouseReleased", x: tp.x, y: tp.y, button: "left", buttons: 0, clickCount: 1, pointerType: "mouse" }); - return { trusted: true, from: fp, to: tp, steps }; + return { input: "chrome", from: fp, to: tp, steps }; +} + +async function chromeInputUpload(params) { + const tab = await getTabByParams(params); + if (params.foreground) await bringToFront(tab); + await attachDebugger(tab.id); + if (!(params.selector || params.uid)) throw new Error("chrome.upload: selector or uid required"); + const paths = Array.isArray(params.paths) ? params.paths.map(String) : []; + if (!paths.length) throw new Error("chrome.upload: no file paths provided"); + const expression = `(() => { + const selector = ${JSON.stringify(params.selector ?? null)}; + const uid = ${JSON.stringify(params.uid ?? null)}; + const state = window.__PI_CHROME_STATE__; + const el = uid && state && state.elements ? state.elements[uid] : (selector ? document.querySelector(selector) : null); + if (!el || el.tagName !== "INPUT" || el.type !== "file") throw new Error("Target must be <input type=file>"); + el.scrollIntoView({ block: "center", inline: "center", behavior: "instant" }); + return el; + })()`; + const evaluated = await cdp(tab.id, "Runtime.evaluate", { expression, objectGroup: "pi-chrome-upload", includeCommandLineAPI: false, returnByValue: false }); + if (evaluated.exceptionDetails) throw new Error(evaluated.exceptionDetails.text || "Could not resolve file input"); + const objectId = evaluated.result?.objectId; + if (!objectId) throw new Error("Could not resolve file input object"); + await cdp(tab.id, "DOM.enable", {}).catch(() => undefined); + const requested = await cdp(tab.id, "DOM.requestNode", { objectId }); + if (!requested.nodeId) throw new Error("Could not resolve file input node"); + await cdp(tab.id, "DOM.setFileInputFiles", { nodeId: requested.nodeId, files: paths }); + await cdp(tab.id, "Runtime.callFunctionOn", { + objectId, + functionDeclaration: `function() { this.dispatchEvent(new Event("input", { bubbles: true })); this.dispatchEvent(new Event("change", { bubbles: true })); return this.files ? this.files.length : 0; }`, + returnByValue: true, + }).catch(() => undefined); + await cdp(tab.id, "Runtime.releaseObject", { objectId }).catch(() => undefined); + return { input: "chrome", uploaded: paths.map((path) => ({ path })) }; } // =============================================================== @@ -720,41 +711,28 @@ async function dispatch(action, params) { ]); case "page.evaluate": return evaluateInTab(params); - case "page.click": { - if (await wantsTrusted(params)) return trustedClick(params); - const synth = await executeActionInTab(params, clickPage, [params.selector ?? null, params.uid ?? null, params.x ?? null, params.y ?? null]); - return await maybeUpgradeToTrusted("click", params, synth, () => trustedClick(params)); - } + case "page.click": + return chromeInputClick(params); case "page.hover": - if (await wantsTrusted(params)) return trustedHover(params); - return executeActionInTab(params, hoverPage, [params.selector ?? null, params.uid ?? null, params.x ?? null, params.y ?? null]); + return chromeInputHover(params); case "page.drag": - if (await wantsTrusted(params)) return trustedDrag(params); - return executeActionInTab(params, dragPage, [params.fromUid ?? null, params.fromSelector ?? null, params.fromX ?? null, params.fromY ?? null, params.toUid ?? null, params.toSelector ?? null, params.toX ?? null, params.toY ?? null, params.steps ?? 12]); + return chromeInputDrag(params); case "page.upload": - return executeActionInTab(params, uploadFiles, [params.selector ?? null, params.uid ?? null, params.files || []]); - case "page.type": { - if (await wantsTrusted(params)) return trustedType(params); - const synth = await executeActionInTab(params, typeIntoPage, [params.selector ?? null, params.uid ?? null, params.text || "", Boolean(params.pressEnter)]); - return await maybeUpgradeToTrusted("type", params, synth, () => trustedType(params)); - } + return chromeInputUpload(params); + case "page.type": + return chromeInputType(params); case "page.fill": - if (await wantsTrusted(params)) return trustedFill(params); - return executeActionInTab(params, fillPage, [params.selector ?? null, params.uid ?? null, params.text || "", params.submit === true]); + return chromeInputFill(params); case "page.key": - if (await wantsTrusted(params)) return trustedKey(params); - return executeActionInTab(params, pressKeyInPage, [params.key]); + return chromeInputKey(params); case "page.scroll": - if (await wantsTrusted(params)) return trustedScroll(params); - return executeActionInTab(params, scrollPage, [params.selector ?? null, params.uid ?? null, params.deltaY ?? 0, params.deltaX ?? 0, params.steps ?? null]); + return chromeInputScroll(params); case "page.tap": - return trustedTap(params); - case "trusted.mode": - return setTrustedMode(params.mode); - case "trusted.status": - return trustedStatus(); - case "trusted.debug": - return trustedDebug(params); + return chromeInputTap(params); + case "input.status": + return inputStatus(); + case "input.debug": + return inputDebug(params); case "page.console.list": return executeInTab(params, listConsoleMessages, [params.clear === true]); case "page.network.list": @@ -1503,31 +1481,31 @@ async function clickPage(selector, uid, x, y) { defaultPrevented = dispatchPointerLikeEvent(point.element, "click", point.x, point.y, prevX, prevY) || defaultPrevented; state.pointer = { x: point.x, y: point.y, t: performance.now() }; // Heuristic: if the clicked thing looks like a media play affordance and the page has paused - // audio/video, the synthetic click may not unlock autoplay. Surface a warning. + // audio/video, the DOM-event click may not unlock autoplay. Surface a warning. let autoplayHint; const labelRaw = (point.element.getAttribute("aria-label") || point.element.textContent || "").trim(); const label = labelRaw.toLowerCase(); if (/^(play|start|begin|next|continue|unmute)/.test(label)) { const idleMedia = Array.from(document.querySelectorAll("audio,video")).some((m) => m.paused); - if (idleMedia) autoplayHint = "This element looks like a media affordance and the page has paused media. Synthetic clicks do not satisfy user-activation gates; audio/video may not start."; + if (idleMedia) autoplayHint = "This element looks like a media affordance and the page has paused media. DOM-event clicks do not satisfy user-activation gates; audio/video may not start."; } const pageMutated = pageHash() !== before; - // Smart-auto retry hint: only set when synthetic produced no observable change AND the + // Smart-auto retry hint: only set when DOM-event path produced no observable change AND the // element looks gated, OR the page just emitted a user-activation rejection. The dispatcher - // uses this to decide whether to retry with trusted mode. - let suggestTrusted = false; + // uses this to decide whether to retry with Chrome input. + let suggestChromeInput = false; let suggestReason; if (!pageMutated) { - if (autoplayHint) { suggestTrusted = true; suggestReason = "play/media affordance + idle media"; } + if (autoplayHint) { suggestChromeInput = true; suggestReason = "play/media affordance + idle media"; } else if (/copy(\s|$)|paste|share|download|fullscreen|sign in with|continue with|allow|enable/i.test(label)) { - suggestTrusted = true; suggestReason = `label '${labelRaw.slice(0, 40)}' looks gated`; + suggestChromeInput = true; suggestReason = `label '${labelRaw.slice(0, 40)}' looks gated`; } else { // Inspect recent console errors for activation-gate rejections. const recent = (state.console || []).slice(-8); const hit = recent.find((e) => /NotAllowedError|Document is not focused|requires transient activation|gesture is required/.test( (e.args || []).map((a) => typeof a === "string" ? a : (a && a.message) || JSON.stringify(a)).join(" ") )); - if (hit) { suggestTrusted = true; suggestReason = "recent console error indicates user-activation gate"; } + if (hit) { suggestChromeInput = true; suggestReason = "recent console error indicates user-activation gate"; } } } return { @@ -1537,13 +1515,13 @@ async function clickPage(selector, uid, x, y) { uid, tag: point.element.tagName, label: labelRaw.slice(0, 80) || undefined, - isTrusted: false, + input: "dom", defaultPrevented, elementVisible: visible, occludedBy: occluded || undefined, pageMutated, autoplayHint, - suggestTrusted: suggestTrusted || undefined, + suggestChromeInput: suggestChromeInput || undefined, suggestReason, }; } @@ -1561,7 +1539,7 @@ async function hoverPage(selector, uid, x, y) { } // Small dwell so hover-intent handlers fire. await sleepPage(rand(80, 220)); - return { x: point.x, y: point.y, selector, uid, tag: point.element.tagName, defaultPrevented, isTrusted: false }; + return { x: point.x, y: point.y, selector, uid, tag: point.element.tagName, defaultPrevented, input: "dom" }; } async function dragPage(fromUid, fromSelector, fromX, fromY, toUid, toSelector, toX, toY, steps) { @@ -1627,7 +1605,7 @@ async function dragPage(fromUid, fromSelector, fromX, fromY, toUid, toSelector, to: { x: to.x, y: to.y }, steps: n, pageMutated: pageHash() !== before, - note: "Synthetic drag with HTML5 DragEvent + shared DataTransfer. isTrusted is still false.", + note: "DOM-event drag with HTML5 DragEvent + shared DataTransfer.", }; } @@ -1677,7 +1655,7 @@ async function scrollPage(selector, uid, deltaY, deltaX, steps) { deltaX: movedX, deltaY: movedY, steps: n, scrollTop: target.scrollTop, scrollLeft: target.scrollLeft, pageMutated: pageHash() !== before, - isTrusted: false, + input: "dom", }; } @@ -1800,18 +1778,18 @@ async function typeIntoPage(selector, uid, text, pressEnter) { const finalValue = "value" in element ? element.value : element.textContent; const valueMatches = "value" in element ? element.value.includes(text) : (element.textContent || "").includes(text); const pageMutated = pageHash() !== before; - // Smart-auto retry hint when typing didn't land at all (e.g., editor blocks synthetic input). - let suggestTrusted = false, suggestReason; + // Smart-auto retry hint when typing didn't land at all (e.g., editor blocks DOM-event input). + let suggestChromeInput = false, suggestReason; if (text.length > 0 && initialValue === finalValue) { - suggestTrusted = true; - suggestReason = "value did not change — editor likely rejects synthetic input"; + suggestChromeInput = true; + suggestReason = "value did not change — editor likely rejects DOM-event input"; } return { selector, uid, length: text.length, pressEnter, - isTrusted: false, + input: "dom", valueMatches, pageMutated, - suggestTrusted: suggestTrusted || undefined, + suggestChromeInput: suggestChromeInput || undefined, suggestReason, }; } @@ -1836,7 +1814,7 @@ function fillPage(selector, uid, text, submit) { if (submit) pressKeyInPage("Enter"); return { selector, uid, length: String(text).length, submit, - isTrusted: false, + input: "dom", valueMatches: "value" in element ? element.value === String(text) : undefined, pageMutated: pageHash() !== before, }; @@ -1890,7 +1868,7 @@ async function pressKeyInPage(key) { } return { key: normalized, - isTrusted: false, + input: "dom", defaultPrevented: down.defaultPrevented || up.defaultPrevented, pageMutated: pageHash() !== before, }; diff --git a/extensions/chrome-profile-bridge/index.ts b/extensions/chrome-profile-bridge/index.ts index 16320bf..6763ad6 100644 --- a/extensions/chrome-profile-bridge/index.ts +++ b/extensions/chrome-profile-bridge/index.ts @@ -1,6 +1,4 @@ import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"; -import { getSettingsListTheme } from "@earendil-works/pi-coding-agent"; -import { Container, type SettingItem, SettingsList, Text } from "@earendil-works/pi-tui"; import { Type } from "typebox"; import { existsSync, readFileSync, statSync } from "node:fs"; import { mkdir, writeFile } from "node:fs/promises"; @@ -114,7 +112,7 @@ function summarizeActionResult(result: unknown): string | undefined { parts.push(`occluded by <${o.tag ?? "?"}${o.id ? "#" + o.id : ""}>`); } if (r.valueMatches === false) parts.push("input value did not stick"); - if (r.autoplayHint) parts.push("autoplay-gated affordance — synthetic click may not start media"); + if (r.autoplayHint) parts.push("autoplay-gated affordance"); return parts.length ? parts.join("; ") : undefined; } @@ -127,12 +125,32 @@ function readRequestBody(request: IncomingMessage): Promise<string> { }); } +function corsHeadersFor(request: IncomingMessage): Record<string, string> { + const origin = String(request.headers.origin ?? ""); + if (!origin.startsWith("chrome-extension://")) return {}; + return { + "access-control-allow-origin": origin, + "access-control-allow-methods": "GET,POST,OPTIONS", + "access-control-allow-headers": "content-type", + "access-control-expose-headers": "x-pi-chrome-version", + "vary": "origin", + }; +} + +function isBrowserOriginAllowed(request: IncomingMessage): boolean { + const origin = String(request.headers.origin ?? ""); + if (origin) return origin.startsWith("chrome-extension://"); + const secFetchSite = String(request.headers["sec-fetch-site"] ?? ""); + return !secFetchSite || secFetchSite === "none" || secFetchSite === "same-origin"; +} + +function isLocalProcessRequest(request: IncomingMessage): boolean { + return !request.headers.origin && !request.headers["sec-fetch-site"]; +} + function sendJson(response: ServerResponse, status: number, body: unknown, extraHeaders?: Record<string, string>): void { response.writeHead(status, { "content-type": "application/json; charset=utf-8", - "access-control-allow-origin": "*", - "access-control-allow-methods": "GET,POST,OPTIONS", - "access-control-allow-headers": "content-type", "cache-control": "no-store", ...(extraHeaders ?? {}), }); @@ -159,7 +177,7 @@ class ChromeProfileBridge { get connected(): boolean { // MV3 service workers can pause between polls/alarms. Treat a recent poll as - // connected without sending a synthetic command; real chrome_* tool calls are + // connected without sending a probe command; real chrome_* tool calls are // the authoritative end-to-end health check. return this.lastSeenAt !== undefined && Date.now() - this.lastSeenAt < 5 * 60_000; } @@ -278,16 +296,25 @@ class ChromeProfileBridge { } private async handle(request: IncomingMessage, response: ServerResponse): Promise<void> { + const url = new URL(request.url ?? "/", this.url); + const corsHeaders = corsHeadersFor(request); if (request.method === "OPTIONS") { - sendJson(response, 200, { ok: true }); + if (!isBrowserOriginAllowed(request)) { + sendJson(response, 403, { ok: false, error: "browser origin not allowed" }); + return; + } + sendJson(response, 200, { ok: true }, corsHeaders); return; } - const url = new URL(request.url ?? "/", this.url); if (request.method === "GET" && url.pathname === "/status") { sendJson(response, 200, this.status()); return; } if (request.method === "POST" && url.pathname === "/command") { + if (!isLocalProcessRequest(request)) { + sendJson(response, 403, { ok: false, error: "Chrome commands are accepted only from local Pi processes" }); + return; + } const body = JSON.parse(await readRequestBody(request)) as { action?: string; params?: Record<string, unknown>; @@ -306,6 +333,10 @@ class ChromeProfileBridge { return; } if (request.method === "GET" && url.pathname === "/next") { + if (!isBrowserOriginAllowed(request)) { + sendJson(response, 403, { ok: false, error: "browser origin not allowed" }); + return; + } this.lastSeenAt = Date.now(); this.clientName = url.searchParams.get("name") ?? undefined; let aborted = false; @@ -334,23 +365,27 @@ class ChromeProfileBridge { command ? { type: "command", command, expectedExtensionVersion: currentVersion } : { type: "none", expectedExtensionVersion: currentVersion }, - { "x-pi-chrome-version": currentVersion }, + { ...corsHeaders, "x-pi-chrome-version": currentVersion }, ); return; } if (request.method === "POST" && url.pathname === "/result") { + if (!isBrowserOriginAllowed(request)) { + sendJson(response, 403, { ok: false, error: "browser origin not allowed" }); + return; + } this.lastSeenAt = Date.now(); const result = JSON.parse(await readRequestBody(request)) as BridgeResult; const pending = this.pending.get(result.id); if (!pending) { - sendJson(response, 404, { ok: false, error: "unknown command id" }); + sendJson(response, 404, { ok: false, error: "unknown command id" }, corsHeaders); return; } clearTimeout(pending.timer); this.pending.delete(result.id); if (result.ok) pending.resolve(result.result); else pending.reject(new Error(result.error ?? "Chrome extension command failed")); - sendJson(response, 200, { ok: true }); + sendJson(response, 200, { ok: true }, corsHeaders); return; } sendJson(response, 404, { error: "not found" }); @@ -399,6 +434,34 @@ export default function (pi: ExtensionAPI): void { const bridge = new ChromeProfileBridge(DEFAULT_HOST, DEFAULT_PORT); let backgroundDefault = false; + let chromeAuthorizedUntil: number | "indefinite" | undefined; + + const authSummary = (): string => { + if (chromeAuthorizedUntil === "indefinite") return "authorized indefinitely"; + if (typeof chromeAuthorizedUntil === "number") { + const remainingMs = chromeAuthorizedUntil - Date.now(); + if (remainingMs > 0) return `authorized for ~${Math.ceil(remainingMs / 60_000)}m`; + } + return "locked"; + }; + + const chromeControlAuthorized = (): boolean => { + if (chromeAuthorizedUntil === "indefinite") return true; + if (typeof chromeAuthorizedUntil === "number" && chromeAuthorizedUntil > Date.now()) return true; + chromeAuthorizedUntil = undefined; + return false; + }; + + const requireChromeControlAuthorized = (): void => { + if (!chromeControlAuthorized()) { + throw new Error("Chrome control locked. Ask the user to run /chrome authorize before using chrome_* tools."); + } + }; + + const authorizedBridgeSend = (action: string, params: Record<string, unknown>, timeoutMs = DEFAULT_TIMEOUT_MS): Promise<unknown> => { + requireChromeControlAuthorized(); + return bridge.send(action, params, timeoutMs); + }; // Translate the public `background` parameter (default false = visible/foreground) into the // service worker's wire-level `foreground` flag, accepting legacy `foreground` as a fallback. @@ -420,8 +483,8 @@ export default function (pi: ExtensionAPI): void { ctx.ui.setStatus("chrome", `Chrome bridge :${DEFAULT_PORT}`); ctx.ui.notify( status.mode === "client" - ? `pi-chrome connected (sharing the Chrome connection an earlier pi session opened).` - : `pi-chrome is ready and waiting for the Chrome companion to connect. Run /chrome onboard to install it.`, + ? `pi-chrome connected (sharing the Chrome connection an earlier pi session opened). Run /chrome authorize before using chrome_* tools.` + : `pi-chrome is ready and waiting for the Chrome companion to connect. Run /chrome onboard to install it, then /chrome authorize to allow chrome_* tools.`, "info", ); }); @@ -436,19 +499,20 @@ export default function (pi: ExtensionAPI): void { Chrome control is available through the chrome_* tools via a companion Chrome extension installed in the user's normal Chrome profile. Tools target the existing signed-in profile, no CDP, no throwaway profile. Capability model (important): -- Default input path is **synthetic DOM events** (\`isTrusted=false\`). Synthetic events drive React/Vue/Angular state fine, but they do NOT satisfy Chrome's user-activation gates: audio/video autoplay, clipboard write, file pickers, fullscreen, and Web Push prompts will NOT open from a synthetic chrome_click. -- **Trusted escape hatch**: chrome_click / chrome_type / chrome_key / chrome_fill / chrome_hover / chrome_drag / chrome_scroll all accept \`trusted: true\`, which dispatches through chrome.debugger / CDP. Trusted events are browser-trusted (\`isTrusted=true\`) and **bypass page CSP entirely** because they're injected at the input layer, not via JS. Default mode is \`auto\`: synthetic first, silent CDP retry only when the click looks gated. If a synthetic click/type produced no \`pageMutated\` or you got a CSP/eval error from chrome_evaluate, escalate to \`trusted: true\` yourself — don't ask the user. -- \`chrome_evaluate\` and \`chrome_snapshot\` run in MAIN world via the **Function constructor**, which requires \`'unsafe-eval'\` in the page CSP. Pages with strict CSP (e.g. github.com, many bank/SaaS apps) will throw \`EvalError: ... 'unsafe-eval' is not an allowed source of script\` and chrome_snapshot will return empty. On those pages, drive the page with \`chrome_screenshot\` (extension API, not gated by CSP) + \`chrome_click\`/\`chrome_type\`/\`chrome_key\` with \`trusted: true\` and viewport coordinates. \`chrome_navigate\`, \`chrome_screenshot\`, \`chrome_tab\`, and trusted input all keep working under any CSP. -- Tool results include \`pageMutated\`, \`defaultPrevented\`, \`elementVisible\`, \`occludedBy\`, and (for type/fill) \`valueMatches\`. If \`pageMutated\` is false after a click that should have changed something, the click likely didn't take effect — do NOT just retry the same way; either escalate to \`trusted: true\` or check the snapshot for occlusion. +- Interactive controls (click/type/fill/key/hover/drag/scroll/tap) use Chrome's real input layer via chrome.debugger / CDP. Events satisfy normal user-activation gates. +- Input bypasses page CSP because it is injected at browser input layer, not page JavaScript. Chrome may show the “Pi Chrome Connector started debugging this browser” banner while attached. +- \`chrome_evaluate\` and \`chrome_snapshot\` run in MAIN world via the **Function constructor**, which requires \`'unsafe-eval'\` in the page CSP. Pages with strict CSP (e.g. github.com, many bank/SaaS apps) will throw \`EvalError: ... 'unsafe-eval' is not an allowed source of script\` and chrome_snapshot will return empty. On those pages, drive the page with \`chrome_screenshot\` + viewport-coordinate \`chrome_click\`/\`chrome_type\`/\`chrome_key\`. \`chrome_navigate\`, \`chrome_screenshot\`, \`chrome_tab\`, and Chrome input all keep working under any CSP. +- Tool results include \`pageMutated\`, \`defaultPrevented\`, \`elementVisible\`, \`occludedBy\`, and (for type/fill) \`valueMatches\`. If an action result indicates no page change or occlusion, inspect current page state instead of repeating blindly. Usage rules: -1. \`chrome_snapshot\` before clicking/typing; pass \`uid\` over \`selector\`. -2. \`includeSnapshot=true\` on click/type/fill to verify in one round trip. -3. If \`chrome_evaluate\` returns null when you expected a value, the expression evaluated to null/undefined in the page; surface the value via \`JSON.stringify\` to confirm. -4. \`chrome_navigate\` supports an optional \`initScript\` that runs at document_start in MAIN world for the next navigation (good for seeding localStorage or stubbing Date.now). -5. By default chrome_* tools focus Chrome so the user can watch; pass \`background=true\` or run /chrome quiet to silence the whole session. -6. If you hit an autoplay/clipboard/file-picker gate, tell the user; this bridge cannot satisfy it. (Generic clicks/typing/CSP gates are fine — escalate to \`trusted: true\`.) -7. Run /chrome doctor when in doubt about connectivity or capabilities. +1. If a chrome_* tool says Chrome control is locked, ask the user to run \`/chrome authorize\` before retrying. +2. \`chrome_snapshot\` before clicking/typing; pass \`uid\` over \`selector\`. +3. \`includeSnapshot=true\` on click/type/fill to verify in one round trip. +4. If \`chrome_evaluate\` returns null when you expected a value, the expression evaluated to null/undefined in the page; surface the value via \`JSON.stringify\` to confirm. +5. \`chrome_navigate\` supports an optional \`initScript\` that runs at document_start in MAIN world for the next navigation (good for seeding localStorage or stubbing Date.now). +6. By default chrome_* tools focus Chrome so the user can watch; pass \`background=true\` or run /chrome background on for session-wide background execution. +7. If you hit a native file-picker or privileged browser prompt gate, tell the user; generic clicks/typing/CSP gates are handled by Chrome input. +8. Run /chrome doctor when in doubt about connectivity or capabilities. </chrome-profile-bridge>`; return { systemPrompt: event.systemPrompt + primer }; }); @@ -514,105 +578,12 @@ Usage rules: lines.push(`… Skipped the remaining checks until you reload the Chrome extension.`); } - // Real-input mode probe (plain English for the user). - if (extensionAlive && !versionMismatch) { - try { - const status = (await bridge.send("trusted.status", {}, 5_000)) as { - mode?: string; - attachedTabs?: number[]; - permissionGranted?: boolean; - }; - if (status.permissionGranted) { - const banner = status.attachedTabs && status.attachedTabs.length ? ` (‘Pi Chrome Connector started debugging this browser’ banner up on ${status.attachedTabs.length} tab(s))` : ""; - const note = - status.mode === "auto" - ? " Clicks/keys are quiet by default; if a site rejects a quiet click, pi-chrome retries it once with a real-looking click. The Chrome banner shows only when that retry happens." - : status.mode === "on" - ? " Every click and keystroke uses a real-looking event. The Chrome banner stays up on every tab pi-chrome touches." - : " All clicks are quiet, no banner. Some sites (sign-ins, copy buttons, file pickers, paywalls) may silently ignore them. Run /chrome clicks if a site isn’t responding."; - const label = status.mode === "auto" ? "auto (smart upgrade)" : status.mode === "on" ? "on (always real-looking)" : "off (always quiet)"; - lines.push(`✓ Click mode: ${label}${banner}.${note}`); - } else { - lines.push(`⚠ Can't send real-looking clicks yet — the companion extension is missing a permission. Open chrome://extensions, click reload on 'Pi Chrome Connector', and accept the new permission prompt.`); - } - } catch (error) { - lines.push(`⚠ Couldn't check click mode: ${(error as Error).message}`); - } - } - ctx.ui.notify(lines.join("\n"), "info"); }; - // Click realism handler. With no args, cycles auto → on → off → auto. Explicit args jump - // directly. 'status' prints the current mode without changing it. - const CLICKS_CYCLE = ["auto", "on", "off"] as const; - const CLICKS_DESC: Record<string, string> = { - auto: "Quiet by default; pi-chrome retries once with a real-looking click if a site rejects the quiet one. The Chrome banner appears only when that retry happens.", - off: "All clicks are quiet, no banner. Some sites (sign-ins, copy buttons, file pickers, paywalls) may silently ignore these clicks.", - on: "Every click and keystroke looks real to websites. Chrome shows a 'Pi Chrome Connector started debugging this browser' banner on every tab pi-chrome touches.", - }; - const CLICKS_LABEL: Record<string, string> = { - auto: "auto (smart upgrade)", - off: "off (always quiet)", - on: "on (always real-looking)", - }; - - const trustedHandler = async (ctx: ExtensionContext, args: string) => { - const rawArg = (args || "").trim().toLowerCase(); - - let status: { mode: string; attachedTabs: number[]; permissionGranted: boolean } | undefined; - try { - status = (await bridge.send("trusted.status", {}, 5_000)) as typeof status; - } catch (error) { - ctx.ui.notify(`Couldn't check current click mode: ${(error as Error).message}`, "warning"); - return; - } - if (!status) return; - - if (!status.permissionGranted) { - ctx.ui.notify( - "pi-chrome can't drive real-looking clicks yet — the companion extension is missing a permission. Open chrome://extensions, click reload on 'Pi Chrome Connector', and accept the new permission prompt that appears.", - "warning", - ); - return; - } - - const current = status.mode; - const attached = status.attachedTabs?.length ? ` (banner up on ${status.attachedTabs.length} tab(s))` : ""; - - if (rawArg === "status") { - ctx.ui.notify(`Click mode is ${CLICKS_LABEL[current] ?? current}${attached}. ${CLICKS_DESC[current] ?? ""}`, "info"); - return; - } - - // No argument = cycle to the next mode. - let target = rawArg; - if (!target) { - const idx = CLICKS_CYCLE.indexOf(current as typeof CLICKS_CYCLE[number]); - target = CLICKS_CYCLE[(idx + 1 + CLICKS_CYCLE.length) % CLICKS_CYCLE.length]; - } - - if (!["on", "off", "auto"].includes(target)) { - ctx.ui.notify(`Unknown click mode '${rawArg}'. Pick one of: auto | off | on | status.`, "warning"); - return; - } - - if (target === current) { - ctx.ui.notify(`Click mode is already ${CLICKS_LABEL[current] ?? current}.`, "info"); - return; - } - - try { - await bridge.send("trusted.mode", { mode: target }, 5_000); - ctx.ui.notify(`Click mode → ${CLICKS_LABEL[target] ?? target}. ${CLICKS_DESC[target] ?? ""}`, "info"); - } catch (error) { - ctx.ui.notify(`Couldn't switch click mode: ${(error as Error).message}`, "warning"); - } - }; - - // Quiet (Chrome focus) handler. No args = toggle. Explicit on/off/status. - const QUIET_DESC: Record<string, string> = { - on: "pi-chrome works in the background; Chrome won't pop up or steal focus.", + // Run-in-background (Chrome focus) handler. No args = toggle. Explicit on/off/status. + const BACKGROUND_DESC: Record<string, string> = { + on: "pi-chrome runs in the background; Chrome won't pop up or steal focus.", off: "Chrome pops to the front and switches tabs so you can watch what pi-chrome is doing.", }; @@ -621,7 +592,7 @@ Usage rules: const currentLabel = backgroundDefault ? "on" : "off"; if (arg === "status") { - ctx.ui.notify(`Quiet mode is ${currentLabel}. ${QUIET_DESC[currentLabel]}`, "info"); + ctx.ui.notify(`Run in background is ${currentLabel}. ${BACKGROUND_DESC[currentLabel]}`, "info"); return; } @@ -629,12 +600,47 @@ Usage rules: else if (arg === "off" || arg === "false" || arg === "0") backgroundDefault = false; else if (arg === "toggle" || arg === "") backgroundDefault = !backgroundDefault; else { - ctx.ui.notify(`Unknown quiet mode '${arg}'. Pick one of: on | off | toggle | status.`, "warning"); + ctx.ui.notify(`Unknown background setting '${arg}'. Pick one of: on | off | toggle | status.`, "warning"); return; } const nextLabel = backgroundDefault ? "on" : "off"; - ctx.ui.notify(`Quiet mode → ${nextLabel}. ${QUIET_DESC[nextLabel]}`, "info"); + ctx.ui.notify(`Run in background → ${nextLabel}. ${BACKGROUND_DESC[nextLabel]}`, "info"); + }; + + const authorizeFor = async (ctx: ExtensionContext, label: string, until: number | "indefinite") => { + const ok = await ctx.ui.confirm( + "Authorize pi-chrome control?", + `This Pi session will be allowed to inspect and control your existing Chrome profile for ${label}.\n\nChrome actions use your signed-in browser state and real input. Only approve if you trust the current agent/task.`, + ); + if (!ok) { + ctx.ui.notify("Chrome control remains locked.", "info"); + return; + } + chromeAuthorizedUntil = until; + ctx.ui.notify(`Chrome control authorized for ${label}.`, "info"); + }; + + const parseAuthorizeArg = (arg: string): { label: string; until: number | "indefinite" } | undefined => { + const normalized = arg.trim().toLowerCase() || "15m"; + if (normalized === "indefinite" || normalized === "forever") return { label: "indefinitely", until: "indefinite" }; + const minutes = normalized.endsWith("m") ? Number(normalized.slice(0, -1)) : Number(normalized); + if (!Number.isFinite(minutes) || minutes <= 0) return undefined; + return { label: `${minutes} minutes`, until: Date.now() + minutes * 60_000 }; + }; + + const authorizeHandler = async (ctx: ExtensionContext, args: string) => { + const grant = parseAuthorizeArg(args); + if (!grant) { + ctx.ui.notify("Unknown authorize duration. Use minutes (15m, 30m, 45) or indefinite.", "warning"); + return; + } + return authorizeFor(ctx, grant.label, grant.until); + }; + + const revokeHandler = (ctx: ExtensionContext) => { + chromeAuthorizedUntil = undefined; + ctx.ui.notify("Chrome control locked. Run /chrome authorize to allow chrome_* tools again.", "info"); }; const onboardHandler = async (ctx: ExtensionContext) => { @@ -672,12 +678,8 @@ Usage rules: } catch { parts.push(`✗ Chrome not responding`); } - try { - const t = (await bridge.send("trusted.status", {}, 3_000)) as { mode?: string; attachedTabs?: number[] }; - const banner = t.attachedTabs?.length ? `, banner on ${t.attachedTabs.length} tab(s)` : ""; - parts.push(`clicks: ${t.mode ?? "?"}${banner}`); - } catch {} - parts.push(`quiet: ${backgroundDefault ? "on" : "off"}`); + parts.push(`auth: ${authSummary()}`); + parts.push(`background: ${backgroundDefault ? "on" : "off"}`); return parts.join(" · "); }; @@ -685,84 +687,63 @@ Usage rules: ctx.ui.notify(await statusSummary(), "info"); }; - // Interactive dialog: each row is a setting whose value cycles with Space/Enter. Enter on - // the last value also saves; Esc / 'q' closes. The description below changes with the - // current value so users always see what the active setting means. - const openSettingsDialog = async (ctx: ExtensionContext): Promise<void> => { - // Read current click mode (might fail if extension permission missing). - let clicksMode: string = "auto"; - let permissionGranted = false; - try { - const t = (await bridge.send("trusted.status", {}, 5_000)) as { mode?: string; permissionGranted?: boolean }; - clicksMode = t.mode ?? "auto"; - permissionGranted = !!t.permissionGranted; - } catch {} - - const clicksItem: SettingItem = { - id: "clicks", - label: "Click realism", - currentValue: clicksMode, - values: ["auto", "on", "off"], - description: permissionGranted - ? (CLICKS_DESC[clicksMode] ?? "") - : "Real-looking clicks unavailable: reload the Chrome extension in chrome://extensions and accept the new permission prompt.", - }; - const quietItem: SettingItem = { - id: "quiet", - label: "Quiet mode", - currentValue: backgroundDefault ? "on" : "off", - values: ["on", "off"], - description: QUIET_DESC[backgroundDefault ? "on" : "off"] ?? "", - }; - const items: SettingItem[] = [clicksItem, quietItem]; - - await ctx.ui.custom<void>((_tui, theme, _kb, done) => { - const container = new Container(); - container.addChild(new Text(theme.fg("accent", theme.bold("pi-chrome settings")), 1, 1)); - container.addChild(new Text(theme.fg("muted", "\u2191\u2193 navigate · space/enter cycle · esc close"), 1, 0)); - - let list: SettingsList; - list = new SettingsList( - items, - Math.min(items.length + 2, 8), - getSettingsListTheme(), - (id, newValue) => { - if (id === "clicks") { - if (!permissionGranted) { - ctx.ui.notify("Click mode locked: reload the Chrome extension first.", "warning"); - // Revert by snapping back to the previous value. - list.updateValue("clicks", clicksItem.currentValue); - return; - } - // Mutate description so the help text matches the new value. - clicksItem.currentValue = newValue; - clicksItem.description = CLICKS_DESC[newValue] ?? ""; - list.invalidate(); - void bridge.send("trusted.mode", { mode: newValue }, 5_000).catch((err) => { - ctx.ui.notify(`Couldn't switch click mode: ${(err as Error).message}`, "warning"); - }); - } else if (id === "quiet") { - backgroundDefault = newValue === "on"; - quietItem.currentValue = newValue; - quietItem.description = QUIET_DESC[newValue] ?? ""; - list.invalidate(); - } - }, - () => done(undefined), - ); - container.addChild(list); + const openAuthorizeMenu = async (ctx: ExtensionContext): Promise<void> => { + while (true) { + const choice = await ctx.ui.select("Authorize Chrome control", [ + "15 minutes", + "30 minutes", + "Indefinite", + "Custom minutes", + ]); + if (!choice) return; + switch (choice) { + case "15 minutes": return authorizeHandler(ctx, "15m"); + case "30 minutes": return authorizeHandler(ctx, "30m"); + case "Indefinite": return authorizeHandler(ctx, "indefinite"); + case "Custom minutes": { + const value = await ctx.ui.input("Authorize for how many minutes?", "45"); + if (!value) continue; + return authorizeHandler(ctx, value); + } + } + } + }; - return { - render: (w) => container.render(w), - invalidate: () => container.invalidate(), - handleInput: (data: string) => list.handleInput(data), - }; - }); + const openBackgroundMenu = async (ctx: ExtensionContext): Promise<void> => { + const choice = await ctx.ui.select("Background / watch mode", [ + "Use Chrome in background", + "Use Chrome in foreground", + ]); + if (!choice) return; + switch (choice) { + case "Use Chrome in background": return backgroundHandler(ctx, "on"); + case "Use Chrome in foreground": return backgroundHandler(ctx, "off"); + } + }; + + const openCommandMenu = async (ctx: ExtensionContext): Promise<void> => { + while (true) { + const choice = await ctx.ui.select(`pi-chrome\n${await statusSummary()}`, [ + "Authorize Chrome control…", + "Lock Chrome control", + "Doctor / troubleshoot", + "Background / watch mode…", + "Install / onboard extension", + ]); + if (!choice) return; + switch (choice) { + case "Authorize Chrome control…": await openAuthorizeMenu(ctx); continue; + case "Lock Chrome control": return revokeHandler(ctx); + case "Doctor / troubleshoot": return doctorHandler(ctx); + case "Background / watch mode…": await openBackgroundMenu(ctx); continue; + case "Install / onboard extension": return onboardHandler(ctx); + } + } }; pi.registerCommand("chrome", { description: - "All pi-chrome controls in one place.\n /chrome status — one-line snapshot of connection + current modes.\n /chrome doctor — full health check.\n /chrome onboard — install the Chrome companion extension.\n /chrome clicks [auto|off|on|status] — how realistic should pi-chrome's clicks be.\n /chrome quiet [on|off|status|toggle] — whether Chrome pops to the front when pi-chrome acts.\nRun with no arguments for an interactive picker that shows current state.", + "All pi-chrome controls in one place.\n /chrome authorize [15m|30m|<minutes>|indefinite] — allow this Pi session to use chrome_* tools.\n /chrome revoke — lock Chrome control.\n /chrome status — one-line snapshot of connection, auth, and background setting.\n /chrome doctor — full health check.\n /chrome onboard — install the Chrome companion extension.\n /chrome background [on|off|status|toggle] — whether pi-chrome runs without focusing Chrome.\nRun with no arguments for an interactive picker that shows current state.", getArgumentCompletions: (prefix) => { const raw = prefix; const trimmedRight = raw.replace(/\s+$/, ""); @@ -779,25 +760,25 @@ Usage rules: let candidates: Item[] = []; if (path.length === 0) { candidates = [ - { fullValue: "status", label: "status", description: "One-line summary: connection + click mode + quiet mode." }, + { fullValue: "authorize", label: "authorize", description: "Allow this Pi session to use chrome_* tools." }, + { fullValue: "revoke", label: "revoke", description: "Lock Chrome control for this Pi session." }, + { fullValue: "status", label: "status", description: "One-line summary: connection, auth, and background setting." }, { fullValue: "doctor", label: "doctor", description: "Full health check. Tells you if Chrome is connected and what's wrong if it isn't." }, { fullValue: "onboard", label: "onboard", description: "Install the Chrome companion extension (first-time setup)." }, - { fullValue: "clicks", label: "clicks", description: "How realistic should pi-chrome's clicks be? auto / off / on." }, - { fullValue: "quiet", label: "quiet", description: "Should Chrome pop to the front when pi-chrome acts, or work silently?" }, + { fullValue: "background", label: "background", description: "Run pi-chrome in the background without focusing Chrome?" }, ]; - } else if (path[0] === "clicks" && path.length === 1) { + } else if (path[0] === "authorize" && path.length === 1) { candidates = [ - { fullValue: "clicks auto", label: "auto", description: "Default. Quiet clicks; upgrade to real-looking ones only when a site rejects them." }, - { fullValue: "clicks off", label: "off", description: "Always quiet. No banner. Some sites won't accept the clicks." }, - { fullValue: "clicks on", label: "on", description: "Always real-looking. Chrome shows a banner. Best for stubborn sites." }, - { fullValue: "clicks status", label: "status", description: "Show the current click mode." }, + { fullValue: "authorize 15m", label: "15m", description: "Authorize Chrome control for 15 minutes." }, + { fullValue: "authorize 30m", label: "30m", description: "Authorize Chrome control for 30 minutes." }, + { fullValue: "authorize indefinite", label: "indefinite", description: "Authorize Chrome control until revoked or Pi exits." }, ]; - } else if (path[0] === "quiet" && path.length === 1) { + } else if (path[0] === "background" && path.length === 1) { candidates = [ - { fullValue: "quiet on", label: "on", description: "Work silently. Chrome stays in the background. Your editor keeps focus." }, - { fullValue: "quiet off", label: "off", description: "Bring Chrome to the front so you can watch (default)." }, - { fullValue: "quiet toggle", label: "toggle", description: "Flip whichever way it's currently set." }, - { fullValue: "quiet status", label: "status", description: "Show the current setting." }, + { fullValue: "background on", label: "on", description: "Run in background. Chrome stays in the background. Your editor keeps focus." }, + { fullValue: "background off", label: "off", description: "Bring Chrome to the front so you can watch (default)." }, + { fullValue: "background toggle", label: "toggle", description: "Flip whichever way it's currently set." }, + { fullValue: "background status", label: "status", description: "Show the current setting." }, ]; } if (candidates.length === 0) return null; @@ -808,31 +789,28 @@ Usage rules: handler: async (args, ctx) => { const tokens = (args || "").trim().split(/\s+/).filter(Boolean); if (tokens.length === 0) { - await openSettingsDialog(ctx); + await openCommandMenu(ctx); return; } const [head, ...rest] = tokens; const subArgs = rest.join(" "); switch (head) { + case "authorize": return authorizeHandler(ctx, subArgs); + case "revoke": return revokeHandler(ctx); case "status": return statusHandler(ctx); case "doctor": return doctorHandler(ctx); case "onboard": return onboardHandler(ctx); - case "clicks": - case "trusted": // legacy alias - return trustedHandler(ctx, subArgs); - case "quiet": - case "background": // legacy alias + case "background": return backgroundHandler(ctx, subArgs); case "settings": { - // Legacy nested form: /chrome settings background ... or /chrome settings trusted ... + // Legacy nested form: /chrome settings background ... const [setting, ...settingArgs] = rest; if (setting === "background") return backgroundHandler(ctx, settingArgs.join(" ")); - if (setting === "trusted") return trustedHandler(ctx, settingArgs.join(" ")); - ctx.ui.notify(`'/chrome settings' was removed. Use /chrome clicks or /chrome quiet directly.`, "warning"); + ctx.ui.notify(`'/chrome settings' was removed. Use /chrome background directly.`, "warning"); return; } default: - ctx.ui.notify(`Unknown subcommand '${head}'. Try: /chrome status | doctor | onboard | clicks | quiet.`, "warning"); + ctx.ui.notify(`Unknown subcommand '${head}'. Try: /chrome authorize | revoke | status | doctor | onboard | background.`, "warning"); } }, }); @@ -844,7 +822,7 @@ Usage rules: "Start/check the local bridge used by the companion Chrome extension. This does not launch a separate Chrome profile; install the unpacked Chrome extension in your existing Chrome profile to connect.", promptSnippet: "Show instructions for connecting Pi to the user's existing Chrome profile via the companion extension.", parameters: Type.Object({ - port: Type.Optional(Type.Number({ description: "Ignored unless PI_CHROME_BRIDGE_PORT is set before Pi starts." })), + port: Type.Optional(Type.Number({ description: "Ignored. The bundled Chrome extension polls 127.0.0.1:17318." })), url: Type.Optional(Type.String({ description: "Optional URL to open in the existing Chrome profile after the extension is connected." })), userDataDir: Type.Optional(Type.String({ description: "Ignored. This bridge intentionally uses the user's existing Chrome profile through the companion extension." })), useDefaultProfile: Type.Optional(Type.Boolean({ description: "Ignored; existing-profile access comes from the companion Chrome extension." })), @@ -852,7 +830,7 @@ Usage rules: }), async execute(_id, params, _signal, _onUpdate, ctx): Promise<ToolTextResult> { if (params.url && bridge.connected) { - const result = await bridge.send("tab.new", { url: params.url }, DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("tab.new", { url: params.url }, DEFAULT_TIMEOUT_MS); return { content: [{ type: "text", text: `Chrome bridge connected; opened ${params.url}` }], details: { status: bridge.status(), result } }; } return { @@ -887,7 +865,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send(`tab.${params.action}`, params, DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend(`tab.${params.action}`, params, DEFAULT_TIMEOUT_MS); if (params.action === "list") { const tabs = result as Array<{ id: number; title: string; url: string; active: boolean; windowId: number }>; const text = tabs.map((tab) => `${tab.id}\t${tab.active ? "*" : " "}\t${tab.title || "(untitled)"}\t${tab.url}`).join("\n") || "No tabs."; @@ -918,7 +896,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const snapshot = await bridge.send( + const snapshot = await authorizedBridgeSend( "page.snapshot", withBackground({ ...params, maxElements: params.maxElements ?? MAX_ELEMENTS }), DEFAULT_TIMEOUT_MS, @@ -948,7 +926,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.navigate", withBackground(params), (params.timeoutMs ?? 15_000) + 2_000); + const result = await authorizedBridgeSend("page.navigate", withBackground(params), (params.timeoutMs ?? 15_000) + 2_000); return { content: [{ type: "text", text: `Navigated to ${params.url}${params.initScript ? " (with initScript)" : ""}` }], details: { result: result as Json } }; }, }); @@ -972,7 +950,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const value = await bridge.send("page.evaluate", withBackground(params), DEFAULT_TIMEOUT_MS); + const value = await authorizedBridgeSend("page.evaluate", withBackground(params), DEFAULT_TIMEOUT_MS); const text = value === undefined ? "undefined" : typeof value === "string" @@ -986,7 +964,7 @@ Usage rules: name: "chrome_click", label: "Chrome Click", description: - "Click a snapshot uid, CSS selector, or viewport coordinate. Default 'auto' mode runs synthetic DOM events first and silently retries with trusted CDP only when the click looks gated (no page change + affordance label matches play/copy/share/sign-in/etc, or a recent NotAllowedError). The 'started debugging' banner appears only when the retry actually happens. Pass trusted=true to force CDP for this call (banner appears immediately). Pass trusted=false to skip retry. Pass includeSnapshot=true to return a fresh snapshot after the click.", + "Click a snapshot uid, CSS selector, or viewport coordinate using Chrome's real input layer. Pass includeSnapshot=true to return a fresh snapshot after the click.", promptSnippet: "Click page elements in Chrome by snapshot uid, selector, or viewport coordinate.", parameters: Type.Object({ uid: Type.Optional(Type.String({ description: "Stable element uid from chrome_snapshot. Prefer uid over selector after taking a snapshot." })), @@ -1001,12 +979,11 @@ Usage rules: background: Type.Optional( Type.Boolean({ description: "If true, click silently without focusing Chrome. Default false." }), ), - trusted: Type.Optional(Type.Boolean({ description: "If true, dispatch through chrome.debugger / CDP so the event is browser-trusted (isTrusted=true, user-activation satisfied). Triggers Chrome's 'started debugging this browser' banner." })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const raw = await bridge.send("page.click", withBackground(params), DEFAULT_TIMEOUT_MS); + const raw = await authorizedBridgeSend("page.click", withBackground(params), DEFAULT_TIMEOUT_MS); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const target = params.uid ?? params.selector ?? `${params.x},${params.y}`; @@ -1019,7 +996,7 @@ Usage rules: name: "chrome_type", label: "Chrome Type", description: - "Focus an optional snapshot uid or CSS selector, then type text. Default 'auto' mode runs synthetic per-character keydown/beforeinput/input/keyup first; if the input value doesn't change at all (editor rejected synthetic input) the call is silently retried through chrome.debugger so each keystroke is browser-trusted (isTrusted=true). Pass trusted=true to force CDP for this call. Pass trusted=false to skip retry. Pass includeSnapshot=true to return a fresh snapshot after typing.", + "Focus an optional snapshot uid or CSS selector, then type text using Chrome's real keyboard input. Pass includeSnapshot=true to return a fresh snapshot after typing.", promptSnippet: "Type text into Chrome, optionally focusing a snapshot uid or selector first.", parameters: Type.Object({ text: Type.String(), @@ -1034,12 +1011,11 @@ Usage rules: background: Type.Optional( Type.Boolean({ description: "If true, type silently without focusing Chrome. Default false." }), ), - trusted: Type.Optional(Type.Boolean({ description: "If true, dispatch through chrome.debugger / CDP so each keystroke is browser-trusted. Triggers Chrome's debugger banner." })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const raw = await bridge.send("page.type", withBackground(params), DEFAULT_TIMEOUT_MS); + const raw = await authorizedBridgeSend("page.type", withBackground(params), DEFAULT_TIMEOUT_MS); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const into = params.uid || params.selector ? ` into ${params.uid ?? params.selector}` : ""; @@ -1052,7 +1028,7 @@ Usage rules: name: "chrome_fill", label: "Chrome Fill", description: - "Set the full value of a text input, textarea, or contenteditable element using framework-aware native value setters and input/change events. Accepts a snapshot uid or CSS selector. Pass includeSnapshot=true to verify after filling.", + "Set the full value of a text input, textarea, or contenteditable element using Chrome click/select/delete/type input. Accepts a snapshot uid or CSS selector. Pass includeSnapshot=true to verify after filling.", promptSnippet: "Fill a Chrome form field by snapshot uid or selector, optionally returning a fresh snapshot.", parameters: Type.Object({ text: Type.String(), @@ -1067,12 +1043,11 @@ Usage rules: background: Type.Optional( Type.Boolean({ description: "If true, fill silently without focusing Chrome. Default false." }), ), - trusted: Type.Optional(Type.Boolean({ description: "If true, dispatch through chrome.debugger / CDP for browser-trusted input. Triggers Chrome's debugger banner." })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const raw = await bridge.send("page.fill", withBackground(params), DEFAULT_TIMEOUT_MS); + const raw = await authorizedBridgeSend("page.fill", withBackground(params), DEFAULT_TIMEOUT_MS); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const into = params.uid || params.selector ? ` into ${params.uid ?? params.selector}` : ""; @@ -1094,7 +1069,7 @@ Usage rules: ctrlKey: Type.Optional(Type.Boolean()), altKey: Type.Optional(Type.Boolean()), metaKey: Type.Optional(Type.Boolean()), - }, { description: "Modifier keys to hold while pressing the key (chord). Only honoured for trusted-mode presses; synthetic path ignores." })), + }, { description: "Modifier keys to hold while pressing the key (chord)." })), includeSnapshot: Type.Optional(Type.Boolean({ description: "If true, include a fresh chrome_snapshot result after the keypress." })), maxElements: Type.Optional(Type.Number({ default: MAX_ELEMENTS, description: "Max elements in the included snapshot." })), targetId: Type.Optional(Type.String()), @@ -1103,12 +1078,11 @@ Usage rules: background: Type.Optional( Type.Boolean({ description: "If true, send the key silently without focusing Chrome. Default false." }), ), - trusted: Type.Optional(Type.Boolean({ description: "If true, dispatch through chrome.debugger / CDP so the keystroke is browser-trusted." })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const raw = await bridge.send("page.key", withBackground(params), DEFAULT_TIMEOUT_MS); + const raw = await authorizedBridgeSend("page.key", withBackground(params), DEFAULT_TIMEOUT_MS); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const base = `Pressed ${params.key}.`; @@ -1133,7 +1107,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.waitFor", params, (params.timeoutMs ?? 10_000) + 2_000); + const result = await authorizedBridgeSend("page.waitFor", params, (params.timeoutMs ?? 10_000) + 2_000); return { content: [{ type: "text", text: `Observed ${params.kind}: ${params.value}` }], details: { result: result as Json } }; }, }); @@ -1154,7 +1128,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.console.list", withBackground(params), DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("page.console.list", withBackground(params), DEFAULT_TIMEOUT_MS); return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; }, }); @@ -1176,7 +1150,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.network.list", withBackground(params), DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("page.network.list", withBackground(params), DEFAULT_TIMEOUT_MS); return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; }, }); @@ -1196,7 +1170,7 @@ Usage rules: port: Type.Optional(Type.Number()), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.network.get", withBackground(params), DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("page.network.get", withBackground(params), DEFAULT_TIMEOUT_MS); return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; }, }); @@ -1226,7 +1200,7 @@ Usage rules: const cwd = workspaceCwd(ctx); const defaultPath = join(cwd, ".pi", "chrome-screenshots", `${new Date().toISOString().replace(/[:.]/g, "-")}.${format}`); const outputPath = params.path ? resolve(cwd, params.path) : defaultPath; - const result = (await bridge.send("page.screenshot", withBackground(params), params.fullPage ? 120_000 : DEFAULT_TIMEOUT_MS)) as { + const result = (await authorizedBridgeSend("page.screenshot", withBackground(params), params.fullPage ? 120_000 : DEFAULT_TIMEOUT_MS)) as { dataUrl?: string; tab?: unknown; fullPage?: boolean; @@ -1263,7 +1237,7 @@ Usage rules: pi.registerTool({ name: "chrome_hover", label: "Chrome Hover", - description: "Hover over an element (synthetic pointerover/mouseover/pointermove) by uid, selector, or x/y. Triggers CSS :hover state and any JS hover handlers; isTrusted is false.", + description: "Hover over an element by uid, selector, or x/y using Chrome pointer movement.", promptSnippet: "Hover a Chrome element to trigger :hover / mouseover handlers.", parameters: Type.Object({ uid: Type.Optional(Type.String()), @@ -1274,10 +1248,9 @@ Usage rules: urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), background: Type.Optional(Type.Boolean()), - trusted: Type.Optional(Type.Boolean({ description: "If true, dispatch through chrome.debugger / CDP for browser-trusted hover." })), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.hover", withBackground(params), DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("page.hover", withBackground(params), DEFAULT_TIMEOUT_MS); return { content: [{ type: "text", text: `Hovered ${params.uid ?? params.selector ?? `${params.x},${params.y}`}` }], details: { result: result as Json } }; }, }); @@ -1285,7 +1258,7 @@ Usage rules: pi.registerTool({ name: "chrome_drag", label: "Chrome Drag", - description: "Synthetic drag from one uid/selector/point to another. Dispatches pointerdown → humanised pointermove path → dragstart/drag/dragenter/dragover/dragleave/drop/dragend with a shared HTML5 DataTransfer, then pointerup. isTrusted=false.", + description: "Drag from one uid/selector/point to another using Chrome pointer input.", promptSnippet: "Drag a Chrome element from one point to another.", parameters: Type.Object({ fromUid: Type.Optional(Type.String()), @@ -1301,10 +1274,9 @@ Usage rules: urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), background: Type.Optional(Type.Boolean()), - trusted: Type.Optional(Type.Boolean({ description: "If true, dispatch through chrome.debugger / CDP so the drag is browser-trusted (real HTML5 dragstart/drop with native DataTransfer)." })), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.drag", withBackground(params), DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("page.drag", withBackground(params), DEFAULT_TIMEOUT_MS); return { content: [{ type: "text", text: `Dragged from ${params.fromUid ?? params.fromSelector} to ${params.toUid ?? params.toSelector}` }], details: { result: result as Json } }; }, }); @@ -1313,7 +1285,7 @@ Usage rules: name: "chrome_tap", label: "Chrome Tap (Touch)", description: - "Dispatch a real browser-trusted touchstart/touchend tap via chrome.debugger (CDP Input.dispatchTouchEvent). Use for sites that gate on TouchEvent rather than MouseEvent (mobile-first PWAs, swipe carousels). Always uses the trusted CDP path — the 'started debugging' banner appears.", + "Dispatch a real touchstart/touchend tap through Chrome's input layer. Use for sites that gate on TouchEvent rather than MouseEvent (mobile-first PWAs, swipe carousels). Chrome may show its debugging banner while attached.", promptSnippet: "Tap (real touch) a Chrome element by snapshot uid, selector, or coordinate.", parameters: Type.Object({ uid: Type.Optional(Type.String()), @@ -1326,7 +1298,7 @@ Usage rules: background: Type.Optional(Type.Boolean()), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.tap", withBackground(params), DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("page.tap", withBackground(params), DEFAULT_TIMEOUT_MS); const target = params.uid ?? params.selector ?? `${params.x},${params.y}`; return { content: [{ type: "text", text: `Tapped ${target} (touch)` }], details: { result: result as Json } }; }, @@ -1347,10 +1319,9 @@ Usage rules: urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), background: Type.Optional(Type.Boolean()), - trusted: Type.Optional(Type.Boolean({ description: "If true, dispatch wheel events through chrome.debugger / CDP for browser-trusted scrolling." })), }), async execute(_id, params): Promise<ToolTextResult> { - const result = await bridge.send("page.scroll", withBackground(params), DEFAULT_TIMEOUT_MS); + const result = await authorizedBridgeSend("page.scroll", withBackground(params), DEFAULT_TIMEOUT_MS); return { content: [{ type: "text", text: `Scrolled dy=${params.deltaY ?? 0} dx=${params.deltaX ?? 0}` }], details: { result: result as Json } }; }, }); @@ -1358,7 +1329,7 @@ Usage rules: pi.registerTool({ name: "chrome_upload_file", label: "Chrome Upload File", - description: "Programmatically set the files of an <input type=file> element from local file paths. Uses DataTransfer to populate input.files and dispatches input+change events. Does NOT open the native file picker; works with React/Vue/Angular controlled inputs.", + description: "Attach local files to an <input type=file> element using Chrome DevTools file-input control. Does NOT open the native file picker; works with React/Vue/Angular controlled inputs.", promptSnippet: "Attach local files to a Chrome <input type=file> without opening the native file picker.", parameters: Type.Object({ uid: Type.Optional(Type.String()), @@ -1370,17 +1341,10 @@ Usage rules: background: Type.Optional(Type.Boolean()), }), async execute(_id, params, _signal, _onUpdate, ctx): Promise<ToolTextResult> { - const { readFile } = await import("node:fs/promises"); - const { basename } = await import("node:path"); const cwd = workspaceCwd(ctx); - const files: Array<{ name: string; type: string; base64: string }> = []; - for (const p of params.paths) { - const abs = resolve(cwd, p); - const buf = await readFile(abs); - files.push({ name: basename(abs), type: "application/octet-stream", base64: buf.toString("base64") }); - } - const result = await bridge.send("page.upload", withBackground({ ...params, files }), DEFAULT_TIMEOUT_MS); - return { content: [{ type: "text", text: `Uploaded ${files.length} file(s) to ${params.uid ?? params.selector}` }], details: { result: result as Json } }; + const paths = params.paths.map((p) => resolve(cwd, p)); + const result = await authorizedBridgeSend("page.upload", withBackground({ ...params, paths }), DEFAULT_TIMEOUT_MS); + return { content: [{ type: "text", text: `Uploaded ${paths.length} file(s) to ${params.uid ?? params.selector}` }], details: { result: result as Json } }; }, }); } diff --git a/package.json b/package.json index 37935a6..0d55105 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "pi-chrome", - "version": "0.15.2", + "version": "0.15.15", "scripts": { "version": "node scripts/sync-manifest-version.js", "prepublishOnly": "node scripts/sync-manifest-version.js" @@ -36,7 +36,10 @@ "stagehand-alternative" ], "license": "MIT", - "author": { "name": "tianrendong", "company": "Earendil Inc." }, + "author": { + "name": "tianrendong", + "company": "Earendil Inc." + }, "homepage": "https://github.com/tianrendong/pi-chrome#readme", "repository": { "type": "git", |
