diff options
| author | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-14 22:46:31 -0400 |
|---|---|---|
| committer | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-14 22:46:31 -0400 |
| commit | 58e86b70a0c7f6bf8e44f8b658e8194444ade943 (patch) | |
| tree | b56f638493c4967544b39c465596dddac1d1b4a9 | |
| parent | 1ec189f8f7f137555e8ae90d6753ee226eefa457 (diff) | |
Audit pi-chrome docs
| -rw-r--r-- | CHANGELOG.md | 5 | ||||
| -rw-r--r-- | CONTRIBUTING.md | 6 | ||||
| -rw-r--r-- | README.md | 24 | ||||
| -rw-r--r-- | docs/COMPARISON.md | 6 | ||||
| -rw-r--r-- | docs/EXAMPLES.md | 8 | ||||
| -rw-r--r-- | docs/FAQ.md | 21 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/browser-extension/manifest.json | 2 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/browser-extension/service_worker.js | 12 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/index.ts | 11 | ||||
| -rw-r--r-- | package.json | 4 |
10 files changed, 45 insertions, 54 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index 719b171..4c60542 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable user-facing changes to `pi-chrome`. +## 0.15.17 — 2026-05-14 + +- **Docs accuracy pass.** Updated README, FAQ, comparison, contributing notes, and package metadata for the current real-input-only, terminal-authorized tool surface. +- **Input verification fix.** `includeSnapshot=true` now works for `chrome_click`, `chrome_type`, `chrome_fill`, and `chrome_key`, returning the Chrome-input result plus a fresh snapshot. + ## 0.15.16 — 2026-05-14 - **Visible `/chrome` loading state.** Bare `/chrome` and `/chrome status` now immediately say “Checking Chrome connection…” before probing the companion extension, so a slow Chrome bridge no longer looks like the command did nothing. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c16b5fe..3b8ff0f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,8 +5,8 @@ Thanks for considering a contribution. pi-chrome aims to be the **de-facto brows ## Non-negotiables 1. **No re-login.** Every change must keep working against the user's already-signed-in Chrome profile. Anything that requires a fresh profile or extra auth steps is out of scope. -2. **Honest result envelopes.** Every action tool returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy` (when relevant), `valueMatches` (for input). Agents need to know **why** something didn't take effect. -3. **Quiet by default, trusted by opt-in.** Synthetic DOM events first. CDP/`chrome.debugger` only when explicitly requested (`trusted: true`) or when the smart-auto heuristic detects an obvious user-activation gate. +2. **Verifiable action results.** Input tools must return structured details and support `includeSnapshot` where verification matters. Agents need enough evidence to avoid blind retries. +3. **Chrome real input.** Interactive controls use Chrome's input layer through `chrome.debugger`; do not re-expose synthetic/untrusted input as public UX. 4. **Benchmarks gate features.** Add a page in `test-suite/` that fails before your change and passes after. We accept PRs faster when there's a green/red verdict to point at. ## Local dev @@ -25,7 +25,7 @@ python3 -m http.server 8765 1. Register in `extensions/chrome-profile-bridge/index.ts` (the `register*Tool` calls near line 840+). 2. Implement the handler in `extensions/chrome-profile-bridge/browser-extension/service_worker.js`. -3. Return a `pageMutated` + relevant fields. +3. Return structured details and support `includeSnapshot` for user-visible state changes when relevant. 4. Add a benchmark page under `test-suite/challenges/` and a manifest entry. 5. Update `README.md` "What an agent gets" table. 6. Add a `CHANGELOG.md` entry. @@ -1,7 +1,7 @@ # pi-chrome > **The fastest way to give a [Pi](https://pi.dev) agent your real Chrome.** -> No CDP. No throwaway profile. No re-login. Watch it work — or run silent. +> No remote-debug port. No throwaway profile. No re-login. Watch it work — or run silent. **MIT · 0 runtime deps · loopback-only bridge (`127.0.0.1:17318`) · inspect [`extensions/chrome-profile-bridge/browser-extension/`](./extensions/chrome-profile-bridge/browser-extension) before loading.** Verify connectivity in one command: `/chrome doctor`. @@ -12,7 +12,7 @@ Agent: chrome_tab(list) → chrome_snapshot(uid:…) → chrome_screenshot(...) You: [keeps coding — agent never asked you to log in] ``` -`pi-chrome` ships **20+ browser tools** for Pi agents, backed by a small MIT-licensed Chrome extension that runs inside the Chrome profile **you already use** — including every site you're already signed into. +`pi-chrome` ships **19 browser tools** for Pi agents, backed by a small MIT-licensed Chrome extension that runs inside the Chrome profile **you already use** — including every site you're already signed into. --- @@ -30,7 +30,7 @@ Then in Pi: On macOS this opens `chrome://extensions`, reveals the bundled `browser-extension/` folder in Finder, and copies its path to your clipboard. In Chrome: **Developer mode** → **Load unpacked** → paste the path. Done. -Verify, then authorize current Pi session in Pi: +Verify, then authorize current Pi session from the terminal: ```text /chrome doctor @@ -120,27 +120,23 @@ You: [files the ticket with the folder attached] --- -## Honest results +## Verifiable actions -Most browser-automation libraries return `void` or a generic ack. `pi-chrome` returns a structured envelope on every interaction: +Input tools return structured details such as the coordinates used, target tag, uploaded paths, key pressed, or scroll distance. For click/type/fill/key calls, pass `includeSnapshot: true` to get a fresh page snapshot in the same result: ```text -chrome_click(occluded-button) → - "Clicked el-3 — pageMutated=false; occluded by <div#overlay>" +chrome_click(uid:"el-3", includeSnapshot:true) → + result: { input:"chrome", x:412, y:238, tag:"BUTTON" } + snapshot: { title, url, text, elements:[...] } ``` -```text -chrome_type(react-input, "hello") → - "Typed into el-7 — valueMatches=true; pageMutated=true" -``` - -This is why agents using pi-chrome don't get stuck in retry loops on broken sites. They get the **reason** the action didn't land and can fix course in one turn. +Agents can verify page state immediately instead of blindly retrying. --- ## What an agent gets -**20 tools**, grouped by job. Every one runs against your already-open tabs. +**19 tools**, grouped by job. Every one runs against your already-open tabs. | Category | Tools | | --------------- | ---------------------------------------------------------------------------------------------- | diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md index d30cb6a..7dfe5ed 100644 --- a/docs/COMPARISON.md +++ b/docs/COMPARISON.md @@ -52,7 +52,7 @@ We benchmark in public — see [`../test-suite/`](../test-suite). Where exact sc 1. **Profile attach, not driver launch.** Every other driver fights cookie persistence, login walls, MFA, and extension state. pi-chrome inherits all of it because it *is* your Chrome. 2. **Chrome input against your real profile.** Interactive tools use CDP input for reliability while still controlling the Chrome profile you already use. 3. **Extension bridge transport.** No `--remote-debugging-port`, no throwaway Chromium. Survives Chrome auto-updates. Works alongside your normal Chrome usage. -4. **Honest result envelopes.** Every action returns `pageMutated`, `defaultPrevented`, `elementVisible`, `occludedBy`, `valueMatches`. Competitors return `void` or generic acks; agents loop blindly on broken clicks. +4. **Structured action results.** Input tools return target coordinates/tags and can include a fresh snapshot (`includeSnapshot`) so agents can verify state instead of blindly retrying. 5. **Multi-session shared bridge.** Planner + worker + audit Pi sessions all drive the same Chrome concurrently. 6. **Stable element uids.** `chrome_snapshot` returns deterministic uids you can pass to subsequent actions — similar to BrowserGym's `bid`, but built into the snapshot tool itself. @@ -101,9 +101,9 @@ These wrap a driver with an LLM loop. They are **higher-level than pi-chrome** a `pi-chrome` exposes tools that any Pi agent can call. If you want to use it from outside Pi: -1. The local bridge speaks HTTP JSON-RPC at `127.0.0.1:17318` (default). The API is internal but stable across patch versions. +1. The local bridge speaks HTTP JSON over `127.0.0.1:17318` (default). The API is internal; use the Pi tool surface unless you are building an adapter. 2. Tool surface mirrors Playwright closely (click/type/navigate/snapshot/screenshot/evaluate/wait_for) so adapter code is short. -3. Honest envelopes (`pageMutated`, `valueMatches`, `occludedBy`) let agent harnesses skip retry/heal logic. +3. `includeSnapshot` on input tools lets agent harnesses verify state after actions. If you want a first-class pi-chrome adapter for Browser Use / Stagehand / LangGraph, file an issue with your use case. diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md index a5cccf4..b3c6286 100644 --- a/docs/EXAMPLES.md +++ b/docs/EXAMPLES.md @@ -119,10 +119,8 @@ On my staging app: ### React controlled inputs ```text -chrome_fill (not chrome_type) for React inputs — it uses the -framework-aware native value setter so the form's state actually updates. -After each fill, the result envelope's valueMatches=true confirms the -component re-rendered with the new value. +Use `chrome_fill` for React inputs when you want to replace the full value. +Pass `includeSnapshot=true` to verify the component re-rendered with the new value. ``` ### File upload without the native picker @@ -160,7 +158,7 @@ Interactive tools use Chrome's real input layer by default: clicks, typing, fill - sign-in flows - guarded buttons - audio/video controls -- fullscreen / permission prompts +- fullscreen and other user-activation checks - pages with strict CSP or user-activation checks Chrome may show its debugger banner while pi-chrome is attached. diff --git a/docs/FAQ.md b/docs/FAQ.md index 33902d3..f7f871f 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -32,9 +32,9 @@ Chrome control is also locked per Pi session until you run `/chrome authorize`; Yes. The first session opens the local bridge; later sessions detect it and pipe their commands through the same bridge. Each Pi session must be authorized with `/chrome authorize` before its chrome_* tools work. -## Why can't this be on the Chrome Web Store? +## Why ship as an unpacked extension? -Web Store extensions cannot communicate with a local process bridge controlled by another tool — Google's policy. pi-chrome must ship as an unpacked extension you load yourself. The upside: you can read the source. The downside: each Chrome update may prompt you to re-confirm. +pi-chrome ships as an unpacked extension so the source and broad browser permissions are easy to inspect and update with the npm package. The downside: you load it manually from `chrome://extensions` and reload it after package updates. ## What happens when I update pi-chrome? @@ -42,8 +42,8 @@ Web Store extensions cannot communicate with a local process bridge controlled b ## What's the install footprint? -- Pi side: one extension that registers ~20 tools and a few slash commands. -- Chrome side: one unpacked extension, ~5000 LOC of plain JavaScript, no dependencies. +- Pi side: one extension that registers 19 tools and a few slash commands. +- Chrome side: one unpacked extension, ~2000 LOC of plain JavaScript, no dependencies. ## Can I script it without Pi? @@ -53,18 +53,11 @@ The Pi-facing tools are thin wrappers around an HTTP bridge at `127.0.0.1:17318` Not always. `chrome_evaluate` and `chrome_snapshot` run in the page's MAIN world through the Function constructor, so pages whose CSP blocks `'unsafe-eval'` can reject them. `chrome_screenshot`, `chrome_navigate`, tab tools, and real Chrome input still work because they use extension/browser APIs rather than page JavaScript. -## Why does my click return `pageMutated=false`? +## How do I tell whether a click or type worked? -Either: -- The element was occluded (look for `occludedBy: <selector>` in the envelope). -- The click handler called `event.preventDefault()` and the page intentionally ignored it. -- The target changed after your snapshot; take a fresh snapshot or screenshot. +Use `includeSnapshot=true` on `chrome_click`, `chrome_type`, `chrome_fill`, or `chrome_key`. The tool returns the Chrome-input result plus a fresh snapshot, so the agent can verify text, URL, visible elements, or form values before continuing. -The result envelope tells you which one. **Don't blind-retry.** - -## Why does `chrome_type` return `valueMatches=false`? - -The field rejected or transformed the typed value. Common culprits: contenteditable rich-text editors, native date pickers, masked-input libraries, or masks. Try `chrome_fill`, then verify with `includeSnapshot=true`. +If the page did not change, take a fresh snapshot or screenshot and check for overlays, disabled controls, stale element uids, or app-side validation. ## How do I attach a file to a React file input? diff --git a/extensions/chrome-profile-bridge/browser-extension/manifest.json b/extensions/chrome-profile-bridge/browser-extension/manifest.json index 66f396e..76d4305 100644 --- a/extensions/chrome-profile-bridge/browser-extension/manifest.json +++ b/extensions/chrome-profile-bridge/browser-extension/manifest.json @@ -1,7 +1,7 @@ { "manifest_version": 3, "name": "Pi Chrome Connector", - "version": "0.15.16", + "version": "0.15.17", "description": "Lets Pi control tabs in Chrome via a local connector at 127.0.0.1.", "permissions": [ "tabs", diff --git a/extensions/chrome-profile-bridge/browser-extension/service_worker.js b/extensions/chrome-profile-bridge/browser-extension/service_worker.js index 08d712b..88391a7 100644 --- a/extensions/chrome-profile-bridge/browser-extension/service_worker.js +++ b/extensions/chrome-profile-bridge/browser-extension/service_worker.js @@ -712,7 +712,7 @@ async function dispatch(action, params) { case "page.evaluate": return evaluateInTab(params); case "page.click": - return chromeInputClick(params); + return withOptionalSnapshot(params, chromeInputClick); case "page.hover": return chromeInputHover(params); case "page.drag": @@ -720,11 +720,11 @@ async function dispatch(action, params) { case "page.upload": return chromeInputUpload(params); case "page.type": - return chromeInputType(params); + return withOptionalSnapshot(params, chromeInputType); case "page.fill": - return chromeInputFill(params); + return withOptionalSnapshot(params, chromeInputFill); case "page.key": - return chromeInputKey(params); + return withOptionalSnapshot(params, chromeInputKey); case "page.scroll": return chromeInputScroll(params); case "page.tap": @@ -932,8 +932,8 @@ async function evaluateInTab(params) { return v; } -async function executeActionInTab(params, func, args) { - const result = await executeInTab(params, func, args); +async function withOptionalSnapshot(params, actionFn) { + const result = await actionFn(params); if (params.includeSnapshot) { const snapshot = await executeInTab({ ...params, foreground: false }, snapshotPage, [params.maxElements || 80, null, null, null]); return { result, snapshot }; diff --git a/extensions/chrome-profile-bridge/index.ts b/extensions/chrome-profile-bridge/index.ts index 6f2739a..5c72da6 100644 --- a/extensions/chrome-profile-bridge/index.ts +++ b/extensions/chrome-profile-bridge/index.ts @@ -8,9 +8,8 @@ import { dirname, join, resolve } from "node:path"; /** * Existing-profile Chrome bridge for pi. * - * This is intentionally not a Chrome DevTools Protocol integration. CDP cannot attach to - * already-running normal Chrome windows and recent Chrome builds block default-profile - * remote debugging. Instead, install the companion Chrome extension from the + * This is intentionally not a remote-debugging-port integration. Chrome blocks default-profile + * remote debugging in many normal launches, so pi-chrome uses a companion extension from the * browser-extension folder bundled next to this Pi extension. * * The companion extension runs inside the user's real Chrome profile and polls this local @@ -496,18 +495,18 @@ export default function (pi: ExtensionAPI): void { pi.on("before_agent_start", (event) => { const primer = ` <chrome-profile-bridge> -Chrome control is available through the chrome_* tools via a companion Chrome extension installed in the user's normal Chrome profile. Tools target the existing signed-in profile, no CDP, no throwaway profile. +Chrome control is available through the chrome_* tools via a companion Chrome extension installed in the user's normal Chrome profile. Tools target the existing signed-in profile: no remote-debug port, no throwaway profile. Capability model (important): - Interactive controls (click/type/fill/key/hover/drag/scroll/tap) use Chrome's real input layer via chrome.debugger / CDP. Events satisfy normal user-activation gates. - Input bypasses page CSP because it is injected at browser input layer, not page JavaScript. Chrome may show the “Pi Chrome Connector started debugging this browser” banner while attached. - \`chrome_evaluate\` and \`chrome_snapshot\` run in MAIN world via the **Function constructor**, which requires \`'unsafe-eval'\` in the page CSP. Pages with strict CSP (e.g. github.com, many bank/SaaS apps) will throw \`EvalError: ... 'unsafe-eval' is not an allowed source of script\` and chrome_snapshot will return empty. On those pages, drive the page with \`chrome_screenshot\` + viewport-coordinate \`chrome_click\`/\`chrome_type\`/\`chrome_key\`. \`chrome_navigate\`, \`chrome_screenshot\`, \`chrome_tab\`, and Chrome input all keep working under any CSP. -- Tool results include \`pageMutated\`, \`defaultPrevented\`, \`elementVisible\`, \`occludedBy\`, and (for type/fill) \`valueMatches\`. If an action result indicates no page change or occlusion, inspect current page state instead of repeating blindly. +- Input tools return structured details and support \`includeSnapshot=true\` on click/type/fill/key. Use the fresh snapshot to verify state instead of repeating blindly. Usage rules: 1. If a chrome_* tool says Chrome control is locked, ask the user to run \`/chrome authorize\` before retrying. 2. \`chrome_snapshot\` before clicking/typing; pass \`uid\` over \`selector\`. -3. \`includeSnapshot=true\` on click/type/fill to verify in one round trip. +3. \`includeSnapshot=true\` on click/type/fill/key to verify in one round trip. 4. If \`chrome_evaluate\` returns null when you expected a value, the expression evaluated to null/undefined in the page; surface the value via \`JSON.stringify\` to confirm. 5. \`chrome_navigate\` supports an optional \`initScript\` that runs at document_start in MAIN world for the next navigation (good for seeding localStorage or stubbing Date.now). 6. By default chrome_* tools focus Chrome so the user can watch; pass \`background=true\` or run /chrome background on for session-wide background execution. diff --git a/package.json b/package.json index 594c9a5..faa04d6 100644 --- a/package.json +++ b/package.json @@ -1,11 +1,11 @@ { "name": "pi-chrome", - "version": "0.15.16", + "version": "0.15.17", "scripts": { "version": "node scripts/sync-manifest-version.js", "prepublishOnly": "node scripts/sync-manifest-version.js" }, - "description": "Give a Pi agent your real, signed-in Chrome. No CDP, no throwaway profile, no re-login. 20+ tools (click, type, navigate, screenshot, network capture, file upload, drag, touch) with honest result envelopes — and a built-in browser-control benchmark suite.", + "description": "Give a Pi agent your real, signed-in Chrome. No remote-debug port, no throwaway profile, no re-login. 19 tools for click, type, navigate, screenshot, network capture, file upload, drag, and touch.", "keywords": [ "pi", "pi-package", |
