diff options
| author | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-09-09 23:50:16 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2026-09-09 23:50:16 -0400 |
| commit | f746631041e15e0c71d464d830ca2df7603f1476 (patch) | |
| tree | 6e00b9d63a630a674b6a3bf6936bd85f82d4aade | |
| parent | 641ae30f7963ad962e162ff0dfbd80966b09a516 (diff) | |
| parent | c1f230eeb76aaa50e98d3fe7d838573612b6e36f (diff) | |
Merge pull request #13 from tianrendong/fix/hard-background-policy
Enforce hard background and release 0.15.48
| -rw-r--r-- | CHANGELOG.md | 8 | ||||
| -rw-r--r-- | README.md | 10 | ||||
| -rw-r--r-- | SECURITY.md | 2 | ||||
| -rw-r--r-- | docs/ARCHITECTURE.md | 20 | ||||
| -rw-r--r-- | docs/COMPARISON.md | 2 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/browser-extension/manifest.json | 2 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/browser-extension/service_worker.js | 144 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/index.ts | 195 | ||||
| -rw-r--r-- | package-lock.json | 4 | ||||
| -rw-r--r-- | package.json | 4 | ||||
| -rw-r--r-- | test-suite/README.md | 9 | ||||
| -rw-r--r-- | test-suite/challenges/43-hard-background.html | 42 | ||||
| -rw-r--r-- | test-suite/manifest.json | 75 | ||||
| -rw-r--r-- | test-suite/unit/automation-target.test.mjs | 2 | ||||
| -rw-r--r-- | test-suite/unit/background-policy.test.mjs | 469 |
15 files changed, 808 insertions, 180 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index cca1965..a5b8675 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,14 @@ All notable user-facing changes to `pi-chrome`. +## 0.15.48 — 2026-09-09 + +- **Existing background mode is now hard background.** `/chrome background on` (still the default) overrides per-call foreground requests, keeps new tabs inactive, and blocks `chrome_tab activate`. Use the existing `/chrome background off` for foreground/watch mode; no new command or lock state. +- **Screenshots without tab activation.** PNG/JPEG and full-page tiles use CDP instead of `captureVisibleTab`. Debugger/capture failures never fall back to switching tabs. Old companions reject background creation/capture with a reload instruction. Screenshot tools now require debugger access. +- **Trusted input preserved.** Background policy does not replace Chrome input with synthetic events. It blocks explicit focus/activation, not page/native/Chrome/OS side effects; inactive-page rendering and focus-gated workflows can still vary by environment. +- **Regression coverage.** Added policy/worker/screenshot unit tests and challenge 43 for inactive-tab visibility plus trusted input. Full-page capture restores both scroll axes best-effort after success or failure. +- **Live validation and unresolved limitation.** Chrome 152/macOS checks passed for inactive tab creation, blocked activation, background PNG/JPEG/full-page capture, and scroll restoration. The trusted-click check encountered debugger detachment, then a visibility failure on retry; the cause remains unresolved and human interference was not ruled out. This release does not promise zero focus changes during trusted input. Chrome-behind-another-app and macOS Spaces behavior remain unverified. + ## 0.15.47 — 2026-09-09 - **Bounded session cleanup.** On exit, Pi waits up to two seconds for cleanup before stopping the bridge. Reload preserves browser resources; revoke remains non-blocking. @@ -108,13 +108,21 @@ Security details: [`SECURITY.md`](./SECURITY.md). Architecture details: [`docs/A /chrome status # connection + auth + background state /chrome authorize [duration] /chrome revoke -/chrome background on # default: don't steal focus +/chrome background on # default: hard background policy /chrome background off # foreground/watch mode /chrome background status ``` If loaded extension is older than installed `pi-chrome`, `/chrome doctor` tells you to reload it from `chrome://extensions`. +### Background policy + +`/chrome background on` is enforced, not an overridable default. Per-call `background:false` cannot bring Chrome forward, new tabs stay inactive, and `chrome_tab activate` is blocked. Use the existing `/chrome background off` for foreground/watch mode; per-call `background:true` still works when that mode is off. + +Screenshots use CDP without activating background tabs. Debugger/capture failures return errors, never an activation fallback. Reload both Pi and the Chrome companion after upgrading; old companions reject background tab creation/screenshots rather than silently switching tabs. + +This prevents explicit pi-chrome focus/activation, not every Chrome/OS side effect. Trusted input, page popups, native prompts, debugger banners, and macOS Spaces can still affect focus. Inactive pages may throttle rendering or reject focus-gated actions. See [scope and risks](./docs/ARCHITECTURE.md#scope-and-risks). + --- ## Limits diff --git a/SECURITY.md b/SECURITY.md index ffc2c06..3d07518 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -27,7 +27,7 @@ The Chrome extension under `extensions/chrome-profile-bridge/browser-extension/` - Loopback bridge only. No remote port. No telemetry. - Chrome real input layer for interactive controls. - Chrome control locked by default; `/chrome authorize` unlocks current Pi session after terminal confirmation, `/chrome revoke` locks it again. -- Run-in-background optional; tab/window focus is observable by default (the user can see Pi acting). +- Hard background mode is on by default: tools cannot override it to explicitly focus windows or activate tabs. `/chrome background off` allows foreground/watch mode. This is not a security sandbox: trusted input, page scripts, native prompts, and Chrome/OS behavior can still affect focus. ## Custom ports diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2e21499..c35cdd8 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -46,14 +46,26 @@ To point pi-chrome at an existing tab, pass `targetId`, `urlIncludes`, or `title ## Background mode -By default, chrome calls run in background so Chrome does not steal focus. +The existing background setting is a hard session policy, enabled by default. No separate lock/unlock command is needed. ```text -/chrome background on # background mode -/chrome background off # foreground/watch mode +/chrome background on # enforce no explicit window focus/tab activation +/chrome background off # allow foreground/watch mode ``` -Per-call `background: false` brings Chrome forward for that action. Per-call `background: true` forces background. +- With background on, per-call `background:false` and legacy `foreground:true` cannot override the policy. `chrome_tab activate` errors with instructions to ask the user to turn background off. +- With background off, calls may focus Chrome; per-call `background:true` still avoids explicit focus/tab activation. +- Policy is applied in `authorizedBridgeSend` for every tool, including `chrome_launch(url)`, tab creation, and tools without a background parameter. Each session sends its own effective `background`/`foreground` flags through the shared bridge. +- All worker window-focus/tab-activation writes go through a guarded helper. Background tab creation uses `active:false`; implicit automation windows remain `focused:false`. +- Screenshots use CDP `Page.captureScreenshot` with `fromSurface:true` and `captureBeyondViewport:false`. They target a tab, not whichever tab happens to be visible. There is no `captureVisibleTab`/activation fallback on debugger or capture failure. PNG/JPEG output is unchanged; full-page capture retains tiles plus a JSON manifest and restores scroll position best-effort, including on failure. +- Background tab creation and screenshots use internal `tab.new.background` / `page.screenshot.background` wire actions. Old companions reject them before changing tabs; Pi reports a reload instruction instead of retrying an unsafe legacy action. No capability-probe race or extra round trip is needed. +- Real CDP input and existing explicit DOM-fallback controls are unchanged. Background mode never silently substitutes synthetic input to avoid focus. + +### Scope and risks + +This is a policy against **explicit pi-chrome focus/activation**, not an OS focus sandbox. Page scripts (`window.open`, `window.focus`), trusted input, native dialogs, debugger banners, Chrome window/Spaces behavior, and closing an active tab can still change focus or selection. Other sessions and human actions remain independent. Requests already dispatched before a mode change keep their earlier policy. + +Inactive/minimized tabs can throttle timers or rendering, and clipboard/fullscreen/other focus-gated workflows may fail. Screenshots now require debugger attachment, which can conflict with DevTools or other extensions; hidden-tab rendering can differ or be unavailable. No automatic foreground retry is allowed. Full-page capture temporarily scrolls the target page. Reload both Pi and the Chrome companion after upgrading, and live-test tab selection, OS focus, and screenshot fidelity on supported Chrome/OS versions. ## Authorization diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md index 16cd81a..0a70895 100644 --- a/docs/COMPARISON.md +++ b/docs/COMPARISON.md @@ -134,7 +134,7 @@ If your threat model excludes extensions with broad permissions, neither approac ## Public benchmarks worth knowing (for axis 2 / axis 3 comparison) -Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **42 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, iframes, file uploads, strict-CSP screenshot fallback and CDP eval/snapshot bypass, dynamic waits, tab lifecycle, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count, with `core`, `conditional`, and `quality` gate buckets. That's **driver-level** grading. +Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **43 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, iframes, file uploads, strict-CSP screenshot fallback and CDP eval/snapshot bypass, dynamic waits, tab lifecycle, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count, with `core`, `conditional`, and `quality` gate buckets. That's **driver-level** grading. For **agent-level** comparison (axis 2), the public benchmarks worth citing: diff --git a/extensions/chrome-profile-bridge/browser-extension/manifest.json b/extensions/chrome-profile-bridge/browser-extension/manifest.json index 4fe5b17..38d1b7f 100644 --- a/extensions/chrome-profile-bridge/browser-extension/manifest.json +++ b/extensions/chrome-profile-bridge/browser-extension/manifest.json @@ -1,7 +1,7 @@ { "manifest_version": 3, "name": "Pi Chrome Connector", - "version": "0.15.47", + "version": "0.15.48", "description": "Lets Pi control tabs in Chrome via a local connector at 127.0.0.1.", "permissions": [ "tabs", diff --git a/extensions/chrome-profile-bridge/browser-extension/service_worker.js b/extensions/chrome-profile-bridge/browser-extension/service_worker.js index dd0567a..fd6c452 100644 --- a/extensions/chrome-profile-bridge/browser-extension/service_worker.js +++ b/extensions/chrome-profile-bridge/browser-extension/service_worker.js @@ -745,7 +745,7 @@ async function domClickFallback(tabId, params, cause) { async function chromeInputClick(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); try { await attachDebugger(tab.id); const resolved = await resolveTargetInTab(tab.id, params); @@ -782,7 +782,7 @@ async function chromeInputClick(params) { async function chromeInputHover(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); await attachDebugger(tab.id); const resolved = await resolveTargetInTab(tab.id, params); const point = resolved.rect ? pickInsideRect(resolved.rect) : { x: resolved.x, y: resolved.y }; @@ -793,7 +793,7 @@ async function chromeInputHover(params) { async function chromeInputKey(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); await attachDebugger(tab.id); const key = String(params.key || ""); if (!key) throw new Error("chrome.key: missing key"); @@ -831,7 +831,7 @@ async function chromeInputKey(params) { async function chromeInputType(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); await attachDebugger(tab.id); if (params.selector || params.uid) { // Focus target by clicking it first. @@ -894,7 +894,7 @@ async function domFillFallback(tabId, params, cause) { async function chromeInputFill(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); try { await attachDebugger(tab.id); if (!(params.selector || params.uid)) throw new Error("chrome.fill: selector or uid required"); @@ -924,7 +924,7 @@ async function chromeInputFill(params) { async function chromeInputScroll(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); await attachDebugger(tab.id); const resolved = (params.selector || params.uid) ? await resolveTargetInTab(tab.id, params) : { x: 100, y: 100, rect: null }; const x = resolved.rect ? resolved.rect.left + Math.min(resolved.rect.width, 800) / 2 : resolved.x; @@ -974,7 +974,7 @@ async function chromeInputScroll(params) { async function chromeInputTap(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); await attachDebugger(tab.id); const resolved = (params.selector || params.uid || (typeof params.x === "number" && typeof params.y === "number")) ? await resolveTargetInTab(tab.id, params) @@ -990,7 +990,7 @@ async function chromeInputTap(params) { async function chromeInputDrag(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); await attachDebugger(tab.id); const from = await resolveTargetInTab(tab.id, { selector: params.fromSelector ?? null, uid: params.fromUid ?? null, x: params.fromX ?? null, y: params.fromY ?? null }); const to = await resolveTargetInTab(tab.id, { selector: params.toSelector ?? null, uid: params.toUid ?? null, x: params.toX ?? null, y: params.toY ?? null }); @@ -1015,7 +1015,7 @@ async function chromeInputDrag(params) { async function chromeInputUpload(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); await attachDebugger(tab.id); if (!(params.selector || params.uid)) throw new Error("chrome.upload: selector or uid required"); const paths = Array.isArray(params.paths) ? params.paths.map(String) : []; @@ -1208,22 +1208,29 @@ async function dispatch(action, params) { extensionVersion: chrome.runtime.getManifest().version, bridgeUrl: BRIDGE_URL, userAgent: navigator.userAgent, + capabilities: { hardBackground: true }, }; case "tab.list": { const tabs = await chrome.tabs.query({}); return Promise.all(tabs.map(formatTab)); } + case "tab.new.background": + case "page.screenshot.background": + // Older workers reject these action names before touching tabs. Do not replace this with + // a capability probe followed by an old action: a reload/profile change can race the probe. + return dispatch(action.slice(0, -".background".length), { ...params, background: true, foreground: false }); case "tab.new": { // Every Pi-opened tab must join a tab group. There is intentionally no opt-out: an ungrouped // Pi-created tab is easy to lose among user tabs. If grouping fails after creation, close the // tab best-effort before surfacing the error so tab.new never leaves an ungrouped Pi tab. const groupTitle = params.groupTitle || "Pi"; const existingGroup = await findGroupRecordByTitle(groupTitle); - const createParams = { url: params.url || "about:blank", active: true }; + const createParams = { url: params.url || "about:blank", active: foregroundRequested(params) }; if (existingGroup && typeof existingGroup.windowId === "number") createParams.windowId = existingGroup.windowId; const tab = await chrome.tabs.create(createParams); await trackSessionTab(sessionKeyOf(params), tab.id, true); try { + await bringToFront(tab, params); return await groupTab(tab, groupTitle, params.groupColor); } catch (error) { if (typeof tab.id === "number") await chrome.tabs.remove(tab.id).catch(() => {}); @@ -1231,12 +1238,14 @@ async function dispatch(action, params) { } } case "tab.activate": { + if (!foregroundRequested(params)) { + throw new Error("Tab activation is blocked by background mode. Ask the user to run /chrome background off to allow foreground work."); + } // Management actions never auto-create an automation target (createOwnedTarget:false): with // no explicit target they act on an owned target if one exists, else error — they must never // fall back to (or spawn a tab just to touch) the user's active tab. const tab = await getTabByParams(params, { createOwnedTarget: false }); - await chrome.windows.update(tab.windowId, { focused: true }); - return formatTab(await chrome.tabs.update(tab.id, { active: true })); + return formatTab(await bringToFront(tab, params)); } case "tab.group": { const tab = await getTabByParams(params, { createOwnedTarget: false }); @@ -1292,7 +1301,7 @@ async function dispatch(action, params) { // Poll from the service worker via CDP (bypasses CSP). The old approach ran the polling // loop in-page with new Function() for expression checks, which fails under strict CSP. const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); const timeoutMs = params.timeoutMs || 10000; const intervalMs = params.intervalMs || 250; const started = Date.now(); @@ -1316,7 +1325,7 @@ async function dispatch(action, params) { return executeInTab(params, probePage, []); case "page.navigate": { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); if (params.initScript) { // Register a one-shot document_start content script. We register, navigate, wait, then unregister. await registerInitScript(tab.id, params.initScript); @@ -1477,7 +1486,7 @@ const HELPER_FUNCS = [ async function executeInTab(params, func, args) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); // Phase 1: define the helpers and the action function as page globals via CDP // Runtime.evaluate. This bypasses page CSP (no `eval`/`new Function`), which is the @@ -1544,7 +1553,7 @@ function piEvalStringify(v) { // pages that ship `script-src 'self'` without `'unsafe-eval'` (which blocks `eval`/`new Function`). async function evaluateInTab(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); const expression = String(params.expression ?? ""); const stringifySrc = `(${piEvalStringify.toString()})`; // Wrap the user expression so the result is run through piEvalStringify in-page before it @@ -1591,7 +1600,7 @@ async function withOptionalSnapshot(params, actionFn) { // It shares window.__PI_CHROME_STATE__ (same el- uid scheme) with the CDP-injected input helpers. async function snapshotInTab(params) { const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); const args = [ params.maxElements || 80, params.containingText ?? null, @@ -1635,7 +1644,7 @@ async function snapshotInTab(params) { async function inspectInTab(params) { if (!params.uid && !params.selector) throw new Error("chrome_inspect requires uid or selector"); const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); + await bringToFront(tab, params); const args = [params.uid ?? null, params.selector ?? null, params.scrollIntoView === true]; await executeScriptTimed({ target: { tabId: tab.id, frameIds: [0] }, @@ -1705,9 +1714,15 @@ if (chrome.webNavigation && chrome.webNavigation.onCommitted) { }); } -async function bringToFront(tab) { +function foregroundRequested(params) { + // Fail quiet when unspecified, and let background veto even a contradictory foreground flag. + return params?.foreground === true && params.background !== true; +} + +async function bringToFront(tab, params) { + if (!foregroundRequested(params)) return tab; await chrome.windows.update(tab.windowId, { focused: true }); - await chrome.tabs.update(tab.id, { active: true }); + return chrome.tabs.update(tab.id, { active: true }); } function waitForTabComplete(tabId, timeoutMs) { @@ -1727,50 +1742,51 @@ function waitForTabComplete(tabId, timeoutMs) { }); } -async function takeScreenshot(params) { - const tab = await getTabByParams(params); - if (params.foreground) await bringToFront(tab); - let previousActiveId; - if (!tab.active) { - const activeBefore = await chrome.tabs.query({ active: true, windowId: tab.windowId }); - previousActiveId = activeBefore[0]?.id; - await chrome.tabs.update(tab.id, { active: true }); - } +async function captureTabScreenshot(tabId, params) { + const format = params.format || "png"; try { - if (params.fullPage) { - // Tile-stitched full page capture: scroll, capture, paste, repeat. - const tiles = await executeInTab({ ...params, foreground: false }, captureFullPageTiles, []); - // captureFullPageTiles only computes scroll positions / metrics; we capture per scroll here - // (chrome.tabs.captureVisibleTab can't be called from MAIN world). - const captured = []; + await attachDebugger(tabId); + const captureParams = { format, fromSurface: true, captureBeyondViewport: false }; + if (format === "jpeg" && params.quality !== undefined) captureParams.quality = params.quality; + const result = await cdp(tabId, "Page.captureScreenshot", captureParams); + if (typeof result?.data !== "string" || !result.data) throw new Error("CDP returned no screenshot data"); + return `data:image/${format};base64,${result.data}`; + } catch (error) { + // captureVisibleTab requires activation and can race with the user switching tabs. Never + // use it as a fallback, even when debugger attachment or background rendering fails. + throw new Error(`Chrome screenshot via CDP failed; no tab-activation fallback was attempted. ${error?.message || error}`); + } +} + +async function takeScreenshot(params) { + const tab = await bringToFront(await getTabByParams(params), params); + if (params.fullPage) { + // Preserve the existing tile + manifest contract. Every tile captures the same resolved tab, + // without activation; selector/title changes during capture must not retarget later tiles. + const targetParams = { ...params, targetId: tab.id, foreground: false }; + const tiles = await executeInTab(targetParams, captureFullPageTiles, []); + const captured = []; + try { for (const tile of tiles.tiles) { - await executeInTab({ ...params, foreground: false }, scrollToY, [tile.scrollY]); - // Small settle delay; many sites have on-scroll animations / lazy-load. - await sleep(120); - const dataUrl = await chrome.tabs.captureVisibleTab(tab.windowId, { - format: params.format || "png", - quality: params.format === "jpeg" ? params.quality : undefined, - }); - captured.push({ y: tile.y, dataUrl }); + await executeInTab(targetParams, scrollToY, [tile.scrollY]); + await sleep(120); // Let scroll/lazy-load handlers settle. + captured.push({ y: tile.y, dataUrl: await captureTabScreenshot(tab.id, params) }); } - await executeInTab({ ...params, foreground: false }, scrollToY, [tiles.originalScrollY]); - return { - fullPage: true, - tab: await formatTab(tab), - dimensions: { width: tiles.width, height: tiles.height, viewportHeight: tiles.viewportHeight, dpr: tiles.dpr }, - tiles: captured, - }; - } - const dataUrl = await chrome.tabs.captureVisibleTab(tab.windowId, { - format: params.format || "png", - quality: params.format === "jpeg" ? params.quality : undefined, - }); - return { dataUrl, tab: await formatTab(tab) }; - } finally { - if (previousActiveId !== undefined && previousActiveId !== tab.id) { - await chrome.tabs.update(previousActiveId, { active: true }).catch(() => undefined); + } finally { + // A failed tile must not strand the page at a new scroll position. Restore both axes + // best-effort, without masking the capture error if the tab/debugger is gone. + await executeInTab(targetParams, scrollToY, [tiles.originalScrollY, tiles.originalScrollX]).catch(() => undefined); } + return { + fullPage: true, + method: "cdp", + tab: await formatTab(tab), + dimensions: { width: tiles.width, height: tiles.height, viewportHeight: tiles.viewportHeight, dpr: tiles.dpr }, + tiles: captured, + }; } + const dataUrl = await captureTabScreenshot(tab.id, params); + return { dataUrl, method: "cdp", tab: await formatTab(tab) }; } // --------------------------------------------------------------------------- @@ -2197,8 +2213,7 @@ function probePage() { } function captureFullPageTiles() { - // Returns the *plan* for tile capture; the actual chrome.tabs.captureVisibleTab calls happen - // in the SW. We just report the scroll positions and metrics. + // Returns the plan for CDP tile capture in the worker: scroll positions and page metrics. const html = document.documentElement; const body = document.body; const width = Math.max(html.scrollWidth, body ? body.scrollWidth : 0, innerWidth); @@ -2206,17 +2221,18 @@ function captureFullPageTiles() { const viewportHeight = innerHeight; const dpr = window.devicePixelRatio || 1; const originalScrollY = scrollY; + const originalScrollX = scrollX; const tiles = []; let y = 0; while (y < height) { tiles.push({ y, scrollY: y }); y += viewportHeight; } - return { width, height, viewportHeight, dpr, originalScrollY, tiles }; + return { width, height, viewportHeight, dpr, originalScrollY, originalScrollX, tiles }; } -function scrollToY(y) { - window.scrollTo({ top: y, left: 0, behavior: "instant" }); +function scrollToY(y, x = 0) { + window.scrollTo({ top: y, left: x, behavior: "instant" }); return { scrollY }; } diff --git a/extensions/chrome-profile-bridge/index.ts b/extensions/chrome-profile-bridge/index.ts index fdfa39c..9546b9c 100644 --- a/extensions/chrome-profile-bridge/index.ts +++ b/extensions/chrome-profile-bridge/index.ts @@ -64,6 +64,7 @@ const DEFAULT_PORT = Number(process.env.PI_CHROME_BRIDGE_PORT ?? "17318"); const DEFAULT_TIMEOUT_MS = 30_000; const MAX_TEXT_CHARS = 30_000; const MAX_ELEMENTS = 80; +const BACKGROUND_PARAM_DESCRIPTION = "If true, avoid explicit Chrome focus/tab activation for this call. /chrome background on (default) enforces this for every call and ignores false. Ask the user to run /chrome background off to allow foreground work."; function truncateText(text: string, maxChars = MAX_TEXT_CHARS): string { if (text.length <= maxChars) return text; @@ -685,7 +686,7 @@ export default function (pi: ExtensionAPI): void { globalState[PI_CHROME_GLOBAL_KEY] = { version: PI_CHROME_VERSION, root: currentRoot, token: instanceToken }; const bridge = new ChromeProfileBridge(DEFAULT_HOST, DEFAULT_PORT); - let backgroundDefault = true; + let backgroundEnabled = true; let chromeAuthorizedUntil: number | "indefinite" | undefined; // Restore an authorization that survived a /reload. Drop it if it already expired. const persistedAuth = globalState[PI_CHROME_AUTH_KEY]; @@ -881,14 +882,21 @@ export default function (pi: ExtensionAPI): void { }, Math.max(0, until - Date.now())); }; - const authorizedBridgeSend = (action: string, params: Record<string, unknown>, timeoutMs = DEFAULT_TIMEOUT_MS, signal?: AbortSignal): Promise<unknown> => { + const authorizedBridgeSend = async (action: string, params: Record<string, unknown>, timeoutMs = DEFAULT_TIMEOUT_MS, signal?: AbortSignal): Promise<unknown> => { requireChromeControlAuthorized(); - // Scope the service worker's dedicated automation tab/window to this session. Forwarded on - // every action so tab resolution, navigation, and cleanup all agree on which target is ours. + // Background on is a session policy, not a default that tool arguments can override. + // Apply it here so tab.new, chrome_launch(url), and tools without a background parameter + // cannot bypass it. Background off still permits per-call background:true. + const typed = params as { background?: boolean; foreground?: boolean }; + const requestedBackground = typed.background ?? (typed.foreground !== undefined ? !typed.foreground : false); + const background = backgroundEnabled || requestedBackground; + if (action === "tab.activate" && background) { + throw new Error("Tab activation is blocked by background mode. Ask the user to run /chrome background off to allow foreground work."); + } + // Scope every action to this session's dedicated automation target and tab group. const sessionKey = sessionKeyFor(sessionCtx); - let wireParams: Record<string, unknown> = sessionKey !== undefined && params.sessionKey === undefined - ? { ...params, sessionKey } - : params; + let wireParams: Record<string, unknown> = { ...params, background, foreground: !background }; + if (sessionKey !== undefined && params.sessionKey === undefined) wireParams.sessionKey = sessionKey; const sessionTitle = sessionCtx !== undefined ? sessionGroupTitle(sessionCtx) : undefined; // Any tab Pi opens through tab.new/tab.group must use THIS session's group, even if a caller // passes group:false or a custom groupTitle. This central guard covers chrome_tab plus internal @@ -904,21 +912,21 @@ export default function (pi: ExtensionAPI): void { if (shouldJoinGroup) { wireParams = { ...wireParams, sessionGroupTitle: sessionTitle, joinSessionGroup: true }; } - return bridge.send(action, wireParams, timeoutMs, signal); - }; - - // Translate the public `background` parameter (default on = silent/background) into the - // service worker's wire-level `foreground` flag, accepting legacy `foreground` as a fallback. - const withBackground = <T extends Record<string, unknown>>(params: T): T => { - const typed = params as { background?: boolean; foreground?: boolean }; - const explicit = - typed.background !== undefined - ? typed.background - : typed.foreground !== undefined - ? !typed.foreground - : undefined; - const background = explicit ?? backgroundDefault; - return { ...params, foreground: !background } as T; + // Older companions ignore background for tab creation and activate tabs for screenshots. + // Dedicated wire actions make them fail closed, with no probe/action race or extra round trip. + // These are internal protocol aliases, not new tools or /chrome commands. + const wireAction = background && (action === "tab.new" || action === "page.screenshot") + ? `${action}.background` + : action; + try { + return await bridge.send(wireAction, wireParams, timeoutMs, signal); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + if (wireAction !== action && message.includes(`Unknown action: ${wireAction}`)) { + throw new Error("Hard background requires an updated Chrome companion extension. Reload Pi Chrome Connector at chrome://extensions, then retry."); + } + throw error; + } }; pi.on("session_start", async (_event, ctx) => { @@ -974,7 +982,7 @@ Usage rules: 3. \`includeSnapshot=true\` on click/type/fill/key to verify in one round trip. 4. If \`chrome_evaluate\` returns null when you expected a value, the expression evaluated to null/undefined in the page; surface the value via \`JSON.stringify\` to confirm. 5. \`chrome_navigate\` supports an optional \`initScript\` that runs at document_start in MAIN world for the next navigation (good for seeding localStorage or stubbing Date.now). -6. By default chrome_* tools run in the background without focusing Chrome; pass \`background=false\` or run /chrome background off when the user wants to watch Chrome work. +6. /chrome background on (default) is a hard policy: per-call \`background=false\` cannot override it, new tabs stay inactive, and \`chrome_tab activate\` is blocked. Ask the user to run /chrome background off when they want foreground/watch mode. With background off, per-call \`background=true\` still avoids explicit focus/tab activation. Screenshots use CDP without activating background tabs; debugger failures never fall back to tab activation. Page scripts, trusted input, native prompts, and Chrome/OS behavior can still affect focus. 7. If you hit a native file-picker or privileged browser prompt gate, tell the user; generic clicks/typing/CSP gates are handled by Chrome input. 8. Run /chrome doctor when in doubt about connectivity or capabilities. </chrome-profile-bridge>`; @@ -1045,30 +1053,30 @@ Usage rules: ctx.ui.notify(lines.join("\n"), "info"); }; - // Run-in-background (Chrome focus) handler. No args = toggle. Explicit on/off/status. + // Existing background setting is the hard policy. No args = toggle; no separate lock mode. const BACKGROUND_DESC: Record<string, string> = { - on: "pi-chrome runs in the background; Chrome won't pop up or steal focus.", - off: "Chrome pops to the front and switches tabs so you can watch what pi-chrome is doing.", + on: "Hard background: pi-chrome will not explicitly focus windows or activate tabs; per-call foreground overrides and tab activation are blocked. Page/Chrome behavior can still affect focus.", + off: "Foreground/watch mode: Chrome may come forward and switch tabs. Per-call background:true still avoids explicit focus/tab activation.", }; const backgroundHandler = async (ctx: ExtensionContext, args: string) => { const arg = (args || "").trim().toLowerCase(); - const currentLabel = backgroundDefault ? "on" : "off"; + const currentLabel = backgroundEnabled ? "on" : "off"; if (arg === "status") { ctx.ui.notify(`Run in background is ${currentLabel}. ${BACKGROUND_DESC[currentLabel]}`, "info"); return; } - if (arg === "on" || arg === "true" || arg === "1") backgroundDefault = true; - else if (arg === "off" || arg === "false" || arg === "0") backgroundDefault = false; - else if (arg === "toggle" || arg === "") backgroundDefault = !backgroundDefault; + if (arg === "on" || arg === "true" || arg === "1") backgroundEnabled = true; + else if (arg === "off" || arg === "false" || arg === "0") backgroundEnabled = false; + else if (arg === "toggle" || arg === "") backgroundEnabled = !backgroundEnabled; else { ctx.ui.notify(`Unknown background setting '${arg}'. Pick one of: on | off | toggle | status.`, "warning"); return; } - const nextLabel = backgroundDefault ? "on" : "off"; + const nextLabel = backgroundEnabled ? "on" : "off"; ctx.ui.notify(`Run in background → ${nextLabel}. ${BACKGROUND_DESC[nextLabel]}`, "info"); }; @@ -1151,7 +1159,7 @@ Usage rules: parts.push(`✗ Chrome not responding`); } parts.push(`auth: ${authSummary()}`); - parts.push(`background: ${backgroundDefault ? "on" : "off"}`); + parts.push(`background: ${backgroundEnabled ? "on (hard)" : "off"}`); return parts.join(" · "); }; @@ -1217,7 +1225,7 @@ Usage rules: pi.registerCommand("chrome", { description: - "All pi-chrome controls in one place.\n /chrome authorize [15m|30m|<minutes>|indefinite] — allow this Pi session to use chrome_* tools.\n /chrome revoke — lock Chrome control.\n /chrome status — one-line snapshot of connection, auth, and background setting.\n /chrome doctor — full health check.\n /chrome onboard — install the Chrome companion extension.\n /chrome background [on|off|status|toggle] — whether pi-chrome runs without focusing Chrome.\nRun with no arguments for an interactive picker that shows current state.", + "All pi-chrome controls in one place.\n /chrome authorize [15m|30m|<minutes>|indefinite] — allow this Pi session to use chrome_* tools.\n /chrome revoke — lock Chrome control.\n /chrome status — one-line snapshot of connection, auth, and background setting.\n /chrome doctor — full health check.\n /chrome onboard — install the Chrome companion extension.\n /chrome background [on|off|status|toggle] — enforce no explicit focus/tab activation, or allow foreground/watch mode.\nRun with no arguments for an interactive picker that shows current state.", getArgumentCompletions: (prefix) => { const raw = prefix; const trimmedRight = raw.replace(/\s+$/, ""); @@ -1239,7 +1247,7 @@ Usage rules: { fullValue: "status", label: "status", description: "One-line summary: connection, auth, and background setting." }, { fullValue: "doctor", label: "doctor", description: "Full health check. Tells you if Chrome is connected and what's wrong if it isn't." }, { fullValue: "onboard", label: "onboard", description: "Install the Chrome companion extension (first-time setup)." }, - { fullValue: "background", label: "background", description: "Run pi-chrome in the background without focusing Chrome?" }, + { fullValue: "background", label: "background", description: "Enforce hard background or allow foreground/watch mode." }, ]; } else if (path[0] === "authorize" && path.length === 1) { candidates = [ @@ -1249,7 +1257,7 @@ Usage rules: ]; } else if (path[0] === "background" && path.length === 1) { candidates = [ - { fullValue: "background on", label: "on", description: "Run in background. Chrome stays in the background. Your editor keeps focus. (default)" }, + { fullValue: "background on", label: "on", description: "Hard background: block explicit focus/tab activation and per-call foreground overrides. (default)" }, { fullValue: "background off", label: "off", description: "Bring Chrome to the front so you can watch." }, { fullValue: "background toggle", label: "toggle", description: "Flip whichever way it's currently set." }, { fullValue: "background status", label: "status", description: "Show the current setting." }, @@ -1333,7 +1341,7 @@ Usage rules: pi.registerTool({ name: "chrome_tab", label: "Chrome Tab", - description: "List, create, activate, close, group, ungroup, or inspect tabs in the user's existing Chrome profile via the companion extension. New/grouped tabs always use this session's Pi tab group. activate/close/group/ungroup require a target (targetId/urlIncludes/titleIncludes); with no target they act on this session's pi-chrome automation tab if one exists, and otherwise error rather than touching the user's active tab.", + description: "List, create, activate, close, group, ungroup, or inspect tabs in the user's existing Chrome profile via the companion extension. New/grouped tabs always use this session's Pi tab group. Background mode keeps new tabs inactive and blocks activate; ask the user to run /chrome background off for foreground/watch mode. activate/close/group/ungroup require a target (targetId/urlIncludes/titleIncludes); with no target they act on this session's pi-chrome automation tab if one exists, and otherwise error rather than touching the user's active tab.", promptSnippet: "List/open/activate/close/group existing Chrome tabs through the companion extension.", parameters: Type.Object({ action: StringEnum(tabActionValues), @@ -1369,7 +1377,7 @@ Usage rules: name: "chrome_snapshot", label: "Chrome Snapshot", description: - "Inspect a page in the user's existing Chrome profile. Default output is a concise, agent-friendly observation with structural layout/context, stable uids, visible actions, form fields, page hints, and changes since the previous snapshot. Use mode/query/nearUid to zoom instead of dumping the whole page. Runs in the background by default; pass background=false to bring Chrome to the foreground so the user can watch.", + "Inspect a page in the user's existing Chrome profile. Default output is a concise, agent-friendly observation with structural layout/context, stable uids, visible actions, form fields, page hints, and changes since the previous snapshot. Use mode/query/nearUid to zoom instead of dumping the whole page. Background mode (default) blocks explicit focus/tab activation even with background=false. Ask the user to run /chrome background off for foreground/watch mode.", promptSnippet: "Observe the current Chrome page: concise summary, structural layout, visible actions, forms, page map, query matches, and stable uids.", parameters: Type.Object({ targetId: Type.Optional(Type.String()), @@ -1382,16 +1390,14 @@ Usage rules: containingText: Type.Optional(Type.String({ description: "Only return elements whose label/text contains this string (case-insensitive). Useful when the page has many controls." })), roleFilter: Type.Optional(Type.String({ description: "Only return elements matching this ARIA role or tag name (case-insensitive). e.g. 'button', 'link', 'textbox'." })), nearUid: Type.Optional(Type.String({ description: "Sort elements by proximity to this snapshot uid. Useful for finding controls near a known anchor." })), - background: Type.Optional( - Type.Boolean({ description: "If true (the default), run silently in the background without focusing Chrome; pass false so Chrome focuses + the tab activates and the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { const snapshot = await authorizedBridgeSend( "page.snapshot", - withBackground({ ...params, maxElements: params.maxElements ?? MAX_ELEMENTS }), + { ...params, maxElements: params.maxElements ?? MAX_ELEMENTS }, DEFAULT_TIMEOUT_MS, signal, ); @@ -1412,16 +1418,14 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true (the default), run silently in the background without focusing Chrome; pass false so Chrome focuses + the tab activates and the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { const snapshot = await authorizedBridgeSend( "page.snapshot", - withBackground({ ...params, mode: params.mode || "auto", maxElements: params.maxElements ?? MAX_ELEMENTS }), + { ...params, mode: params.mode || "auto", maxElements: params.maxElements ?? MAX_ELEMENTS }, DEFAULT_TIMEOUT_MS, signal, ); @@ -1442,15 +1446,13 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true (the default), run silently in the background without focusing Chrome; pass false so Chrome focuses + the tab activates and the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { try { - const inspect = await authorizedBridgeSend("page.inspect", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const inspect = await authorizedBridgeSend("page.inspect", params, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: formatChromeInspect(inspect) }], details: { inspect } }; } catch (error) { const message = error instanceof Error ? error.message : String(error); @@ -1460,13 +1462,13 @@ Usage rules: // but still gives useful nearby candidates instead of failing the workflow. const snapshot = await authorizedBridgeSend( "page.snapshot", - withBackground({ + { ...params, mode: "interactive", maxElements: MAX_ELEMENTS, nearUid: params.uid, query: params.selector, - }), + }, DEFAULT_TIMEOUT_MS, signal, ); @@ -1480,7 +1482,7 @@ Usage rules: name: "chrome_navigate", label: "Chrome Navigate", description: - "Navigate a Chrome tab to a URL via the companion extension. With no target, navigation goes to pi-chrome's own dedicated automation window/tab — it never replaces the user's active tab. Pass targetId/urlIncludes/titleIncludes only to act on a specific existing tab. Runs in the background by default; pass background=false to focus Chrome and activate the tab so the user can watch. Optionally waits for load completion.", + "Navigate a Chrome tab to a URL via the companion extension. With no target, navigation goes to pi-chrome's own dedicated automation window/tab — it never replaces the user's active tab. Pass targetId/urlIncludes/titleIncludes only to act on a specific existing tab. Background mode (default) blocks explicit focus/tab activation even with background=false; /chrome background off allows foreground/watch mode. Optionally waits for load completion.", promptSnippet: "Navigate a Chrome tab in the user's existing profile.", parameters: Type.Object({ url: Type.String(), @@ -1490,14 +1492,12 @@ Usage rules: waitUntilLoad: Type.Optional(Type.Boolean({ default: true })), timeoutMs: Type.Optional(Type.Number({ default: 15_000 })), initScript: Type.Optional(Type.String({ description: "Optional JavaScript source to run in MAIN world at document_start of the next navigation. Useful for seeding localStorage, stubbing Date.now(), or defining navigator.webdriver=undefined. Requires the companion extension's webNavigation permission." })), - background: Type.Optional( - Type.Boolean({ description: "If true, navigate silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.navigate", withBackground(params), (params.timeoutMs ?? 15_000) + 2_000, signal); + const result = await authorizedBridgeSend("page.navigate", params, (params.timeoutMs ?? 15_000) + 2_000, signal); return { content: [{ type: "text", text: `Navigated to ${params.url}${params.initScript ? " (with initScript)" : ""}` }], details: { result: result as Json } }; }, }); @@ -1506,7 +1506,7 @@ Usage rules: name: "chrome_evaluate", label: "Chrome Evaluate", description: - "Evaluate JavaScript in an existing Chrome tab through the companion extension. Runs in the page context and returns JSON-serializable values when possible. Runs in the background by default; pass background=false to focus Chrome and activate the tab.", + "Evaluate JavaScript in an existing Chrome tab through the companion extension. Runs in the page context and returns JSON-serializable values when possible. Background mode (default) blocks explicit focus/tab activation even with background=false; /chrome background off allows foreground/watch mode.", promptSnippet: "Evaluate JavaScript in the active Chrome tab through the companion extension.", parameters: Type.Object({ expression: Type.String(), @@ -1514,14 +1514,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true, evaluate silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const value = await authorizedBridgeSend("page.evaluate", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const value = await authorizedBridgeSend("page.evaluate", params, DEFAULT_TIMEOUT_MS, signal); const text = value === undefined ? "undefined" : typeof value === "string" @@ -1548,14 +1546,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true, click silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const raw = await authorizedBridgeSend("page.click", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const raw = await authorizedBridgeSend("page.click", params, DEFAULT_TIMEOUT_MS, signal); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const target = params.uid ?? params.selector ?? `${params.x},${params.y}`; @@ -1580,14 +1576,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true, type silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const raw = await authorizedBridgeSend("page.type", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const raw = await authorizedBridgeSend("page.type", params, DEFAULT_TIMEOUT_MS, signal); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const into = params.uid || params.selector ? ` into ${params.uid ?? params.selector}` : ""; @@ -1614,14 +1608,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true, fill silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const raw = await authorizedBridgeSend("page.fill", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const raw = await authorizedBridgeSend("page.fill", params, DEFAULT_TIMEOUT_MS, signal); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const into = params.uid || params.selector ? ` into ${params.uid ?? params.selector}` : ""; @@ -1635,7 +1627,7 @@ Usage rules: name: "chrome_key", label: "Chrome Key", description: - "Send a keyboard key to an existing Chrome tab (Enter, Escape, Tab, Backspace, Delete, ArrowUp/Down/Left/Right, or one character). Runs in the background by default; pass background=false to focus Chrome and activate the tab so the user can watch. Pass includeSnapshot=true to verify after the keypress.", + "Send a keyboard key to an existing Chrome tab (Enter, Escape, Tab, Backspace, Delete, ArrowUp/Down/Left/Right, or one character). Background mode (default) blocks explicit focus/tab activation even with background=false; /chrome background off allows foreground/watch mode. Pass includeSnapshot=true to verify after the keypress.", promptSnippet: "Press keys in Chrome through the companion extension.", parameters: Type.Object({ key: Type.String(), @@ -1650,14 +1642,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true, send the key silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const raw = await authorizedBridgeSend("page.key", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const raw = await authorizedBridgeSend("page.key", params, DEFAULT_TIMEOUT_MS, signal); const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json; const summary = summarizeActionResult(result); const base = `Pressed ${params.key}.`; @@ -1699,12 +1689,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean({ description: "If true, run silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." })), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.console.list", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.console.list", params, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; }, }); @@ -1721,12 +1711,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean({ description: "If true, run silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." })), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.network.list", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.network.list", params, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; }, }); @@ -1741,12 +1731,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean({ description: "If true, run silently without focusing Chrome. Defaults to on (the session background setting); pass false to focus Chrome so the user can watch." })), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.network.get", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.network.get", params, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: truncateText(safeJson(result)) }], details: { result: result as Json } }; }, }); @@ -1755,19 +1745,17 @@ Usage rules: name: "chrome_screenshot", label: "Chrome Screenshot", description: - "Capture a screenshot of an existing Chrome tab via the companion extension and save it to disk. Chrome's extension screenshot API requires the target tab to be the active tab in its window. Runs in the background by default (the tab is briefly activated within its window for the capture, then the previous active tab is restored); pass background=false to focus Chrome so the user can watch.", + "Capture a screenshot of a Chrome tab via CDP and save it to disk without activating background tabs. Requires debugger access; failures never fall back to activating a tab. Background mode (default) ignores background=false; /chrome background off allows foreground/watch mode.", promptSnippet: "Capture Chrome screenshots and save them under .pi/chrome-screenshots by default.", parameters: Type.Object({ path: Type.Optional(Type.String({ description: "Output path. Defaults to .pi/chrome-screenshots/<timestamp>.<format>." })), format: Type.Optional(StringEnum(imageFormatValues)), - quality: Type.Optional(Type.Number({ description: "JPEG quality 0-100." })), - fullPage: Type.Optional(Type.Boolean({ description: "Not supported by the extension bridge yet; viewport screenshots are captured." })), + quality: Type.Optional(Type.Number({ minimum: 0, maximum: 100, description: "JPEG quality 0-100." })), + fullPage: Type.Optional(Type.Boolean({ description: "Capture full-page tiles plus a JSON manifest. Temporarily scrolls the target page; does not activate background tabs." })), targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional( - Type.Boolean({ description: "If true (the default), capture silently without focusing the Chrome window (the target tab is briefly activated within its window for the capture, then restored); pass false to focus Chrome." }), - ), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), host: Type.Optional(Type.String()), port: Type.Optional(Type.Number()), }), @@ -1776,8 +1764,9 @@ Usage rules: const cwd = workspaceCwd(ctx); const defaultPath = join(cwd, ".pi", "chrome-screenshots", `${new Date().toISOString().replace(/[:.]/g, "-")}.${format}`); const outputPath = params.path ? resolve(cwd, params.path) : defaultPath; - const result = (await authorizedBridgeSend("page.screenshot", withBackground(params), params.fullPage ? 120_000 : DEFAULT_TIMEOUT_MS, signal)) as { + const result = (await authorizedBridgeSend("page.screenshot", params, params.fullPage ? 120_000 : DEFAULT_TIMEOUT_MS, signal)) as { dataUrl?: string; + method?: string; tab?: unknown; fullPage?: boolean; dimensions?: { width: number; height: number; viewportHeight: number; dpr: number }; @@ -1800,13 +1789,13 @@ Usage rules: await writeFile(outputPath + ".json", JSON.stringify({ width, height, viewportHeight, dpr, tiles: manifest }, null, 2)); return { content: [{ type: "text", text: `Saved ${result.tiles.length} full-page tile(s) for ${width}×${height}px page. Manifest: ${outputPath}.json` }], - details: { manifest: outputPath + ".json", tiles: manifest, dimensions: result.dimensions, tab: result.tab } as unknown as Record<string, unknown>, + details: { manifest: outputPath + ".json", tiles: manifest, dimensions: result.dimensions, tab: result.tab, method: result.method } as unknown as Record<string, unknown>, }; } if (!result.dataUrl) throw new Error("Screenshot returned no dataUrl"); const base64 = result.dataUrl.replace(/^data:image\/(?:png|jpeg);base64,/, ""); await writeFile(outputPath, Buffer.from(base64, "base64")); - return { content: [{ type: "text", text: `Saved Chrome screenshot to ${outputPath}` }], details: { path: outputPath, format, tab: result.tab } }; + return { content: [{ type: "text", text: `Saved Chrome screenshot to ${outputPath}` }], details: { path: outputPath, format, tab: result.tab, method: result.method } }; }, }); @@ -1823,10 +1812,10 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean()), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.hover", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.hover", params, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: `Hovered ${params.uid ?? params.selector ?? `${params.x},${params.y}`}` }], details: { result: result as Json } }; }, }); @@ -1849,10 +1838,10 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean()), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.drag", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.drag", params, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: `Dragged from ${params.fromUid ?? params.fromSelector} to ${params.toUid ?? params.toSelector}` }], details: { result: result as Json } }; }, }); @@ -1871,10 +1860,10 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean()), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.tap", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.tap", params, DEFAULT_TIMEOUT_MS, signal); const target = params.uid ?? params.selector ?? `${params.x},${params.y}`; return { content: [{ type: "text", text: `Tapped ${target} (touch)` }], details: { result: result as Json } }; }, @@ -1894,10 +1883,10 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean()), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), }), async execute(_id, params, signal): Promise<ToolTextResult> { - const result = await authorizedBridgeSend("page.scroll", withBackground(params), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.scroll", params, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: `Scrolled dy=${params.deltaY ?? 0} dx=${params.deltaX ?? 0}` }], details: { result: result as Json } }; }, }); @@ -1914,12 +1903,12 @@ Usage rules: targetId: Type.Optional(Type.String()), urlIncludes: Type.Optional(Type.String()), titleIncludes: Type.Optional(Type.String()), - background: Type.Optional(Type.Boolean()), + background: Type.Optional(Type.Boolean({ description: BACKGROUND_PARAM_DESCRIPTION })), }), async execute(_id, params, signal, _onUpdate, ctx): Promise<ToolTextResult> { const cwd = workspaceCwd(ctx); const paths = params.paths.map((p) => resolve(cwd, p)); - const result = await authorizedBridgeSend("page.upload", withBackground({ ...params, paths }), DEFAULT_TIMEOUT_MS, signal); + const result = await authorizedBridgeSend("page.upload", { ...params, paths }, DEFAULT_TIMEOUT_MS, signal); return { content: [{ type: "text", text: `Uploaded ${paths.length} file(s) to ${params.uid ?? params.selector}` }], details: { result: result as Json } }; }, }); diff --git a/package-lock.json b/package-lock.json index 3b570c2..1e05f8e 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "pi-chrome", - "version": "0.15.47", + "version": "0.15.48", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "pi-chrome", - "version": "0.15.47", + "version": "0.15.48", "license": "MIT", "peerDependencies": { "@earendil-works/pi-coding-agent": "*", diff --git a/package.json b/package.json index bd08334..c1f3b84 100644 --- a/package.json +++ b/package.json @@ -1,8 +1,8 @@ { "name": "pi-chrome", - "version": "0.15.47", + "version": "0.15.48", "scripts": { - "test": "node test-suite/unit/csp-eval.test.mjs && node test-suite/unit/automation-target.test.mjs && node test-suite/unit/session-cleanup.test.mjs", + "test": "node test-suite/unit/csp-eval.test.mjs && node test-suite/unit/automation-target.test.mjs && node test-suite/unit/session-cleanup.test.mjs && node test-suite/unit/background-policy.test.mjs", "version": "node scripts/sync-manifest-version.js", "prepublishOnly": "node scripts/sync-manifest-version.js" }, diff --git a/test-suite/README.md b/test-suite/README.md index 47e737f..59d2388 100644 --- a/test-suite/README.md +++ b/test-suite/README.md @@ -176,6 +176,15 @@ The dashboard renders this from `manifest.json`. In brief: 40. dynamic wait/readiness 41. explicit tab lifecycle 42. strict CSP eval/snapshot via CDP (regression guard for the CSP bypass) +43. hard background: inactive-tab visibility and trusted input + +### Hard-background regression (43) + +Keep `/chrome background on`. Open challenge 43, then manually select a different tab in the **same window** and return to Pi. Target the challenge URL explicitly, arm its probe, and run its manifest recipe without selecting the target again. Inspect the saved screenshot for the colored target. The page detects visibility changes and synthetic input; it cannot establish OS focus or macOS Spaces behavior. + +Separately live-test with Chrome behind another app, a minimized window, multiple windows, and macOS Spaces. Confirm the foreground app/window and each active tab stay unchanged during background new-tab creation, viewport/full-page screenshots, and trusted input. Confirm `/chrome background off` permits explicit activation. Native prompts/page popups are outside this focus policy, not a zero-focus guarantee. + +`npm test` runs mocked regressions for every registered tool's policy, old-companion rejection, worker focus writes, CDP capture success/failure, trusted input, and full-page scroll restoration. No live Chrome is required; Node.js 22.13+ is required for TypeScript stripping. ## Design notes diff --git a/test-suite/challenges/43-hard-background.html b/test-suite/challenges/43-hard-background.html new file mode 100644 index 0000000..3fe9a34 --- /dev/null +++ b/test-suite/challenges/43-hard-background.html @@ -0,0 +1,42 @@ +<!doctype html> +<meta charset="utf-8"> +<title>43 hard background</title> +<link rel="stylesheet" href="../_style.css"> +<script src="../_lib.js"></script> +<body> +<main> + <h1>Hard-background target</h1> + <p>Setup: keep <code>/chrome background on</code>. Select another tab in this same Chrome window so this target is inactive, then return to Pi. Do not select this tab again until verification finishes.</p> + <p>Runner: target this URL explicitly. Evaluate <code>armBackgroundProbe()</code>, snapshot and screenshot with <code>background:false</code>, then click Verify with <code>domFallback:false</code> and <code>background:false</code>. These foreground requests must not activate this tab. Read <code>window.__verdict</code> without activating it.</p> + <p>This page grades tab visibility and trusted input, not OS focus or macOS Spaces. Inspect the screenshot for the colored target below; test OS focus separately. Reload to reset.</p> + <div style="background:#2463aa;color:white;padding:32px;border:16px solid #e5a323">SCREENSHOT TARGET: hard background</div> + <button id="verify">Verify hidden trusted click</button> +</main> +<script> +Challenge.init({ id: "hard-background", instructions: "inactive tab → arm → snapshot/screenshot → trusted click; no activation" }); +let armed = false; +window.armBackgroundProbe = () => { + if (armed) throw new Error("Already armed; reload to reset"); + if (!document.hidden) throw new Error("Select another tab in this window before arming"); + armed = true; + Challenge.log("armed", { visibility: document.visibilityState, focused: document.hasFocus() }); + return { armed, visibility: document.visibilityState }; +}; +document.addEventListener("visibilitychange", () => { + Challenge.log("visibilitychange", { visibility: document.visibilityState, armed }); + if (armed && !document.hidden) Challenge.fail("Target became visible after arming: background activation leaked"); +}); +for (const name of ["focus", "blur"]) { + window.addEventListener(name, () => Challenge.log(name, { visibility: document.visibilityState, focused: document.hasFocus() })); +} +document.getElementById("verify").addEventListener("click", (event) => { + Challenge.log("verify", { trusted: event.isTrusted, visibility: document.visibilityState, armed }); + const failures = []; + if (!armed) failures.push("Probe was not armed while inactive"); + if (!event.isTrusted) failures.push("Verify click isTrusted=false: synthetic input substituted"); + if (!document.hidden) failures.push("Verify target is not hidden"); + if (failures.length) Challenge.fail(...failures); + else Challenge.pass("Trusted click received while target remained hidden"); +}); +</script> +</body> diff --git a/test-suite/manifest.json b/test-suite/manifest.json index 0ebb313..4f72c85 100644 --- a/test-suite/manifest.json +++ b/test-suite/manifest.json @@ -1565,6 +1565,9 @@ "difficulty": "L2", "gate": "core", "goal": "Create a new tab, inspect it, close it, return to the parent tab, and verify tab lifecycle state.", + "prerequisites": [ + "Ask the user to run /chrome background off for this explicit activation test. Background on intentionally blocks chrome_tab activate." + ], "expected": { "synthetic": "CONDITIONAL", "trusted": "PASS", @@ -1677,5 +1680,77 @@ ], "manualBaseline": "unverified", "gradeSource": "page" + }, + { + "id": "hard-background", + "file": "challenges/43-hard-background.html", + "category": "focus-keyboard", + "difficulty": "L2", + "gate": "conditional", + "goal": "Snapshot and screenshot an inactive target with background:false, then send a trusted click without activating it. Inspect the screenshot for the colored target.", + "expected": { + "synthetic": "FAIL", + "trusted": "PASS", + "manual": "FAIL" + }, + "prerequisites": [ + "Keep /chrome background on. Reload both Pi and the updated Chrome companion.", + "Open the target, manually select a different tab in the same window, then return to Pi before arming. Do not select the target until grading finishes." + ], + "recipe": [ + { + "tool": "chrome_evaluate", + "params": { + "urlIncludes": "/43-hard-background.html", + "expression": "armBackgroundProbe()", + "background": true + } + }, + { + "tool": "chrome_snapshot", + "params": { + "urlIncludes": "/43-hard-background.html", + "background": false + } + }, + { + "tool": "chrome_screenshot", + "params": { + "urlIncludes": "/43-hard-background.html", + "path": ".pi/chrome-screenshots/hard-background.png", + "background": false + } + }, + { + "tool": "chrome_click", + "params": { + "urlIncludes": "/43-hard-background.html", + "selector": "#verify", + "background": false, + "domFallback": false, + "includeSnapshot": true + } + }, + { + "tool": "chrome_evaluate", + "params": { + "urlIncludes": "/43-hard-background.html", + "expression": "JSON.stringify({v:window.__verdict,r:window.__reason,e:window.__events})", + "background": true + } + } + ], + "requires": { + "cdp": true + }, + "tags": ["background", "visibility", "screenshot", "trusted-input"], + "notes": [ + "The page grades visibility and isTrusted, not OS focus or macOS Spaces. The runner must also inspect the screenshot for target fidelity.", + "Manual FAIL is intentional: a normal human click requires selecting the hidden tab. Synthetic FAIL detects replacing Chrome input with DOM events.", + "Selecting the target while armed causes sticky FAIL, including after PASS. Read results through background tools before returning to this tab." + ], + "flakeRisk": "medium", + "manualBaseline": "unverified", + "gradeSource": "both" } ] diff --git a/test-suite/unit/automation-target.test.mjs b/test-suite/unit/automation-target.test.mjs index f838fa6..7f46597 100644 --- a/test-suite/unit/automation-target.test.mjs +++ b/test-suite/unit/automation-target.test.mjs @@ -345,7 +345,7 @@ async function run() { ok(state.tabs.has(state.userArticle.id), "tab.close: user's active tab was NOT closed"); ok(state.windows.size === windowsBefore && state.tabs.size === tabsBefore, "tab.close: did not spawn a throwaway tab/window"); - await throwsWith(() => w.dispatch("tab.activate", { sessionKey: SK }), /no automation tab yet|Pass targetId/, "tab.activate: errors with no target/owned target"); + await throwsWith(() => w.dispatch("tab.activate", { sessionKey: SK, foreground: true }), /no automation tab yet|Pass targetId/, "tab.activate: errors with no target/owned target"); // Once an automation target exists, management actions operate on it (not on the user tab). const nav = await w.dispatch("page.navigate", { url: "https://pi.test/manage", waitUntilLoad: false, sessionKey: SK }); diff --git a/test-suite/unit/background-policy.test.mjs b/test-suite/unit/background-policy.test.mjs new file mode 100644 index 0000000..e20d0ab --- /dev/null +++ b/test-suite/unit/background-policy.test.mjs @@ -0,0 +1,469 @@ +// Runtime regressions for the Pi session policy and the real worker dispatch/input paths. +// Chrome APIs are mocked: these tests prove explicit focus calls, not OS/Spaces behavior. +import assert from "node:assert/strict"; +import fs from "node:fs"; +import vm from "node:vm"; +import path from "node:path"; +import { stripTypeScriptTypes } from "node:module"; +import { test } from "node:test"; + +const indexSource = fs.readFileSync(new URL("../../extensions/chrome-profile-bridge/index.ts", import.meta.url), "utf8"); +const workerSource = fs.readFileSync(new URL("../../extensions/chrome-profile-bridge/browser-extension/service_worker.js", import.meta.url), "utf8"); +const clone = (value) => JSON.parse(JSON.stringify(value)); +function section(start, end) { + const from = indexSource.indexOf(start); + const to = indexSource.indexOf(end, from); + assert.ok(from >= 0 && to > from, `Missing source section: ${start}`); + return indexSource.slice(from, to); +} + +// Load the shipped policy, existing command handler, and actual tool registrations. Only the +// bridge/Pi UI/typebox/formatting/filesystem boundaries are replaced; no live server is opened. +function piHarness({ session = "alpha", send } = {}) { + const calls = [], tools = new Map(), notices = [], writes = []; + let authorized = true; + const ctx = { key: `session:${session}`, title: `Pi Session: ${session}`, cwd: "/fixture", ui: { notify: (...args) => notices.push(args) } }; + const bridge = { + connected: true, status: () => ({}), + async send(action, params, timeout, signal) { + calls.push({ action, params: clone(params), timeout, signal }); + if (signal?.aborted) throw new Error("Chrome command aborted"); + if (send) return send(action, params, timeout, signal); + if (action === "tab.version") return { capabilities: { hardBackground: true } }; + if (action === "tab.list") return []; + if (action.startsWith("page.screenshot")) return { dataUrl: "data:image/png;base64,dGVzdA==", method: "cdp" }; + return {}; + }, + }; + const Type = new Proxy({}, { get: () => (value = {}) => value }); + const sandbox = { + console, Buffer, Type, + Date, + DEFAULT_HOST: "127.0.0.1", DEFAULT_PORT: 17318, DEFAULT_TIMEOUT_MS: 30000, MAX_ELEMENTS: 80, + BACKGROUND_PARAM_DESCRIPTION: "background policy", + ChromeProfileBridge: function () { return bridge; }, + requireChromeControlAuthorized() { if (!authorized) throw new Error("Chrome control locked"); }, + sessionCtx: ctx, sessionKeyFor: (c) => c?.key, sessionGroupTitle: (c) => c.title, + chromeToolsRegistered: false, StringEnum: () => ({}), + tabActionValues: [], snapshotModeValues: [], waitForValues: [], imageFormatValues: [], + safeJson: JSON.stringify, truncateText: (s) => s, formatChromeSnapshot: JSON.stringify, + formatChromeInspect: JSON.stringify, summarizeActionResult: () => "", formatIncludedSnapshotText: (_r, text) => text, + workspaceCwd: () => ctx.cwd, ...path, + mkdir: async () => {}, writeFile: async (...args) => writes.push(args), + }; + const registrations = indexSource.slice(indexSource.indexOf("function registerChromeTools(pi:"), indexSource.lastIndexOf("\n}")); + vm.runInNewContext(stripTypeScriptTypes([ + section("const bridge = new ChromeProfileBridge(", "\n\tlet chromeAuthorizedUntil:"), + section("const authorizedBridgeSend =", '\n\tpi.on("session_start",'), + section("const BACKGROUND_DESC:", "\n\tconst authorizeFor ="), + registrations, + "globalThis.send = authorizedBridgeSend; globalThis.background = backgroundHandler;", + ].join("\n")), sandbox); + sandbox.registerChromeTools({ registerTool: (tool) => tools.set(tool.name, tool) }); + return { + calls, tools, notices, writes, ctx, + send: async (...args) => sandbox.send(...args), + background: (arg) => sandbox.background(ctx, arg), + tool: (name, params = {}, signal) => tools.get(name).execute("test", params, signal, undefined, ctx), + authorize: (value) => { authorized = value; }, + }; +} + +const flagCases = [ + [{}, false], [{ background: false }, false], [{ background: true }, true], + [{ foreground: true }, false], [{ foreground: false }, true], + [{ background: false, foreground: false }, false], [{ background: true, foreground: true }, true], +]; + +test("session background on overrides all per-call/legacy foreground flags; off preserves per-call background", async () => { + const h = piHarness(); + for (const mode of ["on", "off"]) { + await h.background(mode); + for (const [flags, requestedBackground] of flagCases) { + const params = Object.freeze({ ...flags, targetId: "2" }); + await h.send("page.snapshot", params); + const wire = h.calls.at(-1).params; + assert.equal(wire.background, mode === "on" || requestedBackground); + assert.equal(wire.foreground, !wire.background); + assert.equal(wire.sessionKey, "session:alpha"); + assert.equal(wire.sessionGroupTitle, "Pi Session: alpha"); + assert.equal(wire.joinSessionGroup, true); + assert.deepEqual(params, { ...flags, targetId: "2" }, "caller params not mutated"); + } + } +}); + +test("every registered page tool, tab.new, and chrome_launch(url) use the central policy", async () => { + const h = piHarness(); + const cases = [ + ["chrome_launch", { url: "https://example.test" }, "tab.new"], + ["chrome_tab", { action: "new", group: false, groupTitle: "other" }, "tab.new"], + ["chrome_snapshot", {}, "page.snapshot"], ["chrome_find", { query: "button" }, "page.snapshot"], + ["chrome_inspect", { uid: "el-1" }, "page.inspect"], ["chrome_navigate", { url: "https://example.test" }, "page.navigate"], + ["chrome_evaluate", { expression: "1" }, "page.evaluate"], ["chrome_click", { uid: "el-1" }, "page.click"], + ["chrome_type", { text: "abc" }, "page.type"], ["chrome_fill", { text: "abc" }, "page.fill"], + ["chrome_key", { key: "Enter" }, "page.key"], ["chrome_wait_for", { kind: "expression", value: "true" }, "page.waitFor"], + ["chrome_list_console_messages", {}, "page.console.list"], ["chrome_list_network_requests", {}, "page.network.list"], + ["chrome_get_network_request", { requestId: "1" }, "page.network.get"], ["chrome_screenshot", {}, "page.screenshot"], + ["chrome_hover", {}, "page.hover"], ["chrome_drag", {}, "page.drag"], ["chrome_tap", {}, "page.tap"], + ["chrome_scroll", {}, "page.scroll"], ["chrome_upload_file", { paths: ["fixture.txt"] }, "page.upload"], + ]; + for (const [name, params, action] of cases) { + await h.tool(name, { ...params, background: false, foreground: true }); + const call = h.calls.at(-1); + assert.equal(call.action, action === "tab.new" || action === "page.screenshot" ? `${action}.background` : action, name); + assert.equal(call.params.foreground, false, name); + assert.equal(call.params.background, true, name); + if (action === "tab.new") assert.equal(call.params.groupTitle, "Pi Session: alpha"); + } + assert.equal(h.writes[0][1].toString(), "test", "screenshot tool still writes decoded bytes"); +}); + +test("activation is rejected before dispatch; existing on/off/toggle/status commands suffice", async () => { + const h = piHarness(); + await assert.rejects(h.tool("chrome_tab", { action: "activate", targetId: "2", background: false }), /background mode.*\/chrome background off/); + assert.equal(h.calls.length, 0); + await h.background("off"); + await h.tool("chrome_tab", { action: "activate", targetId: "2" }); + assert.equal(h.calls.at(-1).params.foreground, true); + await h.background("toggle"); + await h.background("status"); + assert.match(h.notices.at(-1)[0], /on.*Hard background/); + for (const arg of ["lock", "unlock", "invalid"]) { + await h.background(arg); + assert.match(h.notices.at(-1)[0], /Unknown background setting/); + await assert.rejects(h.send("tab.activate", {}), /background mode/); + } + await h.background(""); + await h.send("tab.activate", {}); + for (const arg of ["true", "1", "on"]) { + await h.background(arg); + await assert.rejects(h.send("tab.activate", {}), /background mode/); + } + for (const arg of ["false", "0", "off"]) { + await h.background(arg); + await h.send("tab.activate", {}); + } +}); + +test("older companions reject background-only wire actions; launch and screenshots never retry unsafe actions", async () => { + const h = piHarness({ send: async (action) => { throw new Error(`Unknown action: ${action}`); } }); + for (const [tool, params] of [ + ["chrome_tab", { action: "new" }], ["chrome_launch", { url: "https://example.test" }], + ["chrome_screenshot", {}], ["chrome_screenshot", { fullPage: true }], + ]) { + await assert.rejects(h.tool(tool, params), /Reload Pi Chrome Connector/); + } + assert.deepEqual(h.calls.map((c) => c.action), ["tab.new.background", "tab.new.background", "page.screenshot.background", "page.screenshot.background"]); +}); + +test("transport failures propagate without fallback; single dispatch preserves authorization/timeout/signal", async () => { + const offline = piHarness({ send: async () => { throw new Error("offline"); } }); + await assert.rejects(offline.send("tab.new", {}), /offline/); + assert.equal(offline.calls.length, 1); + const h = piHarness(), controller = new AbortController(); + await h.send("tab.new", {}, 900, controller.signal); + assert.equal(h.calls.length, 1); + assert.equal(h.calls[0].timeout, 900); + assert.equal(h.calls[0].signal, controller.signal); + controller.abort(); + await assert.rejects(h.send("tab.new", {}, 900, controller.signal), /Chrome command aborted/); + h.authorize(false); + await assert.rejects(h.send("tab.new", {}), /Chrome control locked/); + assert.equal(h.calls.length, 2, "locked call does not reach bridge"); +}); + +test("shared-owner sessions retain independent policies; per-call background still works with mode off", async () => { + const sharedCalls = []; + const send = async (action, params) => { sharedCalls.push({ action, params }); return {}; }; + const a = piHarness({ session: "A", send }), b = piHarness({ session: "B", send }); + await b.background("off"); + await a.send("tab.new", { background: false }); + await b.send("tab.new", {}); + assert.equal(sharedCalls[0].params.sessionKey, "session:A"); + assert.equal(sharedCalls[0].params.foreground, false); + assert.equal(sharedCalls[0].action, "tab.new.background"); + assert.equal(sharedCalls[1].params.sessionKey, "session:B"); + assert.equal(sharedCalls[1].params.foreground, true); + assert.equal(sharedCalls[1].action, "tab.new"); + await b.send("tab.new", { background: true }); + assert.equal(sharedCalls[2].action, "tab.new.background"); + assert.equal(sharedCalls[2].params.foreground, false); +}); + +function workerHarness({ withWindows = true } = {}) { + const calls = []; + const tabs = new Map([ + [1, { id: 1, windowId: 1, active: true, groupId: -1, url: "https://user.test/", title: "User" }], + [2, { id: 2, windowId: 1, active: false, groupId: -1, url: "https://target.test/", title: "Target" }], + ]); + let focusedWindow = 1, nextTab = 3, nextWindow = 2; + const storage = {}, groups = new Map(); + const listener = { addListener() {}, removeListener() {} }; + const chrome = { + runtime: { id: "test", getManifest: () => ({ version: "0.0.0" }), onInstalled: listener, onStartup: listener }, + alarms: { onAlarm: listener, create() {} }, action: { onClicked: listener }, webNavigation: { onCommitted: listener }, + storage: { session: { get: async (key) => ({ [key]: storage[key] }), set: async (value) => Object.assign(storage, value) } }, + tabs: { + onUpdated: listener, + query: async (query = {}) => [...tabs.values()].filter((t) => (!query.active || t.active) && (query.windowId === undefined || query.windowId === t.windowId)).map(clone), + get: async (id) => { if (!tabs.has(id)) throw new Error("No tab"); return clone(tabs.get(id)); }, + create: async (params) => { + calls.push({ action: "tabs.create", params }); + const tab = { id: nextTab++, windowId: params.windowId ?? 1, active: params.active, url: params.url, groupId: -1 }; + if (tab.active) for (const t of tabs.values()) if (t.windowId === tab.windowId) t.active = false; + tabs.set(tab.id, tab); + return clone(tab); + }, + update: async (id, params) => { + calls.push({ action: "tabs.update", id, params }); + const tab = tabs.get(id); + if (params.active) for (const t of tabs.values()) if (t.windowId === tab.windowId) t.active = false; + Object.assign(tab, params); + return clone(tab); + }, + remove: async (id) => tabs.delete(id), + group: async ({ groupId, tabIds }) => { + const id = groupId ?? groups.size + 1; + if (!groups.has(id)) groups.set(id, { id, windowId: tabs.get(tabIds[0]).windowId }); + for (const tid of tabIds) tabs.get(tid).groupId = id; + return id; + }, + ungroup: async (id) => { tabs.get(id).groupId = -1; }, + captureVisibleTab: async () => { calls.push({ action: "captureVisibleTab" }); throw new Error("activation-based capture forbidden"); }, + }, + tabGroups: { + query: async ({ windowId } = {}) => [...groups.values()].filter((g) => windowId === undefined || g.windowId === windowId).map(clone), + get: async (id) => groups.get(id), + update: async (id, props) => Object.assign(groups.get(id), props), + }, + windows: { + update: async (id, params) => { calls.push({ action: "windows.update", id, params }); if (params.focused) focusedWindow = id; }, + }, + debugger: { + onDetach: listener, getTargets: (cb) => cb([]), attach: async () => {}, detach: async () => {}, + sendCommand: (target, method, params, callback) => { + calls.push({ action: "cdp", target, method, params }); + try { callback(h.cdpResult(method, params)); } + catch (error) { chrome.runtime.lastError = { message: error.message }; callback(); chrome.runtime.lastError = undefined; } + }, + }, + scripting: { executeScript: async () => [{ result: { ok: true, value: { elements: [], url: "https://target.test/" } } }] }, + }; + if (withWindows) chrome.windows.create = async (params) => { + calls.push({ action: "windows.create", params }); + const id = nextWindow++; + if (params.focused) focusedWindow = id; + const tab = { id: nextTab++, windowId: id, active: true, url: params.url, groupId: -1 }; + tabs.set(tab.id, tab); + return { id, tabs: [clone(tab)] }; + }; + const w = { + chrome, console, setTimeout, clearTimeout, setInterval: () => 0, clearInterval() {}, + fetch: async () => { throw new Error("no network in unit tests"); }, navigator: { userAgent: "unit-test" }, + }; + w.self = w; + vm.runInNewContext(workerSource, w); + w.sleep = async () => {}; // Skip human timing, not CDP dispatch. + w.resolveTargetInTab = async () => ({ found: true, x: 40, y: 50, tag: "BUTTON" }); + const h = { + w, chrome, calls, tabs, focusedWindow: () => focusedWindow, + cdpResult: (method) => method === "Page.captureScreenshot" ? { data: "c2NyZWVuc2hvdA==" } + : method === "DOM.requestNode" ? { nodeId: 1 } + : { result: { type: "boolean", value: true, objectId: "file-input" } }, + }; + return h; +} +function assertNoFocus(h) { + assert.equal(h.focusedWindow(), 1); + assert.equal(h.tabs.get(1).active, true); + assert.ok(!h.calls.some((c) => c.action === "windows.update" || c.action === "captureVisibleTab" || (c.action === "tabs.update" && c.params.active) || (c.action === "tabs.create" && c.params.active))); +} + +test("Pi-to-worker background aliases preserve grouping, screenshot output, and ownership-aware cleanup", async () => { + const h = workerHarness(); + const pi = piHarness({ send: (action, params) => h.w.dispatch(action, params) }); + const opened = await pi.tool("chrome_tab", { action: "new", group: false, groupTitle: "wrong", background: false }); + assert.equal(opened.details.result.group.title, "Pi Session: alpha"); + const launched = await pi.tool("chrome_launch", { url: "https://fixture.test" }); + const shot = await pi.tool("chrome_screenshot", { targetId: "2", background: false }); + assert.equal(shot.details.method, "cdp"); + assert.equal(pi.writes[0][1].toString(), "screenshot"); + assertNoFocus(h); + await h.w.dispatch("automation.cleanup", { sessionKey: "session:alpha" }); + assert.ok(!h.tabs.has(opened.details.result.tab.id)); + assert.ok(!h.tabs.has(launched.details.result.tab.id)); + assert.ok(h.tabs.has(1) && h.tabs.has(2), "user/adopted tabs survive cleanup"); + assert.equal(h.tabs.get(2).groupId, -1, "adopted tab is safely ungrouped"); +}); + +test("worker advertises support, keeps new tabs inactive, and rejects activation before target resolution", async () => { + const h = workerHarness(); + assert.equal((await h.w.dispatch("tab.version", {})).capabilities.hardBackground, true); + for (const flags of [{}, { foreground: false }, { background: true, foreground: true }]) { + const result = await h.w.dispatch("tab.new", { ...flags, sessionKey: "alpha", groupTitle: "Pi Session: alpha" }); + assert.equal(result.tab.active, false); + await assert.rejects(h.w.dispatch("tab.activate", { ...flags, targetId: "missing" }), /background mode/); + } + assertNoFocus(h); + const active = await h.w.dispatch("tab.activate", { targetId: "2", foreground: true, background: false }); + assert.equal(active.id, 2); + assert.equal(active.active, true); + const opened = await h.w.dispatch("tab.new", { foreground: true }); + assert.equal(opened.tab.active, true); +}); + +test("background-only worker aliases force safe behavior even with foreground flags", async () => { + const h = workerHarness(); + const opened = await h.w.dispatch("tab.new.background", { foreground: true, background: false }); + assert.equal(opened.tab.active, false); + const shot = await h.w.dispatch("page.screenshot.background", { targetId: "2", foreground: true, background: false }); + assert.equal(shot.method, "cdp"); + assertNoFocus(h); +}); + +test("implicit automation windows stay unfocused; tab fallback stays inactive", async () => { + for (const withWindows of [true, false]) { + const h = workerHarness({ withWindows }); + const tab = await h.w.dispatch("page.navigate", { url: "https://fixture.test", waitUntilLoad: false, background: true, foreground: true, sessionKey: "alpha" }); + assert.notEqual(tab.id, 1); + assertNoFocus(h); + assert.ok(h.calls.filter((c) => c.action === "windows.create").every((c) => c.params.focused === false)); + } +}); + +test("all worker page/input paths veto conflicting foreground flags without replacing trusted input", async () => { + const h = workerHarness(); + const actions = [ + ["navigate", { url: "https://target.test/", waitUntilLoad: false }], ["snapshot", {}], ["inspect", { uid: "el-1" }], + ["evaluate", { expression: "true" }], ["waitFor", { kind: "expression", value: "true" }], + ["console.list", {}], ["network.list", {}], ["network.get", { requestId: "1" }], + ["click", { x: 40, y: 50, includeSnapshot: true }], ["fill", { selector: "input", text: "a", submit: true }], + ["type", { selector: "input", text: "a", pressEnter: true }], ["key", { key: "Enter" }], ["hover", { x: 40, y: 50 }], + ["drag", { fromX: 40, fromY: 50, toX: 80, toY: 90 }], ["tap", { x: 40, y: 50 }], + ["scroll", { deltaY: 10 }], ["upload", { selector: "input", paths: ["/fixture.txt"] }], + ]; + for (const [action, params] of actions) { + const result = await h.w.dispatch(`page.${action}`, { targetId: "2", ...params, background: true, foreground: true, domFallback: false }); + if (["click", "fill", "type", "key", "hover", "drag", "tap", "scroll"].includes(action)) assert.equal((result.result ?? result).input, "chrome", action); + assertNoFocus(h); + } + const methods = new Set(h.calls.filter((c) => c.action === "cdp").map((c) => c.method)); + for (const method of ["Input.dispatchMouseEvent", "Input.dispatchKeyEvent", "Input.dispatchTouchEvent", "DOM.setFileInputFiles"]) assert.ok(methods.has(method), method); + h.chrome.debugger = undefined; + for (const action of ["click", "fill"]) { + await assert.rejects(h.w.dispatch(`page.${action}`, { targetId: "2", selector: "input", text: "a", background: true, domFallback: false }), /chrome.debugger API unavailable/); + } + assertNoFocus(h); +}); + +test("CDP PNG/JPEG screenshots preserve active tabs and pass viewport/quality options", async () => { + for (const targetId of ["1", "2"]) { + for (const options of [{}, { format: "png", quality: 70 }, { format: "jpeg", quality: 0 }, { format: "jpeg", quality: 83 }]) { + const h = workerHarness(); + const result = await h.w.dispatch("page.screenshot", { targetId, background: true, foreground: true, ...options }); + const capture = h.calls.find((c) => c.method === "Page.captureScreenshot"); + const format = options.format ?? "png"; + assert.equal(result.dataUrl, `data:image/${format};base64,c2NyZWVuc2hvdA==`); + assert.equal(result.method, "cdp"); + assert.equal(result.tab.id, Number(targetId)); + assert.equal(capture.target.tabId, Number(targetId)); + assert.deepEqual(clone(capture.params), { format, fromSurface: true, captureBeyondViewport: false, ...(format === "jpeg" ? { quality: options.quality } : {}) }); + assertNoFocus(h); + } + } +}); + +test("screenshot failures never fall back to activation/visible-tab capture", async () => { + for (const failure of ["attach", "capture", "empty", "missing"]) { + const h = workerHarness(); + if (failure === "attach") h.chrome.debugger.attach = async () => { throw new Error("permission denied"); }; + else h.cdpResult = () => { + if (failure === "capture") throw new Error("capture failed"); + return failure === "empty" ? { data: "" } : undefined; + }; + await assert.rejects(h.w.dispatch("page.screenshot", { targetId: "2", background: true }), /no tab-activation fallback/); + assertNoFocus(h); + } +}); + +test("screenshot does not restore/override a user tab switch during capture; watch mode explicitly activates", async () => { + const h = workerHarness(); + h.cdpResult = () => { + h.tabs.get(1).active = false; + h.tabs.get(2).active = true; // Human selection while CDP is pending. + return { data: "dGVzdA==" }; + }; + const result = await h.w.dispatch("page.screenshot", { targetId: "1", background: true }); + assert.equal(result.tab.id, 1); + assert.equal(h.tabs.get(2).active, true); + assert.ok(!h.calls.some((c) => c.action === "tabs.update" || c.action === "windows.update")); + + const watch = workerHarness(); + const shot = await watch.w.dispatch("page.screenshot", { targetId: "2", foreground: true, background: false }); + assert.equal(shot.tab.active, true); + assert.equal(watch.calls.filter((c) => c.action === "tabs.update" && c.params.active).length, 1); + assert.equal(watch.calls.filter((c) => c.action === "windows.update").length, 1); + assert.ok(!watch.calls.some((c) => c.action === "captureVisibleTab")); +}); + +test("challenge 43 grader rejects activation/synthetic input and retains sticky failures", () => { + const lib = fs.readFileSync(new URL("../_lib.js", import.meta.url), "utf8"); + const html = fs.readFileSync(new URL("../challenges/43-hard-background.html", import.meta.url), "utf8"); + const script = html.match(/<script>([\s\S]*?)<\/script>/)[1]; + function page() { + const listeners = new Map(); + const add = (name, fn) => listeners.set(name, fn); + const document = { + hidden: true, get visibilityState() { return this.hidden ? "hidden" : "visible"; }, hasFocus: () => false, + body: { insertBefore() {} }, createElement: () => ({ style: {} }), + getElementById: (id) => id === "verify" ? { addEventListener: add } : null, + addEventListener: add, + }; + const p = { document, addEventListener: add, location: { search: "" }, URLSearchParams, + performance: { now: () => 1 }, localStorage: { setItem() {} } }; + p.window = p; + vm.runInNewContext(`${lib}\n${script}`, p); + return { p, document, fire: (name, event = {}) => listeners.get(name)(event) }; + } + const visible = page(); + visible.document.hidden = false; + assert.throws(() => visible.p.armBackgroundProbe(), /Select another tab/); + const good = page(); + good.p.armBackgroundProbe(); + assert.throws(() => good.p.armBackgroundProbe(), /Already armed/); + good.fire("click", { isTrusted: true }); + assert.equal(good.p.__verdict, "PASS"); + good.document.hidden = false; + good.fire("visibilitychange"); + assert.equal(good.p.__verdict, "FAIL", "late activation must invalidate an earlier pass"); + const synthetic = page(); + synthetic.p.armBackgroundProbe(); + synthetic.fire("click", { isTrusted: false }); + synthetic.fire("click", { isTrusted: true }); + assert.equal(synthetic.p.__verdict, "FAIL", "trusted retry cannot hide synthetic input"); +}); + +test("full-page tiles remain CDP-only, pin the resolved target, and restore both scroll axes after success/failure", async () => { + for (const fail of [false, true]) { + const h = workerHarness(), scrolls = []; + h.w.executeInTab = async (params, fn, args) => { + assert.equal(params.targetId, 2); + assert.equal(params.foreground, false); + if (fn.name === "captureFullPageTiles") return { width: 800, height: 1200, viewportHeight: 600, dpr: 2, originalScrollY: 73, originalScrollX: 31, tiles: [{ y: 0, scrollY: 0 }, { y: 600, scrollY: 600 }] }; + scrolls.push(clone(args)); + }; + let captures = 0; + h.cdpResult = () => { if (++captures === 2 && fail) throw new Error("tile failed"); return { data: "dGlsZQ==" }; }; + const shot = h.w.dispatch("page.screenshot", { urlIncludes: "target.test", fullPage: true, background: true, foreground: true }); + if (fail) await assert.rejects(shot, /tile failed/); + else { + const result = await shot; + assert.equal(result.method, "cdp"); + assert.equal(result.tiles.length, 2); + assert.equal(result.dimensions.dpr, 2); + } + assert.deepEqual(scrolls, [[0], [600], [73, 31]]); + assertNoFocus(h); + } +}); |
