diff options
| -rw-r--r-- | README.md | 4 | ||||
| -rw-r--r-- | docs/ARCHITECTURE.md | 6 | ||||
| -rw-r--r-- | docs/FAQ.md | 2 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/browser-extension/service_worker.js | 145 | ||||
| -rw-r--r-- | extensions/chrome-profile-bridge/index.ts | 54 | ||||
| -rw-r--r-- | package.json | 2 | ||||
| -rw-r--r-- | test-suite/unit/automation-target.test.mjs | 4 | ||||
| -rw-r--r-- | test-suite/unit/background-policy.test.mjs | 14 | ||||
| -rw-r--r-- | test-suite/unit/chrome-command.test.mjs | 3 | ||||
| -rw-r--r-- | test-suite/unit/hidden-input.test.mjs | 98 | ||||
| -rw-r--r-- | test-suite/unit/shared-window.test.mjs | 61 | ||||
| -rw-r--r-- | test-suite/unit/user-window.test.mjs | 119 |
12 files changed, 326 insertions, 186 deletions
@@ -77,9 +77,9 @@ Run `/chrome revoke` when finished. Use `/chrome authorize` again whenever you w ## Safety - **Per-session approval.** Pi's Chrome tools require `/chrome authorize`. `/chrome revoke` locks them and requests cleanup of that session's owned automation tabs. Cleanup preserves existing user tabs. -- **Separate targets by default.** Page actions without an explicit target use a session-owned automation window or tab. The agent can deliberately target an existing tab when your task calls for it. +- **Separate targets by default.** Page actions without an explicit target use a session-owned automation tab, opened inactive in your window inside the session's tab group (collapsed), or in a new window only when none is open. The agent can deliberately target an existing tab when your task calls for it. - **Local transport, not a sandbox.** The bridge binds to `127.0.0.1:17318` and rejects browser-origin command requests. It does not authenticate arbitrary non-browser local callers; it is not protection against hostile processes on your machine. -- **Background mode:** `/chrome background on` (default) blocks pi-chrome tools from directly bringing Chrome to the front or switching your selected tab. Use `/chrome background off` to allow those actions, and `/chrome background status` to check the setting. +- **Background mode:** off by default (fork). Calls work in the background unless an agent passes `background:false` to bring its tab forward; inactive tabs take clicks, typing and screenshots without that. `/chrome background on` blocks bringing Chrome to the front or switching your selected tab outright; `/chrome background status` shows the setting. ### Limits diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index c35cdd8..8d4c72b 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -46,15 +46,15 @@ To point pi-chrome at an existing tab, pass `targetId`, `urlIncludes`, or `title ## Background mode -The existing background setting is a hard session policy, enabled by default. No separate lock/unlock command is needed. +The background setting is a hard session policy when on. In this fork it is off by default: every call still defaults to background, and an agent may pass `background:false` (or call `chrome_tab activate`) when it needs a tab showing. An inactive tab takes trusted CDP input and renders screenshots (measured 2026-11-14), so that is rarely needed. ```text /chrome background on # enforce no explicit window focus/tab activation -/chrome background off # allow foreground/watch mode +/chrome background off # default: agents may bring a tab forward when they ask ``` - With background on, per-call `background:false` and legacy `foreground:true` cannot override the policy. `chrome_tab activate` errors with instructions to ask the user to turn background off. -- With background off, calls may focus Chrome; per-call `background:true` still avoids explicit focus/tab activation. +- With background off, calls stay in the background unless they pass `background:false`; `chrome_tab activate` defaults to foreground. - Policy is applied in `authorizedBridgeSend` for every tool, including `chrome_launch(url)`, tab creation, and tools without a background parameter. Each session sends its own effective `background`/`foreground` flags through the shared bridge. - All worker window-focus/tab-activation writes go through a guarded helper. Background tab creation uses `active:false`; implicit automation windows remain `focused:false`. - Screenshots use CDP `Page.captureScreenshot` with `fromSurface:true` and `captureBeyondViewport:false`. They target a tab, not whichever tab happens to be visible. There is no `captureVisibleTab`/activation fallback on debugger or capture failure. PNG/JPEG output is unchanged; full-page capture retains tiles plus a JSON manifest and restores scroll position best-effort, including on failure. diff --git a/docs/FAQ.md b/docs/FAQ.md index bf46a51..cfa2cfa 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -36,7 +36,7 @@ Subagent sessions that run inside an authorized Pi process (for example with pi- ## Does pi-chrome navigate my current tab? -No. The first chrome_* action that has no explicit target opens a **dedicated automation window** that pi-chrome owns (falling back to a dedicated tab only if a separate window can't be created), and reuses it for the rest of the session. Your existing tabs and windows are never reused or overwritten. Pass `targetId`/`urlIncludes`/`titleIncludes` to deliberately act on a tab you already have open. +No. The first chrome_* action that has no explicit target opens a **dedicated automation tab** that pi-chrome owns, inactive, in your window and in the session's own (collapsed) tab group, and reuses it for the rest of the session. A window is created only when none is open. Your existing tabs and windows are never reused or overwritten. Pass `targetId`/`urlIncludes`/`titleIncludes` to deliberately act on a tab you already have open. The window survives `/reload` and Chrome service-worker restarts because ownership is tracked by id and mirrored to `chrome.storage.session`. It is closed when you run `/chrome revoke` and on real session end (not on `/reload`); cleanup only ever closes that session's own window/tab. diff --git a/extensions/chrome-profile-bridge/browser-extension/service_worker.js b/extensions/chrome-profile-bridge/browser-extension/service_worker.js index 9ac247c..dc9f45d 100644 --- a/extensions/chrome-profile-bridge/browser-extension/service_worker.js +++ b/extensions/chrome-profile-bridge/browser-extension/service_worker.js @@ -226,36 +226,34 @@ function isPiChromeOwnedTarget(tabId, sessionKey) { return false; } -// Create a fresh automation target for `sessionKey`. If this session already has a tab group, -// create the tab inside that group's window so one Pi session keeps one Chrome tab group (Chrome -// groups cannot span windows). If it has none, give the session its own window: a window shows -// only one tab, so sessions sharing a window would hide each other's tabs, and a hidden tab gets no -// real input and blank screenshots (measured 2026-11-08). Fall back to a tab. When the tab is -// created in a pre-existing window, leave windowId unset so cleanup only closes our tab. +// The window agents' tabs go in: the user's last focused normal window, so every session's tabs +// sit together in one window (as tab groups) instead of a window each. An inactive tab still takes +// trusted input and renders screenshots (measured 2026-11-14), so sharing a window costs nothing. +// Null when no normal window is open; the caller then creates one. +async function agentWindowId() { + if (typeof chrome.windows?.getLastFocused !== "function") return null; + const win = await chrome.windows.getLastFocused({ windowTypes: ["normal"] }).catch(() => null); + if (win && typeof win.id === "number" && !win.incognito) return win.id; + const all = typeof chrome.windows.getAll === "function" ? await chrome.windows.getAll({ windowTypes: ["normal"] }).catch(() => []) : []; + const other = all.find((w) => typeof w.id === "number" && !w.incognito); + return other ? other.id : null; +} + +// Create a fresh automation target for `sessionKey`: an inactive tab in the session's group's +// window if it has a group (Chrome groups cannot span windows), else in the user's window. A new +// window only when Brave has none. When the tab goes in a pre-existing window, windowId stays unset. async function createAutomationTarget(sessionKey, groupTitle) { const existingGroup = groupTitle ? await findGroupRecordByTitle(groupTitle) : null; - if (existingGroup && typeof existingGroup.windowId === "number") { - const tab = await chrome.tabs.create({ url: "about:blank", active: false, windowId: existingGroup.windowId }); - automationTargets.set(sessionKey, { windowId: undefined, tabId: typeof tab.id === "number" ? tab.id : undefined }); + const windowId = existingGroup && typeof existingGroup.windowId === "number" ? existingGroup.windowId : await agentWindowId(); + if (windowId === null && typeof chrome.windows?.create === "function") { + const win = await chrome.windows.create({ url: "about:blank", focused: false }); + const created = win && Array.isArray(win.tabs) ? win.tabs[0] : undefined; + if (!created || typeof created.id !== "number") throw new Error("Chrome created a window without a tab"); + automationTargets.set(sessionKey, { windowId: typeof win.id === "number" ? win.id : undefined, tabId: created.id }); await persistAutomationTargets(); - return tab; + return created; } - if (chrome.windows && typeof chrome.windows.create === "function") { - try { - const win = await chrome.windows.create({ url: "about:blank", focused: false }); - const created = win && Array.isArray(win.tabs) ? win.tabs[0] : undefined; - if (created && typeof created.id === "number") { - automationTargets.set(sessionKey, { windowId: typeof win.id === "number" ? win.id : undefined, tabId: created.id }); - await persistAutomationTargets(); - return created; - } - } catch { - // Window creation can fail (policy, headless, etc.); fall back to a dedicated tab below. - } - } - // Tab fallback: the tab lives in a pre-existing (user/shared) window we did NOT create, so we - // must leave windowId unset — cleanup then closes only our tab, never the user's window. - const tab = await chrome.tabs.create({ url: "about:blank", active: false }); + const tab = await chrome.tabs.create({ url: "about:blank", active: false, ...(windowId !== null ? { windowId } : {}) }); automationTargets.set(sessionKey, { windowId: undefined, tabId: typeof tab.id === "number" ? tab.id : undefined }); await persistAutomationTargets(); return tab; @@ -442,7 +440,12 @@ async function attachDebugger(tabId) { const entry = { detachAt: Date.now() + INPUT_IDLE_DETACH_MS, pointer: { x: 120 + Math.random() * 200, y: 80 + Math.random() * 120 }, debuggee: attachedDebuggee || { tabId } }; attachedTabs.set(tabId, entry); // Page events are what tell us a page opened a dialog; without them nobody answers it. - await cdpRaw(tabId, "Page.enable", {}).catch(() => undefined); + // Agents' tabs are usually inactive, so also let the page believe it has focus: menus and pickers + // that close on blur stay open. Ends when the debugger detaches. + await Promise.all([ + cdpRaw(tabId, "Page.enable", {}).catch(() => undefined), + cdpRaw(tabId, "Emulation.setFocusEmulationEnabled", { enabled: true }).catch(() => undefined), + ]); return entry; } @@ -480,10 +483,11 @@ async function inputDebug(params) { }; } -// Chrome drops trusted CDP input to hidden pages: mouse presses and keys are silently ignored and -// mouse moves can hang waiting for a frame. A page is hidden when its tab is inactive, its window is -// minimized, or (macOS) its window became a native window-tab behind a full-screen window. Check -// before sending input so the caller gets an actionable error instead of a false success. +// A page is hidden when its tab is inactive, its window is minimized, or (macOS) its window is +// covered or on another Space. Measured 2026-11-14 in Brave: an inactive tab still takes trusted +// CDP clicks, key presses, text and mouse moves, and screenshots render it live; only mouseWheel +// hangs waiting for a frame that never comes, so chrome.scroll scrolls hidden pages by script. +// A minimized window is untested, so input there still fails up front with an actionable error. async function pageVisibilityState(tabId) { try { const evaluated = await cdp(tabId, "Runtime.evaluate", { expression: "document.visibilityState", returnByValue: true }); @@ -501,8 +505,20 @@ function hiddenTabReason(tab, win) { return "its window is covered by another window or is on another Space (Chrome treats fully covered windows as hidden)"; } +// The tab and window when trusted input to the page may be dropped (hidden in a minimized window), +// else null. +async function inputRefusingWindow(tab) { + if ((await pageVisibilityState(tab.id)) !== "hidden") return null; + const fresh = await chrome.tabs.get(tab.id).catch(() => tab); + const win = typeof chrome.windows?.get === "function" ? await chrome.windows.get(fresh?.windowId ?? tab.windowId).catch(() => null) : null; + return win?.state === "minimized" ? { fresh, win } : null; +} + async function assertTabVisibleForInput(tab, params, label) { - let state = await pageVisibilityState(tab.id); + const refusing = await inputRefusingWindow(tab); + if (!refusing) return; + const { fresh, win } = refusing; + let state = "hidden"; if (state !== "hidden") return; if (foregroundRequested(params)) { // bringToFront just ran; give Chrome a moment to report the page visible. @@ -512,11 +528,9 @@ async function assertTabVisibleForInput(tab, params, label) { } if (state !== "hidden") return; } - const fresh = await chrome.tabs.get(tab.id).catch(() => tab); - const win = typeof chrome.windows?.get === "function" ? await chrome.windows.get(fresh?.windowId ?? tab.windowId).catch(() => null) : null; const error = new Error( - `${label}: tab ${tab.id} is hidden because ${hiddenTabReason(fresh, win)}. Chrome ignores trusted input to hidden pages, so the events would be dropped. ` + - `Pass background:false for this call (or run /chrome background off) to bring the tab forward. chrome_click and chrome_fill can instead use their DOM fallback (untrusted page events).`, + `${label}: tab ${tab.id} is hidden because ${hiddenTabReason(fresh, win)}. Input to a minimized window may be dropped. ` + + `Pass background:false for this call to bring the tab forward. chrome_click and chrome_fill can instead use their DOM fallback (untrusted page events).`, ); error.hiddenTab = true; error.windowState = win?.state; @@ -1368,6 +1382,22 @@ async function chromeInputFill(params) { } } +// Runs in the page: scroll the innermost scrollable element under (x, y), else the document. +function scrollHiddenPage(x, y, dx, dy) { + const canScroll = (el) => { + const style = getComputedStyle(el); + const yOk = dy !== 0 && /(auto|scroll|overlay)/.test(style.overflowY) && el.scrollHeight > el.clientHeight; + const xOk = dx !== 0 && /(auto|scroll|overlay)/.test(style.overflowX) && el.scrollWidth > el.clientWidth; + return yOk || xOk; + }; + let el = document.elementFromPoint(x, y); + while (el && el !== document.body && el !== document.documentElement && !canScroll(el)) el = el.parentElement; + const target = el && el !== document.body && el !== document.documentElement ? el : (document.scrollingElement || document.documentElement); + const before = { x: target.scrollLeft, y: target.scrollTop }; + target.scrollBy({ left: dx, top: dy, behavior: "instant" }); + return { scrolled: target === document.scrollingElement ? "document" : (target.tagName || "").toLowerCase(), before, after: { x: target.scrollLeft, y: target.scrollTop } }; +} + async function chromeInputScroll(params) { const tab = await getTabByParams(params); await bringToFront(tab, params); @@ -1377,6 +1407,14 @@ async function chromeInputScroll(params) { const x = resolved.rect ? resolved.rect.left + Math.min(resolved.rect.width, 800) / 2 : resolved.x; const y = resolved.rect ? resolved.rect.top + Math.min(resolved.rect.height, 600) / 2 : resolved.y; const totalY = params.deltaY || 0, totalX = params.deltaX || 0; + // Chrome never answers a mouseWheel sent to a hidden page, so scroll it by script instead. + if ((await pageVisibilityState(tab.id)) === "hidden") { + const evaluated = await cdp(tab.id, "Runtime.evaluate", { + expression: `(${scrollHiddenPage.toString()})(${JSON.stringify(x)}, ${JSON.stringify(y)}, ${JSON.stringify(totalX)}, ${JSON.stringify(totalY)})`, + returnByValue: true, + }); + return { input: "script", reason: "tab is hidden (not the window's active tab), where Chrome drops scroll-wheel events", deltaX: totalX, deltaY: totalY, ...(evaluated?.result?.value || {}) }; + } // Profile mimics a trackpad flick: short ramp-up (~15% of events), then geometric decay // with a ~12% drop per event. Gives momentum tail tests something to find, and the small // tail deltas (a handful of <20px events) put IntersectionObserver thresholds in range. @@ -1627,11 +1665,12 @@ async function chromeInputUpload(params) { } // The target opens the native picker when clicked. Intercept the chooser so no OS dialog appears, -// then attach the files to the input Chrome reports. The click is trusted Chrome input when the page -// is visible; a hidden page drops trusted input, so there we call element.click() with a CDP user -// gesture instead (an untrusted event, like chrome_click's DOM fallback) unless domFallback:false. +// then attach the files to the input Chrome reports. The click is trusted Chrome input, also on an +// inactive tab; in a minimized window, where trusted input may be dropped, we call element.click() +// with a CDP user gesture instead (an untrusted event, like chrome_click's DOM fallback) unless +// domFallback:false. async function uploadViaFileChooser(tab, params, paths, objectId, info) { - const hidden = (await pageVisibilityState(tab.id)) === "hidden"; + const hidden = Boolean(await inputRefusingWindow(tab)); if (hidden && params.domFallback === false) await assertTabVisibleForInput(tab, params, "chrome.upload"); await cdp(tab.id, "Page.enable", {}); await cdp(tab.id, "Page.setInterceptFileChooserDialog", { enabled: true }); @@ -1657,7 +1696,7 @@ async function uploadViaFileChooser(tab, params, paths, objectId, info) { input: "chrome", mode: "file-chooser", trigger, - reason: trigger === "activation-click" ? "target tab is hidden; Chrome drops trusted input to hidden pages" : undefined, + reason: trigger === "activation-click" ? "target tab is hidden in a minimized window, where trusted input may be dropped" : undefined, inFrame: info.inFrame || undefined, uploaded: paths.map((path) => ({ path })), }; @@ -1856,17 +1895,25 @@ async function groupTab(tab, title, color) { if (!tab || typeof tab.id !== "number") throw new Error("No tab to group"); const groupTitle = cleanGroupTitle(title); let groupId = tab.groupId; + let createdGroup = false; if (typeof groupId !== "number" || groupId < 0) { // Some browsers omit windowId on freshly created tabs; read it back instead of guessing. const windowId = typeof tab.windowId === "number" ? tab.windowId : (await chrome.tabs.get(tab.id).catch(() => null))?.windowId; const existing = await findGroupByTitle(windowId, groupTitle); // Without createProperties.windowId Chrome creates the group in the *current* (user's) window // and moves the tab there, pulling Pi's automation tab out of its own window as a hidden tab. + createdGroup = existing === null; groupId = existing !== null ? await chrome.tabs.group({ groupId: existing, tabIds: [tab.id] }) : await chrome.tabs.group(typeof windowId === "number" ? { tabIds: [tab.id], createProperties: { windowId } } : { tabIds: [tab.id] }); } - await chrome.tabGroups.update(groupId, { title: groupTitle, color: cleanGroupColor(color), collapsed: false }); + // A new group starts collapsed to one label in the user's window, unless its tab is the one + // showing (collapsing would switch the window to another tab). Leave an existing group as the + // user set it. + const fresh = await chrome.tabs.get(tab.id).catch(() => tab); + const update = { title: groupTitle, color: cleanGroupColor(color) }; + if (createdGroup) update.collapsed = fresh?.active !== true; + await chrome.tabGroups.update(groupId, update); const grouped = await chrome.tabs.get(tab.id); return { tab: await formatTab(grouped), group: await groupRecord(groupId) }; } @@ -1898,19 +1945,20 @@ async function dispatch(action, params) { const existingGroup = await findGroupRecordByTitle(groupTitle); const createParams = { url: params.url || "about:blank", active: foregroundRequested(params) }; if (existingGroup && typeof existingGroup.windowId === "number") createParams.windowId = existingGroup.windowId; - // No group of this session yet: use the window of its automation tab if it has one, else open - // the tab in a window of its own. Never the user's focused window, where Chrome would put it. + // No group of this session yet: use the window of its automation tab if it has one, else the + // user's window. A window of its own only when Brave has no window open. let ownWindow = false; if (createParams.windowId === undefined) { const home = await resolveOwnedAutomationTarget(sessionKeyOf(params)).catch(() => null); - if (home && typeof home.windowId === "number") createParams.windowId = home.windowId; + const windowId = home && typeof home.windowId === "number" ? home.windowId : await agentWindowId(); + if (windowId !== null) createParams.windowId = windowId; else ownWindow = typeof chrome.windows?.create === "function"; } // Register the creation as in flight until the tab is tracked, so early capture (which only // runs in tracked tabs) waits for it instead of skipping the new tab's first page load. const creation = (async () => { const created = ownWindow - ? (await chrome.windows.create({ url: createParams.url, focused: false })).tabs[0] + ? (await chrome.windows.create({ url: createParams.url, focused: createParams.active })).tabs[0] : await chrome.tabs.create(createParams); await trackSessionTab(sessionKeyOf(params), created.id, true); return created; @@ -1944,7 +1992,7 @@ async function dispatch(action, params) { } case "tab.activate": { if (!foregroundRequested(params)) { - throw new Error("Tab activation is blocked by background mode. Ask the user to run /chrome background off to allow foreground work."); + throw new Error("Tab activation is blocked: the call asked for background, or the user turned background mode on (/chrome background on)."); } // Management actions never auto-create an automation target (createOwnedTarget:false): with // no explicit target they act on an owned target if one exists, else error — they must never @@ -2565,6 +2613,9 @@ function foregroundRequested(params) { async function bringToFront(tab, params) { if (!foregroundRequested(params)) return tab; await chrome.windows.update(tab.windowId, { focused: true }); + if (typeof tab.groupId === "number" && tab.groupId >= 0 && chrome.tabGroups) { + await chrome.tabGroups.update(tab.groupId, { collapsed: false }).catch(() => undefined); + } return chrome.tabs.update(tab.id, { active: true }); } diff --git a/extensions/chrome-profile-bridge/index.ts b/extensions/chrome-profile-bridge/index.ts index 97b9f33..8a5e02d 100644 --- a/extensions/chrome-profile-bridge/index.ts +++ b/extensions/chrome-profile-bridge/index.ts @@ -73,7 +73,7 @@ const DEFAULT_PORT = REMOTE_BRIDGE?.port ?? Number(process.env.PI_CHROME_BRIDGE_ const DEFAULT_TIMEOUT_MS = 30_000; const MAX_TEXT_CHARS = 30_000; const MAX_ELEMENTS = 80; -const BACKGROUND_PARAM_DESCRIPTION = "If true, avoid explicit Chrome focus/tab activation for this call. /chrome background on (default) enforces this for every call and ignores false. Ask the user to run /chrome background off to allow foreground work."; +const BACKGROUND_PARAM_DESCRIPTION = "Default true: work on the tab where it is, without changing what the user sees (an inactive tab takes clicks, typing and screenshots fine). Pass false only when you need the tab showing: it activates the tab and raises its window over what the user is doing. Refused if the user has turned background mode on with /chrome background on."; function resolveRemoteBridge(raw: string | undefined): { host: string; port: number } | undefined { const value = raw?.trim(); @@ -270,7 +270,7 @@ function describeTypeEvidence(result: unknown, typedLength: number, pressEnter: && typeof r.existingTextLengthBefore === "number" && r.existingTextLengthBefore === r.valueLengthAfter && (r.valueRedacted === true || r.valueBefore === r.valueAfter); if (unchanged) { - lines.push("\u26a0 The field value did not change: the keystrokes did not reach this field. It may not have keyboard focus (background/hidden tabs often cannot take focus); focus it with a uid/selector click, or ask the user to run /chrome background off, then verify."); + lines.push("\u26a0 The field value did not change: the keystrokes did not reach this field. It may not have keyboard focus; focus it with a uid/selector click and verify, or retry with background:false to bring the tab forward."); } else if (r.insertedAt === "caret-middle" && typedLength > 0) { lines.push("⚠ Text was spliced into the middle of existing content; chrome_type does NOT replace. Use chrome_fill (or chrome_type replace=true) to replace a field's value."); if (pressEnter) lines.push("⚠ If Enter submitted the form, it submitted the spliced value above, not just your text."); @@ -922,7 +922,8 @@ export default function (pi: ExtensionAPI): void { if (!alreadyLoaded) globalState[PI_CHROME_GLOBAL_KEY] = { version: PI_CHROME_VERSION, root: currentRoot, token: instanceToken }; const bridge = new ChromeProfileBridge(DEFAULT_HOST, DEFAULT_PORT); - let backgroundEnabled = true; + // Off by default: agents may bring a tab forward when they need to (background:false). + let backgroundEnabled = false; let chromeAuthorizedUntil: number | "indefinite" | undefined; // Restore an authorization that survived a /reload. Drop it if it already expired. const persistedAuth = globalState[PI_CHROME_AUTH_KEY]; @@ -1126,14 +1127,16 @@ export default function (pi: ExtensionAPI): void { const authorizedBridgeSend = async (action: string, params: Record<string, unknown>, timeoutMs = DEFAULT_TIMEOUT_MS, signal?: AbortSignal): Promise<unknown> => { requireChromeControlAuthorized(); - // Background on is a session policy, not a default that tool arguments can override. - // Apply it here so tab.new, chrome_launch(url), and tools without a background parameter - // cannot bypass it. Background off still permits per-call background:true. + // Background on (opt-in, /chrome background on) is a session policy that tool arguments cannot + // override. Apply it here so tab.new, chrome_launch(url), and tools without a background + // parameter cannot bypass it. Off (the default) lets each call choose. const typed = params as { background?: boolean; foreground?: boolean }; - const requestedBackground = typed.background ?? (typed.foreground !== undefined ? !typed.foreground : false); + // Unspecified means background, so nothing changes what the user sees unless an agent asks; + // tab.activate exists to bring a tab forward, so it defaults the other way. + const requestedBackground = typed.background ?? (typed.foreground !== undefined ? !typed.foreground : action !== "tab.activate"); const background = backgroundEnabled || requestedBackground; if (action === "tab.activate" && background) { - throw new Error("Tab activation is blocked by background mode. Ask the user to run /chrome background off to allow foreground work."); + throw new Error("Tab activation is blocked: the user turned background mode on (/chrome background on)."); } // Scope every action to this session's dedicated automation target and tab group. const sessionKey = sessionKeyFor(sessionCtx); @@ -1273,7 +1276,7 @@ Capability model (important): - \`chrome_evaluate\` and \`chrome_snapshot\` run in MAIN world via **CDP \`Runtime.evaluate\`**, which is not subject to the page's Content-Security-Policy. They work even on strict-CSP pages (e.g. github.com, many bank/SaaS apps) that block \`'unsafe-eval'\`. \`chrome_navigate initScript\` likewise injects at document_start via CDP and bypasses CSP. \`chrome_screenshot\`, \`chrome_tab\`, and Chrome input also work under any CSP. - Input tools return structured details and support \`includeSnapshot=true\` on click/type/fill/key. Use the fresh snapshot to verify state instead of repeating blindly. If the action started a navigation, the snapshot waits (up to 5s) for the new page. - \`chrome_type\` never replaces: with a uid/selector it appends at the end of the field; without one it types at the current caret. Check its before/after report. Use \`chrome_fill\` to replace a field. -- Chrome ignores real input to hidden pages (inactive tabs, minimized windows, windows behind a macOS full-screen window). Input tools report this instead of silently doing nothing; retry with background:false only when bringing the tab forward is acceptable. +- Inactive (hidden) tabs work: clicks, typing, keys and screenshots reach them without bringing them forward, and chrome_scroll scrolls them by script. Only a minimized window refuses input; pass background:false to bring a tab forward when you need it showing. - \`chrome_cdp\` runs any raw CDP method on a tab (emulation, cookies, PDF, accessibility tree, etc.) when no dedicated chrome_* tool fits; \`chrome_cdp_targets\` diagnoses debugger/overlay conflicts, and \`chrome_clear_overlay\` closes a password manager prompt that blocks a tab (unless you are waiting for the user to use it). Usage rules: @@ -1282,11 +1285,11 @@ Usage rules: 3. \`includeSnapshot=true\` on click/type/fill/key to verify in one round trip. 4. If \`chrome_evaluate\` returns null when you expected a value, the expression evaluated to null/undefined in the page; surface the value via \`JSON.stringify\` to confirm. 5. \`chrome_navigate\` supports an optional \`initScript\` that runs at document_start in MAIN world for the next navigation (good for seeding localStorage or stubbing Date.now). -6. /chrome background on (default) is a hard policy: per-call \`background=false\` cannot override it, new tabs stay inactive, and \`chrome_tab activate\` is blocked. Ask the user to run /chrome background off when they want foreground/watch mode. With background off, per-call \`background=true\` still avoids explicit focus/tab activation. Screenshots use CDP without activating background tabs; debugger failures never fall back to tab activation. Page scripts, trusted input, native prompts, and Chrome/OS behavior can still affect focus. +6. By default every call works in the background: tabs open inactive and nothing changes what the user sees. Pass \`background: false\` on a call (or use \`chrome_tab activate\`) only when you need the tab showing, for example a page that misbehaves while hidden or something the user should watch; it activates the tab and raises its window over what the user is doing. If the user has run /chrome background on, that is refused and you must work without it. 7. If you hit a native file-picker or privileged browser prompt gate, tell the user; generic clicks/typing/CSP gates are handled by Chrome input. 8. Run /chrome doctor when in doubt about connectivity or capabilities. 9. A page can open dialogs you cannot see: alert, confirm, prompt, and the \"Leave site? Changes you made may not be saved\" prompt of a page with unsaved changes. They are dismissed for you (stay, cancel). \`chrome_navigate\` reports one by failing, and \`leaveIfAsked: true\` leaves anyway; use that only when you know nothing unsaved matters, and ask the user when unsure. If a click or key press that should have navigated did nothing, a Leave-site prompt may have stopped it: check the page, then ask the user before discarding anything. -10. Your session has its own browser window, with your tabs in your own colored tab group. Only the active tab of a window is visible, and a hidden tab gets no real input and blank screenshots, so reuse your tab (navigate it) rather than opening new ones; open a second tab only when you need two pages at once, and expect only the active one to take input. Do not open new windows unless the user asks. +10. Your tabs open in the user's Brave window, inactive, in your session's own tab group (collapsed to one label). An inactive tab takes clicks, typing and screenshots normally, so you do not need it showing. Reuse your tab (navigate it) rather than opening new ones, and close tabs you opened when you are done with them. Do not open new windows unless the user asks. 11. If a tool fails saying another extension (a password manager such as 1Password) has a prompt open on the tab, decide whether it is in your way or part of the task. If it is just in your way, call \`chrome_clear_overlay\` for the tab. If you need the user to sign in, authorize or approve something with it, leave it alone and tell the user what you are waiting for. Navigating the tab away also frees it.`; // Shared handlers, dispatched by the unified /chrome command below. @@ -1295,7 +1298,7 @@ Usage rules: const lines: string[] = [ `pi-chrome v${PI_CHROME_VERSION}`, `• Authorization: ${authSummary()}.`, - `• Background: ${backgroundEnabled ? "on (hard)" : "off (foreground/watch mode)"}.`, + `• Background: ${backgroundEnabled ? "on (hard)" : "off (agents may bring a tab forward with background:false; default)"}.`, ]; const status = bridge.status(); const roleLabel = status.remote @@ -1364,8 +1367,7 @@ Usage rules: lines.push(`• Pi hasn't opened its own tab in this session yet.`); } else if (own.visibility === "hidden") { lines.push( - `⚠ Pi's tab is hidden because ${own.hiddenReason ?? "its window is not visible"}. Chrome ignores clicks and typing in hidden tabs, so those tools will fail until it's visible.`, - ` Fix: bring Pi's window forward or uncover it, or run /chrome background off to let Pi bring it forward.`, + `⚠ Pi's tab is hidden because ${own.hiddenReason ?? "its window is not visible"}. Clicks, typing and screenshots still work there; only a minimized window refuses input.`, ); } else if (own.visibility === "visible") { lines.push(`✓ Pi's tab is visible, so clicks and typing will reach it.`); @@ -1383,7 +1385,7 @@ Usage rules: // Existing background setting is the hard policy. No args = toggle; no separate lock mode. const BACKGROUND_DESC: Record<string, string> = { on: "Hard background: pi-chrome will not explicitly focus windows or activate tabs; per-call foreground overrides and tab activation are blocked. Page/Chrome behavior can still affect focus.", - off: "Foreground/watch mode: Chrome may come forward and switch tabs. Per-call background:true still avoids explicit focus/tab activation.", + off: "Default. Calls work in the background unless an agent passes background:false to bring its tab forward.", }; const backgroundHandler = async (ctx: ExtensionContext, args: string) => { @@ -1577,8 +1579,8 @@ Usage rules: ]; } else if (path[0] === "background" && path.length === 1) { candidates = [ - { fullValue: "background on", label: "on", description: "Hard background: block explicit focus/tab activation and per-call foreground overrides. (default)" }, - { fullValue: "background off", label: "off", description: "Bring Chrome to the front so you can watch." }, + { fullValue: "background on", label: "on", description: "Hard background: block explicit focus/tab activation and per-call foreground overrides." }, + { fullValue: "background off", label: "off", description: "Let agents bring a tab forward when they ask to. (default)" }, { fullValue: "background toggle", label: "toggle", description: "Flip whichever way it's currently set." }, { fullValue: "background status", label: "status", description: "Show the current setting." }, ]; @@ -1661,7 +1663,7 @@ Usage rules: pi.registerTool({ name: "chrome_tab", label: "Chrome Tab", - description: "List, create, activate, close, group, ungroup, or inspect tabs in the user's existing Chrome profile via the companion extension. New/grouped tabs always use this session's Pi tab group, in the session's own window; prefer navigating a tab you already have to opening another, since only a window's active tab is visible. Background mode keeps new tabs inactive and blocks activate; ask the user to run /chrome background off for foreground/watch mode. activate/close/group/ungroup require a target (targetId/urlIncludes/titleIncludes); with no target they act on this session's pi-chrome automation tab if one exists, and otherwise error rather than touching the user's active tab.", + description: "List, create, activate, close, group, ungroup, or inspect tabs in the user's existing Chrome profile via the companion extension. New/grouped tabs always use this session's Pi tab group, inactive, in the user's window; prefer navigating a tab you already have to opening another. Inactive tabs take input and screenshots fine; activate only when you need the tab showing (it changes what the user sees). activate/close/group/ungroup require a target (targetId/urlIncludes/titleIncludes); with no target they act on this session's pi-chrome automation tab if one exists, and otherwise error rather than touching the user's active tab.", promptSnippet: "List/open/activate/close/group existing Chrome tabs through the companion extension.", parameters: Type.Object({ action: StringEnum(tabActionValues), @@ -1697,7 +1699,7 @@ Usage rules: name: "chrome_snapshot", label: "Chrome Snapshot", description: - "Inspect a page in the user's existing Chrome profile. Default output is a concise, agent-friendly observation with structural layout/context, stable uids, visible actions, form fields, page hints, and changes since the previous snapshot. Use mode/query/nearUid to zoom instead of dumping the whole page. Background mode (default) blocks explicit focus/tab activation even with background=false. Ask the user to run /chrome background off for foreground/watch mode.", + "Inspect a page in the user's existing Chrome profile. Default output is a concise, agent-friendly observation with structural layout/context, stable uids, visible actions, form fields, page hints, and changes since the previous snapshot. Use mode/query/nearUid to zoom instead of dumping the whole page. Works on inactive tabs without bringing them forward.", promptSnippet: "Observe the current Chrome page: concise summary, structural layout, visible actions, forms, page map, query matches, and stable uids.", parameters: Type.Object({ targetId: Type.Optional(Type.String()), @@ -1802,7 +1804,7 @@ Usage rules: name: "chrome_navigate", label: "Chrome Navigate", description: - "Navigate a Chrome tab to a URL via the companion extension. With no target, navigation goes to pi-chrome's own dedicated automation window/tab — it never replaces the user's active tab. Pass targetId/urlIncludes/titleIncludes only to act on a specific existing tab. Background mode (default) blocks explicit focus/tab activation even with background=false; /chrome background off allows foreground/watch mode. Optionally waits for load completion. Some pages ask \"Leave site? Changes you made may not be saved\" before navigating away; you cannot see that prompt, so it is answered for you: the default is to stay, and the call then fails at once saying so, with the tab unchanged and nothing lost. Retry with leaveIfAsked: true only if you know nothing unsaved matters (you just saved or submitted, or the tab is a throwaway you opened); if you are not sure, ask the user, because leaving discards unsaved work for good. leaveIfAsked applies to this one call only. Any other JavaScript dialog (alert, confirm, prompt) that appears during the navigation is dismissed and reported the same way.", + "Navigate a Chrome tab to a URL via the companion extension. With no target, navigation goes to pi-chrome's own dedicated automation window/tab — it never replaces the user's active tab. Pass targetId/urlIncludes/titleIncludes only to act on a specific existing tab. Works on inactive tabs without bringing them forward. Optionally waits for load completion. Some pages ask \"Leave site? Changes you made may not be saved\" before navigating away; you cannot see that prompt, so it is answered for you: the default is to stay, and the call then fails at once saying so, with the tab unchanged and nothing lost. Retry with leaveIfAsked: true only if you know nothing unsaved matters (you just saved or submitted, or the tab is a throwaway you opened); if you are not sure, ask the user, because leaving discards unsaved work for good. leaveIfAsked applies to this one call only. Any other JavaScript dialog (alert, confirm, prompt) that appears during the navigation is dismissed and reported the same way.", promptSnippet: "Navigate a Chrome tab in the user's existing profile; a page's \"Leave site?\" prompt makes the call fail unless leaveIfAsked is true.", parameters: Type.Object({ url: Type.String(), @@ -1827,7 +1829,7 @@ Usage rules: name: "chrome_evaluate", label: "Chrome Evaluate", description: - "Evaluate JavaScript in an existing Chrome tab through the companion extension. Runs in the page context and returns JSON-serializable values when possible. Background mode (default) blocks explicit focus/tab activation even with background=false; /chrome background off allows foreground/watch mode.", + "Evaluate JavaScript in an existing Chrome tab through the companion extension. Runs in the page context and returns JSON-serializable values when possible. Works on inactive tabs without bringing them forward.", promptSnippet: "Evaluate JavaScript in the active Chrome tab through the companion extension.", parameters: Type.Object({ expression: Type.String(), @@ -1861,7 +1863,7 @@ Usage rules: selector: Type.Optional(Type.String({ description: "CSS selector to click. Prefer uid from chrome_snapshot when available." })), x: Type.Optional(Type.Number({ description: "Viewport x coordinate if uid/selector is omitted." })), y: Type.Optional(Type.Number({ description: "Viewport y coordinate if uid/selector is omitted." })), - domFallback: Type.Optional(Type.Boolean({ description: "If true (default), fall back to DOM-dispatched click if Chrome's CDP input path is blocked (extension overlay, debugger failure, or a hidden tab that ignores trusted input)." })), + domFallback: Type.Optional(Type.Boolean({ description: "If true (default), fall back to DOM-dispatched click if Chrome's CDP input path is blocked (extension overlay, debugger failure, or a minimized window that refuses trusted input)." })), includeSnapshot: Type.Optional(Type.Boolean({ description: "If true, include a fresh chrome_snapshot result after the click." })), maxElements: Type.Optional(Type.Number({ default: MAX_ELEMENTS, description: "Max elements in the included snapshot." })), targetId: Type.Optional(Type.String()), @@ -1926,7 +1928,7 @@ Usage rules: selector: Type.Optional(Type.String({ description: "CSS selector to fill if uid is omitted." })), perCharacter: Type.Optional(Type.Boolean({ default: false, description: "Send individual key events even in contenteditables. Default: one native text insertion for contenteditables; key events for other fields." })), submit: Type.Optional(Type.Boolean({ description: "If true, press Enter after filling." })), - domFallback: Type.Optional(Type.Boolean({ description: "If true (default), fall back to DOM value-setting if Chrome's CDP input path is blocked (extension overlay, debugger failure, or a hidden tab that ignores trusted input)." })), + domFallback: Type.Optional(Type.Boolean({ description: "If true (default), fall back to DOM value-setting if Chrome's CDP input path is blocked (extension overlay, debugger failure, or a minimized window that refuses trusted input)." })), includeSnapshot: Type.Optional(Type.Boolean({ description: "If true, include a fresh chrome_snapshot result after filling." })), maxElements: Type.Optional(Type.Number({ default: MAX_ELEMENTS, description: "Max elements in the included snapshot." })), targetId: Type.Optional(Type.String()), @@ -1951,7 +1953,7 @@ Usage rules: name: "chrome_key", label: "Chrome Key", description: - "Send a keyboard key to an existing Chrome tab (Enter, Escape, Tab, Backspace, Delete, ArrowUp/Down/Left/Right, or one character). Background mode (default) blocks explicit focus/tab activation even with background=false; /chrome background off allows foreground/watch mode. Pass includeSnapshot=true to verify after the keypress.", + "Send a keyboard key to an existing Chrome tab (Enter, Escape, Tab, Backspace, Delete, ArrowUp/Down/Left/Right, or one character). Works on inactive tabs without bringing them forward. Pass includeSnapshot=true to verify after the keypress.", promptSnippet: "Press keys in Chrome through the companion extension.", parameters: Type.Object({ key: Type.String(), @@ -2069,7 +2071,7 @@ Usage rules: name: "chrome_screenshot", label: "Chrome Screenshot", description: - "Capture a screenshot of a Chrome tab via CDP and save it to disk without activating background tabs. Requires debugger access; failures never fall back to activating a tab. Background mode (default) ignores background=false; /chrome background off allows foreground/watch mode.", + "Capture a screenshot of a Chrome tab via CDP and save it to disk without activating background tabs. Requires debugger access; failures never fall back to activating a tab. Inactive tabs render fine; no need to bring them forward.", promptSnippet: "Capture Chrome screenshots and save them under .pi/chrome-screenshots by default.", parameters: Type.Object({ path: Type.Optional(Type.String({ description: "Output path. Defaults to .pi/chrome-screenshots/<timestamp>.<format>." })), @@ -2259,7 +2261,7 @@ Usage rules: name: "chrome_cdp", label: "Chrome CDP Call", description: - "Low-level escape hatch: run one Chrome DevTools Protocol (CDP) method against a tab (no target = this session's automation tab), e.g. Emulation.setDeviceMetricsOverride, Network.getCookies, DOM.getDocument, Page.printToPDF, Accessibility.getFullAXTree. Put CDP fields inside params. Nothing is filtered against a safe list: destructive methods run as given. Prefer dedicated chrome_* tools when they cover the task. Background mode blocks Page.bringToFront and Target.activateTarget, but other methods can still change what the user sees. Screenshot/binary payloads and results over 256 KiB are summarised instead of returned in full. Domain events are not streamed back; only the method's direct result is returned.", + "Low-level escape hatch: run one Chrome DevTools Protocol (CDP) method against a tab (no target = this session's automation tab), e.g. Emulation.setDeviceMetricsOverride, Network.getCookies, DOM.getDocument, Page.printToPDF, Accessibility.getFullAXTree. Put CDP fields inside params. Nothing is filtered against a safe list: destructive methods run as given. Prefer dedicated chrome_* tools when they cover the task. Page.bringToFront and Target.activateTarget are blocked unless the call passes background:false, and other methods can still change what the user sees. Screenshot/binary payloads and results over 256 KiB are summarised instead of returned in full. Domain events are not streamed back; only the method's direct result is returned.", promptSnippet: "Run a raw Chrome DevTools Protocol method against a Chrome tab (low-level escape hatch).", parameters: Type.Object({ method: Type.String({ description: "CDP method name, e.g. \"Runtime.evaluate\" or \"Emulation.setDeviceMetricsOverride\"." }), diff --git a/package.json b/package.json index 7f7e204..4995d3c 100644 --- a/package.json +++ b/package.json @@ -2,7 +2,7 @@ "name": "pi-chrome", "version": "0.15.57", "scripts": { - "test": "node test-suite/unit/csp-eval.test.mjs && node test-suite/unit/automation-target.test.mjs && node test-suite/unit/session-cleanup.test.mjs && node test-suite/unit/background-policy.test.mjs && node test-suite/unit/input-reliability.test.mjs && node test-suite/unit/chrome-command.test.mjs && node test-suite/unit/bridge-resilience.test.mjs && node test-suite/unit/cdp-passthrough.test.mjs && node test-suite/unit/type-evidence.test.mjs && node test-suite/unit/hidden-input.test.mjs && node test-suite/unit/console-capture.test.mjs && node test-suite/unit/prompt-section.test.mjs && node test-suite/unit/upload-guard.test.mjs && node test-suite/unit/grant-scope.test.mjs && node test-suite/unit/leave-site-prompt.test.mjs && node test-suite/unit/shared-window.test.mjs && node test-suite/unit/overlay.test.mjs && node test-suite/unit/sandboxed-snapshot.test.mjs", + "test": "node test-suite/unit/csp-eval.test.mjs && node test-suite/unit/automation-target.test.mjs && node test-suite/unit/session-cleanup.test.mjs && node test-suite/unit/background-policy.test.mjs && node test-suite/unit/input-reliability.test.mjs && node test-suite/unit/chrome-command.test.mjs && node test-suite/unit/bridge-resilience.test.mjs && node test-suite/unit/cdp-passthrough.test.mjs && node test-suite/unit/type-evidence.test.mjs && node test-suite/unit/hidden-input.test.mjs && node test-suite/unit/console-capture.test.mjs && node test-suite/unit/prompt-section.test.mjs && node test-suite/unit/upload-guard.test.mjs && node test-suite/unit/grant-scope.test.mjs && node test-suite/unit/leave-site-prompt.test.mjs && node test-suite/unit/user-window.test.mjs && node test-suite/unit/overlay.test.mjs && node test-suite/unit/sandboxed-snapshot.test.mjs", "version": "node scripts/sync-manifest-version.js", "prepublishOnly": "node scripts/sync-manifest-version.js" }, diff --git a/test-suite/unit/automation-target.test.mjs b/test-suite/unit/automation-target.test.mjs index 73fbc77..1ac3fd6 100644 --- a/test-suite/unit/automation-target.test.mjs +++ b/test-suite/unit/automation-target.test.mjs @@ -9,6 +9,10 @@ // Like csp-eval.test.mjs we load the *real* worker into a vm sandbox with a stateful chrome.* // mock, then exercise the real helpers and the real dispatch() paths. Chrome state (tabs/windows/ // storage.session) can be shared across two sandbox loads to simulate a service-worker restart. +// +// Its chrome.windows mock has no getLastFocused, so the worker takes the path for a browser with no +// window to share and gives each target a window of its own. Placement in the user's window is +// covered by user-window.test.mjs. import vm from "node:vm"; import fs from "node:fs"; diff --git a/test-suite/unit/background-policy.test.mjs b/test-suite/unit/background-policy.test.mjs index ea08be0..98eaa7c 100644 --- a/test-suite/unit/background-policy.test.mjs +++ b/test-suite/unit/background-policy.test.mjs @@ -71,7 +71,7 @@ function piHarness({ session = "alpha", send } = {}) { } const flagCases = [ - [{}, false], [{ background: false }, false], [{ background: true }, true], + [{}, true], [{ background: false }, false], [{ background: true }, true], [{ foreground: true }, false], [{ foreground: false }, true], [{ background: false, foreground: false }, false], [{ background: true, foreground: true }, true], ]; @@ -96,6 +96,7 @@ test("session background on overrides all per-call/legacy foreground flags; off test("every registered page tool, tab.new, and chrome_launch(url) use the central policy", async () => { const h = piHarness(); + await h.background("on"); const cases = [ ["chrome_launch", { url: "https://example.test" }, "tab.new"], ["chrome_tab", { action: "new", group: false, groupTitle: "other" }, "tab.new"], @@ -143,7 +144,11 @@ test("typing options and verification pass through real tool registrations witho test("activation is rejected before dispatch; existing on/off/toggle/status commands suffice", async () => { const h = piHarness(); - await assert.rejects(h.tool("chrome_tab", { action: "activate", targetId: "2", background: false }), /background mode.*\/chrome background off/); + await h.tool("chrome_tab", { action: "activate", targetId: "2" }); + assert.equal(h.calls.at(-1).params.foreground, true, "activation is allowed by default"); + h.calls.length = 0; + await h.background("on"); + await assert.rejects(h.tool("chrome_tab", { action: "activate", targetId: "2", background: false }), /background mode on/); assert.equal(h.calls.length, 0); await h.background("off"); await h.tool("chrome_tab", { action: "activate", targetId: "2" }); @@ -199,9 +204,9 @@ test("shared-owner sessions retain independent policies; per-call background sti const sharedCalls = []; const send = async (action, params) => { sharedCalls.push({ action, params }); return {}; }; const a = piHarness({ session: "A", send }), b = piHarness({ session: "B", send }); - await b.background("off"); + await a.background("on"); await a.send("tab.new", { background: false }); - await b.send("tab.new", {}); + await b.send("tab.new", { background: false }); assert.equal(sharedCalls[0].params.sessionKey, "session:A"); assert.equal(sharedCalls[0].params.foreground, false); assert.equal(sharedCalls[0].action, "tab.new.background"); @@ -314,6 +319,7 @@ function assertNoFocus(h) { test("Pi-to-worker background aliases preserve grouping, screenshot output, and ownership-aware cleanup", async () => { const h = workerHarness(); const pi = piHarness({ send: (action, params) => h.w.dispatch(action, params) }); + await pi.background("on"); const opened = await pi.tool("chrome_tab", { action: "new", group: false, groupTitle: "wrong", background: false }); assert.equal(opened.details.result.group.title, "Pi Session: alpha"); const launched = await pi.tool("chrome_launch", { url: "https://fixture.test" }); diff --git a/test-suite/unit/chrome-command.test.mjs b/test-suite/unit/chrome-command.test.mjs index dac719a..28bfe8a 100644 --- a/test-suite/unit/chrome-command.test.mjs +++ b/test-suite/unit/chrome-command.test.mjs @@ -174,8 +174,7 @@ test("Doctor warns when this session's Pi tab is hidden, and stays quiet when it await hidden.run("doctor"); const report = hidden.notices.at(-1)[0]; assert.match(report, /⚠ Pi's tab is hidden because its window is covered by another window/); - assert.match(report, /Chrome ignores clicks and typing in hidden tabs/); - assert.match(report, /\/chrome background off/); + assert.match(report, /Clicks, typing and screenshots still work there/); const none = harness({ send: withStatus({ windowId: null, tabId: null }) }); await none.run("doctor"); diff --git a/test-suite/unit/hidden-input.test.mjs b/test-suite/unit/hidden-input.test.mjs index 874b5e5..b49229c 100644 --- a/test-suite/unit/hidden-input.test.mjs +++ b/test-suite/unit/hidden-input.test.mjs @@ -1,6 +1,8 @@ -// Hidden-tab input guard, full-screen automation targets, input.debug without a target, upload -// targets (label/wrapper/iframe/file-chooser), and screenshot pruning. Chrome and pages are mocked; -// the hidden-page behavior itself (Chrome drops trusted input) was measured live on Chrome/macOS. +// Hidden-tab input, automation targets beside a full-screen window, input.debug without a target, +// upload targets (label/wrapper/iframe/file-chooser), and screenshot pruning. Chrome and pages are +// mocked. Measured live in Brave on 2026-11-14: an inactive tab takes trusted clicks, keys, text and +// mouse moves and renders screenshots; a mouseWheel there hangs. Minimized windows are untested, so +// input there still fails up front. import assert from "node:assert/strict"; import fs from "node:fs"; import os from "node:os"; @@ -54,50 +56,68 @@ function worker({ visibility = ["visible"], tab = { id: 2, windowId: 1, active: return { w, chrome, calls, inputs: () => calls.filter((c) => c.method.startsWith("Input.")) }; } -// ---- hidden-tab guard ---- -test("typing into a hidden tab fails fast with the reason instead of silently dropping keys", async () => { +// ---- hidden-tab input ---- +const minimized = { id: 1, state: "minimized" }; + +test("typing into an inactive tab sends trusted input", async () => { const h = worker({ visibility: ["hidden"] }); - await assert.rejects(h.w.dispatch("page.type", { selector: "#q", text: "hi", background: true }), (error) => { - assert.match(error.message, /chrome\.type: tab 2 is hidden because it is an inactive tab in its window/); - assert.match(error.message, /background:false/); - return true; - }); - assert.equal(h.inputs().length, 0, "no trusted input sent to a hidden page"); + const result = await h.w.dispatch("page.type", { selector: "#q", text: "hi", background: true }); + assert.ok(h.inputs().length > 0, "trusted input sent to the inactive tab"); + assert.notEqual(result.input, "dom-fallback"); }); -for (const [action, params] of [["key", { key: "Enter" }], ["hover", { x: 1, y: 1 }], ["scroll", { deltaY: 10 }], ["tap", { x: 1, y: 1 }], ["drag", { fromX: 1, fromY: 1, toX: 2, toY: 2 }]]) { - test(`${action} refuses hidden tabs before sending input`, async () => { +for (const [action, params] of [["key", { key: "Enter" }], ["hover", { x: 1, y: 1 }], ["tap", { x: 1, y: 1 }], ["drag", { fromX: 1, fromY: 1, toX: 2, toY: 2 }]]) { + test(`${action} sends trusted input to an inactive tab and refuses a minimized window`, async () => { const h = worker({ visibility: ["hidden"] }); - await assert.rejects(h.w.dispatch(`page.${action}`, { ...params, background: true }), new RegExp(`chrome\\.${action}: tab 2 is hidden`)); - assert.equal(h.inputs().length, 0); + let moves = 0; + h.w.cdpMoveTo = async () => { moves++; }; // hover is only a mouse move, stubbed by the harness + await h.w.dispatch(`page.${action}`, { ...params, background: true }); + assert.ok(h.inputs().length + moves > 0); + const m = worker({ visibility: ["hidden"], win: minimized }); + await assert.rejects(m.w.dispatch(`page.${action}`, { ...params, background: true }), new RegExp(`chrome\\.${action}: tab 2 is hidden because its window is minimized`)); + assert.equal(m.inputs().length, 0); }); } -test("click and fill use their DOM fallback immediately for hidden tabs, or reject with domFallback:false", async () => { - const click = worker({ visibility: ["hidden"] }); +test("scroll on an inactive tab scrolls by script, since Chrome never answers its wheel events", async () => { + const h = worker({ visibility: ["hidden"] }); + const result = await h.w.dispatch("page.scroll", { deltaY: 300, background: true }); + assert.equal(result.input, "script"); + assert.equal(h.inputs().length, 0, "no mouseWheel sent"); + assert.ok(h.calls.some((c) => c.method === "Runtime.evaluate" && /scrollBy/.test(c.params.expression))); + const visible = worker(); + assert.equal((await visible.w.dispatch("page.scroll", { deltaY: 300, background: true })).input, "chrome"); +}); + +test("click and fill send trusted input to an inactive tab; in a minimized window they use their DOM fallback, or reject with domFallback:false", async () => { + const inactive = worker({ visibility: ["hidden"] }); + assert.equal((await inactive.w.dispatch("page.click", { selector: "button", background: true })).input, "chrome"); + assert.ok(inactive.inputs().length > 0); + + const click = worker({ visibility: ["hidden"], win: minimized }); const clicked = await click.w.dispatch("page.click", { selector: "button", background: true }); assert.equal(clicked.input, "dom-fallback"); - assert.match(clicked.reason, /hidden/); + assert.match(clicked.reason, /minimized/); assert.equal(click.inputs().length, 0); - await assert.rejects(worker({ visibility: ["hidden"] }).w.dispatch("page.click", { selector: "button", background: true, domFallback: false }), /is hidden/); + await assert.rejects(worker({ visibility: ["hidden"], win: minimized }).w.dispatch("page.click", { selector: "button", background: true, domFallback: false }), /is hidden/); - const fill = worker({ visibility: ["hidden"] }); + const fill = worker({ visibility: ["hidden"], win: minimized }); fill.w.domFillFallback = async (_tabId, _params, cause) => ({ input: "dom-fallback", reason: String(cause.message) }); const filled = await fill.w.dispatch("page.fill", { selector: "#q", text: "x", background: true }); assert.equal(filled.input, "dom-fallback"); - assert.match(filled.reason, /hidden/); + assert.match(filled.reason, /minimized/); assert.equal(fill.inputs().length, 0); }); -test("hidden reasons distinguish minimized and macOS full-screen windows", async () => { - const minimized = worker({ visibility: ["hidden"], win: { id: 1, state: "minimized" } }); - await assert.rejects(minimized.w.dispatch("page.key", { key: "a", background: true }), /its window is minimized/); - const fullscreen = worker({ visibility: ["hidden"], tab: { id: 2, windowId: 1, active: true, url: "https://fixture.test/" }, win: { id: 1, state: "fullscreen" } }); - await assert.rejects(fullscreen.w.dispatch("page.key", { key: "a", background: true }), (error) => { - assert.match(error.message, /behind a full-screen window/); - assert.equal(error.windowState, "fullscreen"); +test("a hidden page in a full-screen or covered window still takes input; only minimized is refused", async () => { + const m = worker({ visibility: ["hidden"], win: minimized }); + await assert.rejects(m.w.dispatch("page.key", { key: "a", background: true }), (error) => { + assert.match(error.message, /its window is minimized/); + assert.equal(error.windowState, "minimized"); return true; }); + const fullscreen = worker({ visibility: ["hidden"], tab: { id: 2, windowId: 1, active: true, url: "https://fixture.test/" }, win: { id: 1, state: "fullscreen" } }); + assert.equal((await fullscreen.w.dispatch("page.key", { key: "a", background: true })).input, "chrome"); }); test("visible pages, failed probes, and foreground tabs that become visible proceed with trusted input", async () => { @@ -105,20 +125,18 @@ test("visible pages, failed probes, and foreground tabs that become visible proc assert.equal((await visible.w.dispatch("page.key", { key: "a", background: true })).input, "chrome"); const unknown = worker({ visibility: [new Error("probe failed")] }); assert.equal((await unknown.w.dispatch("page.key", { key: "a", background: true })).input, "chrome"); - const foreground = worker({ visibility: ["hidden", "hidden", "visible"] }); + const foreground = worker({ visibility: ["hidden", "hidden", "visible"], win: minimized }); foreground.w.bringToFront = async (tab) => tab; assert.equal((await foreground.w.dispatch("page.key", { key: "a", foreground: true })).input, "chrome"); }); // ---- automation target placement ---- -test("automation targets get their own window even while the user's window is full screen", async () => { - // Chrome disables macOS automatic window tabbing, so a new window is never merged into the user's - // full-screen window; putting Pi's tab in the user's window would only lose isolation. +test("automation targets go in the user's window, inactive, even while it is full screen", async () => { for (const state of ["fullscreen", "maximized"]) { const h = worker({ win: { id: 1, state } }); await h.w.createAutomationTarget(`s-${state}`, undefined); - assert.deepEqual(clone(h.calls.find((c) => c.method === "windows.create").props), { url: "about:blank", focused: false }, state); - assert.equal(h.calls.some((c) => c.method === "tabs.create"), false, state); + assert.deepEqual(clone(h.calls.find((c) => c.method === "tabs.create").props), { url: "about:blank", active: false, windowId: 1 }, state); + assert.equal(h.calls.some((c) => c.method === "windows.create"), false, state); } }); @@ -137,8 +155,8 @@ test("input.debug without a target does not call tabs.get(-1) and reports the se }); // ---- upload targets ---- -function uploadWorker({ visibility = "visible", chooserMode = "selectMultiple", opensChooser = true } = {}) { - const h = worker({ visibility: [visibility] }); +function uploadWorker({ visibility = "visible", chooserMode = "selectMultiple", opensChooser = true, win } = {}) { + const h = worker({ visibility: [visibility], ...(win ? { win } : {}) }); const fileInput = { tagName: "INPUT", type: "file", multiple: true, isConnected: true }; const frameInput = { tagName: "INPUT", type: "file", multiple: false, isConnected: true }; const button = { tagName: "BUTTON", isConnected: true, querySelectorAll: () => [] }; @@ -209,13 +227,15 @@ test("upload buttons open an intercepted file chooser with a trusted click on vi assert.ok(h.inputs().some((c) => c.params.type === "mousePressed")); }); -test("hidden pages use a user-gesture element.click() for upload buttons unless domFallback:false", async () => { - const h = uploadWorker({ visibility: "hidden" }); +test("upload buttons in a minimized window use a user-gesture element.click() unless domFallback:false", async () => { + const inactive = uploadWorker({ visibility: "hidden" }); + assert.equal((await inactive.w.dispatch("page.upload", { selector: "#button", paths: ["/a.txt"], background: true })).trigger, "chrome", "an inactive tab gets the trusted click"); + const h = uploadWorker({ visibility: "hidden", win: minimized }); const result = await h.w.dispatch("page.upload", { selector: "#button", paths: ["/a.txt"], background: true }); assert.equal(result.trigger, "activation-click"); assert.match(result.reason, /hidden/); assert.equal(h.inputs().length, 0); - const strict = uploadWorker({ visibility: "hidden" }); + const strict = uploadWorker({ visibility: "hidden", win: minimized }); await assert.rejects(strict.w.dispatch("page.upload", { selector: "#button", paths: ["/a.txt"], background: true, domFallback: false }), /is hidden/); assert.equal(setFiles(strict).length, 0); }); diff --git a/test-suite/unit/shared-window.test.mjs b/test-suite/unit/shared-window.test.mjs deleted file mode 100644 index 66041f4..0000000 --- a/test-suite/unit/shared-window.test.mjs +++ /dev/null @@ -1,61 +0,0 @@ -// Each session has its own window: a window shows only one tab, so sessions sharing one hid each -// other's tabs (no real input, blank screenshots). A session's extra tabs go into its own window. -import assert from "node:assert/strict"; -import fs from "node:fs"; -import vm from "node:vm"; -import { test } from "node:test"; - -const workerSource = fs.readFileSync(new URL("../../extensions/chrome-profile-bridge/browser-extension/service_worker.js", import.meta.url), "utf8"); - -function worker({ groups = [], windows = { 5: { id: 5, type: "normal" } } } = {}) { - const calls = []; - const listener = { addListener() {}, removeListener() {} }; - let nextTab = 100; - const w = { - console, setTimeout, clearTimeout, setInterval: () => 0, navigator: { userAgent: "unit-test" }, fetch: async () => { throw new Error("offline"); }, - chrome: { - runtime: { id: "test", getManifest: () => ({ version: "0.0.0" }), onInstalled: listener, onStartup: listener }, - alarms: { create() {}, onAlarm: listener }, action: { onClicked: listener }, webNavigation: { onCommitted: listener }, - debugger: { onDetach: listener, onEvent: listener, getTargets: (cb) => cb([]) }, - storage: { session: { get: async () => ({}), set: async (v) => { calls.push({ method: 'storage.set', json: JSON.stringify(v) }); } } }, - tabGroups: { query: async () => groups }, - windows: { - get: async (id) => windows[id] ?? Promise.reject(new Error("No window")), - create: async (props) => { calls.push({ method: "windows.create", props }); return { id: 9, tabs: [{ id: 900, windowId: 9 }] }; }, - }, - tabs: { - create: async (props) => { calls.push({ method: "tabs.create", props }); return { id: nextTab++, windowId: props.windowId }; }, - query: async () => [], get: async () => null, onUpdated: listener, - }, - }, - }; - w.self = w; - vm.runInNewContext(workerSource, w); - return { w, calls }; -} - -const piGroup = (windowId, name = "Pi Session: claude-code 1 other") => ({ id: 1, windowId, title: name }); -const ops = (calls) => calls.filter((c) => c.method !== "storage.set").map((c) => c.method); - -test("a new session gets its own window even when another session's window exists", async () => { - const { w, calls } = worker({ groups: [piGroup(5)] }); - await w.createAutomationTarget("claude-code 2 mine", "Pi Session: claude-code 2 mine"); - assert.deepEqual(ops(calls), ["windows.create"]); -}); - -test("a session that already has a group adds its tab to that group's window", async () => { - const { w, calls } = worker({ groups: [piGroup(5, "Pi Session: claude-code 2 mine")] }); - await w.createAutomationTarget("claude-code 2 mine", "Pi Session: claude-code 2 mine"); - assert.deepEqual(ops(calls), ["tabs.create"]); - assert.equal(calls.find((c) => c.method === "tabs.create").props.windowId, 5); -}); - -test("chrome_tab new for a session with no group goes to the session's own window, not the focused one", async () => { - const { w, calls } = worker({ groups: [piGroup(5)] }); - w.groupTab = async (tab) => tab; - w.bringToFront = async () => {}; - await w.dispatch("tab.new", { url: "https://x.test/", groupTitle: "Pi Session: claude-code 2 mine", sessionKey: "claude-code 2 mine" }); - assert.deepEqual(ops(calls), ["windows.create"]); - assert.equal(calls[0].props.url, "https://x.test/"); - assert.equal(calls[0].props.focused, false); -}); diff --git a/test-suite/unit/user-window.test.mjs b/test-suite/unit/user-window.test.mjs new file mode 100644 index 0000000..af4e939 --- /dev/null +++ b/test-suite/unit/user-window.test.mjs @@ -0,0 +1,119 @@ +// Agents' tabs go in the user's window, inactive, in per-session tab groups that start collapsed. +// An inactive tab takes trusted input and renders screenshots (measured 2026-11-14), so there is no +// need for a window per session. A window is created only when Brave has none. +import assert from "node:assert/strict"; +import fs from "node:fs"; +import vm from "node:vm"; +import { test } from "node:test"; + +const workerSource = fs.readFileSync(new URL("../../extensions/chrome-profile-bridge/browser-extension/service_worker.js", import.meta.url), "utf8"); + +function worker({ groups = [], windows = { 5: { id: 5, type: "normal" } }, lastFocused = 5, tabs = {} } = {}) { + const calls = []; + const listener = { addListener() {}, removeListener() {} }; + let nextTab = 100; + const w = { + console, setTimeout, clearTimeout, setInterval: () => 0, navigator: { userAgent: "unit-test" }, fetch: async () => { throw new Error("offline"); }, + chrome: { + runtime: { id: "test", getManifest: () => ({ version: "0.0.0" }), onInstalled: listener, onStartup: listener }, + alarms: { create() {}, onAlarm: listener }, action: { onClicked: listener }, webNavigation: { onCommitted: listener }, + debugger: { onDetach: listener, onEvent: listener, getTargets: (cb) => cb([]) }, + storage: { session: { get: async () => ({}), set: async (v) => { calls.push({ method: 'storage.set', json: JSON.stringify(v) }); } } }, + tabGroups: { query: async () => groups, update: async (id, props) => { calls.push({ method: "tabGroups.update", id, props }); return { id, ...props }; } }, + windows: { + get: async (id) => windows[id] ?? Promise.reject(new Error("No window")), + getLastFocused: async () => (lastFocused === null ? Promise.reject(new Error("No window")) : windows[lastFocused]), + getAll: async () => Object.values(windows), + update: async (id, props) => { calls.push({ method: "windows.update", id, props }); return { id }; }, + create: async (props) => { calls.push({ method: "windows.create", props }); return { id: 9, tabs: [{ id: 900, windowId: 9 }] }; }, + }, + tabs: { + create: async (props) => { calls.push({ method: "tabs.create", props }); return { id: nextTab++, windowId: props.windowId }; }, + query: async () => [], get: async (id) => tabs[id] ?? null, onUpdated: listener, + group: async () => 77, + update: async (id, props) => { calls.push({ method: "tabs.update", id, props }); return { id, ...props }; }, + }, + }, + }; + w.self = w; + vm.runInNewContext(workerSource, w); + return { w, calls }; +} + +const piGroup = (windowId, name = "Pi Session: claude-code 1 other") => ({ id: 1, windowId, title: name }); +const ops = (calls) => calls.filter((c) => c.method !== "storage.set").map((c) => c.method); +const noWindows = { windows: {}, lastFocused: null }; + +test("a new session's tab goes in the user's window, inactive", async () => { + const { w, calls } = worker({ groups: [piGroup(5)] }); + await w.createAutomationTarget("claude-code 2 mine", "Pi Session: claude-code 2 mine"); + assert.deepEqual(ops(calls), ["tabs.create"]); + const props = calls.find((c) => c.method === "tabs.create").props; + assert.equal(props.windowId, 5); + assert.equal(props.active, false); +}); + +test("a window is created only when Brave has none", async () => { + const { w, calls } = worker(noWindows); + await w.createAutomationTarget("claude-code 2 mine", "Pi Session: claude-code 2 mine"); + assert.deepEqual(ops(calls), ["windows.create"]); + assert.equal(calls[0].props.focused, false); +}); + +test("a session that already has a group adds its tab to that group's window", async () => { + const { w, calls } = worker({ groups: [piGroup(7, "Pi Session: claude-code 2 mine")], windows: { 5: { id: 5 }, 7: { id: 7 } } }); + await w.createAutomationTarget("claude-code 2 mine", "Pi Session: claude-code 2 mine"); + assert.deepEqual(ops(calls), ["tabs.create"]); + assert.equal(calls.find((c) => c.method === "tabs.create").props.windowId, 7); +}); + +test("chrome_tab new for a session with no group goes in the user's window, inactive", async () => { + const { w, calls } = worker({ groups: [piGroup(5)] }); + w.groupTab = async (tab) => tab; + w.bringToFront = async () => {}; + await w.dispatch("tab.new", { url: "https://x.test/", groupTitle: "Pi Session: claude-code 2 mine", sessionKey: "claude-code 2 mine" }); + assert.deepEqual(ops(calls), ["tabs.create"]); + assert.equal(calls[0].props.windowId, 5); + assert.equal(calls[0].props.active, false); +}); + +test("chrome_tab new with no window open creates an unfocused one", async () => { + const { w, calls } = worker(noWindows); + w.groupTab = async (tab) => tab; + w.bringToFront = async () => {}; + await w.dispatch("tab.new", { url: "https://x.test/", groupTitle: "Pi Session: claude-code 2 mine", sessionKey: "claude-code 2 mine" }); + assert.deepEqual(ops(calls), ["windows.create"]); + assert.equal(calls[0].props.focused, false); +}); + +test("a new group starts collapsed; an existing group keeps the user's setting", async () => { + const { w, calls } = worker({ tabs: { 300: { id: 300, windowId: 5, active: false, groupId: -1 }, 301: { id: 301, windowId: 5, active: false, groupId: 77 } } }); + w.groupRecord = async (id) => ({ id }); + w.formatTab = async (t) => t; + await w.groupTab({ id: 300, windowId: 5, groupId: -1 }, "Pi Session: x", "blue"); + await w.groupTab({ id: 301, windowId: 5, groupId: 77 }, "Pi Session: x", "blue"); + const updates = calls.filter((c) => c.method === "tabGroups.update").map((c) => c.props.collapsed); + assert.deepEqual(updates, [true, undefined]); +}); + +test("a new group holding the tab that is showing is not collapsed", async () => { + const { w, calls } = worker({ tabs: { 300: { id: 300, windowId: 5, active: true, groupId: -1 } } }); + w.groupRecord = async (id) => ({ id }); + w.formatTab = async (t) => t; + await w.groupTab({ id: 300, windowId: 5, groupId: -1 }, "Pi Session: x", "blue"); + assert.equal(calls.find((c) => c.method === "tabGroups.update").props.collapsed, false); +}); + +test("bringing a tab forward expands its group", async () => { + const { w, calls } = worker(); + await w.bringToFront({ id: 300, windowId: 5, groupId: 77 }, { foreground: true, background: false }); + assert.deepEqual(ops(calls), ["windows.update", "tabGroups.update", "tabs.update"]); + assert.equal(calls[1].props.collapsed, false); +}); + +test("input is sent to an inactive tab; only a minimized window refuses it", async () => { + const { w } = worker({ windows: { 5: { id: 5, state: "normal" }, 6: { id: 6, state: "minimized" } }, tabs: { 300: { id: 300, windowId: 5, active: false }, 301: { id: 301, windowId: 6, active: true } } }); + w.pageVisibilityState = async () => "hidden"; + await w.assertTabVisibleForInput({ id: 300, windowId: 5 }, {}, "chrome.click"); + await assert.rejects(w.assertTabVisibleForInput({ id: 301, windowId: 6 }, {}, "chrome.click"), /minimized/); +}); |