aboutsummaryrefslogtreecommitdiff
path: root/src/domains
AgeCommit message (Collapse)Author
2026-08-02refactor(cdp): extract attachTo, getWindowId, evaluateJson helpersamankumarsingh77
Target.attachToTarget was decoded at 5 sites, Browser.getWindowForTarget at 5, and Runtime.evaluate unwrapped at another. Each is now one typed helper. Deletes the dead cdp/window-binding.ts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25feat: pin the agent to a chosen browser profile (#16)Aman Kumar
* feat: pin the agent to a chosen browser profile (#15) Harness tabs were created with a bare Target.createTarget, which places them in Chrome's defaultBrowserContextId — and that value follows window focus. Which profile the agent acted as therefore depended on which browser window the user last clicked, so the same task could run as a work account on one run and a personal account on the next. /browser-profile lists the connected browser's profiles as "Name (email)" and pins one to ~/.pi/agent/browser-harness.json, so the choice outlives the session. The same picker appears on first browser_setup when nothing is pinned; with no interactive UI, or on cancel, setup continues with a note and the previous behavior. Chrome offers no direct route to another profile: Target.createTarget rejects a foreign browserContextId, and openerId does not inherit the opener's context. The window is opened through the browser's own command line (--profile-directory, handed to the running browser by Chromium's ProcessSingleton) and identified by a unique file:// sentinel page, since a bare about:blank#token argument is discarded. Further tabs come from window.open evaluated with userGesture: true — the only CDP-reachable way to place a tab in a non-default browser context — correlated by a unique token so concurrent isolated tabs stay distinct. A failed handshake is reported, never silently redirected to another profile. All four tab-creation sites now funnel through cdp/target-factory, so open_urls and the isolated tabs behind web_search / read_page cannot run with a different profile's cookies than the visible tabs. Also fixed along the way: snap Chromium is discovered, Windows uses %LOCALAPPDATA%, browsers launched with an explicit --user-data-dir are found, the running browser is chosen by ranking candidates against the connected user-data-dir rather than taking the first process found, and an installed-but-closed browser no longer reads as running. CI runs the profile suites and a real-browser end-to-end test on ubuntu, macOS, and Windows — profile discovery is the one part of the harness whose behavior genuinely differs per OS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: single-flight harness window creation Isolated tabs run concurrently (browser_read_page is registered unserialized), so several could each find "no harness window" and create one — two blank windows before this feature, two LAUNCHED profile windows after it. ensureHarnessWindow now shares one in-flight creation per client. Sharing introduces a second hazard: freshlyCreated is a claim on the seed tab, and if every waiter received it, they would all drive the same tab. Only the caller that started the creation keeps the claim; the rest open their own tab beside it. Covered by test/manual/profile-concurrency-test.ts (five concurrent isolated tabs, pinned and unpinned: distinct targets, one window, correct profile), wired into CI on Linux only — the coordination is pure JS and identical across platforms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: tilde expansion differs by purpose on Windows CI on windows-latest caught this: join("~", "x") yields "~\x" there, and expandTilde only handled "~/". The two uses are not the same, so they no longer share a helper. The agent dir stays byte-compatible with pi's own expandTildePath ("~/" only, every platform) — being more lenient would put the pin file somewhere pi never looks. $CHROME_USER_DATA_DIR is ours to interpret, so it accepts "~\" as well, which is what a Windows user would type. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22feat: add forms domain and durable window binding (v0.9.0) (#14)Aman Kumar
- Add forms domain with a universal field setter that auto-detects element type and fires framework-compatible input events (React native-setter trick) - Add window-binding helper to capture and bind the real Chrome windowId as the session's durable window identity on every new-window code path - Wire supporting changes across CDP session/ownership, client, navigate, isolated-tab, prompt, and state - Add manual tests for forms and window binding - Bump version 0.8.3 -> 0.9.0 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08feat: add expand/collapse (Ctrl+O) rendering to web_search and read_pageamankumarsingh77
Both tools dumped full output into the TUI. Add a shared renderExpandableText helper (src/domains/render.ts) mirroring browser_execute_js: a compact preview by default, full body on expand via keyHint("app.tools.expand"). Tools stash a {summary, body, fullOutputPath} render payload in details; the full text still goes to the LLM via ToolOk.text. Also surfaces engine + result count in the web_search summary line (closes the S4 observability gap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08feat: add browser_read_page reader-mode toolamankumarsingh77
URL (or owned targetId) → clean main-article text with nav/ads/footer boilerplate stripped, via a dependency-free readability heuristic. DOM walk lives in an in-page capture expression; scoring/selection is a pure, fixture- tested function (link-density + boilerplate-ancestor filtering, body-text fallback for structure-less pages). Reuses the isolated-attach lifecycle; registered unserialized (safe concurrent read). 10 extractor unit scenarios. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08feat: add browser_web_search tool (real-Chrome SERP scrape, no API key)amankumarsingh77
Thin query→ranked-links primitive. Isolated-attach tab lifecycle mirroring browser_open_urls (never touches the user's current tab); pure Google SERP parser with semantic (a h3) extraction and redirect unwrapping; CAPTCHA and no-results surfaced as invalid_state + details.reason. Fixture-based parser tests (9 scenarios). Establishes the isolated-tab lifecycle and tsx+fixture test convention that browser_read_page and deep-research build on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05fix: make dispatch_key fire legacy keyCode handlers and tighten execute_js ↵Aman Kumar
IIFE wrap (#9) - browser_dispatch_key now sets keyCode/which on the synthesized KeyboardEvent (both ref and selector paths) so handlers checking e.keyCode === 13 fire. - browser_execute_js IIFE auto-wrap triggers only when the source starts with a return statement, not on any substring "return " (which silently produced undefined for bare expressions). - browser_fill guidelines point at dispatch_key for tag-input submission and the snapshot+click recipe for custom dropdowns. Bump to 0.8.1. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03refactor: extract event queue from transport into dedicated moduleamankumarsingh77
2026-06-29feat: address elements by stable refs across re-renders (#8)Aman Kumar
Interaction tools could only target elements by raw viewport coordinates (browser_click) or model-guessed CSS selectors (browser_fill/focus/ select_option/upload). Neither survives a re-render: on React/SPA forms, every save or edit reflows the page, so captured @(x,y) coordinates go stale and guessed selectors miss — leaving the agent to fall back to hand-rolled execute_js DOM walks and unnoticed failed saves. Add a stable ref system, mirroring Playwright MCP / browser-use: - snapshot assigns each interactive element a ref (e1, e2, …), shown as [eN] in the outline, and publishes a per-tab ref → backendNodeId map - interaction tools accept an optional `ref` that re-resolves the element at call time (via DOM.resolveNode / getBoxModel), so it stays valid across re-renders; coordinates/selectors remain as fallbacks - mutating tools append a compact diff of changed / newly-appeared (*) / removed interactive elements, so silent failed saves surface immediately - stale refs return an actionable "re-snapshot" error instead of a miss Ref map is replaced per snapshot (bounded memory); fill keeps the native _valueTracker write path so React controlled inputs still update. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21release: v0.6.0 — per-tab isolation, Unix socket daemon, on-demand init (#6)Aman Kumar
* feat: Unix socket daemon with auto-reconnect for persistent Chrome CDP Eliminates the 'Allow Remote Debugging' Chrome prompt that appeared on every pi session by multiplexing pi instances through a single persistent daemon process. Key additions: - Unix socket IPC (net.createServer + readline JSON framing) - CdpBridge with ID multiplexer + event router - DaemonTransport implementing CdpTransport interface - Auto-spawn with ensureDaemon() (lazy, checks socket) Fixes since initial implementation: - Bridge auto-reconnects to Chrome with exponential backoff (1s-60s). Previously the bridge tried once at startup and permanently failed if Chrome wasn't ready — causing 'Chrome not connected' on every browser tool call. - Daemon spawn uses project-local tsx binary instead of global --import flag, fixing silent spawn failure (ERR_MODULE_NOT_FOUND). - Stale ctx crash: persistState in onOwnershipChange wrapped in try/catch to prevent CDP event consumer crashes. 49 test assertions across 4 test files. Zero TypeScript errors. * feat: on-demand browser initialization Browser daemon no longer auto-starts on session_start. Instead: - session_start creates the client lazily but does NOT connect or spawn the daemon. Zero Chrome prompts for non-browser pi sessions. - Browser tool wrapper checks for daemon socket before each call. Socket missing → clear error: 'Run /browser-setup first'. Socket exists → proceeds silently (daemon already running). - /browser-setup is the explicit, one-time init point. It spawns the daemon and connects to Chrome only when the user opts in. - SKILL.md updated with Connection Setup instructions telling agents to direct users to /browser-setup when socket is missing. * fix: checkChromeRunning on macOS — match by substring not exact ps -A -o comm= returns full executable paths on modern macOS (e.g. /Applications/Google Chrome.app/Contents/MacOS/Google Chrome), not the basename. The old exact-match check (browserProcesses.includes(l)) never matched, causing false 'No browser instance running' errors. Changed to substring includes() with helper/renderer/crashpad filtering. * feat: browser_setup as agent-callable tool The agent can now self-recover from 'Browser harness not initialized' errors by calling browser_setup directly — no user mediation needed. - Extracted shared performSetup() from the slash command into setup.ts - Created src/domains/setup.ts with ensureAlive:false to bypass the socket guard (this tool IS the setup, can't require itself) - Registered browser_setup in the tool registry - Updated SKILL.md: agent calls browser_setup directly, not ask user - Removed stale test/manual/spawn-test.ts * release: v0.6.0 — per-tab isolation, Unix socket daemon, on-demand init
2026-05-07release: v0.5.0 — browser_console tool for JS error/console diagnosticsamankumarsingh77
2026-05-06release: v0.4.0 — accessibility snapshot, network capture, snapshot-first ↵amankumarsingh77
prompts Adds two new tools and pivots the agent's default workflow away from screenshots toward structured page data. browser_snapshot — CDP accessibility tree (roles, names, states, hierarchy) with click coordinates @(x,y) attached to every interactive element via DOM.getBoxModel (parallel, 1.5s aggregate budget). Pass directly to browser_click — no screenshot round-trip. Optional includeScreenshot:true for visual confirmation. format:"json" returns a slim structure with full box per node. browser_network_requests — replaces the deprecated browser_get_network_log placeholder. Filters by urlPattern (substring or /regex/), method, status, resource type, recency. Optional response-body capture under a 5s aggregate budget with 50KB per-body cap. Buffer is page-scoped (cleared on tab switch), 500 records/tab, with a bufferOverflowed flag on drain. CDP plumbing: in-process aggregator in src/cdp/network-buffer.ts wired into the existing single-consumer event loop. Accessibility domain enabled alongside Page/DOM/Runtime/Network on every attach. Ctrl+O expand/collapse on three tools' renderResult — browser_snapshot, browser_network_requests, browser_execute_js. Compact-by-default views with keyHint("app.tools.expand") that respects user keybinding remaps. Tool prompts pivoted snapshot-first: browser_screenshot is documented as LAST RESORT for visual verification only; browser_snapshot is the default for understanding pages; browser_execute_js is the default for surgical reads. browser_click and browser_open_urls guidance follows. SKILL.md collapsed from ~342 to ~53 lines — keeping only what isn't already in tool prompts: the decision tree, connection rules, browser_run_script daemon bindings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05feat: tab ownership — isolate harness tabs from the user's browsingamankumarsingh77
Track which page targets this harness session opened so the agent can never attach to, switch to, or close a tab the user opened themselves. New tabs are spawned in a dedicated Chrome window (newWindow:true on first open, openerId chaining after) so harness activity stays visually separate. - new OwnershipRegistry (src/cdp/ownership.ts) tracks owned targetIds and the harness window seed; mutations notify a single listener that persists to BrowserState (ownedTargetIds, harnessWindowTargetId) - attachFirstPage prefers an already-owned live target; otherwise opens a fresh harness-owned tab in its own window instead of grabbing whatever page the user is on - session subscribes Target.setDiscoverTargets and reaps targetDestroyed events so the registry stays in sync with reality - listTabs returns owned:boolean per tab and reconciles dead ids - browser_list_tabs defaults to scope:"owned"; pass scope:"all" to inspect the user's other tabs (read-only — they cannot be switched to or closed) - browser_switch_tab refuses non-owned tabs, with a clear remediation hint - browser_new_tab opens inside the harness window via openerId, falling back to a fresh window when the seed has been closed - new browser_close_tab tool — owned tabs only, supports unique hex prefix Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05fix: prevent TUI crash when screenshot path overflows terminal width (#2)Aman Kumar
The Image text-fallback render does not respect the rendered width, so a long file path can blow past the terminal width and crash the host TUI. Wrap the component to truncate each rendered line with truncateToWidth so it fits the available width. Co-authored-by: amankumarsingh77 <amankumarsingh7702@gmal.com>
2026-05-04feat: parallel tool execution with automatic mutation serializationamankumarsingh77
- Add lightweight async mutex (src/util/mutex.ts) for FIFO serialization - Add serialized?: boolean to BrowserToolDefinition - Wire mutex acquisition into registerBrowserTool execute wrapper - Expose mutationMutex() on BrowserClient - Mark all mutation tools (click, type, scroll, navigate, switch_tab, etc.) as serialized - Leave observation tools (screenshot, page_info, execute_js, http_get, etc.) unguarded - Update prompt and skill docs to reflect parallel-safe behavior - Fix scroll tool: add Page.bringToFront, explicit button:none/buttons:0/pointerType:mouse - Update .gitignore to ignore scripts/ and data/ folders
2026-05-03feat: render screenshots inline in TUI via Image component; add ↵amankumarsingh77
parallelization guidance to SKILL.md - browser_screenshot now has renderResult that creates an Image component for inline display in Kitty/iTerm2/Ghostty/WezTerm terminals - Falls back to dim text placeholder on terminals without image support - Updated system prompt table description to mention TUI inline rendering - Added Parallelization section to SKILL.md teaching the agent to batch independent operations (browser_open_urls, browser_http_get, etc.)
2026-05-02release: v0.3.1 — audit fixes for list_tabs IDs, switch_tab prefix ↵amankumarsingh77
matching, download auto-mkdir, session recovery, docs updates Bug fixes: - list_tabs: show full 32-char targetIds (was truncated to 8 chars) - switch_tab: prefix matching fallback for hex prefixes >= 8 chars - download: auto-create directory with mkdir -p - ensureAlive: Runtime.evaluate probe to recover from crashed page targets Docs: - dispatch_key: clarify DOM-only event behavior - navigate: warn about Google anti-bot limitation - network_log: update deprecated promptSnippet - SKILL.md: document actual daemon API (evaluateJs, pageInfo, session.call) - SKILL.md: add troubleshooting for sharp/maxDim and Google navigation - sharp: add to optionalDependencies for screenshot maxDim auto-resize
2026-05-02feat(domains): migrate execute_js and run_script with path allowlist + ↵amankumarsingh77
timeout + return validation
2026-05-02feat(domains): migrate http_get (timeout covers body) and get_network_log ↵amankumarsingh77
(deprecation note)
2026-05-02feat(domains): migrate drag_and_dropamankumarsingh77
2026-05-02feat(domains): migrate viewport_resizeamankumarsingh77
2026-05-02feat(domains): migrate upload_file (safeJs), download (validated dir), ↵amankumarsingh77
print_to_pdf
2026-05-02feat(domains): migrate tab tools (list, current, switch, new)amankumarsingh77
2026-05-02feat(domains): migrate go_back, go_forward, reloadamankumarsingh77
2026-05-02feat(domains): migrate navigate (NavOutcome) and open_urlsamankumarsingh77
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02feat(domains): migrate screenshot, add typed sharp shim, restore click ↵amankumarsingh77
crosshair overlay
2026-05-02feat(domains): migrate handle_dialogamankumarsingh77
2026-05-02feat(domains): migrate page_info, wait, wait_for_load with typed timeoutamankumarsingh77
2026-05-02feat(domains): migrate scroll with W3C deltaY semantics and dialog guardamankumarsingh77
2026-05-02feat(domains): migrate type, press_key, dispatch_key with safeJs and ↵amankumarsingh77
matched-count validation
2026-05-02feat(domains): migrate browser_click to defineBrowserTool — vertical sliceamankumarsingh77
Implements Task 9: adds src/domains/click.ts with the new architecture, updates registry.ts to register clickTool, removes the legacy browser_click block from tools.ts, and wires BrowserClient alongside BrowserDaemon in index.ts so both systems coexist during migration. Also fixes AnyBrowserToolDefinition variance: uses never-arg renderCall in the erased type and a boundary cast in registerBrowserTool so concrete tool definitions (with specific Static<S> args) are assignable to the registry array without widening.