diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 170 |
1 files changed, 170 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..a97d105 --- /dev/null +++ b/README.md @@ -0,0 +1,170 @@ +# pi-castle-cdp-search + +Web search for [pi](https://pi.dev) through a Chrome that is **already running** +on the wire network, driven over the Chrome DevTools Protocol. + +It never launches a browser. It attaches to one a person or a launchd job +started — by default castle's shared Chrome on `10.88.0.25:9223` — opens a +background tab, reads the results page, and closes the tab again. + +Because the search runs in a real browser with a real profile, it reaches pages +that reject datacentre traffic, and it costs nothing per query. + +## The tool + +One tool, `castle_cdp_search`. Deliberately **not** named `web_search`, so it +coexists with `pi-web-access` rather than shadowing it — both stay registered and +the model picks. + +| Parameter | Type | Notes | +|---|---|---| +| `query` | string, required | As you would type it into a search box | +| `numResults` | integer, optional | 1–20, default 10 | +| `recency` | `day` \| `week` \| `month` \| `year`, optional | Omit for no time limit | + +It returns titles, URLs and snippets — **search results only**. It does not fetch +the linked pages; follow up with a read/fetch tool for full content. + +## Which browser it drives + +Exactly the same configuration as [`pi-browser-harness`](../pi-browser-harness), +on purpose: one decision governs both packages, so they can never end up driving +different browsers in the same session. + +| `BU_CDP_HTTP` | Behaviour | +|---|---| +| unset | the `/browser-target` choice stored by pi-browser-harness, else castle (`10.88.0.25:9223`) | +| `host:port` | that endpoint | +| `local` (also `localhost`, `off`, `none`, `0`, `no`) | this machine's own browser on `127.0.0.1:9222` | + +Precedence is `BU_CDP_HTTP` > `/browser-target` > built-in default, matching the +harness. The `/browser-target` file (`~/.pi/agent/browser-target.json`) is read +only, never written — the harness owns that setting. + +**One deliberate difference from the harness.** When the configured endpoint is +unreachable, this extension reports that and stops. The harness falls back from +its built-in default to a local browser; here that fallback would mean silently +running searches through the operator's own signed-in Chrome without saying so. +Say `local` explicitly if that is what you want. + +The endpoint is re-resolved from `/json/version` on every connect, never cached: +castle recycles Chrome hourly and Chrome re-mints its browser UUID each launch, +so a pinned `ws://` URL breaks on the first recycle. A dropped socket reconnects +transparently on the next search. + +## Behaviour worth knowing + +- **Nothing is dialled at load.** pi runs extension factories in invocations that + never start a session, so the connection is made on first search and closed by + an idempotent `session_shutdown` handler. +- **One fresh background tab per search**, closed on every path including errors + and aborts. `background: true` so it does not steal focus from whoever is + looking at that screen. +- **Two concurrent searches**, queued beyond that. +- **20-second deadline** per search, covering connect, navigate and extract + together. Esc aborts an in-flight navigation. +- **Every failure throws.** pi only sets `isError: true` when `execute()` throws; + a returned `{ error }` object would read to the model as a search that simply + found nothing. + +## What it touches + +The browser is shared and belongs to a person. Worth being aware of: + +- Queries the agent runs land in that browser profile's history and cookies, and + in the Google account's search history if that profile is signed in. +- Tabs open and close on someone's screen. `background: true` keeps them from + stealing focus, but they are visible. +- Searching hard trips Google's rate limiter for the whole host — a handful of + queries in a few seconds is enough to earn a `/sorry/` page that affects the + human using that browser too. The cap of two concurrent searches limits this; + it does not eliminate it. Measured once: roughly 30 queries in a few minutes + cost about 90 minutes of blocking. +- **A decaying block does not look like a block.** Once the `/sorry/` page stops + being served, Google returns an *empty results page* for a while instead — + which is indistinguishable from a query that genuinely has no hits. It + surfaces as `NoResultsError`, not `SearchChallengeError`. If several unrelated + queries all come back empty, that is rate limiting; wait rather than retrying. + +## When a captcha appears + +Google will eventually serve a `/sorry/` interstitial, a consent wall, or a +recaptcha — especially if searches come in fast. The extension detects this and +throws a `SearchChallengeError` naming the challenge, the browser, and the URL a +human has to visit. + +The agent cannot solve it. The guidelines tell it to stop searching and hand off +to the user, who opens that URL in the browser at the endpoint, clears the +challenge, and lets the agent retry. The cookie is profile-wide, so one pass +unblocks later searches. + +The tab is closed rather than left open on the challenge page: leaving it would +let the operator solve it in place, but would also litter a shared browser with +abandoned tabs on every failure. The URL in the error is enough. + +## Search parameters, and two surprises + +Plain searches use `udm=14` — Google's "Web" tab: no AI overview, no carousels, +just ranked links, which is both cheaper to parse and closer to what was asked +for. Verified against the real browser: + +- `tbs=qdr:*`, the parameter Google's own Tools menu writes, renders an **empty + page** for this profile — `#search` present, no `#rso`, no results. The older + `as_qdr=*` works and genuinely filters. +- Any date restriction combined with `udm=14` also renders that empty page. + +So a time-limited search drops `udm` and uses `as_qdr`. The extractor handles +both layouts. + +## Result extraction + +`src/extract.ts` avoids Google's generated class names (`MjjYud`, `kb0PBd`, …) +entirely. It prefers the `data-snhf` / `data-sncf` hooks and otherwise falls back +to a structural walk: from each `<h3>`, take the enclosing link, climb until the +text grows past the header's, and abandon the climb if a second `<h3>` comes into +scope. Checked against both the classic SERP and the `udm=14` layout. + +Google's markup will change anyway. When it does, a search returns +`NoResultsError` whose message distinguishes "results container rendered but +unparseable" (extractor needs updating) from "no results area at all" (query or +parameters). + +## Install + +``` +pi install /Volumes/Sense/src/soarez/pi-castle-cdp-search +``` + +Or, once it is on castle alongside the harness: + +``` +pi install ssh://sz@10.88.0.25/Users/sz/repos/pi-castle-cdp-search.git +``` + +The ssh form is what pi-browser-harness uses, and it is the portable one: pi +records local-path installs by resolved absolute path, and `settings.json` is +shared across hosts through the dotfiles symlink. + +To try it without installing: + +``` +pi -e ./src/index.ts +``` + +## Development + +``` +npm install +npm run typecheck +npm test # unit tests, no browser needed +``` + +The tests cover endpoint resolution, URL building, formatting, the challenge +error, and the concurrency semaphore. The semaphore tests point at an +unreachable endpoint on purpose — the questions there are about slot +bookkeeping, and hitting a real browser to test a counter would be slow, flaky, +and rude to whoever is using it. + +Requires Node 22+, for the global `WebSocket`. There are no runtime +dependencies; pi supplies `@earendil-works/pi-coding-agent`, +`@earendil-works/pi-ai` and `typebox` to extensions through its own loader. |