diff options
Diffstat (limited to 'castle-search/README.md')
| -rw-r--r-- | castle-search/README.md | 224 |
1 files changed, 224 insertions, 0 deletions
diff --git a/castle-search/README.md b/castle-search/README.md new file mode 100644 index 0000000..6dac8a7 --- /dev/null +++ b/castle-search/README.md @@ -0,0 +1,224 @@ +# pi-castle-cdp-search + +Web search for [pi](https://pi.dev) through a Chrome that is **already running** +on the wire network, driven over the Chrome DevTools Protocol. + +It never launches a browser. It attaches to one a person or a launchd job +started — by default castle's shared Chrome on `10.88.0.25:9223` — opens a +background tab, reads the results page, and closes the tab again. + +Because the search runs in a real browser with a real profile, it reaches pages +that reject datacentre traffic, and it costs nothing per query. + +## The tool + +One tool, `castle_cdp_search`. Deliberately **not** named `web_search`, so it +coexists with `pi-web-access` rather than shadowing it — both stay registered and +the model picks. + +| Parameter | Type | Notes | +|---|---|---| +| `query` | string, required | As you would type it into a search box | +| `numResults` | integer, optional | 1–20, default 10 | +| `recency` | `day` \| `week` \| `month` \| `year`, optional | Omit for no time limit | +| `engine` | `google` \| `duckduckgo` \| `bing` \| `brave`, optional | Overrides the configured default for one call | + +It returns titles, URLs and snippets — **search results only**. It does not fetch +the linked pages; follow up with a read/fetch tool for full content. + +## Engines + +Four, with independent indexes and independent rate limits. That last part is +the point: when one starts serving captchas, the others generally still work, +and the tool tells the agent so inside the error. + +| Engine | Recency windows | Notes | +|---|---|---| +| `google` | day, week, month, year | Best results; blocks aggressively under automation | +| `duckduckgo` | day, week, month, year | No-JS endpoint — the most stable markup here, least likely to challenge | +| `bing` | day, week, month | No year window exists | +| `brave` | none | Independent index, unwrapped links; challenges quickly | + +Aliases are accepted: `ddg`, `duck`, `g`, `b`. + +**An unsupported recency window is refused, not ignored.** Asking Bing for the +past year raises `RecencyUnsupportedError` naming the engines that can do it. +Silently returning unfiltered results would be indistinguishable from success. + +### Switching engines + +Exactly like switching the CDP host, plus a per-call override: + +``` +/search-engine show the current engine and where it came from +/search-engine ddg persist a choice (machine-wide, survives restarts) +/search-engine default back to Google +CCS_SEARCH_ENGINE=bing pi … pin for one process +``` + +Precedence: **`engine` parameter > `CCS_SEARCH_ENGINE` > `/search-engine` > +Google.** The stored choice lives in +`~/.pi/agent/castle-cdp-search-engine.json`, written atomically. + +Note the tool parameter sits *above* the environment variable, which is the +opposite of how the CDP target treats an explicit endpoint. That is deliberate: +for the browser, an env var must win because quietly driving a different machine +means automating the operator's own signed-in Chrome — a safety property. +Choosing a different search engine carries no such hazard, and letting the agent +fall back when one engine is blocked is the single most useful thing it can do +with this tool. + +An unrecognised engine name is reported rather than skipped, so a typo in +`CCS_SEARCH_ENGINE` never silently searches Google instead. + +## Which browser it drives + +Exactly the same configuration as [`pi-browser-harness`](../pi-browser-harness), +on purpose: one decision governs both packages, so they can never end up driving +different browsers in the same session. + +| `BU_CDP_HTTP` | Behaviour | +|---|---| +| unset | the `/browser-target` choice stored by pi-browser-harness, else castle (`10.88.0.25:9223`) | +| `host:port` | that endpoint | +| `local` (also `localhost`, `off`, `none`, `0`, `no`) | this machine's own browser on `127.0.0.1:9222` | + +Precedence is `BU_CDP_HTTP` > `/browser-target` > built-in default, matching the +harness. The `/browser-target` file (`~/.pi/agent/browser-target.json`) is read +only, never written — the harness owns that setting. + +**One deliberate difference from the harness.** When the configured endpoint is +unreachable, this extension reports that and stops. The harness falls back from +its built-in default to a local browser; here that fallback would mean silently +running searches through the operator's own signed-in Chrome without saying so. +Say `local` explicitly if that is what you want. + +The endpoint is re-resolved from `/json/version` on every connect, never cached: +castle recycles Chrome hourly and Chrome re-mints its browser UUID each launch, +so a pinned `ws://` URL breaks on the first recycle. A dropped socket reconnects +transparently on the next search. + +## Behaviour worth knowing + +- **Nothing is dialled at load.** pi runs extension factories in invocations that + never start a session, so the connection is made on first search and closed by + an idempotent `session_shutdown` handler. +- **One fresh background tab per search**, closed on every path including errors + and aborts. `background: true` so it does not steal focus from whoever is + looking at that screen. +- **Two concurrent searches**, queued beyond that. +- **20-second deadline** per call, covering the queue wait, connect, navigate and + extract together. Esc aborts an in-flight navigation. +- **Every failure throws.** pi only sets `isError: true` when `execute()` throws; + a returned `{ error }` object would read to the model as a search that simply + found nothing. + +## What it touches + +The browser is shared and belongs to a person. Worth being aware of: + +- Queries the agent runs land in that browser profile's history and cookies, and + in the search engine's account history if that profile is signed in. +- Tabs open and close on someone's screen. `background: true` keeps them from + stealing focus, but they are visible. +- Searching hard trips rate limiters for the whole host, and the challenge + affects the human using that browser too. Measured during development: + **Google** tolerated a lot, then roughly 30 queries in a few minutes cost about + 90 minutes of blocking. **Brave** is far stricter — it challenges after as few + as one or two queries and clears again in roughly 15–20 minutes. **DuckDuckGo** + and **Bing** never challenged once across the whole of development, which is + why DuckDuckGo is the one to reach for when you need to run several searches + in a row. +- **A decaying block does not look like a block.** Once the challenge page stops + being served, engines return an *empty results page* for a while instead — + indistinguishable from a query with no hits. It surfaces as `NoResultsError`, + not `SearchChallengeError`. If one engine comes back empty and another answers + the same query, that is rate limiting. + +## When a captcha appears + +Every engine eventually serves an interstitial: Google's `/sorry/` page, Brave's +"Verifying you're not a bot", a consent wall, a Cloudflare challenge. The +extension detects these and throws `SearchChallengeError` naming the challenge, +the engine, the browser, and the URL a human would have to visit. + +The error tells the agent to **try another engine first**, because a challenge on +one engine says nothing about the others and that is a fix it can apply itself. +Only when engines run out should it ask the user to clear the challenge in the +browser. The cookie is profile-wide, so one pass unblocks later searches. + +The tab is closed rather than left open on the challenge page: leaving it would +let the operator solve it in place, but would also litter a shared browser with +abandoned tabs on every failure. The URL in the error is enough. + +## Search parameters, and three traps + +Everything below was observed against castle's real Chrome, not inferred. All +three fail *silently* — the results look entirely plausible, just wrong. + +1. **Google: `tbs=qdr:*` renders an empty page.** That is the parameter Google's + own Tools menu writes. The older `as_qdr=*` works. +2. **Google: any date filter combined with `udm=14` renders that same empty + page.** So a time-limited search drops `udm` and uses the classic layout. +3. **Bing: `count=` silently cancels `filters=`.** With `ex1:"ez1"` alone every + result is hours old; add `count` in either order and months-old results come + back, unfiltered and unremarkable-looking. So Bing drops `count` whenever a + date filter is present and the caller slices the list instead. + +Each has a regression test asserting the parameter combination, since none of +them would announce itself if it regressed. + +## Result extraction + +Two modes, because the engines genuinely differ: + +- **items** — DuckDuckGo (`.result`), Bing (`li.b_algo`) and Brave + (`.snippet[data-type=web]`) each have a clean per-result container. +- **headings** — Google has no stable container; its class names (`MjjYud`, + `kb0PBd`, `yuRUbf`) are generated. Results are found from each `<h3>` outward, + climbing to the enclosing block but abandoning the climb if a second `<h3>` + comes into scope. + +DuckDuckGo and Bing both route outbound links through redirectors +(`duckduckgo.com/l/?uddg=…`, `bing.com/ck/a?…&u=a1<base64url>`); both are +unwrapped in the page so the agent gets real URLs. Google and Brave link +straight out. + +Markup will change. When it does, `NoResultsError` distinguishes "container +rendered but unparseable" (that engine's extractor needs updating) from "no +container at all" (query, or rate limiting). + +## Install + +``` +pi install ssh://sz@10.88.0.25/Users/sz/repos/pi-castle-cdp-search.git +``` + +The ssh form is what pi-browser-harness uses, and it is the portable one: pi +records local-path installs by resolved absolute path, and `settings.json` is +shared across hosts through the dotfiles symlink. + +To try it without installing: + +``` +pi -e ./src/index.ts +``` + +## Development + +``` +npm install +npm run typecheck +npm test # unit tests, no browser needed +``` + +The tests cover engine selection and precedence, URL building for all four +engines (including the three traps above), recency refusal, formatting, the +challenge error, and the concurrency semaphore. The semaphore tests point at an +unreachable endpoint on purpose — the questions there are about slot +bookkeeping, and hitting a real browser to test a counter would be slow, flaky, +and rude to whoever is using it. + +Requires Node 22+, for the global `WebSocket`. There are no runtime +dependencies; pi supplies `@earendil-works/pi-coding-agent`, +`@earendil-works/pi-ai` and `typebox` to extensions through its own loader. |