path: root/README.md

# pi-castle-cdp-search

Web search for [pi](https://pi.dev) through a Chrome that is **already running**
on the wire network, driven over the Chrome DevTools Protocol.

It never launches a browser. It attaches to one a person or a launchd job
started — by default castle's shared Chrome on `10.88.0.25:9223` — opens a
background tab, reads the results page, and closes the tab again.

Because the search runs in a real browser with a real profile, it reaches pages
that reject datacentre traffic, and it costs nothing per query.

## The tool

One tool, `castle_cdp_search`. Deliberately **not** named `web_search`, so it
coexists with `pi-web-access` rather than shadowing it — both stay registered and
the model picks.

| Parameter | Type | Notes |
|---|---|---|
| `query` | string, required | As you would type it into a search box |
| `numResults` | integer, optional | 1–20, default 10 |
| `recency` | `day` \| `week` \| `month` \| `year`, optional | Omit for no time limit |
| `engine` | `google` \| `duckduckgo` \| `bing` \| `brave`, optional | Overrides the configured default for one call |

It returns titles, URLs and snippets — **search results only**. It does not fetch
the linked pages; follow up with a read/fetch tool for full content.

## Engines

Four, with independent indexes and independent rate limits. That last part is
the point: when one starts serving captchas, the others generally still work,
and the tool tells the agent so inside the error.

| Engine | Recency windows | Notes |
|---|---|---|
| `google` | day, week, month, year | Best results; blocks aggressively under automation |
| `duckduckgo` | day, week, month, year | No-JS endpoint — the most stable markup here, least likely to challenge |
| `bing` | day, week, month | No year window exists |
| `brave` | none | Independent index, unwrapped links; challenges quickly |

Aliases are accepted: `ddg`, `duck`, `g`, `b`.

**An unsupported recency window is refused, not ignored.** Asking Bing for the
past year raises `RecencyUnsupportedError` naming the engines that can do it.
Silently returning unfiltered results would be indistinguishable from success.

### Switching engines

Exactly like switching the CDP host, plus a per-call override:

```
/search-engine                 show the current engine and where it came from
/search-engine ddg             persist a choice (machine-wide, survives restarts)
/search-engine default         back to Google
CCS_SEARCH_ENGINE=bing pi …    pin for one process
```

Precedence: **`engine` parameter > `CCS_SEARCH_ENGINE` > `/search-engine` >
Google.** The stored choice lives in
`~/.pi/agent/castle-cdp-search-engine.json`, written atomically.

Note the tool parameter sits *above* the environment variable, which is the
opposite of how the CDP target treats an explicit endpoint. That is deliberate:
for the browser, an env var must win because quietly driving a different machine
means automating the operator's own signed-in Chrome — a safety property.
Choosing a different search engine carries no such hazard, and letting the agent
fall back when one engine is blocked is the single most useful thing it can do
with this tool.

An unrecognised engine name is reported rather than skipped, so a typo in
`CCS_SEARCH_ENGINE` never silently searches Google instead.

## Which browser it drives

Exactly the same configuration as [`pi-browser-harness`](../pi-browser-harness),
on purpose: one decision governs both packages, so they can never end up driving
different browsers in the same session.

| `BU_CDP_HTTP` | Behaviour |
|---|---|
| unset | the `/browser-target` choice stored by pi-browser-harness, else castle (`10.88.0.25:9223`) |
| `host:port` | that endpoint |
| `local` (also `localhost`, `off`, `none`, `0`, `no`) | this machine's own browser on `127.0.0.1:9222` |

Precedence is `BU_CDP_HTTP` > `/browser-target` > built-in default, matching the
harness. The `/browser-target` file (`~/.pi/agent/browser-target.json`) is read
only, never written — the harness owns that setting.

**One deliberate difference from the harness.** When the configured endpoint is
unreachable, this extension reports that and stops. The harness falls back from
its built-in default to a local browser; here that fallback would mean silently
running searches through the operator's own signed-in Chrome without saying so.
Say `local` explicitly if that is what you want.

The endpoint is re-resolved from `/json/version` on every connect, never cached:
castle recycles Chrome hourly and Chrome re-mints its browser UUID each launch,
so a pinned `ws://` URL breaks on the first recycle. A dropped socket reconnects
transparently on the next search.

## Behaviour worth knowing

- **Nothing is dialled at load.** pi runs extension factories in invocations that
  never start a session, so the connection is made on first search and closed by
  an idempotent `session_shutdown` handler.
- **One fresh background tab per search**, closed on every path including errors
  and aborts. `background: true` so it does not steal focus from whoever is
  looking at that screen.
- **Two concurrent searches**, queued beyond that.
- **20-second deadline** per call, covering the queue wait, connect, navigate and
  extract together. Esc aborts an in-flight navigation.
- **Every failure throws.** pi only sets `isError: true` when `execute()` throws;
  a returned `{ error }` object would read to the model as a search that simply
  found nothing.

## What it touches

The browser is shared and belongs to a person. Worth being aware of:

- Queries the agent runs land in that browser profile's history and cookies, and
  in the search engine's account history if that profile is signed in.
- Tabs open and close on someone's screen. `background: true` keeps them from
  stealing focus, but they are visible.
- Searching hard trips rate limiters for the whole host, and the challenge
  affects the human using that browser too. Measured during development:
  **Google** tolerated a lot, then roughly 30 queries in a few minutes cost about
  90 minutes of blocking. **Brave** is far stricter — it challenges after as few
  as one or two queries and clears again in roughly 15–20 minutes. **DuckDuckGo**
  and **Bing** never challenged once across the whole of development, which is
  why DuckDuckGo is the one to reach for when you need to run several searches
  in a row.
- **A decaying block does not look like a block.** Once the challenge page stops
  being served, engines return an *empty results page* for a while instead —
  indistinguishable from a query with no hits. It surfaces as `NoResultsError`,
  not `SearchChallengeError`. If one engine comes back empty and another answers
  the same query, that is rate limiting.

## When a captcha appears

Every engine eventually serves an interstitial: Google's `/sorry/` page, Brave's
"Verifying you're not a bot", a consent wall, a Cloudflare challenge. The
extension detects these and throws `SearchChallengeError` naming the challenge,
the engine, the browser, and the URL a human would have to visit.

The error tells the agent to **try another engine first**, because a challenge on
one engine says nothing about the others and that is a fix it can apply itself.
Only when engines run out should it ask the user to clear the challenge in the
browser. The cookie is profile-wide, so one pass unblocks later searches.

The tab is closed rather than left open on the challenge page: leaving it would
let the operator solve it in place, but would also litter a shared browser with
abandoned tabs on every failure. The URL in the error is enough.

## Search parameters, and three traps

Everything below was observed against castle's real Chrome, not inferred. All
three fail *silently* — the results look entirely plausible, just wrong.

1. **Google: `tbs=qdr:*` renders an empty page.** That is the parameter Google's
   own Tools menu writes. The older `as_qdr=*` works.
2. **Google: any date filter combined with `udm=14` renders that same empty
   page.** So a time-limited search drops `udm` and uses the classic layout.
3. **Bing: `count=` silently cancels `filters=`.** With `ex1:"ez1"` alone every
   result is hours old; add `count` in either order and months-old results come
   back, unfiltered and unremarkable-looking. So Bing drops `count` whenever a
   date filter is present and the caller slices the list instead.

Each has a regression test asserting the parameter combination, since none of
them would announce itself if it regressed.

## Result extraction

Two modes, because the engines genuinely differ:

- **items** — DuckDuckGo (`.result`), Bing (`li.b_algo`) and Brave
  (`.snippet[data-type=web]`) each have a clean per-result container.
- **headings** — Google has no stable container; its class names (`MjjYud`,
  `kb0PBd`, `yuRUbf`) are generated. Results are found from each `<h3>` outward,
  climbing to the enclosing block but abandoning the climb if a second `<h3>`
  comes into scope.

DuckDuckGo and Bing both route outbound links through redirectors
(`duckduckgo.com/l/?uddg=…`, `bing.com/ck/a?…&u=a1<base64url>`); both are
unwrapped in the page so the agent gets real URLs. Google and Brave link
straight out.

Markup will change. When it does, `NoResultsError` distinguishes "container
rendered but unparseable" (that engine's extractor needs updating) from "no
container at all" (query, or rate limiting).

## Install

```
pi install ssh://sz@10.88.0.25/Users/sz/repos/pi-castle-cdp-search.git
```

The ssh form is what pi-browser-harness uses, and it is the portable one: pi
records local-path installs by resolved absolute path, and `settings.json` is
shared across hosts through the dotfiles symlink.

To try it without installing:

```
pi -e ./src/index.ts
```

## Development

```
npm install
npm run typecheck
npm test          # unit tests, no browser needed
```

The tests cover engine selection and precedence, URL building for all four
engines (including the three traps above), recency refusal, formatting, the
challenge error, and the concurrency semaphore. The semaphore tests point at an
unreachable endpoint on purpose — the questions there are about slot
bookkeeping, and hitting a real browser to test a counter would be slow, flaky,
and rude to whoever is using it.

Requires Node 22+, for the global `WebSocket`. There are no runtime
dependencies; pi supplies `@earendil-works/pi-coding-agent`,
`@earendil-works/pi-ai` and `typebox` to extensions through its own loader.