| Age | Commit message (Collapse) | Author |
|---|
|
Google, Bing and Brave all put a generated answer above their results now,
and none of it is being read. Investigated against castle's Chrome; nothing
in src/ changed, because the cost turned out to be worth deciding on rather
than absorbing.
Two things make it expensive. Google's AI Overview exists only on the
classic layout — udm=14 is the "Web" tab, and having no overview is part of
what that tab is — so enabling it means giving back the clean page the
extractor was built around. And all three stream in after the results are
already parseable: Google settles ~2.3s in, Brave was still growing at 5s,
against an extractor that returns at ~0.5s. The wait is often wasted, since
the blocks are query-dependent — no Google overview for "how does postgres
MVCC vacuum work" after 15s, no Bing answer for "who founded anthropic".
Selectors, the traps behind them (#m-x-content briefly holds ~29k
characters of unrelated page; Google's "Show more" hides only the
disclaimer), the measured timings, three possible designs and their
tradeoffs are all in the doc, so this does not have to be re-measured if
the answer turns out to be yes.
|
|
Concrete numbers from development rather than a vague warning: Google
tolerates a lot then blocks for ~90 minutes, Brave challenges after one or
two queries and clears in ~15-20, and DuckDuckGo and Bing never challenged
at all. That last fact is the useful one — it says which engine to reach
for when several searches are needed in a row.
|
|
Brave's source name was leaking into every snippet ("Medium March 27,
2025 - ..."), because the .sitename/.netloc selectors guessed for it match
nothing. Rather than guess again, use the ordering: an engine rendering
"source / breadcrumb / title / description" puts all its metadata before
the title, so everything after the title line is the description. Class
names churn; that ordering does not. The named selectors stay as the
fallback for when the title is not on a line of its own.
Verified against Brave, and Google/DuckDuckGo/Bing re-checked for
regressions.
|
|
ctx.ui.notify is a no-op when hasUI is false, so `pi -p /search-engine`
changed the stored engine and printed nothing — the command looked broken.
Print mode now falls back to stdout. JSON mode deliberately does not: its
stdout is a structured event stream and a stray line would corrupt it.
|
|
Engine resolution mirrors the browser target: CCS_SEARCH_ENGINE, then a
/search-engine choice persisted machine-wide, then Google. A per-call
`engine` parameter sits above both so the agent can fall back when one
engine starts serving captchas — the one case where the model, not the
operator, has to make the call. Unlike the CDP target there is no safety
argument for the environment winning: driving the wrong browser means
automating someone's signed-in Chrome, choosing a different index does not.
An unsupported recency window is refused, naming the engines that support
it, rather than dropped. Silently returning unfiltered results is
indistinguishable from success, which is the failure this whole design is
trying to avoid.
Extraction grows a second mode. DuckDuckGo, Bing and Brave have clean
per-result containers; Google does not, so its heading-walk stays as its
own path rather than being bent into the item shape. DuckDuckGo and Bing
route links through redirectors, unwrapped in the page.
Third silent-failure trap found, alongside Google's two: on Bing, `count`
cancels `filters`. With ex1:"ez1" alone every result is hours old; add
count in either order and months-old results return, looking perfectly
ordinary. Bing now drops count whenever a date filter is present.
Challenge detection widened to Brave's "Verifying you're not a bot" and
"Quick check before you continue searching", which the previous Google-
shaped matcher missed entirely — found by tripping it.
|
|
Attaches to a browser that is already running — never launches one —
using the same endpoint configuration as pi-browser-harness, so a single
/browser-target choice governs both packages.
One tool, castle_cdp_search, deliberately not named web_search so it
coexists with pi-web-access rather than shadowing it.
Notes from validating against castle's Chrome:
- tbs=qdr:*, the parameter Google's own Tools menu writes, renders an
empty page on this profile; the older as_qdr=* works. Any date filter
combined with udm=14 is also empty, so recency drops udm.
- Target.createTarget must not be raced against the abort signal:
raceAbort abandons the promise but cannot cancel the command, and the
command's side effect is a tab nothing is left holding.
- A search cancelled while queued has to be removed from the semaphore
queue, or the slot handed to it later is never counted back.
- A decaying rate-limit block stops serving /sorry/ and returns an empty
results page instead, indistinguishable from a genuine zero-hit search.
|
