sz-chrome-remote-cdp.git/castle-search/src/search.ts, branch master

castle-search: resolve Google links lazily, in Chrome, not eagerly over plain HTTP

2026-11-06T03:33:30+00:00

The Node-side resolver sent one curl-like request per result with no cookies, JS or browser; Google answers that with its 'Verifying your request' countdown after a handful, and the testing itself triggered it. A click or direct navigation in Chrome redirects at once (measured). Search now leaves Google links wrapped and prints the breadcrumb Google shows for each. castle_cdp_resolve follows one wrapped link in a Chrome tab when the agent picks it, accepts only google.com/goto?url=, and reports the verification page instead of waiting it out.

castle-search: read Brave again, and report genuine zero-hit searches as such

2026-11-06T03:18:21+00:00

Brave: #results no longer exists on the live page; fall back to #main/main. It was reported as 'no results area at all', i.e. a rate limit. DuckDuckGo: its 'No results found' notice is itself a .result, so an empty search looked like broken markup. emptyMarker lets the probe say the engine itself reported no matches, and the error tells the caller to loosen the query instead of switching engines.

castle-search: read Google results again by resolving its /goto links

2026-11-06T03:17:16+00:00

Google now wraps every result as google.com/goto?url=. The self-host filter discarded all of them, so every Google search failed with 'nothing could be parsed'. Accept /goto, resolve it node-side from the 302 Location, and when Google answers its 'Verifying your request' interstitial instead, keep the wrapped link and put the displayed breadcrumb in the snippet.

Write up what AI answer blocks would cost before building one

2026-08-04T14:08:47+00:00

Google, Bing and Brave all put a generated answer above their results now, and none of it is being read. Investigated against castle's Chrome; nothing in src/ changed, because the cost turned out to be worth deciding on rather than absorbing. Two things make it expensive. Google's AI Overview exists only on the classic layout — udm=14 is the "Web" tab, and having no overview is part of what that tab is — so enabling it means giving back the clean page the extractor was built around. And all three stream in after the results are already parseable: Google settles ~2.3s in, Brave was still growing at 5s, against an extractor that returns at ~0.5s. The wait is often wasted, since the blocks are query-dependent — no Google overview for "how does postgres MVCC vacuum work" after 15s, no Bing answer for "who founded anthropic". Selectors, the traps behind them (#m-x-content briefly holds ~29k characters of unrelated page; Google's "Show more" hides only the disclaimer), the measured timings, three possible designs and their tradeoffs are all in the doc, so this does not have to be re-measured if the answer turns out to be yes.