castle-search: resolve Google links lazily, in Chrome, not eagerly over plain HTTP

2026-11-06T03:33:30+00:00

The Node-side resolver sent one curl-like request per result with no cookies, JS or browser; Google answers that with its 'Verifying your request' countdown after a handful, and the testing itself triggered it. A click or direct navigation in Chrome redirects at once (measured). Search now leaves Google links wrapped and prints the breadcrumb Google shows for each. castle_cdp_resolve follows one wrapped link in a Chrome tab when the agent picks it, accepts only google.com/goto?url=, and reports the verification page instead of waiting it out.

Write up what AI answer blocks would cost before building one

2026-08-04T14:08:47+00:00

Google, Bing and Brave all put a generated answer above their results now, and none of it is being read. Investigated against castle's Chrome; nothing in src/ changed, because the cost turned out to be worth deciding on rather than absorbing. Two things make it expensive. Google's AI Overview exists only on the classic layout — udm=14 is the "Web" tab, and having no overview is part of what that tab is — so enabling it means giving back the clean page the extractor was built around. And all three stream in after the results are already parseable: Google settles ~2.3s in, Brave was still growing at 5s, against an extractor that returns at ~0.5s. The wait is often wasted, since the blocks are query-dependent — no Google overview for "how does postgres MVCC vacuum work" after 15s, no Bing answer for "who founded anthropic". Selectors, the traps behind them (#m-x-content briefly holds ~29k characters of unrelated page; Google's "Show more" hides only the disclaimer), the measured timings, three possible designs and their tradeoffs are all in the doc, so this does not have to be re-measured if the answer turns out to be yes.

sz-chrome-remote-cdp.git/castle-search/src/errors.ts, branch master

castle-search: resolve Google links lazily, in Chrome, not eagerly over plain HTTP

Write up what AI answer blocks would cost before building one