diff options
Diffstat (limited to 'castle-search/README.md')
| -rw-r--r-- | castle-search/README.md | 20 |
1 files changed, 18 insertions, 2 deletions
diff --git a/castle-search/README.md b/castle-search/README.md index 23a36e6..f3fb5e9 100644 --- a/castle-search/README.md +++ b/castle-search/README.md @@ -184,8 +184,24 @@ Two modes, because the engines genuinely differ: DuckDuckGo and Bing both route outbound links through redirectors (`duckduckgo.com/l/?uddg=…`, `bing.com/ck/a?…&u=a1<base64url>`); both are -unwrapped in the page so the agent gets real URLs. Google and Brave link -straight out. +unwrapped in the page so the agent gets real URLs. Brave links straight out. + +**Google wraps every result** as `google.com/goto?url=<opaque token>` (rolled +out June–August 2026; Google called it "technical measures against evolving +forms of abuse" and has not documented it). The token is not reversible and the +target is not in the DOM. So search leaves Google links wrapped, prints the +breadcrumb Google shows for each, and `castle_cdp_resolve` follows one link in +Chrome — one tab, closed afterwards — when the agent decides it wants it. +Resolve lazily: a plain HTTP client (curl, Node `fetch`) is answered with a +"Verifying your request" page and a 17–19 s countdown after a handful of +requests, while a click or a direct navigation in Chrome redirects at once +(measured 2026-11-06). If Chrome does get the countdown, `GoogleVerificationError` +says so; it is never waited out or clicked through. Never resolve links with +plain HTTP: that was tried first, and the testing itself triggered the check. + +An engine's own "no results" page is not an extractor failure. DuckDuckGo's +notice is itself a `.result`, so `emptyMarker` identifies it and the error says +the query was too narrow rather than blaming markup or a rate limit. Markup will change. When it does, `NoResultsError` distinguishes "container rendered but unparseable" (that engine's extractor needs updating) from "no |