diff options
| author | Igor Soarez <igor@soarez.org> | 2026-11-06 03:33:30 +0000 |
|---|---|---|
| committer | Igor Soarez <igor@soarez.org> | 2026-11-06 03:33:30 +0000 |
| commit | 561f0ee076c684a79fd5671af1bad0894f33e4de (patch) | |
| tree | 7849677412a8b5f80835fa812f2e255f222a2c82 /castle-search/README.md | |
| parent | 147b3b1507ebcdf5912fbd4c474c9dfb35265148 (diff) | |
The Node-side resolver sent one curl-like request per result with no
cookies, JS or browser; Google answers that with its 'Verifying your
request' countdown after a handful, and the testing itself triggered it.
A click or direct navigation in Chrome redirects at once (measured).
Search now leaves Google links wrapped and prints the breadcrumb Google
shows for each. castle_cdp_resolve follows one wrapped link in a Chrome tab
when the agent picks it, accepts only google.com/goto?url=<token>, and
reports the verification page instead of waiting it out.
Diffstat (limited to 'castle-search/README.md')
| -rw-r--r-- | castle-search/README.md | 20 |
1 files changed, 18 insertions, 2 deletions
diff --git a/castle-search/README.md b/castle-search/README.md index 23a36e6..f3fb5e9 100644 --- a/castle-search/README.md +++ b/castle-search/README.md @@ -184,8 +184,24 @@ Two modes, because the engines genuinely differ: DuckDuckGo and Bing both route outbound links through redirectors (`duckduckgo.com/l/?uddg=…`, `bing.com/ck/a?…&u=a1<base64url>`); both are -unwrapped in the page so the agent gets real URLs. Google and Brave link -straight out. +unwrapped in the page so the agent gets real URLs. Brave links straight out. + +**Google wraps every result** as `google.com/goto?url=<opaque token>` (rolled +out June–August 2026; Google called it "technical measures against evolving +forms of abuse" and has not documented it). The token is not reversible and the +target is not in the DOM. So search leaves Google links wrapped, prints the +breadcrumb Google shows for each, and `castle_cdp_resolve` follows one link in +Chrome — one tab, closed afterwards — when the agent decides it wants it. +Resolve lazily: a plain HTTP client (curl, Node `fetch`) is answered with a +"Verifying your request" page and a 17–19 s countdown after a handful of +requests, while a click or a direct navigation in Chrome redirects at once +(measured 2026-11-06). If Chrome does get the countdown, `GoogleVerificationError` +says so; it is never waited out or clicked through. Never resolve links with +plain HTTP: that was tried first, and the testing itself triggered the check. + +An engine's own "no results" page is not an extractor failure. DuckDuckGo's +notice is itself a `.result`, so `emptyMarker` identifies it and the error says +the query was too narrow rather than blaming markup or a rate limit. Markup will change. When it does, `NoResultsError` distinguishes "container rendered but unparseable" (that engine's extractor needs updating) from "no |
