aboutsummaryrefslogtreecommitdiff
path: root/castle-search/README.md
diff options
context:
space:
mode:
authorIgor Soarez <igor@soarez.org>2026-11-06 03:33:30 +0000
committerIgor Soarez <igor@soarez.org>2026-11-06 03:33:30 +0000
commit561f0ee076c684a79fd5671af1bad0894f33e4de (patch)
tree7849677412a8b5f80835fa812f2e255f222a2c82 /castle-search/README.md
parent147b3b1507ebcdf5912fbd4c474c9dfb35265148 (diff)
castle-search: resolve Google links lazily, in Chrome, not eagerly over plain HTTPHEADmaster
The Node-side resolver sent one curl-like request per result with no cookies, JS or browser; Google answers that with its 'Verifying your request' countdown after a handful, and the testing itself triggered it. A click or direct navigation in Chrome redirects at once (measured). Search now leaves Google links wrapped and prints the breadcrumb Google shows for each. castle_cdp_resolve follows one wrapped link in a Chrome tab when the agent picks it, accepts only google.com/goto?url=<token>, and reports the verification page instead of waiting it out.
Diffstat (limited to 'castle-search/README.md')
-rw-r--r--castle-search/README.md20
1 files changed, 18 insertions, 2 deletions
diff --git a/castle-search/README.md b/castle-search/README.md
index 23a36e6..f3fb5e9 100644
--- a/castle-search/README.md
+++ b/castle-search/README.md
@@ -184,8 +184,24 @@ Two modes, because the engines genuinely differ:
DuckDuckGo and Bing both route outbound links through redirectors
(`duckduckgo.com/l/?uddg=…`, `bing.com/ck/a?…&u=a1<base64url>`); both are
-unwrapped in the page so the agent gets real URLs. Google and Brave link
-straight out.
+unwrapped in the page so the agent gets real URLs. Brave links straight out.
+
+**Google wraps every result** as `google.com/goto?url=<opaque token>` (rolled
+out June–August 2026; Google called it "technical measures against evolving
+forms of abuse" and has not documented it). The token is not reversible and the
+target is not in the DOM. So search leaves Google links wrapped, prints the
+breadcrumb Google shows for each, and `castle_cdp_resolve` follows one link in
+Chrome — one tab, closed afterwards — when the agent decides it wants it.
+Resolve lazily: a plain HTTP client (curl, Node `fetch`) is answered with a
+"Verifying your request" page and a 17–19 s countdown after a handful of
+requests, while a click or a direct navigation in Chrome redirects at once
+(measured 2026-11-06). If Chrome does get the countdown, `GoogleVerificationError`
+says so; it is never waited out or clicked through. Never resolve links with
+plain HTTP: that was tried first, and the testing itself triggered the check.
+
+An engine's own "no results" page is not an extractor failure. DuckDuckGo's
+notice is itself a `.result`, so `emptyMarker` identifies it and the error says
+the query was too narrow rather than blaming markup or a rate limit.
Markup will change. When it does, `NoResultsError` distinguishes "container
rendered but unparseable" (that engine's extractor needs updating) from "no