aboutsummaryrefslogtreecommitdiff
path: root/castle-search/src/resolve.ts
AgeCommit message (Collapse)Author
19 hourscastle-search: resolve Google links lazily, in Chrome, not eagerly over ↵HEADmasterIgor Soarez
plain HTTP The Node-side resolver sent one curl-like request per result with no cookies, JS or browser; Google answers that with its 'Verifying your request' countdown after a handful, and the testing itself triggered it. A click or direct navigation in Chrome redirects at once (measured). Search now leaves Google links wrapped and prints the breadcrumb Google shows for each. castle_cdp_resolve follows one wrapped link in a Chrome tab when the agent picks it, accepts only google.com/goto?url=<token>, and reports the verification page instead of waiting it out.
19 hourscastle-search: read Google results again by resolving its /goto linksIgor Soarez
Google now wraps every result as google.com/goto?url=<opaque token>. The self-host filter discarded all of them, so every Google search failed with 'nothing could be parsed'. Accept /goto, resolve it node-side from the 302 Location, and when Google answers its 'Verifying your request' interstitial instead, keep the wrapped link and put the displayed breadcrumb in the snippet.