| Age | Commit message (Collapse) | Author |
|---|
|
plain HTTP
The Node-side resolver sent one curl-like request per result with no
cookies, JS or browser; Google answers that with its 'Verifying your
request' countdown after a handful, and the testing itself triggered it.
A click or direct navigation in Chrome redirects at once (measured).
Search now leaves Google links wrapped and prints the breadcrumb Google
shows for each. castle_cdp_resolve follows one wrapped link in a Chrome tab
when the agent picks it, accepts only google.com/goto?url=<token>, and
reports the verification page instead of waiting it out.
|
|
Brave: #results no longer exists on the live page; fall back to #main/main.
It was reported as 'no results area at all', i.e. a rate limit.
DuckDuckGo: its 'No results found' notice is itself a .result, so an empty
search looked like broken markup. emptyMarker lets the probe say the engine
itself reported no matches, and the error tells the caller to loosen the
query instead of switching engines.
|
|
Google now wraps every result as google.com/goto?url=<opaque token>. The
self-host filter discarded all of them, so every Google search failed with
'nothing could be parsed'. Accept /goto, resolve it node-side from the
302 Location, and when Google answers its 'Verifying your request'
interstitial instead, keep the wrapped link and put the displayed
breadcrumb in the snippet.
|
|
|
|
|
|
Google, Bing and Brave all put a generated answer above their results now,
and none of it is being read. Investigated against castle's Chrome; nothing
in src/ changed, because the cost turned out to be worth deciding on rather
than absorbing.
Two things make it expensive. Google's AI Overview exists only on the
classic layout — udm=14 is the "Web" tab, and having no overview is part of
what that tab is — so enabling it means giving back the clean page the
extractor was built around. And all three stream in after the results are
already parseable: Google settles ~2.3s in, Brave was still growing at 5s,
against an extractor that returns at ~0.5s. The wait is often wasted, since
the blocks are query-dependent — no Google overview for "how does postgres
MVCC vacuum work" after 15s, no Bing answer for "who founded anthropic".
Selectors, the traps behind them (#m-x-content briefly holds ~29k
characters of unrelated page; Google's "Show more" hides only the
disclaimer), the measured timings, three possible designs and their
tradeoffs are all in the doc, so this does not have to be re-measured if
the answer turns out to be yes.
|
