Write up what AI answer blocks would cost before building one

Google, Bing and Brave all put a generated answer above their results now,
and none of it is being read. Investigated against castle's Chrome; nothing
in src/ changed, because the cost turned out to be worth deciding on rather
than absorbing.

Two things make it expensive. Google's AI Overview exists only on the
classic layout — udm=14 is the "Web" tab, and having no overview is part of
what that tab is — so enabling it means giving back the clean page the
extractor was built around. And all three stream in after the results are
already parseable: Google settles ~2.3s in, Brave was still growing at 5s,
against an extractor that returns at ~0.5s. The wait is often wasted, since
the blocks are query-dependent — no Google overview for "how does postgres
MVCC vacuum work" after 15s, no Bing answer for "who founded anthropic".

Selectors, the traps behind them (#m-x-content briefly holds ~29k
characters of unrelated page; Google's "Show more" hides only the
disclaimer), the measured timings, three possible designs and their
tradeoffs are all in the doc, so this does not have to be re-measured if
the answer turns out to be yes.

1 files changed, 70 insertions, 0 deletions

diff --git a/castle-search/test/format.test.ts b/castle-search/test/format.test.ts
new file mode 100644
index 0000000..74e37e1
--- /dev/null
+++ b/castle-search/test/format.test.ts
@@ -0,0 +1,70 @@
+import assert from "node:assert/strict";
+import { test } from "node:test";
+
+import { formatResults } from "../src/format.ts";
+import { SearchChallengeError } from "../src/errors.ts";
+
+const results = [
+  { title: "First", url: "https://example.com/1", snippet: "A snippet." },
+  { title: "Second", url: "https://example.com/2", snippet: "" },
+];
+
+test("results are numbered, with the URL on its own line", () => {
+  const text = formatResults({
+    query: "q",
+    searchUrl: "https://www.google.com/search?q=q",
+    finalUrl: "https://www.google.com/search?q=q",
+    endpoint: "10.88.0.25:9223 (built-in default)",
+    engine: "Google",
+    engineSource: "built-in default",
+    results,
+  });
+  // The engine has to be named on every result set: the agent may have switched
+  // engines to get around a block, and two indexes are not interchangeable.
+  assert.match(
+    text,
+    /^Search results for "q" — 2 hits from Google \(built-in default\), via Chrome at 10\.88\.0\.25:9223 \(built-in default\)$/m,
+  );
+  assert.match(text, /^1\. First$/m);
+  assert.match(text, /^ {3}https:\/\/example\.com\/1$/m);
+  assert.match(text, /^2\. Second$/m);
+  // A missing snippet must not leave a stray blank indented line.
+  assert.ok(!/\n {3}\n/.test(text));
+});
+
+test("a redirect is reported, because the query URL is then not where we looked", () => {
+  const text = formatResults({
+    query: "q",
+    searchUrl: "https://www.google.com/search?q=q",
+    finalUrl: "https://www.google.com/search?q=q&sei=abc",
+    endpoint: "e",
+    engine: "Bing",
+    engineSource: "/search-engine",
+    results,
+  });
+  assert.match(text, /^Landed on: https:\/\/www\.google\.com\/search\?q=q&sei=abc$/m);
+});
+
+test("the challenge error tells the agent to hand off to a human, with the URL", () => {
+  const err = new SearchChallengeError({
+    challenge: "captcha-widget",
+    pageUrl: "https://www.google.com/sorry/index?continue=x",
+    endpoint: "10.88.0.25:9223 (built-in default)",
+    engine: "Google",
+    alternatives: ["duckduckgo", "bing", "brave"],
+    detail: "Our systems have detected unusual traffic",
+  });
+  assert.equal(err.name, "SearchChallengeError");
+  assert.ok(err instanceof Error, "must be throwable as an Error");
+  assert.match(err.message, /captcha-widget/);
+  assert.match(err.message, /https:\/\/www\.google\.com\/sorry\/index\?continue=x/);
+  assert.match(err.message, /Ask the user/);
+  assert.match(err.message, /10\.88\.0\.25:9223/);
+  // Switching engines is the agent's own move and must be offered before the
+  // human handoff, since a block on one engine says nothing about the others.
+  for (const alt of ["duckduckgo", "bing", "brave"]) assert.match(err.message, new RegExp(alt));
+  assert.ok(
+    err.message.indexOf("retry the same query with engine") < err.message.indexOf("Ask the user"),
+    "the self-service fallback must come before the human handoff",
+  );
+});