aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorTianren Dong <tianrendong.dtr@gmail.com>2026-05-16 14:12:13 -0400
committerTianren Dong <tianrendong.dtr@gmail.com>2026-05-16 14:12:13 -0400
commit0c3613891604d10e332d31762c9c3345c357a3bb (patch)
tree4cf73bbea05328e2dfff01e69714a6959954f956 /docs
parentb792f22f2c2ef4dfb7bd300e7381a186e34e0c18 (diff)
0.15.26: update docs for benchmark gates
Diffstat (limited to 'docs')
-rw-r--r--docs/COMPARISON.md4
-rw-r--r--docs/EXAMPLES.md4
-rw-r--r--docs/FAQ.md8
3 files changed, 11 insertions, 5 deletions
diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md
index 7dfe5ed..3f8db12 100644
--- a/docs/COMPARISON.md
+++ b/docs/COMPARISON.md
@@ -134,7 +134,7 @@ If your threat model excludes extensions with broad permissions, neither approac
## Public benchmarks worth knowing (for axis 2 / axis 3 comparison)
-Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering real input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, file uploads, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count. That's **driver-level** grading.
+Pi-chrome itself ships a benchmark suite ([`../test-suite/`](../test-suite)) of **41 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks** covering trusted input, pointer humanization, keyboard fidelity, drag/drop, Shadow DOM, iframes, file uploads, strict-CSP screenshot fallback, dynamic waits, tab lifecycle, network observability, fingerprint leaks, and agent-safety honeypots. Scoring tracks expected outcomes per challenge instead of raw PASS count, with `core`, `conditional`, and `quality` gate buckets. That's **driver-level** grading.
For **agent-level** comparison (axis 2), the public benchmarks worth citing:
@@ -156,7 +156,7 @@ Cite live leaderboards rather than hard-coded numbers; agent scores shift monthl
## Reproducing pi-chrome's driver-level claims
-Run [`../test-suite/`](../test-suite) against any browser-control tool. Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events`, so any tool (Playwright, Puppeteer, Selenium, Stagehand, pi-chrome) can grade itself deterministically.
+Run [`../test-suite/`](../test-suite) against any browser-control tool. Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events`, so any tool (Playwright, Puppeteer, Selenium, Stagehand, pi-chrome) can grade itself deterministically. Headline release scoring should use the `core` gate; `conditional` depends on declared environment capabilities, and `quality` tracks adversarial/humanization regressions.
```bash
cd test-suite && python3 -m http.server 8765
diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md
index b3c6286..bf3aa0a 100644
--- a/docs/EXAMPLES.md
+++ b/docs/EXAMPLES.md
@@ -159,6 +159,8 @@ Interactive tools use Chrome's real input layer by default: clicks, typing, fill
- guarded buttons
- audio/video controls
- fullscreen and other user-activation checks
-- pages with strict CSP or user-activation checks
+- pages where DOM injection/evaluate is limited, if the agent can use screenshots + coordinates
+
+Strict CSP note: `chrome_snapshot`/`chrome_evaluate` may be blocked on pages that disallow `unsafe-eval`; `chrome_screenshot`, tab/navigation tools, and real input still work.
Chrome may show its debugger banner while pi-chrome is attached.
diff --git a/docs/FAQ.md b/docs/FAQ.md
index f7f871f..f821da7 100644
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -14,9 +14,9 @@ By default no — extensions need explicit "Allow in incognito" permission. Togg
## Will sites detect that I'm automating?
-Interactive controls use Chrome's real input layer via CDP: pointer paths are humanized, key cadence has variance, and normal user-activation gates are satisfied. Some detectors check for the `chrome.debugger` API attached and Chrome will show the "Chrome is being debugged" banner.
+Interactive controls use Chrome's real input layer via CDP, so normal user-activation gates are satisfied and input is closer to real browser use than DOM-dispatched events. pi-chrome also shapes pointer/keyboard/scroll behavior, but this is not a guarantee of undetectability. Some detectors check for the `chrome.debugger` API attached, and Chrome will show the "Chrome is being debugged" banner.
-The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals.
+The [`test-suite/`](../test-suite) grades browser-control behavior against common detection signals. Its `quality` bucket is adversarial signal, not a blanket promise that every site will treat automation as human.
## Why do I see a banner saying "Pi Chrome Connector started debugging this browser"?
@@ -49,6 +49,10 @@ pi-chrome ships as an unpacked extension so the source and broad browser permiss
The Pi-facing tools are thin wrappers around an HTTP bridge at `127.0.0.1:17318`. You could call it directly from any process, but the API is internal and may change. If you need a stable scripting interface, file an issue and we'll consider stabilizing.
+## What can humans do that pi-chrome cannot?
+
+pi-chrome controls web pages through Chrome extension APIs, page inspection, screenshots, and browser input. It is not full OS-level human control. Known gaps include native Chrome/OS dialogs (print/save-as, some permission bubbles, password-manager prompts), arbitrary OS app interaction, visual CAPTCHA challenges, hardware-backed auth (passkeys/security keys/biometrics), rich multi-touch/pinch/stylus gestures, and DOM inspection inside cross-origin iframes. Some of these can still be handled with screenshot + coordinate input or user assistance, but they are not first-class deterministic workflows.
+
## Does `chrome_evaluate` work on strict-CSP pages?
Not always. `chrome_evaluate` and `chrome_snapshot` run in the page's MAIN world through the Function constructor, so pages whose CSP blocks `'unsafe-eval'` can reject them. `chrome_screenshot`, `chrome_navigate`, tab tools, and real Chrome input still work because they use extension/browser APIs rather than page JavaScript.