aboutsummaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorTianren Dong <tianrendong.dtr@gmail.com>2026-05-16 14:12:13 -0400
committerTianren Dong <tianrendong.dtr@gmail.com>2026-05-16 14:12:13 -0400
commit0c3613891604d10e332d31762c9c3345c357a3bb (patch)
tree4cf73bbea05328e2dfff01e69714a6959954f956 /README.md
parentb792f22f2c2ef4dfb7bd300e7381a186e34e0c18 (diff)
0.15.26: update docs for benchmark gates
Diffstat (limited to 'README.md')
-rw-r--r--README.md17
1 files changed, 13 insertions, 4 deletions
diff --git a/README.md b/README.md
index 03cdb3b..361a689 100644
--- a/README.md
+++ b/README.md
@@ -149,6 +149,10 @@ Agents can verify page state immediately instead of blindly retrying.
Each tool is documented inline in Pi — agents see the parameters and gotchas (Chrome input, CSP limits, file upload behavior) without trial-and-error.
+### Known limits vs. human Chrome use
+
+pi-chrome is strongest on web-page workflows exposed through DOM, screenshots, tabs, and Chrome input. It is not a full human/OS substitute. Current limitations include native Chrome/OS surfaces (print/save dialogs, permission bubbles, password-manager prompts), cross-origin iframe DOM access, rich multitouch/pinch/stylus gestures, visual CAPTCHA/bot challenges, hardware-backed auth (passkeys/security keys/biometrics), and arbitrary OS app interaction. For strict-CSP pages, use screenshot + coordinate input when `chrome_snapshot`/`chrome_evaluate` are blocked.
+
---
## Click & input behavior
@@ -213,9 +217,13 @@ Multiple Pi sessions (planner / worker / audit) can all drive the same Chrome at
## Built-in benchmark suite
-[`test-suite/`](./test-suite) is a benchmark for **any** browser-control agent (not just pi-chrome). It includes **38 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks**.
+[`test-suite/`](./test-suite) is a benchmark for **any** browser-control agent (not just pi-chrome). It includes **41 primitive challenges** plus **4 hermetic BrowserGym-style long-horizon tasks**.
+
+Scoring tracks expected outcomes per challenge rather than raw PASS count, so tools are judged against their declared browser-control capability. Unit challenges are split into gate buckets:
-Scoring tracks expected outcomes per challenge rather than raw PASS count, so tools are judged against their declared browser-control capability.
+- `core` — expected release blockers for normal trusted-mode browser control.
+- `conditional` — capability/environment gated (clipboard, touch, dialogs, native UI, etc.).
+- `quality` — adversarial humanization/fingerprint signals; report trends, don't block general release by default.
Each challenge exposes `window.__verdict` / `window.__reason` / `window.__events` and a manifest entry with expected results per mode.
@@ -224,7 +232,7 @@ cd test-suite && python3 -m http.server 8765
# open http://127.0.0.1:8765/ in the Chrome window pi-chrome controls
```
-Categories: `real-input`, `pointer-humanization`, `keyboard`, `activation-gates`, `scroll`, `drag-drop`, `clipboard`, `native-controls`, `frameworks`, `editing`, `dom-complexity`, `frames`, `files`, `observability`, `fingerprint`, `agent-safety`.
+Categories include: `trusted-input`, `pointer-humanization`, `keyboard`, `focus-keyboard`, `activation-gates`, `scroll`, `drag-drop`, `clipboard`, `native-controls`, `frameworks`, `editing`, `dom-complexity`, `frames`, `files`, `observability`, `csp`, `lazy-loading`, `dialogs`, `popups`, `spa-routing`, `fingerprint`, and `agent-safety`.
If you build a competing tool, please open a PR with your scores. We benchmark in public.
@@ -247,7 +255,8 @@ There is no network exposure in the default configuration; the bridge binds to l
`pi-chrome` is actively shipped. Things on the near roadmap:
- More observability tools (DOM mutation streams, performance traces)
-- First-class iframe + Shadow-DOM uid stability across snapshots
+- First-class cross-origin iframe + Shadow-DOM uid stability across snapshots
+- Native-browser surface coverage where extension APIs allow it (downloads, permissions, context menus)
- Web Push & service worker introspection
- Recorder mode that emits agent prompts from your own clicks