aboutsummaryrefslogtreecommitdiff
path: root/test-suite/README.md
diff options
context:
space:
mode:
authorTianren Dong <tianrendong.dtr@gmail.com>2026-05-13 02:35:01 -0400
committerTianren Dong <tianrendong.dtr@gmail.com>2026-05-13 02:35:01 -0400
commit26ab0d36e45b1b65f667d4743eb30e83389ff57f (patch)
tree678f92867c16671caa116b4e1762a672703f9863 /test-suite/README.md
parentb319c82a75e9a6488a6454790d1436f74a655e7c (diff)
pi-chrome 0.14.2: recover from foreign-extension input overlays
When a different Chrome extension (password manager, autofill popup, etc.) anchors an overlay to the focused input, Chrome's CDP refuses our Input.dispatchMouseEvent with 'Cannot access a chrome-extension:// URL of different extension'. The CDP target list shows the foreign chrome-extension:// URL as type:'other' attached to the same tab. New behaviour in cdp() wrapper: 1. On that specific error for any Input.* call, snapshot CDP targets to identify the offending extension id and log it (foreign-ext- detected event added to ring buffer). 2. Send Esc via Input.dispatchKeyEvent (key dispatch routes by focus, not coordinates, so it bypasses the overlay) to dismiss the popup. 3. Retry the original Input.* call once. 4. If retry still fails, throw a clear error naming the foreign extension id and listing three concrete workarounds: disable that extension on the page, use Tab-to-focus instead of clicking, or run /chrome quiet off.
Diffstat (limited to 'test-suite/README.md')
-rw-r--r--test-suite/README.md184
1 files changed, 124 insertions, 60 deletions
diff --git a/test-suite/README.md b/test-suite/README.md
index 241976c..bb8becb 100644
--- a/test-suite/README.md
+++ b/test-suite/README.md
@@ -1,8 +1,13 @@
-# pi-chrome anti-automation test suite
+# pi-chrome browser-control benchmark
-Static pages that each block a different automated-interaction signal. Drive them
-with the `chrome_*` tools to discover where the companion-extension's synthetic
-events still look like a bot, and to verify any humanization fixes.
+Static benchmark pages for evaluating tools that let agents control Chrome. The
+suite covers two related questions:
+
+1. **Can the agent complete normal browser work?** Forms, scroll containers,
+ contenteditable, files, frames, Shadow DOM, network/console inspection.
+2. **Does the interaction look like a human/browser action when sites care?**
+ `isTrusted`, user activation, pointer paths, key cadence, native controls,
+ drag/drop, touch, paste, and scroll momentum.
## Run
@@ -14,61 +19,120 @@ python3 -m http.server 8765
Each challenge page exposes:
-- `window.__challenge` — short id
-- `window.__verdict` — `"PENDING" | "PASS" | "FAIL"`
-- `window.__reason` — array of strings, why it failed (or how it passed)
+- `window.__challenge` — id
+- `window.__verdict` — `"PENDING" | "PASS" | "FAIL" | "SKIP" | "WARN"`
+- `window.__reason` — array of reasons
- `window.__events` — raw event log for forensics
-`PASS` means the page believes a real human did the thing.
-`FAIL` means the page caught the automation.
-
-## Aggregate runner
-
-`index.html` lists every challenge, loads each one in an iframe, lets you (or
-the agent) drive the interaction, and tallies verdicts.
-
-Recommended agent flow:
-
-1. `chrome_navigate` to `http://127.0.0.1:8765/`.
-2. For each challenge link, open it, run the listed interaction with the
- `chrome_*` tools, then read `window.__verdict` / `window.__reason` via
- `chrome_evaluate`.
-3. Compare against the manual baseline (do the same interaction by hand and
- confirm `PASS`).
-
-## Challenges
-
-| id | file | what it blocks |
-|----|------|----------------|
-| `is-trusted-click` | `challenges/01-is-trusted-click.html` | click handler ignores `event.isTrusted === false` |
-| `is-trusted-keyboard` | `challenges/02-is-trusted-keyboard.html` | input rejects if `keydown.isTrusted` false |
-| `webdriver-flag` | `challenges/03-webdriver-flag.html` | `navigator.webdriver` truthy / Chrome runtime quirks |
-| `mouse-entropy` | `challenges/04-mouse-entropy.html` | requires organic mousemove path before click |
-| `event-timing` | `challenges/05-event-timing.html` | rejects when pointerdown→pointerup gap < N ms or always-equal |
-| `click-coordinates` | `challenges/06-click-coordinates.html` | rejects clicks that land on exact element center |
-| `pointer-properties` | `challenges/07-pointer-properties.html` | demands pressure / non-zero movementX/Y / pointerId variety |
-| `keyboard-cadence` | `challenges/08-keyboard-cadence.html` | rejects same-tick `keydown`/`keyup` and missing per-char events |
-| `composition-input` | `challenges/09-composition-input.html` | listens for native `keydown` + `keypress` + `input` sequence |
-| `user-activation` | `challenges/10-user-activation.html` | feature gates: clipboard.writeText, fullscreen |
-| `honeypot` | `challenges/11-honeypot.html` | hidden field must stay empty |
-| `fingerprint` | `challenges/12-fingerprint.html` | UA-CH, languages, plugins, permissions consistency |
-| `focus-order` | `challenges/13-focus-order.html` | requires `pointerdown`-driven focus, not direct `.focus()` |
-| `wheel-scroll` | `challenges/14-wheel-scroll.html` | rejects scrollTop jumps without `wheel`/`scroll` events |
-| `drag-drop-datatransfer` | `challenges/15-drag-drop-datatransfer.html` | requires full HTML5 drag cycle with populated `DataTransfer` |
-| `contenteditable-selection` | `challenges/16-contenteditable-selection.html` | requires `selectionchange` + monotonic caret advance per keystroke |
-| `paste-clipboard` | `challenges/17-paste-clipboard.html` | demands real OS paste with `clipboardData` + `inputType=insertFromPaste` |
-| `native-select` | `challenges/18-native-select.html` | rejects `<option>` clicks / `.value=` assignment; needs trusted picker change |
-| `hover-dwell` | `challenges/19-hover-dwell.html` | reveal gated on ≥600ms hover with pointermove activity, latency after reveal |
-| `react-value-tracker` | `challenges/20-react-value-tracker.html` | input must mutate via native `value` setter so React's `_valueTracker` is stale |
-| `keyboard-modifiers` | `challenges/21-keyboard-modifiers.html` | Shift+a chord: modifier ordering, `shiftKey`, `code=KeyA`, `getModifierState` |
-| `touch-events` | `challenges/22-touch-events.html` | real `TouchEvent`+`Touch` (radius, force) — synthetic pointerType=touch fails |
-| `stack-trace-fingerprint` | `challenges/23-stack-trace-fingerprint.html` | inspects handler call stack for extension/eval/Runtime.evaluate frames |
-| `viewport-edge-clicks` | `challenges/24-viewport-edge-clicks.html` | rejects pointer events with clientX/Y outside viewport or (0,0) default |
-| `pointer-continuity` | `challenges/25-pointer-continuity.html` | two clicks far apart need bridging pointermoves with mid-span samples |
-| `mousemove-rate` | `challenges/26-mousemove-rate.html` | mousemove Δt distribution must match ~60–250Hz with jitter |
-| `scroll-momentum` | `challenges/27-scroll-momentum.html` | wheel ΔY trace must show peak + decaying momentum tail ≥100ms |
-| `intersection-visibility` | `challenges/28-intersection-visibility.html` | IntersectionObserver gradient + rAF scroll frames — no teleport |
-
-## What each `PASS` would require in `service_worker.js`
-
-See `notes/bypass-ideas.md`.
+`manifest.json` is the source of truth for benchmark metadata: category, goal,
+expected result per mode, prerequisites, flake risk, manual baseline status, and
+canonical tool recipe. `manifest.schema.json` documents the manifest shape.
+Recipes express tool intent; runners may need to adapt descriptive selectors
+(e.g. shadow/iframe notation) and expand path placeholders like `$PWD`.
+
+## Modes / expected outcomes
+
+The same page can have different expected results depending on tool capability:
+
+- `synthetic` — DOM-dispatched events / framework-aware setters. Fast and quiet.
+- `trusted` — browser-trusted input, usually via `chrome.debugger`/CDP. Can show
+ Chrome's debugging banner.
+- `manual` — human baseline in same browser/profile.
+
+Expected values in `manifest.json`:
+
+- `PASS` / `FAIL` — deterministic target for that mode.
+- `CONDITIONAL` — depends on browser policy, OS, device capability, permissions,
+ or an unreleased tool primitive. Inspect `prerequisites`, `notes`, and
+ `flakeRisk`.
+
+Manual baselines are tracked separately with `manualBaseline`. `unverified`
+means the manual expectation is a target, not a recorded contract.
+
+## Recommended agent flow
+
+1. Navigate to dashboard:
+ `http://127.0.0.1:8765/`.
+2. Pick mode (`synthetic`, `trusted`, or `manual`) and clear local verdicts.
+3. For each manifest row:
+ - `chrome_navigate` to `http://127.0.0.1:8765/<file>`.
+ - `chrome_snapshot` before acting; prefer snapshot `uid` over raw selector.
+ - Execute the listed `recipe`, adapting descriptive frame/shadow selectors to
+ whatever selectors/uids the tool exposes.
+ - Read:
+ ```js
+ JSON.stringify({
+ v: window.__verdict,
+ r: window.__reason,
+ e: window.__events?.slice(-20)
+ })
+ ```
+4. Return to dashboard and compare actual verdicts with expected values.
+5. Copy JSON report from dashboard for PRs or regression notes.
+
+## Challenge categories
+
+- `trusted-input` — browser-trusted click/key events.
+- `pointer-humanization` — paths, coordinates, movement continuity/rate.
+- `keyboard` / `focus-keyboard` — typing fidelity, modifiers, Tab flows.
+- `activation-gates` — clipboard/fullscreen/user activation.
+- `scroll` / `scroll-visibility` — wheel events, momentum, IntersectionObserver.
+- `drag-drop` — HTML5 drag/drop + `DataTransfer`.
+- `clipboard` — OS/browser paste path.
+- `native-controls` — controls that should use browser UI/keyboard semantics.
+- `frameworks` / `editing` — React-style value tracking, contenteditable.
+- `dom-complexity` / `frames` — Shadow DOM and iframe targeting.
+- `files` — file attachment to `<input type=file>`.
+- `observability` — console/network capture tools.
+- `fingerprint` — environment and stack fingerprint probes.
+- `agent-safety` — hidden honeypots and safe target selection.
+
+## Current challenge inventory
+
+The dashboard renders this from `manifest.json`. In brief:
+
+1. trusted click
+2. trusted keyboard
+3. webdriver/runtime flags
+4. mouse entropy before click
+5. click timing
+6. click coordinate variation
+7. pointer event properties
+8. keyboard cadence
+9. beforeinput/input order
+10. user activation gates
+11. honeypot safety
+12. fingerprint consistency
+13. focus order
+14. wheel scroll
+15. drag/drop `DataTransfer`
+16. contenteditable selection
+17. paste clipboard
+18. native select
+19. hover dwell
+20. React value tracker
+21. keyboard modifiers
+22. touch events
+23. stack trace fingerprint
+24. viewport click coordinates
+25. pointer continuity
+26. mousemove rate
+27. scroll momentum
+28. intersection visibility
+29. Shadow DOM controls
+30. iframe targeting
+31. file upload
+32. keyboard Tab navigation
+33. network/console capture
+
+## Design notes
+
+- A failure is useful only when compared to expected mode. Example: synthetic
+ `isTrusted` failing is expected and validates that the test detects quiet DOM
+ events.
+- Some tests are capability-gated. Example: touch tests should be `SKIP`/manual
+ conditional on non-touch hardware.
+- Fingerprint tests should warn before blocking. Real Chrome profiles can use
+ software WebGL in VMs, remote desktops, or policy-constrained environments.
+- `notes/bypass-ideas.md` is historical guidance for older synthetic-only
+ versions. Prefer `manifest.json` for current expected outcomes.