diff options
Diffstat (limited to 'test-suite/README.md')
| -rw-r--r-- | test-suite/README.md | 184 |
1 files changed, 124 insertions, 60 deletions
diff --git a/test-suite/README.md b/test-suite/README.md index 241976c..bb8becb 100644 --- a/test-suite/README.md +++ b/test-suite/README.md @@ -1,8 +1,13 @@ -# pi-chrome anti-automation test suite +# pi-chrome browser-control benchmark -Static pages that each block a different automated-interaction signal. Drive them -with the `chrome_*` tools to discover where the companion-extension's synthetic -events still look like a bot, and to verify any humanization fixes. +Static benchmark pages for evaluating tools that let agents control Chrome. The +suite covers two related questions: + +1. **Can the agent complete normal browser work?** Forms, scroll containers, + contenteditable, files, frames, Shadow DOM, network/console inspection. +2. **Does the interaction look like a human/browser action when sites care?** + `isTrusted`, user activation, pointer paths, key cadence, native controls, + drag/drop, touch, paste, and scroll momentum. ## Run @@ -14,61 +19,120 @@ python3 -m http.server 8765 Each challenge page exposes: -- `window.__challenge` — short id -- `window.__verdict` — `"PENDING" | "PASS" | "FAIL"` -- `window.__reason` — array of strings, why it failed (or how it passed) +- `window.__challenge` — id +- `window.__verdict` — `"PENDING" | "PASS" | "FAIL" | "SKIP" | "WARN"` +- `window.__reason` — array of reasons - `window.__events` — raw event log for forensics -`PASS` means the page believes a real human did the thing. -`FAIL` means the page caught the automation. - -## Aggregate runner - -`index.html` lists every challenge, loads each one in an iframe, lets you (or -the agent) drive the interaction, and tallies verdicts. - -Recommended agent flow: - -1. `chrome_navigate` to `http://127.0.0.1:8765/`. -2. For each challenge link, open it, run the listed interaction with the - `chrome_*` tools, then read `window.__verdict` / `window.__reason` via - `chrome_evaluate`. -3. Compare against the manual baseline (do the same interaction by hand and - confirm `PASS`). - -## Challenges - -| id | file | what it blocks | -|----|------|----------------| -| `is-trusted-click` | `challenges/01-is-trusted-click.html` | click handler ignores `event.isTrusted === false` | -| `is-trusted-keyboard` | `challenges/02-is-trusted-keyboard.html` | input rejects if `keydown.isTrusted` false | -| `webdriver-flag` | `challenges/03-webdriver-flag.html` | `navigator.webdriver` truthy / Chrome runtime quirks | -| `mouse-entropy` | `challenges/04-mouse-entropy.html` | requires organic mousemove path before click | -| `event-timing` | `challenges/05-event-timing.html` | rejects when pointerdown→pointerup gap < N ms or always-equal | -| `click-coordinates` | `challenges/06-click-coordinates.html` | rejects clicks that land on exact element center | -| `pointer-properties` | `challenges/07-pointer-properties.html` | demands pressure / non-zero movementX/Y / pointerId variety | -| `keyboard-cadence` | `challenges/08-keyboard-cadence.html` | rejects same-tick `keydown`/`keyup` and missing per-char events | -| `composition-input` | `challenges/09-composition-input.html` | listens for native `keydown` + `keypress` + `input` sequence | -| `user-activation` | `challenges/10-user-activation.html` | feature gates: clipboard.writeText, fullscreen | -| `honeypot` | `challenges/11-honeypot.html` | hidden field must stay empty | -| `fingerprint` | `challenges/12-fingerprint.html` | UA-CH, languages, plugins, permissions consistency | -| `focus-order` | `challenges/13-focus-order.html` | requires `pointerdown`-driven focus, not direct `.focus()` | -| `wheel-scroll` | `challenges/14-wheel-scroll.html` | rejects scrollTop jumps without `wheel`/`scroll` events | -| `drag-drop-datatransfer` | `challenges/15-drag-drop-datatransfer.html` | requires full HTML5 drag cycle with populated `DataTransfer` | -| `contenteditable-selection` | `challenges/16-contenteditable-selection.html` | requires `selectionchange` + monotonic caret advance per keystroke | -| `paste-clipboard` | `challenges/17-paste-clipboard.html` | demands real OS paste with `clipboardData` + `inputType=insertFromPaste` | -| `native-select` | `challenges/18-native-select.html` | rejects `<option>` clicks / `.value=` assignment; needs trusted picker change | -| `hover-dwell` | `challenges/19-hover-dwell.html` | reveal gated on ≥600ms hover with pointermove activity, latency after reveal | -| `react-value-tracker` | `challenges/20-react-value-tracker.html` | input must mutate via native `value` setter so React's `_valueTracker` is stale | -| `keyboard-modifiers` | `challenges/21-keyboard-modifiers.html` | Shift+a chord: modifier ordering, `shiftKey`, `code=KeyA`, `getModifierState` | -| `touch-events` | `challenges/22-touch-events.html` | real `TouchEvent`+`Touch` (radius, force) — synthetic pointerType=touch fails | -| `stack-trace-fingerprint` | `challenges/23-stack-trace-fingerprint.html` | inspects handler call stack for extension/eval/Runtime.evaluate frames | -| `viewport-edge-clicks` | `challenges/24-viewport-edge-clicks.html` | rejects pointer events with clientX/Y outside viewport or (0,0) default | -| `pointer-continuity` | `challenges/25-pointer-continuity.html` | two clicks far apart need bridging pointermoves with mid-span samples | -| `mousemove-rate` | `challenges/26-mousemove-rate.html` | mousemove Δt distribution must match ~60–250Hz with jitter | -| `scroll-momentum` | `challenges/27-scroll-momentum.html` | wheel ΔY trace must show peak + decaying momentum tail ≥100ms | -| `intersection-visibility` | `challenges/28-intersection-visibility.html` | IntersectionObserver gradient + rAF scroll frames — no teleport | - -## What each `PASS` would require in `service_worker.js` - -See `notes/bypass-ideas.md`. +`manifest.json` is the source of truth for benchmark metadata: category, goal, +expected result per mode, prerequisites, flake risk, manual baseline status, and +canonical tool recipe. `manifest.schema.json` documents the manifest shape. +Recipes express tool intent; runners may need to adapt descriptive selectors +(e.g. shadow/iframe notation) and expand path placeholders like `$PWD`. + +## Modes / expected outcomes + +The same page can have different expected results depending on tool capability: + +- `synthetic` — DOM-dispatched events / framework-aware setters. Fast and quiet. +- `trusted` — browser-trusted input, usually via `chrome.debugger`/CDP. Can show + Chrome's debugging banner. +- `manual` — human baseline in same browser/profile. + +Expected values in `manifest.json`: + +- `PASS` / `FAIL` — deterministic target for that mode. +- `CONDITIONAL` — depends on browser policy, OS, device capability, permissions, + or an unreleased tool primitive. Inspect `prerequisites`, `notes`, and + `flakeRisk`. + +Manual baselines are tracked separately with `manualBaseline`. `unverified` +means the manual expectation is a target, not a recorded contract. + +## Recommended agent flow + +1. Navigate to dashboard: + `http://127.0.0.1:8765/`. +2. Pick mode (`synthetic`, `trusted`, or `manual`) and clear local verdicts. +3. For each manifest row: + - `chrome_navigate` to `http://127.0.0.1:8765/<file>`. + - `chrome_snapshot` before acting; prefer snapshot `uid` over raw selector. + - Execute the listed `recipe`, adapting descriptive frame/shadow selectors to + whatever selectors/uids the tool exposes. + - Read: + ```js + JSON.stringify({ + v: window.__verdict, + r: window.__reason, + e: window.__events?.slice(-20) + }) + ``` +4. Return to dashboard and compare actual verdicts with expected values. +5. Copy JSON report from dashboard for PRs or regression notes. + +## Challenge categories + +- `trusted-input` — browser-trusted click/key events. +- `pointer-humanization` — paths, coordinates, movement continuity/rate. +- `keyboard` / `focus-keyboard` — typing fidelity, modifiers, Tab flows. +- `activation-gates` — clipboard/fullscreen/user activation. +- `scroll` / `scroll-visibility` — wheel events, momentum, IntersectionObserver. +- `drag-drop` — HTML5 drag/drop + `DataTransfer`. +- `clipboard` — OS/browser paste path. +- `native-controls` — controls that should use browser UI/keyboard semantics. +- `frameworks` / `editing` — React-style value tracking, contenteditable. +- `dom-complexity` / `frames` — Shadow DOM and iframe targeting. +- `files` — file attachment to `<input type=file>`. +- `observability` — console/network capture tools. +- `fingerprint` — environment and stack fingerprint probes. +- `agent-safety` — hidden honeypots and safe target selection. + +## Current challenge inventory + +The dashboard renders this from `manifest.json`. In brief: + +1. trusted click +2. trusted keyboard +3. webdriver/runtime flags +4. mouse entropy before click +5. click timing +6. click coordinate variation +7. pointer event properties +8. keyboard cadence +9. beforeinput/input order +10. user activation gates +11. honeypot safety +12. fingerprint consistency +13. focus order +14. wheel scroll +15. drag/drop `DataTransfer` +16. contenteditable selection +17. paste clipboard +18. native select +19. hover dwell +20. React value tracker +21. keyboard modifiers +22. touch events +23. stack trace fingerprint +24. viewport click coordinates +25. pointer continuity +26. mousemove rate +27. scroll momentum +28. intersection visibility +29. Shadow DOM controls +30. iframe targeting +31. file upload +32. keyboard Tab navigation +33. network/console capture + +## Design notes + +- A failure is useful only when compared to expected mode. Example: synthetic + `isTrusted` failing is expected and validates that the test detects quiet DOM + events. +- Some tests are capability-gated. Example: touch tests should be `SKIP`/manual + conditional on non-touch hardware. +- Fingerprint tests should warn before blocking. Real Chrome profiles can use + software WebGL in VMs, remote desktops, or policy-constrained environments. +- `notes/bypass-ideas.md` is historical guidance for older synthetic-only + versions. Prefer `manifest.json` for current expected outcomes. |
