1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
|
# pi-chrome browser-control benchmark
Static benchmark pages for evaluating tools that let agents control Chrome. The suite has two layers:
1. **Unit challenges** (`manifest.json`) — MiniWoB-style capability probes for
forms, scroll containers, contenteditable, files, frames, Shadow DOM,
network/console inspection, `isTrusted`, user activation, pointer paths, key
cadence, native controls, drag/drop, touch, paste, and scroll momentum.
2. **Long-horizon hermetic tasks** (`task-manifest.json`) — WebArena /
BrowserGym-inspired multi-step tasks with fresh run IDs and deterministic
programmatic graders.
## Run
```bash
cd test-suite
python3 -m http.server 8765
# open http://127.0.0.1:8765/ in the Chrome window pi-chrome controls
```
Each challenge page exposes:
- `window.__challenge` — id
- `window.__verdict` — `"PENDING" | "PASS" | "FAIL" | "SKIP" | "WARN"`
- `window.__reason` — array of reasons
- `window.__events` — raw event log for forensics
`manifest.json` is the source of truth for unit-challenge metadata: category,
gate bucket, goal, expected result per mode, prerequisites, flake risk, manual
baseline status, and canonical tool recipe. `manifest.schema.json` documents the
manifest shape. Recipes express tool intent; runners may need to adapt
descriptive selectors (e.g. shadow/iframe notation), dynamic tab ids, and expand
path placeholders like `$PWD`.
`task-manifest.json` is the source of truth for long-horizon tasks: BrowserGym-style
`taskId`, seed, viewport, goal object, difficulty tier, max steps, declared
action subsets, reset/setup URL, validate hook, optional cheat recipe, and
programmatic grader expression. `task-manifest.schema.json` documents this shape.
`browsergym-action-space.json` records BrowserGym-compatible action subsets.
## Modes / expected outcomes
The same page can have different expected results depending on tool capability:
- `synthetic` — DOM-dispatched events / framework-aware setters. Fast and quiet.
- `trusted` — browser-trusted input, usually via `chrome.debugger`/CDP. Can show
Chrome's debugging banner.
- `manual` — human baseline in same browser/profile.
Expected values in `manifest.json`:
- `PASS` / `FAIL` — deterministic target for that mode.
- `CONDITIONAL` — depends on browser policy, OS, device capability, permissions,
or an unreleased tool primitive. Inspect `prerequisites`, `notes`, and
`flakeRisk`.
Manual baselines are tracked separately with `manualBaseline`. `unverified`
means the manual expectation is a target, not a recorded contract.
## Gate buckets
Each unit challenge has a `gate` field:
- `core` — required release blocker for normal trusted-mode pi-chrome shipping.
- `conditional` — blocks only when declared prerequisites/capabilities are present
(clipboard, touch, dialogs, native UI, etc.).
- `quality` — adversarial humanization/fingerprint signal. Track regressions, but
do not block general ship without an explicit product decision.
## Recommended unit-challenge agent flow
1. Navigate to dashboard:
`http://127.0.0.1:8765/`.
2. Pick mode (`synthetic`, `trusted`, or `manual`) and clear local verdicts.
3. For each manifest row:
- `chrome_navigate` to `http://127.0.0.1:8765/<file>`.
- `chrome_snapshot` before acting; prefer snapshot `uid` over raw selector.
- Execute the listed `recipe`, adapting descriptive frame/shadow selectors to
whatever selectors/uids the tool exposes.
- Read:
```js
JSON.stringify({
v: window.__verdict,
r: window.__reason,
e: window.__events?.slice(-20)
})
```
4. Return to dashboard and compare actual verdicts with expected values.
5. Copy JSON report from dashboard for PRs or regression notes.
## Recommended long-horizon task flow
1. Load `task-manifest.json`.
2. Replace `$RUN_ID` in `startUrl` with a fresh value.
3. Navigate to the start URL and read the visible task instruction.
4. Solve using normal browser tools only; avoid direct state mutation unless the
benchmark mode explicitly allows evaluate-based actions.
5. Click **Grade now** or evaluate the task grader expression:
```js
JSON.stringify({ v: window.__taskVerdict, r: window.__taskReason })
```
6. Record action count, observations used, tools used, verdict, and reason.
## Design principles copied from browser-agent benchmarks
- Prefer hermetic sites and deterministic graders over live sites and LLM judges.
- Report action API and observation format; these strongly affect scores.
- Use difficulty tiers: L1 atomic, L2 compositional, L3 cross-page/context-rich.
- Include tedious cross-page memory and exact-value transfer tasks; short unit
probes hide these failures.
- Keep synthetic-event-gated tests because extension bridges face failures that
CDP/Playwright-style benchmarks usually do not measure.
## Challenge categories
- `trusted-input` — browser-trusted click/key events.
- `pointer-humanization` — paths, coordinates, movement continuity/rate.
- `keyboard` / `focus-keyboard` — typing fidelity, modifiers, Tab flows.
- `activation-gates` — clipboard/fullscreen/user activation.
- `scroll` / `scroll-visibility` — wheel events, momentum, IntersectionObserver.
- `drag-drop` — HTML5 drag/drop + `DataTransfer`.
- `clipboard` — OS/browser paste path.
- `native-controls` — controls that should use browser UI/keyboard semantics.
- `frameworks` / `editing` — React-style value tracking, contenteditable.
- `dom-complexity` / `frames` — Shadow DOM and iframe targeting.
- `files` — file attachment to `<input type=file>`.
- `observability` — console/network capture tools.
- `csp` — strict Content Security Policy: screenshot/coordinate fallback (39) and the CDP eval/snapshot bypass that works under `script-src 'self'` without `unsafe-eval` (42).
- `lazy-loading` — dynamic DOM readiness and wait behavior.
- `fingerprint` — environment and stack fingerprint probes.
- `agent-safety` — hidden honeypots and safe target selection.
## Current challenge inventory
The dashboard renders this from `manifest.json`. In brief:
1. trusted click
2. trusted keyboard
3. webdriver/runtime flags
4. mouse entropy before click
5. click timing
6. click coordinate variation
7. pointer event properties
8. keyboard cadence
9. beforeinput/input order
10. user activation gates
11. honeypot safety
12. fingerprint consistency
13. focus order
14. wheel scroll
15. drag/drop `DataTransfer`
16. contenteditable selection
17. paste clipboard
18. native select
19. hover dwell
20. React value tracker
21. keyboard modifiers
22. touch events
23. stack trace fingerprint
24. viewport click coordinates
25. pointer continuity
26. mousemove rate
27. scroll momentum
28. intersection visibility
29. Shadow DOM controls
30. iframe targeting
31. file upload
32. keyboard Tab navigation
33. network/console capture
34. dialog handling
35. target blank popup
36. modal focus trap
37. autocomplete combobox
38. SPA route change
39. strict CSP screenshot/coordinate fallback
40. dynamic wait/readiness
41. explicit tab lifecycle
42. strict CSP eval/snapshot via CDP (regression guard for the CSP bypass)
43. hard background: inactive-tab visibility and trusted input
### Hard-background regression (43)
Keep `/chrome background on`. Open challenge 43, then manually select a different tab in the **same window** and return to Pi. Target the challenge URL explicitly, arm its probe, and run its manifest recipe without selecting the target again. Inspect the saved screenshot for the colored target. The page detects visibility changes and synthetic input; it cannot establish OS focus or macOS Spaces behavior.
Separately live-test with Chrome behind another app, a minimized window, multiple windows, and macOS Spaces. Confirm the foreground app/window and each active tab stay unchanged during background new-tab creation, viewport/full-page screenshots, and trusted input. Confirm `/chrome background off` permits explicit activation. Native prompts/page popups are outside this focus policy, not a zero-focus guarantee.
`npm test` runs mocked regressions for every registered tool's policy, old-companion rejection, worker focus writes, CDP capture success/failure, trusted input, and full-page scroll restoration. No live Chrome is required; Node.js 22.13+ is required for TypeScript stripping.
## Design notes
- A failure is useful only when compared to expected mode. Example: synthetic
`isTrusted` failing is expected and validates that the test detects quiet DOM
events.
- Some tests are capability-gated. Example: touch tests should be `SKIP`/manual
conditional on non-touch hardware.
- Fingerprint tests should warn before blocking. Real Chrome profiles can use
software WebGL in VMs, remote desktops, or policy-constrained environments.
- `notes/bypass-ideas.md` is historical guidance for older synthetic-only
versions. Prefer `manifest.json` for current expected outcomes.
- `notes/browsergym-compat.md` defines the reset/step/validate/observation/BID
contract for external BrowserGym-style agents.
- `notes/runner-spec.md`, `notes/scoring.md`, and `notes/profiles.md` define
runner output, scoring, retry policy, and environment metadata.
|
