pi-chrome: snapshot pre-migration state

4 files changed, 154 insertions, 0 deletions

diff --git a/test-suite/notes/browsergym-compat.md b/test-suite/notes/browsergym-compat.md
new file mode 100644
index 0000000..1eaea41
--- /dev/null
+++ b/test-suite/notes/browsergym-compat.md
@@ -0,0 +1,70 @@
+# BrowserGym compatibility contract
+
+Pi-chrome benchmark tasks mirror BrowserGym without importing Gymnasium.
+
+## Task fields
+
+Each `task-manifest.json` entry maps to `AbstractBrowserTask` concepts:
+
+- `taskId` / `id` — stable namespaced id
+- `seed` — deterministic state seed
+- `viewport`, `slowMoMs`, `timeoutMs`, `locale`, `timezoneId`
+- `goal` and `goalObject` — text and OpenAI-style message objects
+- `setupUrl` — page URL that performs setup from `task`, `run`, `seed`
+- `validateScript` — JS validate hook returning reward contract
+- `cheatScript` — gold recipe used to sanity-check grader
+- `maxSteps`, `humanBaseline`, `difficulty`, `tags`
+- `actionSubsets`, `allowedActions` — BrowserGym HighLevelActionSet metadata
+
+## Reset / step return
+
+Runner should expose:
+
+```text
+reset() -> (obs, info)
+step(action) -> (obs, reward, terminated, truncated, info)
+```
+
+`terminated` means task done. `truncated` means max step or timeout.
+
+## Validate return
+
+In-page validators return:
+
+```js
+{ reward, done, message, info }
+```
+
+`reward` is incremental. Current hermetic tasks use binary terminal reward. `info.rubric` carries per-check details.
+
+## Observation schema target
+
+Runner observations should include BrowserGym-like keys:
+
+- `chat_messages`
+- `goal`, `goal_object`
+- `open_pages_urls`, `open_pages_titles`, `active_page_index`
+- `url`, `screenshot`, `dom_object`, `axtree_object`
+- `extra_element_properties`
+- `focused_element_bid`
+- `last_action`, `last_action_error`
+- `elapsed_time`
+
+## BID plan
+
+BrowserGym uses `bid` attributes as stable element handles. Pi-chrome currently returns snapshot `uid`. Compatibility path:
+
+1. During snapshot, assign every interactive element a deterministic `bid`.
+2. Return both `uid` and `bid`.
+3. Add `visibility`, `bbox`, `clickable`, `set_of_marks` metadata.
+4. Maintain uid↔bid mapping so agents can use either addressing mode.
+
+Target attributes:
+
+- `bid`
+- `browsergym_visibility_ratio`
+- `browsergym_set_of_marks`
+
+## Action subsets
+
+See `../browsergym-action-space.json` for BrowserGym-compatible subsets and function signatures.
diff --git a/test-suite/notes/profiles.md b/test-suite/notes/profiles.md
new file mode 100644
index 0000000..8f55797
--- /dev/null
+++ b/test-suite/notes/profiles.md
@@ -0,0 +1,22 @@
+# Calibration profiles
+
+Benchmark results depend on Chrome profile, OS, input device, and extension mode.
+Record these with every run:
+
+- OS and version
+- Chrome version
+- pi-chrome package version
+- companion extension version
+- trusted mode: off / auto / on
+- input hardware: trackpad vs detent mouse, touch support
+- permissions: clipboard, fullscreen, downloads
+- viewport size and deviceScaleFactor
+- loaded privacy/security extensions
+
+Known environment-sensitive areas:
+
+- WebGL fingerprint can warn/fail in VMs, remote desktops, GPU-disabled Chrome.
+- Scroll momentum expects trackpad-like decay; detent mouse wheels may fail manually.
+- Touch tests require touch events enabled/supported.
+- Dialog/download/file-picker tests require native browser UI handling or manual intervention.
+- Stack traces vary across Chrome versions; tests should only fail concrete automation URLs/Runtime frames.
diff --git a/test-suite/notes/runner-spec.md b/test-suite/notes/runner-spec.md
new file mode 100644
index 0000000..b040cb4
--- /dev/null
+++ b/test-suite/notes/runner-spec.md
@@ -0,0 +1,26 @@
+# Recipe runner spec
+
+Future Node/Pi runner should:
+
+1. Read `manifest.json`.
+2. Probe environment capabilities (`trusted`, `clipboard`, `touch`, `dialogs`, `downloads`, `fileSystem`).
+3. For each entry:
+   - skip if `requires` is unsatisfied and expected is `CONDITIONAL`
+   - navigate to challenge URL
+   - execute `recipe` in order
+   - wait `verdictDelayMs || 500`
+   - evaluate `JSON.stringify({v:window.__verdict,r:window.__reason,d:window.Challenge?.state?.details,e:window.__events?.slice(-20)})`
+   - compare to `expected[mode]`
+4. Output:
+   - Markdown summary
+   - JSON details
+   - JUnit XML for CI
+
+Runner must adapt recipe intent:
+
+- expand `${REPO_ROOT}` path placeholders
+- adapt unsupported shadow/iframe selector notation to snapshot uids or evaluate fallback
+- preserve hook install ordering for console/network capture tests
+- record whether trusted/CDP path was used
+
+Long-horizon task runner should read `task-manifest.json`, replace `$RUN_ID`, solve task, then evaluate task grader expression.
diff --git a/test-suite/notes/scoring.md b/test-suite/notes/scoring.md
new file mode 100644
index 0000000..970a706
--- /dev/null
+++ b/test-suite/notes/scoring.md
@@ -0,0 +1,36 @@
+# Benchmark scoring
+
+## Unit challenges (`manifest.json`)
+
+Score actual verdict against `expected[mode]`, not raw PASS count.
+
+- `PASS` / `FAIL` expected: exact match required.
+- `CONDITIONAL`: exclude from headline score unless environment prerequisite is declared present.
+- `SKIP` / `WARN`: report separately.
+
+Recommended flake policy:
+
+- Run each non-destructive challenge up to 2 retries.
+- Take best verdict only for known-flaky timing/scroll tests.
+- Always keep all reasons/events in detailed report.
+
+Difficulty weighting:
+
+- L1 = 1
+- L2 = 2
+- L3 = 3
+
+Report both unweighted and weighted score. Also report per-category score.
+
+## Long-horizon tasks (`task-manifest.json`)
+
+Task score is deterministic grader PASS / not PASS. Record:
+
+- action count
+- wall time
+- tools used
+- observation mode (`snapshot`, screenshot, accessibility, evaluate)
+- whether direct state mutation/evaluate was allowed
+- final grader reason
+
+Do not use LLM judge for hermetic tasks unless grader cannot express the target.