pi-chrome: snapshot pre-migration state

1 files changed, 36 insertions, 0 deletions

diff --git a/test-suite/notes/scoring.md b/test-suite/notes/scoring.md
new file mode 100644
index 0000000..970a706
--- /dev/null
+++ b/test-suite/notes/scoring.md
@@ -0,0 +1,36 @@
+# Benchmark scoring
+
+## Unit challenges (`manifest.json`)
+
+Score actual verdict against `expected[mode]`, not raw PASS count.
+
+- `PASS` / `FAIL` expected: exact match required.
+- `CONDITIONAL`: exclude from headline score unless environment prerequisite is declared present.
+- `SKIP` / `WARN`: report separately.
+
+Recommended flake policy:
+
+- Run each non-destructive challenge up to 2 retries.
+- Take best verdict only for known-flaky timing/scroll tests.
+- Always keep all reasons/events in detailed report.
+
+Difficulty weighting:
+
+- L1 = 1
+- L2 = 2
+- L3 = 3
+
+Report both unweighted and weighted score. Also report per-category score.
+
+## Long-horizon tasks (`task-manifest.json`)
+
+Task score is deterministic grader PASS / not PASS. Record:
+
+- action count
+- wall time
+- tools used
+- observation mode (`snapshot`, screenshot, accessibility, evaluate)
+- whether direct state mutation/evaluate was allowed
+- final grader reason
+
+Do not use LLM judge for hermetic tasks unless grader cannot express the target.