aboutsummaryrefslogtreecommitdiff
path: root/test-suite/notes/scoring.md
blob: 970a7060d3586a1942cb97edbc2ceb5d694220a6 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Benchmark scoring

## Unit challenges (`manifest.json`)

Score actual verdict against `expected[mode]`, not raw PASS count.

- `PASS` / `FAIL` expected: exact match required.
- `CONDITIONAL`: exclude from headline score unless environment prerequisite is declared present.
- `SKIP` / `WARN`: report separately.

Recommended flake policy:

- Run each non-destructive challenge up to 2 retries.
- Take best verdict only for known-flaky timing/scroll tests.
- Always keep all reasons/events in detailed report.

Difficulty weighting:

- L1 = 1
- L2 = 2
- L3 = 3

Report both unweighted and weighted score. Also report per-category score.

## Long-horizon tasks (`task-manifest.json`)

Task score is deterministic grader PASS / not PASS. Record:

- action count
- wall time
- tools used
- observation mode (`snapshot`, screenshot, accessibility, evaluate)
- whether direct state mutation/evaluate was allowed
- final grader reason

Do not use LLM judge for hermetic tasks unless grader cannot express the target.