blob: 970a7060d3586a1942cb97edbc2ceb5d694220a6 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
# Benchmark scoring
## Unit challenges (`manifest.json`)
Score actual verdict against `expected[mode]`, not raw PASS count.
- `PASS` / `FAIL` expected: exact match required.
- `CONDITIONAL`: exclude from headline score unless environment prerequisite is declared present.
- `SKIP` / `WARN`: report separately.
Recommended flake policy:
- Run each non-destructive challenge up to 2 retries.
- Take best verdict only for known-flaky timing/scroll tests.
- Always keep all reasons/events in detailed report.
Difficulty weighting:
- L1 = 1
- L2 = 2
- L3 = 3
Report both unweighted and weighted score. Also report per-category score.
## Long-horizon tasks (`task-manifest.json`)
Task score is deterministic grader PASS / not PASS. Record:
- action count
- wall time
- tools used
- observation mode (`snapshot`, screenshot, accessibility, evaluate)
- whether direct state mutation/evaluate was allowed
- final grader reason
Do not use LLM judge for hermetic tasks unless grader cannot express the target.
|