blob: 5edb7f52192235481b9a8e18bfc1096cbda52e5a (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
# Benchmark scoring
## Unit challenges (`manifest.json`)
Score actual verdict against `expected[mode]`, not raw PASS count.
Gate buckets:
- `core`: headline release gate. Exact expected verdict required.
- `conditional`: gate only when declared capability/prerequisite is present; otherwise report skipped/conditional.
- `quality`: adversarial humanization/fingerprint signal. Report trend and regressions, but do not block general release unless explicitly promoted.
Expected values:
- `PASS` / `FAIL` expected: exact match required inside active gate bucket.
- `CONDITIONAL`: exclude from core headline score unless environment prerequisite is declared present.
- `SKIP` / `WARN`: report separately.
Recommended flake policy:
- Run each non-destructive challenge up to 2 retries.
- Take best verdict only for known-flaky timing/scroll tests.
- Always keep all reasons/events in detailed report.
Difficulty weighting:
- L1 = 1
- L2 = 2
- L3 = 3
Report both unweighted and weighted score. Also report per-category score.
## Long-horizon tasks (`task-manifest.json`)
Task score is deterministic grader PASS / not PASS. Record:
- action count
- wall time
- tools used
- observation mode (`snapshot`, screenshot, accessibility, evaluate)
- whether direct state mutation/evaluate was allowed
- final grader reason
Do not use LLM judge for hermetic tasks unless grader cannot express the target.
|
