diff options
| author | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-16 14:01:00 -0400 |
|---|---|---|
| committer | Tianren Dong <tianrendong.dtr@gmail.com> | 2026-05-16 14:01:00 -0400 |
| commit | b792f22f2c2ef4dfb7bd300e7381a186e34e0c18 (patch) | |
| tree | a31a4e6d1213478f5ac96e2208d1a1c16fc825b9 /test-suite/notes | |
| parent | f133011c8ecda06ae63068649a9714c88a7b901e (diff) | |
0.15.25: fix reload after stale singleton
Diffstat (limited to 'test-suite/notes')
| -rw-r--r-- | test-suite/notes/runner-spec.md | 7 | ||||
| -rw-r--r-- | test-suite/notes/scoring.md | 12 |
2 files changed, 15 insertions, 4 deletions
diff --git a/test-suite/notes/runner-spec.md b/test-suite/notes/runner-spec.md index b040cb4..24c97ab 100644 --- a/test-suite/notes/runner-spec.md +++ b/test-suite/notes/runner-spec.md @@ -3,13 +3,14 @@ Future Node/Pi runner should: 1. Read `manifest.json`. -2. Probe environment capabilities (`trusted`, `clipboard`, `touch`, `dialogs`, `downloads`, `fileSystem`). +2. Probe environment capabilities (`trusted`, `clipboard`, `touch`, `dialogs`, `downloads`, `fileSystem`, `strictCspFallback`). 3. For each entry: + - read `gate` (`core`, `conditional`, `quality`) and score in that bucket - skip if `requires` is unsatisfied and expected is `CONDITIONAL` - navigate to challenge URL - execute `recipe` in order - wait `verdictDelayMs || 500` - - evaluate `JSON.stringify({v:window.__verdict,r:window.__reason,d:window.Challenge?.state?.details,e:window.__events?.slice(-20)})` + - evaluate `JSON.stringify({v:window.__verdict,r:window.__reason,d:window.Challenge?.state?.details,e:window.__events?.slice(-20)})`, except strict-CSP pages where eval is expected to fail; read persisted verdict from dashboard/localStorage after leaving the CSP page - compare to `expected[mode]` 4. Output: - Markdown summary @@ -21,6 +22,8 @@ Runner must adapt recipe intent: - expand `${REPO_ROOT}` path placeholders - adapt unsupported shadow/iframe selector notation to snapshot uids or evaluate fallback - preserve hook install ordering for console/network capture tests +- substitute dynamic tab ids from `chrome_tab list` recipes +- support screenshot/coordinate fallback for strict CSP pages where snapshot/evaluate are blocked - record whether trusted/CDP path was used Long-horizon task runner should read `task-manifest.json`, replace `$RUN_ID`, solve task, then evaluate task grader expression. diff --git a/test-suite/notes/scoring.md b/test-suite/notes/scoring.md index 970a706..5edb7f5 100644 --- a/test-suite/notes/scoring.md +++ b/test-suite/notes/scoring.md @@ -4,8 +4,16 @@ Score actual verdict against `expected[mode]`, not raw PASS count. -- `PASS` / `FAIL` expected: exact match required. -- `CONDITIONAL`: exclude from headline score unless environment prerequisite is declared present. +Gate buckets: + +- `core`: headline release gate. Exact expected verdict required. +- `conditional`: gate only when declared capability/prerequisite is present; otherwise report skipped/conditional. +- `quality`: adversarial humanization/fingerprint signal. Report trend and regressions, but do not block general release unless explicitly promoted. + +Expected values: + +- `PASS` / `FAIL` expected: exact match required inside active gate bucket. +- `CONDITIONAL`: exclude from core headline score unless environment prerequisite is declared present. - `SKIP` / `WARN`: report separately. Recommended flake policy: |