diff options
Diffstat (limited to 'test-suite/notes/runner-spec.md')
| -rw-r--r-- | test-suite/notes/runner-spec.md | 7 |
1 files changed, 5 insertions, 2 deletions
diff --git a/test-suite/notes/runner-spec.md b/test-suite/notes/runner-spec.md index b040cb4..24c97ab 100644 --- a/test-suite/notes/runner-spec.md +++ b/test-suite/notes/runner-spec.md @@ -3,13 +3,14 @@ Future Node/Pi runner should: 1. Read `manifest.json`. -2. Probe environment capabilities (`trusted`, `clipboard`, `touch`, `dialogs`, `downloads`, `fileSystem`). +2. Probe environment capabilities (`trusted`, `clipboard`, `touch`, `dialogs`, `downloads`, `fileSystem`, `strictCspFallback`). 3. For each entry: + - read `gate` (`core`, `conditional`, `quality`) and score in that bucket - skip if `requires` is unsatisfied and expected is `CONDITIONAL` - navigate to challenge URL - execute `recipe` in order - wait `verdictDelayMs || 500` - - evaluate `JSON.stringify({v:window.__verdict,r:window.__reason,d:window.Challenge?.state?.details,e:window.__events?.slice(-20)})` + - evaluate `JSON.stringify({v:window.__verdict,r:window.__reason,d:window.Challenge?.state?.details,e:window.__events?.slice(-20)})`, except strict-CSP pages where eval is expected to fail; read persisted verdict from dashboard/localStorage after leaving the CSP page - compare to `expected[mode]` 4. Output: - Markdown summary @@ -21,6 +22,8 @@ Runner must adapt recipe intent: - expand `${REPO_ROOT}` path placeholders - adapt unsupported shadow/iframe selector notation to snapshot uids or evaluate fallback - preserve hook install ordering for console/network capture tests +- substitute dynamic tab ids from `chrome_tab list` recipes +- support screenshot/coordinate fallback for strict CSP pages where snapshot/evaluate are blocked - record whether trusted/CDP path was used Long-horizon task runner should read `task-manifest.json`, replace `$RUN_ID`, solve task, then evaluate task grader expression. |