diff options
Diffstat (limited to 'skills/pi-review-panel')
| -rw-r--r-- | skills/pi-review-panel/SKILL.md | 182 | ||||
| -rwxr-xr-x | skills/pi-review-panel/run-member.sh | 88 |
2 files changed, 270 insertions, 0 deletions
diff --git a/skills/pi-review-panel/SKILL.md b/skills/pi-review-panel/SKILL.md new file mode 100644 index 0000000..1320ad4 --- /dev/null +++ b/skills/pi-review-panel/SKILL.md @@ -0,0 +1,182 @@ +--- +name: pi-review-panel +description: >- + Reviews whatever the user specifies — a diff, a design doc, a proposal, a + file, a question — with a panel of external models driven by the `pi` CLI, + then verifies and synthesizes their findings into one report, with per-model + cost. ONLY use this skill when the user explicitly asks for it (e.g. "run a + pi review panel", "panel review", "review this with the pi models") — never + invoke it on your own initiative for ordinary review requests. The user may + specify a subset of the default provider/model pairs, or a different set + entirely. +--- + +# pi review panel + +Review something with several external models via `pi`, each run +independently, then verify their claims and synthesize. + +The fiddly parts (hang-avoidance, cost accounting, output extraction) are +already solved in `run-member.sh` next to this file. Use it. Don't hand-roll +`pi` invocations — the flag set and the redirection shape are load-bearing and +the failure modes are silent hangs. If you're curious why, the reasons are +documented at the top of that script. + +## 0. Panel composition + +The default panel is eight runs: + +| provider | model | output name | +|---|---|---| +| fireworks | `accounts/fireworks/models/kimi-k3` | fw-kimi-k3 | +| openrouter | `moonshotai/kimi-k3` | or-kimi-k3 | +| fireworks | `accounts/fireworks/models/glm-5p2` | fw-glm-5p2 | +| openrouter | `z-ai/glm-5.2` | or-glm-5.2 | +| fireworks | `accounts/fireworks/models/deepseek-v4-flash-0731` | fw-deepseek-v4 | +| openrouter | `deepseek/deepseek-v4-flash-0731` | or-deepseek-v4 | +| openrouter | `google/gemini-3.7-flash` | or-gemini-3.7-flash | +| openrouter | `z-ai/glm-5.3` | or-glm-5.3 | + +If the user names specific providers/models, use exactly those instead — don't +silently add the defaults back. Credentials are already configured in `pi`; no +setup needed. + +## 1. Establish what is under review + +**Review what the user specified.** It is whatever they point at — a document, +a proposal, a single file, a design question, a diff. Take their framing as +given; the panel machinery is indifferent to what is in the brief. + +Only when they didn't specify, default to the diff: uncommitted changes if any +(`git status`, `git diff`), otherwise the current branch against its merge-base +with the default branch. Ask only if genuinely ambiguous. + +State at the top of the final report exactly what was reviewed (commit range, +"working tree", or the file path). + +**Snapshot it.** If the target is a file in a repo someone else may be editing, +record its mtime and content hash now, and re-check before you synthesize. A +target that changes mid-run invalidates parts of the panel's reasoning and you +need to know that happened. + +## 2. Write a review brief + +Create a scratch directory `$P` with `$P/out/`. Write `$P/brief.md`: + +- One paragraph of project context (what the project is, from its README). +- The material under review, inline in a fenced block. Inline it even if you + also give the path — reviewers should not have to find it. +- Absolute paths of the relevant files, so reviewers can read the surrounding + context with their tools. +- The **question the user is actually asking**, up front. "Find bugs in this + diff" and "can you beat this design" produce very different reviews; the + brief must not leave reviewers to guess which one this is. +- The **criteria**, stated as hard requirements the reviewer must judge + against. If the user gave criteria, use their words verbatim. +- Reviewer instructions, matched to that question: act as a strict senior + reviewer; look for errors, incorrect claims (especially documented numbers, + or behavior the artifact does not actually produce), missing cases, + unnecessary complexity; for each finding give location, severity (blocker / + major / minor / nit), and a one-paragraph justification; **verify + empirically before asserting** — say which interpreter/version/command + produced each claim; end with a single verdict line; deliver the full + markdown review as the final answer, asking no questions and offering no + follow-up work. +- Tell them everything they can reach is **read-only**. `pi` has edit and write + tools and will use them; a review never needs files mutated. + +A useful verdict line is a fixed pair, e.g. `VERDICT: SHIP` / +`VERDICT: DO NOT SHIP`, or for a design review `VERDICT: PROPOSAL IS BEST` / +`VERDICT: BETTER IDEA FOUND — <name>`. Fixed strings make the tally mechanical. + +## 3. Run the panel + +Launch all members concurrently. `$SK` is this skill's directory: + +```bash +"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/kimi-k3" fw-kimi-k3 > "$P/out/_fw-kimi-k3.log" 2>&1 & +"$SK/run-member.sh" "$P" openrouter "moonshotai/kimi-k3" or-kimi-k3 > "$P/out/_or-kimi-k3.log" 2>&1 & +"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/glm-5p2" fw-glm-5p2 > "$P/out/_fw-glm-5p2.log" 2>&1 & +"$SK/run-member.sh" "$P" openrouter "z-ai/glm-5.2" or-glm-5.2 > "$P/out/_or-glm-5.2.log" 2>&1 & +"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/deepseek-v4-flash-0731" fw-deepseek-v4 > "$P/out/_fw-deepseek-v4.log" 2>&1 & +"$SK/run-member.sh" "$P" openrouter "deepseek/deepseek-v4-flash-0731" or-deepseek-v4 > "$P/out/_or-deepseek-v4.log" 2>&1 & +"$SK/run-member.sh" "$P" openrouter "google/gemini-3.7-flash" or-gemini-3.7-flash > "$P/out/_or-gemini-3.7-flash.log" 2>&1 & +"$SK/run-member.sh" "$P" openrouter "z-ai/glm-5.3" or-glm-5.3 > "$P/out/_or-glm-5.3.log" 2>&1 & +``` + +Expect **5–30 minutes**. Poll with a loop that also watches disk, since the raw +streams are large while runs are in flight: + +```bash +while pgrep -f run-member.sh > /dev/null; do + echo "alive=$(pgrep -f run-member.sh | wc -l) freeGB=$(df -g "$P" | tail -1 | awk '{print $4}')" + sleep 60 +done +``` + +**Do the work you can do while they run.** This is the whole reason to launch +early: don't sit on the poll loop. Verify the target yourself, build the +measurement harness, establish the baseline. You will need your own evidence in +step 5 regardless, and by the time the reviews land you want to be ready to +adjudicate them, not starting from scratch. + +A run is good if `$P/out/<name>.md` is non-empty and ends with a verdict line. +Retry a failure once. If it fails twice, note it in the report with its cost and +continue — one dead provider must not sink the panel, and a duplicated model on +the other provider means coverage usually survives. + +## 4. Cost accounting + +`run-member.sh` writes `$P/out/<name>.cost.json` per run, summed from the +provider's own per-message `usage.cost` — actual billed figures, not estimates. +Report them by default; the user should never have to ask what a panel cost. + +```bash +jq -s -r 'sort_by(.cost)|reverse[] + | [.name,.provider,(.exit|tostring),(.seconds|tostring),(.turns|tostring), + (.input|tostring),(.output|tostring),(.cost*10000|round/10000|tostring)] | @tsv' \ + "$P"/out/*.cost.json | column -t +jq -s '{runs:length, total_cost:(map(.cost)|add)}' "$P"/out/*.cost.json +``` + +Include failed attempts as their own rows — a timed-out run bills for every +token it generated and can easily be the most expensive line in the table. If a +retry ran, keep both attempts (rename the first `<name>.attempt1.cost.json` +before retrying, or it is overwritten). + +Cost per useful finding varies by two orders of magnitude across these models. +Worth a sentence in the report when the spread is large — it informs whether +the panel should shrink next time. + +## 5. Verify, then synthesize + +The panel's output is raw material, not findings. Reviewers state confident, +specific, mutually contradictory things, and the disagreements are where the +real answer usually is. + +- **Reproduce every substantive claim yourself** before it goes in the report. + Models hallucinate line numbers, APIs and measurements. Where reviewers + disagree, run the experiment that decides it and name who was right — that + adjudication is often the most valuable part of the report. +- Consensus is weak evidence, not proof: on a version-specific question, four + reviewers agreeing can all be testing one version. A finding raised by one + reviewer and confirmed by you outranks one raised by four and confirmed by + nobody. +- Watch for claims that are true of a variant but not of the thing actually + written — a reviewer "verifying" a snippet they silently corrected. +- Beware your own harness. If you extract code to test it standalone, a missing + import can surface as a swallowed exception that looks exactly like the bug + you're hunting. Sanity-check that the baseline still passes the cases it + should. + +The report: + +- Findings deduplicated, ranked by severity, each tagged with how many members + raised it and whether **you** confirmed it, on what. +- A "Dropped" section naming what you could not confirm and why. This is not + optional — it's what makes the confirmed findings trustworthy. +- The verdict tally and any disagreement worth the author's attention. +- The cost table from step 4. +- Paths to the per-model raw outputs. `$P` is session scratch and will not + survive; if the findings matter, copy the reviews and any verification + scripts somewhere durable in the project and link those instead. diff --git a/skills/pi-review-panel/run-member.sh b/skills/pi-review-panel/run-member.sh new file mode 100755 index 0000000..7578c1e --- /dev/null +++ b/skills/pi-review-panel/run-member.sh @@ -0,0 +1,88 @@ +#!/bin/bash +# Run one pi-review-panel member. +# +# run-member.sh <workdir> <provider> <model> <name> [timeout_seconds] +# +# Reads <workdir>/brief.md, writes: +# <workdir>/out/<name>.md the review markdown +# <workdir>/out/<name>.cost.json tokens + USD cost, from the provider +# <workdir>/out/<name>.jsonl filtered event stream (kept for debugging) +# <workdir>/out/<name>.err stderr +# +# Exit status is pi's. 124 means the timeout fired — the run is a write-off, +# but it still cost money; the cost file is written either way. +# +# Six mechanics here were each established the hard way. Do not "simplify" +# them away: +# +# 1. `--offline` is required. pi's startup model-catalog fetch intermittently +# hangs forever; --offline skips it. Symptom without it: the process +# produces no output at all, not even the session line, and never exits. +# +# 2. pi's stdout must be redirected to a FILE, never piped. A pipeline +# deadlocks: `timeout` kills pi, but the downstream reader never sees EOF +# and hangs past its own timeout. This is why message_update filtering +# happens after the run, not inline. +# +# 3. `--mode json` re-emits the entire message on every token delta, so the +# raw stream grows quadratically — 790 MB and 3.1 GB on real runs. It is +# filtered to the events that matter and the raw file is deleted at once. +# Budget ~5 GB of scratch per concurrent member while runs are in flight. +# +# 4. Costs must be summed over every assistant message, not read off the last +# one. Each message carries only its own usage. +# +# 5. pi's stdin must be /dev/null. In -p mode pi also reads stdin when it is +# not a terminal, and waits for EOF. Started from a shell whose stdin never +# closes — an agent's tool shell, a backgrounded job — it hangs forever +# with no output at all, which looks exactly like the hang in (1). Found +# 2026-09-14: two members timed out after 30 minutes having never called +# the API. +# +# 6. Byte counts are stripped of whitespace before jq reads them. macOS `wc -c` +# pads its output (" 6"), jq's tonumber rejects that, and the whole +# cost file silently came out empty. + +set -u + +P="${1:?workdir}"; provider="${2:?provider}"; model="${3:?model}"; name="${4:?name}" +limit="${5:-1800}" + +mkdir -p "$P/out" +start=$SECONDS + +timeout "$limit" pi --provider "$provider" --model "$model" \ + -p --no-session -nc --no-extensions -ns --offline \ + --thinking high --mode json \ + "Read the file $P/brief.md and do exactly what it says. Deliver the full markdown review as your final answer." \ + < /dev/null > "$P/out/$name.raw" 2> "$P/out/$name.err" +rc=$? +elapsed=$((SECONDS - start)) + +raw_bytes=$(wc -c < "$P/out/$name.raw" | tr -d '[:space:]') +grep -v '"type":"message_update"' "$P/out/$name.raw" > "$P/out/$name.jsonl" +rm -f "$P/out/$name.raw" + +# The review: text blocks of the last assistant message. +jq -rs ' + [.[] | select(.type=="agent_end")] | last | .messages + | [.[] | select(.role=="assistant")] | last + | [.content[] | select(.type=="text") | .text] | join("") +' "$P/out/$name.jsonl" > "$P/out/$name.md" 2>/dev/null + +jq -rs --arg name "$name" --arg model "$model" --arg provider "$provider" \ + --arg rc "$rc" --arg elapsed "$elapsed" --arg raw "$raw_bytes" ' + [.[] | select(.type=="message_end" and .message.role=="assistant") | .message.usage] + | {name: $name, provider: $provider, model: $model, exit: ($rc|tonumber), + seconds: ($elapsed|tonumber), turns: length, + input: (map(.input) | add // 0), output: (map(.output) | add // 0), + cacheRead: (map(.cacheRead) | add // 0), + cacheWrite: (map(.cacheWrite) | add // 0), + rawStreamBytes: ($raw|tonumber), + cost: (map(.cost.total) | add // 0)} +' "$P/out/$name.jsonl" > "$P/out/$name.cost.json" 2>/dev/null + +printf '%-16s exit:%-4s %5ss review:%7s bytes cost:$%s\n' \ + "$name" "$rc" "$elapsed" "$(wc -c < "$P/out/$name.md" | tr -d '[:space:]')" \ + "$(jq -r '.cost' "$P/out/$name.cost.json" 2>/dev/null)" +exit $rc |
