summaryrefslogtreecommitdiff
path: root/skills/pi-review-panel
diff options
context:
space:
mode:
authorIgor Soarez <igor@soarez.org>2026-09-17 19:45:23 +0100
committerIgor Soarez <igor@soarez.org>2026-09-17 19:45:23 +0100
commit13cc065ec014b86369dfe4ad60d1a43fa06fd251 (patch)
tree684690c61b7b59758c2aac132c65b81c86de0608 /skills/pi-review-panel
dotfiles
Shell, editor and terminal configuration, the tools that go with them, and szecret, an age-based secret manager whose store lives elsewhere. History restarts here. What came before was not written to be read by anyone else, and carried machine names, addresses and an encrypted secret store; it is kept privately instead. What is left is meant to be safe on any machine, including one whose owner is not me: see CLAUDE.md for the rule it is written to.
Diffstat (limited to 'skills/pi-review-panel')
-rw-r--r--skills/pi-review-panel/SKILL.md182
-rwxr-xr-xskills/pi-review-panel/run-member.sh88
2 files changed, 270 insertions, 0 deletions
diff --git a/skills/pi-review-panel/SKILL.md b/skills/pi-review-panel/SKILL.md
new file mode 100644
index 0000000..1320ad4
--- /dev/null
+++ b/skills/pi-review-panel/SKILL.md
@@ -0,0 +1,182 @@
+---
+name: pi-review-panel
+description: >-
+ Reviews whatever the user specifies — a diff, a design doc, a proposal, a
+ file, a question — with a panel of external models driven by the `pi` CLI,
+ then verifies and synthesizes their findings into one report, with per-model
+ cost. ONLY use this skill when the user explicitly asks for it (e.g. "run a
+ pi review panel", "panel review", "review this with the pi models") — never
+ invoke it on your own initiative for ordinary review requests. The user may
+ specify a subset of the default provider/model pairs, or a different set
+ entirely.
+---
+
+# pi review panel
+
+Review something with several external models via `pi`, each run
+independently, then verify their claims and synthesize.
+
+The fiddly parts (hang-avoidance, cost accounting, output extraction) are
+already solved in `run-member.sh` next to this file. Use it. Don't hand-roll
+`pi` invocations — the flag set and the redirection shape are load-bearing and
+the failure modes are silent hangs. If you're curious why, the reasons are
+documented at the top of that script.
+
+## 0. Panel composition
+
+The default panel is eight runs:
+
+| provider | model | output name |
+|---|---|---|
+| fireworks | `accounts/fireworks/models/kimi-k3` | fw-kimi-k3 |
+| openrouter | `moonshotai/kimi-k3` | or-kimi-k3 |
+| fireworks | `accounts/fireworks/models/glm-5p2` | fw-glm-5p2 |
+| openrouter | `z-ai/glm-5.2` | or-glm-5.2 |
+| fireworks | `accounts/fireworks/models/deepseek-v4-flash-0731` | fw-deepseek-v4 |
+| openrouter | `deepseek/deepseek-v4-flash-0731` | or-deepseek-v4 |
+| openrouter | `google/gemini-3.7-flash` | or-gemini-3.7-flash |
+| openrouter | `z-ai/glm-5.3` | or-glm-5.3 |
+
+If the user names specific providers/models, use exactly those instead — don't
+silently add the defaults back. Credentials are already configured in `pi`; no
+setup needed.
+
+## 1. Establish what is under review
+
+**Review what the user specified.** It is whatever they point at — a document,
+a proposal, a single file, a design question, a diff. Take their framing as
+given; the panel machinery is indifferent to what is in the brief.
+
+Only when they didn't specify, default to the diff: uncommitted changes if any
+(`git status`, `git diff`), otherwise the current branch against its merge-base
+with the default branch. Ask only if genuinely ambiguous.
+
+State at the top of the final report exactly what was reviewed (commit range,
+"working tree", or the file path).
+
+**Snapshot it.** If the target is a file in a repo someone else may be editing,
+record its mtime and content hash now, and re-check before you synthesize. A
+target that changes mid-run invalidates parts of the panel's reasoning and you
+need to know that happened.
+
+## 2. Write a review brief
+
+Create a scratch directory `$P` with `$P/out/`. Write `$P/brief.md`:
+
+- One paragraph of project context (what the project is, from its README).
+- The material under review, inline in a fenced block. Inline it even if you
+ also give the path — reviewers should not have to find it.
+- Absolute paths of the relevant files, so reviewers can read the surrounding
+ context with their tools.
+- The **question the user is actually asking**, up front. "Find bugs in this
+ diff" and "can you beat this design" produce very different reviews; the
+ brief must not leave reviewers to guess which one this is.
+- The **criteria**, stated as hard requirements the reviewer must judge
+ against. If the user gave criteria, use their words verbatim.
+- Reviewer instructions, matched to that question: act as a strict senior
+ reviewer; look for errors, incorrect claims (especially documented numbers,
+ or behavior the artifact does not actually produce), missing cases,
+ unnecessary complexity; for each finding give location, severity (blocker /
+ major / minor / nit), and a one-paragraph justification; **verify
+ empirically before asserting** — say which interpreter/version/command
+ produced each claim; end with a single verdict line; deliver the full
+ markdown review as the final answer, asking no questions and offering no
+ follow-up work.
+- Tell them everything they can reach is **read-only**. `pi` has edit and write
+ tools and will use them; a review never needs files mutated.
+
+A useful verdict line is a fixed pair, e.g. `VERDICT: SHIP` /
+`VERDICT: DO NOT SHIP`, or for a design review `VERDICT: PROPOSAL IS BEST` /
+`VERDICT: BETTER IDEA FOUND — <name>`. Fixed strings make the tally mechanical.
+
+## 3. Run the panel
+
+Launch all members concurrently. `$SK` is this skill's directory:
+
+```bash
+"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/kimi-k3" fw-kimi-k3 > "$P/out/_fw-kimi-k3.log" 2>&1 &
+"$SK/run-member.sh" "$P" openrouter "moonshotai/kimi-k3" or-kimi-k3 > "$P/out/_or-kimi-k3.log" 2>&1 &
+"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/glm-5p2" fw-glm-5p2 > "$P/out/_fw-glm-5p2.log" 2>&1 &
+"$SK/run-member.sh" "$P" openrouter "z-ai/glm-5.2" or-glm-5.2 > "$P/out/_or-glm-5.2.log" 2>&1 &
+"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/deepseek-v4-flash-0731" fw-deepseek-v4 > "$P/out/_fw-deepseek-v4.log" 2>&1 &
+"$SK/run-member.sh" "$P" openrouter "deepseek/deepseek-v4-flash-0731" or-deepseek-v4 > "$P/out/_or-deepseek-v4.log" 2>&1 &
+"$SK/run-member.sh" "$P" openrouter "google/gemini-3.7-flash" or-gemini-3.7-flash > "$P/out/_or-gemini-3.7-flash.log" 2>&1 &
+"$SK/run-member.sh" "$P" openrouter "z-ai/glm-5.3" or-glm-5.3 > "$P/out/_or-glm-5.3.log" 2>&1 &
+```
+
+Expect **5–30 minutes**. Poll with a loop that also watches disk, since the raw
+streams are large while runs are in flight:
+
+```bash
+while pgrep -f run-member.sh > /dev/null; do
+ echo "alive=$(pgrep -f run-member.sh | wc -l) freeGB=$(df -g "$P" | tail -1 | awk '{print $4}')"
+ sleep 60
+done
+```
+
+**Do the work you can do while they run.** This is the whole reason to launch
+early: don't sit on the poll loop. Verify the target yourself, build the
+measurement harness, establish the baseline. You will need your own evidence in
+step 5 regardless, and by the time the reviews land you want to be ready to
+adjudicate them, not starting from scratch.
+
+A run is good if `$P/out/<name>.md` is non-empty and ends with a verdict line.
+Retry a failure once. If it fails twice, note it in the report with its cost and
+continue — one dead provider must not sink the panel, and a duplicated model on
+the other provider means coverage usually survives.
+
+## 4. Cost accounting
+
+`run-member.sh` writes `$P/out/<name>.cost.json` per run, summed from the
+provider's own per-message `usage.cost` — actual billed figures, not estimates.
+Report them by default; the user should never have to ask what a panel cost.
+
+```bash
+jq -s -r 'sort_by(.cost)|reverse[]
+ | [.name,.provider,(.exit|tostring),(.seconds|tostring),(.turns|tostring),
+ (.input|tostring),(.output|tostring),(.cost*10000|round/10000|tostring)] | @tsv' \
+ "$P"/out/*.cost.json | column -t
+jq -s '{runs:length, total_cost:(map(.cost)|add)}' "$P"/out/*.cost.json
+```
+
+Include failed attempts as their own rows — a timed-out run bills for every
+token it generated and can easily be the most expensive line in the table. If a
+retry ran, keep both attempts (rename the first `<name>.attempt1.cost.json`
+before retrying, or it is overwritten).
+
+Cost per useful finding varies by two orders of magnitude across these models.
+Worth a sentence in the report when the spread is large — it informs whether
+the panel should shrink next time.
+
+## 5. Verify, then synthesize
+
+The panel's output is raw material, not findings. Reviewers state confident,
+specific, mutually contradictory things, and the disagreements are where the
+real answer usually is.
+
+- **Reproduce every substantive claim yourself** before it goes in the report.
+ Models hallucinate line numbers, APIs and measurements. Where reviewers
+ disagree, run the experiment that decides it and name who was right — that
+ adjudication is often the most valuable part of the report.
+- Consensus is weak evidence, not proof: on a version-specific question, four
+ reviewers agreeing can all be testing one version. A finding raised by one
+ reviewer and confirmed by you outranks one raised by four and confirmed by
+ nobody.
+- Watch for claims that are true of a variant but not of the thing actually
+ written — a reviewer "verifying" a snippet they silently corrected.
+- Beware your own harness. If you extract code to test it standalone, a missing
+ import can surface as a swallowed exception that looks exactly like the bug
+ you're hunting. Sanity-check that the baseline still passes the cases it
+ should.
+
+The report:
+
+- Findings deduplicated, ranked by severity, each tagged with how many members
+ raised it and whether **you** confirmed it, on what.
+- A "Dropped" section naming what you could not confirm and why. This is not
+ optional — it's what makes the confirmed findings trustworthy.
+- The verdict tally and any disagreement worth the author's attention.
+- The cost table from step 4.
+- Paths to the per-model raw outputs. `$P` is session scratch and will not
+ survive; if the findings matter, copy the reviews and any verification
+ scripts somewhere durable in the project and link those instead.
diff --git a/skills/pi-review-panel/run-member.sh b/skills/pi-review-panel/run-member.sh
new file mode 100755
index 0000000..7578c1e
--- /dev/null
+++ b/skills/pi-review-panel/run-member.sh
@@ -0,0 +1,88 @@
+#!/bin/bash
+# Run one pi-review-panel member.
+#
+# run-member.sh <workdir> <provider> <model> <name> [timeout_seconds]
+#
+# Reads <workdir>/brief.md, writes:
+# <workdir>/out/<name>.md the review markdown
+# <workdir>/out/<name>.cost.json tokens + USD cost, from the provider
+# <workdir>/out/<name>.jsonl filtered event stream (kept for debugging)
+# <workdir>/out/<name>.err stderr
+#
+# Exit status is pi's. 124 means the timeout fired — the run is a write-off,
+# but it still cost money; the cost file is written either way.
+#
+# Six mechanics here were each established the hard way. Do not "simplify"
+# them away:
+#
+# 1. `--offline` is required. pi's startup model-catalog fetch intermittently
+# hangs forever; --offline skips it. Symptom without it: the process
+# produces no output at all, not even the session line, and never exits.
+#
+# 2. pi's stdout must be redirected to a FILE, never piped. A pipeline
+# deadlocks: `timeout` kills pi, but the downstream reader never sees EOF
+# and hangs past its own timeout. This is why message_update filtering
+# happens after the run, not inline.
+#
+# 3. `--mode json` re-emits the entire message on every token delta, so the
+# raw stream grows quadratically — 790 MB and 3.1 GB on real runs. It is
+# filtered to the events that matter and the raw file is deleted at once.
+# Budget ~5 GB of scratch per concurrent member while runs are in flight.
+#
+# 4. Costs must be summed over every assistant message, not read off the last
+# one. Each message carries only its own usage.
+#
+# 5. pi's stdin must be /dev/null. In -p mode pi also reads stdin when it is
+# not a terminal, and waits for EOF. Started from a shell whose stdin never
+# closes — an agent's tool shell, a backgrounded job — it hangs forever
+# with no output at all, which looks exactly like the hang in (1). Found
+# 2026-09-14: two members timed out after 30 minutes having never called
+# the API.
+#
+# 6. Byte counts are stripped of whitespace before jq reads them. macOS `wc -c`
+# pads its output (" 6"), jq's tonumber rejects that, and the whole
+# cost file silently came out empty.
+
+set -u
+
+P="${1:?workdir}"; provider="${2:?provider}"; model="${3:?model}"; name="${4:?name}"
+limit="${5:-1800}"
+
+mkdir -p "$P/out"
+start=$SECONDS
+
+timeout "$limit" pi --provider "$provider" --model "$model" \
+ -p --no-session -nc --no-extensions -ns --offline \
+ --thinking high --mode json \
+ "Read the file $P/brief.md and do exactly what it says. Deliver the full markdown review as your final answer." \
+ < /dev/null > "$P/out/$name.raw" 2> "$P/out/$name.err"
+rc=$?
+elapsed=$((SECONDS - start))
+
+raw_bytes=$(wc -c < "$P/out/$name.raw" | tr -d '[:space:]')
+grep -v '"type":"message_update"' "$P/out/$name.raw" > "$P/out/$name.jsonl"
+rm -f "$P/out/$name.raw"
+
+# The review: text blocks of the last assistant message.
+jq -rs '
+ [.[] | select(.type=="agent_end")] | last | .messages
+ | [.[] | select(.role=="assistant")] | last
+ | [.content[] | select(.type=="text") | .text] | join("")
+' "$P/out/$name.jsonl" > "$P/out/$name.md" 2>/dev/null
+
+jq -rs --arg name "$name" --arg model "$model" --arg provider "$provider" \
+ --arg rc "$rc" --arg elapsed "$elapsed" --arg raw "$raw_bytes" '
+ [.[] | select(.type=="message_end" and .message.role=="assistant") | .message.usage]
+ | {name: $name, provider: $provider, model: $model, exit: ($rc|tonumber),
+ seconds: ($elapsed|tonumber), turns: length,
+ input: (map(.input) | add // 0), output: (map(.output) | add // 0),
+ cacheRead: (map(.cacheRead) | add // 0),
+ cacheWrite: (map(.cacheWrite) | add // 0),
+ rawStreamBytes: ($raw|tonumber),
+ cost: (map(.cost.total) | add // 0)}
+' "$P/out/$name.jsonl" > "$P/out/$name.cost.json" 2>/dev/null
+
+printf '%-16s exit:%-4s %5ss review:%7s bytes cost:$%s\n' \
+ "$name" "$rc" "$elapsed" "$(wc -c < "$P/out/$name.md" | tr -d '[:space:]')" \
+ "$(jq -r '.cost' "$P/out/$name.cost.json" 2>/dev/null)"
+exit $rc