diff options
| -rw-r--r-- | skills/pi-review-panel/SKILL.md | 170 | ||||
| -rwxr-xr-x | skills/pi-review-panel/run-member.sh | 88 | ||||
| -rw-r--r-- | skills/pi-subagent/SKILL.md | 209 | ||||
| -rwxr-xr-x | skills/pi-subagent/run-agent.sh | 189 | ||||
| -rwxr-xr-x | skills/pi-subagent/status.sh | 85 |
5 files changed, 530 insertions, 211 deletions
diff --git a/skills/pi-review-panel/SKILL.md b/skills/pi-review-panel/SKILL.md index 1320ad4..f850ebb 100644 --- a/skills/pi-review-panel/SKILL.md +++ b/skills/pi-review-panel/SKILL.md @@ -16,17 +16,16 @@ description: >- Review something with several external models via `pi`, each run independently, then verify their claims and synthesize. -The fiddly parts (hang-avoidance, cost accounting, output extraction) are -already solved in `run-member.sh` next to this file. Use it. Don't hand-roll -`pi` invocations — the flag set and the redirection shape are load-bearing and -the failure modes are silent hangs. If you're curious why, the reasons are -documented at the top of that script. +This is a **fan-out** in the sense of the pi-subagent skill, which does the +running: read its SKILL.md first and follow it for the workdir, launching, +monitoring, retries, cost table and verification. `$SA` below is that skill's +directory. What follows is only what a review panel adds. ## 0. Panel composition The default panel is eight runs: -| provider | model | output name | +| provider | model | name | |---|---|---| | fireworks | `accounts/fireworks/models/kimi-k3` | fw-kimi-k3 | | openrouter | `moonshotai/kimi-k3` | or-kimi-k3 | @@ -38,145 +37,70 @@ The default panel is eight runs: | openrouter | `z-ai/glm-5.3` | or-glm-5.3 | If the user names specific providers/models, use exactly those instead — don't -silently add the defaults back. Credentials are already configured in `pi`; no -setup needed. +silently add the defaults back. A provider this machine lacks +(`pi --offline --list-models <model>` shows nothing for it) is dropped and +named in the report, not substituted. ## 1. Establish what is under review **Review what the user specified.** It is whatever they point at — a document, a proposal, a single file, a design question, a diff. Take their framing as -given; the panel machinery is indifferent to what is in the brief. +given. Only when they didn't specify, default to the diff: uncommitted changes if any (`git status`, `git diff`), otherwise the current branch against its merge-base with the default branch. Ask only if genuinely ambiguous. State at the top of the final report exactly what was reviewed (commit range, -"working tree", or the file path). - -**Snapshot it.** If the target is a file in a repo someone else may be editing, -record its mtime and content hash now, and re-check before you synthesize. A -target that changes mid-run invalidates parts of the panel's reasoning and you -need to know that happened. - -## 2. Write a review brief - -Create a scratch directory `$P` with `$P/out/`. Write `$P/brief.md`: - -- One paragraph of project context (what the project is, from its README). -- The material under review, inline in a fenced block. Inline it even if you - also give the path — reviewers should not have to find it. -- Absolute paths of the relevant files, so reviewers can read the surrounding - context with their tools. -- The **question the user is actually asking**, up front. "Find bugs in this - diff" and "can you beat this design" produce very different reviews; the - brief must not leave reviewers to guess which one this is. -- The **criteria**, stated as hard requirements the reviewer must judge - against. If the user gave criteria, use their words verbatim. -- Reviewer instructions, matched to that question: act as a strict senior - reviewer; look for errors, incorrect claims (especially documented numbers, - or behavior the artifact does not actually produce), missing cases, - unnecessary complexity; for each finding give location, severity (blocker / - major / minor / nit), and a one-paragraph justification; **verify - empirically before asserting** — say which interpreter/version/command - produced each claim; end with a single verdict line; deliver the full - markdown review as the final answer, asking no questions and offering no - follow-up work. -- Tell them everything they can reach is **read-only**. `pi` has edit and write - tools and will use them; a review never needs files mutated. - -A useful verdict line is a fixed pair, e.g. `VERDICT: SHIP` / -`VERDICT: DO NOT SHIP`, or for a design review `VERDICT: PROPOSAL IS BEST` / -`VERDICT: BETTER IDEA FOUND — <name>`. Fixed strings make the tally mechanical. +"working tree", or the file path), and snapshot it as pi-subagent §1 says. + +## 2. Write the review brief + +`$P/brief.md`, following pi-subagent §2, with these specifics: + +- The question up front: "find bugs in this diff" and "can you beat this + design" are different reviews. +- The reviewer stance: a strict senior reviewer, looking for errors, incorrect + claims (especially documented numbers, or behavior the artifact does not + actually produce), missing cases, unnecessary complexity. +- For each finding: location, severity (blocker / major / minor / nit), and a + one-paragraph justification. +- Everything they can reach is **read-only**. Reviewers need to run things to + verify, so this is read-only by instruction (pi-subagent §3, level 2), not + `--tools`. +- A fixed verdict pair as the last line: `VERDICT: SHIP` / + `VERDICT: DO NOT SHIP`, or for a design review `VERDICT: PROPOSAL IS BEST` / + `VERDICT: BETTER IDEA FOUND — <name>`. ## 3. Run the panel -Launch all members concurrently. `$SK` is this skill's directory: +All members at once, one line per row of the table: ```bash -"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/kimi-k3" fw-kimi-k3 > "$P/out/_fw-kimi-k3.log" 2>&1 & -"$SK/run-member.sh" "$P" openrouter "moonshotai/kimi-k3" or-kimi-k3 > "$P/out/_or-kimi-k3.log" 2>&1 & -"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/glm-5p2" fw-glm-5p2 > "$P/out/_fw-glm-5p2.log" 2>&1 & -"$SK/run-member.sh" "$P" openrouter "z-ai/glm-5.2" or-glm-5.2 > "$P/out/_or-glm-5.2.log" 2>&1 & -"$SK/run-member.sh" "$P" fireworks "accounts/fireworks/models/deepseek-v4-flash-0731" fw-deepseek-v4 > "$P/out/_fw-deepseek-v4.log" 2>&1 & -"$SK/run-member.sh" "$P" openrouter "deepseek/deepseek-v4-flash-0731" or-deepseek-v4 > "$P/out/_or-deepseek-v4.log" 2>&1 & -"$SK/run-member.sh" "$P" openrouter "google/gemini-3.7-flash" or-gemini-3.7-flash > "$P/out/_or-gemini-3.7-flash.log" 2>&1 & -"$SK/run-member.sh" "$P" openrouter "z-ai/glm-5.3" or-glm-5.3 > "$P/out/_or-glm-5.3.log" 2>&1 & +"$SA/run-agent.sh" --deliver "the full markdown review" "$P" fireworks "accounts/fireworks/models/kimi-k3" fw-kimi-k3 > "$P/out/_fw-kimi-k3.log" 2>&1 & +# ...and so on for each member ``` -Expect **5–30 minutes**. Poll with a loop that also watches disk, since the raw -streams are large while runs are in flight: +Expect 5–30 minutes; use them as pi-subagent §5 says. A member succeeded if +its review ends with a verdict line. A provider that fails twice doesn't sink +the panel: the same model on the other provider usually keeps the coverage. -```bash -while pgrep -f run-member.sh > /dev/null; do - echo "alive=$(pgrep -f run-member.sh | wc -l) freeGB=$(df -g "$P" | tail -1 | awk '{print $4}')" - sleep 60 -done -``` - -**Do the work you can do while they run.** This is the whole reason to launch -early: don't sit on the poll loop. Verify the target yourself, build the -measurement harness, establish the baseline. You will need your own evidence in -step 5 regardless, and by the time the reviews land you want to be ready to -adjudicate them, not starting from scratch. - -A run is good if `$P/out/<name>.md` is non-empty and ends with a verdict line. -Retry a failure once. If it fails twice, note it in the report with its cost and -continue — one dead provider must not sink the panel, and a duplicated model on -the other provider means coverage usually survives. - -## 4. Cost accounting - -`run-member.sh` writes `$P/out/<name>.cost.json` per run, summed from the -provider's own per-message `usage.cost` — actual billed figures, not estimates. -Report them by default; the user should never have to ask what a panel cost. - -```bash -jq -s -r 'sort_by(.cost)|reverse[] - | [.name,.provider,(.exit|tostring),(.seconds|tostring),(.turns|tostring), - (.input|tostring),(.output|tostring),(.cost*10000|round/10000|tostring)] | @tsv' \ - "$P"/out/*.cost.json | column -t -jq -s '{runs:length, total_cost:(map(.cost)|add)}' "$P"/out/*.cost.json -``` +## 4. Synthesize -Include failed attempts as their own rows — a timed-out run bills for every -token it generated and can easily be the most expensive line in the table. If a -retry ran, keep both attempts (rename the first `<name>.attempt1.cost.json` -before retrying, or it is overwritten). - -Cost per useful finding varies by two orders of magnitude across these models. -Worth a sentence in the report when the spread is large — it informs whether -the panel should shrink next time. - -## 5. Verify, then synthesize - -The panel's output is raw material, not findings. Reviewers state confident, -specific, mutually contradictory things, and the disagreements are where the -real answer usually is. - -- **Reproduce every substantive claim yourself** before it goes in the report. - Models hallucinate line numbers, APIs and measurements. Where reviewers - disagree, run the experiment that decides it and name who was right — that - adjudication is often the most valuable part of the report. -- Consensus is weak evidence, not proof: on a version-specific question, four - reviewers agreeing can all be testing one version. A finding raised by one - reviewer and confirmed by you outranks one raised by four and confirmed by - nobody. -- Watch for claims that are true of a variant but not of the thing actually - written — a reviewer "verifying" a snippet they silently corrected. -- Beware your own harness. If you extract code to test it standalone, a missing - import can surface as a swallowed exception that looks exactly like the bug - you're hunting. Sanity-check that the baseline still passes the cases it - should. +Verify first (pi-subagent §7): the reviews are raw material, and the places +where reviewers contradict each other are where the real answer usually is. The report: +- What was reviewed. - Findings deduplicated, ranked by severity, each tagged with how many members raised it and whether **you** confirmed it, on what. -- A "Dropped" section naming what you could not confirm and why. This is not - optional — it's what makes the confirmed findings trustworthy. -- The verdict tally and any disagreement worth the author's attention. -- The cost table from step 4. -- Paths to the per-model raw outputs. `$P` is session scratch and will not - survive; if the findings matter, copy the reviews and any verification - scripts somewhere durable in the project and link those instead. +- A "Dropped" section naming what you could not confirm and why. Not optional — + it's what makes the confirmed findings trustworthy. +- The verdict tally and any disagreement worth the author's attention, with who + was right where you settled it. +- The cost table (`"$SA/status.sh" "$P"`), and a sentence on cost per useful + finding when the spread is large — it informs whether the panel should + shrink next time. +- Paths to the per-model reviews, copied somewhere durable if the findings + matter (pi-subagent §9). diff --git a/skills/pi-review-panel/run-member.sh b/skills/pi-review-panel/run-member.sh deleted file mode 100755 index 7578c1e..0000000 --- a/skills/pi-review-panel/run-member.sh +++ /dev/null @@ -1,88 +0,0 @@ -#!/bin/bash -# Run one pi-review-panel member. -# -# run-member.sh <workdir> <provider> <model> <name> [timeout_seconds] -# -# Reads <workdir>/brief.md, writes: -# <workdir>/out/<name>.md the review markdown -# <workdir>/out/<name>.cost.json tokens + USD cost, from the provider -# <workdir>/out/<name>.jsonl filtered event stream (kept for debugging) -# <workdir>/out/<name>.err stderr -# -# Exit status is pi's. 124 means the timeout fired — the run is a write-off, -# but it still cost money; the cost file is written either way. -# -# Six mechanics here were each established the hard way. Do not "simplify" -# them away: -# -# 1. `--offline` is required. pi's startup model-catalog fetch intermittently -# hangs forever; --offline skips it. Symptom without it: the process -# produces no output at all, not even the session line, and never exits. -# -# 2. pi's stdout must be redirected to a FILE, never piped. A pipeline -# deadlocks: `timeout` kills pi, but the downstream reader never sees EOF -# and hangs past its own timeout. This is why message_update filtering -# happens after the run, not inline. -# -# 3. `--mode json` re-emits the entire message on every token delta, so the -# raw stream grows quadratically — 790 MB and 3.1 GB on real runs. It is -# filtered to the events that matter and the raw file is deleted at once. -# Budget ~5 GB of scratch per concurrent member while runs are in flight. -# -# 4. Costs must be summed over every assistant message, not read off the last -# one. Each message carries only its own usage. -# -# 5. pi's stdin must be /dev/null. In -p mode pi also reads stdin when it is -# not a terminal, and waits for EOF. Started from a shell whose stdin never -# closes — an agent's tool shell, a backgrounded job — it hangs forever -# with no output at all, which looks exactly like the hang in (1). Found -# 2026-09-14: two members timed out after 30 minutes having never called -# the API. -# -# 6. Byte counts are stripped of whitespace before jq reads them. macOS `wc -c` -# pads its output (" 6"), jq's tonumber rejects that, and the whole -# cost file silently came out empty. - -set -u - -P="${1:?workdir}"; provider="${2:?provider}"; model="${3:?model}"; name="${4:?name}" -limit="${5:-1800}" - -mkdir -p "$P/out" -start=$SECONDS - -timeout "$limit" pi --provider "$provider" --model "$model" \ - -p --no-session -nc --no-extensions -ns --offline \ - --thinking high --mode json \ - "Read the file $P/brief.md and do exactly what it says. Deliver the full markdown review as your final answer." \ - < /dev/null > "$P/out/$name.raw" 2> "$P/out/$name.err" -rc=$? -elapsed=$((SECONDS - start)) - -raw_bytes=$(wc -c < "$P/out/$name.raw" | tr -d '[:space:]') -grep -v '"type":"message_update"' "$P/out/$name.raw" > "$P/out/$name.jsonl" -rm -f "$P/out/$name.raw" - -# The review: text blocks of the last assistant message. -jq -rs ' - [.[] | select(.type=="agent_end")] | last | .messages - | [.[] | select(.role=="assistant")] | last - | [.content[] | select(.type=="text") | .text] | join("") -' "$P/out/$name.jsonl" > "$P/out/$name.md" 2>/dev/null - -jq -rs --arg name "$name" --arg model "$model" --arg provider "$provider" \ - --arg rc "$rc" --arg elapsed "$elapsed" --arg raw "$raw_bytes" ' - [.[] | select(.type=="message_end" and .message.role=="assistant") | .message.usage] - | {name: $name, provider: $provider, model: $model, exit: ($rc|tonumber), - seconds: ($elapsed|tonumber), turns: length, - input: (map(.input) | add // 0), output: (map(.output) | add // 0), - cacheRead: (map(.cacheRead) | add // 0), - cacheWrite: (map(.cacheWrite) | add // 0), - rawStreamBytes: ($raw|tonumber), - cost: (map(.cost.total) | add // 0)} -' "$P/out/$name.jsonl" > "$P/out/$name.cost.json" 2>/dev/null - -printf '%-16s exit:%-4s %5ss review:%7s bytes cost:$%s\n' \ - "$name" "$rc" "$elapsed" "$(wc -c < "$P/out/$name.md" | tr -d '[:space:]')" \ - "$(jq -r '.cost' "$P/out/$name.cost.json" 2>/dev/null)" -exit $rc diff --git a/skills/pi-subagent/SKILL.md b/skills/pi-subagent/SKILL.md new file mode 100644 index 0000000..c5f1bfb --- /dev/null +++ b/skills/pi-subagent/SKILL.md @@ -0,0 +1,209 @@ +--- +name: pi-subagent +description: >- + Delegates work to one or more headless `pi` subagents on any provider/model, + run in the background, and collects each one's final answer plus its actual + billed cost. Covers a single delegated task, the same task fanned out to + several models for independent answers, or a job split into separate tasks + run in parallel, followed by verifying what came back. Use when the user + asks for subagents, to hand something to another model, to get several + models' takes, or to parallelize work across pi runs — and when another + skill (such as pi-review-panel) builds on it. Spawning costs money: never + start subagents on your own initiative without the user asking. +--- + +# pi subagents + +Run `pi` headless as a subagent: a fresh process, a written brief, no memory of +this conversation, one final answer. You stay the orchestrator. Subagents +produce raw material; you check it before it reaches the user. + +The fiddly parts (hang avoidance, output extraction, cost accounting, keeping +failed attempts) are solved in `run-agent.sh` next to this file. Use it. Don't +hand-roll `pi -p` invocations — the flag set and the redirection shape are +load-bearing, and the failure modes are silent hangs. The reasons are at the +top of the script. `$SK` below is this skill's directory. + +## 0. Shape the job + +Pick one of three shapes, and say which to the user if it isn't obvious: + +| shape | what | when | +|---|---|---| +| **delegate** | one task, one model | a self-contained job worth running elsewhere: a long investigation, a second opinion from a specific model, work you want off your own context | +| **fan-out** | one brief, N models | you want independent answers to compare: reviews, designs, hard questions. Disagreement is the signal | +| **split** | N briefs, one model each | a job that divides cleanly into parts that don't depend on each other's results | + +A subagent knows nothing you don't write down, can't ask you anything, and +can't see your other subagents. If the parts of a split need each other's +output, it's a sequence, not a split — run it in stages. + +**Models.** Use exactly the provider/model pairs the user names; don't add +others. If they named none, ask — or, for a fan-out, offer the default panel in +the pi-review-panel skill. Check a name resolves first: +`pi --offline --list-models <search>`. Credentials are already configured in +`pi`. + +## 1. Set up a workdir + +```bash +P="$(mktemp -d -t pi-subagent)"; mkdir -p "$P/out" +``` + +`$P` is scratch. One workdir per job; several briefs can live in it +(`$P/brief-<part>.md`), and every run writes `$P/out/<name>.*`. + +**Snapshot the inputs.** If the subagents will read files someone else may be +editing — including you — record their content hash now and re-check before you +use the answers. Input that changes mid-run invalidates parts of their +reasoning, and you need to know that happened. + +## 2. Write the brief + +The brief is the whole interface. Write it as if for a capable contractor who +has never seen the project and will not get to ask questions: + +- **The task, first sentence.** What you actually want. "Find bugs in this + diff" and "propose a better design than this one" produce very different + work; don't leave it to be guessed. +- **Context.** One paragraph on the project (from its README) and on why this + task exists. +- **The material, inline** in fenced blocks, even if you also give paths — the + subagent should not have to go looking for the thing it's about. +- **Absolute paths** of relevant files and directories, so it can read the + surroundings with its tools. +- **Criteria**, as hard requirements. If the user gave criteria, quote them + verbatim. +- **Evidence rules.** Verify empirically before asserting, and say which + command, interpreter, or version produced each claim. Distinguish what was + run from what was reasoned. +- **What it may touch** — see §3 — stated plainly. `pi` has edit and write + tools and will use them unless told, or prevented. +- **The deliverable**: format, sections, length. End with a **fixed final + line** where answers will be compared — `VERDICT: SHIP` / + `VERDICT: DO NOT SHIP`, `ANSWER: <x>`, `STATUS: DONE` / + `STATUS: BLOCKED — <why>`. Fixed strings make the tally and the success check + mechanical. +- **No questions, no follow-ups.** Deliver the full result as the final answer; + don't ask for confirmation, don't offer further work. + +## 3. Decide what it may touch + +Subagents share this machine and its filesystem. Constrain them in this order +of preference: + +1. **Read-only by construction**: `--tools read,grep,find,ls`. No shell, so it + can't run or change anything. Right for pure reading and reasoning. +2. **Read-only by instruction**: default tools (bash included), with the brief + saying everything is read-only. Needed when the subagent must run things to + verify claims — tests, interpreters, measurements. Tell it to put scratch + files under its own `$P/scratch-<name>/` and nowhere else. +3. **Writes, isolated**: give each writing subagent its own git worktree and + pass it as `--cwd`; never let two writers share a tree, and never point one + at the user's working copy. Worktrees go under + `~/x/worktrees/<repo>/<name>`. You review and bring over the result; the + subagent doesn't commit, push or post anything — say so in the brief. + +Context files (AGENTS.md/CLAUDE.md) are off by default, so the brief is the +only instruction a subagent gets. Pass `--context-files` when it works inside +a repo whose conventions it must follow. + +## 4. Launch + +Always in the background, so you can keep working: + +```bash +"$SK/run-agent.sh" "$P" <provider> "<model>" <name> > "$P/out/_<name>.log" 2>&1 & +``` + +Options go before the positionals: `--brief FILE` (default `$P/brief.md`), +`--timeout SECS` (default 1800), `--thinking LEVEL` (default `high`), +`--tools LIST`, `--cwd DIR`, `--context-files`, `--deliver TEXT`. Names must be +unique within a workdir; a natural scheme is `<provider-abbrev>-<model>` for a +fan-out and `<part>` for a split. + +A fan-out of one brief to three models: + +```bash +"$SK/run-agent.sh" "$P" fireworks "accounts/fireworks/models/kimi-k3" fw-kimi-k3 > "$P/out/_fw-kimi-k3.log" 2>&1 & +"$SK/run-agent.sh" "$P" openrouter "z-ai/glm-5.3" or-glm-5.3 > "$P/out/_or-glm-5.3.log" 2>&1 & +"$SK/run-agent.sh" "$P" openrouter "google/gemini-3.7-flash" or-gemini > "$P/out/_or-gemini.log" 2>&1 & +``` + +A split, each part with its own brief: + +```bash +for part in parser cli docs; do + "$SK/run-agent.sh" --brief "$P/brief-$part.md" "$P" openrouter "z-ai/glm-5.3" "$part" \ + > "$P/out/_$part.log" 2>&1 & +done +``` + +## 5. Monitor — and work meanwhile + +Expect minutes to tens of minutes per run. Check progress with: + +```bash +"$SK/status.sh" "$P" # once: running names, then the table so far +"$SK/status.sh" --wait 60 "$P" # block until all are done (exit 0) +``` + +`status.sh` exits 3 while anything is still running, and also prints free disk +space — the event streams are big while runs are in flight. + +**Don't sit on the wait loop.** Launching early is only worth it if you use the +time: do your own investigation of the same question, build whatever harness +you'll need to check the answers, establish baselines. You need your own +evidence in §7 regardless. When you do block, give the shell call a timeout +and loop, rather than one unbounded wait. + +## 6. Handle failures + +A run succeeded if `$P/out/<name>.md` is non-empty and (when you asked for one) +ends with the fixed final line. Exit 124 is the timeout firing. `<name>.err` +and `<name>.jsonl` say what happened otherwise. + +Retry a failure once, under the same name: `run-agent.sh` moves the failed +run's files to `<name>.attempt<N>.*` first, cost included. If it fails twice, +report it with its cost and carry on — one dead provider must not sink the job. +For a split, a failed part is a gap you either fill yourself or report. + +## 7. Verify, then use + +What comes back is raw material, not results: + +- **Reproduce every substantive claim** before passing it on. Models + hallucinate line numbers, APIs, file contents and measurements, with + confidence. +- **Disagreement is where the answer is.** When answers conflict, run the + experiment that decides it and say who was right. +- **Consensus is weak evidence.** Several models agreeing can all be making the + same mistake, or testing the same wrong version. One claim you confirmed + outranks four you didn't. +- **Check what was actually tested.** Watch for a claim true of a variant but + not of the thing written — a subagent "verifying" code it silently fixed + first. +- **Distrust your own harness too.** A test extracted to run standalone can + fail for reasons of its own (a missing import, a swallowed exception) that + look exactly like the bug under discussion. Check the baseline passes what it + should. +- **Written work**, from a worktree: read the whole diff and run the tests + yourself before bringing any of it over. Never merge it unseen. + +Say what you dropped and why, alongside what you kept — that's what makes the +kept part trustworthy. + +## 8. Report cost + +Every report that used subagents ends with the cost table; the user should +never have to ask. `status.sh` prints it: per run, from the provider's own +per-message `usage.cost` — billed figures, not estimates — failed attempts as +their own rows, then the total. A timed-out run bills for every token it +generated and can be the most expensive line. When cost per useful result +varies a lot across models, say so: it informs who to use next time. + +## 9. Keep what matters + +`$P` won't survive the session. If the answers matter, copy the `.md` outputs, +the cost files, and any verification scripts somewhere durable in the project, +and link those instead of `$P`. diff --git a/skills/pi-subagent/run-agent.sh b/skills/pi-subagent/run-agent.sh new file mode 100755 index 0000000..6312eb3 --- /dev/null +++ b/skills/pi-subagent/run-agent.sh @@ -0,0 +1,189 @@ +#!/bin/bash +# Run one pi subagent, headless, to completion. +# +# run-agent.sh [options] <workdir> <provider> <model> <name> +# +# Options: +# --brief FILE task brief to hand the subagent (default: <workdir>/brief.md) +# --timeout SECS hard wall-clock limit (default: 1800) +# --thinking LEVEL off|minimal|low|medium|high|xhigh|max (default: high) +# --tools LIST pi tool allowlist, e.g. read,grep,find,ls for a subagent +# that must not run or change anything (default: pi's own) +# --cwd DIR directory the subagent runs in (default: the current one) +# --context-files let pi load AGENTS.md/CLAUDE.md from --cwd (default: off) +# --deliver TEXT what the final answer must be (default: "your complete +# result, as markdown") +# +# Writes, under <workdir>/out/: +# <name>.md the subagent's final answer: text of its last message +# <name>.cost.json tokens, seconds, exit status, USD cost from the provider +# <name>.jsonl the event stream without per-token deltas, for debugging +# <name>.err pi's stderr +# <name>.running exists only while the run is in flight; holds the PID +# +# A previous run under the same name is never overwritten: its files are moved +# to <name>.attempt<N>.* first, so a retry keeps the bill of the failure. +# +# Exit status is pi's, except that a run pi calls a success is turned into 1 +# when its answer is empty or its last message ended in a provider error: pi +# exits 0 for both (an invalid API key, for one), and the cost file records +# the stop reason and error. 124 means the timeout fired -- the run is a +# write-off, but it still cost money; the cost file is written either way. +# +# Six mechanics here were each established the hard way. Do not "simplify" +# them away: +# +# 1. `--offline` is required. pi's startup model-catalog fetch intermittently +# hangs forever; --offline skips it. Symptom without it: the process +# produces no output at all, not even the session line, and never exits. +# +# 2. pi's stdout must be redirected to a FILE, never piped. A pipeline +# deadlocks: `timeout` kills pi, but the downstream reader never sees EOF +# and hangs past its own timeout. This is why filtering happens after the +# run, not inline. +# +# 3. The raw stream is filtered and deleted at once. Older pi re-emitted the +# entire message on every token delta, so it grew quadratically -- 790 MB +# and 3.1 GB on real runs. pi 1.x documents message_update as delta-only, +# but the stream is still mostly deltas nobody reads. Budget generous +# scratch per concurrent subagent until that is confirmed on the version +# in use. +# +# 4. Costs must be summed over every assistant message, not read off the last +# one. Each message carries only its own usage. +# +# 5. pi's stdin must be /dev/null. In -p mode pi also reads stdin when it is +# not a terminal, and waits for EOF. Started from a shell whose stdin never +# closes -- an agent's tool shell, a backgrounded job -- it hangs forever +# with no output at all, which looks exactly like the hang in (1). +# +# 6. Byte counts are stripped of whitespace before jq reads them. macOS `wc -c` +# pads its output (" 6"), jq's tonumber rejects that, and the whole +# cost file silently came out empty. +# +# `timeout` is GNU coreutils and macOS has none unless coreutils is installed, +# where it may be `gtimeout`. Without either, perl's alarm does the same job: it +# survives exec, and SIGALRM ends pi. That exit (142) is reported as 124. +# +# Extensions and skills are always off (-ne -ns): an extension can block on UI +# that a headless run has no way to answer, and the brief should say everything +# the subagent needs to know. + +set -u + +brief=""; limit=1800; thinking=high; tools=""; cwd=""; context="" +deliver="your complete result, as markdown" +while [[ $# -gt 0 ]] +do + case "$1" in + --brief) brief="$2"; shift 2 ;; + --timeout) limit="$2"; shift 2 ;; + --thinking) thinking="$2"; shift 2 ;; + --tools) tools="$2"; shift 2 ;; + --cwd) cwd="$2"; shift 2 ;; + --context-files) context=y; shift ;; + --deliver) deliver="$2"; shift 2 ;; + -h|--help) sed -n '2,/^$/p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + --) shift; break ;; + -*) echo "run-agent.sh: unknown option $1" >&2; exit 2 ;; + *) break ;; + esac +done + +P="${1:?workdir}"; provider="${2:?provider}"; model="${3:?model}"; name="${4:?name}" +P="$(cd "$P" && pwd)" || exit 2 +brief="${brief:-$P/brief.md}" +[[ "$brief" == /* ]] || brief="$PWD/$brief" +[[ -s "$brief" ]] || { echo "run-agent.sh: no brief at $brief" >&2; exit 2; } + +out="$P/out" +mkdir -p "$out" + +# Keep any earlier attempt, cost file included. +if [[ -e "$out/$name.cost.json" || -e "$out/$name.md" || -e "$out/$name.err" ]] +then + n=1 + while [[ -e "$out/$name.attempt$n.cost.json" || -e "$out/$name.attempt$n.err" ]] + do + n=$((n + 1)) + done + for ext in md cost.json jsonl err + do + [[ -e "$out/$name.$ext" ]] && mv "$out/$name.$ext" "$out/$name.attempt$n.$ext" + done + if [[ -e "$out/$name.attempt$n.cost.json" ]] + then + tmp="$(mktemp)" + jq --arg a "$name.attempt$n" '.name = $a' "$out/$name.attempt$n.cost.json" > "$tmp" \ + && mv "$tmp" "$out/$name.attempt$n.cost.json" + fi +fi + +args=(--provider "$provider" --model "$model" + -p --no-session -ne -ns --offline + --thinking "$thinking" --mode json) +[[ -n "$context" ]] || args+=(-nc) +[[ -n "$tools" ]] && args+=(--tools "$tools") + +if command -v timeout > /dev/null; then limiter=(timeout "$limit") +elif command -v gtimeout > /dev/null; then limiter=(gtimeout "$limit") +else limiter=(perl -e 'alarm shift; exec @ARGV or die "exec: $!\n"' "$limit") +fi + +echo $$ > "$out/$name.running" +trap 'rm -f "$out/$name.running"' EXIT +start=$SECONDS + +( + if [[ -n "$cwd" ]]; then cd "$cwd" || exit 2; fi + exec "${limiter[@]}" pi "${args[@]}" \ + "Read the file $brief and do exactly what it says. Deliver $deliver as your final answer." \ + < /dev/null > "$out/$name.raw" 2> "$out/$name.err" +) +rc=$? +[[ $rc -eq 142 && "${limiter[0]}" == perl ]] && rc=124 +elapsed=$((SECONDS - start)) + +raw_bytes=$(wc -c < "$out/$name.raw" | tr -d '[:space:]') +grep -v '"type":"message_update"' "$out/$name.raw" > "$out/$name.jsonl" +rm -f "$out/$name.raw" + +# The deliverable: text blocks of the last assistant message. +jq -rs ' + [.[] | select(.type=="agent_end")] | last | .messages + | [.[] | select(.role=="assistant")] | last + | [.content[] | select(.type=="text") | .text] | join("") +' "$out/$name.jsonl" > "$out/$name.md" 2>/dev/null + +# pi exits 0 when the provider refused the request, or the model said nothing. +stop=$(jq -rs '[.[] | select(.type=="message_end" and .message.role=="assistant")] + | last | .message.stopReason // "none"' "$out/$name.jsonl" 2>/dev/null) +if [[ $rc -eq 0 ]] && { [[ "$stop" == error || "$stop" == none ]] || ! grep -q '[^[:space:]]' "$out/$name.md"; } +then + rc=1 +fi + +jq -rs --arg name "$name" --arg model "$model" --arg provider "$provider" \ + --arg rc "$rc" --arg elapsed "$elapsed" --arg raw "$raw_bytes" ' + [.[] | select(.type=="message_end" and .message.role=="assistant") | .message] as $msgs + | ($msgs | last) as $last + | ($msgs | map(.usage)) + | {name: $name, provider: $provider, model: $model, exit: ($rc|tonumber), + stopReason: ($last.stopReason // null), + error: ($last.errorMessage // null | if . then .[0:500] else . end), + seconds: ($elapsed|tonumber), turns: length, + input: (map(.input) | add // 0), output: (map(.output) | add // 0), + cacheRead: (map(.cacheRead) | add // 0), + cacheWrite: (map(.cacheWrite) | add // 0), + rawStreamBytes: ($raw|tonumber), + cost: (map(.cost.total) | add // 0)} +' "$out/$name.jsonl" > "$out/$name.cost.json" 2>/dev/null + +printf '%-20s exit:%-4s stop:%-8s %5ss answer:%7s bytes cost:$%s\n' \ + "$name" "$rc" "$stop" "$elapsed" "$(wc -c < "$out/$name.md" | tr -d '[:space:]')" \ + "$(jq -r '.cost' "$out/$name.cost.json" 2>/dev/null)" +if [[ $rc -ne 0 ]] +then + jq -r '.error // empty' "$out/$name.cost.json" 2>/dev/null | head -c 300; echo +fi +exit $rc diff --git a/skills/pi-subagent/status.sh b/skills/pi-subagent/status.sh new file mode 100755 index 0000000..15a75de --- /dev/null +++ b/skills/pi-subagent/status.sh @@ -0,0 +1,85 @@ +#!/bin/bash +# Report on the subagents of one workdir: what is still running, and the +# cost table of everything that finished, failed attempts included. +# +# status.sh <workdir> report once +# status.sh --wait [secs] <workdir> report every <secs> (default 60) until +# nothing is running, then the table +# +# Exit status: 0 when nothing is running, 3 while something still is. +# +# "Running" means <workdir>/out/<name>.running exists and its PID is alive. A +# marker whose process is gone (run-agent.sh was killed) is reported as stale +# and removed. + +set -u + +wait=""; interval=60 +if [[ "${1:-}" == "--wait" ]] +then + wait=y; shift + if [[ "${1:-}" =~ ^[0-9]+$ ]]; then interval="$1"; shift; fi +fi +P="${1:?workdir}" +out="$P/out" + +running() { + local marker pid alive=() + for marker in "$out"/*.running + do + [[ -e "$marker" ]] || continue + pid="$(cat "$marker" 2>/dev/null)" + if [[ -n "$pid" ]] && kill -0 "$pid" 2>/dev/null + then + alive+=("$(basename "$marker" .running)") + else + echo "stale: $(basename "$marker" .running) (process gone)" >&2 + rm -f "$marker" + fi + done + echo "${alive[*]:-}" +} + +free_gb() { + df -k "$P" | awk 'NR==2 {printf "%.1f", $4/1048576}' +} + +if [[ -n "$wait" ]] +then + while : + do + names="$(running)" + [[ -z "$names" ]] && break + echo "$(date +%H:%M:%S) running=$(wc -w <<< "$names" | tr -d ' ') freeGB=$(free_gb): $names" + sleep "$interval" + done +fi + +names="$(running)" +if [[ -n "$names" ]] +then + echo "running: $names" +fi + +shopt -s nullglob +costs=("$out"/*.cost.json) +if [[ ${#costs[@]} -gt 0 ]] +then + { + echo -e "name\tprovider\tmodel\texit\tstop\tsecs\tturns\tin\tout\tanswer_bytes\tcost" + for f in "${costs[@]}" + do + n="$(basename "$f" .cost.json)" + bytes=0 + [[ -e "$out/$n.md" ]] && bytes="$(wc -c < "$out/$n.md" | tr -d '[:space:]')" + jq -r --arg b "$bytes" '[.name,.provider,.model,(.exit|tostring),(.stopReason // "-"),(.seconds|tostring), + (.turns|tostring),(.input|tostring),(.output|tostring),$b, + (.cost*10000|round/10000|tostring)] | @tsv' "$f" 2>/dev/null \ + || echo -e "$n\t?\t?\t?\t?\t?\t?\t?\t?\t$bytes\t(unreadable cost file)" + done | sort -t$'\t' -k11 -rn + } | column -t -s$'\t' + jq -s -r '"runs: \(length) failed: \(map(select(.exit != 0)) | length) total cost: $\(map(.cost) | add | .*10000 | round/10000)"' \ + "${costs[@]}" 2>/dev/null +fi + +[[ -z "$names" ]] || exit 3 |