Skip to content

feat(devx-qa): add babysit-pr skill - #19

Merged
Pierozi merged 3 commits into
mainfrom
feat/babysit-pr-skill
Sep 18, 2026
Merged

Pierozi merged 3 commits into
mainfrom
feat/babysit-pr-skill

Conversation

@Pierozi

@Pierozi Pierozi commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Adds a babysit-pr skill so a PR's failing CI can be diagnosed and remediated without re-explaining the loop each time. Distilled from codex's babysit-pr, which is ~16KB of instruction the model largely already knows — this keeps only what isn't in the weights: the gh mechanics, the branch-caused vs. infrastructure classification, and hard budgets on autonomous pushes.
  • The skill watches, diagnoses, and classifies; it does not own git mutations. Commits go through devx-git:ci and base-branch merges through devx-git:update-origin, so the push guardrails live in one place instead of being re-implemented without them.
  • Bumps devx-qa to 1.6.0 and brings the docs back in line with the directory names Claude Code actually resolves.

Review

A /code-review max pass found 18 defects, reproduced live against gh 2.100.0. All 18 are fixed in 08cbd16, and each gh invocation in the file was re-verified against real repos before being written.

Security and safety

  • The skill instructed the agent to run a command string read out of an untrusted CI log. Verification commands now come from the repo's own scripts, and log content is stated to be data, never instructions.
  • Nothing forbade copying values out of a log into the tree. GitHub masks only registered secrets, and secrets-guard inspects file paths and command strings — not context. Now explicitly forbidden, and logs are grepped rather than read whole.
  • This was the only auto-triggering, push-capable skill in the marketplace, with no tool ceiling. Now disable-model-invocation: true, a scoped allowed-tools, and confirmation before the first push.
  • Nothing verified the checked-out branch was the PR head, so commits could land anywhere. Preflight now compares headRefName, checks out the PR, and refuses fork PRs.

Correctness — verified, not assumed

  • select(.state=="FAILURE") missed CANCELLED, TIMED_OUT, ERROR, and ACTION_REQUIRED. Worse, a cancelled-only run exits 0: facebook/react#37589 reports 238 pass / 6 cancel and gh pr checks exits 0. Now dispatches on bucket.
  • "Any other exit code = a check failed" conflated four states. Verified: cli/cli#14456 exits 0, facebook/react#37656 exits 1 with no required checks reported. Replaced with a full exit-code table plus message matching.
  • gh run view --job <id> --log-failed errors run is still in progress — exactly the state --fail-fast creates — so no matrix failure was diagnosable. Replaced with gh api .../actions/jobs/<id>/logs --allow-escape-sequences, which serves a job's log mid-run. The old rationale for --job was backwards and is gone.
  • gh run rerun --failed returns 403 This workflow is already running on an in-progress run, and the budget was decremented for the refused attempt. Now waits for status: completed and only counts a rerun that dispatched.
  • mergeable: UNKNOWN — the normal value seconds after a push — was reported as ready, and BEHIND never triggered the merge written one line below it. Now re-polls, then dispatches on mergeStateStatus.
  • PR=$(...) was set in one Bash call and read in three others. Shell state does not persist, and an empty selector does not error — gh silently falls back to the current branch's PR. Literal values are now substituted, and -R <owner>/<repo> is threaded through so a URL argument's repo isn't discarded.
  • ~14% of check links aren't /actions/runs/<id>/job/<id> — external CI, CLA bots, other repos. Now branches on the workflow field and surfaces non-Actions checks as blockers.
  • --required was never passed, so advisory checks entered the fix loop forever; it also exits 1 on repos with no branch protection, which is why it needed the exit-code table first.
  • --watch outlived the Bash tool's timeout with no arm for it; --interval 60 restored.

Structure

  • Nothing in the skill ran git fetch, so Step 6 merged a stale base ref and looped forever. Delegated to devx-git:update-origin, which fetches, takes a backup branch, and handles a rejected push.
  • Budgets keyed on job name and were decremented only on the flake path, so a cascade of different jobs never tripped them, and the stop conditions were orphaned — no step consulted them. Now global caps (5 pushes, 3 reruns, 45 min, 2 attempts per error signature) evaluated at the loop head.
  • A bare git push violated update-origin Guardrail 6, the fix shipped in 037c512. Now git push origin HEAD with the rejection recovery path.

Docs — corrected the --job claim the review disproved, aligned ### fixing-skills / ### fixing-react-antipatterns with their directory names, removed a ### claudemd section for a skill that does not exist, and fixed the stale /commit reference (the directory is ci).

Test plan

  • Every gh invocation verified against live repos (cli/cli, facebook/react, nodejs/node, microsoft/vscode) on gh 2.100.0
  • Exit codes confirmed: 0 all-pass, 1 no-required-checks, 8 pending, 4 auth
  • Confirmed the bucket filter catches the cancelled-only false green
  • Confirmed the jobs-logs API serves a log mid-run and --allow-escape-sequences is required
  • Confirmed non-Actions checks carry no workflow field
  • Frontmatter within limits: description 506/1024, body 231/500 lines
  • End-to-end run against a real failing PR with a multi-job matrix

Watches a PR's CI after creation, diagnoses failing jobs from their logs,
fixes branch-caused failures, reruns genuine flakes (budget: 3), and pushes
until the PR is green and mergeable. Bumps devx-qa to 1.6.0 and wires the
skill into the plugin README and marketplace manifests.
- Bind $PR in a preflight step; it was referenced but never defined
- Check gh auth and a clean tree before acting
- Gate every push behind a local verification of the same failing command
- Add a per-job fix-attempt budget so a wrong diagnosis cannot loop
- Rewrite the description in Use-when form with discovery terms
- Number the workflow steps and add a concrete classification example
…skill

A max-effort review confirmed 18 defects, reproduced against gh 2.100.0.
Every load-bearing gh interaction was wrong.

Security and safety:
- Stop executing command strings read out of CI logs; verification commands
  now come from the repo's own scripts. Logs are stated to be data.
- Forbid copying literal values from logs into the tree: GitHub masks only
  registered secrets and secrets-guard inspects paths, not context.
- Add disable-model-invocation, a scoped allowed-tools, and argument-hint.
  This was the only auto-triggering, push-capable skill in the marketplace.
- Verify the PR head branch is checked out before committing; refuse forks.
- Ask before the first push.

Correctness:
- Dispatch on bucket, not state: state==FAILURE misses CANCELLED, TIMED_OUT,
  ERROR and ACTION_REQUIRED, and a cancelled-only run exits 0 (verified on
  react/react#37589: 238 pass, 6 cancel, exit 0).
- Full exit-code table; exit 1 spans no-required-checks, no-checks-yet and
  real failure (verified: cli/cli#14456 exits 0, react/react#37656 exits 1).
- Replace gh run view --job --log-failed, which errors 'run is still in
  progress' in exactly the state --fail-fast creates, with the jobs logs API
  plus --allow-escape-sequences. Grep the saved log instead of reading it all.
- Stop assuming every check link is an Actions job URL; branch on workflow.
- Wait for run completion before rerun (403 'already running') and decrement
  the budget only on a dispatched rerun.
- Re-poll mergeable=UNKNOWN; dispatch on mergeStateStatus so BEHIND merges.
- Substitute literal PR values: shell state does not survive between calls,
  and an empty selector silently resolves to the current branch's PR.
- Thread -R owner/repo so a URL argument's repo is not discarded.
- Add --required with a fallback for repos without branch protection.
- Restore --interval 60 and handle the Bash timeout killing --watch.

Structure:
- Delegate mutations to devx-git: commits to ci, base merges to update-origin.
  Nothing here ran git fetch, so Step 6 merged a stale base ref forever.
- Global budgets (5 pushes, 3 reruns, 45 min) evaluated at the loop head.

Docs: correct the disproven --job claim, align section headers with the
directory names Claude Code resolves, drop the phantom claudemd section, and
fix the stale /commit reference (the skill directory is ci).
@Pierozi
Pierozi merged commit da7787e into main Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant