Refreshable tokens (7/7): guided end-to-end verification scripts by babakks · Pull Request #14456 · cli/cli

babakks · 2026-09-15T01:28:54Z

Part of #14449. Based on #14455 (agent-task capi).

Description

This PR adds guided, interactive end-to-end verification scripts for the refreshable (short-lived) OAuth token feature: one that drives gh commands directly, and one that exercises the git credential helper experience. They are review aids rather than merged automated tests: each walks a reviewer through the scenarios one command at a time, printing Given / When / Then, waiting for confirmation before each run, and auto-checking what it can.

Run them against a build of the stack:

# gh commands directly (pick a storage theme)
script/refreshable-token-e2e.sh --keyring
script/refreshable-token-e2e.sh --config

# git credential helper experience (run in a real terminal, not VS Code)
script/refreshable-token-gitcredential-e2e.sh

Without an option the first script prompts you to pick a storage theme. Both use ./bin/gh by default (set GH_BIN to point at another binary) and authenticate against github.com (set GH_E2E_HOST to override). They drive real gh auth commands and will clobber stored credentials for the host, so run them on a throwaway or reauthenticatable account. The git credential helper script must be run from a real external terminal: the VS Code integrated terminal injects its own credential helper that intercepts git before gh, so the script blocks there (override with GH_E2E_ALLOW_VSCODE=1 only if you know what you are doing).

To exercise the refresh conditions on demand without waiting for a real token to expire, the script forces expiry with a test-only environment hack, and it detects whether a refresh happened from gh's own output and debug logs. These test-only hacks are shown to you as short notes and are removed in the pre-merge polish PR.

How did you test this change?

The main script, script/refreshable-token-e2e.sh, walks the following Given / When / Then scenarios. Each is confirmed interactively and auto-checked where possible; a handful of visual outcomes are left for the reviewer to eyeball.

Login:

Given a clean auth state for the host
When we log in requesting a short-lived credential
Then login succeeds and reports it received a short-lived refreshable token

API transport:

Given a signed-in short-lived credential
When an ordinary command runs while the access token is expired or near expiry
Then the command succeeds and gh refreshes the token first

Given the same short-lived credential with a still-valid access token
When the same command runs
Then the command succeeds without any refresh

Given a token supplied through the GH_TOKEN environment variable
When a command runs while the token in storage is expired or near expiry
Then the command succeeds and no refresh happens, because an environment token is never refreshed

gh auth status:

Given a signed-in short-lived credential
When auth status runs while the access token is expired or near expiry
Then status refreshes the token and shows it as a short-lived refreshable credential

Given the same short-lived credential
When auth status --json hosts runs while the access token is expired or near expiry
Then status refreshes the token and the JSON carries the refreshable fields

Given the same short-lived credential with a still-valid access token
When auth status runs
Then status shows the refreshable credential without refreshing it

Given the same short-lived credential with a still-valid access token
When auth status --json hosts runs
Then the JSON carries the refreshable fields and no refresh happens

Given a signed-in short-lived credential
When auth status --no-refresh runs while the access token is expired or near expiry
Then status still shows the refreshable credential but does not refresh it

Given a signed-in short-lived credential
When auth status --json hosts --no-refresh runs while the access token is expired or near expiry
Then the JSON carries the refreshable fields and no refresh happens

Given an unrelated token supplied through the GH_TOKEN environment variable
When auth status runs
Then the active entry is reported as sourced from GH_TOKEN and used verbatim, never refreshed

Given an unrelated token supplied through the GH_TOKEN environment variable
When auth status --json hosts runs
Then the active entry is reported as sourced from GH_TOKEN with no refreshable fields

gh auth token:

Given a signed-in short-lived credential
When auth token runs while the access token is expired or near expiry
Then gh refreshes the token and prints the refreshed value

Given the same short-lived credential with a still-valid access token
When auth token runs
Then gh prints the current token without refreshing it

Given a signed-in short-lived credential
When auth token --no-refresh runs while the access token is expired or near expiry
Then gh prints the stored token as-is without refreshing it

Given a token supplied through the GH_TOKEN environment variable
When auth token runs while the token in storage is expired or near expiry
Then gh prints the environment token without refreshing it

Given a signed-in short-lived credential
When auth token --secure-storage runs while the access token is expired or near expiry
Then gh refreshes the token and prints the refreshed value

Given the same short-lived credential with a still-valid access token
When auth token --secure-storage runs
Then gh prints the current token without refreshing it

Given a signed-in short-lived credential
When auth token --secure-storage --no-refresh runs while the access token is expired or near expiry
Then no refresh happens, and with keyring storage gh prints the token straight from the keyring, while with config storage gh reports no token found because secure storage is keyring-only and the token lives in config

Given a token supplied through the GH_TOKEN environment variable
When auth token --secure-storage --hostname github.com runs while the access token is expired or near expiry
Then gh ignores GH_TOKEN, refreshes the stored credential, and prints that value

gh auth refresh:

Given a signed-in short-lived credential
When we run auth refresh --short-lived
Then the flow completes and the credential stays short-lived and refreshable

Cleanup:

Given the verification run has finished
When we choose to log out at the cleanup prompt
Then gh removes the account and its stored credential, returning to a clean state

Git credential helper:

The second script, script/refreshable-token-gitcredential-e2e.sh, verifies the same feature from git's point of view, driving gh as git's credential helper. It uses gh auth login's default storage and must be run from a real terminal outside VS Code. It walks these scenarios:

Given a clean auth state for the host, using default storage
When we log in over HTTPS requesting a short-lived credential
Then login succeeds, reports a short-lived refreshable token, and sets git protocol to https

Given a signed-in account on the host
When we run this build's auth setup-git for the host
Then git is configured to call this gh as its credential helper for the host over HTTPS

Given setup-git has run
When we read git's configured credential helper for the host
Then it points at this gh via 'auth git-credential'

Given gh is the credential helper and the access token is still valid
When git asks gh for credentials for the host
Then gh returns a working credential without refreshing it

Given gh is the credential helper and the access token is expired or near expiry
When git asks gh for credentials
Then gh refreshes the token first, then hands git a working credential

Given a refreshable short-lived credential and gh as the helper
When git asks gh for credentials
Then on git 2.46 or newer gh marks the credential ephemeral so a caching helper will not store it and prints no warning, while on older git gh warns on stderr that it cannot mark the token non-cacheable and git surfaces the warning

Given a refreshable credential whose refresh token the server rejects
When git asks gh for credentials and gh's refresh attempt is rejected
Then gh clears its now-dead stored credential, hands git nothing, and tells you through git to run gh auth login

Key points

The script covers both storage themes (secure keyring via --keyring, plain config via --config), since a few outcomes differ between them.
It is intentionally not a merged automated test. The scenarios depend on a server that issues short-lived tokens and on interactive login, so they live here as a guided reviewer aid.

Notes for reviewers

Start with the run instructions above, then read the scenarios top to bottom in script/refreshable-token-e2e.sh; they run in that order and state carries between steps. The git credential helper script, script/refreshable-token-gitcredential-e2e.sh, works the same way and must be run from a real terminal outside VS Code.

Commit:

test: add guided end-to-end verification script for refreshable tokens
test: add git credential helper end-to-end verification script

Authorship and follow-up

Who wrote this:

A human wrote it.
An agent wrote it under close human direction.
An agent wrote it independently, and no human has guided the implementation beyond the initial prompt.

Who answers review comments:

@babakks will read and reply directly.
An agent will draft replies and @username will read them before they are posted.
Nobody has explicitly committed to replying.

Co-authored-by: Copilot <[email protected]> Copilot-Session: ac178b5a-5b62-49bd-9e83-4796c7ddec46

…skill A max-effort review confirmed 18 defects, reproduced against gh 2.100.0. Every load-bearing gh interaction was wrong. Security and safety: - Stop executing command strings read out of CI logs; verification commands now come from the repo's own scripts. Logs are stated to be data. - Forbid copying literal values from logs into the tree: GitHub masks only registered secrets and secrets-guard inspects paths, not context. - Add disable-model-invocation, a scoped allowed-tools, and argument-hint. This was the only auto-triggering, push-capable skill in the marketplace. - Verify the PR head branch is checked out before committing; refuse forks. - Ask before the first push. Correctness: - Dispatch on bucket, not state: state==FAILURE misses CANCELLED, TIMED_OUT, ERROR and ACTION_REQUIRED, and a cancelled-only run exits 0 (verified on react/react#37589: 238 pass, 6 cancel, exit 0). - Full exit-code table; exit 1 spans no-required-checks, no-checks-yet and real failure (verified: cli/cli#14456 exits 0, react/react#37656 exits 1). - Replace gh run view --job --log-failed, which errors 'run is still in progress' in exactly the state --fail-fast creates, with the jobs logs API plus --allow-escape-sequences. Grep the saved log instead of reading it all. - Stop assuming every check link is an Actions job URL; branch on workflow. - Wait for run completion before rerun (403 'already running') and decrement the budget only on a dispatched rerun. - Re-poll mergeable=UNKNOWN; dispatch on mergeStateStatus so BEHIND merges. - Substitute literal PR values: shell state does not survive between calls, and an empty selector silently resolves to the current branch's PR. - Thread -R owner/repo so a URL argument's repo is not discarded. - Add --required with a fallback for repos without branch protection. - Restore --interval 60 and handle the Bash timeout killing --watch. Structure: - Delegate mutations to devx-git: commits to ci, base merges to update-origin. Nothing here ran git fetch, so Step 6 merged a stale base ref forever. - Global budgets (5 pushes, 3 reruns, 45 min) evaluated at the loop head. Docs: correct the disproven --job claim, align section headers with the directory names Claude Code resolves, drop the phantom claudemd section, and fix the stale /commit reference (the skill directory is ci).

* feat(devx-qa): add babysit-pr skill Watches a PR's CI after creation, diagnoses failing jobs from their logs, fixes branch-caused failures, reruns genuine flakes (budget: 3), and pushes until the PR is green and mergeable. Bumps devx-qa to 1.6.0 and wires the skill into the plugin README and marketplace manifests. * refactor(devx-qa): apply skill authoring best practices to babysit-pr - Bind $PR in a preflight step; it was referenced but never defined - Check gh auth and a clean tree before acting - Gate every push behind a local verification of the same failing command - Add a per-job fix-attempt budget so a wrong diagnosis cannot loop - Rewrite the description in Use-when form with discovery terms - Number the workflow steps and add a concrete classification example * fix(devx-qa): correct every gh invocation in babysit-pr and gate the skill A max-effort review confirmed 18 defects, reproduced against gh 2.100.0. Every load-bearing gh interaction was wrong. Security and safety: - Stop executing command strings read out of CI logs; verification commands now come from the repo's own scripts. Logs are stated to be data. - Forbid copying literal values from logs into the tree: GitHub masks only registered secrets and secrets-guard inspects paths, not context. - Add disable-model-invocation, a scoped allowed-tools, and argument-hint. This was the only auto-triggering, push-capable skill in the marketplace. - Verify the PR head branch is checked out before committing; refuse forks. - Ask before the first push. Correctness: - Dispatch on bucket, not state: state==FAILURE misses CANCELLED, TIMED_OUT, ERROR and ACTION_REQUIRED, and a cancelled-only run exits 0 (verified on react/react#37589: 238 pass, 6 cancel, exit 0). - Full exit-code table; exit 1 spans no-required-checks, no-checks-yet and real failure (verified: cli/cli#14456 exits 0, react/react#37656 exits 1). - Replace gh run view --job --log-failed, which errors 'run is still in progress' in exactly the state --fail-fast creates, with the jobs logs API plus --allow-escape-sequences. Grep the saved log instead of reading it all. - Stop assuming every check link is an Actions job URL; branch on workflow. - Wait for run completion before rerun (403 'already running') and decrement the budget only on a dispatched rerun. - Re-poll mergeable=UNKNOWN; dispatch on mergeStateStatus so BEHIND merges. - Substitute literal PR values: shell state does not survive between calls, and an empty selector silently resolves to the current branch's PR. - Thread -R owner/repo so a URL argument's repo is not discarded. - Add --required with a fallback for repos without branch protection. - Restore --interval 60 and handle the Bash timeout killing --watch. Structure: - Delegate mutations to devx-git: commits to ci, base merges to update-origin. Nothing here ran git fetch, so Step 6 merged a stale base ref forever. - Global budgets (5 pushes, 3 reruns, 45 min) evaluated at the loop head. Docs: correct the disproven --job claim, align section headers with the directory names Claude Code resolves, drop the phantom claudemd section, and fix the stale /commit reference (the skill directory is ci).

williammartin · 2026-10-06T11:29:01Z

+# Guided, interactive end-to-end verification for refreshable (short-lived) OAuth tokens.
+#
+# This script is a review aid, not a merged test. It walks a reviewer through the
+# scenarios in files/e2e-verification-scenarios.md one command at a time, using a


This file doesn't exist.

williammartin · 2026-10-06T11:29:41Z

+echo; divider; echo
+
+# ===========================================================================
+# NEW SYSTEM


Not really sure what this means.

williammartin · 2026-10-06T11:30:13Z

+
+label "Expected:"
+assert_err_contains "login reports a short-lived refreshable token" "refreshable token"
+manual "you selected HTTPS as the git protocol"


I don't understand what this is asking because the command is run above?

williammartin · 2026-10-06T11:46:55Z

@@ -0,0 +1,625 @@
+#!/usr/bin/env bash


This is a very cool script. I think it would be useful to get as much as possible into one or more acceptance tests?

williammartin · 2026-10-06T11:49:24Z

+confirm_run
+run_git \
+  "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" \
+  "printf 'protocol=https\\nhost=$HOST\\n' | GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill"


This step fails on git 2.46+ even though gh is doing the right thing.

git credential fill only forwards capabilities to the helper if the caller declares them on stdin. We don't, so gh never sees capability[]=authtype and takes the older-git path. On git 2.53 I get the "cannot mark the short-lived token as non-cacheable" warning, this step FAILs, and the script exits 1. The warning also shows up in the two earlier credential fill steps (they still pass because they don't check for it).

The production path is fine. I checked with a local server that returns a 401: a real git ls-remote sends capability[]=authtype to the helper without being asked, so git fetch/push get the ephemeral credential.

Here's the change I made locally to get it passing: declare the capability on git 2.46+ for every credential fill step, and assert ephemeral=1 instead of leaving it to the reviewer to eyeball.

Diff

diff --git a/script/refreshable-token-gitcredential-e2e.sh b/script/refreshable-token-gitcredential-e2e.sh index 3183950f7..54e6cd638 100755 --- a/script/refreshable-token-gitcredential-e2e.sh +++ b/script/refreshable-token-gitcredential-e2e.sh @@ -279,6 +279,14 @@ if [ -n "$GIT_VER" ]; then fi fi +# git credential fill only forwards capabilities to helpers when the caller +# declares them on stdin (real fetch/push declare authtype themselves), so +# declare authtype on git 2.46+ to exercise the same path as real git traffic. +CAPA="" +if [ "$GIT_NEW" -eq 1 ]; then + CAPA='capability[]=authtype\n' +fi + # --------------------------------------------------------------------------- # Intro # --------------------------------------------------------------------------- @@ -389,12 +397,12 @@ next_step label "Given" "gh is the credential helper and the access token is still valid" label "When" "git asks gh for credentials for https://$HOST" label "Then" "gh returns a working credential without refreshing it" -cmdline "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" +cmdline "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" note " (git prints the resolved credential, including the token.)" confirm_run run_git \ - "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" \ - "printf 'protocol=https\\nhost=$HOST\\n' | GH_DEBUG=api GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" \ + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | GH_DEBUG=api GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" label "Expected:" assert_out_contains "git received a username from the helper" "username=" @@ -405,12 +413,12 @@ next_step label "Given" "gh is the credential helper and the access token is expired or near expiry" label "When" "git asks gh for credentials" label "Then" "gh refreshes the token first, then hands git a working credential" -cmdline "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" +cmdline "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" note " (Expiry is forced here via the GH_AT_EXPIRES_IN test hack.)" confirm_run run_git \ - "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" \ - "printf 'protocol=https\\nhost=$HOST\\n' | GH_AT_EXPIRES_IN=1 GH_DEBUG=api GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" \ + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | GH_AT_EXPIRES_IN=1 GH_DEBUG=api GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" label "Expected:" assert_refresh "gh refreshed the token before handing it to git" @@ -425,16 +433,16 @@ if [ "$GIT_NEW" -eq 1 ]; then else label "Then" "gh warns on stderr that it cannot mark the token non-cacheable, and git surfaces the warning" fi -cmdline "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" +cmdline "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" confirm_run run_git \ - "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" \ - "printf 'protocol=https\\nhost=$HOST\\n' | GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" \ + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" label "Expected:" if [ "$GIT_NEW" -eq 1 ]; then assert_err_absent "no non-cacheable warning on git 2.46 or newer" "$NONCACHEABLE_WARNING" - manual "git shows authtype/ephemeral fields (gh marked the credential non-cacheable)" + assert_out_contains "gh marked the credential ephemeral (non-cacheable)" "ephemeral=1" else assert_err_contains "gh warns it cannot mark the token non-cacheable" "$NONCACHEABLE_WARNING" manual "git surfaced gh's warning above" @@ -445,7 +453,7 @@ next_step label "Given" "a refreshable credential whose refresh token the server rejects" label "When" "git asks gh for credentials and gh's refresh attempt is rejected" label "Then" "gh clears its now-dead stored credential, hands git nothing, and tells you (through git) to run gh auth login" -cmdline "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" +cmdline "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" heading " This step logs you out of $HOST in gh." note " (Access-token and refresh-token expiry are both forced via the GH_AT_EXPIRES_IN" note " and GH_RT_EXPIRES_IN test hacks, so gh attempts a refresh and the refresh is" @@ -456,8 +464,8 @@ note " gh auth login again to keep using gh for $HOST.)" note " (A non-zero exit is expected here: git gets no credential.)" confirm_run run_git \ - "printf 'protocol=https\\nhost=$HOST\\n' | git credential fill" \ - "printf 'protocol=https\\nhost=$HOST\\n' | GH_AT_EXPIRES_IN=1 GH_RT_EXPIRES_IN=1 GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | git credential fill" \ + "printf '${CAPA}protocol=https\\nhost=$HOST\\n' | GH_AT_EXPIRES_IN=1 GH_RT_EXPIRES_IN=1 GIT_TERMINAL_PROMPT=0 $GIT_NEUTRAL credential fill" label "Expected:" assert_err_contains "gh reports the token has expired" "has expired"

babakks and others added 2 commits September 15, 2026 00:55

test: add guided end-to-end verification script for refreshable tokens

1296b9d

Co-authored-by: Copilot <[email protected]> Copilot-Session: ac178b5a-5b62-49bd-9e83-4796c7ddec46

test: add git credential helper end-to-end verification script

71d80e6

Co-authored-by: Copilot <[email protected]> Copilot-Session: ac178b5a-5b62-49bd-9e83-4796c7ddec46

babakks requested a review from a team as a code owner September 15, 2026 01:28

babakks requested review from niik and removed request for a team September 15, 2026 01:28

babakks mentioned this pull request Sep 15, 2026

Refreshable (short-lived) OAuth token support #14449

Open

babakks added this pull request to stack #14457 September 15, 2026 01:31

Pierozi mentioned this pull request Sep 18, 2026

feat(devx-qa): add babysit-pr skill agentic-dev3o/devx-plugins#19

Merged

7 tasks done

suzuki-shunsuke mentioned this pull request Sep 24, 2026

ghtkn's advantages over GitHub CLI's Refreshable (short-lived) OAuth token support suzuki-shunsuke/ghtkn#651

Open

williammartin requested changes Oct 6, 2026

View reviewed changes

Navigation Menu

Repository navigation

Refreshable tokens (7/7): guided end-to-end verification scripts - #14456

Refreshable tokens (7/7): guided end-to-end verification scripts#14456
babakks wants to merge 2 commits into
babakks/refresh-token-c4-agent-task-capifrom
babakks/refresh-token-e2e

babakks commented Sep 15, 2026 •

edited

Loading

Uh oh!

williammartin Oct 6, 2026

Uh oh!

williammartin Oct 6, 2026

Uh oh!

williammartin Oct 6, 2026

Uh oh!

williammartin Oct 6, 2026

Uh oh!

williammartin Oct 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

babakks commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How did you test this change?

Key points

Notes for reviewers

Authorship and follow-up

Uh oh!

williammartin Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

williammartin Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

williammartin Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

williammartin Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

williammartin Oct 6, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

babakks commented Sep 15, 2026 •

edited

Loading