Repository navigation
gh repo sync not working #7574
Description
Activity
@Alizter Thanks for writing in. We have not made any changes to that command in a long while. Do you have any estimation around when this stopped working? I am inclined to think that perhaps this is a token permission issue or a setting changed on the repo in question. I am been unable to reproduce this on any of my forked repos.
Today it appears to work. I would guess then somebody in the GitHub API was tweaking some things causing my queries to fail. Nothing actionable from you guys I suppose.
Reacted by Sam CoeConsider reopening - I have been encountering this off-and-on for at least a week.
Reacted by Andrew PollockI've also been encountering this intermittently:
$ gh repo sync andrewpollock/osv.dev HTTP 404: Not Found (https://api.github.com/repos/andrewpollock/osv.dev/git/refs/heads/master)Yet if I go sync my fork via the web interface, the CLI will stop failing. It's been quite vexing.
In case this helps:
$ DEBUG=1 gh repo sync andrewpollock/osv.dev ⣾* Request at 2023-06-14 09:03:50.551435814 +1000 AEST m=+0.081041738 * Request to https://api.github.com/graphql ⣷* Request took 916.596168ms * Request at 2023-06-14 09:03:51.469220503 +1000 AEST m=+0.998826430 * Request to https://api.github.com/repos/andrewpollock/osv.dev/merge-upstream ⣻* Request took 379.934845ms * Request at 2023-06-14 09:03:51.849800312 +1000 AEST m=+1.379406238 * Request to https://api.github.com/graphql ⣟* Request took 312.245235ms * Request at 2023-06-14 09:03:52.162329298 +1000 AEST m=+1.691935223 * Request to https://api.github.com/repos/google/osv.dev/git/refs/heads/master ⣷* Request took 251.153768ms * Request at 2023-06-14 09:03:52.41393556 +1000 AEST m=+1.943541558 * Request to https://api.github.com/repos/andrewpollock/osv.dev/git/refs/heads/master ⣽* Request took 304.482351ms HTTP 404: Not Found (https://api.github.com/repos/andrewpollock/osv.dev/git/refs/heads/master)Reacted by Sam CoeAfter I've synced by the web UI:
$ DEBUG=1 gh repo sync andrewpollock/osv.dev * Request at 2023-06-14 09:05:27.150894162 +1000 AEST m=+0.065482152 * Request to https://api.github.com/graphql ⣷* Request took 919.547876ms * Request at 2023-06-14 09:05:28.071390589 +1000 AEST m=+0.985978644 * Request to https://api.github.com/repos/andrewpollock/osv.dev/merge-upstream ⣻* Request took 296.000559ms ✓ Synced the "andrewpollock:master" branch from "google:master"Reacted by Sam CoeSeeing as various people have been experiencing this issue on and off, I am going to reopen this issue. I don't think I have seen any report that makes me think this is a bug in
ghspecifically but rather a platform bug. Anyone with evidence to the contrary please let me know as my only current course is to direct this to the appropriate internal platform team.If anyone has the ability to reproduce the error and willing to share their logs from using
GH_DEBUG=api gh repo syncthat would be appreciate in trying to determine exactly what is failing. Please be mindful to redact any private data from the logs before submitting them here.I agree that this sounds like a platform bug based on the nature of the error.
2 remaining items
Hello everyone, thank you for reporting this bug and providing additional information to help out.
I believe this is a platform change, but I also believe you can resolve it through the CLI. If you aren't interested in the details, jump to the Resolution section below.
Investigation
Firstly, the
404 Not Foundonhttps://api.github.com/repos/<org>/<repo>/git/refs/heads/<branch>is a bit of a red herring. This404is a failure toPATCH, while attempting to update the branch to point at the commit ref provided in the body. I believe this is failing because the commit does not exist on the targeted repository. The reason it doesn't exist on the provided repository is due to an earlier failure on the/merge-upstreamendpoint (that is swallowed for historical reasons I don't yet understand). For example, from @andrewpollock's logs:> POST /repos/andrewpollock/osv.dev/merge-upstream HTTP/1.1 > Host: api.github.com ... { "branch": "master" } < HTTP/2.0 422 Unprocessable EntityEdit: My assumption around the
404was incorrect, see the following post for details about why it failed and why the422and404have the same cause.Why is this endpoint failing (intermittently)?
In digging into this, I found a recent change that began requiring
workflowscope on the token if any commits to be merged from upstream included workflow changes.I was able to somewhat prove this locally by forking
ocaml/dune, checking out @Alizter's commit, and seeinggh repo sync williammartin-test-org/dunefail because the next commit included a workflow change. I then reset to and pushedHEAD^1frommainandsynced successfully, seeing that the next commit didn't have a workflow change.I believe this theory is further evidenced by:
- Reports that this occurs intermittently (it would only occur if upstream commits to be merged included a workflow change)
- Reports that using
sync repobutton on the UI succeeds (which would already have the required privileges) - Reports that after using
sync repoon the UI, that the CLI works again (because themerge-upstreamno longer fails and the commit exists afterPATCHoccurs) - Adding the
workflowscope to my token allowed me to sync even with workflow changes upstream.
Resolution
To resolve this you can add the
workflowscope to your token either on initial login like so:gh auth login -s workflowor via refreshing you existing token like so:
gh auth refresh -s workflowCaveat
It's possible that this change introduced other failure cases that I don't yet have the necessary knowledge to understand but I believe that this is almost certainly the common case. If after refreshing your token you still experience errors, please leave a comment.
CLI Changes
From our point of view, we should determine how to handle this error in a more graceful manner, prompting to refresh to include the token. Unfortunately, there are a number of reasons that the API returns
422and the reasons are opaque to clients, so we may need some changes before we can support this.In the meantime, hopefully this issue serves as a place for people to find help.
Reacted by Ben MacLeod, Sebastian Poxhofer, Patrik Ragnarsson and Jake HarringtonReacted by Kayvan SylvanFurther Investigation
Following up on the previous comment I did some further digging into the relationship between the earlier
422and the later404. As it turns out, both occur for the same reason as far as I can tell (missingworkflowscope).I created a commit that changed a workflow that was not associated with a branch and tried to
PATCHupdate thehead/mainref and received a404when using a token withoutworkflowscope:
It was later successful with
workflowscope:
What's happening here is that as described in the oauth scopes doc, the
workflowscope is needed for any workflow file change that isn't in a branch on the repository already. When we attempt to update theref, GitHub is rejecting this as "you don't have permission to point this branch at something that contains a workflow file change" and that's manifesting as a404.I think that ties up the loose ends!
Reacted by Kayvan Sylvan@williammartin - Thank you for the detailed explanation!
The API team have shipped an improved error message for the
/merge-upstreamendpoint:* Request to https://api.github.com/repos/williammartin-test-org/dune/merge-upstream > POST /repos/williammartin-test-org/dune/merge-upstream HTTP/1.1 > Host: api.github.com > Accept: application/vnd.github.merge-info-preview+json, application/vnd.github.nebula-preview > Authorization: token ████████████████████ > Content-Length: 18 > Content-Type: application/json; charset=utf-8 > Time-Zone: Europe/Amsterdam > User-Agent: GitHub CLI 2.30.0 { "branch": "main" } ⣯< HTTP/2.0 422 Unprocessable Entity < Access-Control-Allow-Origin: * < Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset < Content-Length: 245 < Content-Security-Policy: default-src 'none' < Content-Type: application/json; charset=utf-8 < Date: Wed, 21 Jun 2023 11:58:36 GMT < Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin < Server: GitHub.com < Strict-Transport-Security: max-age=31536000; includeSubdomains; preload < Vary: Accept-Encoding, Accept, X-Requested-With < X-Accepted-Oauth-Scopes: < X-Content-Type-Options: nosniff < X-Frame-Options: deny < X-Github-Api-Version-Selected: 2022-11-28 < X-Github-Media-Type: github.merge-info-preview; param=nebula-preview; format=json < X-Github-Request-Id: ECA3:30A7:51E59B8:52A51FF:6492E5EC < X-Oauth-Client-Id: 178c6fc778ccc68e1d6a < X-Oauth-Scopes: admin:public_key, gist, read:org, repo < X-Ratelimit-Limit: 15000 < X-Ratelimit-Remaining: 14989 < X-Ratelimit-Reset: 1687351152 < X-Ratelimit-Resource: core < X-Ratelimit-Used: 11 < X-Xss-Protection: 0 { "message": "refusing to allow an OAuth App to create or update workflow `.github/workflows/bench.yml` without `workflow` scope", "documentation_url": "https://docs.github.com/rest/branches/branches#sync-a-fork-branch-with-the-upstream-repository" }However, they have not yet made a change to the
PATCH /repos/<ORG>/<REPO>/git/refs/heads/mainendpoint.Unfortunately, with our current strategy for trying
/merge-upstreamand then falling back to therefupdate endpoint, we don't expose the original useful error. However, since the missing scope is unrecoverable between attempts, this should give us enough information to bail out early.Resolution
To resolve this you can add the
workflowscope to your token either on initial login like so:gh auth login -s workflowor via refreshing you existing token like so:
gh auth refresh -s workflowCaveat
It's possible that this change introduced other failure cases that I don't yet have the necessary knowledge to understand but I believe that this is almost certainly the common case. If after refreshing your token you still experience errors, please leave a comment.
Is this incompatible with use of
ghfrom GitHub Actions? From what I understand,workflowis an OAuth app scope, not a permission available to the GITHUB_TOKEN.I use
gh repo syncin a workflow to keep a fork in sync with its parent once a day, which started failing intermittently around the same time as this issue. I tried addinggh auth refresh -s workflowto the workflow, but that seems geared towards an interactive login. Instead workflows are encouraged to login a different way:To use gh in GitHub Actions, add GH_TOKEN: ${{ github.token }} to "env".
As a test, I bumped the permissions of my workflow to
permissions: write-alland upstream workflow changes continue to prevent syncs.EDIT: A fine-grained PAT with
contentsandworkflowspermissions does work, but expires in a year.@spencerschrock Sounds like a platform bug that
permissions: write-alldoes not allow syncing. I can ask around internally about it.Update, I heard back from the internal API team and they explained that this behavior is a security feature and is not a bug. So unfortunately the only work around is to generate a new token, perhaps with this, that has the correct scopes.
Reacted by Spencer SchrockI am seeing the same issue after a commit that introduced a workflow file change.
The error still only shows 404 which is perplexing.Here is the full log with the debug=api
https://pastebin.com/45UBLTUiI am seeing the same issue after a commit that introduced a workflow file change. The error still only shows 404 which is perplexing.
Here is the full log with the debug=api https://pastebin.com/45UBLTUi
Same here. Should a new issue be made?
@fzakaria @Kamayana Can you add the workflow scope with
gh auth refresh -s workflowand tell me it's still an issue?@fzakaria I can see this in your
GH_DEBUG=apioutput:{ "message": "refusing to allow an OAuth App to create or update workflow `.github/workflows/publishWheelRelease.yml` without `workflow` scope", "documentation_url": "https://docs.github.com/rest/branches/branches#sync-a-fork-branch-with-the-upstream-repository" }Additionally, I can see that you are using and old version of
gh, and #7612 made this error more obvious. I think it went in in 2.32 and you are using 2.30Yes it worked with that workfow scope. Thank you for looking into it.
I'll update my
ghversion; the discrepancy between the button on the UI and the default on the client still feels like a gap FWIW>I'll update my gh version; the discrepancy between the button on the UI and the default on the client still feels like a gap FWIW
Yeh it's a tricky balance of asking for the minimal set of scopes required for most operations and providing useful errors when required vs just asking for everything. I'm not sure there is a right answer here, but your feedback is useful in helping us figure it out, thanks!
I also faced this issue. Strangely, even after #7612 was merged, I still get the misleading 404 not found error.
$ gh repo sync terraform-linters/magic-modules HTTP 404: Not Found (https://api.github.com/repos/GoogleCloudPlatform/magic-modules/git/refs/heads/tflint_provider)
https://github.com/terraform-linters/magic-modules/actions/runs/13742985933/job/38434464702
After some digging, I found that the regexp introduced in #7612 doesn't work with GitHub App token error messages. The message looks like this and does not match the "workflow scope" (should be "workflows permission").
{ "message": "refusing to allow a GitHub App to create or update workflow `.github/workflows/teamcity-pr-checks.yml` without `workflows` permission", "documentation_url": "https://docs.github.com/rest/branches/branches#sync-a-fork-branch-with-the-upstream-repository", "status": "422" }https://github.com/terraform-linters/magic-modules/actions/runs/13745172118/job/38439197950
@williammartin Would it be ok to open a PR to fix this issue? If so, I can submit a PR with a30afcf in it.
Describe the bug
when the repo doesn't need syncing, the command succeeds. Clicking this URL also correctly fetches the json.
NixOS
Steps to reproduce the behavior
gh repo sync user/repoExpected vs actual behavior
It used to work and sync the repo.