Skip to content

Creating a release with files sometimes fails upoading the files and tanks the release #6651

Closed
Closed
Creating a release with files sometimes fails upoading the files and tanks the release#6651
Labels
bugSomething isn't workingpriority-3Affects a small number of users or is largely cosmetic

Description

@mkolesnik
mkolesnik
opened on Nov 22, 2022

Describe the bug

Sometimes when using gh release create <files>..., uploading files fails (and ends up with a partial release).

Steps to reproduce the behavior

  1. Run gh release create with files
  2. It sometimes fails to upload one (possibly more) of the files

Expected vs actual behavior

As proposed by @mislav on #6645 , the retry mechanism for file uploads could be more robust to handle these failures as well.

However, would you say that gh release create <files>... could have a better retry mechanism built in so that uploads are retried automatically within gh and that your automation doesn't have to do it?

We already have some rudimentary retry logic, but we could expand it to also handle the types of errors you are experiencing:

cli/pkg/cmd/release/shared/upload.go

Lines 140 to 146 in 8891456

_, err := uploadAsset(httpClient, uploadURL, a)
// retry upload several times upon receiving HTTP 5xx
if err == nil || !errors.As(err, &httpError) || httpError.StatusCode < 500 || retries < maxRetries {
return err
}
retries++
time.Sleep(time.Duration(retries) * time.Second)

Ref. https://github.com/golang/net/blob/ecf7fda6a59edbedc58c1207f55285a9ad98e10a/http2/transport.go#L1130

Ref. io.ErrUnexpectedEOF https://pkg.go.dev/io#pkg-variables

Originally posted by @mislav in #6645 (comment)

Logs

Examples of failures from our CI [1]:

1. Post "https://uploads.github.com/repos/submariner-io/releases/releases/83445511/assets?label=&name=subctl-v0.14.0-rc4-windows-amd64.exe.tar.xz": http2: client connection force closed via ClientConn.Close
2. Post "https://uploads.github.com/repos/submariner-io/releases/releases/83445512/assets?label=&name=subctl-v0.14.0-rc4-linux-arm64.tar.xz": unexpected EOF

[1] https://github.com/submariner-io/releases/actions/runs/3491034412/jobs/5843200596#step:4:536

Activity

  1. mkolesnik
    added
    bugSomething isn't working
    on Nov 22, 2022
  2. cliAutomation
    added
    needs-triageneeds to be reviewed
    on Nov 22, 2022
  3. mislav
    added
    priority-3Affects a small number of users or is largely cosmetic
    and removed
    needs-triageneeds to be reviewed
    on Nov 23, 2022

  4. mislav commented on Dec 19, 2022

    @mislav
    mislav
    on Dec 19, 2022
    Contributor

    Fixed in 7caf7da

  5. mislav
    closed this as completedon Dec 19, 2022

  6. vadimkantorov commented on Dec 29, 2022

    @vadimkantorov
    vadimkantorov
    on Dec 29, 2022

    @mislav Another example of error/failure of gh release create: Post "https://uploads.github.com/repos/busytex/busytex/releases/87511879/assets?label=&name=fonts.conf": unexpected EOF

  7. mislav commented on Dec 30, 2022

    @mislav
    mislav
    on Dec 30, 2022
    Contributor

    @vadimkantorov This happened even with the latest release?

  8. vadimkantorov commented on Dec 30, 2022

    @vadimkantorov
    vadimkantorov
    on Dec 30, 2022

    Unlikely. I'm using apk for alpine:

        runs-on: ubuntu-20.04
    container: alpine:3.17
    steps:
      - name: Install Prerequisites
        run:  apk add --update --no-cache libnsl libnsl-dev build-base coreutils cmake git xz wget perl gperf p7zip python3 strace && ln -sf python3 /usr/bin/python

      - name: Install Hub
        run:  apk add --no-cache -X http://dl-cdn.alpinelinux.org/alpine/v3.17/testing github-cli

    and gh available by default on ubuntu-20.04 proper

    Another error (coming from default gh on ubuntu-20.04: HTTP 400: Bad Content-Length: (https://uploads.github.com/repos/busytex/busytex/releases/87606152/assets?label=&name=ubuntu-texlive-base.data)

  9. vadimkantorov commented on Dec 30, 2022

    @vadimkantorov
    vadimkantorov
    on Dec 30, 2022

    I can try using the latest release. Does it exist as static binary that I can just wget on my alpine and ubuntu GH Actions jobs?

    For Bad Content-Length, it could be trying to add an empty asset to the release, but it's strange to fail like so

  10. mislav commented on Jan 9, 2023

    @mislav
    mislav
    on Jan 9, 2023
    Contributor

    @vadimkantorov You can download and untar a precompiled binary for any GitHub CLI release, e.g. https://github.com/cli/cli/releases/tag/v2.21.2

    I think that our Linux binaries should work on Alpine: #2556.

  11. vadimkantorov commented on Jan 11, 2023

    @vadimkantorov
    vadimkantorov
    on Jan 11, 2023

    I think that our Linux binaries should work on Alpine: #2556.

    I will check, but it would be better if there was a simple binary as an asset - that would save decompression. Probably it's quite small anyway. And it would also be better if file names / release notes were clear if the binaries is built fully statically (including musl libc) and are alpine/anylinux compatible or not.

    image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority-3Affects a small number of users or is largely cosmetic

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions