Creating a release with files sometimes fails upoading the files and tanks the release

@mislav

Describe the bug

Sometimes when using gh release create <files>..., uploading files fails (and ends up with a partial release).

Steps to reproduce the behavior

Run gh release create with files
It sometimes fails to upload one (possibly more) of the files

Expected vs actual behavior

As proposed by @mislav on #6645 , the retry mechanism for file uploads could be more robust to handle these failures as well.

However, would you say that gh release create <files>... could have a better retry mechanism built in so that uploads are retried automatically within gh and that your automation doesn't have to do it?

We already have some rudimentary retry logic, but we could expand it to also handle the types of errors you are experiencing:

cli/pkg/cmd/release/shared/upload.go

Lines 140 to 146 in 8891456

_, err := uploadAsset(httpClient, uploadURL, a)

// retry upload several times upon receiving HTTP 5xx

if err == nil || !errors.As(err, &httpError) || httpError.StatusCode < 500 || retries < maxRetries {

return err

}

retries++

time.Sleep(time.Duration(retries) * time.Second)

Ref. https://github.com/golang/net/blob/ecf7fda6a59edbedc58c1207f55285a9ad98e10a/http2/transport.go#L1130

Ref. io.ErrUnexpectedEOF https://pkg.go.dev/io#pkg-variables

Originally posted by @mislav in #6645 (comment)

Logs

Examples of failures from our CI [1]:

1. Post "https://uploads.github.com/repos/submariner-io/releases/releases/83445511/assets?label=&name=subctl-v0.14.0-rc4-windows-amd64.exe.tar.xz": http2: client connection force closed via ClientConn.Close
2. Post "https://uploads.github.com/repos/submariner-io/releases/releases/83445512/assets?label=&name=subctl-v0.14.0-rc4-linux-arm64.tar.xz": unexpected EOF

[1] https://github.com/submariner-io/releases/actions/runs/3491034412/jobs/5843200596#step:4:536

Fixed in 7caf7da

@mislav

@mislav Another example of error/failure of gh release create: Post "https://uploads.github.com/repos/busytex/busytex/releases/87511879/assets?label=&name=fonts.conf": unexpected EOF

@vadimkantorov

@vadimkantorov This happened even with the latest release?

Unlikely. I'm using apk for alpine:

    runs-on: ubuntu-20.04
    container: alpine:3.17
    steps:
      - name: Install Prerequisites
        run:  apk add --update --no-cache libnsl libnsl-dev build-base coreutils cmake git xz wget perl gperf p7zip python3 strace && ln -sf python3 /usr/bin/python

      - name: Install Hub
        run:  apk add --no-cache -X http://dl-cdn.alpinelinux.org/alpine/v3.17/testing github-cli

and gh available by default on ubuntu-20.04 proper

Another error (coming from default gh on ubuntu-20.04: HTTP 400: Bad Content-Length: (https://uploads.github.com/repos/busytex/busytex/releases/87606152/assets?label=&name=ubuntu-texlive-base.data)

I can try using the latest release. Does it exist as static binary that I can just wget on my alpine and ubuntu GH Actions jobs?

For Bad Content-Length, it could be trying to add an empty asset to the release, but it's strange to fail like so

@vadimkantorov

@vadimkantorov You can download and untar a precompiled binary for any GitHub CLI release, e.g. https://github.com/cli/cli/releases/tag/v2.21.2

I think that our Linux binaries should work on Alpine: #2556.

I think that our Linux binaries should work on Alpine: #2556.

I will check, but it would be better if there was a simple binary as an asset - that would save decompression. Probably it's quite small anyway. And it would also be better if file names / release notes were clear if the binaries is built fully statically (including musl libc) and are alpine/anylinux compatible or not.

Navigation Menu

Repository navigation

Creating a release with files sometimes fails upoading the files and tanks the release #6651

Describe the bug

Steps to reproduce the behavior

Expected vs actual behavior

Logs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	_, err := uploadAsset(httpClient, uploadURL, a)
	// retry upload several times upon receiving HTTP 5xx
	if err == nil \|\| !errors.As(err, &httpError) \|\| httpError.StatusCode < 500 \|\| retries < maxRetries {
	return err
	}
	retries++
	time.Sleep(time.Duration(retries) * time.Second)

Creating a release with files sometimes fails upoading the files and tanks the release #6651

Description

Describe the bug

Steps to reproduce the behavior

Expected vs actual behavior

Logs

Activity

mislav commented on Dec 19, 2022

vadimkantorov commented on Dec 29, 2022

mislav commented on Dec 30, 2022

vadimkantorov commented on Dec 30, 2022

vadimkantorov commented on Dec 30, 2022

mislav commented on Jan 9, 2023

vadimkantorov commented on Jan 11, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions