Skip to content

Frequent Code 500 "Backend Error" #1221

Description

@paulshapiro

I've been doing many thousands of uploads of images to Google Cloud Storage and have just now started to get elevated rates of 500 responses (message of "Backend Error". Not sure what's going on. I keep re-running the script and seems to keep failing. It's not clear to me if there's an issue with a specific file I'm attempting to upload (I haven't isolated it to a single file), etc., and I haven't taken the time to implement exponential backoff retry, etc.. Any guidance on how to resolve? Many thanks,
–Paul

Activity

  1. paulshapiro commented on Apr 10, 2016

    @paulshapiro
    Author

    Quick update; I have a little shell script set up which re-runs the upload script, and it keeps snagging very early on without making much progress any more. Could be some issue on the G Cloud server?

  2. stephenplusplus commented on Apr 10, 2016

    @stephenplusplus
    Contributor

    Thanks for reporting. A sample script to reproduce would be helpful. I'll try to throw something together as soon as I can, but feel free to beat me to it.

    Is this still an issue on any previous version of gcloud?

    It might be worth trying disabling resumable uploads and validation during the upload.

  3. paulshapiro commented on Apr 10, 2016

    @paulshapiro
    Author

    I'd had validation disabled but hadn't had resumable actually set to false -- looks like that was the ticket! By the way, it was indeed failing on the same image over and over again. I was doing an http.get(…) piping the response directly to the gcloud write stream. The get URL was 'http://www.moma.org/media/W1siZiIsIjExNTEzIl0sWyJwIiwiY29udmVydCIsIi1yZXNpemUgMjAwMHgyMDAwXHUwMDNlIl1d.jpg?sha=ffa3e3b501ece014' (That URL may expire for you though due to the sha)

    My package.json specifies a gcloud version of ^0.30.2.

    Funny enough, as I write this, it failed on the following URL as well, although it looks like on a retry it started working again. Maybe that's what resumable fixed - starting from scratch on a re-upload attempt?

    http://www.moma.org/media/W1siZiIsIjIxMDE5MSJdLFsicCIsImNvbnZlcnQiLCItcmVzaXplIDIwMDB4MjAwMFx1MDAzZSJdXQ.jpg?sha=126f83c5be2e910c

    Many thanks

  4. stephenplusplus commented on Apr 11, 2016

    @stephenplusplus
    Contributor

    Yeah, the way we support resumable uploads can cause some trouble in certain scenarios. We persist a file in ~/.config that records metadata about each resumable upload that is started.

    Most of the time, everything just works, but in cases where the same file is being written to frequently, the chance for a mismatch in state between the Storage API and gcloud-node goes up. This could confuse the Storage API, making it return that not-very-helpful "Backend Error".

    Turning off resumable basically makes the gcloud-node stream just a throughput. The Storage API will treat every upload as a new object and if anything goes wrong, you should get a more meaningful error.

    Glad that was the fix in this case! Thanks for reporting and let us know again if resumable uploads are creating more trouble.

  5. Kazeki1 commented on Jun 8, 2018

    @Kazeki1

    In my case I am using angular-in-memory-web-api and after npm install --save-dev angular-in-memory-web-api (default is v. 0.6.0) project generates Code 500 Backend error. When I revert to version 0.5.1 or 0.5.4 and error is resolved.

  6. added a commit that references this issue on Jul 23, 2025
  7. added a commit that references this issue on Feb 5, 2026
  8. added a commit that references this issue on Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

api: storageIssues related to the Cloud Storage API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions