Automatic retry request for all rate limit responses

We should have a global setting to automatically retry requests (with exponential back-off) that receive a rate limit response. It should be turned on by default, have a retry limit of 3-5 that is configurable and additionally be configured to be turned off altogether.

Something like:

var gcloud = require('gcloud')({ retryLimit: 5 });

and

var gcloud = require('gcloud')({ autoRetry: false });

We should be clear that this setting is only for responses related to rate limits being reached and not that it will retry every request regardless of response error.

Just look for error code 429: TooManyRequests?

The Google Cloud Storage JSON API per-project usage limit was exceeded. If your application tries to use more than its limit, additional requests will fail. You can see API usage limits on the Quotas pane in the Google Developers Console.

And 503: ServiceUnavailable?

Please reduce your request rate.

I think you may want to also look for 500.

There are inevitable non-zero intermittent 5xx errors with distributed systems like the current supported Google APIs, I would like to retry in such cases, as well as rate limited errors.

Apparently it's there's no way for me to test this locally because I never get rate limited on my machine. I have no idea why that is. I literally just made 100 buckets in ~1 second, no rate limit.

Maybe you can explicitly configure the lower user limit on your project?

@tmatsuo

@tmatsuo Any idea what this error code is? https://travis-ci.org/GoogleCloudPlatform/gcloud-node/builds/56411032

Excerpt:

Error: The service was unable to fulfill your request. Please try again.

Does it always happen?
If not, they are just intermittent errors, and they're not limited only to Pub/Sub.

What was the HTTP status BTW?

No it doesn't always happen. Just trying to track down the code because this occurred with rate limiting PR merged (checking and retrying 500, 503 and 429)