Raise a helpful error if projectID is missing when making a request?

Right now, if I run the following code on an unconfigured machine, I get a scary back-end error...

// (We're not running on App Engine or Compute Engine).
var gcloud = require('gcloud');
var dataset = gcloud.datastore.dataset();  // <-- Here is the issue (projectId === undefined)

dataset.get(datastore.key(['Person', 1], function(result) {
  console.log(result);
});

Seems like if we can't auto-discover the project ID or credentials, we fail with a very long error.

Could we update things so that we raise a specific exception saying "You tried to make a request with a null project ID, meaning we couldn't figure out the project ID and you also didn't tell us the project ID..."

@rdayal

It might also be useful to automatically pull the project ID from the one set in the Cloud SDK (see ~/.config/gcloud/properties which sometimes has a default project ID in config-file format). This is where the Cloud SDK stores the result of gcloud config set project <projectId> -- right @rdayal?

I think it would be nice to fail sooner at the var dataset = gcloud.datastore.dataset() point, because by then, we'll know "whatever API request you try, it ain't gonna work."

OK so you're saying at the time you create a Dataset, if we couldn't figure out the project ID by then, we should raise an exception ? SGTM.

The actual API call that was made in your example was:

{ method: 'POST',
  uri: 'https://www.googleapis.com/datastore/v1beta2/datasets/{projectId}/lookup',
  headers: { 'Content-Type': 'application/x-protobuf' } }

(note the {projectId})

Even if you ran it in an environment where google-auth-library could have detected the projectId and credentials, it still would fail because that placeholder was never replaced.

Before I figure out the right way to solve this, I want to see if we should stand by this: our docs say a projectId is always required. The project id can be detected by google-auth-library, but we still require it in our library to form API calls to the right path, among various other places.

Do we want to push for a completely projectId-configuration-free experience? I can try to extract the logic from the auth library, or send a PR to expose it.

However, if we're okay to require projectId, then I'll just throw if we don't get one or one wasn't inherited.

@ryanseys

Hm, this might be a good time to clarify the whole project ID / dataset ID thing...

In the future, each project will be able to have multiple dataset IDs (we'll likely call them databases). If you "leave it out" we should just substitute the default Dataset ID (which happens to be the project ID).

If we can guess the project ID, I'd really like that... So, some code...

var gcloud = require('gcloud');
// project ID is either null or auto-detected...
// null is OK here, we could still set it further down the line...

var dataset = gcloud.datastore.dataset();
// By this point we should have a dataset ID == project ID.
// If that's not auto-detected or set by now, we should holler.

If we wanted to explore the idea of being able to configure datastore (which I personally like but, as @ryanseys says, it's a breaking change...), this might be....

var gcloud = require('gcloud');
// Auto-detect a project ID, or leave as null.
// null is OK here, we could still set it further down the line...

var datastore = gcloud.datastore();
// pull the ID out of gcloud, or let users specify which ID it is.
// If there's no ID by this point, we should holler, as I don't think there's anything you can do...

var dataset = datastore.dataset();
// dataset ID should be datastore's project ID or specified by the user.
// No need to holler here as that already happened above.

The inheritance of a project ID from a datastore to a child dataset makes sense to me.

If we can rely on env vars to tell us if we're running on G[AC]E, then this will be super easy. If anything is async, it's terribly difficult.

GAE Dev: No magic.
GAE Prod: Env var name: GAE_LONG_APP_ID
GCE: ??

How google-auth-library checks if the user is running on GCE: https://github.com/google/google-auth-library-nodejs/blob/ce7f97409908856e2386a650e2fa4de6b2a8211a/lib/auth/googleauth.js#L158

There's a similar endpoint where we can get the projectId. The problem is that it's an async call, which complicates things.

We wouldn't want to do the async call on require('gcloud'), since the user hasn't asked us to do anything yet.
We can't do it at var datastore = gcloud.datastore();, as this makes a formerly non-async call have async side effects.
We have to do it the last possible chance, for example, on datastore.get(). Our code would have to be re-worked quite a bit to inject the new logic (if necessary) before all API calls. And even then, throwing async exceptions aren't ideal (which we would have to do if we couldn't find a projectId).

Summing all of that up, I propose we continue to require a projectId:

var gcloud = require('gcloud');
var dataset = gcloud.datastore.dataset(); // throws.
var dataset = gcloud.datastore.dataset({ projectId: 'asdf' });

Is this where promises could come in handy? Have a promise for the project ID and evaluate it when needed ?

For Google Compute Engine Linux instances that are "v20131120 or newer" we can run:

$ sudo dmidecode -s bios-vendor | grep Google

on the instance and it should output:

Google

Source: https://cloud.google.com/compute/docs/instances#dmi

Yikes -- sudo required? :-/

I tried it without sudo and got command not found so yeah, probably. :(

Makes sense to require a projectId. If they want a "configuration free" experience, they can put projectId: GCLOUD_PROJECT_ID in their code, and just make sure their instances have that env var configured always. I don't think this is asking too much. The other advantage to this is there is no surprises as to what project is used when they do run their code. If they forget to specify it, they will get an error. Sounds reasonable to me.

regarding autodetect, ~/.config/gcloud/properties is only for some Linux
systems...For ex, will not work in devshell, a GCE jenkins instance via
Bitnami and or Windows 32 and Windows 64 bits...
Most/all of them have different ways of find this info if it exists...
Ugly code for now, but this gives you some implementation in Java:
See setupInitialCommands in
https://github.com/ludoch/gcloud-maven-plugin/blob/master/src/main/java/com/google/appengine/gcloudapp/AbstractGcloudMojo.java

On Tue, May 19, 2015 at 1:50 PM, Ryan Seys [email protected] wrote:

Makes sense to require a projectId. If they want a "configuration free"
experience, they can put projectId: GCLOUD_PROJECT_ID in their code, and
just make sure their instances have that env var configured always. I don't
think this is asking too much. The other advantage to this is there is
no surprises as to what project is used when they do run their code. If
they forget to specify it, they will get an error. Sounds reasonable to me.

—
Reply to this email directly or view it on GitHub
#570 (comment)
.