How do I add an index to the datastore?

I've looked through here, so apologies if I've missed anything: https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.12.0/datastore

I'm on Google Compute (not AppEngine) hence I don't have an index.yaml which is deployed and therefore creates the indexes for me.

So I'm just trying to figure out how to create an index. I have a default namespace, Kind=Item and each entity has `{ user : 'myuser', inserted : '2015-01-31T22:38:43.019Z' }.

Use case: I am trying to get the latest 10 items that belong to a user. When trying to query this, I get 'no matching index found.':

  var query = dataset.createQuery([ 'Item' ])
  query.filter('user =', 'myuser')
  query.order('-inserted')
  query.limit(10)
  dataset.runQuery(query, callback)

I think I'm in the right place in thinking that I need to add an index for (perhaps) Item(user, inserted).

My questions are:

Is there any method that allows me to do this within gcloud-node?
I can't see any docs which show this but I may have overlooked something. Can you point me to the right place if these exist?
If this isn't the right index, is there a way to tell what it needs to be?

Many thanks.

@chilts

Hey @chilts, long time no see!

If you want to try and hit the ground running on this, take a look at the CONTRIBUTING.md doc here to see how we add indexes to run our regression tests. Unfortunately we don't have an extensive guide on how to do it in our docs, but these resources might be able to help you. We are using the gcloud CLI tool to add them to datastore. The docs for that specific command can be found here. The indexes we use for our regression tests can be found at regression/data/index.yaml so they might be a good example for you to base your indexes off of.

Let me know if you get stuck!

@ryanseys

Hi @ryanseys - good to 'see' you. Hope you're doing well.

Thanks for all the info. I finally figured out that I needed some extra stuff for the gcloud cli tool and a indexes.yaml or similar to create them. After some experimentation I found the right index to add. Cheers again.

Documenting for people who come across this issue, firstly, you need to add the 'app' component (since the datastore API is still classed as 'preview' and that is contained there):

gcloud components update app

Then add the indexes.yaml file somewhere into path/ and run this to tell the datastore to add the indexes:

gcloud preview datastore create-indexes path/

Finally, open this page (with your own projectId):
https://console.developers.google.com/project/projectId/datastore/indexes to see when the indexes are being 'created' and when they are 'serving'.

Cheers.

Thank you for documenting your steps for other developers! :)

@chilts

@chilts @ryanseys Is this still the preferred way to create an index? Don't see a way thru the console, but keep coming across this: https://cloud.google.com/datastore/docs/tools/indexconfig#Datastore_Using_automatic_index_configuration which also seems like a lot of steps to create an index.

@pcostell

/cc @pcostell

Right now the general gist is that you need to set an indexes file (.yaml or .xml), defining what you want indexed, and use the SDK to sync that file with the datastore instance.

Looks like that's defined here: https://cloud.google.com/datastore/docs/tools/indexconfig#Datastore_Updating_indexes

@jgeewax

@jgeewax Thx! No worries, got it working with the gCloud CLI
with which I ran: gcloud preview datastore create-indexes index.yaml
against a yaml file like @ https://github.com/GoogleCloudPlatform/gcloud-node/blob/master/system-test/data/index.yaml

The last missing piece of the puzzle for me was to use the AppEngine Datastore Viewer to generate the proper yaml output.

We are working on updating those docs right now to make things better, but here's the gist:

I would recommend using the Datastore emulator: gcloud beta emulators datastore start. This will run a loca version of Datastore on your machine. You should use this for testing your application. Importantly, whenever you issue that requires a query it updates an index.yaml file for you. After testing all the different cases for your application, you can then see the index.yaml file with the proper entries and upload it. This will help you avoid missing any indexes and having to wait for them to be built while your users are unable to run a specific query.

@pcostell

Thx @pcostell. Still learning the "App Engine way" 😄

Is there a way to get list of attributes for datastore records? like the properties that are indexes, or to get list of properties and be able to see the indexed value on them?

@JustinBeckwith

Hey @JustinBeckwith, not sure if you meant to assign to me. Yes, I created this 6 years ago, but don't use AppEngine or datastore anymore. And the issue has been closed since that first day 6 years ago. Nothing to do here I don't think.