Can't connect to local Datastore Emulator

Hi,
I just updated from 0.29 to 30.3 and I can't connect to the Datastore emulator locally.
This is the steps I do:

Launch datastore with:

gcloud beta emulators datastore start --project=my-project-id --data-dir /path/to/some/dir --host-port localhost:8380

It launches and tells me:

API endpoint: http://localhost:8380/datastore
...
Dev App Server is now running

Init gcloud-node with

var config = {
    projectId: 'my-project-id',
    // keyFilename: '/some/path/to/credential.key.json', (tried both with and without)
};

var gcloud = require('gcloud')(config);
var datastore = gcloud.datastore({
        apiEndpoint: "http://localhost:8380"
});

Create an Entity

var key = datastore.key('BlogPost');
datastore.save({
    key: key,
    data: {
        title: 'Hello world'
    }
}, function(err) {
    console.log(key.path);
});

I receive this in err:
"{"code":503,"metadata":{"_internal_repr":{}}}"

I then changed the apiEndPoint to "http://localhost:8380/datastore" (as indicated when launching the emulator) but then it blocks the request (does not come into the callback) and in the terminal it says:

E0414 09:11:46.140609000 4406423552 resolve_address_posix.c:126] getaddrinfo: nodename nor servname provided, or not know

As I said, I tried both with and withouth credentials without any difference.

Some help would be greatly appreciated! :)
Thanks a lot for this great library.

Hi,

I had the same problem, it's due to the fact that the datastore emulator doesn't implements GRPC yet.
you can see the workaround in the changelog :
https://github.com/GoogleCloudPlatform/gcloud-node/releases/tag/v0.30.0

Additionally, we are now using gRPC to give a performance boost to your Datastore calls. As of now, the gcloud SDK does not have a gRPC emulator, meaning if you wish to use a localhost backend instead of the upstream API, you will have to use the gcd.sh tool directly.

@sebelga

Thanks for the kind words @sebelga and for the assist @Thomas-T!

Also tagging duplicate: #1220.

@Thomas-T

Thanks for fast answer and info @Thomas-T ! , I will try the gcd.sh tool and come back if any problem.
Best

Ok I got it working with the gcd.sh (had to first use the "create" command on a folder, then the "start" command on that same folder.

Although it works, on each call to the datastore I get (on the node server console):
E0415 17:55:30.698468000 4406132736 tcp_client_posix.c:173] failed to connect to 'ipv6:[::1]:8080': socket error: connection refused

And on the local datastore console:
INFO: Detected HTTP/2 connection

Not a big deal but just checking if it was normal :)

@pcostell

Thanks for bringing it up!

INFO: Detected HTTP/2 connection

That sounds okay.

E0415 17:55:30.698468000 4406132736 tcp_client_posix.c:173] failed to connect to 'ipv6:[::1]:8080': socket error: connection refused

That's a little more concerning. @pcostell that error is being printed from gRPC. The function calls seem to behave correctly, but that message still prints. Is that anything to be concerned with?

The INFO message is totally fine, it's just logging that it received an HTTP/2 connection instead of an HTTP/1 connection (we should probably make this FINE messages instead).

The ipv6 error definitely seems more alarming, but unfortunately I don't know much about it. Maybe we should open up a bug over on the grpc github?

@murgatroid99

Thanks for checking it out. @murgatroid99 should we open an issue? I thought I'd give you a chance to take a look first.

One last detail. It seems that when connecting locally to this gcd.sh, the generated ID does not work as I receive a "normal" incremental id (1, 2, 3, ... )

To make sure it was not the gcloud library, I removed the "apiEndpoint" and pointed to Google and the generated ID was like before (16 number figure).

Again, no big deal because it's local but just to let you know..

Hi, I'm having the same issue.
If the problem is that the emulator does not implement grpc yet, will it eventually implement it? I wish it does.

If the answer is true, should we keep this issue open?

Thanks,

FYI, I'm getting the same error on a datastore.insert() call:

E0725 18:03:55.390445457  178615 tcp_client_posix.c:173]     failed to connect to 'ipv6:[::1]:8080': socket error: connection refused

This is the output of the emulator:

./gcd/gcd.sh start /tmp/datastore
Jul 25, 2016 6:03:32 PM com.google.appengine.api.datastore.dev.LocalDatastoreService init
INFO: Local Datastore initialized: 
    Type: High Replication
    Storage: /tmp/datastore/WEB-INF/appengine-generated/local_db.bin
Jul 25, 2016 6:03:32 PM io.grpc.internal.ManagedChannelImpl <init>
INFO: [ManagedChannelImpl@1060b431] Created with target localhost:8080
Jul 25, 2016 6:03:32 PM com.google.appengine.api.datastore.dev.LocalDatastoreService load
INFO: The backing store, /tmp/datastore/WEB-INF/appengine-generated/local_db.bin, does not exist. It will be created.
Jul 25, 2016 6:03:32 PM io.gapi.emulators.netty.NettyUtil applyJava7LongHostnameWorkaround
INFO: Applied Java 7 long hostname workaround.
API endpoint: http://localhost:8080/datastore
If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:

  export DATASTORE_EMULATOR_HOST=localhost:8080

Jul 25, 2016 6:03:55 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
INFO: Adding handler(s) to newly registered Channel.
Jul 25, 2016 6:03:55 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected HTTP/2 connection.
Jul 25, 2016 6:03:55 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
INFO: Adding handler(s) to newly registered Channel.
Jul 25, 2016 6:03:55 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected HTTP/2 connection.

It would be great to reopen this issue. It's a major development hindrance.

@pcostell

Can someone open an issue on the grpc repo? https://github.com/grpc/grpc. Please link back to this issue so anyone else who might run into this can follow along. You can tag me and @pcostell as well.

Is this really a gRPC problem? How can you tell?

I was wondering if the error was due to having an empty datastore. Whenever I start the emulator, I get local_db.bin, does not exist. It will be created., which tells me that the datastore is not being created. In fact when I look in the datastore directory, the local_db.bin file is not there. Maybe it's created lazily in the first write?

In any case, I decided to run the tool without using the disk, and it crashed right away:

./gcd/gcd.sh start --store_on_disk=false
Exception in thread "main" java.lang.NullPointerException
    at com.google.cloud.datastore.emulator.CloudDatastore$StartAction.apply(CloudDatastore.java:619)
    at com.google.appengine.tools.util.Parser$ParseResult.applyArgs(Parser.java:48)
    at com.google.cloud.datastore.emulator.CloudDatastore.run(CloudDatastore.java:848)
    at com.google.cloud.datastore.emulator.CloudDatastore.main(CloudDatastore.java:794)

This problem is getting in the way of the Google Cloud Functions Alpha release, as we developers need a way to test things locally.

@pcostell

I can't tell, but that additional information might be helpful. That sounds like an issue with the emulator.

But, the goal is getting the issue in front of the right eyes. I don't believe this to be a problem with this API client library. It seems to be in either the transport layer (grpc) or Datastore. @pcostell is from the Datastore team and advised opening the issue with gRPC. If you need the quickest response possible, heading over there might be best. And when @pcostell catches all of the name dropping I'm doing, maybe he'll take another look and offer help as well.

@pcostell

I was able to find a workaround (thanks for a theory from @pcostell that it may be an ipv6 issue).

First, run the emulator with --no-legacy:

gcloud beta emulators datastore start --no-legacy

Then use 127.0.0.1 instead of localhost:

DATASTORE_EMULATOR_HOST=127.0.0.1:<port> node my_app.js

Of course, replace <port> by whatever was output by gcloud (or force a port in the emulator with --host-port)

It sounds like this may be an ipv4 vs ipv6 issue.

A note if it's helpful, the gcloud emulator supports ipv6 hostnames starting in gcloud 116.0.0:

gcloud beta emulators datastore start --no-legacy --host-port "[::1]:8080"

But it seems like gcloud-node is trying to connect over ipv6 to localhost, whereas the emulator is serving on ipv4.

Is there anything our library can do?

chore(main): release 2.30.3 (#1230)

feat: support create/list datasets on a different project (#1230)

build: do not overwrite custom env vars (#1230)

feat: support callbacks with database getRestoreInfo(), getState(), g…

chore(main): release 9.7.0 (#1230)

fix: remove common protos (#1232)

chore(testproxy): read row should return a single row (#1230)

chore(testproxy): read row should return a single row (googleapis#1230)

Navigation Menu

Repository navigation

Can't connect to local Datastore Emulator #1230

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Can't connect to local Datastore Emulator #1230

Description

Activity

Thomas-T commented on Apr 14, 2016

stephenplusplus commented on Apr 14, 2016

loixlab commented on Apr 14, 2016

loixlab commented on Apr 15, 2016

stephenplusplus commented on Apr 15, 2016

pcostell commented on Apr 15, 2016

stephenplusplus commented on Apr 15, 2016

loixlab commented on Apr 15, 2016

zjaml commented on Jul 4, 2016

fortuna commented on Jul 25, 2016

stephenplusplus commented on Jul 25, 2016

fortuna commented on Jul 25, 2016

stephenplusplus commented on Jul 25, 2016

fortuna commented on Jul 25, 2016

pcostell commented on Jul 26, 2016

stephenplusplus commented on Jul 26, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions