Skip to content
This repository was archived by the owner on Feb 20, 2026. It is now read-only.
This repository was archived by the owner on Feb 20, 2026. It is now read-only.

Errors importing lots of entities to DataStore using the emulator #95

Closed
#544
Closed
Errors importing lots of entities to DataStore using the emulator#95
#544
Assignees
stephenplusplus
Labels
api: datastoreIssues related to the googleapis/nodejs-datastore API.type: questionRequest for information or clarification. Not an issue.

Description

@JustinBeckwith
JustinBeckwith
opened on May 31, 2018

From @glenpike on May 23, 2018 10:26

[x] - Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
[x] - Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
[404] - Check our Troubleshooting guide: https://googlecloudplatform.github.io/google-cloud-node/#/docs/guides/troubleshooting
[404] - Check our FAQ: https://googlecloudplatform.github.io/google-cloud-node/#/docs/guides/faq

If you are still having issues, please be sure to include as much information as possible:

Environment details

  • gcloud SDK: 202.0.0
  • OS: OSX El Capitan (10.11.6) Using about 12GB / 16GB memory
  • Node.js version: v8.11.2
  • npm version: v5.6.0
  • google-cloud-node version:
├─┬ @google-cloud/[email protected]
│ ├─┬ @google-cloud/[email protected]
├─┬ @google-cloud/[email protected]
│ └─┬ @google-cloud/[email protected]
│   ├─┬ @google-cloud/[email protected]
│   ├─┬ @google-cloud/[email protected]
├─┬ @google-cloud/[email protected]
│ ├─┬ @google-cloud/[email protected]

Using DataStore via: [email protected]

Steps to reproduce

Looping through a list of data and creating a model for each one, then calling a function which
uses save

    const { body } = ctx;
    let promises = [];

    // Save new Object
    body.new.forEach((model) => {
        const createdEntity = FromModel.create(model, model.id);
        promises.push(createdEntity.upsert());
    });

    const response = await Promise.all(promises);

Trying to import about 2.5k models, we are getting lots of errors that look like they're coming from grpc maybe? Workaround is to split data into chunks, e.g. 1/4 works.

The log of errors looks like this ('...' is replacing several repeated events):

10:02:46.826Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.826Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.827Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
...
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.841Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.841Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
...
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.858Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.858Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.859Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.873Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.874Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
...
10:02:46.898Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.898Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.898Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.898Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:02:46.899Z ERROR import: 13 INTERNAL: Half-closed without a request
10:02:46.899Z ERROR import: 1 CANCELLED: Received RST_STREAM with error code 8
10:03:43.145Z ERROR import: 4 DEADLINE_EXCEEDED: Deadline Exceeded
10:03:43.145Z ERROR import: 4 DEADLINE_EXCEEDED: Deadline Exceeded

The DEADLINE_EXCEEDED error seems to correspond with this in the emulator:

datastore] May 23, 2018 11:03:16 AM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub$7 run
[datastore] INFO: Time to persist datastore: 198 ms
[datastore] Exception in thread "LocalDatastoreService-1" java.lang.OutOfMemoryError: unable to create new native thread
[datastore] 	at java.lang.Thread.start0(Native Method)
[datastore] 	at java.lang.Thread.start(Thread.java:714)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[datastore] 	at java.lang.Thread.run(Thread.java:745)
[datastore] Exception in thread "LocalDatastoreService-4" java.lang.OutOfMemoryError: unable to create new native thread
[datastore] 	at java.lang.Thread.start0(Native Method)
[datastore] 	at java.lang.Thread.start(Thread.java:714)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
[datastore] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[datastore] 	at java.lang.Thread.run(Thread.java:745)

Copied from original issue: googleapis/google-cloud-node#2822

Activity

  1. JustinBeckwith commented on May 31, 2018

    @JustinBeckwith
    JustinBeckwith
    on May 31, 2018
    ContributorAuthor

    @glenpike does this repo using the datastore service, and not the emulator? I'm trying to figure out if this is an issue with the client library or the emulator :)

  2. JustinBeckwith
    added
    type: questionRequest for information or clarification. Not an issue.
    needs more infoThis issue needs more information from the customer to proceed.
    on May 31, 2018

  3. glenpike commented on Jun 1, 2018

    @glenpike
    glenpike
    on Jun 1, 2018

    Hi @JustinBeckwith - this happens with the emulator, on the 'live' system it's behaving.

  4. sduskis
    removed
    needs more infoThis issue needs more information from the customer to proceed.
    on Dec 4, 2018

  5. ideasculptor commented on Jul 30, 2019

    @ideasculptor
    ideasculptor
    on Jul 30, 2019

    This problem is persistent in the emulator.

    I have a reasonably small production datastore instance - barely over 1GB including indexes. I then exported a small fraction of that data, just a few of the entity types constituting about 125MB of data in the storage bucket. But I have been utterly unable to import that data into the emulator. No matter how much memory I give to the running process, it eventually errors out with OOM errors. (when I gave the docker container I was running it in 8GB of memory, it finally completed). Total size on disk was about 160MB. The runtime memory requirements relative to the total dataset size seem more than a little out of whack.

    I'm just using a basic import command via curl, just like the documentation suggests (which also never make a single mention of memory management). We're talking about tens of thousands of entities here, not millions. This ought to be a trivial workload for any database. Are there any workarounds? I'm on a host with 16GB of memory and plenty of disk space.

  6. tylervick commented on Nov 14, 2019

    @tylervick
    tylervick
    on Nov 14, 2019

    +1 - I encounter this issue using the recommended documentation (posting via cURL). A single kind with a few thousand records (~2GB) should not choke this up. Any ideas?

  7. JustinBeckwith
    assigned
    stephenplusplus
    on Nov 14, 2019

  8. JustinBeckwith commented on Nov 14, 2019

    @JustinBeckwith
    JustinBeckwith
    on Nov 14, 2019
    ContributorAuthor

    @stephenplusplus could I trouble you to take a look?

  9. ohmpatel1997 commented on Nov 15, 2019

    @ohmpatel1997
    ohmpatel1997
    on Nov 15, 2019

    I have encountered the same issue while I was importing data of around 1GB. So the problem was in emulator only. The solution is to increase the allocated memory to JVM. It will eat up your entire CPU but it will work.

    Stop your emulator.

    Use the below command to increase the memory.

    -Xms512m -Xmx1152m -XX:MaxPermSize=256m -XX:MaxNewSize=256m
    -Xms: initial heap size
    -Xmx: Maximum heap size

    Increase it according to your needs and run your import again.

    Hope this works for you...!

  10. stephenplusplus
    closed this as completedin #544on Nov 26, 2019
  11. gcf-label-sync
    added
    api: datastoreIssues related to the googleapis/nodejs-datastore API.
    on Jan 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: datastoreIssues related to the googleapis/nodejs-datastore API.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions