Repository navigation
Datastore error on great amount of data #1386
Description
Activity
Sorry you're having this issue :( Is it possible to put a small sample together, or otherwise the actual gcloud-node code you're using?
hi,
here is a really small test file to verify the behaviour.'use strict'; const gcloud = require('gcloud')({ projectId: [project ID], keyFilename: [keyfilePath] }); const datastore = gcloud.datastore(); const _ = require('lodash'); var str = new Array(500000).join("a"); debugger; function toDatastore(obj, indexedFields) { indexedFields = indexedFields || []; const results = []; Object.keys(obj).forEach((k) => { if (obj[k] === undefined) { return; } results.push({ name: k, value: obj[k], excludeFromIndexes: indexedFields.indexOf(k) < 0 }); }); return results; } function create(kind, data, indexedFields, cb) { const key = datastore.key([kind]); debugger; const entity = { key: key, data: toDatastore(data, indexedFields) }; datastore.save( entity, (error) => { debugger; if (error) { return cb ? cb(error) : error; } return cb ? cb(null) : true; } ); }; create("testKind", { data0: str, data1: str, data3: str, key: '250277808676421', key1: 'b4wbj0b5wy04', key2: 'sdjhfkahsdjkfhjkas', key3: "setstts", key4: "sdfasdfa" }, ["key", "key1", "key2", "key3", "key4"], (err, result) => { debugger; });In this case the error was raised whth string of 500000 chars.
ThanksThanks for putting that together. @pcostell any insight on this one? Thanks!
It looks like you've exceeded the maximum size for an entity: https://cloud.google.com/datastore/docs/concepts/limits
Not sure why you don't get a nicer error though. @stephenplusplus can see what the request/response from Datastore is?
Ah yes that's likely the issue -- although I don't understand why a Bad request turns into the given node stack trace.
Yeah, I don't understand that either. That didn't happen for me from the sample code provided, maybe it was from some temporarily malformed code, @mik115?
No temporary malformed and yes I reached the upper entity limit size of 1MB.
I've already fixed the problem chunking my data and saving them in separate entities.Thanks anyway for the really quick responses
Great :) We're working on providing more helpful error messages in #1378. Feel free to re-open this if you think there's more we can do.
It breaks node process for me as well! With same error, on same line:
Error: Unsupported field value, undefined, was provided. < at Object.encodeValueIdeas how to fix it?
@kirillgroshkov could you share reproduction code?
1 remaining item
- added 6 commits that reference this issue
on Jan 27, 2026 - added a commit that references this issue
on Mar 27, 2026 - added a commit that references this issue
on Mar 27, 2026 - added a commit that references this issue
on May 5, 2026
Hi,
i'm facing a real strange issue.
I've a datastore save request that goes well when the entity is small,
but when we try to save a great amount of data we receive this error
that breaks the node process with this stacktrace:
the error shows up when i try to save an entity with two string field, not indexed of 1MB of size
ie
We chunck our data on string of less than 1MB to respect the maximum string length property on datastore.
I'm pretty sure that the issue is about the amount of data because saving only one chunk no error was raised.
There's an upper limit on the total entity size or there is some error?
Thanks in advance