Repository navigation
Ancestor query with filter on children not returning anything #1237
Description
I am using gcloud-node v.0.30.3 and since it doesn't support gRPC I am using gcd-rpc for my local datastore.
Say I run this query to get all the children:
var query = datastore.createQuery('Contacts').hasAncestor(datastore.key(['ContactsParent', 'my_username']));
datastore.runQuery(query, function (err, entities) {
if (err) {
console.log('error ' + err);
return;
}
console.log(JSON.stringify(entities));
return;
});
This works and returns all the entities. If I go:
var query = datastore.createQuery('Contacts').hasAncestor(datastore.key(['ContactsParent', 'my_username'])).filter('status', 'some status');
datastore.runQuery(query, function (err, entities) {
if (err) {
console.log('error ' + err);
return;
}
console.log(JSON.stringify(entities));
return;
});
Nothing is return even though it should return something. Nothing is printed in the node console.
I have noticed some strange behavior. If I add a limit clause to this query it will return an empty array ( [] is printed in the console), but without it nothing.
My index.yaml file looks like this:
indexes: - kind: Contacts ancestor: yes properties: - name: status
I am expecting with this query that all the children get returned, and then from those children only the ones with a status of 'some status' return. This is not happening.
The records do exist in the datastore, but when I filter they do not. Example of a record:
datastore.save(
{
key: datastore.key(['ContactsParent','my_username', 'Contacts','my_contact']),
method: 'insert',
data: {
status: 'some status'
}
}
}
Activity
stephenplusplus commentedon Apr 15, 2016
cc @pcostell
pcostell commentedon Apr 15, 2016
@stephenplusplus Could you verify what proto gets sent in this case?
pcostell commentedon Apr 15, 2016
Note I tested this manually using direct gRPC to the emulator and it seems to work fine, so I think it may be limited to gcloud-node.
stephenplusplus commentedon Apr 15, 2016
I'm trying this now and get stuck in an infinite loop. The response from the query always has
batch.moreResultsasMORE_RESULTS_AFTER_LIMIT, so our code creates anextQuery, and automatically runs it. TheendCursorvalue is always the same as well, so it never gets anywhere.pcostell commentedon Apr 15, 2016
Didn't we address this in 30.3?
stephenplusplus commentedon Apr 15, 2016
We handle
NOT_FINISHEDby running the query again automatically, regardless of "autoPaginate" being true or false. And we added some perpetual limit logic to make sure we only ask for the right amount of results on those "nextQuery"s. But when we get theMORE_RESULTS_AFTER_LIMITresponse, we still build and run the nextQuery.Here's where our response logic starts-- anything look incorrect?
pcostell commentedon Apr 15, 2016
Doesn't the logic say to create the nextQuery if
more_resultsisNOT_FINISHEDorMORE_RESULTS_AFTER_LIMIT, but only run it if it isNOT_FINISHED?stephenplusplus commentedon Apr 15, 2016
It looks that way because of the code in that function, but when
runQueryis called, it's actually wrapped by another function which automatically runs anynextQuery. This wrapper function is used throughout our API with any method that returns a "nextQuery" (storage.bucket.getFiles(), pubsub.getTopics(), etc). So we had to add in the manual override to continue running if it's "NOT_FINISHED", because the wrapper function wouldn't have continued running the query if the user had "autoPaginate" set to false.Is it possible the emulator should have returned another value in place of "MORE_RESULTS_AFTER_LIMIT"? The code is only running over and over again because the response is telling us there are more results, but they're not coming.
pcostell commentedon Apr 15, 2016
Yes this is definitely the case: the emulator always returns
MORE_RESULTS_AFTER_LIMIT or NOT_FINISHED, but never NO_MORE_RESULTS.
I am working on making the emulator match production Datastore now.
However, I don't think the client should ever be doing this automatically.
This is the point of having NOT_FINISHED. MORE_RESULTS_AFTER_LIMIT should
only be used as a signal to the user of the client library that there might
be another page. For resource reasons the API explicitly only guarantees
that the value of this field is more conservative from a correctness point
of view (it won't say that is is finished if it actually isn't -- except of
course for eventual consistency for eventually consistent queries).
On Fri, Apr 15, 2016, 3:31 PM Stephen Sawchuk [email protected]
wrote:
stephenplusplus commentedon Apr 15, 2016
This is just the default behavior of our API. If there is a "nextQuery" to run, we run it automatically. All methods that act this way can be instructed not to. In the case of
datastore.runQuery(),query.autoPaginate(false)must be set.We've done it many ways in the past, but this is why it was defaulted.
Do we need to rethink that approach for running Datastore queries, or is allowing auto-pagination to be disabled sufficient?
// @jgeewax
susanlinsfu commentedon Apr 15, 2016
Sorry, but from what I understand adding autoPaginate(false) to my query should solve it for now? I have just tried adding it to my query and it still returns an empty array.
pcostell commentedon Apr 16, 2016
@stephenplusplus that makes sense, I think in that case you should only return
nextQueryifmore_results = NOT_FINISHED(and not do the request insiderunQuery, instead depend on the generic pagination).On Fri, Apr 15, 2016, 4:37 PM susanlinsfu [email protected] wrote:
stephenplusplus commentedon Apr 16, 2016
@susanlinsfu regarding the empty result set, I'm not sure why they aren't being returned. Here's a script that I've tried: https://gist.github.com/stephenplusplus/8302ecb5892072f80db17f717a84a898. The query seems to return the results. Does that same script not work for you, or am I maybe not properly recreating your scenario?
susanlinsfu commentedon Apr 16, 2016
@stephenplusplus ahh yes it works for me now, I had a spelling mistake in the word "some stuff". And yes you are right if autopaginate is disabled it works, but if it is not there (or if limit) it will not return any data. Thanks!
8 remaining items
aloisDeLaComble commentedon Jan 30, 2017
autoPaginate(false) does not exist on datastore 0.6.0. Am I doing things wrong or was this method removed ?
If it has been removed, then how to fix the issue that still occurs when mixing hasAncestor with filter ?
dazraf commentedon Jul 7, 2017
I just installed the latest release of gcloud beta tools - this problem is still an issue. Why is it closed?
ziong commentedon Aug 15, 2017
+1
problem not yet solved!
any update?
chore(main): release 8.6.0 (#1237)
test: use pack-n-play for install test (#1237)
chore(main): release 8.6.0 (#1237)
chore(main): release 8.6.0 (#1237)
build(test): recursively find test files; fail on unsupported depende…
feat: publish the Cloud Bigtable Change Streams (#1237)
feat: publish the Cloud Bigtable Change Streams (#1237)
feat: publish the Cloud Bigtable Change Streams (#1237)
feat: publish the Cloud Bigtable Change Streams (googleapis#1237)