Repository navigation
nextQuery always returns a value #1260
Description
Activity
I believe there are issues with the emulator. @pcostell Do you think is related to the issue we found before?
Yes I believe this is the same issue as before. However, even if there are no more results Cloud Datastore will still return an end cursor which may be useful. So I'm re-echoing my concern that nextQuery isn't the right API for when
more_resultsisMORE_RESULTS_AFTER_LIMITorMORE_RESULTS_AFTER_CURSOR. Importantly, unlike other APIs the Cloud Datastoreend_cursoris not a traditionalnext_page_token.What would be the ideal way of handling pagination? Sorry if I missed your explanation before, feel free to link me.
// just dropping a link to #1237 as that was the issue we discovered the API/emulator mismatch
Coming back to this:
I think we need the following:
On more_results = NOT_FINISHED, set next_query. Properly track offset/limit so that this next_query has these set.
On any other more_results, do not set next_query.
This means we can defer to the normal paging tools to handle paging for us. It would also mean that when more_results == MORE_RESULTS_AFTER_LIMIT, next_query would not be set. This means we'll need to expose the end_cursor and more_results so that the user can make an informed decision when choosing to display information to the user.All of this will apply regardless of the auto_paginate setting. Users that set auto_paginate=False will just have to deal with the case where more_results=NOT_FINISHED, however we'll properly setup the next_query for them.
I think I'm stuck on why we wouldn't want to provide
nextQuery. I think we're doing it just to save the user the time and hassle of building their own query. They of course don't have to run it. And if it's poor form for us to run it for them (i.e, how we're treating it like anextPageToken), then we can just always returnnextQuery, but never run it for the user.// @jgeewax
In the case of
more_result=NOT_FINISHED, then we want to runnextQueryfor them (exactly like anextPageToken) -- really in this case we don't ever want to not run it for them, but in theory they can thanks toautoPaginate. However, in most other cases we don't expect the user to run it again with the context of that code (for example, we'd expect the user to serialize the cursor then use it later). In this case, having the query isn't actually helpful, they just need to know how the current position in the query. Cursor functionality expends beyond paging. They can be used for splitting work over queries and defining ranges.In the case of more_result=NOT_FINISHED, then we want to run nextQuery for them (exactly like a nextPageToken) -- really in this case we don't ever want to not run it for them, but in theory they can thanks to autoPaginate
Woops, forgot about that case. We should definitely run it for them 👍
for example, we'd expect the user to serialize the cursor then use it later
Makes a lot of sense. We could still return a nextQuery, but make it more straightforward on getting the endCursor from it. It's currently the un-documented
nextQuery.endVal, but we could expose this withnextQuery.getEndCursor()or justnextQuery.endCursor.I may be wrong here, but I don't think there is harm in leaving the behavior as it is now. I would bet that the "handle everything for me" approach is desirable to a fair amount of users. However, we definitely need to document
nextQuerybetter, explain what a non-autoPagination cursoring flow looks like, and add the above mentioned accessors.I'm sorry I am not quite following. So with the code I provided would there be a way to modify it to know when there are no more results?
Cheers
The way I see it is that our desired behavior is a streaming API. If a user asks for a limit of N, we will give them a stream of up to N entities. We can then add some helpers on top of that to convert the stream to a list.
Cloud Datastore then offers a paging feature, which provides the user a cursor they can use to get extra pages. On top of that, Cloud Datastore has the feature of best effort end condition tracking (
more_results). This is very useful for users who are showing pages to their users as they can get information about whether or not they should show a "Next Page" button (this isMORE_RESULTS_AFTER_LIMIT). This feature is conservative (MORE_RESULTS_AFTER_LIMITmay be returned even if the correct result isNO_MORE_RESULTS).Importantly, Cloud Datastore's paging feature is completely separate from the batching API, which is really just an implementation detail because prior to gRPC we couldn't offer a streaming API.
@susanlinsfu Sorry I took over this thread a bit to discuss proper Cloud Datastore usage. Right now, the
next_queryis still returned even though there are no more results. I'll try to make sure to explicitly send an update when the emulator has bettermore_resultssupport.I think it would be good to hear from JJ. I'll bring it up in the next meeting (Monday) if we don't hear back sooner.
I'd like to pop up a level here and look at the situations that users are actually interested in:
- I want to loop over all results
- I want to loop over all results and break out early if I match a condition
- I want to loop over results, but make
<= NAPI calls - I want to get
<= Nresults (with any number of API calls) - I want to build a UI that shows pages and picks up where I left off
Looking at these cases, I see the following:
- For all results, we have the streaming API which is probably the best way to handle this use case.
- For breaking early, just do the above and call
this.end()as documented. - I don't have an easy way to limit the number of API calls...
- I can tell Datastore to limit things, and then use the streaming API to iterate through the limit
- I should manually track tokens for pages/cursors, but I have quite a bit of code to write
IMO,
nextQueryis nice, but it's an implementation detail. I really think that having to reference it at all means we've kind of built the wrong thing.
Here's what I think we should make things look like:
Give me all the results:
datastore.runQuery(query).on('data', function(entity) { // do something with entity });
Give me results until I change my mind:
datastore.runQuery(query).on('data', function(entity) { // do something with entity if (iFeelLikeIt) this.end(); });
Limit the number of API calls (because that is the same as $$$):
datastore.runQuery(query, {apiCallLimit: 10}).on('data', function(entity) { // do something with entity });
Limit the number of entities (because I only want a max of 10 to show in my list):
query = query.limit(10); datastore.runQuery(query).on('data', function(entity) { // do something with entity });
Page through things as though I'm showing a UI
query = query.limit(10).startCursor(request.get('cursor')); var pageEntities = []; datastore.runQuery(query).on('data', function(entity) { pageEntities.push(entity); }); render_template('template.html', {entities: entities, cursor: query.nextCursor});
Obviously if we do promises, this gets much happier:
query = query.limit(10).startCursor(request.get('cursor')); datastore.runQuery(query).then(function(entities) { render_template('template.html', {entities: entities, cursor: query.nextCursor}); });
I'd agree here, except for
I want to loop over results, but make <= N API calls
How many API calls get made is really an implementation detail, and may depend on many other things (like timeouts and constraints on the server). I think this is reflected in Cloud Datastore's new pricing model, which charges per entity returned and doesn't factor in the number of API calls made (starting July 1).
11 remaining items
I wouldn't recommend downgrading to an older version because you'll also switch to using Cloud Datastore API v1beta2, which has much worse latency characteristics than v1beta3.
@stephenplusplus -- This should work now if @susanlinsfu just provides autoPaginate(false), right? The query should just return all 42 results. Until the emulator returns the proper
more_resultsflag you'll still get a next query, but otherwise it will work, right?Just a note that downgrading shouldn't work. As I understand our setup, the emulator still returned
MORE_RESULTS_AFTER_LIMITeven if the limit hadn't been reached in v1beta2.@pcostell the bare bones question is:
how would I know there are no more results to fetch in the query?
If the emulator only responds with "MORE_RESULTS_AFTER_LIMIT" or "NOT_FINISHED", but never "NO_MORE_RESULTS", our API always return a
nextQuery. So whileautoPaginate(false)has to be used*, a user will still have to keep manually runningnextQueryto see if more results come back. Compared to when running against the cloud API, we won't return anextQuerywhen we getNO_MORE_RESULTS, which is how the user knows there are no more results to fetch.*if
autoPaginate(false)is not used, you'll end up in an infinite loop because we auto-runnextQuery.Thanks for the clarification around the old emulator.
Ok so here's the issue from my side:
You should only auto fetch the next batch if
more_resultsisNOT_FINISHED.A query with
MORE_RESULTS_AFTER_LIMITshould have the next batch fetched ONLY by developers.Then, you can get all 42 entities by just issuing the query:
datastore.createQuery('Messages').filter('sender',username);Could you please explain when each of the responses is returned? (MORE_RESULTS_AFTER_LIMIT, NOT_FINISHED, NO_MORE_RESULTS)
NOT_FINISHED: returned when the query is not complete.
If the user set a limit, we haven't satisfied the limit. If the user did not, there are likely more results. However, this is best effort: it is still possible that
NOT_FINISHEDis returned and the follow up query returns no results but a newmore_resultswith updated information. This will happen if Cloud Datastore decides to end the request based on resource constraints and cannot determine if there are more results after the last result.MORE_RESULTS_AFTER_LIMIT: there may be more results after the user specified limit.
It's possible for this to be returned even though there are not actually results after the limit because we weren't able to determine if there are more results or not.
In
v1beta2and the emulator version ofv1beta3, this is always the result returned for completion.MORE_RESULTS_AFTER_CURSOR: there may be more results after the user specified end cursor.
NO_MORE_RESULTS: there are no more results.
Thank you! From the gcloud-node side:
If a user runs a query with a limit, we return that many results, as well as a nextQuery for them to get more:
runQuery(queryWithLimit, function (err, results, nextQuery) { results.length <= Limit if (nextQuery) { nextQuery.startVal === originalQueryResponse.endCursor } })
If a user runs a query without a limit, we fetch until we get
NO_MORE_RESULTS.runQuery(queryWithoutLimit, function(err, results) { // all results that match the query (no `nextQuery`) })
To handle pagination manually:
runQuery(queryWithoutLimitAndAutoPaginationDisabled, function(err, results, nextQuery) { results.length === However many Datastore returned with until `MORE_RESULTS_AFTER_LIMIT` })
I think this is the case you want us to change-- you're saying we should let the user do all of the pagination manually, as in the last example?
Thanks for your patience!
I'm not sure I'm asking the user to handle pagination manually, but let me restate to hopefully answer the question:
-
gcloud-node should iterate as long as
more_results == NOT_FINISHED. <== note that the way you've builtauto_paginate, I think ifauto_paginate=Falseyou should stop iterating? Which is why I thinknext_queryshould only return IFmore_results == NOT_FINISHEDand there should be a separate mechanism for the other cases. -
When gcloud-node returns to the user,
more_resultsinformation should be returned to them. They can then decide if they want to display the "next page" icon to their user. In the case of the emulator orv1beta2, they'll just always have a "Next Page" icon (they can avoid this by looking at limits if it's really important).
-
I'm going to work on a PR that does that and see if I start to understand what we're going for in the process.
- added a commit that references this issue
on Jul 23, 2025 - added a commit that references this issue
on Feb 3, 2026
I have 42 items which should be returned:
The query first gets the first 10 items, then the next 10, then the next 10, then the remaining 2(42 items). The problem then is when I make the last query to get the last 2 items it returns a nextQuery. As a result it looks as if there is more items even though there are none. I have gone over my code and I can't see anything that is causing this so I am thinking it has something to do with gcloud. I am expecting that when the last query is made no nextQuery should be returned by the datastore. Is this correct?
I am using gcd-rpc emulator and gcloud-node 0.31.0.