Unable to query more than 300 entities after updating to 0.30.0

After updating from 0.29.0 to 0.30.0 one of my queries is now capped at returning 300 entities with no cursor. When I revert back to 0.29.0 it properly returns the limit of 1000. Any ideas why this might be happening?

var query = datastore.createQuery('Payment')
                        .filter('datemade', '>=', new Date(dateStart))
                        .filter('datemade', '<', new Date(dateEnd))
                        .limit(1000);

datastore.runQuery(
    query,
    function(error, entities, cursor) {
        if (error) {
            console.error(error);
        } else {
            console.log(entities.length);
            console.log(cursor);
        }
    }
);

@pcostell

@pcostell here is our logic for continuing to run queries -- does that look correct?

For reference, here is our old logic.

@mcfarljw

Also @mcfarljw, we don't return a third argument to the callback with runQuery, it should just be error and entities, unless you disable autopagination:

var query = datastore.createQuery('Payment')
                        .filter('datemade', '>=', new Date(dateStart))
                        .filter('datemade', '<', new Date(dateEnd))
                        .limit(1000)
                        .autoPaginate(false);

datastore.runQuery(
    query,
    function(error, entities, nextQuery) {
        if (nextQuery) {
            datastore.runQuery(nextQuery, function(error, entities, nextQuery) {/*...*/});
        }
    }
);

@stephenplusplus

@stephenplusplus It looks like your end condition is wrong. It should be "NOT_FINISHED", not "MORE_RESULTS_AFTER_LIMIT"

@mcfarljw

Thanks! PR and patch incoming @mcfarljw -- thanks for reporting!

Fix is released under v0.30.1 -- please let me know if it works!

Thanks for the quick responses! I just loaded up the latest patch v0.30.1, but I'm still not able to all the results that v0.29.0 was returning. There seems to be something else going on here.

I've done a few more tests directly querying some of my entities with nothing more than a limit of 1000 set. Using v0.29.0 it always returns the requested limit, but when using v0.30.1 it returns a significantly fewer amount that varies based on the entity being queried. For example, the Payment entity in the example below will only return up to a limit of 300, if I set it to 301 then the request seems to go on indefinitely without timing out. I just tried a simple limit query on another entity and it wouldn't go past a limit of 96.

datastore.runQuery(
    datastore.createQuery('Payment').limit(1000),
    function(error, entities) {
        console.log(entities.length);
    }
);

I think I've found the problem, but I'll have to take a swing at it tomorrow-- sorry for the delay. I'll send a PR real quick with a rough idea of the solution.

Assuming we fixed it for real this time, it's released in v0.30.2 :)

I just tested it out again using v0.30.2 and was able to query the expected number of entities. Thanks for addressing this in such a timely manner!

Phew! Thank you for reporting it, it was a huge oversight on my part.

👨
➖
👓

chore(main): release 2.30.1 (#1200)

build: update Node.js generator to compile protos (#1200)

fix: batch transaction should use a session from the pool (#1207)

chore: release 9.6.7 (#1200)

fix: openTelemetry doc fails to generate for #1185 (#1200)

Navigation Menu

Repository navigation

Unable to query more than 300 entities after updating to 0.30.0 #1200

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unable to query more than 300 entities after updating to 0.30.0 #1200

Description

Activity

stephenplusplus commented on Apr 4, 2016

stephenplusplus commented on Apr 4, 2016

pcostell commented on Apr 4, 2016

stephenplusplus commented on Apr 4, 2016

stephenplusplus commented on Apr 5, 2016

mcfarljw commented on Apr 5, 2016

stephenplusplus commented on Apr 5, 2016

stephenplusplus commented on Apr 5, 2016

mcfarljw commented on Apr 5, 2016

stephenplusplus commented on Apr 5, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions