BigTableInstance.getRows sometimes returns 0 values silently

I noticed when running getRows that I would sometimes get 0 results. No error was thrown so this took a while to uncover, and eventually I had to do it in batches:

  let rows = [];
  const MAX_KEYS = 1000;

  for (const i = 0; i * MAX_KEYS < rowIds.length; i++) {
    const [newRows] = await eventsTable.getRows({
      decode: false,
      keys: rowIds.slice(i * MAX_KEYS, (i + 1) * MAX_KEYS),
      filter: [{
        column: {
          cellLimit: 1,
        },
      }],
    });

    rows = rows.concat(newRows);
  };

I'm not sure if this is even the right approach, since 1000 seems pretty arbitrary.

Potentially this is related to #1755? However I was only using ~9k keys which where ~100 characters each which should be under the 4mb limit.

@lesv

So I believe in v1 you weren't allowed to send more than 1mb worth of keys (after serialization). I'm not sure if such a limit exists in v2 however.

/cc @lesv

@lesv

@lesv any insight on this one? Thanks!

@callmehiphop

@callmehiphop anyone else from Bigtable you can ping for us?

@garye

/cc @garye @sduskis

I don't know enough about this case to give a clear answer. Is the table populated with data for each of the keys? Has someone tried to reproduce this case? There isn't a known issue for the service failing, but I can to reproduce this with java and see if it succeeds.

@callmehiphop

@callmehiphop wdyt?

@arbesfeld

@arbesfeld are you still seeing this?

I'm not sure - I did a workaround by making requests to Bigtable with smaller chunks of keys and that seemed to fix the issue.

Cool, I think we can close this out. We have a note in our docs advising to chunk up requests as well.

feat: accept google-gax instance as a parameter (#1757)

chore: pin testbench version (#1757)

chore: pin testbench version (googleapis#1757)

fix: Correct project id on client side metrics by avoiding getProject…

Navigation Menu

Repository navigation

BigTableInstance.getRows sometimes returns 0 values silently #1757

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BigTableInstance.getRows sometimes returns 0 values silently #1757

Description

Activity

callmehiphop commented on Oct 31, 2016

stephenplusplus commented on Nov 7, 2016

stephenplusplus commented on Nov 14, 2016

callmehiphop commented on Nov 16, 2016

sduskis commented on Nov 16, 2016

stephenplusplus commented on Nov 29, 2016

callmehiphop commented on Nov 29, 2016

arbesfeld commented on Nov 29, 2016

stephenplusplus commented on Nov 29, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions