Unhandled exception from gRPC for request size

@matanh-tzmedical

From @matanh-tzmedical on November 13, 2017 21:18

When trying to load test the our kubernetes stack by sending thousands of requests to our exposed endpoint I get this error which crashes the Node app:

{ Error: Request payload size exceeds the limit: 10485760 bytes. at /var/www/node_modules/grpc/src/client.js:554:15 code: 3, metadata: Metadata { _internal_repr: {} } }

It seems like the 1MB buffering that the @google-cloud/logging library is supposed to be doing is not happening.

Environment details

OS: Google Cloud Container Cluster - Ubuntu Latest Node image
Node.js version: Docker public image - node:6.10.3
npm version: 3.10.10
@google-cloud/logging-bunyan version: 0.7.0

Steps to reproduce

Enable stackdriver logging on a Google Cloud Kubernetes app.
Log thousands of requests as fast as possible.

Copied from original issue: googleapis/nodejs-logging-bunyan#14

@ofrobots

From @ofrobots on November 13, 2017 23:42

/cc @landrito @lukesneeringer @murgatroid99: Any guesses if this could be an issue in gRPC or in google-gax?

@murgatroid99

From @murgatroid99 on November 15, 2017 0:4

This is a gRPC thing. By default, gRPC sets a limit on the size of (serialized) messages that can be sent or received. This can be modified with the channel arguments "grpc.max_receive_message_length" and "grpc.max_send_message_length", which are defined here. A value of -1 allows for messages of unlimited size in both cases.

@ofrobots

From @ofrobots on November 15, 2017 1:58

So it seems that the google-cloud/common-grpc library sets the defaults to:

var GRPC_SERVICE_OPTIONS = {
  // RE: https://github.com/GoogleCloudPlatform/google-cloud-node/issues/1991
  'grpc.max_send_message_length': -1, // unlimited
  'grpc.max_receive_message_length': -1, // unlimited

  // RE: https://github.com/grpc/grpc/issues/8839
  // RE: https://github.com/grpc/grpc/issues/8382
  // RE: https://github.com/GoogleCloudPlatform/google-cloud-node/issues/1991
  'grpc.initial_reconnect_backoff_ms': 5000
};

So, that is already as expected. It would seem that somehow this is not taking effect.

@ofrobots

From @ofrobots on November 15, 2017 2:19

@stephenplusplus @lukesneeringer It seems that logging may be using the google-gax path. would this config in common-grpc apply at all? I don't see google-gax doing anything with the size limits.

@ofrobots

From @ofrobots on November 15, 2017 4:7

Repro

const bunyan = require('bunyan');
const LB = require('@google-cloud/logging-bunyan');

const lb = new LB({projectId: 'a real project id'});

const logger = bunyan.createLogger({
  name: 'my-service',
  streams: [
    lb.stream()
  ]
});

for (let i = 0; i < 100; i++) {
  logger.info({buffer: new Int8Array(56150)}, 'shields at 99%');
}

google-gax doesn't set any of these defaults, as far as I know. It should probably just copy the settings in common-grpc.

For now, this should work:

const lb = new LB({
  projectId: 'a real project id',
  'grpc.max_send_message_length': -1,
  'grpc.max_receive_message_length': -1,
  'grpc.initial_reconnect_backoff_ms': 5000
});

@ofrobots

From @ofrobots on November 15, 2017 18:31

@stephenplusplus that didn't work actually.

I think this issue belongs upstream in logging. Or perhaps more generally in all gax based in libraries given that there are known issues that caused us to customize the config in common-grpc. It doesn't make sense that we are using different network config for common-grpc libraries vs. gax based libraries.

I am going to move this issue to logging for now.

I don't have any rights on this repo, but consider me assigned to this issue.

@landrito

@landrito @ofrobots @lukesneeringer -- should these defaults be set in GAX or in toolkit?

@lukesneeringer

@lukesneeringer and @alexander-fenster?

Given that there is already case precedent to set max send and receive length is set to be unlimited in google-cloud/common-grpc, I'm fine with setting these defaults in gax.

I'd like to see if this is an important knob to open up to API producers so we can have the default set at client library generation, I'll poll the other languages to see if this is relevant. For now, lets set these in gax.

Since you're here, any head start on where I can make this change?

@stephenplusplus

It would be set in this method: https://github.com/googleapis/gax-nodejs/blob/master/lib/grpc.js#L236

@stephenplusplus any idea why this was not fixed by:

const lb = new LB({
  projectId: 'a real project id',
  'grpc.max_send_message_length': -1,
  'grpc.max_receive_message_length': -1,
  'grpc.initial_reconnect_backoff_ms': 5000
});

I see the same errors even with those setting.

I added a log line here (https://github.com/googleapis/gax-nodejs/blob/0d60fae9b22ac5c53d6eeb45b7ff869ec28ca9c4/lib/grpc.js#L236) and confirmed the grpc settings were recognized by gax:

{ 'grpc.max_send_message_length': -1,
  'grpc.max_receive_message_length': -1,
  'grpc.initial_reconnect_backoff_ms': 5000 }

@alexander-fenster

@alexander-fenster Any updates?

@alexander-fenster

@alexander-fenster any updates on this issue? Thanks!

@alexander-fenster

@alexander-fenster Any updates?

@alexander-fenster

@alexander-fenster ping.

Looking at this old issue... gRPC options can be set in the client constructor and will be passed down to gRPC. We had recently some changes related to that in googleapis/gax-nodejs#328. Closing this for now. If grpc options are not passed down to gRPC correctly please create a new issue.

Navigation Menu

Repository navigation

Unhandled exception from gRPC for request size #480

Environment details

Steps to reproduce

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unhandled exception from gRPC for request size #480

Description

Environment details

Steps to reproduce

Activity

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 16, 2017

stephenplusplus commented on Nov 17, 2017

ofrobots commented on Nov 17, 2017

landrito commented on Nov 17, 2017

stephenplusplus commented on Nov 17, 2017

landrito commented on Nov 17, 2017

stephenplusplus commented on Nov 17, 2017

vchudnov-g commented on Nov 10, 2018

noahdietz commented on Nov 19, 2018

jbolinger commented on Dec 10, 2018

landrito commented on Jan 2, 2019

alexander-fenster commented on Jan 3, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions