Skip to content

gRPC max size limit #1991

Description

@pyhrus

Since grpc version 1.0.0, message size is limit 4MB, and pubsub itself supports payload size upto 10MB. If there is message that is greater than 4MB in queue, pubsub client errors out with Error: Received message larger than max (6374241 vs. 4194304)

Environment details

  • OS: Debian Jessie
  • Node.js version: 6.3.1
  • npm version:3.10.3
  • google-cloud-node version: 0.8.0

Steps to reproduce

  1. Publish message greater than 4MB using HTTP API
  2. Use pubsub client to pull the message

Thanks!

Activity

  1. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    api: pubsubIssues related to the Pub/Sub API.
    on Feb 14, 2017
  2. stephenplusplus commented on Feb 16, 2017

    @stephenplusplus
    Contributor

    @murgatroid99 @jmuk do you know if there's a way to handle this?

  3. jmuk commented on Feb 16, 2017

    @jmuk
    Contributor

    It's really obscure to me how to customize the parameters like this. Maybe the third parameter to the Client constructor might be used (https://github.com/grpc/grpc/blob/master/src/node/src/client.js#L732) but I don't know what values or structures can configure this.

    @murgatroid99, do you know which keys/values we should use here?

  4. murgatroid99 commented on Feb 16, 2017

    @murgatroid99

    The keys and values are defined at https://github.com/grpc/grpc/blob/master/include/grpc/impl/codegen/grpc_types.h#L148. The one you want for this case is "grpc.max_send_message_length".

  5. jmuk commented on Feb 16, 2017

    @jmuk
    Contributor

    Thanks!

    I also filed googleapis/gax-nodejs#107 for this problem.

  6. stephenplusplus commented on Feb 17, 2017

    @stephenplusplus
    Contributor

    PR sent to remove the limit of the max message sent length and max message received length: #2007

  7. arbesfeld commented on Jun 8, 2017

    @arbesfeld
    Contributor

    We still seem to be hitting this 4mb limit while testing with the following versions:

        "@google-cloud/bigtable": "0.10.1",
        "@google-cloud/pubsub": "0.11.0",
        "@google-cloud/storage": "1.1.0"
    

    Any recommendation for how to avoid this GRPC limit?

  8. Michael-Jing commented on Dec 26, 2017

    @Michael-Jing

    I am using the grpcio python lib, and encounter the same issue, any idea to get rid of this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

api: pubsubIssues related to the Pub/Sub API.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions