Repository navigation
ComputeEngineChannelBuilder fails on App Engine #7604
Description
Activity
I'm not that familiar with Direct Path or App Engine, but my reading of this is that it sounds like it's a problem with the change in java-bigtable and not with gRPC itself. cc @dapengzhang0, who might have more knowledge of gRPC + Direct Path.
I think the issue would be the same if another client library activated DirectPath by calling
InstantiatingGrpcChannelProvider.Builder.setAttemptDirectPath(true).If I read the code path correctly, the following will be done:
// from com.google.api.gax.grpc.InstantiatingGrpcChannelProvider#createSingleChannel if (isDirectPathEnabled(serviceAddress) && credentials instanceof ComputeEngineCredentials) { // It is now assumed we run on Compute Engine because we have an instance of ComputeEngineCredentials // but these appear on App Engine, too. // from io.grpc.alts.ComputeEngineChannelBuilder if (!CheckGcpEnvironment.isOnGcp()) { status = Status.INTERNAL.withDescription("Compute Engine Credentials can only be used on Google Cloud Platform"); } }
I think the fault here is to assume that the presence of
ComputeEngineCredentialslead to the presence of/sys/class/dmi/id/product_namewhich is not the case on App Engine.Thanks for the additional info (and the very detailed original report as well). I think I understand the issue better now: the gax + java-bigtable changes now default to directly using
ComputeEngineChannelBuilderhere even on App Engine, and ourComputeEngineChannelCredentialsthen objects, saying that it can only be used on GCP.@WeiranFang Is the change to
ComputeEngineChannelBuilderwhen direct path is enabled intended to also run on App Engine?if (isDirectPathEnabled(serviceAddress) && credentials instanceof ComputeEngineCredentials)
For client to use
ComputeEngineChannelBuilder, the credentials has to be an instance of theComputeEngineCredentials. So there's probably an assumption thatComputeEngineCredentialscan only be used on GCE. IfComputeEngineCredentialscan also be used on App Engine, we might have an incorrect assumption.@apolcyn WDYT?
cc @mohanli-mlFor client to use ComputeEngineChannelBuilder, the credentials has to be an instance of the ComputeEngineCredentials. So there's probably an assumption that ComputeEngineCredentials can only be used on GCE. If ComputeEngineCredentials can also be used on App Engine, we might have an incorrect assumption.
I agree it sounds like the decision to use ComputeEngineChannelBuilder may be overly aggressive in this case.
Note that
ComputeEngineCredentialsbasically requires availability of the metadata server endpoint serving the service account's tokens. Meanwhile,ComputeEngineChannelBuilderrequires/sys/class/dmi/id/product_nameto be available. And it looks like in this case, on AppEngine, the former may be available without the latter, breaking the logic.I'll note that gRPC originally wanted the client libraries to use
GoogleDefaultChannelBuilderwhich would use the appropriate method for the platform. We addedComputeEngineChannelBuilderfor the client libraries to fail when not used on GCE, as that is what was requested by the client libraries. So this appears to be working as intended.Hey Jan, the fix PR has been merged and gax-java 1.60.1 has been released. Can you test if the bug is fixed? Thanks! @janhicken
Perfect, I'm going to do that on Monday and give you a heads up
I first tested with gax-java at
1.60.0and google-cloud-bigtable at1.17.0which reproduced the original error.When still using google-cloud-bigtable at
1.17.0and upgrading gax-java to1.60.1, the issue is gone 👍I think it's safe to close this issue then, thanks to everyone for the quick responses!
Hey Jan, thanks for the tests! Since release is frozen because of thanksgiving this week, we will try to have a new release of bigtable next week.
- locked as resolved and limited conversation to collaborators
on Jun 3, 2021
What version of gRPC-Java are you using?
I'm using version
1.33.1.What is your environment?
App Engine Standard Environment using Java 11.
What did you expect to see?
I am using the Java Bigtable client which sets up a gRPC connection to the Bigtable service. In GAE Java 11, the application default credentials are an instance of
ComputeEngineCredentials, which talk to the project's metadata server to obtain auth tokens. This works fine for many client libraries, however the Bigtable client uses DirectPath since version 1.17.0 by default. This code path includes the usage of gRPC'sComputeEngineChannelBuilder. I would expect this to work on App Engine.What did you see instead?
As the file
/sys/class/dmi/id/product_nameis not available on App Engine Standard instances, an internal error will be produced resulting in an exception when making gRPC calls.Steps to reproduce the bug
Use the Bigtable Java client with version
>= 1.17.0. Create a client and read an example row:This results in the following stack trace:
Moreover, the following warning will be printed to the log: