xds: support istio grpc-agent

I tried to use xDS with istio grpc-agent. But it doesn't work well.

Is your feature request related to a problem?

xDS doesn't work with istio grpc-agent because not supported NameResolver of unix:///
#4750

Then I write a simple NameResolverProvider that it just returns DomainSocketAddress, but it doesn't work.

Feb 02, 2022 7:56:38 PM io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<8>: (xds:///xxx.core.svc.cluster.local:9000)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=NameResolver returned no usable address. addrs=[], attrs={}
io.grpc.xds.XdsNameResolver@7c6c48cc was used, cause=null}

Describe the solution you'd like

Enable xDS to handle unix domain socket with istio grpc-agent like grpc-go.

Additional context

https://istio.io/latest/blog/2021/proxyless-grpc/

I didn't see any references to unix domain socket in the additional context link you cited so I didn't see how it is related.

Are you saying the istio control plane returned unix: addresses as endpoints and the gRPC code dropped them even before using your NameResolver for UDS?

@naoh87

Drat. Well, I guess this will be a forcing function to finishing #4750 and its dependencies. We've slowly been moving in that direction, so luckily the ChannelCredentials paved the way partly already.

Then I write a simple NameResolverProvider that it just returns DomainSocketAddress, but it doesn't work.

That should have worked. We know that would fail if OkHttp were in the mix or if Netty wasn't using epoll, and there are issues with grpc-netty vs grpc-netty-shaded. But you didn't hit any of those (yet) from the error. That error claims your name resolver didn't provide any addresses at all. @naoh87, can you paste code that prepares the data to return to the listener? I think there is something simple going on.

@sanjaypujare, FYI, fixing this means fixing #3085 which means methods like ServerProvider.provider() will no longer be used at all by gRPC, as we'll always need to be prepared to try multiple transports. We can leave it in place temporarily, but it will need to be deleted eventually.

@sanjaypujare, Istio uses a control plane proxy in the local environment and the client connects to it over UDS. See this diagram from the blog post: https://istio.io/latest/blog/2021/proxyless-grpc/architecture.svg

@sanjaypujare

...
@sanjaypujare, Istio uses a control plane proxy in the local environment and the client connects to it over UDS. See this diagram from the blog post: https://istio.io/latest/blog/2021/proxyless-grpc/architecture.svg

Thanks, I did search for "UDS" on that page :-) Should be a good feature in Chrome (and other browsers) to OCR the embedded images in the text search.

I'll take a look, we may need to use TCP for java.

@costinm

@costinm, I should have noticed this earlier when looking at that blog post. It would certainly need a hack today (like making your own NR). If TCP is available, that'd be great. But it also seems like it is time for us to get the plumbing to allow UDS.

The NR itself is very easy and would be relatively (compared to others) small. But without the plumbing it would make multiple (mostly right) assumptions about the environment.

Yes, java support for UDS would be very nice.

Unfortunately the XDS proxy in the agent only support UDS, but at least for testing it should be possible to use a custom bootstrap file that connects directly to istiod.istio-system on the plain text port. The main purpose of the XDS proxy in the agent is to deal with the custom certificate and auth - ideally we shouldn't need it, but right now in Istio we just have too many ways to authenticate and get certificates.

@ejona86

Sorry for my late reply @ejona86

can you paste code that prepares the data to return to the listener?

I tried this NR just call onAddress in start.

listener.onAddresses(
  List.of(new EquivalentAddressGroup(new DomainSocketAddress(targetUri.getPath()))),
  Attributes.EMPTY)

And I tried also add or not add epoll dependency. Both case cause same result.

"io.netty" % "netty-transport-native-epoll" % "4.1.73.Final" classifier "linux-x86_64"

It may not matter I used grpc-java from Scala.

Then I write a simple NameResolverProvider that it just returns DomainSocketAddress, but it doesn't work.

That should have worked.

Will it be enough with only a different NameResolverProvider? Because UDS requires a different socket channel as well (EpollDomainSocketChannel).

Now I'm confused. 😅 I asked if it was enough with only a different NameResolverProvider, and you said yes, showing that SdsClient is already providing an EpollDomainSocketChannel...

...
Will it be enough with only a different NameResolverProvider? Because UDS requires a different socket channel as well (EpollDomainSocketChannel).

yes, ...

But then I do actually need a ManagedChannelProvider as well?

Now I'm confused. 😅 I asked if it was enough with only a different NameResolverProvider, and you said yes, showing that SdsClient is already providing an EpollDomainSocketChannel...

SdsClient has been removed and is a different client (not the XdsClient that we want here). My link was to an older version (1.40.x) if you notice...

...
Will it be enough with only a different NameResolverProvider? Because UDS requires a different socket channel as well (EpollDomainSocketChannel).

yes, ...

But then I do actually need a ManagedChannelProvider as well?

Looks like that because that's the only one that can provide the channelType we were talking about.

@ejona86

Thanks, now it's clear.
Something like this seems to work. but like @ejona86 says it's a bit brittle.
What would a "proper" solution be? I may attempt a PR.

@ejona86

Thanks, now it's clear. Something like this seems to work.

But you also need to register this in ManagedChannelRegistry and would you do that thru the Service Provider interface?

but like @ejona86 says it's a bit brittle. What would a "proper" solution be? I may attempt a PR.

I'll let @ejona86 answer that. It will be good if you can contribute a PR - since this functionality is sorely needed.

@ejona86

But you also need to register this in ManagedChannelRegistry and would you do that thru the Service Provider interface?

Yes.

I'll let @ejona86 answer that. It will be good if you can contribute a PR - since this functionality is sorely needed.

A partly related question; since this will give the client the capability to connect over UDS, should the same be done for server as well? I.e. something like this would be needed somewhere:

NettyServerBuilder.forAddress(new DomainSocketAddress(path))
  .channelType(EpollServerDomainSocketChannel.class)
  ...

...
A partly related question; since this will give the client the capability to connect over UDS, should the same be done for server as well? I.e. something like this would be needed somewhere:
NettyServerBuilder.forAddress(new DomainSocketAddress(path))
  .channelType(EpollServerDomainSocketChannel.class)
  ...

"client" in this context can be confusing. In this case gRPC (the library) only needs the XdsClient to be able to instantiate a gRPC client over UDS to talk to the Istio agent (which is the XDS server although just a proxy). Even if one is creating an XDS managed gRPC server (check out io.grpc.xds.XdsServerBuilder) it still needs only the XdsClient (containing a gRPC client) to talk to the Istio agent (over UDS in this case).

So something like what you are describing i.e. creating a NettyServer using UDS to receive incoming RPCs is not needed.

So something like what you are describing i.e. creating a NettyServer using UDS to receive incoming RPCs is not needed.

Well it wouldn't be needed for the issue discussed here (xDS over UDS), but the fixes discussed here would also allow any other grpc-java client to connect to any server over UDS.

So my question is rather if there should be an easier way to create a server for UDS? As this will allow out-of-the-box support for creating both a client and server over UDS. E.g. something like?

Grpc.newServerBuilderForAddress("unix:/foo.bar", InsecureServerCredentials.create())

...

So my question is rather if there should be an easier way to create a server for UDS? As this will allow out-of-the-box support for creating both a client and server over UDS. E.g. something like?
Grpc.newServerBuilderForAddress("unix:/foo.bar", InsecureServerCredentials.create())

Okay I see what you are saying. But:

at least I can't think of an easier way or a common change that gives us UDS support for both client & server
UDS on the client side is more urgent whereas we don't immediately need UDS on the server side.

@ejona86

but like @ejona86 says it's a bit brittle. What would a "proper" solution be? I may attempt a PR.

I'll let @ejona86 answer that. It will be good if you can contribute a PR - since this functionality is sorely needed.

@ejona86 any thoughts?

@sanjaypujare

The design for #3085 is essentially add Collection<SocketAddress> getSupportedSocketAddresses() to NameResolverProvider and ManagedChannelProvider. ManagedChannelRegistry would use the target string to determine which NameResolverProvider would be used. It will then loop through ManagedChannelProviders in order until one supports all the addresses returned by the NameResolverProvider. And then for this issue we add unix name resolver and a unix variant for Netty.

I think @sanjaypujare was looking at implementing this, although I'd imagine he'd be happy handing it off. He has some stuff written up in a random doc; he could potentially share it with you.

@cfredri4

@cfredri4 if you are interested in submitting a PR I can share my doc with you. Let me know

Navigation Menu

Repository navigation

xds: support istio grpc-agent #8884

Is your feature request related to a problem?

Describe the solution you'd like

Additional context

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

xds: support istio grpc-agent #8884

Description

Is your feature request related to a problem?

Describe the solution you'd like

Additional context

Activity

sanjaypujare commented on Feb 3, 2022

ejona86 commented on Feb 3, 2022

sanjaypujare commented on Feb 3, 2022

costinm commented on Feb 3, 2022

ejona86 commented on Feb 3, 2022

costinm commented on Feb 3, 2022

naoh87 commented on Feb 7, 2022

cfredri4 commented on Mar 8, 2022

8 remaining items

cfredri4 commented on Mar 8, 2022

sanjaypujare commented on Mar 8, 2022

cfredri4 commented on Mar 9, 2022

sanjaypujare commented on Mar 9, 2022

cfredri4 commented on Mar 10, 2022

sanjaypujare commented on Mar 10, 2022

cfredri4 commented on Mar 10, 2022

sanjaypujare commented on Mar 10, 2022

cfredri4 commented on Mar 23, 2022

ejona86 commented on Mar 23, 2022

sanjaypujare commented on Mar 23, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions