Node group based XDS routing

[Lukáš Drbal]

Hello everyone,

We are trying to setup routing via XDS to our GRPC services. Routing should be based on `node.cluster` information provided from client.

Basically we would like to have 2 groups of GRPC clusters (priority and normal) with same endpoints and choose right one by client `node.cluster` identification.

I have very minimal setup [1] which works absolutely as we expected for java client but doesn't work for C++ (and grpc_cli). Node hashing implementation [2]. This is minimal setup to reproducing this behaviour, regular routing is more complicated.

From log perspective it looks like from C++ xds server receive `node.cluster` information just in first request. 

From java I see cluster in all requests:
grpc-default-executor-0] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-1] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-1] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-0] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-1] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-2] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-1] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-2] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group.

But from cli / c++ I see cluster just in first request:
[grpc-default-executor-0] INFO org.example.xds.routing.XdsServer - Routing [priority] to priority group. [grpc-default-executor-0] INFO org.example.xds.routing.XdsServer - Routing [] to normal group. [grpc-default-executor-0] INFO org.example.xds.routing.XdsServer - Routing [] to normal group.

This leads to expected error when c++ client is trying to get priority listeners and routes from default group.

Can somebody give me any hint what's wrong here?

Thanks a lot!

L.

[1] https://github.com/LesTR/xds-routing-test

[2] https://github.com/LesTR/xds-routing-test/blob/master/src/main/java/org/example/xds/routing/XdsServer.java#L61

[Mark D. Roth]

The xDS protocol does not require the node information to be sent by the client for every request on the stream; the client needs to send it only on the first request on the stream.  Quoting this section of the xDS spec:

Only the first request on a stream is guaranteed to carry the node identifier. The subsequent discovery requests on the same stream may carry an empty node identifier. This holds true regardless of the acceptance of the discovery responses on the same stream. The node identifier should always be identical if present more than once on the stream. It is sufficient to only check the first message for the node identifier as a result.

The Java implementation may currently happen to send the node information with every request on the stream, but it's not required to do that, and your xDS server should not expect that behavior.  I think you need to change your xDS server to look at the node information on the first request on the stream and store the node.cluster field so that it knows the value when it sees subsequent requests on the same stream.

I hope this information is helpful.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1de7e140-862f-414f-b25a-7b1afc4069can%40googlegroups.com.

--

Mark D. Roth <ro...@google.com>
Software Engineer
Google, Inc.

[Lukáš Drbal]

Hello Mark,

as first thanks a lot for Your replay, it is more clear for me now.

Do you have any idea how to distinguish connected clients? I was looking for some information which I can use but I don't see anything usable in NodeGroup interface [1]. It provides access just for `Node` protobuf object but I don't see anything usable there.

Any idea or example?

Thanks!

 
[1] https://github.com/envoyproxy/java-control-plane/blob/main/cache/src/main/java/io/envoyproxy/controlplane/cache/NodeGroup.java

[Mark D. Roth]

I am not familiar with java-control-plane, so I can't answer that.  You might try asking in their developer community.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/19380de0-9db6-4095-a120-ae40c453a9e8n%40googlegroups.com.

[Lukáš Drbal]

Ach, sorry, my bad.
Thanks.

L.

[Lukáš Drbal]

Just in case somebody get into same issue and found this thread:

 

I was able to connect all discovery requests from client by adding server interceptor where I marked all requests with client socket information and reuse it later in node hashing.

Details can be found in this commit [1] in my test repo.

Once again thanks for help.

L.

[1] https://github.com/LesTR/xds-routing-test/commit/ebb1a00c39b9ab9db7868f1857ca5166d8e3748e