xds: Initial Request on the ADS Stream contains only one resource

[František Bořánek]

Hi,

I don't know if such behaviour is a bug.

I have server communicates with 7 services configured via XDS and after the xds-client got GO AWAY packet from XDS server 6 channels cannot connect due to error and only one is ok. I did a tcpdump to see a chat between our implementation of XDS and C++gRPC and cannot deside if C++gRPC behave correct.

It relates to https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md#initial-request-on-the-ads-stream

 >      The first request on the ADS stream will include a Node message that identifies the client. As mentioned above, most of the fields for the Node message will be read from the bootstrap file.

I would expect that after reconnect the xds client sends to ADS stream 4 requests (LDS, CDS, EDS, RDS) however it sends 5 requests (LDS, LDS, CDS, EDS, RDS). On our side of xds server we are confused by these 2 LDS requests.

1.

(PROTOBUF) envoy.api.v2.DiscoveryRequest

{

version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f"

node: {

cluster: "default-cluster"

user_agent_name: "gRPC C-core linux"

user_agent_version: "C-core 22.0.0"

client_features: "envoy.lb.does_not_support_overprovisioning"

build_version: "gRPC C-core linux 22.0.0"

}

resource_names: "server-1"

type_url: "type.googleapis.com/envoy.api.v2.Listener"

}

2.

(PROTOBUF) envoy.api.v2.DiscoveryRequest

{

version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f"

resource_names: "server-1"

resource_names: "server-2"

resource_names: "server-3"

resource_names: "server-4"

resource_names: "server-5"

resource_names: "server-6"

resource_names: "server-7"

type_url: "type.googleapis.com/envoy.api.v2.Listener"

}

- Shouldn't it be only one LDS request?

- Shouldn't it the initial request contains all configured resources?

Root of our problem is that we responds only on first request since it have the same version. I know how to fix it on our end, but it is behaviour of gRPC xds-client correct?

[Mark D. Roth]

The gRPC client does not know a priori what set of resource names are available on the xDS server, and even if it did, it would not request all of them proactively, because it may not actually need all of them.  Instead, each time a gRPC channel is created with an "xds:" target URI, that tells gRPC which LDS resource to request, and it requests the resources as they are needed.  Because the application can create new channels at any time, the set of resources requested on the ADS stream can change at any time.

If your application immediately creates 7 channels when it starts up, then the pattern you're seeing is expected.  When the first channel (presumably for "xds:server-1") starts connecting, it will immediately subscribe to the corresponding LDS resource ("server-1").  While it's waiting for that message to be sent, it notices that the other 6 channels have started up, so it realizes that it has to subscribe to the other 6 resources.  By the time the initial message is sent and it's able to send a second message, it knows about all 7 subscriptions, so the second message contains all 7 of them.

Note that once your application receives all 7 resources from the xDS server, if the ADS stream is broken and the client reestablishes it at that point, it should immediately subscribe to all 7 resources in the initial message on the new stream.  That is because it already knows about all of the subscriptions at the point at which it sends the initial message on the new stream.

This communication is fully legal and should work fine with any xDS-compliant server.  As described in this section of the xDS spec, the intent of the version field in the DiscoveryRequest is to provide the server with a way to know when it sends a version that the client considers invalid.  But if the client sends a second DiscoveryRequest with the same version and a different list of resource names, that indicates that it is changing what resources it is subscribing to, and the server cannot ignore that just because the version is the same as one it has previously seen.

I hope this information is helpful.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/e2a5befd-5943-4b40-afc4-4864938b9b88n%40googlegroups.com.

--

Mark D. Roth <ro...@google.com>
Software Engineer
Google, Inc.

[František Bořánek]

It makes sense.

However, in this case, the initial connection was already made and this tcpdump samples recovering of ADS stream. And as you write:

> Note that once your application receives all 7 resources from the xDS server, if the ADS stream is broken and the client reestablishes it at that point, it should immediately subscribe to all 7 resources in the initial message on the new stream.

It behaves as I described. First 1 resource and then 7 resources. Note that the messages have version_info. Brand new channels don't have version_info.

But as you pointed, the channels can be created anytime, thus it's valid bahaviur.

Thanks for replying.

F.

16. 3. 2022 v 19:49, Mark D. Roth <ro...@google.com>:

[Mark D. Roth]

It is valid behavior, but I agree that it's a little sub-optimal for the client to send two messages when it reestablishes the stream.  What language are you using gRPC in?

[František Bořánek]

C++, the latest 1.44.0 release. 

16. 3. 2022 v 21:32, Mark D. Roth <ro...@google.com>:



[Mark D. Roth]

That's very interesting.  Can you run with the following environment variables and send me the log?

GRPC_VERBOSITY=DEBUG GRPC_TRACE=xds_client,xds_resolver

[František Bořánek]

It's not exactly the same version. This one is gRPC 1.43.2, but behaviour do not vary form 1.44.0.

I attached also initial start up, in case you want to compare. Reconnection of xds contains file goaway.from.xds.cpp.grpc.log. 

The lifetime of channels are until shutdown of the application.

[Mark D. Roth]

Thanks for reporting this!  I think I see the problem, and it should be fixed by https://github.com/grpc/grpc/pull/29144.

I don't see a trivial way to write a test for it, since the current behavior is not broken, just slightly sub-optimal.  But feel free to try out the patch and let me know if it fixes the problem.

17. 3. 2022 v 16:52, Mark D. Roth <ro...@google.com>:

[František Bořánek]

Works like a charm. All current channels are included in a first message during reconnection.

Thanks very much.

Here is from debug log:

I0317 23:08:14.625789077      49 xds_client.cc:1287]         [xds_client 0x7f89b40352a0] xds server xds-control-plane:10000: ADS call status received (chand=0x7f89b4042270, ads_calld=0x7f89f00fb600, call=0x7f89d0022e70): status=1, details='Received RST_STREAM with error code 8', error='"OK"'

I0317 23:08:14.625866170      49 xds_client.cc:685]          [xds_client 0x7f89b40352a0] xds server xds-control-plane:10000: start new call from retryable call 0x7f89b4189aa0

I0317 23:08:14.625891228      49 xds_client.cc:938]          [xds_client 0x7f89b40352a0] xds server xds-control-plane:10000: starting ADS call (calld: 0x7f89b8095930, call: 0x7f89b81801f0)

D0317 23:08:14.626050477      49 xds_api.cc:257]             [xds_client 0x7f89b40352a0] constructed ADS request: version_info: "8cbd8378-4243-4cc7-9a2f-1f075055a469"

node: {

  cluster: "default-cluster"

  user_agent_name: "gRPC C-core linux"

  user_agent_version: "C-core 22.0.0"

  client_features: "envoy.lb.does_not_support_overprovisioning"

  5: "gRPC C-core linux 22.0.0"

}

resource_names: "server-1"

resource_names: "server-2"

resource_names: "server-3"

resource_names: "server-4"

resource_names: "server-5"

resource_names: "server-6"

resource_names: "server-7"

type_url: "type.googleapis.com/envoy.api.v2.Listener"

I0317 23:08:14.626073893      49 xds_client.cc:1055]         [xds_client 0x7f89b40352a0] xds server xds-control-plane:10000: sending ADS request: type=envoy.config.listener.v3.Listener version=8cbd8378-4243-4cc7-9a2f-1f075055a469 nonce= error="OK"

17. 3. 2022 v 21:56, Mark D. Roth <ro...@google.com>:

[Mark D. Roth]

Thanks for confirming!  I've just merged the PR, so this change should be in 1.46.