Using xDs as a generic control plane control plane protocol

[Wenjie Zeng]

Hi,

The xDs protocol as a transport is pretty generic -- it can be used to deliver any type of dynamic configs using a streaming API with delta semantics. I was wondering if anyone here has experience using xDs to deliver configs beyond those for Envoy and gRpc.

https://github.com/cncf/xds splits xDs into a transport protocol and data model. The data model defines a data models common for data plane stuff. Therefore, if I have an xDs server deployed, I think of it performing two orthogonal function: (1) the generic xDs transport protocol, and (2) the Envoy/gRpc specific logic to organize/arrange data plane data models (such as dependency graph, delivery order, etc).

If I have other clients that need dynamic config types beyond Envoy/gRpc ones, is it sensible to reuse the transport logic in the xDs server to deliver those other config types? With the right internal architecture, I feel that it's possible to decouple the transport and data model code in the xDs server.

Does anyone have experience using xDs as a generic transport protocol? Any open-source xDs server that are more generic xDs implementation than the reference implementations https://github.com/envoyproxy/go-control-plane or https://github.com/envoyproxy/java-control-plane?

[Jakub Dyszkiewicz]

Hey Wenjie,

we use xDS with go-control-plane to exchange resources between Global CP and Zone CP in Kuma.

We figured out that we have very similar requirements of CP<->CP communication to CP<->Envoy. On top of that, our policies in Kuma are defined using protocol buffers.

The only hard part was that CP->Envoy is one-way communication. In our Global CP <-> Zone CP case, the majority of policies are also synced one way (from Global to Zone).

However, there are some objects that have to be synced from Zone CP to Global CP, mostly for visibility.

With the first implementation, we initiated connections both ways which was an operational nightmare.

We ended up creating a mux service https://github.com/kumahq/kuma/blob/master/api/mesh/v1alpha1/mux.proto that creates logical streams both ways.

https://github.com/kumahq/kuma/blob/master/api/mesh/v1alpha1/kds.proto

You can find an implementation here https://github.com/kumahq/kuma/tree/master/pkg/kds although because of muxing, it's not straightforward.

We are considering simplifying it to something like this https://github.com/kumahq/kuma/issues/3789#issuecomment-1108216355

We also use xDS to serve a list of endpoints to scrape for Prometheus. This one is way simpler to analyze because it's one way

https://github.com/kumahq/kuma/blob/master/pkg/mads/v1/service/mads.go#L16

Let me know if you have more questions!

Jakub

[Wenjie Zeng]

Thank you so much Jakub for your detailed reply! Let me take a look at the links you've provided.