Investigating Sidecar Proxy Approaches

[Joey Bratton]

We're looking into different sidecar approaches for gRPC proxying to provide a common platform for things like service discovery, distributed tracing, etc. Our environment is mostly on the JVM, but we'd like to come up with a design that is polyglot-friendly and doesn't require us to rebuild all of the proxy's features in each new language.

Envoy and Linkerd are both options that are on the table, but I'm also interested in investigating the feasibility of building something new which is based directly on gRPC. In theory, I'd love to be able to build out all of those common proxy features as standard ClientInterceptors from grpc-java so that consumers which are built on the JVM could just create a "fat" client that includes those interceptors directly and a non-JVM consumer would use a "thin" client without any interceptors along with a grpc-java-based coprocess that includes the interceptors in the proxied request.

I've started working on a spike for that approach, but I wanted to get some feedback on the idea and make sure that I'm working with the right abstraction layers. Here are some of my current questions with the approach:

- Does the fallbackHandlerRegistry seem like the best extension point to use to create the catch-all method?

- Ideally the marshalling phase would effectively be a no-op since the proxy wouldn't care about concrete types, so does it make sense to just use Marshaller<InputStream> instances that just return the given instances for parsing and streaming?

- The intent is for the proxy to be unaware of the implementation details of the actual server that the request is being proxied to. Should we just assume the request is a bi-directional streaming request in the proxy in order to support all request/response types? Is that likely to cause a huge performance impact since the framework can't optimize for different request/response types?

- The proxy will potentially maintain connections to many different servers. I haven't dug into this part a ton yet, but I assume this would involve maintaining a TransportManager and using that to create out-of-band transports?

- This initial spike is towards a "client -> client co-process proxy -> server" message flow. Some interceptors may make more sense server-side, which would either lead to a "client -> server co-process proxy -> server" or a "client -> client co-process proxy -> server co-process proxy -> server" flow, but my initial hunch is that those are mainly just slight permutations of the same problem so they wouldn't need drastically different implementations. Does that seem like a reasonable assumption?

Thanks!

- Joey Bratton

[Carl Mastrangelo]

Responses inline

On Sunday, January 22, 2017 at 8:08:37 PM UTC-8, Joey Bratton wrote:

We're looking into different sidecar approaches for gRPC proxying to provide a common platform for things like service discovery, distributed tracing, etc. Our environment is mostly on the JVM, but we'd like to come up with a design that is polyglot-friendly and doesn't require us to rebuild all of the proxy's features in each new language.

Envoy and Linkerd are both options that are on the table, but I'm also interested in investigating the feasibility of building something new which is based directly on gRPC. In theory, I'd love to be able to build out all of those common proxy features as standard ClientInterceptors from grpc-java so that consumers which are built on the JVM could just create a "fat" client that includes those interceptors directly and a non-JVM consumer would use a "thin" client without any interceptors along with a grpc-java-based coprocess that includes the interceptors in the proxied request.

I've started working on a spike for that approach, but I wanted to get some feedback on the idea and make sure that I'm working with the right abstraction layers. Here are some of my current questions with the approach:

- Does the fallbackHandlerRegistry seem like the best extension point to use to create the catch-all method?

Yes.

 

- Ideally the marshalling phase would effectively be a no-op since the proxy wouldn't care about concrete types, so does it make sense to just use Marshaller<InputStream> instances that just return the given instances for parsing and streaming?

We use byte array marshallers (called ByteBufOutputMarshaller) to skip marshalling in benchmarks.  It is just a passthru.  That sounds like what you want.

 

- The intent is for the proxy to be unaware of the implementation details of the actual server that the request is being proxied to. Should we just assume the request is a bi-directional streaming request in the proxy in order to support all request/response types? Is that likely to cause a huge performance impact since the framework can't optimize for different request/response types?

Without knowing the type, you have to assume BIDI.  If you out-of-band knowledge that all client are UNARY, then you can make such optimizations.

 

- The proxy will potentially maintain connections to many different servers. I haven't dug into this part a ton yet, but I assume this would involve maintaining a TransportManager and using that to create out-of-band transports?

What do you mean by out-of-band transports?

 

- This initial spike is towards a "client -> client co-process proxy -> server" message flow. Some interceptors may make more sense server-side, which would either lead to a "client -> server co-process proxy -> server" or a "client -> client co-process proxy -> server co-process proxy -> server" flow, but my initial hunch is that those are mainly just slight permutations of the same problem so they wouldn't need drastically different implementations. Does that seem like a reasonable assumption?

Do you own all the clients and/or servers in your scenario?  That will change the answer to which configuration is the best.  The configuration we use commonly is that both client and server have a co-process proxy.

 

Thanks!

- Joey Bratton

[Joey Bratton]

Awesome, thanks for the quick response! Some additional questions/responses added inline below.

On Monday, January 23, 2017 at 1:31:55 PM UTC-5, Carl Mastrangelo wrote:

Responses inline

On Sunday, January 22, 2017 at 8:08:37 PM UTC-8, Joey Bratton wrote:

We're looking into different sidecar approaches for gRPC proxying to provide a common platform for things like service discovery, distributed tracing, etc. Our environment is mostly on the JVM, but we'd like to come up with a design that is polyglot-friendly and doesn't require us to rebuild all of the proxy's features in each new language.

Envoy and Linkerd are both options that are on the table, but I'm also interested in investigating the feasibility of building something new which is based directly on gRPC. In theory, I'd love to be able to build out all of those common proxy features as standard ClientInterceptors from grpc-java so that consumers which are built on the JVM could just create a "fat" client that includes those interceptors directly and a non-JVM consumer would use a "thin" client without any interceptors along with a grpc-java-based coprocess that includes the interceptors in the proxied request.

I've started working on a spike for that approach, but I wanted to get some feedback on the idea and make sure that I'm working with the right abstraction layers. Here are some of my current questions with the approach:

- Does the fallbackHandlerRegistry seem like the best extension point to use to create the catch-all method?

Yes.

 

- Ideally the marshalling phase would effectively be a no-op since the proxy wouldn't care about concrete types, so does it make sense to just use Marshaller<InputStream> instances that just return the given instances for parsing and streaming?

We use byte array marshallers (called ByteBufOutputMarshaller) to skip marshalling in benchmarks.  It is just a passthru.  That sounds like what you want.

 

- The intent is for the proxy to be unaware of the implementation details of the actual server that the request is being proxied to. Should we just assume the request is a bi-directional streaming request in the proxy in order to support all request/response types? Is that likely to cause a huge performance impact since the framework can't optimize for different request/response types?

Without knowing the type, you have to assume BIDI.  If you out-of-band knowledge that all client are UNARY, then you can make such optimizations.

 

- The proxy will potentially maintain connections to many different servers. I haven't dug into this part a ton yet, but I assume this would involve maintaining a TransportManager and using that to create out-of-band transports?

What do you mean by out-of-band transports?

https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/TransportManager.java#L77

 

 

- This initial spike is towards a "client -> client co-process proxy -> server" message flow. Some interceptors may make more sense server-side, which would either lead to a "client -> server co-process proxy -> server" or a "client -> client co-process proxy -> server co-process proxy -> server" flow, but my initial hunch is that those are mainly just slight permutations of the same problem so they wouldn't need drastically different implementations. Does that seem like a reasonable assumption?

Do you own all the clients and/or servers in your scenario?  That will change the answer to which configuration is the best.  The configuration we use commonly is that both client and server have a co-process proxy.

Is there any public info on the co-process proxy that you guys are using, or is that entirely an internal project?

 

 

Thanks!

- Joey Bratton

[Kun Zhang]

On Monday, January 23, 2017 at 11:26:20 AM UTC-8, Joey Bratton wrote:

Awesome, thanks for the quick response! Some additional questions/responses added inline below.

On Monday, January 23, 2017 at 1:31:55 PM UTC-5, Carl Mastrangelo wrote:

Responses inline

On Sunday, January 22, 2017 at 8:08:37 PM UTC-8, Joey Bratton wrote:

We're looking into different sidecar approaches for gRPC proxying to provide a common platform for things like service discovery, distributed tracing, etc. Our environment is mostly on the JVM, but we'd like to come up with a design that is polyglot-friendly and doesn't require us to rebuild all of the proxy's features in each new language.

Envoy and Linkerd are both options that are on the table, but I'm also interested in investigating the feasibility of building something new which is based directly on gRPC. In theory, I'd love to be able to build out all of those common proxy features as standard ClientInterceptors from grpc-java so that consumers which are built on the JVM could just create a "fat" client that includes those interceptors directly and a non-JVM consumer would use a "thin" client without any interceptors along with a grpc-java-based coprocess that includes the interceptors in the proxied request.

I've started working on a spike for that approach, but I wanted to get some feedback on the idea and make sure that I'm working with the right abstraction layers. Here are some of my current questions with the approach:

- Does the fallbackHandlerRegistry seem like the best extension point to use to create the catch-all method?

Yes.

 

- Ideally the marshalling phase would effectively be a no-op since the proxy wouldn't care about concrete types, so does it make sense to just use Marshaller<InputStream> instances that just return the given instances for parsing and streaming?

We use byte array marshallers (called ByteBufOutputMarshaller) to skip marshalling in benchmarks.  It is just a passthru.  That sounds like what you want.

 

- The intent is for the proxy to be unaware of the implementation details of the actual server that the request is being proxied to. Should we just assume the request is a bi-directional streaming request in the proxy in order to support all request/response types? Is that likely to cause a huge performance impact since the framework can't optimize for different request/response types?

Without knowing the type, you have to assume BIDI.  If you out-of-band knowledge that all client are UNARY, then you can make such optimizations.

 

- The proxy will potentially maintain connections to many different servers. I haven't dug into this part a ton yet, but I assume this would involve maintaining a TransportManager and using that to create out-of-band transports?

What do you mean by out-of-band transports?

https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/TransportManager.java#L77

The out-of-band transports are exclusively for LoadBalancer implementations to talk to an external balancer. It's not for your use case.

A straightforward approach would be creating a channel for each server.

 

 

- This initial spike is towards a "client -> client co-process proxy -> server" message flow. Some interceptors may make more sense server-side, which would either lead to a "client -> server co-process proxy -> server" or a "client -> client co-process proxy -> server co-process proxy -> server" flow, but my initial hunch is that those are mainly just slight permutations of the same problem so they wouldn't need drastically different implementations. Does that seem like a reasonable assumption?

Do you own all the clients and/or servers in your scenario?  That will change the answer to which configuration is the best.  The configuration we use commonly is that both client and server have a co-process proxy.

Is there any public info on the co-process proxy that you guys are using, or is that entirely an internal project?

Sorry, it's entirely internal.

 

 

 

Thanks!

- Joey Bratton