http2 reverse proxy:how to reduce connections with backend server(s)?

[killjason]

(moved from: https://github.com/grpc/grpc-java/issues/2470)

Imagine there are 10k grpc-clients, they established 10k http2 connections(TCP-connections) with the http2 reverse proxy; then http2 reverse proxy create 10k http2 connections(TCP-connections) to the origin(backend) server.
Is it possible to reduce the 10k connections between proxy and origin(backend) server?
for example, can a connection pool be used in reverse proxy to reduce connections with backend server?

This picture can explain better:

this picture is in Nginx blog, Is it possible to do the same thing to reduce connections with backend serevrs using http2-reverse proxy?

[Carl Mastrangelo]

This depends on how homogeneous your backends are.  For example, if your proxy going to the same logical set of backends each time, (even if they are distinct machines) then yes this is possible with gRPC.  In gRPC, a channel represent a higher level concept than a single client.  It represents multiple connections to multiple backends (a.k.a. Servers).

In your case, it seems like you should build a map of hostname (a.k.a. "target") to Channel and pick the correct channel to serve requests to in your proxy.   This works well if you are handling a small number hostnames.  Each channel will have its own tcp connections, but there will be few total channels.

You can do more advanced things too with your host name.  If the backends that you send traffic to route requests based on the host name, but each backend can handle the requests of other host names, then you can reduce the number of connections even further.  For example, if you know that  foo.mydomain.com and bar.mydomain.com both physically point to the same set of servers, then they can both share the same channel.  In your channel, you can override the "authority" field but still reuse the same connection.

We can provide a better answer if you could share a little more detail about what you want to do.

[killjason]

In my case:

1.The backend servers represent a same micro-service(may be contains 10 machines) serving the same APIs.

2.The proxy is based on pure Netty(no gRPC included). but clients and backend servers are developed on gRPC.

I am not sure if one Netty channel can represents multiple connections to multiple backends?

在 2016年12月2日星期五 UTC+8上午1:11:03，Carl Mastrangelo写道：

[Josh Humphries]

On Thu, Dec 1, 2016 at 12:26 PM, killjason <jasonso...@gmail.com> wrote:

In my case:

1.The backend servers represent a same micro-service(may be contains 10 machines) serving the same APIs.

2.The proxy is based on pure Netty(no gRPC included). but clients and backend servers are developed on gRPC.

I am not sure if one Netty channel can represents multiple connections to multiple backends?

If you are using Netty as a layer-4 proxy, then it cannot. But if you use it as a layer-7 proxy, using the HTTP/2 protocol handlers, you can. When a client initiates a new stream, you pick a backend, find a channel to that backend, and create a new stream on that backend channel. You will effectively have a map of incoming channel & stream ID -> outgoing channel & stream ID and use that to proxy frames. This works for gRPC, but would probably be insufficient for general-purpose HTTP/2 where the servers initiate streams (since the proxy won't be able to know which client the server-initiated stream was intended). Admittedly, there will probably be some implementation complexity in properly managing HTTP/2 flow control windows on both sides while avoiding excessive resource usage/buffering.

 

在 2016年12月2日星期五 UTC+8上午1:11:03，Carl Mastrangelo写道：

This depends on how homogeneous your backends are.  For example, if your proxy going to the same logical set of backends each time, (even if they are distinct machines) then yes this is possible with gRPC.  In gRPC, a channel represent a higher level concept than a single client.  It represents multiple connections to multiple backends (a.k.a. Servers).

In your case, it seems like you should build a map of hostname (a.k.a. "target") to Channel and pick the correct channel to serve requests to in your proxy.   This works well if you are handling a small number hostnames.  Each channel will have its own tcp connections, but there will be few total channels.

You can do more advanced things too with your host name.  If the backends that you send traffic to route requests based on the host name, but each backend can handle the requests of other host names, then you can reduce the number of connections even further.  For example, if you know that  foo.mydomain.com and bar.mydomain.com both physically point to the same set of servers, then they can both share the same channel.  In your channel, you can override the "authority" field but still reuse the same connection.

We can provide a better answer if you could share a little more detail about what you want to do.

On Thursday, December 1, 2016 at 8:59:45 AM UTC-8, killjason wrote:

(moved from: https://github.com/grpc/grpc-java/issues/2470)

Imagine there are 10k grpc-clients, they established 10k http2 connections(TCP-connections) with the http2 reverse proxy; then http2 reverse proxy create 10k http2 connections(TCP-connections) to the origin(backend) server.
Is it possible to reduce the 10k connections between proxy and origin(backend) server?
for example, can a connection pool be used in reverse proxy to reduce connections with backend server?

This picture can explain better:

this picture is in Nginx blog, Is it possible to do the same thing to reduce connections with backend serevrs using http2-reverse proxy?

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/dd155eb2-c351-49a0-8687-dffa035c3f2b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[killjason]

@Josh Humphries, your plan is nice, gRPC currently not support init stream in server side, so this limit can be ignored.

在 2016年12月2日星期五 UTC+8上午2:18:54，Josh Humphries写道：

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Louis Ryan]

I would take a look at Lyft's Envoy to perform the same role here too if you're not too married to Java for the proxy

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/4234545b-e25f-4e0c-9994-667a8ab21fe1%40googlegroups.com.

[killjason]

the Lyft's Envoy use the same thread model as Netty: a connection was bound to a single thread during its lifetime.

在 2016年12月3日星期六 UTC+8上午3:35:10，Louis Ryan写道：

[killjason]

seems that Lyft's Envoy did the same way as Josh said when proxying http/2.

在 2016年12月3日星期六 UTC+8上午3:35:10，Louis Ryan写道：

I would take a look at Lyft's Envoy to perform the same role here too if you're not too married to Java for the proxy