2 way communication over single tcp connection

[Kumar Goyal, Rajat]

Hi Grpc team,

 

      I am having a requirement where my grpc-servers are behind firewall. So all incoming connections to these servers are blocked.

I want to use grpc as tunnel, where these servers will initiate the tcp connection with a publicly exposed load balancer. I chose nginx as load balancer for POC. So my idea is :

 

a)      Grpc servers will initiate a long-lived tcp connection with nginx by calling a RPC. Nginx will have all these servers defined under upstream group. This way each upstream server will have 1 connection with nginx.

b)     A Grpc client calls nginx, and nginx will forward the request to any of upstream server. In-doing so nginx should use the connection established in step-a than creating a new connection to upstream server.

But when I tested with above set-up, I see that it is creating a new connection than using the already established connection with upstream server.

 

Can you please suggest if this is possible ? How differently should I run the grpc server / nginx conf. Can any other load-balancer server above purpose than nginx ?

 

Regards,

Rajat

---

This message may contain confidential information protected by law. The contents of this email are to be viewed only by the intended recipient. If you received this message in error, notify the sender immediately and delete the original message without printing. Product descriptions, pricing and similar content is for information only and does not constitute an offer, warranty or guarantee. Contracts with Arcesium are formed only by written documents bearing the signature of its authorized representative.

[Sergii Tkachenko]

Hey Kumar,

Interesting question!

As I understand, your plan is for gRPC servers to initiate a TCP connection to nginx from their side. This is needed because incoming connections are blocked by a firewall, so gRPC servers must initiate outgoing connection first.

Could you please clarify a couple of things about your setup?

> a) Grpc servers will initiate a long-lived tcp connection with nginx by calling a RPC.

How does this happen exactly? How can gRPC servers call an RPC on ngnix, if they're the ones serving this RPCs?

Would you mind sharing your nginx.conf? This will give me a better picture of your setup.

Using gRPC with nginx load balancing in private networks behind firewalls is quite common. Take a looks at this article, if you haven't already: https://www.nginx.com/blog/nginx-1-13-10-grpc/.

What's unusual about your setup, is that the LB cannot talk to gRPC servers first. You might want to consider requesting a firewall rule allowing the LB to talk to gRPC instances.

- Sergii

[Rajat Goyal]

Hi Sergii,

    Please find my nginx conf:

user nginx;

worker_processes  1;

error_log /var/log/nginx/error.log notice;

pid /var/run/nginx.pid;

events {

    worker_connections 1024;

}

http {

   map $upstream_trailer_grpc_status $grpc_status {

    default $upstream_trailer_grpc_status; # We normally expect to receive grpc-status as a trailer

    ''      $sent_http_grpc_status;        # Else use the header, regardless of who generated it

   }

   log_format grpc_json escape=json '{"timestamp":"$time_iso8601","client":"$remote_addr",'

                                  '"uri":"$uri","http-status":$status,'

                                  '"grpc-status":$grpc_status,"upstream":"$upstream_addr",'

                                  '"connection":$connection,"tx-bytes":$bytes_sent}';

   upstream client_a {  #these servers will establish a out-bound connection with nginx on their start

    zone client_a 64k;

   server 192.168.127.129:50051 max_conns=1;

   server 192.168.127.130:50051 max_conns=1;

   keepalive_timeout 365d;

   keepalive 2;

   queue 100;

   }

   keepalive_timeout 365d;

   server {

      listen 5080 http2 so_keepalive=on;

      access_log /var/log/nginx/grpc_log.json grpc_json;

      location /helloworld. {              # this rpc is used for testing client. When client calls nginx, due to keepalive settings nginx is just using below out-bound connection to upstream.

         grpc_socket_keepalive on;

         expires -1;

         grpc_pass grpc://client_a;

      }

      location /manualflowcontrol. {       # this rpc is used for establishing a bi-di connection between upstream grpc-server & nginx, nginx is just passing the request to same server. 

                                       # here also in grpc_pass nginx is creating a new outbound-connection to grpc-server instead of using grpc-server -> nginx established connection.

         grpc_socket_keepalive on;

         expires -1;

         grpc_pass grpc://$remote_addr:50051;

      }

   }

}

Firewall is out of our control as managed by some other company(s).