2-way RPC through firewall that blocks connections in one direction

[Tim K]

Looking for suggestions on how to solve the following transport (not encoding) problem:

- a server somewhere on the Internet, in the cloud. Let's call it "cloud server".

- a server in a corporate data center, behind a firewall that allows only *outgoing* connections. Let's call it "LAN server" for simplicity.

The two servers need to be able to talk to each other:

- the LAN server will push some data to the cloud server (e.g. metrics, error events, etc.); this one is easy the LAN server can just make outgoing HTTPS connections to the cloud server.

- [THE PROBLEM] the cloud server also needs to tell the LAN server to do some things; the cloud server communication is driven by a web GUI for example (or some other client) that needs to be responsive. Now, the cloud server cannot establish connections (HTTPS or otherwise) to the LAN server whenever it needs to talk to the LAN server because of the LAN firewall that blocks incoming requests.

Having the LAN server periodically poll for commands from the cloud server (e.g. 1 sec. or some other interval) is not an option because it will introduce unwanted delays for the clients of the cloud server (e.g. Web UI).

Any suggestions on how to approach the problem? What techniques to look at, e.g. HTTP long polls, websockets, WAMP?

More importantly, any good Go libraries that can abstract all this so I don't "invent" it myself and instead I can focus on solving the actual problem at hand and not the transport problem? Of course, I'd like something that helps as much as it can when it comes to re-establishing any broken connections/tunnels, supports SSL, supports some form of authentication (though that could also be added on top)?

Thanks for your help.

Tim

 

[Tristan Colgate]

GRPC has bi directional streams using http2 over TLS. That might be a good base to build on.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Bourgon]

This isn't really a "transport" problem, so much as it's a "network
architecture" problem.

You have to initiate the connection from behind the firewall, so you
need to design the LAN server to connect to a well-known Cloud server
endpoint at startup, and manage/maintain that persistent connection.
In effect, you're creating your own application-specific "transport",
and you direct all communication over that transport.

As Tristan notes, all else equal, gRPC bidirectional streaming is
probably a good starting point. I'd also look at ZeroMQ or nanomsg,
depending on what kind of messaging semantics you need. Though they
could be massaged to work, I'd avoid HTTP long poll or websockets.

As for how this ties into Go kit — if you build the transport as you
need it, and then write appropriate endpoint bindings for each RPC,
you can leverage all the standard Go kit middlewares and patterns
quite easily. You could use something like [0] as a starting point.

[0] https://github.com/go-kit/kit/blob/master/examples/addsvc/grpc_binding.go

Cheers,
Peter.

[Tim K]

Fair enough maybe "transport" is not the right term and indeed I do recognize it as a network/communication problem. This looks a lot like a form of VPN or ssh (with reverse port forwarding) tunnel that gets initiated from the LAN side then all the traffic flows through that. Of course I would like to avoid truly using VPN or ssh (though that may be good enough as a POC for now) and instead I'd like the tunnel to be negotiated and maintained by the 2 Go servers that need to talk to each other.

Thanks for the gRPC pointer. Are there any real world issues with overly restrictive firewalls when it comes to HTTP/2?

And of course I'd love to hear from other people how they solved such a problem and whether there are already any other Go frameworks that can help with this.

Tim

[Art Mellor]

We solved a similar problem by using the RPC package. We had the LAN server initiate TCP connections and then created RPC Servers (don't use the default one) that used that connection. Now you can make requests in either direction via the underlying TCP connection that was initiated from the LAN server.

It worked really well.