How to communicate with an iex node behind a firewall

[Peter Koppstein]

I'm having trouble setting up a two-computer environment for Elixir because one of the computers is behind a firewall. I've spent some time looking for documentation and hints about doing this, and have made some progress, but the documentation that I've been able to find seems to be inadequate. In any case, I'd certainly appreciate some help!

Please note that I'm familiar with the general procedure for getting two iex nodes to communicate with each other, as well as the details about port 4369 and the trick involving:

   --erl "-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001"

The situation is actually very typical:

• the computer inside the firewall (let's call it inside.edu) can ssh to the computer outside the firewall (let's call it outside.org), and vice versa;
• the router for outside.org also forwards the two above-mentioned ports (4369 and 9001) to outside.org;
• the guardians of inside.edu have inflexible policies regarding port forwarding.

As I understand it, the challenge here is to configure things so that messages from outside.org
can reach inside.edu.

Is there a way to do this using ssh tunneling? Is the iex option "-rsh ssh" relevant?

[The above was originally posted at https://github.com/elixir-lang/elixir/issues/3269, but it was suggested that this would be a more appropriate place.]

[José Valim]

I haven't played with such setup yet so I am afraid I can't help much besides giving some pointers.

1. Did you mean iex --remsh instead of -rsh? If so, that won't make a difference

2. Just to be clear, were you able to get two different nodes in the same network working?

3. Can you provide the exact command you are using to start Elixir?

4. When things are not working, you can start epmd in debug mode. The way Erlang distribution works is that, if you are Node A in Machine X and you want to talk to Node B in Machine Y, first you connect to the EPMD in Machine Y (using port 4396) and you ask all nodes. Once you see all nodes, you connect to each node in the specified inet ports.

So one option is to start epmd with debug mode (run epmd -kill && epmd -debug), see that Node B is registered to epmd on its own machine and then attempt to request a connection from Node A in Machine X. More often than not, connection issues are related due to the naming where Machine X does not even know where Machine Y is.

Erlang was mostly designed to run on local secure networks. With all this said, if things still do not work, you can attempt to start a discussion on erlang-questions mailing list and see if anyone has done ssh tunnelling related deploys.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/4e3adc16-686f-4bfe-8fb1-5bbf6af03363%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Koppstein]

Thank you, José, for your detailed response.  The situation I described (one machine inside a firewall and one machine outside a firewall, with only ssh to work with) is pretty common, so I hope you'll have the time to give the matter some thought.

Rather than answering your questions in detail, I think it will at this stage be more useful to you if I describe the progress I've made with respect to the INSIDE/OUTSIDE problem, as that is in itself fairly involved.

Please also note that the following is of interest as verifies that at least one major hurdle has been overcome: as shown below, the ssh log on the INSIDE machine lights up when the OUTSIDE machine starts iex.  

In the following, INSIDE is the computer inside the firewall, and OUTSIDE is the computer outside the firewall; "INSIDE$" is the command-line prompt of the INSIDE computer.

```

# From now on, lines beginning with # are comments.

# In the following, the input and output on the two machines are shown in chronological order.

# Lines beginning with 'debug1:' are from the "verbose" output of ssh on INSIDE.

# FIRST: I verified that the ~/.erlang.cookie files are identical on the INSIDE and OUTSIDE machines.

# SECOND: On OUTSIDE, I verified that the beam.smp and epmd can accept network connections

# both by using the Firewall GUI, and by using codesign.

# THIRD:

INSIDE$ sudo codesign --force --deep --sign - /usr/local/Cellar/erlang/17.4.1/lib/erlang/erts-6.3.1/bin/beam.smp

INSIDE$ sudo codesign --force --deep --sign - /usr/local/Cellar/erlang/17.4.1/lib/erlang/erts-6.3.1/bin/epmd

# On INSIDE, I setup ~/.ssh/config to include these lines:

Host tunnel

    HostName OUTSIDE

    RemoteForward 4369 localhost:4369

    RemoteForward 9001 localhost:9001

#######################

INSIDE$ ssh -TfNx -v tunnel

Authenticated to OUTSIDE ([173.xx.xx.xxx]:22).

debug1: Remote connections from LOCALHOST:4369 forwarded to local address localhost:4369

debug1: Remote connections from LOCALHOST:9001 forwarded to local address localhost:9001

debug1: Requesting tun unit 2147483647 in mode 1

debug1: sys_tun_open: /dev/tun0 open failed: No such file or directory

Tunnel device open failed.

Could not request tunnel forwarding.

debug1: remote forward success for: listen 4369, connect localhost:4369

debug1: remote forward success for: listen 9001, connect localhost:9001

debug1: All remote forwarding requests processed

INSIDE: iex --name "peak@INSIDE" --erl "-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001"

Eshell V6.1  (abort with ^G)

Interactive Elixir (1.0.3) - press Ctrl+C to exit (type h() ENTER for help)

iex(peak@INSIDE)1> 

#############################

OUTSIDE$ iex --name "peter@OUTSIDE" --erl "-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001"

#############################

# The following "debug" output appears on the INSIDE console:

debug1: client_input_channel_open: ctype forwarded-tcpip rchan 4 win 2097152 max 32768

debug1: client_request_forwarded_tcpip: listen localhost port 4369, originator 127.0.0.1 port 50717

debug1: connect_next: host localhost ([::1]:4369) in progress, fd=6

debug1: channel 0: new [127.0.0.1]

debug1: confirm forwarded-tcpip

debug1: channel 0: connection failed: Connection refused

debug1: connect_next: host localhost ([127.0.0.1]:4369) in progress, fd=7

debug1: channel 0: connected to localhost port 4369

###########################################

iex(peter@OUTSIDE)1> Node.ping :"peak@INSIDE"

:pang

###########################################

iex(peak@INSIDE)1> Node.ping :"peter@OUTSIDE"

:pang

```

[José Valim]

Sorry Peter but I have no idea how to further help.

I believe you already said that OUTSIDE expands to something relevant in "peter@OUTSIDE", right? I.e. the machines can find each other with the host names you have set up.

The last two references I have are:

* epmd docs - http://www.erlang.org/doc/man/epmd.html

* net_adm docs - http://www.erlang.org/doc/man/net_adm.html

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--
You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/843d7c7b-40d9-4258-a4c6-4844f0a84f72%40googlegroups.com.

[Saša Jurić]

My devops skills are fairly basic, so I'm maybe missing something, but let me try to help (and sorry if I'm asking obvious questions). 

Distributed Erlang establishes a TCP connection between two nodes. Are you able to do:

INSIDE$ telnet OUTSIDE 4396

INSIDE$ telnet OUTSIDE 9001

and vice versa from the OUTSIDE to INSIDE?

If that doesn't work, neither will distributed Erlang.

I'm not sure if both directions need to work, but it should definitely work in the direction you're establishing the connection. So if you're doing Node.connect(:foo@OUTSIDE), then telnet from INSIDE to OUTSIDE must also work.

Also, just to be sure, node address must be resolvable by the local host, and the remote host you're using must correspond to the one used when starting the remote node.

In other words, if I start f...@127.0.0.1 then Node.connect(:"f...@bar.baz") will not work even if bar.baz resolves to the proper host.

Finally, I'm not really sure that using distributed Erlang in different networks is a good idea. If you're planning on relying on services such as :pg2, :global, and similar, those things are chatty, and they may work poorly on a slower network. Also, distributed Erlang uses a single connection for all communication (including net_tick monitoring), which may lead to some unexpected disconnections or delays if you're passing huge volumes of data.

If you just need to pass some data occasionally, then you might be better off by just exposing some HTTP(S) or TCP/SSL endpoint, and use that for data exchange.

[Peter Koppstein]

On Thu, Apr 16, 2015 at 5:49 PM, Saša Jurić <sasa....@gmail.com> wrote:

... let me try to help 

Thanks for your interest!

 

Distributed Erlang establishes a TCP connection between two nodes. Are you able to do:

INSIDE$ telnet OUTSIDE 4396

INSIDE$ telnet OUTSIDE 9001

Please note that epmd uses port 4369 by default (not 4396). 

The good news is that:

1)  "INSIDE$ telnet OUTSIDE 4369" confirms that INSIDE can reach port OUTSIDE:4369.  

2) http://www.yougetsignal.com/tools/open-ports also confirms that OUTSIDE:4369 is "open".  

3) if I run iex on OUTSIDE with the parameter:

 --erl "-kernel inet_dist_listen_min 9001 inet_dist_listen_max 9001"

then "INSIDE$ telnet OUTSIDE 9001" also shows signs of life.

4) The open-ports tool also confirms the port is "open".

 

and vice versa from the OUTSIDE to INSIDE?

There's the rub. If I have the "tunnel" setup as previously described, and if iex is running on INSIDE with the min/max parameters as shown above, then: 

OUTSIDE$ telnet INSIDE 4369 # hangs

OUTSIDE$ telnet INSIDE 9001 # "Connection refused"

Stepping back for a minute, it seems to me that it should be possible to configure ssh to achieve the desired goal, but it MIGHT be necessary to have some control over the port choices that the various processes make for different purposes.  (For example, epmd has a "-port PORT" option.)

I could probably figure it out if I had a better understanding of (a) which ports are being "listened" to by which processes; (b) which processes are attempting to send messages to which ports; and (c) how to specify which ports should be used by the various processes.

Thank you again. 

[Peter Koppstein]

On Thu, Apr 16, 2015 at 5:49 PM, Saša Jurić <sasa....@gmail.com> wrote:

...and vice versa from the OUTSIDE to INSIDE?

Of course, if that were possible, there'd be no issue.  That is, INSIDE is behind a firewall, and ssh tunnelling is not magic.  Normally, one would use tunneling like this:

1) On OUTSIDE, arrange for a port (say OUTSIDE:43690) to be mapped to INSIDE:4369

2) Then on OUTSIDE, the way to communicate with INSIDE:4369 is like this:

OUTSIDE$ telnet OUTSIDE 43690  

That's why I suspect that some way to determine the port numbers (as with epmd -port PORT) is the key here.

(By the way, epmd heeds the setting of the environment variable: ERL_EPMD_PORT.)

Thanks.

[Saša Jurić]

Yeah, I’m not sure if it can work if different epmd instances listen on different ports, but I never had this situation, so I don’t really have any first hand experience there.

Assuming I’m not wrong, I think you’ll likely need to resolve this outside of Erlang, and somehow open needed ports on OUTSIDE and INSIDE.

I had a similar case once upon a time, when I connected two different Erlang nodes from different networks, and we resorted to a small VPN for that. But in hindsight, it was a mistake to even use distributed Erlang there, because I was just doing data transfer (so no RPC was needed), and this could (and should) have been done through some data-only interface exposed through TCP/SSL. That would in fact be a simpler solution, with less setup required, and it would also be more secure, since RPC from the outside to the inside wouldn’t be possible.

So I guess my advice is to consider whether distributed Erlang is really suitable for your particular use case.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-talk/mt7iH6hzvew/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-ta...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAMCq_EXSs0L2OTBTQs2be%2Bw-0Tt-55UoQTCcj%2BNAsein46cmcg%40mail.gmail.com.