1-to-1 connections & possibly hidden nodes

[Roberto Ostinelli]

All,
Is it possible to have a 1-to-1 connection from a node in a cluster to a single Erlang node which does _not_ connect to the mesh, but only to a very specific Erlang node?

I've go a very CPU-heavy application and am trying to find ways to offload the CPU computations to Erlang nodes (tha don't need to be in a cluster, except for a single box they refer to).

Basically something like this:

<NODE A in cluster> --------- <NODE B in cluster>
        \                     /
         \                   /
          \                 /
          <NODE B in cluster>
           *       *       *
          *        *        *
         *         *         *
      <NODE B1> <NODE B2> <NODE B>

The distributed cluster is composed of nodes A, B and C, but nodes B1, B2 and B3 can communicate directly with node B and that node only. Node B can offload CPU computations to B1, B2, B3 by sending erlang messages to them and receive results back.

Every node in the cluster needs ~50x the CPU power, therefore I'm trying to avoid having the standard cluster grow so big that it would be generating way too much noise in a full mesh network (especially since it really isn't needed).

I can of course use a simple custom TCP binary channel, but was wandering if something already exists that I could leverage. I've seen that hidden nodes "Hidden nodes always establish hidden connections to all other nodes except for nodes in the same global group" [1] so if I understand this correctly nodes B1-3 would still connect to A and B, even though the cluster wouldn't know about them.

Thank you for any considerations you might have,
r.

 [1] http://erlang.org/doc/man/erl.html

[Roberto Ostinelli]

...And of course the diagram would be (without the copy-pasta issues):

<NODE A in cluster> --------- <NODE C in cluster>
        \                     /
         \                   /
          \                 /
          <NODE B in cluster>
           *       *       *
          *        *        *
         *         *         *
      <NODE B1> <NODE B2> <NODE B3>

[Stanislav Ledenev]

If I understand you correctly, erl -connect_all false is what you need.

пн, 12 июл. 2021 г. в 21:27, Roberto Ostinelli <osti...@gmail.com>:

[Fred Hebert]

Another approach that I recall working is one that is cookie-based. Each cookie you set represents cluster membership, so nodes that share a cookie will connect to each other.

If you create another node that has a distinct cookie, it won't connect to anything else. If I recall properly, you can however call erlang:set_cookie(OtherNode, Cookie) to make a single caller able to use a distinct cookie to talk to OtherNode, meaning it will connect to it, but won't be able to transitively forward the connection unless the peer also shares that cookie.

This lets you use non-hidden nodes that are parts of clusters but still have them able to privately talk to special nodes by using one-off cookies that act as a way to wall off connection propagation.

[Roberto Ostinelli]

Hi Stanislav,

Your suggestion seems to work. I've started 3 nodes one@ and two@ connected in a cluster, and three@ with the option you specify. I get:

(one@dev.local)1> net_kernel:connect_node('t...@dev.local').
true
(one@dev.local)2> nodes().
['t...@dev.local']

(two@dev.local)1> nodes().
['o...@dev.local']

When I then connect three@ to one@:

(three@dev.local)1> net_kernel:connect_node('o...@dev.local').
true
(three@dev.local)2> nodes().
['o...@dev.local']

(one@dev.local)3> nodes().
['t...@dev.local','th...@dev.local']

While two@ doesn't change:

(two@dev.local)2> nodes().
['o...@dev.local']

What is the advantage over using a hidden node? Here below is the output with one@ and two@ in a full cluster, while three@ has been started with the -hidden option and has been connected only to one@:

(one@dev.local)1> nodes().
['t...@dev.local']
(one@dev.local)2> nodes(hidden).
['th...@dev.local']

(one@dev.local)3> register(shell, self()).
true

(two@dev.local)1> nodes().
['o...@dev.local']
(two@dev.local)2> nodes(hidden).
[]

(three@dev.local)1> {shell, 'o...@dev.local'} ! hello.
hello

(one@dev.local)4> flush().
Shell got hello
ok

Thank you for your help!

r.

[Roberto Ostinelli]

This is pretty interesting Fred, thank you for sharing. I didn't know about this possibility. Are you aware of any pitfalls of using this method? I will investigate it too.

[Roberto Ostinelli]

On a side note, both with the -hidden and the -connect_all false options sets, if you try to send a message from the hidden / not meshed node to a node not part of the known nodes, a connection will be established still (i.e. the sending succeeds). I guess the sending operation first tries to amend a connection if unset?

(two@dev.local)1> nodes().
['o...@dev.local']

_Without_ doing a net_kernel:connect_node/1:

(three@dev.local)2> {shell, 't...@dev.local'} ! hello.
hello

(two@dev.local)3> nodes().
['o...@dev.local','th...@dev.local']

[Stanislav Ledenev]

As I understand, cookies are for (simple) authentication purposes.
Nodes with different cookies won't be able to interact with each other but
connection attempt would be made with each call to a node. 

Another approach that I recall working is one that is cookie-based. Each cookie you set represents cluster membership, so nodes that share a cookie will connect to each other.

If you create another node that has a distinct cookie, it won't connect to anything else. If I recall properly, you can however call erlang:set_cookie(OtherNode, Cookie) to make a single caller able to use a distinct cookie to talk to OtherNode, meaning it will connect to it, but won't be able to transitively forward the connection unless the peer also shares that cookie.

This lets you use non-hidden nodes that are parts of clusters but still have them able to privately talk to special nodes by using one-off cookies that act as a way to wall off connection propagation

Here is my _suggestion_: I think "-hidden_node" is about publishing information
about a node and "-connect_all " is about a process of establishing connection when node is up.

[Michael Truog]

On 7/12/21 3:02 PM, Roberto Ostinelli wrote:

What is the advantage over using a hidden node? Here below is the output with one@ and two@ in a full cluster, while three@ has been started with the -hidden option and has been connected only to one@:

(one@dev.local)1> nodes().
['t...@dev.local']
(one@dev.local)2> nodes(hidden).
['th...@dev.local']

(one@dev.local)3> register(shell, self()).
true

(two@dev.local)1> nodes().
['o...@dev.local']
(two@dev.local)2> nodes(hidden).
[]

(three@dev.local)1> {shell, 'o...@dev.local'} ! hello.
hello

(one@dev.local)4> flush().
Shell got hello
ok

Hi Roberto,

It is best if you have the kernel app 'dist_auto_connect' setting set to 'never' if you want to ensure each node connection is created programmatically (to avoid unintended node connections due to messages).  The function net_kernel:connect_node/1 is creating a visible node connection, ensuring that your connection makes your node part of the fully connected set of visible nodes.  If you used net_kernel:hidden_connect_node/1 a hidden node connection would be created and you will not get any additional node connections from that function call.  However, net_kernel:hidden_connect_node/1 remains undocumented so it isn't clearly part of the public interface.  CloudI (https://cloudi.org) uses it and it is the preferred function for connecting to an Erlang node from a script, so I would like to think the function isn't disappearing in the future.

Best Regards,
Michael

[Roberto Ostinelli]

Thank you all for the tips!