mariadb/galera replication over wan with firewalls

[tippy2k]

Hello everyone,

I am trying to setup a mariadb/galera cluster across two datacenters.  I have tried a few configurations but I have not been having any luck.  

Here is what I am trying to achieve. I am currently operating in a private datacenter and I have three database nodes that connect to each other over a local network.  I am trying to migrate my setup to a co-location facility, and I have duplicated my setup there.  I want to be able to synchronize the two cluster together.

Starting with my current firewall I created NAT rules to map ports of three public ip addresses to ports on the original cluster.  These ports are 3306, 4567, 4568, and 4444.  I changed my configs to use the NAT'ed public ip addresses instead of the local ip addresses in wsrep_cluster_address and wsrep_node_address.  

I stopped the cluster and started it up by bootstrapping the first node.  When I joined the second node I saw this error in the second node's error log:

150820 13:40:35 [Warning] WSREP: Failed to prepare for incremental state transfer: Failed to open IST listener at tcp://x.x.x.x:4568', asio error 'Cannot assign requested address': 99 (Cannot assign requested address)

The second node does join the cluster (although it takes a while).  My guess is that IST fails to bind to the public ip because the node does not have that ip address assigned to it.

Is it possible to make a node listen on tcp://0.0.0.0.4568 so it will work correctly behind a firewall?  If not what are my options?  hand off public ip addresses to the nodes in each datacenter?  SSH tunnels?  VPN?  If there is anyone who has dealt with the kind of setup I would greatly appreciate and advice.

Thanks,

Mark

[alexey.y...@galeracluster.com]

> <tcp://64.28.63.52:4568'>, asio error 'Cannot assign requested

> address': 99
> (Cannot assign requested address)
>
> The second node does join the cluster (although it takes a while). My
> guess is that IST fails to bind to the public ip because the node does
> not
> have that ip address assigned to it.
>
> Is it possible to make a node listen on tcp://0.0.0.0.4568

Try setting ist.recv_addr in wsrep_provider_options to tcp://0.0.0.0.

But I think the secret sauce here is to use domain names instead of IP
addresses and custom /etc/hosts to resolve those names to proper
addresses depending on the node location.

[tippy2k]

Thanks for the reply.  I went ahead and created a vpn.  It is working ok, but has introduced a whole new world of possible issues.  If I can't keep the connection stable i will try the settings that you mention.  I never thought about tricking out dns with etc/hosts.  I am not sure if i can get my mind around what would be put in each machines hosts file.  But definitely worth considering.

Thanks,

Mark

[Jani Baramidze]

But after setting ist.recv_addr to 0.0.0.0, binding succeeds but other machine in cluster then tries to connect to 0.0.0.0 as well:

150902  9:51:18 [ERROR] WSREP: IST failed: IST sender, failed to connect 'tcp://0.0.0.0:4568': Connection refused: 111 (Connection refused)

at galera/src/ist.cpp:Sender():576

I think it's used as connect address as well. how can I fix it? I also want replication over WAN. 

https://github.com/codership/galera/issues/105

I think this issue is related.