Setting up a galera cluster in multiple regions on aws

[Charles Johnson]

I am building a MariaDB Galera Cluster, on AWS, EC2. Using MariaDB 10.5 and Galera 4.

I have two nodes in the California region, named galera_ca and galera_ca2, and a third node in Oregon name galera_or

The security groups between the servers are wide open, and the ping time between OR and CA is 12ms

I started galera_ca using galera_new_cluster

I started galera_ca2 using systemctl start mariadb

The two California nodes connect and share data just fine.

wsrep_cluster_size indicates that I have two nodes in the cluster.

I started galera_or using systemctl start mariadb

Oregon then hangs on starting for a very long time, and wsrep_cluster_size shows three nodes, but the server cannot be connected to.

Eventually systemctl start mariadb returns this message:

Job for mariadb.service failed because a fatal signal was delivered to the control process. See "systemctl status mariadb.service" and "journalctl -xe" for details.

systemctl status mariadb.service doesn't show anything exceptional,

journalctl -xe shows:

requested state transfer from 'any', but it is impossible to select State Transfer donor: Resource temporarily unavailable

wsrep_cluster_size still shows three nodes, but mariadb is not running in OR.

If I try to shut down either of the California servers systemctl stop mariadb it never returns.

All three servers have basically the same configuration:

[galera]

wsrep_on=ON

wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

wsrep_node_name='galera_ca'

wsrep_node_address="INTERNAL ADDRESS OF THIS NODE"

wsrep_cluster_name='galera-experiment'

wsrep_cluster_address="gcomm://EXTERNAL ADDRESS OF NODE CA, EXTERNAL ADDRESS OF NODE CA2, EXTERNAL ADDRESS OF NODE OR"

wsrep_provider_options="gcache.size=300M; gcache.page_size=300M"

wsrep_slave_threads=4

wsrep_sst_method=rsync

binlog_format=row

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

bind-address=0.0.0.0

All tutorials and demonstrations have the servers all in the same VPC and the configuration files all use the internal IP addresses. 

I can't find any examples where the external IPs are used and the nodes exist in different regions.

[Victor Lindellee]

Hi Charles!

Galera cannot guess what ip you want to sync data over, so be explicit and set

wsrep_node_incoming_address

wsrep_node_address

wsrep_sst_receive_address

https://galeracluster.com/library/documentation/mysql-wsrep-options.html

regards
Victor

--
You received this message because you are subscribed to the Google Groups "codership" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codership-tea...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codership-team/6766b69d-7019-4861-a9a3-a03d2019a4dao%40googlegroups.com.

[charles...@civi.com]

I’ve reduced the cluster to two nodes: galera_ca and galera_or

I have verified the network between the two are wide open, I can ssh, open the remote db, and perform operations.

 

I start galera_ca with:

sudo galera_new_cluster

 

And it runs just fine

 

and I start galera_or with:

sudo systemctl start mariadb

 

galera_or runs fine on its own, I have verified that.

But, when I fill in server.cnf it won’t start, even though galera_ca is showing 2 nodes in wsrep_cluster_size

 

 

 

[mariadb]

lower_case_table_names=1

 

[galera]

wsrep_on=ON

wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

 

wsrep_node_name='galera_ca'

wsrep_node_address="CA_EXTERNAL_IP"

wsrep_node_incoming_address=" CA_EXTERNAL_IP "

wsrep_sst_receive_address=" CA_EXTERNAL_IP "

 

wsrep_cluster_name='galera-experiment'

wsrep_cluster_address="gcomm:// CA_EXTERNAL_IP,OR_EXTERNAL_IP"

 

wsrep_provider_options="gcache.size=300M; gcache.page_size=300M"

wsrep_slave_threads=4

wsrep_sst_method=rsync

 

binlog_format=row

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

bind-address=0.0.0.0

 

 

 

[mariadb]

lower_case_table_names=1

 

[galera]

wsrep_on=ON

wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

 

wsrep_node_name='galera_or'

wsrep_node_address=" OR_EXTERNAL_IP "

wsrep_node_incoming_address=" OR_EXTERNAL_IP "

wsrep_sst_receive_address=" OR_EXTERNAL_IP "

 

wsrep_cluster_name='galera-experiment'

wsrep_cluster_address="gcomm:// CA_EXTERNAL_IP, OR_EXTERNAL_IP "

 

wsrep_provider_options="gcache.size=300M; gcache.page_size=300M"

wsrep_slave_threads=4

wsrep_sst_method=rsync

 

binlog_format=row

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

bind-address=0.0.0.0

 

 

Charles Johnson

PO Box 4689

Greenwich, CT 06831

866-930-4526

www.civi.com

[Charles Johnson]

I found the answer in the documentation for wsrep_node_address: 

on Amazon EC2, you must use the global DNS name, instead of the internal IP address.