Problem with adding node to MariaDB/Galera cluster

[Marko Sutic]

Hello,

we are trying to initiate MariaDB/Galera replication over WAN but cannot add node to the configuration.

This is the error from donor log:

...

160426 20:22:48 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '142.255.0.34:4444/xtrabackup_sst' --auth 'wsrep:*********' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf'  --binlog 'replication-master_mysql-bin' --gtid '9d1051bb-9a12-11e5-9c03-2a73c0b193e5:316073097''

160426 20:22:48 [Note] WSREP: sst_donor_thread signaled with 0

160426 20:22:48 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '142.255.0.34:4444/xtrabackup_sst' --auth 'wsrep:*********' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf'  --binlog 'replication-master_mysql-bin' --gtid '9d1051bb-9a12-11e5-9c03-2a73c0b193e5:316073097'

160426 20:22:48 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '142.255.0.34:4444/xtrabackup_sst' --auth 'wsrep:*********' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf'  --binlog 'replication-master_mysql-bin' --gtid '9d1051bb-9a12-11e5-9c03-2a73c0b193e5:316073097': 1 (Operation not permitted)

160426 20:22:48 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '142.255.0.34:4444/xtrabackup_sst' --auth 'wsrep:***********' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf'  --binlog 'replication-master_mysql-bin' --gtid '9d1051bb-9a12-11e5-9c03-2a73c0b193e5:316073097'

160426 20:22:48 [Warning] WSREP: 2.0 (prod-galera2): State transfer to 1.0 (prod-aws-galera1) failed: -1 (Operation not permitted)

160426 20:22:48 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 316073097)

...

We are using wsrep_sst_method=xtrabackup-v2 method for SST.

Wsrep parameters we're using:

wsrep_auto_increment_control = 1

wsrep_causal_reads = 0

wsrep_certify_nonPK = 1

wsrep_cluster_address = 'gcomm://142.255.0.34,111.222.333.444,111.222.333.555'

wsrep_cluster_name = 'prod_cluster'

wsrep_convert_LOCK_to_trx = 0

wsrep_drupal_282555_workaround = 0

wsrep_node_address = '142.255.0.34'

wsrep_node_name='prod_galera1'

wsrep_node_incoming_address = '142.255.0.34'

wsrep_provider = /usr/lib/galera/libgalera_smm.so

wsrep_provider_options = 'gcache.size = 1G'

wsrep_slave_threads = 50

wsrep_sst_method = xtrabackup-v2

I have changed IP addresses and names in the question.

How to overcome this problem and successfully start SST transfer?

Where should we look to troubleshoot problem?

Thanks.

Regards,
Marko

[Marko Sutic]

Version:

+------------------------------------+

| version()                          |

+------------------------------------+

| 10.0.17-MariaDB-1~wheezy-wsrep-log |

+------------------------------------+

1 row in set (0.00 sec)

| wsrep_provider_name          | Galera                                              |

| wsrep_provider_vendor        | Codership Oy <in...@codership.com>                   |

| wsrep_provider_version       | 3.9(rXXXX)            

Regards,

Marko