Galera three nodes : flow control pause time high

[nguyen quoc viet]

Hi Everybody,

        - I have setup three node DB (MariaDB verion: 10.2.23 on Centos 7.6 64 bit ) and galera 25.3.26, after setup everything work fine. But Suddenly one month ago, three nodes sometimes not sync. 

        - In front of Galera cluster, I use maxscale (2.3.5) for load blancing and routing. 

 

       - I had check and collect in time of failure, maxscale treat (node 2 is master, node 1 and node 3 are slaver): 

           a. Receive Queue Node 1 increase in time of failure 

           b. Receive Queue Node 3 :  not increase as node 1 

           c. Sent Queue Node 2 :  increase in time of failure. 

           d. Cert_failures and bf aborts is zero ( on three node)

      - When failures , I have Stop mariadb on node 1 and galera cluster come back as Normal State.

      - After fail, I also check RAM, CPU, Network , IOPS Disk  all of them and log mariaDB. But I don't detect clue for troubleshooting. 

      - Pls give me some recommend for troubleshooting, thanks for reply.

      Send Queue

      Receive Queue

      Flow control

      

Config on Node 1

      

[mysqld]

log-error=/var/log/mysql/mysql-error.log

bind-address=0.0.0.0

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

# Galera cluster configuration

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_address="gcomm://node1,node2,node3"

wsrep_cluster_name="clustername"

#wsrep_sst_method=rsync

wsrep_sst_method=mariabackup

wsrep_sst_auth=xxxxx:xxxx

#wsrep_debug = On

wsrep_desync=on

wsrep_slave_threads=48

wsrep_provider_options="gcache.size=2048M"

# Cluster node configuration

wsrep_node_address="node1_address"

wsrep_node_name="node1"

#Replication from master 

server_id=3

slave-skip-errors=1062,1032

log-slave-updates=1

log-bin=binlog

binlog_format=row

wsrep_gtid_mode=1

wsrep_gtid_domain_id=1

gtid_domain_id=201

 

Config on Node 2

[mysqld]

log-error=/var/log/mysql/mysql-error.log

bind-address=0.0.0.0

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

# Galera cluster configuration

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_address="gcomm://node1,node2,node3"

wsrep_cluster_name="clustername"

#wsrep_sst_method=rsync

wsrep_sst_method=mariabackup

wsrep_sst_auth=xxxxx:xxxx

#wsrep_debug = On

wsrep_desync=on

wsrep_slave_threads=48

wsrep_provider_options="gcache.size=2048M"

# Cluster node configuration

wsrep_node_address="node2_address"

wsrep_node_name="node2"

#Replication from master 

server_id=3

slave-skip-errors=1062,1032

log-slave-updates=1

log-bin=binlog

binlog_format=row

wsrep_gtid_mode=1

wsrep_gtid_domain_id=1

gtid_domain_id=201

   

Config on Node 3

[mysqld]

log-error=/var/log/mysql/mysql-error.log

bind-address=0.0.0.0

default_storage_engine=InnoDB

innodb_autoinc_lock_mode=2

# Galera cluster configuration

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_cluster_address="gcomm://node1,node2,node3"

wsrep_cluster_name="clustername"

#wsrep_sst_method=rsync

wsrep_sst_method=mariabackup

wsrep_sst_auth=xxxxx:xxxx

#wsrep_debug = On

wsrep_desync=on

wsrep_slave_threads=48

wsrep_provider_options="gcache.size=2048M"

# Cluster node configuration

wsrep_node_address="node3_address"

wsrep_node_name="node3"

#Replication from master 

server_id=3

slave-skip-errors=1062,1032

log-slave-updates=1

log-bin=binlog

binlog_format=row

wsrep_gtid_mode=1

wsrep_gtid_domain_id=1

gtid_domain_id=201