What is the proper way to stop mysql service for the galera cluster servers?

[Yung Hee Lee]

Hi,

I have two cluster nodes.

When I stop the first node with cmd 'service mysql stop', the wsrep processes still running:

mysql    15877  0.0  0.4 528760 73384 ?        Ssl  14:28   0:00 /usr/sbin/mysqld --wsrep_start_position=72cca273-7401-11e6-884b-5ed348c1d32c:96628

 

mysql    15886  0.1  0.0 113256  1712 ?        S    14:28   0:01 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address 132.25.123.456 --datadir /var/lib/mysql/ --parent 15877

mysql    15916  0.0  0.0 114644  1204 ?        S    14:28   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql    16076  0.0  0.0 146472  1880 ?        S    14:29   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql    16077  0.0  0.0 146472  1892 ?        S    14:29   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql    16078  0.0  0.0 146472  1880 ?        S    14:29   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

mysql    16079  0.0  0.0 146472  1864 ?        S    14:29   0:00 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

Same things happen when I stop the mysql service at node2.

I have to kill all the processes via command 'kill PID' then only I can restart the node1. 

Themysql service in node2 cannot be restarted as it will receive timeout error. However, the rsync processes running fine. I wonder it is because I manually kill the wsrep processes until it creates some conflict?

May I know what is the proper way to have a restart?

[node2]# systemctl status mariadb.service



● mariadb.service - MariaDB database server

   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)

  Drop-In: /etc/systemd/system/mariadb.service.d

           └─migrated-from-my.cnf-settings.conf

   Active: failed (Result: timeout) since Thu 2016-09-08 16:16:19 MYT; 11min ago

  Process: 32753 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)

  Process: 32750 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

 Main PID: 414

   CGroup: /system.slice/mariadb.service

           ├─ 414 /usr/sbin/mysqld --wsrep_start_position=72cca273-7401-11e6-884b-5ed348c1d32c:97396

           ├─ 422 /bin/bash -ue /usr//bin/wsrep_sst_rsync --role joiner --address node2 --datadir /var/lib/mysql/ --parent 414

           ├─ 453 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 669 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 670 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 671 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 672 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 673 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 674 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 675 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 676 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 679 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 680 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 681 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 683 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 684 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 685 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 686 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           ├─ 687 rsync --daemon --no-detach --port 4444 --config /var/lib/mysql//rsync_sst.conf

           └─4131 sleep 1




Sep 08 16:13:55 sweb1.genusis.com rsyncd[672]: receiving file list

Sep 08 16:13:55 sweb1.genusis.com rsyncd[667]: receiving file list

Sep 08 16:13:55 sweb1.genusis.com rsyncd[677]: sent 54 bytes  received 203 bytes  total size 61

Sep 08 16:13:55 sweb1.genusis.com rsyncd[668]: sent 54 bytes  received 203 bytes  total size 61

Sep 08 16:14:30 sweb1.genusis.com rsyncd[667]: sent 1688 bytes  received 18228898 bytes  total size 18221850

Sep 08 16:14:48 sweb1.genusis.com systemd[1]: mariadb.service start operation timed out. Terminating.

Sep 08 16:16:19 sweb1.genusis.com systemd[1]: mariadb.service stop-final-sigterm timed out. Skipping SIGKILL. Entering failed mode.

Sep 08 16:16:19 sweb1.genusis.com systemd[1]: Failed to start MariaDB database server.

Sep 08 16:16:19 sweb1.genusis.com systemd[1]: Unit mariadb.service entered failed state.

 

Sep 08 16:16:19 sweb1.genusis.com systemd[1]: mariadb.service failed.

[Suresh Kuna]

Did we stopped the apps and monitoring systems before the shutdown of the mysql services.