Proper way to remove and rejoin a node to cluster

[Joe White]

Hello,

I've been getting mixed results when removing a node (node A) from a 3 node cluster for a reboot, then rejoin the cluster. A few times node A just joins the cluster upon starting mariadb, but often times the /usr/sbin/mariadbd --user=mysql --wsrep_recover --disable-log-error process just hangs resulting in one of two scenarios:

1. Mariadb never starts on Node A. There is no movement in mysqld.log.
2. Node A shows as a member of the cluster when running select * from information_schema.WSREP_MEMBERSHIP from an existing node in the cluster, however mariadb on node A never starts. Looking at the logs on Node A, I see Server status change joiner -> initializing. 

The process on Node A before rebooting is:

* stop all applications that could be writing to NodeA

* run SHUTDOWN WAIT FOR ALL REPLICAS; in the sql terminal

* reboot the machine

Any advice would be greatly appreciated!

[Alexey Yurchenko]

Hi,

First thing to understand is that normal MariaDB/Galera startup on reboot happens in two phases:

1) preparation: mysqld is started with --wsrep-recover option to only to recover innodb tablespace, find its GTID and shut down. During that phase no cluster connection is made and other nodes are unaware of it.

2) operation: mysqld is started with --wsrep-initial-position=<GTID> and connects to cluster. Here it may find that it needs to copy state snapshot from another node before proceeding, but in any case next thing it needs to initialize innodb storage engine once again.

And judging by you report it is innodb initialization where it gets stuck - whether it is phase 1 or 2. So I'd suggest that you look into that direction. There were reports of mysteriously long innodb initializations. It may be a combo of innodb configuration and hardware.

And yes, it is unrelated to how you shutdown the server, but of course a graceful shutdown makes for quicker restart.

KInd regards.

Alex