Auto-bootstrap and graceful restarts

[Ragnar Rova]

Hi,

1. How can pc.recovery=true be used to aid graceful cluster restarts? Today pc.recovery only helps me when there is a full cluster crash / power outage etc, since the gvwstate.dat file is removed on graceful shutdown. http://galeracluster.com/documentation-webpages/galeraparameters.html#pc-recovery specifically mentions: "Graceful full cluster restarts without the need for explicitly bootstrapping a new Primary Component." How can pc.recovery help here? That line confuses me. And why again is gvwstate.dat removed on shutdown?

2. In my startup script (mentioned in earlier post), I interview all nodes what sequence number they are on by either running mysqld --wsrep-recover or selecting wsrep_last_commited, depending on if the node is running or not. This method is only used if more than a majority of nodes are stopped or if gvwstate.dat does not exist on all nodes, so that there is a real need for bootstrap, otherwise I just start the stopped nodes. I automatically pick the node with the highest seqno as bootstrap node, preferring running nodes when there are several with equal seqno, and aborting if not all nodes have the same state uuid. Reading http://galeracluster.com/2013/10/order-of-business/ and and this seems correct. Have I missed anything?

3. Given the above, has anyone worked out a solution of rebooting a cluster gracefully without a control node or manual operator coordinating the process? I think not and that the order-of-business article mentions that this was never intended. If I really want some sort of automated restart of a 3-node cluster I have to use a fourth independent coordinator node which does the restart of all nodes. Maybe this is less important of a case the more you think of it. For unexpected shutdowns pc.recovery does its job and otherwise my start script from 2. helps the operator. 

I know it might look like i'm adding myself to the miracle-expecters group here ;) Appreciate some clarifications on the above.

Thanks!

[alexey.y...@galeracluster.com]

On 2015-06-10 00:04, Ragnar Rova wrote:
> Hi,
>
> 1. How can pc.recovery=true be used to aid graceful cluster restarts?
> Today
> pc.recovery only helps me when there is a full cluster crash / power
> outage
> etc, since the gvwstate.dat file is removed on graceful shutdown.
> http://galeracluster.com/documentation-webpages/galeraparameters.html#pc-recovery
> specifically mentions: "Graceful full cluster restarts without the need
> for
> explicitly bootstrapping a new Primary Component." How can pc.recovery
> help
> here? That line confuses me. And why again is gvwstate.dat removed on
> shutdown?

It is an error in documentation. At the moment pc.recovery=true helps
only in the case of total cluster crash. This is precisely because
gvwstate.dat is removed on graceful shutdown. Which in turn was made out
of conservative considerations that it may interfere with the now
canonical way to start the cluster with --wsrep-new-cluster option.
(Initial feature request was to facilitate unattended cluster restart in
case of power outage)

It may be that we've been too conservative about that and now we are
thinking of relaxing this.

You are not adding yourself to the miracle-expecters group. However, it
would be nice to understand, what is the use case for whole cluster
restart? The whole premise behind the cluster is that you never have to
shut the whole service down. Why do you need a 4th node? What's wrong
with restarting one node at a time?

Regards,
Alex

[Ragnar Rova]

The whole premise behind the cluster is that you never have to
shut the whole service down. Why do you need a 4th node? What's wrong
with restarting one node at a time?

One case, albeit a bit contrived is:

Benign but clueless operator needs to do shutdown on 3 machines running galera cluster. Let's say it's done one by one, maybe to do some HW maintenance or whatever. They are managed with SysV init scripts (RHEL6). So `service mysql stop` is called, and each node gracefully leaves the cluster. When machines are booted again, the cluster cannot automatically start. MySQL is set to start on boot, and has wsrep_cluster_address set to each of the other nodes. But when they start, they will just block forever since no bootstrap node is up.

What I can do now is to clearly document that any restarts should anyhow be done one-by-one (agreed this is HA, so that should be the procedure anyhow),

I was also looking for feedback on the script in 2. since I don't want the bootstrap node selection to be done manually. Are there some quirks here? I am never comparing sequence numbers on a system under load where seqnos might update since the auto-bootstrap mechanism is only triggered if a majority is down, so that the primary component is anyhow lost already.