Preventing Access to node spending replication data

[Christian Becker]

Hi There,

we have a Galera Cluster with 3 Nodes. One of the Node was destroyed during a hardware crash. So now we've started a full replication of this node.

So far so good. But the first node doesn't reject requests to it. There are hundreds of processes accessing the database and getting stuck in "opening tables" state. So our services didn't have a chance to recognize, that the Node is down and no failover will happen.

Sometimes nodes are getting a "too many connections" error.

Is it possible that our system isn't well configured? I would expect, that the first and the third node will reject any connection to them until replication is finished.

Bests

Chris

[hunter86bg]

Hello Chris,

could you check the following:
1. What is the SST method defined on both nodes ? I'm currently using xtrabackup-v2 which is non-blocking method,but there is also rsync and mysqldump methods
2. Please check that in the logs on node 3 (damaged one) you should see how the node has status of  "JOINER" while node 1 into "DONOR"

To prevent your case , I'm using a xinetd service to run a custom script that provides response OK / Error to the Haproxy's check mechanism (my load balancer).

Also consider enabling the following option on the donor: wsrep-sst-donor-rejects-queries