Restarting a node in Galera Cluster takes 560 seconds!

70 views
Skip to first unread message

Snorri Bergmann

unread,
Sep 14, 2022, 10:39:43 AM9/14/22
to codership
Hi.

I just set up a three node Galera cluster (10.9.2-MariaDB) last week. Everything went smoothly, I could sync nodes in seconds and bounce at will.

Yesterday I encountered a major problem. Now If I restart mariadb on one of the nodes I can see in the logs that it joins the cluster  immediately. However the time it takes WSREP to request state transfer from the donor is always 560 seconds (give or take 2 secs). Otherwise, the cluster operates normally (replication is immediate etc.) 

Here is a snippet from the log when the server web-cluster-db2 has been restarted (note the lag time in bold). Does anyone have a clue what's going on?

Yhanks in advance,
-S

Snippet from the log: 

View:
  id: ffad2f07-3412-11ed-802e-e3c787820b3f:541
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(3):
        0: ae01cc87-3420-11ed-b896-efa50b185b45, web-cluster-db1
        1: c60d47de-3420-11ed-8445-16d58f164eb1, web-cluster-db3
        2: cca24476-342d-11ed-9fbf-4f610f46cd7c, web-cluster-db2
=================================================
2022-09-14 13:04:43 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-09-14 13:04:43 2 [Note] WSREP: Lowest cert index boundary for CC from group: 518
2022-09-14 13:04:43 2 [Note] WSREP: Min available from gcache for CC from group: 1
2022-09-14 13:04:46 0 [Note] WSREP: (ae01cc87-b896, 'tcp://0.0.0.0:4567') turning message relay requesting off
2022-09-14 13:14:06 0 [Note] WSREP: Member 2.0 (web-cluster-db2) requested state transfer from '*any*'. Selected 1.0 (web-cluster-db3)(SYNCED) as donor.
2022-09-14 13:14:06 0 [Note] WSREP: 1.0 (web-cluster-db3): State transfer to 2.0 (web-cluster-db2) complete.
2022-09-14 13:14:06 0 [Note] WSREP: Member 1.0 (web-cluster-db3) synced with group.
2022-09-14 13:14:08 0 [Note] WSREP: 2.0 (web-cluster-db2): State transfer from 1.0 (web-cluster-db3) complete.
2022-09-14 13:14:08 0 [Note] WSREP: Member 2.0 (web-cluster-db2) synced with group.

Reply all
Reply to author
Forward
0 new messages