Unable to join in second node into the cluster

[Jan Hrnčíř]

Hello community, 

we fail in every attempt to join the existing Galera-4 cluster with second node.

System info: 

CentOS 7

Galera-4 - 26.4.6

MariaDB 10.4.15

Whole SST phase is done correctly but when comes to IST phase the joining node is not able to sync with the cluster.

Joining node:

[Note] WSREP: Cert. index preload up to 332924104

[Note] WSREP: ####### IST applying starts with 332924105

[Note] WSREP: ####### IST current seqno initialized to 332190849

[Note] WSREP: Receiving IST...  0.0% (    0/16495 events) complete.

[Note] WSREP: IST preload starting at 332190849

[Note] WSREP: Service thread queue flushed.

[Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:332190848, protocol version: 5

[ERROR] WSREP: RecordSet checksum does not match:

computed: 595a41c4 c8331684 918397e5 214fe656

found:    573532d1 8f59a603 a6fc2a05 63c528d3: 22 (Invalid argument)

         at galerautils/src/gu_rset.cpp:checksum():551

2020-11-30 16:33:58 0 [ERROR] WSREP: Writeset deserialization failed: Writeset checksum failed: 22 (Invalid argument)

         at galera/src/write_set_ng.hpp:checksum_fin():864

         at galera/src/trx_handle.hpp:unserialize():533

WS flags:      0

Trx proto:     5

Trx source:    00000000-0000-0000-0000-000000000000

Trx conn_id:   18446744073709551615

Trx trx_id:    18446744073709551615

Trx last_seen: -1

[ERROR] WSREP: got exception while reading IST stream: Writeset checksum failed: 22 (Invalid argument)

         at galera/src/write_set_ng.hpp:checksum_fin():864

         at galera/src/trx_handle.hpp:unserialize():533

[ERROR] WSREP: IST didn't contain all write sets, expected last: 332207343 last received: 332190876

[ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)

         at galera/src/replicator_smm.hpp:pop_front():315. Null event.

Donor:

2020-11-30 20:15:21 0 [ERROR] WSREP: async IST sender failed to serve tcp://x.x.x.x:4568: Unrecognized writeset version: -1: 71 (Protocol error)

         at galera/src/write_set_ng.hpp:version():96

2020-11-30 20:15:21 0 [Note] WSREP: async IST sender served

Wsrep configuration:

wsrep_provider_options="cert.log_conflicts=ON; gcs.fc_limit=400; gcs.fc_factor=0.9; gcache.size=8192M; gcache.page_size=1024M; evs.keepalive_period=PT3S; evs.inactive_check_period=PT10S; evs.suspect_timeout=PT30S; evs.inactive_timeout=PT1M"

Thanks in advance

JH

[Michal Lohniský]

Hi, we have same problem. Did you solve it?

[Michal Lohniský]

JH:

> Hi Michal,

> Well, it's been a while. I suppose I made a full resync / new node...

> What prevents you from recreating the node ?

> Regards,

We solved it by rising gcache size:

wsrep_provider_options="gcache.size=5G; gcache.page_size=5G"

and restarting the node. Hopefully it will help someone :-).