Why does this node lose membership/get partitioned?

45 views

Skip to first unread message

Nik

unread,

Jun 29, 2017, 11:58:07 AM6/29/17

to codership

Hi,

I'm trying to understand why a node in my cluster was no longer recognized as a member of the component during some network hiccups. There are 5 nodes total, and 2 of them (fb1284f8,bd9d2616) were having networking issues at the time. I was expecting that those two would be dropped from the the cluster, which I see occuring, and continue processing since 3 nodes were still up. But then all of a sudden another node (636f2a08) decides that it is partitioned from the cluster, which triggers none of my nodes to be in the primary component and no writes/reads allowed altogether.

I see connection issues being recorded for nodes fb1284f8 and bd9d2616, but nothing regarding connection issues between 636f2a08 and 5c03655f or 900d7d8d . So i don't see why 636f2a08 is reporting its view initially as all other nodes were partitioned. It seems like it corrects itself and declares 5c03655f and 900d7d8d as stable, yet its still set as non-primary component, and does not allow any read/writes until fb1284f8 and bd9d2616 were able to connect again.

Was 636f2a08 actually having network issues with 5c03655f and 900d7d8d at exactly 02:22:03? Which was why it could not see those nodes at the time? But if so why would it stay as non-primary until fb1284f8 and bd9d2616 were able to connect again?

Here are the logs for node 636f2a08, lemme know if you need further logs from this or any other nodes.

2017-06-27T02:20:57-04:00 db2 mysqld: 170627 2:20:57 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer fb1284f8 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:20:57-04:00 db2 mysqld: 170627 2:20:57 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.20:4567

2017-06-27T02:20:58-04:00 db2 mysqld: 170627 2:20:58 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') reconnecting to fb1284f8 (ssl://10.5.5.20:4567), attempt 0

2017-06-27T02:20:59-04:00 db2 mysqld: 170627 2:20:59 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:00-04:00 db2 mysqld: 170627 2:21:00 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') reconnecting to bd9d2616 (ssl://10.5.5.21:4567), attempt 0

2017-06-27T02:21:01-04:00 db2 mysqld: 170627 2:21:01 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:03-04:00 db2 mysqld: 170627 2:21:03 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:05-04:00 db2 mysqld: 170627 2:21:05 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:07-04:00 db2 mysqld: 170627 2:21:07 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:09-04:00 db2 mysqld: 170627 2:21:09 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:11-04:00 db2 mysqld: 170627 2:21:11 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:13-04:00 db2 mysqld: 170627 2:21:13 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:15-04:00 db2 mysqld: 170627 2:21:15 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:17-04:00 db2 mysqld: 170627 2:21:17 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:17-04:00 db2 mysqld: 170627 2:21:17 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:34329 local endpoint ssl://10.0.15.20:4567 cipher: AES128-SHA compression:

2017-06-27T02:21:17-04:00 db2 mysqld: 170627 2:21:17 [Warning] WSREP: write_handler(): protocol is shutdown (336396495: 'error:140D00CF:SSL routines:SSL_write:protocol is shutdown')

2017-06-27T02:21:17-04:00 db2 mysqld: 170627 2:21:17 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:47568 local endpoint ssl://10.0.15.20:4567 cipher: AES128-SHA compression:

2017-06-27T02:21:18-04:00 db2 mysqld: 170627 2:21:18 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-06-27T02:21:18-04:00 db2 mysqld: 170627 2:21:18 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:4567 local endpoint ssl://10.0.15.20:53870 cipher: AES128-SHA compression:

2017-06-27T02:21:19-04:00 db2 mysqld: 170627 2:21:19 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:22-04:00 db2 mysqld: 170627 2:21:22 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer fb1284f8 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:23-04:00 db2 mysqld: 170627 2:21:23 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:23-04:00 db2 mysqld: 170627 2:21:23 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') reconnecting to fb1284f8 (ssl://10.5.5.20:4567), attempt 0

2017-06-27T02:21:26-04:00 db2 mysqld: 170627 2:21:26 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:27-04:00 db2 mysqld: 170627 2:21:27 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:30-04:00 db2 mysqld: 170627 2:21:30 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:31-04:00 db2 mysqld: 170627 2:21:31 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:31-04:00 db2 mysqld: 170627 2:21:31 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:47576 local endpoint ssl://10.0.15.20:4567 cipher: AES128-SHA compression:

2017-06-27T02:21:32-04:00 db2 mysqld: 170627 2:21:32 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:34341 local endpoint ssl://10.0.15.20:4567 cipher: AES128-SHA compression:

2017-06-27T02:21:32-04:00 db2 mysqld: 170627 2:21:32 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection established to bd9d2616 ssl://10.5.5.21:4567

2017-06-27T02:21:33-04:00 db2 mysqld: 170627 2:21:33 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:4567 local endpoint ssl://10.0.15.20:53877 cipher: AES128-SHA compression:

2017-06-27T02:21:35-04:00 db2 mysqld: 170627 2:21:35 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:47579 local endpoint ssl://10.0.15.20:4567 cipher: AES128-SHA compression:

2017-06-27T02:21:38-04:00 db2 mysqld: 170627 2:21:38 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') reconnecting to bd9d2616 (ssl://10.5.5.21:4567), attempt 0

2017-06-27T02:21:40-04:00 db2 mysqld: 170627 2:21:40 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:41-04:00 db2 mysqld: 170627 2:21:41 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:44-04:00 db2 mysqld: 170627 2:21:44 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:45-04:00 db2 mysqld: 170627 2:21:45 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:48-04:00 db2 mysqld: 170627 2:21:48 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:49-04:00 db2 mysqld: 170627 2:21:49 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:52-04:00 db2 mysqld: 170627 2:21:52 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:53-04:00 db2 mysqld: 170627 2:21:53 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:56-04:00 db2 mysqld: 170627 2:21:56 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:21:57-04:00 db2 mysqld: 170627 2:21:57 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:22:00-04:00 db2 mysqld: 170627 2:22:00 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-06-27T02:22:01-04:00 db2 mysqld: 170627 2:22:01 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: view(view_id(NON_PRIM,5c03655f,1304) memb {

2017-06-27T02:22:03-04:00 db2 mysqld: 636f2a08,1

2017-06-27T02:22:03-04:00 db2 mysqld: } joined {

2017-06-27T02:22:03-04:00 db2 mysqld: } left {

2017-06-27T02:22:03-04:00 db2 mysqld: } partitioned {

2017-06-27T02:22:03-04:00 db2 mysqld: 5c03655f,1

2017-06-27T02:22:03-04:00 db2 mysqld: 900d7d8d,2

2017-06-27T02:22:03-04:00 db2 mysqld: bd9d2616,3

2017-06-27T02:22:03-04:00 db2 mysqld: fb1284f8,3

2017-06-27T02:22:03-04:00 db2 mysqld: })

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: Flow-control interval: [16, 16]

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: Received NON-PRIMARY.

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 117467613)

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: New cluster view: global state: 6d20929f-638c-11e4-bd79-4664d0d5b3d0:117467613, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: view(view_id(NON_PRIM,636f2a08,1305) memb {

2017-06-27T02:22:03-04:00 db2 mysqld: 636f2a08,1

2017-06-27T02:22:03-04:00 db2 mysqld: } joined {

2017-06-27T02:22:03-04:00 db2 mysqld: } left {

2017-06-27T02:22:03-04:00 db2 mysqld: } partitioned {

2017-06-27T02:22:03-04:00 db2 mysqld: 5c03655f,1

2017-06-27T02:22:03-04:00 db2 mysqld: 900d7d8d,2

2017-06-27T02:22:03-04:00 db2 mysqld: bd9d2616,3

2017-06-27T02:22:03-04:00 db2 mysqld: fb1284f8,3

2017-06-27T02:22:03-04:00 db2 mysqld: })

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: Flow-control interval: [16, 16]

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: Received NON-PRIMARY.

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: declaring 5c03655f at ssl://10.0.5.20:4567 stable

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: declaring 900d7d8d at ssl://10.6.5.20:4567 stable

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: view(view_id(NON_PRIM,5c03655f,1306) memb {

2017-06-27T02:22:03-04:00 db2 mysqld: 5c03655f,1

2017-06-27T02:22:03-04:00 db2 mysqld: 636f2a08,1

2017-06-27T02:22:03-04:00 db2 mysqld: 900d7d8d,2

2017-06-27T02:22:03-04:00 db2 mysqld: } joined {

2017-06-27T02:22:03-04:00 db2 mysqld: } left {

2017-06-27T02:22:03-04:00 db2 mysqld: } partitioned {

2017-06-27T02:22:03-04:00 db2 mysqld: bd9d2616,3

2017-06-27T02:22:03-04:00 db2 mysqld: fb1284f8,3

2017-06-27T02:22:03-04:00 db2 mysqld: })

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 3

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: Flow-control interval: [28, 28]

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: Received NON-PRIMARY.

2017-06-27T02:22:03-04:00 db2 mysqld: 170627 2:22:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2017-06-27T02:22:04-04:00 db2 mysqld: 170627 2:22:04 [Note] WSREP: (636f2a08, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

Nik

unread,

Jul 7, 2017, 10:35:04 AM7/7/17

to codership

Had this happen again but with another node this time (5c03655f), just decided it was no longer a member and so lost my primary cluster. Would anyone be able to explain why a node would lose membership with the remaining nodes (636f2a08 / 900d7d8d) if there are no logs showing a connection issue between them?

2017-07-07T02:10:06-04:00 db1 mysqld: 170707 2:10:06 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer fb1284f8 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:10:06-04:00 db1 mysqld: 170707 2:10:06 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.20:4567

2017-07-07T02:10:07-04:00 db1 mysqld: 170707 2:10:07 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to fb1284f8 (ssl://10.5.5.20:4567), attempt 0

2017-07-07T02:10:10-04:00 db1 mysqld: 170707 2:10:10 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:10:11-04:00 db1 mysqld: 170707 2:10:11 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:51330 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:10:11-04:00 db1 mysqld: 170707 2:10:11 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:10:13-04:00 db1 mysqld: 170707 2:10:13 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:4567 local endpoint ssl://10.0.5.20:59095 cipher: AES128-SHA compression:

2017-07-07T02:10:14-04:00 db1 mysqld: 170707 2:10:14 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:10:14-04:00 db1 mysqld: 170707 2:10:14 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:55508 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:10:14-04:00 db1 mysqld: 170707 2:10:14 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:10:14-04:00 db1 mysqld: 170707 2:10:14 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to bd9d2616 ssl://10.5.5.21:4567

2017-07-07T02:10:18-04:00 db1 mysqld: 170707 2:10:18 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting off

2017-07-07T02:10:41-04:00 db1 mysqld: 170707 2:10:41 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.20:4567

2017-07-07T02:10:41-04:00 db1 mysqld: 170707 2:10:41 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:51335 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:10:42-04:00 db1 mysqld: 170707 2:10:42 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to fb1284f8 (ssl://10.5.5.20:4567), attempt 0

2017-07-07T02:10:44-04:00 db1 mysqld: 170707 2:10:44 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:4567 local endpoint ssl://10.0.5.20:59097 cipher: AES128-SHA compression:

2017-07-07T02:10:44-04:00 db1 mysqld: 170707 2:10:44 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:10:47-04:00 db1 mysqld: 170707 2:10:47 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting off

2017-07-07T02:11:00-04:00 db1 mysqld: 170707 2:11:00 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:00-04:00 db1 mysqld: 170707 2:11:00 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer fb1284f8 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:00-04:00 db1 mysqld: 170707 2:11:00 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.20:4567 ssl://10.5.5.21:4567

2017-07-07T02:11:01-04:00 db1 mysqld: 170707 2:11:01 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to bd9d2616 (ssl://10.5.5.21:4567), attempt 0

2017-07-07T02:11:01-04:00 db1 mysqld: 170707 2:11:01 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to fb1284f8 (ssl://10.5.5.20:4567), attempt 0

2017-07-07T02:11:04-04:00 db1 mysqld: 170707 2:11:04 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:04-04:00 db1 mysqld: 170707 2:11:04 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:08-04:00 db1 mysqld: 170707 2:11:08 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:08-04:00 db1 mysqld: 170707 2:11:08 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:11-04:00 db1 mysqld: 170707 2:11:11 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:55515 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:11-04:00 db1 mysqld: 170707 2:11:11 [Warning] WSREP: write_handler(): protocol is shutdown (336396495: 'error:140D00CF:SSL routines:SSL_write:protocol is shutdown')

2017-07-07T02:11:11-04:00 db1 mysqld: 170707 2:11:11 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:4567 local endpoint ssl://10.0.5.20:50118 cipher: AES128-SHA compression:

2017-07-07T02:11:11-04:00 db1 mysqld: 170707 2:11:11 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:4567 local endpoint ssl://10.0.5.20:59105 cipher: AES128-SHA compression:

2017-07-07T02:11:11-04:00 db1 mysqld: 170707 2:11:11 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:55519 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:11-04:00 db1 mysqld: 170707 2:11:11 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:51344 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:14-04:00 db1 mysqld: 170707 2:11:14 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr timed out, no messages seen in PT3S

2017-07-07T02:11:15-04:00 db1 mysqld: 170707 2:11:15 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer fb1284f8 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:15-04:00 db1 mysqld: 170707 2:11:15 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:19-04:00 db1 mysqld: 170707 2:11:19 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:19-04:00 db1 mysqld: 170707 2:11:19 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:23-04:00 db1 mysqld: 170707 2:11:23 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:23-04:00 db1 mysqld: 170707 2:11:23 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:27-04:00 db1 mysqld: 170707 2:11:27 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:27-04:00 db1 mysqld: 170707 2:11:27 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:51356 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:27-04:00 db1 mysqld: 170707 2:11:27 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:55531 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:27-04:00 db1 mysqld: 170707 2:11:27 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:4567 local endpoint ssl://10.0.5.20:50126 cipher: AES128-SHA compression:

2017-07-07T02:11:27-04:00 db1 mysqld: 170707 2:11:27 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:11:27-04:00 db1 mysqld: 170707 2:11:27 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to bd9d2616 ssl://10.5.5.21:4567

2017-07-07T02:11:30-04:00 db1 mysqld: 170707 2:11:30 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting off

2017-07-07T02:11:31-04:00 db1 mysqld: 170707 2:11:31 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer fb1284f8 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:31-04:00 db1 mysqld: 170707 2:11:31 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:31-04:00 db1 mysqld: 170707 2:11:31 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.20:4567 ssl://10.5.5.21:4567

2017-07-07T02:11:32-04:00 db1 mysqld: 170707 2:11:32 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to fb1284f8 (ssl://10.5.5.20:4567), attempt 0

2017-07-07T02:11:32-04:00 db1 mysqld: 170707 2:11:32 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to bd9d2616 (ssl://10.5.5.21:4567), attempt 0

2017-07-07T02:11:35-04:00 db1 mysqld: 170707 2:11:35 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:35-04:00 db1 mysqld: 170707 2:11:35 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:39-04:00 db1 mysqld: 170707 2:11:39 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:39-04:00 db1 mysqld: 170707 2:11:39 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer 00000000 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:40-04:00 db1 mysqld: 170707 2:11:40 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:51362 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:40-04:00 db1 mysqld: 170707 2:11:40 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:55537 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:40-04:00 db1 mysqld: 170707 2:11:40 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:11:40-04:00 db1 mysqld: 170707 2:11:40 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to bd9d2616 ssl://10.5.5.21:4567

2017-07-07T02:11:40-04:00 db1 mysqld: 170707 2:11:40 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:4567 local endpoint ssl://10.0.5.20:59118 cipher: AES128-SHA compression:

2017-07-07T02:11:42-04:00 db1 mysqld: 170707 2:11:42 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:11:46-04:00 db1 mysqld: 170707 2:11:46 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting off

2017-07-07T02:11:46-04:00 db1 mysqld: 170707 2:11:46 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:46-04:00 db1 mysqld: 170707 2:11:46 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.21:4567

2017-07-07T02:11:47-04:00 db1 mysqld: 170707 2:11:47 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to bd9d2616 (ssl://10.5.5.21:4567), attempt 0

2017-07-07T02:11:49-04:00 db1 mysqld: 170707 2:11:49 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:4567 local endpoint ssl://10.0.5.20:50135 cipher: AES128-SHA compression:

2017-07-07T02:11:49-04:00 db1 mysqld: 170707 2:11:49 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:55539 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:11:49-04:00 db1 mysqld: 170707 2:11:49 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to bd9d2616 ssl://10.5.5.21:4567

2017-07-07T02:11:50-04:00 db1 mysqld: 170707 2:11:50 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to bd9d2616 ssl://10.5.5.21:4567

2017-07-07T02:11:53-04:00 db1 mysqld: 170707 2:11:53 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting off

2017-07-07T02:11:58-04:00 db1 mysqld: 170707 2:11:58 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:11:58-04:00 db1 mysqld: 170707 2:11:58 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.21:4567

2017-07-07T02:11:59-04:00 db1 mysqld: 170707 2:11:59 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to bd9d2616 (ssl://10.5.5.21:4567), attempt 0

2017-07-07T02:11:59-04:00 db1 mysqld: 170707 2:11:59 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer fb1284f8 with addr ssl://10.5.5.20:4567 timed out, no messages seen in PT3S

2017-07-07T02:12:00-04:00 db1 mysqld: 170707 2:12:00 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:51364 local endpoint ssl://10.0.5.20:4567 cipher: AES128-SHA compression:

2017-07-07T02:12:00-04:00 db1 mysqld: 170707 2:12:00 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to fb1284f8 (ssl://10.5.5.20:4567), attempt 0

2017-07-07T02:12:00-04:00 db1 mysqld: 170707 2:12:00 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.21:4567 local endpoint ssl://10.0.5.20:50138 cipher: AES128-SHA compression:

2017-07-07T02:12:00-04:00 db1 mysqld: 170707 2:12:00 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:12:01-04:00 db1 mysqld: 170707 2:12:01 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to bd9d2616 ssl://10.5.5.21:4567

2017-07-07T02:12:02-04:00 db1 mysqld: 170707 2:12:02 [Note] WSREP: SSL handshake successful, remote endpoint ssl://10.5.5.20:4567 local endpoint ssl://10.0.5.20:59125 cipher: AES128-SHA compression:

2017-07-07T02:12:05-04:00 db1 mysqld: 170707 2:12:05 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting off

2017-07-07T02:12:05-04:00 db1 mysqld: 170707 2:12:05 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection established to fb1284f8 ssl://10.5.5.20:4567

2017-07-07T02:12:05-04:00 db1 mysqld: 170707 2:12:05 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

2017-07-07T02:12:08-04:00 db1 mysqld: 170707 2:12:08 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting off

2017-07-07T02:12:09-04:00 db1 mysqld: 170707 2:12:09 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') connection to peer bd9d2616 with addr ssl://10.5.5.21:4567 timed out, no messages seen in PT3S

2017-07-07T02:12:09-04:00 db1 mysqld: 170707 2:12:09 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.5.5.21:4567

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: (5c03655f, 'ssl://0.0.0.0:4567') reconnecting to bd9d2616 (ssl://10.5.5.21:4567), attempt 0

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: view(view_id(NON_PRIM,5c03655f,1309) memb {

2017-07-07T02:12:10-04:00 db1 mysqld: 5c03655f,1

2017-07-07T02:12:10-04:00 db1 mysqld: } joined {

2017-07-07T02:12:10-04:00 db1 mysqld: } left {

2017-07-07T02:12:10-04:00 db1 mysqld: } partitioned {

2017-07-07T02:12:10-04:00 db1 mysqld: 636f2a08,1

2017-07-07T02:12:10-04:00 db1 mysqld: 900d7d8d,2

2017-07-07T02:12:10-04:00 db1 mysqld: bd9d2616,3

2017-07-07T02:12:10-04:00 db1 mysqld: fb1284f8,3

2017-07-07T02:12:10-04:00 db1 mysqld: })

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: Flow-control interval: [16, 16]

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: Received NON-PRIMARY.

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 118842273)

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: New cluster view: global state: 6d20929f-638c-11e4-bd79-4664d0d5b3d0:118842273, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: view(view_id(NON_PRIM,5c03655f,1310) memb {

2017-07-07T02:12:10-04:00 db1 mysqld: 5c03655f,1

2017-07-07T02:12:10-04:00 db1 mysqld: } joined {

2017-07-07T02:12:10-04:00 db1 mysqld: } left {

2017-07-07T02:12:10-04:00 db1 mysqld: } partitioned {

2017-07-07T02:12:10-04:00 db1 mysqld: 636f2a08,1

2017-07-07T02:12:10-04:00 db1 mysqld: 900d7d8d,2

2017-07-07T02:12:10-04:00 db1 mysqld: bd9d2616,3

2017-07-07T02:12:10-04:00 db1 mysqld: fb1284f8,3

2017-07-07T02:12:10-04:00 db1 mysqld: })

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: Flow-control interval: [16, 16]

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: Received NON-PRIMARY.

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Warning] WSREP: Send action {(nil), 823, TORDERED} returned -107 (Transport endpoint is not connected)

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Warning] WSREP: Send action {(nil), 819, TORDERED} returned -107 (Transport endpoint is not connected)

2017-07-07T02:12:10-04:00 db1 mysqld: 170707 2:12:10 [Warning] WSREP: Send action {(nil), 1325, TORDERED} returned -107 (Transport endpoint is not connected)