Cant add new node

[nathan barrett]

Hello, i have a three node mariadb cluster setup, and im trying to add a fourth node into the mix. but hwne i try to join the node, it wont join, i get the following in the log:

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') turning message relay requesting on, nonlive peers: tcp://23.238.33.9

4:4567

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') turning message relay requesting off

150128 10:31:07 [Note] WSREP: declaring 21a0cb35-a6bf-11e4-935f-4b4e35321d91 sta

ble

150128 10:31:07 [Note] WSREP: declaring 7069370e-a61e-11e4-844b-a6332fb3990c sta

ble

150128 10:31:07 [Note] WSREP: declaring 8c5c3a7b-a5f1-11e4-a7c2-63386a2874ef sta

ble

150128 10:31:07 [Note] WSREP: Node 7069370e-a61e-11e4-844b-a6332fb3990c state pr

im

150128 10:31:07 [Note] WSREP: view(view_id(PRIM,21a0cb35-a6bf-11e4-935f-4b4e3532

1d91,25) memb {

        21a0cb35-a6bf-11e4-935f-4b4e35321d91,0

        7069370e-a61e-11e4-844b-a6332fb3990c,0

        7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be,0

        8c5c3a7b-a5f1-11e4-a7c2-63386a2874ef,0

} joined {

} left {

} partitioned {

})

150128 10:31:07 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_i

dx = 2, memb_num = 4

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: sent state msg: 21eda816-a6bf-11e4

-9d36-1b2dbcd40ca4

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: got state msg: 21eda816-a6bf-11e4-

9d36-1b2dbcd40ca4 from 0 (db4)

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: got state msg: 21eda816-a6bf-11e4-

9d36-1b2dbcd40ca4 from 1 (db1)

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: got state msg: 21eda816-a6bf-11e4-

9d36-1b2dbcd40ca4 from 2 (db2)

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: got state msg: 21eda816-a6bf-11e4-

9d36-1b2dbcd40ca4 from 3 (db3)

150128 10:31:07 [Note] WSREP: Quorum results:

        version    = 3,

        component  = PRIMARY,

        conf_id    = 24,

        members    = 3/4 (joined/total),

        act_id     = 1649106,

        last_appl. = 1649089,

        protocols  = 0/5/3 (gcs/repl/appl),

        group UUID = 23e9cb9d-a5d5-11e4-b7c5-173bcc032850

150128 10:31:07 [Note] WSREP: Flow-control interval: [32, 32]

150128 10:31:07 [Note] WSREP: New cluster view: global state: 23e9cb9d-a5d5-11e4

-b7c5-173bcc032850:1649106, view# 25: Primary, number of nodes: 4, my index: 2,

protocol version 3

150128 10:31:07 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notifica

tion.

150128 10:31:07 [Note] WSREP: REPL Protocols: 5 (3, 1)

150128 10:31:07 [Note] WSREP: Service thread queue flushed.

150128 10:31:07 [Note] WSREP: Assign initial position for certification: 1649106

, protocol version: 3

150128 10:31:07 [Note] WSREP: Service thread queue flushed.

150128 10:31:07 [Warning] WSREP: Releasing seqno 1649106 before 1649107 was assi

gned.

150128 10:31:07 [Note] WSREP: declaring 7069370e-a61e-11e4-844b-a6332fb3990c sta

ble

150128 10:31:07 [Note] WSREP: declaring 8c5c3a7b-a5f1-11e4-a7c2-63386a2874ef sta

ble

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') turning message relay requesting on, nonlive peers: tcp://23.238.33.9

4:4567

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: Node 7069370e-a61e-11e4-844b-a6332fb3990c state pr

im

150128 10:31:07 [Note] WSREP: view(view_id(PRIM,7069370e-a61e-11e4-844b-a6332fb3

990c,26) memb {

        7069370e-a61e-11e4-844b-a6332fb3990c,0

        7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be,0

        8c5c3a7b-a5f1-11e4-a7c2-63386a2874ef,0

} joined {

} left {

} partitioned {

        21a0cb35-a6bf-11e4-935f-4b4e35321d91,0

})

150128 10:31:07 [Note] WSREP: forgetting 21a0cb35-a6bf-11e4-935f-4b4e35321d91 (t

cp://23.238.33.94:4567)

150128 10:31:07 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_i

dx = 1, memb_num = 3

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') address 'tcp://23.254.255.157:4567' pointing to uuid 7c0cbd8c-a61e-11

e4-acdd-6bbe2455f9be is blacklisted, skipping

150128 10:31:07 [Note] WSREP: (7c0cbd8c-a61e-11e4-acdd-6bbe2455f9be, 'tcp://0.0.

0.0:4567') turning message relay requesting off

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: sent state msg: ae08c7db-a702-11e4

-9fb7-b38ff46f7b2b

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: got state msg: ae08c7db-a702-11e4-

9fb7-b38ff46f7b2b from 0 (db1)

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: got state msg: ae08c7db-a702-11e4-

9fb7-b38ff46f7b2b from 1 (db2)

150128 10:31:07 [Note] WSREP: STATE EXCHANGE: got state msg: ae08c7db-a702-11e4-

9fb7-b38ff46f7b2b from 2 (db3)

150128 10:31:07 [Note] WSREP: Quorum results:

        version    = 3,

        component  = PRIMARY,

        conf_id    = 25,

        members    = 3/3 (joined/total),

        act_id     = 1649122,

        last_appl. = 1649089,

        protocols  = 0/5/3 (gcs/repl/appl),

        group UUID = 23e9cb9d-a5d5-11e4-b7c5-173bcc032850

150128 10:31:07 [Note] WSREP: Flow-control interval: [28, 28]

150128 10:31:07 [Note] WSREP: New cluster view: global state: 23e9cb9d-a5d5-11e4

-b7c5-173bcc032850:1649122, view# 26: Primary, number of nodes: 3, my index: 1,

protocol version 3

150128 10:31:07 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notifica

tion.

150128 10:31:07 [Note] WSREP: REPL Protocols: 5 (3, 1)

150128 10:31:07 [Note] WSREP: Service thread queue flushed.

150128 10:31:07 [Note] WSREP: Assign initial position for certification: 1649122

, protocol version: 3

150128 10:31:07 [Note] WSREP: Service thread queue flushed.

150128 10:31:07 [Warning] WSREP: Releasing seqno 1649122 before 1649123 was assi

gned.

150128 10:31:13 [Note] WSREP:  cleaning up 21a0cb35-a6bf-11e4-935f-4b4e35321d91

(tcp://23.238.33.94:4567)

[Donovan Sydow]

Nate,

Looks like there may be communication issues between the fourth node and the others. Without the configuration it is hard to tell, but it looks like communication is blocked to this other member node: tcp://23.238.33.94:4567. I would check firewall, routing, and verify reachability in both directions of all the nodes from this new one. Keep in mind with an even number of nodes, you run a high risk of going into "split-brain" mode in case there is a problem, so I would recommend setting garbd on another host as well.

Cheers,

Donovan