While starting 2nd node of the galera cluster getting the following messages:

143 views
Skip to first unread message

subhajit chakraborty

unread,
May 9, 2016, 3:48:31 PM5/9/16
to codership
160510 00:44:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160510 00:44:22 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.oTAUUp' --pid-file='/var/lib/mysql/CHNCTLDBS02-recover.pid'
2016-05-10 00:44:23 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-05-10 00:44:23 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30) starting as process 6070 ...
160510 00:44:25 mysqld_safe WSREP: Recovered position cb7f6c86-1602-11e6-ae42-b399ef55c380:0
2016-05-10 00:44:25 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-05-10 00:44:25 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30) starting as process 6105 ...
2016-05-10 00:44:25 6105 [Note] WSREP: Read nil XID from storage engines, skipping position init
2016-05-10 00:44:25 6105 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-3/libgalera_smm.so'
2016-05-10 00:44:25 6105 [Note] WSREP: wsrep_load(): Galera 3.16(r5c765eb) by Codership Oy <in...@codership.com> loaded successfully.
2016-05-10 00:44:25 6105 [Note] WSREP: CRC-32C: using hardware acceleration.
2016-05-10 00:44:25 6105 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2016-05-10 00:44:25 6105 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.32.134.10; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 300M; gcache.size = 300M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false;
2016-05-10 00:44:25 6105 [Note] WSREP: Service thread queue flushed.
2016-05-10 00:44:25 6105 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2016-05-10 00:44:25 6105 [Note] WSREP: wsrep_sst_grab()
2016-05-10 00:44:25 6105 [Note] WSREP: Start replication
2016-05-10 00:44:25 6105 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2016-05-10 00:44:25 6105 [Note] WSREP: protonet asio version 0
2016-05-10 00:44:25 6105 [Note] WSREP: Using CRC-32C for message checksums.
2016-05-10 00:44:25 6105 [Note] WSREP: backend: asio
2016-05-10 00:44:25 6105 [Note] WSREP: gcomm thread scheduling priority set to other:0
2016-05-10 00:44:25 6105 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2016-05-10 00:44:25 6105 [Note] WSREP: restore pc from disk failed
2016-05-10 00:44:25 6105 [Note] WSREP: GMCast version 0
2016-05-10 00:44:25 6105 [Note] WSREP: (3e62f331, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2016-05-10 00:44:25 6105 [Note] WSREP: (3e62f331, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2016-05-10 00:44:25 6105 [Note] WSREP: EVS version 0
2016-05-10 00:44:25 6105 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer '10.32.134.12:,10.32.134.9:,10.32.134.10:'
2016-05-10 00:44:25 6105 [Warning] WSREP: (3e62f331, 'tcp://0.0.0.0:4567') address 'tcp://10.32.134.10:4567' points to own listening address, blacklisting
2016-05-10 00:44:25 6105 [Note] WSREP: (3e62f331, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2016-05-10 00:44:25 6105 [Note] WSREP: declaring ae5cff97 at tcp://10.32.134.12:4567 stable
2016-05-10 00:44:25 6105 [Note] WSREP: Node ae5cff97 state prim
2016-05-10 00:44:25 6105 [Note] WSREP: view(view_id(PRIM,3e62f331,2) memb {
        3e62f331,0
        ae5cff97,0
} joined {
} left {
} partitioned {
})
2016-05-10 00:44:25 6105 [Note] WSREP: save pc into disk
2016-05-10 00:44:25 6105 [Note] WSREP: discarding pending addr without UUID: tcp://10.32.134.9:4567
2016-05-10 00:44:25 6105 [Note] WSREP: gcomm: connected
2016-05-10 00:44:25 6105 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2016-05-10 00:44:25 6105 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2016-05-10 00:44:25 6105 [Note] WSREP: Opened channel 'galera_cluster'
2016-05-10 00:44:25 6105 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2016-05-10 00:44:25 6105 [Note] WSREP: Waiting for SST to complete.
2016-05-10 00:44:25 6105 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 3eaf82e9-161a-11e6-b40d-92912b7a0a86
2016-05-10 00:44:25 6105 [Note] WSREP: STATE EXCHANGE: sent state msg: 3eaf82e9-161a-11e6-b40d-92912b7a0a86
2016-05-10 00:44:25 6105 [Note] WSREP: STATE EXCHANGE: got state msg: 3eaf82e9-161a-11e6-b40d-92912b7a0a86 from 0 (CHNCTLDBS02)
2016-05-10 00:44:25 6105 [Note] WSREP: STATE EXCHANGE: got state msg: 3eaf82e9-161a-11e6-b40d-92912b7a0a86 from 1 (CHNCTLDBS03)
2016-05-10 00:44:25 6105 [Note] WSREP: Quorum results:
        version    = 4,
        component  = PRIMARY,
        conf_id    = 1,
        members    = 1/2 (joined/total),
        act_id     = 0,
        last_appl. = -1,
        protocols  = 0/7/3 (gcs/repl/appl),
        group UUID = cb7f6c86-1602-11e6-ae42-b399ef55c380
2016-05-10 00:44:25 6105 [Note] WSREP: Flow-control interval: [23, 23]
2016-05-10 00:44:25 6105 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
2016-05-10 00:44:25 6105 [Note] WSREP: State transfer required:
        Group state: cb7f6c86-1602-11e6-ae42-b399ef55c380:0
        Local state: 00000000-0000-0000-0000-000000000000:-1
2016-05-10 00:44:25 6105 [Note] WSREP: New cluster view: global state: cb7f6c86-1602-11e6-ae42-b399ef55c380:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
2016-05-10 00:44:25 6105 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-05-10 00:44:25 6105 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.32.134.10' --datadir '/var/lib/mysql/' --defaults-file '/usr/my.cnf' --defaults-group-suffix '' --parent '6105'  '' '
2016-05-10 00:44:25 6105 [Note] WSREP: Prepared SST request: rsync|10.32.134.10:4444/rsync_sst
2016-05-10 00:44:25 6105 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-05-10 00:44:25 6105 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-05-10 00:44:25 6105 [Note] WSREP: Service thread queue flushed.
2016-05-10 00:44:25 6105 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2016-05-10 00:44:25 6105 [Note] WSREP: Service thread queue flushed.
2016-05-10 00:44:25 6105 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (cb7f6c86-1602-11e6-ae42-b399ef55c380): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2016-05-10 00:44:25 6105 [Note] WSREP: Member 0.0 (CHNCTLDBS02) requested state transfer from '*any*'. Selected 1.0 (CHNCTLDBS03)(SYNCED) as donor.
2016-05-10 00:44:25 6105 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
2016-05-10 00:44:25 6105 [Note] WSREP: Requesting state transfer: success, donor: 1
2016-05-10 00:44:27 6105 [Warning] WSREP: 1.0 (CHNCTLDBS03): State transfer to 0.0 (CHNCTLDBS02) failed: -255 (Unknown error 255)
2016-05-10 00:44:27 6105 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort.
2016-05-10 00:44:27 6105 [Note] WSREP: gcomm: terminating thread
2016-05-10 00:44:27 6105 [Note] WSREP: gcomm: joining thread
2016-05-10 00:44:27 6105 [Note] WSREP: gcomm: closing backend
2016-05-10 00:44:28 6105 [Note] WSREP: view(view_id(NON_PRIM,3e62f331,2) memb {
        3e62f331,0
} joined {
} left {
} partitioned {
        ae5cff97,0
})
2016-05-10 00:44:28 6105 [Note] WSREP: (3e62f331, 'tcp://0.0.0.0:4567') turning message relay requesting off
2016-05-10 00:44:28 6105 [Note] WSREP: view((empty))
2016-05-10 00:44:28 6105 [Note] WSREP: gcomm: closed
2016-05-10 00:44:28 6105 [Note] WSREP: /usr/sbin/mysqld: Terminated.
160510 00:44:28 mysqld_safe mysqld from pid file /var/lib/mysql/CHNCTLDBS02.pid ended
WSREP_SST: [ERROR] Parent mysqld process (PID:6105) terminated unexpectedly. (20160510 00:44:28.956)
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 6144 (20160510 00:44:28.958)
WSREP_SST: [INFO] Joiner cleanup done. (20160510 00:44:29.462)



Please help.
Reply all
Reply to author
Forward
0 new messages