ERROR! The server quit without updating PID file (/var/lib/mysql/s2.hidb.bimsym.com.pid).
When i check .err file it shows below error.
161008 07:22:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
161008 07:22:03 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.AZGzjH' --pid-file='/var/lib/mysql/s2.xxxxx.xxxxx.com-recover.pid'
2016-10-08 07:22:04 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-10-08 07:22:04 0 [Note] /usr/sbin/mysqld (mysqld 5.6.33) starting as process 11766 ...
161008 07:22:08 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
2016-10-08 07:22:09 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-10-08 07:22:09 0 [Note] /usr/sbin/mysqld (mysqld 5.6.33) starting as process 11807 ...
2016-10-08 07:22:09 11807 [Note] WSREP: Setting wsrep_ready to 0
2016-10-08 07:22:09 11807 [Note] WSREP: Read nil XID from storage engines, skipping position init
2016-10-08 07:22:09 11807 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-3/libgalera_smm.so'
2016-10-08 07:22:09 11807 [Note] WSREP: wsrep_load(): Galera 3.18(r3b41103) by Codership Oy <
in...@codership.com> loaded successfully.
2016-10-08 07:22:09 11807 [Note] WSREP: CRC-32C: using hardware acceleration.
2016-10-08 07:22:09 11807 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2016-10-08 07:22:09 11807 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 67.xxx.xxx.xxx.xxx; base_port = 4567; cert.log_conflicts = ON; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0;
gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false
2016-10-08 07:22:09 11807 [Note] WSREP: Service thread queue flushed.
2016-10-08 07:22:09 11807 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2016-10-08 07:22:09 11807 [Note] WSREP: wsrep_sst_grab()
2016-10-08 07:22:09 11807 [Note] WSREP: Start replication
2016-10-08 07:22:09 11807 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2016-10-08 07:22:09 11807 [Note] WSREP: protonet asio version 0
2016-10-08 07:22:09 11807 [Note] WSREP: Using CRC-32C for message checksums.
2016-10-08 07:22:09 11807 [Note] WSREP: backend: asio
2016-10-08 07:22:09 11807 [Note] WSREP: gcomm thread scheduling priority set to other:0
2016-10-08 07:22:09 11807 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2016-10-08 07:22:09 11807 [Note] WSREP: restore pc from disk failed
2016-10-08 07:22:09 11807 [Note] WSREP: GMCast version 0
2016-10-08 07:22:09 11807 [Note] WSREP: (738de7af, 'tcp://
0.0.0.0:4567') multicast: , ttl: 1
2016-10-08 07:22:09 11807 [Note] WSREP: EVS version 0
2016-10-08 07:22:09 11807 [Note] WSREP: gcomm: connecting to group 'xxxxx_cluster', peer '69.xxx.xxx.xxx.xxx:,67.xxx.xxx.xxx.xxx:,208.xxx.xxx.xxx.xxx:'
2016-10-08 07:22:09 11807 [Note] WSREP: (738de7af, 'tcp://
0.0.0.0:4567') connection established to 738de7af tcp://67.xxx.xxx.xxx.xxx:4567
2016-10-08 07:22:09 11807 [Warning] WSREP: (738de7af, 'tcp://
0.0.0.0:4567') address 'tcp://67.xxx.xxx.xxx.xxx:4567' points to own listening address, blacklisting
2016-10-08 07:22:09 11807 [Note] WSREP: (738de7af, 'tcp://
0.0.0.0:4567') connection established to 738de7af tcp://67.xxx.xxx.xxx.xxx:4567
2016-10-08 07:22:09 11807 [Note] WSREP: (738de7af, 'tcp://
0.0.0.0:4567') connection established to 318a4221 tcp://69.xxx.xxx.xxx.xxx:4567
2016-10-08 07:22:09 11807 [Note] WSREP: (738de7af, 'tcp://
0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2016-10-08 07:22:09 11807 [Note] WSREP: (738de7af, 'tcp://
0.0.0.0:4567') connection established to 18805553 tcp://208.xxx.xxx.xxx.xxx:4567
2016-10-08 07:22:09 11807 [Note] WSREP: declaring 18805553 at tcp://208.xxx.xxx.xxx.xxx:4567 stable
2016-10-08 07:22:09 11807 [Note] WSREP: declaring 318a4221 at tcp://69.xxx.xxx.xxx.xxx:4567 stable
2016-10-08 07:22:09 11807 [Note] WSREP: Node 18805553 state prim
2016-10-08 07:22:10 11807 [Note] WSREP: view(view_id(PRIM,18805553,37) memb {
18805553,0
318a4221,0
738de7af,0
} joined {
} left {
} partitioned {
})
2016-10-08 07:22:10 11807 [Note] WSREP: save pc into disk
2016-10-08 07:22:10 11807 [Note] WSREP: gcomm: connected
2016-10-08 07:22:10 11807 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2016-10-08 07:22:10 11807 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2016-10-08 07:22:10 11807 [Note] WSREP: Opened channel 'xxxxx_cluster'
2016-10-08 07:22:10 11807 [Note] WSREP: Waiting for SST to complete.
2016-10-08 07:22:10 11807 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
2016-10-08 07:22:10 11807 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2016-10-08 07:22:10 11807 [Note] WSREP: STATE EXCHANGE: sent state msg: e9acb8da-8d27-11e6-b2d0-6310c7f6d02c
2016-10-08 07:22:10 11807 [Note] WSREP: STATE EXCHANGE: got state msg: e9acb8da-8d27-11e6-b2d0-6310c7f6d02c from 0 (
s3.xxxxx.xxxxx.com)
2016-10-08 07:22:10 11807 [Note] WSREP: STATE EXCHANGE: got state msg: e9acb8da-8d27-11e6-b2d0-6310c7f6d02c from 1 (
s1.xxxxx.xxxxx.com)
2016-10-08 07:22:10 11807 [Note] WSREP: STATE EXCHANGE: got state msg: e9acb8da-8d27-11e6-b2d0-6310c7f6d02c from 2 (
s2.xxxx.xxxxx.com)
2016-10-08 07:22:10 11807 [Note] WSREP: Quorum results:
version = 4,
component = PRIMARY,
conf_id = 176,
members = 2/3 (joined/total),
act_id = 92667,
last_appl. = -1,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = aab00320-83e1-11e6-9065-6bee750e7c1c
2016-10-08 07:22:10 11807 [Note] WSREP: Flow-control interval: [28, 28]
2016-10-08 07:22:10 11807 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 92667)
2016-10-08 07:22:10 11807 [Note] WSREP: State transfer required:
Group state: aab00320-83e1-11e6-9065-6bee750e7c1c:92667
Local state: 00000000-0000-0000-0000-000000000000:-1
2016-10-08 07:22:10 11807 [Note] WSREP: New cluster view: global state: aab00320-83e1-11e6-9065-6bee750e7c1c:92667, view# 177: Primary, number of nodes: 3, my index: 2, protocol version 3
2016-10-08 07:22:10 11807 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-10-08 07:22:10 11807 [Note] WSREP: Setting wsrep_ready to 0
2016-10-08 07:22:10 11807 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '67.xxx.xxx.xxx.xxx' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '11807' '' '
2016-10-08 07:22:10 11807 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '67.xxx.xxx.xxx.xxx' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '11807' ''
Read: '(null)'
2016-10-08 07:22:10 11807 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '67.xxx.xxx.xxx.xxx' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '11807' '' : 2 (No such file or directory)
2016-10-08 07:22:10 11807 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2016-10-08 07:22:10 11807 [ERROR] Aborting
2016-10-08 07:22:12 11807 [Note] WSREP: waiting for client connections to close: 2
2016-10-08 07:22:12 11807 [Note] WSREP: Closing send monitor...
2016-10-08 07:22:12 11807 [Note] WSREP: Closed send monitor.
2016-10-08 07:22:12 11807 [Note] WSREP: gcomm: terminating thread
2016-10-08 07:22:12 11807 [Note] WSREP: gcomm: joining thread
2016-10-08 07:22:12 11807 [Note] WSREP: gcomm: closing backend
2016-10-08 07:22:12 11807 [Note] WSREP: view(view_id(NON_PRIM,18805553,37) memb {
738de7af,0
} joined {
} left {
} partitioned {
18805553,0
318a4221,0
})
2016-10-08 07:22:12 11807 [Note] WSREP: view((empty))
2016-10-08 07:22:12 11807 [Note] WSREP: gcomm: closed
2016-10-08 07:22:12 11807 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2016-10-08 07:22:12 11807 [Note] WSREP: Flow-control interval: [16, 16]
2016-10-08 07:22:12 11807 [Note] WSREP: Received NON-PRIMARY.
2016-10-08 07:22:12 11807 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 92667)
2016-10-08 07:22:12 11807 [Note] WSREP: Received self-leave message.
2016-10-08 07:22:12 11807 [Note] WSREP: Flow-control interval: [0, 0]
2016-10-08 07:22:12 11807 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2016-10-08 07:22:12 11807 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 92667)
2016-10-08 07:22:12 11807 [Note] WSREP: RECV thread exiting 0: Success
2016-10-08 07:22:12 11807 [Note] WSREP: recv_thread() joined.
2016-10-08 07:22:12 11807 [Note] WSREP: Closing replication queue.
2016-10-08 07:22:12 11807 [Note] WSREP: Closing slave action queue.
2016-10-08 07:22:12 11807 [Note] WSREP: Service disconnected.
2016-10-08 07:22:12 11807 [Note] WSREP: closing wsrep thread 2
2016-10-08 07:22:12 11807 [Note] WSREP: closing wsrep thread 1
2016-10-08 07:22:12 11807 [Note] WSREP: WSREP rollback thread wakes for signal
2016-10-08 07:22:12 11807 [Note] WSREP: WSREP rollback thread has empty abort queue
2016-10-08 07:22:12 11807 [Note] WSREP: rollbacker thread exiting
2016-10-08 07:22:13 11807 [Note] WSREP: Some threads may fail to exit.
2016-10-08 07:22:13 11807 [Note] Binlog end
2016-10-08 07:22:13 11807 [Note] /usr/sbin/mysqld: Shutdown complete
Error in my_thread_global_end(): 2 threads didn't exit
161008 07:22:18 mysqld_safe mysqld from pid file /var/lib/mysql/s2.xxxxx.xxxxx.com.pid ended
Please help me to resolve this error.