percona issue joining node

[pavithra kumar]

2016-05-19 23:24:13 2547 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.0.1.225:4444/xtrabackup_sst//1

2016-05-19 23:24:13 2547 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

2016-05-19 23:24:13 2547 [Note] WSREP: REPL Protocols: 7 (3, 2)

2016-05-19 23:24:13 2547 [Note] WSREP: Service thread queue flushed.

2016-05-19 23:24:13 2547 [Note] WSREP: Assign initial position for certification: 2183133, protocol version: 3

2016-05-19 23:24:13 2547 [Note] WSREP: Service thread queue flushed.

2016-05-19 23:24:13 2547 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (b4821329-1e39-11e6-8378-baa69bfe7dc9) does not match group state UUID (e8e2feed-1753-11e6-bfe3-972d646081e6): 1 (Operation not permitted)

         at galera/src/replicator_str.cpp:prepare_for_IST():463. IST will be unavailable.

2016-05-19 23:24:13 2547 [Note] WSREP: Member 0.0 (ip-10-0-1-225.ap-southeast-1.compute.internal) requested state transfer from '*any*'. Selected 1.0 (ip-10-0-1-224.ap-southeast-1.compute.internal)(SYNCED) as donor.

2016-05-19 23:24:13 2547 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2183133)

2016-05-19 23:24:13 2547 [Note] WSREP: Requesting state transfer: success, donor: 1

WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /var/lib/mysql//.sst from previous state transfer (20160519 23:24:13.932)

WSREP_SST: [INFO] Proceeding with SST (20160519 23:24:13.934)

WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160519 23:24:13.935)

WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20160519 23:24:13.937)

removed `/var/lib/mysql/auto.cnf'

removed `/var/lib/mysql/ib_logfile0'

removed directory: `/var/lib/mysql/test'

removed `/var/lib/mysql/performance_schema/host_cache.frm'

removed `/var/lib/mysql/performance_schema/rwlock_instances.frm'

removed `/var/lib/mysql/performance_schema/events_waits_summary_global_by_event_name.frm'

removed `/var/lib/mysql/performance_schema/events_statements_summary_by_account_by_event_name.frm'

removed `/var/lib/mysql/performance_schema/accounts.frm'

removed `/var/lib/mysql/performance_schema/table_lock_waits_summary_by_table.frm'

removed `/var/lib/mysql/performance_schema/setup_objects.frm'

WSREP_SST: [INFO] Waiting for SST streaming to complete! (20160519 23:24:13.964)

2016-05-19 23:24:15 2547 [Note] WSREP: (529594ab, 'tcp://0.0.0.0:4567') turning message relay requesting off

WSREP_SST: [ERROR] xtrabackup_checkpoints missing, failed innobackupex/SST on donor (20160519 23:24:23.935)

WSREP_SST: [ERROR] Cleanup after exit with status:2 (20160519 23:24:23.936)

2016-05-19 23:24:23 2547 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.1.225' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '2547'  '' : 2 (No such file or directory)

2016-05-19 23:24:23 2547 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

2016-05-19 23:24:23 2547 [ERROR] WSREP: SST failed: 2 (No such file or directory)

2016-05-19 23:24:23 2547 [ERROR] Aborting

2016-05-19 23:24:23 2547 [Warning] WSREP: 1.0 (ip-10-0-1-224.ap-southeast-1.compute.internal): State transfer to 0.0 (ip-10-0-1-225.ap-southeast-1.compute.internal) failed: -22 (Invalid argument)

2016-05-19 23:24:23 2547 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.

2016-05-19 23:24:23 2547 [Note] WSREP: gcomm: terminating thread

2016-05-19 23:24:23 2547 [Note] WSREP: gcomm: joining thread

2016-05-19 23:24:23 2547 [Note] WSREP: gcomm: closing backend

2016-05-19 23:24:23 2547 [Note] WSREP: view(view_id(NON_PRIM,529594ab,51) memb {

        529594ab,0

} joined {

} left {

} partitioned {

        56560110,0

        6e832bd2,0

})

2016-05-19 23:24:23 2547 [Note] WSREP: view((empty))

2016-05-19 23:24:23 2547 [Note] WSREP: gcomm: closed

2016-05-19 23:24:23 2547 [Note] WSREP: /usr/sbin/mysqld: Terminated.

160519 23:24:23 mysqld_safe mysqld from pid file /var/lib/mysql/ip-10-0-1-225.ap-southeast-1.compute.internal.pid ended

[krunal....@percona.com]

There is failure in SST.

Given that you are using Xtrabackup-v2 check its log files on DONOR and JOINER for more details.

Regards,

Krunal