Cant join node cluster

[syam...@genusis.com]

I received this error:

2018-02-20 16:24:58 140441185429632 [Note] WSREP: save pc into disk

2018-02-20 16:24:58 140441185429632 [Note] WSREP: gcomm: connected

2018-02-20 16:24:58 140441185429632 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636

2018-02-20 16:24:58 140441185429632 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)

2018-02-20 16:24:58 140441185429632 [Note] WSREP: Opened channel 'MariaDB_Cluster'

2018-02-20 16:24:58 140435481675520 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2

2018-02-20 16:24:58 140441185429632 [Note] WSREP: Waiting for SST to complete.

2018-02-20 16:24:58 140435481675520 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 89fb25a6-1617-11e8-xxx

2018-02-20 16:24:58 140435481675520 [Note] WSREP: STATE EXCHANGE: sent state msg: 89fb25a6-1617-11e8-xxx

2018-02-20 16:24:58 140435481675520 [Note] WSREP: STATE EXCHANGE: got state msg: 89fb25a6-1617-11e8-xxx from 0 (sdb6)

2018-02-20 16:24:58 140435481675520 [Note] WSREP: STATE EXCHANGE: got state msg: 89fb25a6-1617-11e8-xxx from 1 (sdb3)

2018-02-20 16:24:58 140435481675520 [Note] WSREP: Quorum results:

        version    = 4,

        component  = PRIMARY,

        conf_id    = 5,

        members    = 1/2 (joined/total),

        act_id     = 4,

        last_appl. = -1,

        protocols  = 0/8/3 (gcs/repl/appl),

        group UUID = ec73d912-160a-xxx

2018-02-20 16:24:58 140435481675520 [Note] WSREP: Flow-control interval: [724, 724]

2018-02-20 16:24:58 140435481675520 [Note] WSREP: Trying to continue unpaused monitor

2018-02-20 16:24:58 140435481675520 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4)

2018-02-20 16:24:58 140440934733568 [Note] WSREP: State transfer required:

        Group state: ec73d912-160a-xxx:4

        Local state: 00000000-0000-0000-0000-000000000000:-1

2018-02-20 16:24:58 140440934733568 [Note] WSREP: New cluster view: global state: ec73d912-xxx:4, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 3

2018-02-20 16:24:58 140440934733568 [Warning] WSREP: Gap in state sequence. Need state transfer.

2018-02-20 16:24:58 140440934733568 [Note] WSREP: Setting wsrep_ready to 0

2018-02-20 16:24:58 140435473282816 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.0.25' --datadir '/var/lib/mysql/'   --parent '88274' --binlog 'mysql-bin' '

/usr//bin/wsrep_sst_common: line 265: [: gzip: binary operator expected

WSREP_SST: [INFO] Streaming with xbstream (20180220 16:24:59.162)

WSREP_SST: [INFO] Using socat as streamer (20180220 16:24:59.163)

WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20180220 16:24:59.167)

WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f  -i 10 -N joiner -F '%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p'  |  gzip -dc | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20180220 16:24:59.188)

2018-02-20 16:25:01 140435490068224 [Note] WSREP: (89ae7932, 'tcp://0.0.0.0:4567') connection to peer 89ae7932 with addr tcp://10.0.0.1:4567 timed out, no messages seen in PT3S

2018-02-20 16:25:01 140435490068224 [Note] WSREP: (89ae7932, 'tcp://0.0.0.0:4567') turning message relay requesting off

   joiner: => Rate:[   0 B/s] Avg:[   0 B/s] Elapsed:0:01:20

Terminated

WSREP_SST: [ERROR] Removing /tmp/tmp.gupXSUIQVe/xtrabackup_galera_info file due to signal (20180220 16:26:28.552)

WSREP_SST: [INFO] NOTE: Joiner-Recv-gtid took 89 seconds (20180220 16:26:28.557)

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 143 32 143 143 (20180220 16:26:28.559)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20180220 16:26:28.561)

2018-02-20 16:26:28 140435473282816 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.0.1' --datadir '/var/lib/mysql/'   --parent '88274' --binlog 'mysql-bin'

        Read: '(null)'

2018-02-20 16:26:28 140435473282816 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.0.1' --datadir '/var/lib/mysql/'   --parent '88274' --binlog 'mysql-bin' : 32 (Broken pipe)

2018-02-20 16:26:28 140440934733568 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.

2018-02-20 16:26:28 140440934733568 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit

Please advise me what should I do?

Im using centos 7

[krunal....@percona.com]

Which version of PXC are you using ?

Did you accidentally modified wsrep_sst_common script.

/usr//bin/wsrep_sst_common: line 265: [: gzip: binary operator expected

Can you check what is causing this script error.

Regards,

Krunal

[syam...@genusis.com]

Im using this version 2.4.9

 

Actually I replaced wsrep_sst_common. 

Anyway I no longer receive that error. But I keep on receiving this error:

innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)

xb_stream_write_data() failed.

compress: write to the destination stream failed.

innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)

[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.

[01] xtrabackup: Error: failed to copy datafile.

this is the log from the donor

2018-02-25 19:08:34 140545137559296 [Note] WSREP: Quorum results:

version    = 4,

component  = PRIMARY,

conf_id    = 1,

members    = 1/2 (joined/total),

act_id     = 121482637,

last_appl. = 121482594,

protocols  = 0/8/3 (gcs/repl/appl),

group UUID = 9908c53c-e085-11e7-ba6d-ffdb1c0f3abc

2018-02-25 19:08:34 140545137559296 [Note] WSREP: Flow-control interval: [724, 724]

2018-02-25 19:08:34 140545137559296 [Note] WSREP: Trying to continue unpaused monitor

2018-02-25 19:08:34 140486438315776 [Note] WSREP: New cluster view: global state: 9908c53c-e085-11e7-ba6d-ffdb1c0f3abc:121482637, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3

2018-02-25 19:08:35 140486438315776 [Note] WSREP: REPL Protocols: 8 (3, 2)

2018-02-25 19:08:35 140486438315776 [Note] WSREP: Assign initial position for certification: 121482637, protocol version: 3

2018-02-25 19:08:35 140550615521024 [Note] WSREP: Service thread queue flushed.

2018-02-25 19:08:35 140545137559296 [Note] WSREP: Member 0.1 (db2) requested state transfer from '*any*'. Selected 1.1 (db1)(SYNCED) as donor.

2018-02-25 19:08:35 140545137559296 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 121482661)

2018-02-25 19:08:35 140486180591360 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.0.0.xx:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'    --binlog 'mysql-bin' --gtid '9908c53c-e085-11e7-ba6d-ffdb1c0f3abc:121482661' --gtid-domain-id '508''

2018-02-25 19:08:35 140486438315776 [Note] WSREP: sst_donor_thread signaled with 0

WSREP_SST: [INFO] Streaming with xbstream (20180225 19:08:35.598)

WSREP_SST: [INFO] Using socat as streamer (20180225 19:08:35.600)

WSREP_SST: [INFO] Using /tmp/tmp.NGqpn5wBKz as xtrabackup temporary directory (20180225 19:08:35.613)

WSREP_SST: [INFO] Using /tmp/tmp.HtFCfrP0YN as innobackupex temporary directory (20180225 19:08:35.615)

WSREP_SST: [INFO] Streaming GTID file before SST (20180225 19:08:35.620)

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} |  gzip | socat -u stdio TCP:10.0.0.xx:4444 ; RC=( ${PIPESTATUS[@]} ) (20180225 19:08:35.625)

WSREP_SST: [INFO] NOTE: donor-gtid took 0 seconds (20180225 19:08:35.630)

WSREP_SST: [INFO] Sleeping before data transfer for SST (20180225 19:08:35.643)

2018-02-25 19:08:37 140545145952000 [Note] WSREP: (eb53eac9, 'tcp://0.0.0.0:4567') turning message relay requesting off

WSREP_SST: [INFO] Streaming the backup to joiner at 10.0.0.xx 4444 (20180225 19:08:45.646)

WSREP_SST: [INFO] Evaluating innobackupex    --no-version-check  $tmpopts $INNOEXTRA --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log | gzip | pv -f  -i 10 -N donor -F '%N => Rate:%r Avg:%a Elapsed:%t %e Bytes: %b %p'  -s 53136775168 | socat -u stdio TCP:10.0.0.xx:4444; RC=( ${PIPESTATUS[@]} ) (20180225 19:08:45.651)

2018/02/25 19:10:04 socat[26943] E write(6, 0x5580030a3da0, 8192): Broken pipe

    donor: => Rate:[9.88MiB/s] Avg:[9.88MiB/s] Elapsed:0:01:18

2018-02-25 19:10:04 140486300354304 [Warning] Aborted connection 947 to db: 'unconnected' user: 'backupuser' host: 'localhost' (Got an error reading communication packets)

WSREP_SST: [INFO] NOTE: donor-SST took 79 seconds (20180225 19:10:04.361)

WSREP_SST: [ERROR] innobackupex finished with error: 1.  Check /var/lib/mysql//innobackup.backup.log (20180225 19:10:04.363)

WSREP_SST: [ERROR] Cleanup after exit with status:22 (20180225 19:10:04.365)

WSREP_SST: [INFO] Cleaning up temporary directories (20180225 19:10:04.368)

2018-02-25 19:10:04 140486180591360 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.0.0.xx:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'    --binlog 'mysql-bin' --gtid '9908c53c-e085-11e7-ba6d-ffdb1c0f3abc:121482661' --gtid-domain-id '508'

2018-02-25 19:10:04 140486180591360 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.0.0.xx:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'    --binlog 'mysql-bin' --gtid '9908c53c-e085-11e7-ba6d-ffdb1c0f3abc:121482661' --gtid-domain-id '508': 22 (Invalid argument)

2018-02-25 19:10:04 140486180591360 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.0.0.xx:4444/xtrabackup_sst//1' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'    --binlog 'mysql-bin' --gtid '9908c53c-e085-11e7-ba6d-ffdb1c0f3abc:121482661' --gtid-domain-id '508'

2018-02-25 19:10:04 140545137559296 [Warning] WSREP: 1.1 (db1): State transfer to 0.1 (db2) failed: -22 (Invalid argument)

[syam...@genusis.com]

Someone please help me

[krunal....@percona.com]

Hello,

* I am still wondering which version of PXC are you using?

* I presume you have restored back to the original script of wsrep-sst-common and no more custom changes are present in the installation.

* Backup stage of XB failed. Logs for the same are present here /var/lib/mysql//innobackup.backup.log.

  Can you share this file to understand why XB failed?

Regards,

Krunal

On Tuesday, March 6, 2018 at 10:37:32 AM UTC+5:30, Sym wrote:

Someone please help me

[Sym]

Im not using PXC. Im using mariadb galera cluster

 /var/lib/mysql//innobackup.backup.log.

180309 11:41:20 [04]        ...done

180309 11:41:20 [04] Compressing and streaming ./genapi/sms_mo.ibd

180309 11:41:20 >> log scanned up to (150710542218)

innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)

xb_stream_write_data() failed.

compress: write to the destination stream failed.

innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)

[05] xtrabackup: Error: xtrabackup_copy_datafile() failed.

[05] xtrabackup: Error: failed to copy datafile.

[krunal....@percona.com]

Hi,

You can find multiple references to the said issue on the internet.

This is one such issue with solution explained.

https://www.datavail.com/blog/how-to-resolve-systemd-issue-with-percona-xtradb-cluster-on-centos-7/

I am not sure which version of MariaDB you are using but the latest version PXC has fixed lot of these issues in their flavor

so maybe manually porting some of these fixes (mainly in scripts) may help resolve the said issues.

Regards,

Krunal