Hi. Installed the latest version of PXC yesterday in our test environment. We've been using xtrabackup for SST and has been working well up to now. It seems to be broken in the latest version though. Here is what I have installed:
Two issues... 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of us probably have this configured in my.cnf instead, innobackupex will bomb out. I added datadir to wsrep.cnf also and it was able to continue 2) after the db was fully transferred to the recipient node (based on size of datadir), the SST process terminates prematurely with the following. innobackex completed OK though. First log is from donor, second from recipient:
DONOR: 121116 8:45:32 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'donor' --address '10.1.8.233:4444/xtrabackup_sst' --auth '***:***' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/wsrep.cnf' --gtid 'f472d7be-2e86-11e2-0800-cf63dfd4bfa3:421807'' 121116 8:45:32 [Note] WSREP: sst_donor_thread signaled with 0 mktemp: invalid option -- - Usage: mktemp [-V] | [-dqtu] [-p prefix] [template] 121116 9:21:09 [Note] WSREP: Provider paused at f472d7be-2e86-11e2-0800-cf63dfd4bfa3:421807 121116 9:25:11 [Note] WSREP: Provider resumed. /usr//bin/wsrep_sst_xtrabackup: line 37: $1: unbound variable 121116 9:25:13 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup --role 'donor' --address '10.1.8.233:4444/xtrabackup_sst' --auth '***:***' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/wsrep.cnf' --gtid 'f472d7be-2e86-11e2-0800-cf63dfd4bfa3:421807' 121116 9:25:13 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'donor' --address '10.1.8.233:4444/xtrabackup_sst' --auth '***:***' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/wsrep.cnf' --gtid 'f472d7be-2e86-11e2-0800-cf63dfd4bfa3:421807': 1 (Operation not permitted) 121116 9:25:13 [Warning] WSREP: 0 (): State transfer to 1 () failed: -1 (Operation not permitted) 121116 9:25:13 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 421807) 121116 9:25:13 [Note] WSREP: Member 0 () synced with group. 121116 9:25:13 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 421807) 121116 9:25:13 [Note] WSREP: Synchronized with group, ready for connections 121116 9:25:13 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 121116 9:25:14 [Note] WSREP: view(view_id(PRIM,03efd9aa-2ff2-11e2-0800-8f099c644f65,7) memb { 03efd9aa-2ff2-11e2-0800-8f099c644f65,
RECIPIENT: 121116 8:45:31 [Note] WSREP: Prepared SST request: xtrabackup|10.1.8.233:4444/xtrabackup_sst 121116 8:45:31 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 121116 8:45:31 [Note] WSREP: Assign initial position for certification: 421807, protocol version: 2 121116 8:45:31 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (f472d7be-2e86-11e2-0800-cf63dfd4bfa3): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable. 121116 8:45:31 [Note] WSREP: Node 1 () requested state transfer from '*any*'. Selected 0 ()(SYNCED) as donor. 121116 8:45:31 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 421807) 121116 8:45:31 [Note] WSREP: Requesting state transfer: success, donor: 0 tar: xtrabackup_checkpoints: time stamp 2012-11-16 09:25:11 is 1 s in the future tar: xtrabackup_binary: time stamp 2012-11-16 09:25:13 is 1 s in the future 121116 9:25:12 [Warning] WSREP: 0 (): State transfer to 1 () failed: -1 (Operation not permitted) 121116 9:25:12 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():712: Will never receive state. Need to abort. 121116 9:25:12 [Note] WSREP: gcomm: terminating thread 121116 9:25:12 [Note] WSREP: gcomm: joining thread 121116 9:25:12 [Note] WSREP: gcomm: closing backend 121116 9:25:13 [Note] WSREP: view(view_id(NON_PRIM,03efd9aa-2ff2-11e2-0800-8f099c644f65,6) memb { e1c9287a-2ff3-11e2-0800-a5873d6c135d,
I can confirm the 1st case, though it is more nefarious then described. WSRP wsrep_sst_xtrabackup will use the last included cnf file. It does not appear to use them as a unit.
On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
> Hi. Installed the latest version of PXC yesterday in our test > environment. We've been using xtrabackup for SST and has been working well > up to now. It seems to be broken in the latest version though. Here is > what I have installed:
> Two issues... > 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of > us probably have this configured in my.cnf instead, innobackupex will bomb > out. I added datadir to wsrep.cnf also and it was able to continue > 2) after the db was fully transferred to the recipient node (based on size > of datadir), the SST process terminates prematurely with the following. > innobackex completed OK though. First log is from donor, second from > recipient:
On Friday, November 16, 2012 2:39:38 PM UTC-6, Michael Eklund wrote:
> I can confirm the 1st case, though it is more nefarious then described. > WSRP wsrep_sst_xtrabackup will use the last included cnf file. It does > not appear to use them as a unit.
> Mike E.
> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>> Hi. Installed the latest version of PXC yesterday in our test >> environment. We've been using xtrabackup for SST and has been working well >> up to now. It seems to be broken in the latest version though. Here is >> what I have installed:
>> Two issues... >> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of >> us probably have this configured in my.cnf instead, innobackupex will bomb >> out. I added datadir to wsrep.cnf also and it was able to continue >> 2) after the db was fully transferred to the recipient node (based on >> size of datadir), the SST process terminates prematurely with the >> following. innobackex completed OK though. First log is from donor, >> second from recipient:
On Friday, November 16, 2012 2:48:46 PM UTC-6, Michael Eklund wrote:
> actually, my datadir was fine though, problem is using the wrong cnf file, > it wont the right innodb* settings if you use custom ones.
> On Friday, November 16, 2012 2:39:38 PM UTC-6, Michael Eklund wrote:
>> I can confirm the 1st case, though it is more nefarious then described. >> WSRP wsrep_sst_xtrabackup will use the last included cnf file. It does >> not appear to use them as a unit.
>> Mike E.
>> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>>> Hi. Installed the latest version of PXC yesterday in our test >>> environment. We've been using xtrabackup for SST and has been working well >>> up to now. It seems to be broken in the latest version though. Here is >>> what I have installed:
>>> Two issues... >>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of >>> us probably have this configured in my.cnf instead, innobackupex will bomb >>> out. I added datadir to wsrep.cnf also and it was able to continue >>> 2) after the db was fully transferred to the recipient node (based on >>> size of datadir), the SST process terminates prematurely with the >>> following. innobackex completed OK though. First log is from donor, >>> second from recipient:
> it is due to recursive calls in mysys/default.c::search_default_file_with_ext > which are not accounted for in wsrep mysql patches.
> Mike E.
> On Friday, November 16, 2012 2:48:46 PM UTC-6, Michael Eklund wrote:
>> actually, my datadir was fine though, problem is using the wrong cnf >> file, it wont the right innodb* settings if you use custom ones.
>> On Friday, November 16, 2012 2:39:38 PM UTC-6, Michael Eklund wrote:
>>> I can confirm the 1st case, though it is more nefarious then described. >>> WSRP wsrep_sst_xtrabackup will use the last included cnf file. It does >>> not appear to use them as a unit.
>>> Mike E.
>>> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>>>> Hi. Installed the latest version of PXC yesterday in our test >>>> environment. We've been using xtrabackup for SST and has been working well >>>> up to now. It seems to be broken in the latest version though. Here is >>>> what I have installed:
>>>> Two issues... >>>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most >>>> of us probably have this configured in my.cnf instead, innobackupex will >>>> bomb out. I added datadir to wsrep.cnf also and it was able to continue >>>> 2) after the db was fully transferred to the recipient node (based on >>>> size of datadir), the SST process terminates prematurely with the >>>> following. innobackex completed OK though. First log is from donor, >>>> second from recipient:
On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
> Hi. Installed the latest version of PXC yesterday in our test > environment. We've been using xtrabackup for SST and has been working well > up to now. It seems to be broken in the latest version though. Here is > what I have installed:
> Two issues... > 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of > us probably have this configured in my.cnf instead, innobackupex will bomb > out. I added datadir to wsrep.cnf also and it was able to continue > 2) after the db was fully transferred to the recipient node (based on size > of datadir), the SST process terminates prematurely with the following. > innobackex completed OK though. First log is from donor, second from > recipient:
On Friday, November 16, 2012 4:46:59 PM UTC-5, Michael Eklund wrote:
> I do not see the second problem in my environment. It may be due to > clocks not being in sync.
> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>> Hi. Installed the latest version of PXC yesterday in our test >> environment. We've been using xtrabackup for SST and has been working well >> up to now. It seems to be broken in the latest version though. Here is >> what I have installed:
>> Two issues... >> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of >> us probably have this configured in my.cnf instead, innobackupex will bomb >> out. I added datadir to wsrep.cnf also and it was able to continue >> 2) after the db was fully transferred to the recipient node (based on >> size of datadir), the SST process terminates prematurely with the >> following. innobackex completed OK though. First log is from donor, >> second from recipient:
On Friday, November 16, 2012 4:55:59 PM UTC-5, Freejack wrote:
> I suspected that as well so I synced and tried again with the same results.
> On Friday, November 16, 2012 4:46:59 PM UTC-5, Michael Eklund wrote:
>> I do not see the second problem in my environment. It may be due to >> clocks not being in sync.
>> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>>> Hi. Installed the latest version of PXC yesterday in our test >>> environment. We've been using xtrabackup for SST and has been working well >>> up to now. It seems to be broken in the latest version though. Here is >>> what I have installed:
>>> Two issues... >>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of >>> us probably have this configured in my.cnf instead, innobackupex will bomb >>> out. I added datadir to wsrep.cnf also and it was able to continue >>> 2) after the db was fully transferred to the recipient node (based on >>> size of datadir), the SST process terminates prematurely with the >>> following. innobackex completed OK though. First log is from donor, >>> second from recipient:
On Monday, November 26, 2012 12:22:35 AM UTC-5, amol wrote:
> any luck with this guys, i am facing the same errors in my donor and > receiver nodes with the same Percona versions
> On Friday, November 16, 2012 4:55:59 PM UTC-5, Freejack wrote:
>> I suspected that as well so I synced and tried again with the same >> results.
>> On Friday, November 16, 2012 4:46:59 PM UTC-5, Michael Eklund wrote:
>>> I do not see the second problem in my environment. It may be due to >>> clocks not being in sync.
>>> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>>>> Hi. Installed the latest version of PXC yesterday in our test >>>> environment. We've been using xtrabackup for SST and has been working well >>>> up to now. It seems to be broken in the latest version though. Here is >>>> what I have installed:
>>>> Two issues... >>>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most >>>> of us probably have this configured in my.cnf instead, innobackupex will >>>> bomb out. I added datadir to wsrep.cnf also and it was able to continue >>>> 2) after the db was fully transferred to the recipient node (based on >>>> size of datadir), the SST process terminates prematurely with the >>>> following. innobackex completed OK though. First log is from donor, >>>> second from recipient:
> I had to go back to using codership's patched mysql build for now. Couldn't figure out why the SST fails at the end.
> On Monday, November 26, 2012 12:22:35 AM UTC-5, amol wrote:
> any luck with this guys, i am facing the same errors in my donor and receiver nodes with the same Percona versions
> On Friday, November 16, 2012 4:55:59 PM UTC-5, Freejack wrote:
> I suspected that as well so I synced and tried again with the same results.
> On Friday, November 16, 2012 4:46:59 PM UTC-5, Michael Eklund wrote:
> I do not see the second problem in my environment. It may be due to clocks not being in sync.
> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
> Hi. Installed the latest version of PXC yesterday in our test environment. We've been using xtrabackup for SST and has been working well up to now. It seems to be broken in the latest version though. Here is what I have installed:
> Two issues...
> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of us probably have this configured in my.cnf instead, innobackupex will bomb out. I added datadir to wsrep.cnf also and it was able to continue
> 2) after the db was fully transferred to the recipient node (based on size of datadir), the SST process terminates prematurely with the following. innobackex completed OK though. First log is from donor, second from recipient:
> RECIPIENT:
> 121116 8:45:31 [Note] WSREP: Prepared SST request: xtrabackup|10.1.8.233:4444/xtrabackup_sst
> 121116 8:45:31 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
> 121116 8:45:31 [Note] WSREP: Assign initial position for certification: 421807, protocol version: 2
> 121116 8:45:31 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (f472d7be-2e86-11e2-0800-cf63dfd4bfa3): 1 (Operation not permitted)
> at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable.
> 121116 8:45:31 [Note] WSREP: Node 1 () requested state transfer from '*any*'. Selected 0 ()(SYNCED) as donor.
> 121116 8:45:31 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 421807)
> 121116 8:45:31 [Note] WSREP: Requesting state transfer: success, donor: 0
> tar: xtrabackup_checkpoints: time stamp 2012-11-16 09:25:11 is 1 s in the future
> tar: xtrabackup_binary: time stamp 2012-11-16 09:25:13 is 1 s in the future
> 121116 9:25:12 [Warning] WSREP: 0 (): State transfer to 1 () failed: -1 (Operation not permitted)
> 121116 9:25:12 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():712: Will never receive state. Need to abort.
> 121116 9:25:12 [Note] WSREP: gcomm: terminating thread
> 121116 9:25:12 [Note] WSREP: gcomm: joining thread
> 121116 9:25:12 [Note] WSREP: gcomm: closing backend
> 121116 9:25:13 [Note] WSREP: view(view_id(NON_PRIM,03efd9aa-2ff2-11e2-0800-8f099c644f65,6) memb {
> e1c9287a-2ff3-11e2-0800-a5873d6c135d,
> } joined {
> } left {
> } partitioned {
> 03efd9aa-2ff2-11e2-0800-8f099c644f65,
> })
> 121116 9:25:13 [Note] WSREP: view((empty))
> 121116 9:25:13 [Note] WSREP: gcomm: closed
> 121116 9:25:13 [Note] WSREP: /usr/sbin/mysqld: Terminated.
> WSREP_SST: [ERROR] Parent mysqld process (PID:6995) terminated unexpectedly. (20121116 09:25:13.866)
> 121116 09:25:13 mysqld_safe mysqld from pid file /var/lib/mysql/r8-qadb-node1.pid ended
> Thanks!
> -- > You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
> To post to this group, send email to percona-discussion@googlegroups.com.
I used the PXC on all nodes without success. Reverted all nodes back to codership mysql. Just install rpm's. The only thing you need to change in the config is to point to where the galera lib is. I believe codership and Percona install the library to different places.
On Monday, November 26, 2012 1:09:40 PM UTC-5, amol wrote:
> did you use that build on all 3 nodes? or just the node that was not > startiing? > do you have instructions on how to get the build ? and install..
> Thanks once again..
> On Nov 26, 2012, at 12:52 PM, Freejack <free...@gmail.com <javascript:>> > wrote:
> I had to go back to using codership's patched mysql build for now. > Couldn't figure out why the SST fails at the end.
> On Monday, November 26, 2012 12:22:35 AM UTC-5, amol wrote:
>> any luck with this guys, i am facing the same errors in my donor and >> receiver nodes with the same Percona versions
>> On Friday, November 16, 2012 4:55:59 PM UTC-5, Freejack wrote:
>>> I suspected that as well so I synced and tried again with the same >>> results.
>>> On Friday, November 16, 2012 4:46:59 PM UTC-5, Michael Eklund wrote:
>>>> I do not see the second problem in my environment. It may be due to >>>> clocks not being in sync.
>>>> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>>>>> Hi. Installed the latest version of PXC yesterday in our test >>>>> environment. We've been using xtrabackup for SST and has been working well >>>>> up to now. It seems to be broken in the latest version though. Here is >>>>> what I have installed:
>>>>> Two issues... >>>>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most >>>>> of us probably have this configured in my.cnf instead, innobackupex will >>>>> bomb out. I added datadir to wsrep.cnf also and it was able to continue >>>>> 2) after the db was fully transferred to the recipient node (based on >>>>> size of datadir), the SST process terminates prematurely with the >>>>> following. innobackex completed OK though. First log is from donor, >>>>> second from recipient:
>>>>> RECIPIENT: >>>>> 121116 8:45:31 [Note] WSREP: Prepared SST request: xtrabackup| >>>>> 10.1.8.233:4444/xtrabackup_sst >>>>> 121116 8:45:31 [Note] WSREP: wsrep_notify_cmd is not defined, >>>>> skipping notification. >>>>> 121116 8:45:31 [Note] WSREP: Assign initial position for >>>>> certification: 421807, protocol version: 2 >>>>> 121116 8:45:31 [Warning] WSREP: Failed to prepare for incremental >>>>> state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) >>>>> does not match group state UUID (f472d7be-2e86-11e2-0800-cf63dfd4bfa3): 1 >>>>> (Operation not permitted) >>>>> at galera/src/replicator_str.cpp:prepare_for_IST():440. IST >>>>> will be unavailable. >>>>> 121116 8:45:31 [Note] WSREP: Node 1 () requested state transfer from >>>>> '*any*'. Selected 0 ()(SYNCED) as donor. >>>>> 121116 8:45:31 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 421807) >>>>> 121116 8:45:31 [Note] WSREP: Requesting state transfer: success, >>>>> donor: 0 >>>>> tar: xtrabackup_checkpoints: time stamp 2012-11-16 09:25:11 is 1 s in >>>>> the future >>>>> tar: xtrabackup_binary: time stamp 2012-11-16 09:25:13 is 1 s in the >>>>> future >>>>> 121116 9:25:12 [Warning] WSREP: 0 (): State transfer to 1 () failed: >>>>> -1 (Operation not permitted) >>>>> 121116 9:25:12 [ERROR] WSREP: >>>>> gcs/src/gcs_group.c:gcs_group_handle_join_msg():712: Will never receive >>>>> state. Need to abort. >>>>> 121116 9:25:12 [Note] WSREP: gcomm: terminating thread >>>>> 121116 9:25:12 [Note] WSREP: gcomm: joining thread >>>>> 121116 9:25:12 [Note] WSREP: gcomm: closing backend >>>>> 121116 9:25:13 [Note] WSREP: >>>>> view(view_id(NON_PRIM,03efd9aa-2ff2-11e2-0800-8f099c644f65,6) memb { >>>>> e1c9287a-2ff3-11e2-0800-a5873d6c135d, >>>>> } joined { >>>>> } left { >>>>> } partitioned { >>>>> 03efd9aa-2ff2-11e2-0800-8f099c644f65, >>>>> }) >>>>> 121116 9:25:13 [Note] WSREP: view((empty)) >>>>> 121116 9:25:13 [Note] WSREP: gcomm: closed >>>>> 121116 9:25:13 [Note] WSREP: /usr/sbin/mysqld: Terminated. >>>>> WSREP_SST: [ERROR] Parent mysqld process (PID:6995) terminated >>>>> unexpectedly. (20121116 09:25:13.866) >>>>> 121116 09:25:13 mysqld_safe mysqld from pid file >>>>> /var/lib/mysql/r8-qadb-node1.pid ended
>>>>> Thanks!
> -- > You received this message because you are subscribed to the Google Groups > "Percona Discussion" group. > To post to this group, send email to percona-d...@googlegroups.com<javascript:> > .
oh..so you are saying that you gave up on PXC and moved to another cluster software?
ok i was thinking you patched the same installation of PXC but packaged by codership..etc..
ok i think i'll try to revert back to 5.5.27 and see if that help...
thanks for you help and advice..
On Nov 26, 2012, at 1:15 PM, Freejack <freej...@gmail.com> wrote:
> I used the PXC on all nodes without success. Reverted all nodes back to codership mysql. Just install rpm's. The only thing you need to change in the config is to point to where the galera lib is. I believe codership and Percona install the library to different places.
> On Monday, November 26, 2012 1:09:40 PM UTC-5, amol wrote:
> did you use that build on all 3 nodes? or just the node that was not startiing?
> do you have instructions on how to get the build ? and install..
> Thanks once again..
> On Nov 26, 2012, at 12:52 PM, Freejack <free...@gmail.com> wrote:
>> I had to go back to using codership's patched mysql build for now. Couldn't figure out why the SST fails at the end.
>> On Monday, November 26, 2012 12:22:35 AM UTC-5, amol wrote:
>> any luck with this guys, i am facing the same errors in my donor and receiver nodes with the same Percona versions
>> On Friday, November 16, 2012 4:55:59 PM UTC-5, Freejack wrote:
>> I suspected that as well so I synced and tried again with the same results.
>> On Friday, November 16, 2012 4:46:59 PM UTC-5, Michael Eklund wrote:
>> I do not see the second problem in my environment. It may be due to clocks not being in sync.
>> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>> Hi. Installed the latest version of PXC yesterday in our test environment. We've been using xtrabackup for SST and has been working well up to now. It seems to be broken in the latest version though. Here is what I have installed:
>> Two issues...
>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of us probably have this configured in my.cnf instead, innobackupex will bomb out. I added datadir to wsrep.cnf also and it was able to continue
>> 2) after the db was fully transferred to the recipient node (based on size of datadir), the SST process terminates prematurely with the following. innobackex completed OK though. First log is from donor, second from recipient:
>> RECIPIENT:
>> 121116 8:45:31 [Note] WSREP: Prepared SST request: xtrabackup|10.1.8.233:4444/xtrabackup_sst
>> 121116 8:45:31 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
>> 121116 8:45:31 [Note] WSREP: Assign initial position for certification: 421807, protocol version: 2
>> 121116 8:45:31 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (f472d7be-2e86-11e2-0800-cf63dfd4bfa3): 1 (Operation not permitted)
>> at galera/src/replicator_str.cpp:prepare_for_IST():440. IST will be unavailable.
>> 121116 8:45:31 [Note] WSREP: Node 1 () requested state transfer from '*any*'. Selected 0 ()(SYNCED) as donor.
>> 121116 8:45:31 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 421807)
>> 121116 8:45:31 [Note] WSREP: Requesting state transfer: success, donor: 0
>> tar: xtrabackup_checkpoints: time stamp 2012-11-16 09:25:11 is 1 s in the future
>> tar: xtrabackup_binary: time stamp 2012-11-16 09:25:13 is 1 s in the future
>> 121116 9:25:12 [Warning] WSREP: 0 (): State transfer to 1 () failed: -1 (Operation not permitted)
>> 121116 9:25:12 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():712: Will never receive state. Need to abort.
>> 121116 9:25:12 [Note] WSREP: gcomm: terminating thread
>> 121116 9:25:12 [Note] WSREP: gcomm: joining thread
>> 121116 9:25:12 [Note] WSREP: gcomm: closing backend
>> 121116 9:25:13 [Note] WSREP: view(view_id(NON_PRIM,03efd9aa-2ff2-11e2-0800-8f099c644f65,6) memb {
>> e1c9287a-2ff3-11e2-0800-a5873d6c135d,
>> } joined {
>> } left {
>> } partitioned {
>> 03efd9aa-2ff2-11e2-0800-8f099c644f65,
>> })
>> 121116 9:25:13 [Note] WSREP: view((empty))
>> 121116 9:25:13 [Note] WSREP: gcomm: closed
>> 121116 9:25:13 [Note] WSREP: /usr/sbin/mysqld: Terminated.
>> WSREP_SST: [ERROR] Parent mysqld process (PID:6995) terminated unexpectedly. (20121116 09:25:13.866)
>> 121116 09:25:13 mysqld_safe mysqld from pid file /var/lib/mysql/r8-qadb-node1.pid ended
>> Thanks!
>> -- >> You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>> To post to this group, send email to percona-d...@googlegroups.com.
> -- > You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
> To post to this group, send email to percona-discussion@googlegroups.com.
Not really another cluster software. PXC(luster) is built on codership's galera multi-master code. It's more of a generic version of mysql but with the same clustering capability.
On Monday, November 26, 2012 1:18:51 PM UTC-5, amol wrote:
> oh..so you are saying that you gave up on PXC and moved to another cluster > software? > ok i was thinking you patched the same installation of PXC but packaged by > codership..etc..
> ok i think i'll try to revert back to 5.5.27 and see if that help...
> thanks for you help and advice..
> On Nov 26, 2012, at 1:15 PM, Freejack <free...@gmail.com <javascript:>> > wrote:
> I used the PXC on all nodes without success. Reverted all nodes back to > codership mysql. Just install rpm's. The only thing you need to change in > the config is to point to where the galera lib is. I believe codership and > Percona install the library to different places.
> On Monday, November 26, 2012 1:09:40 PM UTC-5, amol wrote:
>> did you use that build on all 3 nodes? or just the node that was not >> startiing? >> do you have instructions on how to get the build ? and install..
>> Thanks once again..
>> On Nov 26, 2012, at 12:52 PM, Freejack <free...@gmail.com> wrote:
>> I had to go back to using codership's patched mysql build for now. >> Couldn't figure out why the SST fails at the end.
>> On Monday, November 26, 2012 12:22:35 AM UTC-5, amol wrote:
>>> any luck with this guys, i am facing the same errors in my donor and >>> receiver nodes with the same Percona versions
>>> On Friday, November 16, 2012 4:55:59 PM UTC-5, Freejack wrote:
>>>> I suspected that as well so I synced and tried again with the same >>>> results.
>>>> On Friday, November 16, 2012 4:46:59 PM UTC-5, Michael Eklund wrote:
>>>>> I do not see the second problem in my environment. It may be due to >>>>> clocks not being in sync.
>>>>> On Friday, November 16, 2012 9:43:19 AM UTC-6, Freejack wrote:
>>>>>> Hi. Installed the latest version of PXC yesterday in our test >>>>>> environment. We've been using xtrabackup for SST and has been working well >>>>>> up to now. It seems to be broken in the latest version though. Here is >>>>>> what I have installed:
>>>>>> Two issues... >>>>>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most >>>>>> of us probably have this configured in my.cnf instead, innobackupex will >>>>>> bomb out. I added datadir to wsrep.cnf also and it was able to continue >>>>>> 2) after the db was fully transferred to the recipient node (based on >>>>>> size of datadir), the SST process terminates prematurely with the >>>>>> following. innobackex completed OK though. First log is from donor, >>>>>> second from recipient:
>>>>>> RECIPIENT: >>>>>> 121116 8:45:31 [Note] WSREP: Prepared SST request: xtrabackup| >>>>>> 10.1.8.233:4444/xtrabackup_sst >>>>>> 121116 8:45:31 [Note] WSREP: wsrep_notify_cmd is not defined, >>>>>> skipping notification. >>>>>> 121116 8:45:31 [Note] WSREP: Assign initial position for >>>>>> certification: 421807, protocol version: 2 >>>>>> 121116 8:45:31 [Warning] WSREP: Failed to prepare for incremental >>>>>> state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) >>>>>> does not match group state UUID (f472d7be-2e86-11e2-0800-cf63dfd4bfa3): 1 >>>>>> (Operation not permitted) >>>>>> at galera/src/replicator_str.cpp:prepare_for_IST():440. IST >>>>>> will be unavailable. >>>>>> 121116 8:45:31 [Note] WSREP: Node 1 () requested state transfer from >>>>>> '*any*'. Selected 0 ()(SYNCED) as donor. >>>>>> 121116 8:45:31 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 421807) >>>>>> 121116 8:45:31 [Note] WSREP: Requesting state transfer: success, >>>>>> donor: 0 >>>>>> tar: xtrabackup_checkpoints: time stamp 2012-11-16 09:25:11 is 1 s in >>>>>> the future >>>>>> tar: xtrabackup_binary: time stamp 2012-11-16 09:25:13 is 1 s in the >>>>>> future >>>>>> 121116 9:25:12 [Warning] WSREP: 0 (): State transfer to 1 () failed: >>>>>> -1 (Operation not permitted) >>>>>> 121116 9:25:12 [ERROR] WSREP: >>>>>> gcs/src/gcs_group.c:gcs_group_handle_join_msg():712: Will never receive >>>>>> state. Need to abort. >>>>>> 121116 9:25:12 [Note] WSREP: gcomm: terminating thread >>>>>> 121116 9:25:12 [Note] WSREP: gcomm: joining thread >>>>>> 121116 9:25:12 [Note] WSREP: gcomm: closing backend >>>>>> 121116 9:25:13 [Note] WSREP: >>>>>> view(view_id(NON_PRIM,03efd9aa-2ff2-11e2-0800-8f099c644f65,6) memb { >>>>>> e1c9287a-2ff3-11e2-0800-a5873d6c135d, >>>>>> } joined { >>>>>> } left { >>>>>> } partitioned { >>>>>> 03efd9aa-2ff2-11e2-0800-8f099c644f65, >>>>>> }) >>>>>> 121116 9:25:13 [Note] WSREP: view((empty)) >>>>>> 121116 9:25:13 [Note] WSREP: gcomm: closed >>>>>> 121116 9:25:13 [Note] WSREP: /usr/sbin/mysqld: Terminated. >>>>>> WSREP_SST: [ERROR] Parent mysqld process (PID:6995) terminated >>>>>> unexpectedly. (20121116 09:25:13.866) >>>>>> 121116 09:25:13 mysqld_safe mysqld from pid file >>>>>> /var/lib/mysql/r8-qadb-node1.pid ended
>>>>>> Thanks!
>> -- >> You received this message because you are subscribed to the Google Groups >> "Percona Discussion" group. >> To post to this group, send email to percona-d...@googlegroups.com.
> -- > You received this message because you are subscribed to the Google Groups > "Percona Discussion" group. > To post to this group, send email to percona-d...@googlegroups.com<javascript:> > .
On Friday, 16 November 2012 23:43:19 UTC+8, Freejack wrote:
> Hi. Installed the latest version of PXC yesterday in our test > environment. We've been using xtrabackup for SST and has been working well > up to now. It seems to be broken in the latest version though. Here is > what I have installed:
> Two issues... > 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of > us probably have this configured in my.cnf instead, innobackupex will bomb > out. I added datadir to wsrep.cnf also and it was able to continue > 2) after the db was fully transferred to the recipient node (based on size > of datadir), the SST process terminates prematurely with the following. > innobackex completed OK though. First log is from donor, second from > recipient:
Just wanted to warn whoever is concerned that this is not a real fix. The script will pass, but it will create a file that won't be used. There is no need to call mktemp in the first place - pidfile is stored in a predefined, unchangeable location. A properly fixed version of the script is available as a separate download here: https://launchpad.net/codership-mysql/5.5/5.5.28-23.7/+download/wsrep... .
Regards,
Alex
On 2012-12-25 03:22, Syahrul Sazli Shaharir wrote:
> and the sync succeeded right afterwards. Hope it helps - thanks.
> On Friday, 16 November 2012 23:43:19 UTC+8, Freejack wrote:
>> Hi. Installed the latest version of PXC yesterday in our test
>> environment. We've been using xtrabackup for SST and has been >> working well
>> up to now. It seems to be broken in the latest version though.
>> Here is
>> what I have installed:
>> Two issues...
>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As >> most of
>> us probably have this configured in my.cnf instead, innobackupex >> will bomb
>> out. I added datadir to wsrep.cnf also and it was able to continue
>> 2) after the db was fully transferred to the recipient node (based >> on size
>> of datadir), the SST process terminates prematurely with the >> following.
>> innobackex completed OK though. First log is from donor, second >> from
>> recipient:
On Tuesday, December 25, 2012 8:55:51 PM UTC-5, Alexey Yurchenko wrote:
> Hi,
> Just wanted to warn whoever is concerned that this is not a real fix. > The script will pass, but it will create a file that won't be used. > There is no need to call mktemp in the first place - pidfile is stored > in a predefined, unchangeable location. A properly fixed version of the > script is available as a separate download here:
> > and the sync succeeded right afterwards. Hope it helps - thanks.
> > On Friday, 16 November 2012 23:43:19 UTC+8, Freejack wrote:
> >> Hi. Installed the latest version of PXC yesterday in our test > >> environment. We've been using xtrabackup for SST and has been > >> working well > >> up to now. It seems to be broken in the latest version though. > >> Here is > >> what I have installed:
> >> Two issues... > >> 1) innobackupex looks in the wsrep.cnf file for the datadir. As > >> most of > >> us probably have this configured in my.cnf instead, innobackupex > >> will bomb > >> out. I added datadir to wsrep.cnf also and it was able to continue > >> 2) after the db was fully transferred to the recipient node (based > >> on size > >> of datadir), the SST process terminates prematurely with the > >> following. > >> innobackex completed OK though. First log is from donor, second > >> from > >> recipient:
On Saturday, November 17, 2012 2:43:19 AM UTC+11, Freejack wrote:
> Hi. Installed the latest version of PXC yesterday in our test > environment. We've been using xtrabackup for SST and has been working well > up to now. It seems to be broken in the latest version though. Here is > what I have installed:
> Two issues... > 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of > us probably have this configured in my.cnf instead, innobackupex will bomb > out. I added datadir to wsrep.cnf also and it was able to continue > 2) after the db was fully transferred to the recipient node (based on size > of datadir), the SST process terminates prematurely with the following. > innobackex completed OK though. First log is from donor, second from > recipient:
> 130228 17:42:08 [Warning] WSREP: 0 (ip-10-132-23-68): State transfer to 1 > (ip-10-132-23-69) failed: -1 (Operation not permitted) > 130228 17:42:08 [ERROR] WSREP: > gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive > state. Need to abort. > 130228 17:42:08 [Note] WSREP: gcomm: terminating thread > 130228 17:42:08 [Note] WSREP: gcomm: joining thread > 130228 17:42:08 [Note] WSREP: gcomm: closing backend > WSREP_SST: [ERROR] Error while getting st data from donor node: 1, 2 > (20130228 17:42:08.675) > 130228 17:42:08 [ERROR] WSREP: Process completed with error: > wsrep_sst_xtrabackup --role 'joiner' --address '10.132.23.69' --auth > 'root:C02G327PDHJQ' --datadir '/var/lib/mysql/' --defaults-file > '/etc/mysql/my.cnf' --parent '12178': 32 (Broken pipe) > 130228 17:42:08 [ERROR] WSREP: Failed to read uuid:seqno from joiner > script. > 130228 17:42:08 [ERROR] WSREP: SST failed: 32 (Broken pipe) > 130228 17:42:08 [ERROR] Aborting
> i just thinks , that is because the script "wsrep_sst_xtrabackup" can not > be find by percona
> could u help me to fix it ?
> thanks.
> On Saturday, November 17, 2012 2:43:19 AM UTC+11, Freejack wrote:
>> Hi. Installed the latest version of PXC yesterday in our test >> environment. We've been using xtrabackup for SST and has been working well >> up to now. It seems to be broken in the latest version though. Here is >> what I have installed:
>> Two issues... >> 1) innobackupex looks in the wsrep.cnf file for the datadir. As most of >> us probably have this configured in my.cnf instead, innobackupex will bomb >> out. I added datadir to wsrep.cnf also and it was able to continue >> 2) after the db was fully transferred to the recipient node (based on >> size of datadir), the SST process terminates prematurely with the >> following. innobackex completed OK though. First log is from donor, >> second from recipient:
This is most likely permissions problem, not related to OP. Make sure that
1) selinux/apparmor is disabled (not only in config, but in real life)
2) xtrabackup is installed and is in mysqld's PATH
3) you can run the failed SST command manually: cut and paste it from the error log.
On 2013-05-01 16:11, adi.hoch...@fiverr.com wrote:
>> 130228 17:42:08 [Warning] WSREP: 0 (ip-10-132-23-68): State transfer >> to 1
>> (ip-10-132-23-69) failed: -1 (Operation not permitted)
>> 130228 17:42:08 [ERROR] WSREP:
>> gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never >> receive
>> state. Need to abort.
>> 130228 17:42:08 [Note] WSREP: gcomm: terminating thread
>> 130228 17:42:08 [Note] WSREP: gcomm: joining thread
>> 130228 17:42:08 [Note] WSREP: gcomm: closing backend
>> WSREP_SST: [ERROR] Error while getting st data from donor node: 1, 2
>> (20130228 17:42:08.675)
>> 130228 17:42:08 [ERROR] WSREP: Process completed with error:
>> wsrep_sst_xtrabackup --role 'joiner' --address '10.132.23.69' --auth
>> 'root:C02G327PDHJQ' --datadir '/var/lib/mysql/' --defaults-file
>> '/etc/mysql/my.cnf' --parent '12178': 32 (Broken pipe)
>> 130228 17:42:08 [ERROR] WSREP: Failed to read uuid:seqno from joiner
>> script.
>> 130228 17:42:08 [ERROR] WSREP: SST failed: 32 (Broken pipe)
>> 130228 17:42:08 [ERROR] Aborting
>> i just thinks , that is because the script "wsrep_sst_xtrabackup" can >> not
>> be find by percona
>> could u help me to fix it ?
>> thanks.
>> On Saturday, November 17, 2012 2:43:19 AM UTC+11, Freejack wrote:
>>> Hi. Installed the latest version of PXC yesterday in our test
>>> environment. We've been using xtrabackup for SST and has been >>> working well
>>> up to now. It seems to be broken in the latest version though.
>>> Here is
>>> what I have installed:
>>> Two issues...
>>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As >>> most of
>>> us probably have this configured in my.cnf instead, innobackupex >>> will bomb
>>> out. I added datadir to wsrep.cnf also and it was able to continue
>>> 2) after the db was fully transferred to the recipient node (based >>> on
>>> size of datadir), the SST process terminates prematurely with the
>>> following. innobackex completed OK though. First log is from >>> donor,
>>> second from recipient:
I just meet the similar error:
[Process completed with error: wsrep_sst_xtrabackup --role 'donor' --address '10.109.37.4:4444/xtrabackup_sst' --auth 'root:xxx' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --gtid 'edefb40a-bc62-11e2-0800-9c175fdcc5a1:14': 2 (No such file or directory)]
could somebody please help me~~~
below is my configuration and logs. Thank you !
*<My Environment>*
*OS*: Ubuntu 12.10 x86_64
*Percona version*: 5.5.30 [DEB]
*my.cnf permission*:root:root 644
Disable *appamor(# */etc/init.d/appamor stop*)*
*<MySQL my.cnf>*
*node1(10.109.37.3):*
130514 18:35:38 [Note] WSREP: sst_donor_thread signaled with 0
130514 18:35:38 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'donor' --address '10.109.37.4:4444/xtrabackup_sst' --auth 'root:xxx' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --gtid 'edefb40a-bc62-11e2-0800-9c175fdcc5a1:14': 2 (No such file or directory)
130514 18:35:38 [Warning] WSREP: 1 (openstack-slave2): State transfer to 0 (openstack-slave2) failed: -1 (Operation not permitted)
*node2*(10.109.37.4)
/var/log/mysql/error.log
130514 18:35:39 [Note] WSREP: New cluster view: global state: edefb40a-bc62-11e2-0800-9c175fdcc5a1:14, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 2
130514 18:35:39 [Warning] WSREP: Gap in state sequence. Need state transfer.
130514 18:35:41 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130514 18:35:41 [Note] WSREP: Assign initial position for certification: 14, protocol version: 2
130514 18:35:41 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (edefb40a-bc62-11e2-0800-9c175fdcc5a1): 1 (Operation not permitted)
at galera/src/replicator_str.cpprepare_for_IST():436. IST will be unavailable.
130514 18:35:41 [Note] WSREP: Node 0 (openstack-slave2) requested state transfer from '*any*'. Selected 1 (openstack-slave2)(SYNCED) as donor.
130514 18:35:41 [Note] WSREP: Requesting state transfer: success, donor: 1
130514 18:35:41 [Warning] WSREP: 1 (openstack-slave2): State transfer to 0 (openstack-slave2) failed: -1 (Operation not permitted)
130514 18:35:41 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.
Alexey Yurchenko於 2013年5月2日星期四UTC+8上午12時42分49秒寫道：
> This is most likely permissions problem, not related to OP. Make sure > that > 1) selinux/apparmor is disabled (not only in config, but in real life) > 2) xtrabackup is installed and is in mysqld's PATH > 3) you can run the failed SST command manually: cut and paste it from > the error log.
> On 2013-05-01 16:11, adi.h...@fiverr.com <javascript:> wrote: > > I experience same issue. > > Any Idea what should i do?
> > On Thursday, February 28, 2013 8:43:51 AM UTC+2, Peter Pang wrote:
> >> hi mate, i meet this problem with latest version of percona cluster
> >> 130228 17:42:08 [Warning] WSREP: 0 (ip-10-132-23-68): State transfer > >> to 1 > >> (ip-10-132-23-69) failed: -1 (Operation not permitted) > >> 130228 17:42:08 [ERROR] WSREP: > >> gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never > >> receive > >> state. Need to abort. > >> 130228 17:42:08 [Note] WSREP: gcomm: terminating thread > >> 130228 17:42:08 [Note] WSREP: gcomm: joining thread > >> 130228 17:42:08 [Note] WSREP: gcomm: closing backend > >> WSREP_SST: [ERROR] Error while getting st data from donor node: 1, 2 > >> (20130228 17:42:08.675) > >> 130228 17:42:08 [ERROR] WSREP: Process completed with error: > >> wsrep_sst_xtrabackup --role 'joiner' --address '10.132.23.69' --auth > >> 'root:C02G327PDHJQ' --datadir '/var/lib/mysql/' --defaults-file > >> '/etc/mysql/my.cnf' --parent '12178': 32 (Broken pipe) > >> 130228 17:42:08 [ERROR] WSREP: Failed to read uuid:seqno from joiner > >> script. > >> 130228 17:42:08 [ERROR] WSREP: SST failed: 32 (Broken pipe) > >> 130228 17:42:08 [ERROR] Aborting
> >> i just thinks , that is because the script "wsrep_sst_xtrabackup" can > >> not > >> be find by percona
> >> could u help me to fix it ?
> >> thanks.
> >> On Saturday, November 17, 2012 2:43:19 AM UTC+11, Freejack wrote:
> >>> Hi. Installed the latest version of PXC yesterday in our test > >>> environment. We've been using xtrabackup for SST and has been > >>> working well > >>> up to now. It seems to be broken in the latest version though. > >>> Here is > >>> what I have installed:
> >>> Two issues... > >>> 1) innobackupex looks in the wsrep.cnf file for the datadir. As > >>> most of > >>> us probably have this configured in my.cnf instead, innobackupex > >>> will bomb > >>> out. I added datadir to wsrep.cnf also and it was able to continue > >>> 2) after the db was fully transferred to the recipient node (based > >>> on > >>> size of datadir), the SST process terminates prematurely with the > >>> following. innobackex completed OK though. First log is from > >>> donor, > >>> second from recipient:
