Regression with Xtrabackup affecting PXC SST

47 views
Skip to first unread message

Raghavendra D Prabhu

unread,
May 16, 2013, 3:57:33 AM5/16/13
to percona-discussion
Hi,

PXC/Galera Users:
There is a regression with XB -
https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which
directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.





Regards,
--
Raghavendra Prabhu
Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
Skype: percona.raghavendrap
GPG: 0xD72BE977
Contact: http://wnohang.net/contact

Make plans to attend Percona Live London MySQL Conference
2013.<http://www.percona.com/live/london-2013/>

Raghavendra D Prabhu

unread,
May 16, 2013, 4:01:31 AM5/16/13
to percona-discussion
Hi,


* On Thu, May 16, 2013 at 01:27:33PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>Hi,
>
>PXC/Galera Users: There is a regression with XB -
>https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which

Correction. The right bug is https://bugs.launchpad.net/percona-xtrabackup/+bug/1180672

>directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
>2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.
>
>
>
>
>
>Regards,
>--
>Raghavendra Prabhu
>Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
>Skype: percona.raghavendrap
>GPG: 0xD72BE977
>Contact: http://wnohang.net/contact
>
>Make plans to attend Percona Live London MySQL Conference
>2013.<http://www.percona.com/live/london-2013/>
>
>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.
>To post to this group, send email to percona-d...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.

Amol

unread,
May 16, 2013, 11:57:57 PM5/16/13
to percona-d...@googlegroups.com
Thanks for informing us, will hold off the upgrade


On Thursday, May 16, 2013 4:01:31 AM UTC-4, Raghavendra Prabhu wrote:
Hi,


* On Thu, May 16, 2013 at 01:27:33PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>Hi,
>
>PXC/Galera Users: There is a regression with XB -
>https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which

Correction. The right bug is https://bugs.launchpad.net/percona-xtrabackup/+bug/1180672

>directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
>2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.
>
>
>
>
>
>Regards,
>--
>Raghavendra Prabhu
>Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
>Skype: percona.raghavendrap
>GPG: 0xD72BE977
>Contact: http://wnohang.net/contact
>
>Make plans to attend Percona Live London MySQL Conference
>2013.<http://www.percona.com/live/london-2013/>
>
>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.

Raghavendra D Prabhu

unread,
May 17, 2013, 6:33:35 AM5/17/13
to percona-discussion, codersh...@googlegroups.com
Hi,

* On Thu, May 16, 2013 at 01:31:31PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>Hi,
>
>
>* On Thu, May 16, 2013 at 01:27:33PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>>Hi,
>>
>>PXC/Galera Users: There is a regression with XB -
>>https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which
>
>Correction. The right bug is https://bugs.launchpad.net/percona-xtrabackup/+bug/1180672
>
>>directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
>>2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.
>>
>>
>>
>>
>>
>>Regards,
>>--
>>Raghavendra Prabhu
>>Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
>>Skype: percona.raghavendrap
>>GPG: 0xD72BE977
>>Contact: http://wnohang.net/contact
>>
>>Make plans to attend Percona Live London MySQL Conference
>>2013.<http://www.percona.com/live/london-2013/>
>>
>>--
>>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.
>>To post to this group, send email to percona-d...@googlegroups.com.
>>For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
>

Fix for this bug has been committed by Xtrabackup developers
(thanks!). It should be pushed during Friday US time. You can
track its milestone here -- https://launchpad.net/percona-xtrabackup/+milestone/2.1.2 (It also has a few other critical bugs fixed).

Ignacio Nin

unread,
May 18, 2013, 12:28:15 AM5/18/13
to percona-d...@googlegroups.com
Hello,

We've published Percona XtraBackup 2.1.2, which fixes (among others) bug
#1180672.

Please find the details at
http://www.mysqlperformanceblog.com/2013/05/18/percona-xtrabackup-2-1-2-for-mysql-available-for-download/.

Regards,


El 17/05/13 07:33, Raghavendra D Prabhu escribi�:
Ignacio Nin
Build Engineer, Percona Inc.

http://www.percona.com/
(+1 877) 862-4316 ext 524
skype: ignacio-nin
ignac...@percona.com
http://www.mysqlperformanceblog.com/

Got MySQL support?
http://tinyurl.com/mysql-support

Jervin R

unread,
May 22, 2013, 3:30:21 AM5/22/13
to percona-d...@googlegroups.com
So there is another one - do not upgrade to 2.1.2 yet - https://bugs.launchpad.net/percona-xtrabackup/+bug/1182698


On Saturday, May 18, 2013 12:28:15 PM UTC+8, Ignacio Nin wrote:
Hello,

We've published Percona XtraBackup 2.1.2, which fixes (among others) bug
#1180672.

Please find the details at
http://www.mysqlperformanceblog.com/2013/05/18/percona-xtrabackup-2-1-2-for-mysql-available-for-download/.

Regards,


El 17/05/13 07:33, Raghavendra D Prabhu escribi�:
>>> send an email to percona-discussion+unsub...@googlegroups.com.

Raghavendra D Prabhu

unread,
May 22, 2013, 5:04:07 AM5/22/13
to percona-d...@googlegroups.com
Hi,

* On Wed, May 22, 2013 at 12:30:21AM -0700, Jervin R <jervi...@percona.com> wrote:
>So there is another one - do not upgrade to 2.1.2 yet -
>https://bugs.launchpad.net/percona-xtrabackup/+bug/1182698
>

Yes, it has been confirmed now and the bug has been identified.
So, before upgrade from 2.0.x wait for XB 2.1.3 (which may be
released soon because of this critical bug), or if already on XB 2.1.2, downgrade (till
XB 2.1.3 is released).

>On Saturday, May 18, 2013 12:28:15 PM UTC+8, Ignacio Nin wrote:
>>
>> Hello,
>>
>> We've published Percona XtraBackup 2.1.2, which fixes (among others) bug
>> #1180672.
>>
>> Please find the details at
>>
>> http://www.mysqlperformanceblog.com/2013/05/18/percona-xtrabackup-2-1-2-for-mysql-available-for-download/.
>>
>>
>> Regards,
>>
>>
>> El 17/05/13 07:33, Raghavendra D Prabhu escribi�:
>> > Hi,
>> >
>> > * On Thu, May 16, 2013 at 01:31:31PM +0530, Raghavendra D Prabhu
>> > <raghavend...@percona.com <javascript:>> wrote:
>> >> Hi,
>> >>
>> >>
>> >> * On Thu, May 16, 2013 at 01:27:33PM +0530, Raghavendra D Prabhu
>> >> <raghavend...@percona.com <javascript:>> wrote:
>> >>> Hi,
>> >>>
>> >>> PXC/Galera Users: There is a regression with XB -
>> >>> https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which
>> >>
>> >> Correction. The right bug is
>> >> https://bugs.launchpad.net/percona-xtrabackup/+bug/1180672
>> >>
>> >>> directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
>> >>> 2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Regards,
>> >>> --
>> >>> Raghavendra Prabhu
>> >>> Percona LLC. - http://www.percona.com / Blog:
>> >>> http://www.mysqlperformanceblog.com/
>> >>> Skype: percona.raghavendrap
>> >>> GPG: 0xD72BE977
>> >>> Contact: http://wnohang.net/contact
>> >>>
>> >>> Make plans to attend Percona Live London MySQL Conference
>> >>> 2013.<http://www.percona.com/live/london-2013/>
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups "Percona Discussion" group.
>> >>> To unsubscribe from this group and stop receiving emails from it,
>> >>> send an email to percona-discuss...@googlegroups.com<javascript:>.
>>
>> >>> To post to this group, send email to
>> >>> percona-d...@googlegroups.com <javascript:>.
>> >>> For more options, visit https://groups.google.com/groups/opt_out.
>> >>>
>> >>>
>> >>
>> >>
>> >
>> > Fix for this bug has been committed by Xtrabackup developers
>> > (thanks!). It should be pushed during Friday US time. You can track
>> > its milestone here --
>> > https://launchpad.net/percona-xtrabackup/+milestone/2.1.2 (It also has
>> > a few other critical bugs fixed).
>> >
>> >>
>> >>
>> >>
>> >
>> >
>> >
>> >
>> > Regards,
>>
>>
>> --
>> Ignacio Nin
>> Build Engineer, Percona Inc.
>>
>> http://www.percona.com/
>> (+1 877) 862-4316 ext 524
>> skype: ignacio-nin
>> ignac...@percona.com <javascript:>
>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discuss...@googlegroups.com.
>To post to this group, send email to percona-d...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.
>
>




Sanket Gupta

unread,
May 22, 2013, 5:22:45 AM5/22/13
to percona-d...@googlegroups.com
I am not sure how packaging \releases work but should we not unpublish this release as this is a important bug that can cause major issues
I personally wasted 5 hours getting this to work and then finally did a downgrade to 2.0.7


>>> send an email to percona-discussion+unsub...@googlegroups.com<javascript:>.


>>> To post to this group, send email to
>>> percona-d...@googlegroups.com <javascript:>.

>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>
>>
>
>  Fix for this bug has been committed by Xtrabackup developers
>  (thanks!). It should be pushed during Friday US time. You can  track
> its milestone here --
> https://launchpad.net/percona-xtrabackup/+milestone/2.1.2 (It also has
> a few other critical bugs fixed).
>
>>
>>
>>
>
>
>
>
> Regards,


--
Ignacio Nin
Build Engineer, Percona Inc.

http://www.percona.com/
(+1 877) 862-4316 ext 524
skype: ignacio-nin
ignac...@percona.com <javascript:>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.
To post to this group, send email to percona-discussion@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.



Regards,
--
Raghavendra Prabhu
Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
Skype: percona.raghavendrap
GPG: 0xD72BE977
Contact: http://wnohang.net/contact

Make plans to attend Percona Live London MySQL Conference
2013.<http://www.percona.com/live/london-2013/>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsub...@googlegroups.com.
To post to this group, send email to percona-discussion@googlegroups.com.

Laurent MINOST

unread,
May 22, 2013, 5:51:57 AM5/22/13
to percona-d...@googlegroups.com
Totally agree on Sanket's opinion and moreover I think that : non-regression tests and release cycle should be IMO strongly reviewed on Percona's side to avoid such problems and above all to show a better outside image of the company ...
This is only from my point of view but with all the problems we know from previous weeks for different releases, this strongly affects the professional/serious and quality image I had of Percona.

Regards,

Laurent
>>> send an email to percona-discussion+unsubscribe@googlegroups.com<javascript:>.


>>> To post to this group, send email to
>>> percona-d...@googlegroups.com <javascript:>.

>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>
>>
>
>  Fix for this bug has been committed by Xtrabackup developers
>  (thanks!). It should be pushed during Friday US time. You can  track
> its milestone here --
> https://launchpad.net/percona-xtrabackup/+milestone/2.1.2 (It also has
> a few other critical bugs fixed).
>
>>
>>
>>
>
>
>
>
> Regards,


--
Ignacio Nin
Build Engineer, Percona Inc.

http://www.percona.com/
(+1 877) 862-4316 ext 524
skype: ignacio-nin
ignac...@percona.com <javascript:>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to percona-d...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.



Regards,
--
Raghavendra Prabhu
Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
Skype: percona.raghavendrap
GPG: 0xD72BE977
Contact: http://wnohang.net/contact

Make plans to attend Percona Live London MySQL Conference
2013.<http://www.percona.com/live/london-2013/>

--
You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to percona-d...@googlegroups.com.

Stewart Smith

unread,
May 22, 2013, 8:17:23 PM5/22/13
to Sanket Gupta, percona-d...@googlegroups.com
Sanket Gupta <sanket...@gmail.com> writes:
> I am not sure how packaging \releases work but should we not unpublish this
> release as this is a important bug that can cause major issues
> I personally wasted 5 hours getting this to work and then finally did a
> downgrade to 2.0.7

It's pretty hard to un-intentionally get an old release - the download
page will have giant warnings about downloading something that isn't the
latest. Many releases have bugs that are critical/high priority for some
segment of people so I personally prefer to keep things up, at least for
completeness than to remove them (as otherwise there's an argument to
make to remove every single old version).

--
Stewart Smith

Stewart Smith

unread,
May 22, 2013, 8:37:49 PM5/22/13
to Laurent MINOST, percona-d...@googlegroups.com
Laurent MINOST <lol...@gmail.com> writes:
> Totally agree on Sanket's opinion and moreover I think that :
> non-regression tests and release cycle should be IMO strongly reviewed on
> Percona's side to avoid such problems and above all to show a better
> outside image of the company ...
> This is only from my point of view but with all the problems we know from
> previous weeks for different releases, this strongly affects the
> professional/serious and quality image I had of Percona.

There's been a couple of interesting things here:
1) We had a bug late in the cycle with testing against current PXC which
went unnoticed, which is how bug #1180672 sneaked in. We've fixed
this problem and we're trying to come up with better was (both human
and machine) to ensure this doesn't happen again.
2) Bug #1180905 relating to innodb_plugin in 5.1 is a bit mystereous as
this is a combination we actively test... so we obviously have a bug
in our testing system we need to fix.
3) The SST hang (bug #1182698) has shown a deficiency in our automated
test coverage. We were not automatically using the latest XB as an
SST method in PXC as part of our testing before a XB release. We will
obviously need to fix this.

One of these three we should have caught with our existing tests,
another is a bug in the test system itself and the other is a gap in our
other test coverage. As is more often the case then not, all the bad
things happen at once and we got bitten by all three :(

At this point in time we have more automated regression tests, code
review and development processes to ensure quality than we have ever had
in the past (and I do plan to write up some blog posts on
this). However... we still have things that slip through. While some of
these bugs could have been avoided with more automated testing, some
could possibly also have been caught in one of the many pre-release
releases of XB 2.1. I'm currently trying to work out ways to make it
easier for people to test upcoming XB releases so that we can find these
gaps in our testing sooner rather than later. It appears that the vast
majority of people do *not* test any of the alpha, beta or release
candidate releases and instead just wait for GA.

Before any code hits the source trees for XtraBackup, we test across an
array of roughly 130 MySQL version and Operating System combinations
with now what numbers about 120 regression tests. There are about 8x
more regression tests now than there were when XtraBackup 1.6 was new
(and many of those have been improved since then too). This is in
addition to stringent code review.

I often joke that doing a build of XtraBackup is a distributed denial of
service attack against build infrastructure for the number of jobs it
kicks off... and we've had to actively optimize the execution of the
test suite as to not be drowned by it completely.

I *completely* understand your frustration. We see each release with a
regression as a failure on our part. We are actively working on both
what caused these specific regressions and in the general case. Stay
tuned :)

--
Stewart Smith
Reply all
Reply to author
Forward
0 new messages