Regression with Xtrabackup affecting PXC SST

Showing 1-11 of 11 messages
Regression with Xtrabackup affecting PXC SST Raghavendra Prabhu 5/16/13 12:57 AM
Hi,

PXC/Galera Users:
There is a regression with XB -
https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which
directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.





Regards,
--
Raghavendra Prabhu
Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
Skype: percona.raghavendrap
GPG: 0xD72BE977
Contact: http://wnohang.net/contact

Make plans to attend Percona Live London MySQL Conference
2013.<http://www.percona.com/live/london-2013/>
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Raghavendra Prabhu 5/16/13 1:01 AM
Hi,


* On Thu, May 16, 2013 at 01:27:33PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>Hi,
>
>PXC/Galera Users: There is a regression with XB -
>https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which

Correction. The right bug is https://bugs.launchpad.net/percona-xtrabackup/+bug/1180672

>directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
>2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.
>
>
>
>
>
>Regards,
>--
>Raghavendra Prabhu
>Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
>Skype: percona.raghavendrap
>GPG: 0xD72BE977
>Contact: http://wnohang.net/contact
>
>Make plans to attend Percona Live London MySQL Conference
>2013.<http://www.percona.com/live/london-2013/>
>
>--
>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsubscribe@googlegroups.com.
>To post to this group, send email to percona-d...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Amol 5/16/13 8:57 PM
Thanks for informing us, will hold off the upgrade
Re: Re: [percona-group] Regression with Xtrabackup affecting PXC SST Raghavendra Prabhu 5/17/13 3:33 AM
Hi,

* On Thu, May 16, 2013 at 01:31:31PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>Hi,
>
>
>* On Thu, May 16, 2013 at 01:27:33PM +0530, Raghavendra D Prabhu <raghavend...@percona.com> wrote:
>>Hi,
>>
>>PXC/Galera Users: There is a regression with XB -
>>https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which
>
>Correction. The right bug is https://bugs.launchpad.net/percona-xtrabackup/+bug/1180672
>
>>directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
>>2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.
>>
>>
>>
>>
>>
>>Regards,
>>--
>>Raghavendra Prabhu
>>Percona LLC. - http://www.percona.com / Blog: http://www.mysqlperformanceblog.com/
>>Skype: percona.raghavendrap
>>GPG: 0xD72BE977
>>Contact: http://wnohang.net/contact
>>
>>Make plans to attend Percona Live London MySQL Conference
>>2013.<http://www.percona.com/live/london-2013/>
>>
>>--
>>You received this message because you are subscribed to the Google Groups "Percona Discussion" group.
>>To unsubscribe from this group and stop receiving emails from it, send an email to percona-discussion+unsubscribe@googlegroups.com.
>>To post to this group, send email to percona-d...@googlegroups.com.
>>For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
>

  Fix for this bug has been committed by Xtrabackup developers
  (thanks!). It should be pushed during Friday US time. You can
  track its milestone here -- https://launchpad.net/percona-xtrabackup/+milestone/2.1.2 (It also has a few other critical bugs fixed).
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Ignacio Nin 5/17/13 9:28 PM
Hello,

We've published Percona XtraBackup 2.1.2, which fixes (among others) bug
#1180672.

Please find the details at
http://www.mysqlperformanceblog.com/2013/05/18/percona-xtrabackup-2-1-2-for-mysql-available-for-download/.

Regards,


El 17/05/13 07:33, Raghavendra D Prabhu escribi�:
Ignacio Nin
Build Engineer, Percona Inc.

http://www.percona.com/
(+1 877) 862-4316 ext 524
skype: ignacio-nin
ignac...@percona.com
http://www.mysqlperformanceblog.com/

Got MySQL support?
http://tinyurl.com/mysql-support

Re: [percona-group] Regression with Xtrabackup affecting PXC SST Jervin R 5/22/13 12:30 AM
So there is another one - do not upgrade to 2.1.2 yet - https://bugs.launchpad.net/percona-xtrabackup/+bug/1182698


On Saturday, May 18, 2013 12:28:15 PM UTC+8, Ignacio Nin wrote:
Hello,

We've published Percona XtraBackup 2.1.2, which fixes (among others) bug
#1180672.

Please find the details at
http://www.mysqlperformanceblog.com/2013/05/18/percona-xtrabackup-2-1-2-for-mysql-available-for-download/.

Regards,


El 17/05/13 07:33, Raghavendra D Prabhu escribi�:
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Raghavendra Prabhu 5/22/13 2:04 AM
Hi,

* On Wed, May 22, 2013 at 12:30:21AM -0700, Jervin R <jervi...@percona.com> wrote:
>So there is another one - do not upgrade to 2.1.2 yet -
>https://bugs.launchpad.net/percona-xtrabackup/+bug/1182698
>

Yes, it has been confirmed now and the bug has been identified.
So, before upgrade from 2.0.x wait for XB 2.1.3 (which may be
released soon because of this critical bug), or if already on XB 2.1.2, downgrade (till
XB 2.1.3 is released).

>On Saturday, May 18, 2013 12:28:15 PM UTC+8, Ignacio Nin wrote:
>>
>> Hello,
>>
>> We've published Percona XtraBackup 2.1.2, which fixes (among others) bug
>> #1180672.
>>
>> Please find the details at
>>
>> http://www.mysqlperformanceblog.com/2013/05/18/percona-xtrabackup-2-1-2-for-mysql-available-for-download/.
>>
>>
>> Regards,
>>
>>
>> El 17/05/13 07:33, Raghavendra D Prabhu escribi�:
>> > Hi,
>> >
>> > * On Thu, May 16, 2013 at 01:31:31PM +0530, Raghavendra D Prabhu
>> > <raghavend...@percona.com <javascript:>> wrote:
>> >> Hi,
>> >>
>> >>
>> >> * On Thu, May 16, 2013 at 01:27:33PM +0530, Raghavendra D Prabhu
>> >> <raghavend...@percona.com <javascript:>> wrote:
>> >>> Hi,
>> >>>
>> >>> PXC/Galera Users: There is a regression with XB -
>> >>> https://bugs.launchpad.net/percona-xtrabackup/+bug/1180662 - which
>> >>
>> >> Correction. The right bug is
>> >> https://bugs.launchpad.net/percona-xtrabackup/+bug/1180672
>> >>
>> >>> directly affects Xtrabackup SST. So, if you do upgrade Xtrabackup to
>> >>> 2.1.1, don't use Xtrabackup SST till the bugs is fixed/pushed.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Regards,
>> >>> --
>> >>> Raghavendra Prabhu
>> >>> Percona LLC. - http://www.percona.com / Blog:
>> >>> http://www.mysqlperformanceblog.com/
>> >>> Skype: percona.raghavendrap
>> >>> GPG: 0xD72BE977
>> >>> Contact: http://wnohang.net/contact
>> >>>
>> >>> Make plans to attend Percona Live London MySQL Conference
>> >>> 2013.<http://www.percona.com/live/london-2013/>
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups "Percona Discussion" group.
>> >>> To unsubscribe from this group and stop receiving emails from it,
>> >>> send an email to percona-discussion+unsubscribe@googlegroups.com<javascript:>.
>>
>> >>> To post to this group, send email to
>> >>> percona-d...@googlegroups.com <javascript:>.
>> >>> For more options, visit https://groups.google.com/groups/opt_out.
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >  Fix for this bug has been committed by Xtrabackup developers
>> >  (thanks!). It should be pushed during Friday US time. You can  track
>> > its milestone here --
>> > https://launchpad.net/percona-xtrabackup/+milestone/2.1.2 (It also has
>> > a few other critical bugs fixed).
>> >
>> >>
>> >>
>> >>
>> >
>> >
>> >
>> >
>> > Regards,
>>
>>
>> --
>> Ignacio Nin
>> Build Engineer, Percona Inc.
>>
>> http://www.percona.com/
>> (+1 877) 862-4316 ext 524
>> skype: ignacio-nin
>> ignac...@percona.com <javascript:>
>> http://www.mysqlperformanceblog.com/
>>
>> Got MySQL support?
>> http://tinyurl.com/mysql-support
>>
>>
>
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Sanket Gupta 5/22/13 2:22 AM
I am not sure how packaging \releases work but should we not unpublish this release as this is a important bug that can cause major issues
I personally wasted 5 hours getting this to work and then finally did a downgrade to 2.0.7
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Laurent MINOST 5/22/13 2:51 AM
Totally agree on Sanket's opinion and moreover I think that : non-regression tests and release cycle should be IMO strongly reviewed on Percona's side to avoid such problems and above all to show a better outside image of the company ...
This is only from my point of view but with all the problems we know from previous weeks for different releases, this strongly affects the professional/serious and quality image I had of Percona.

Regards,

Laurent
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Stewart Smith 5/22/13 5:17 PM
Sanket Gupta <sanket...@gmail.com> writes:
> I am not sure how packaging \releases work but should we not unpublish this
> release as this is a important bug that can cause major issues
> I personally wasted 5 hours getting this to work and then finally did a
> downgrade to 2.0.7

It's pretty hard to un-intentionally get an old release - the download
page will have giant warnings about downloading something that isn't the
latest. Many releases have bugs that are critical/high priority for some
segment of people so I personally prefer to keep things up, at least for
completeness than to remove them (as otherwise there's an argument to
make to remove every single old version).

--
Stewart Smith
Re: [percona-group] Regression with Xtrabackup affecting PXC SST Stewart Smith 5/22/13 5:37 PM
Laurent MINOST <lol...@gmail.com> writes:
> Totally agree on Sanket's opinion and moreover I think that :
> non-regression tests and release cycle should be IMO strongly reviewed on
> Percona's side to avoid such problems and above all to show a better
> outside image of the company ...
> This is only from my point of view but with all the problems we know from
> previous weeks for different releases, this strongly affects the
> professional/serious and quality image I had of Percona.

There's been a couple of interesting things here:
1) We had a bug late in the cycle with testing against current PXC which
   went unnoticed, which is how bug #1180672 sneaked in. We've fixed
   this problem and we're trying to come up with better was (both human
   and machine) to ensure this doesn't happen again.
2) Bug #1180905 relating to innodb_plugin in 5.1 is a bit mystereous as
   this is a combination we actively test... so we obviously have a bug
   in our testing system we need to fix.
3) The SST hang (bug #1182698) has shown a deficiency in our automated
   test coverage. We were not automatically using the latest XB as an
   SST method in PXC as part of our testing before a XB release. We will
   obviously need to fix this.

One of these three we should have caught with our existing tests,
another is a bug in the test system itself and the other is a gap in our
other test coverage. As is more often the case then not, all the bad
things happen at once and we got bitten by all three :(

At this point in time we have more automated regression tests, code
review and development processes to ensure quality than we have ever had
in the past (and I do plan to write up some blog posts on
this). However... we still have things that slip through. While some of
these bugs could have been avoided with more automated testing, some
could possibly also have been caught in one of the many pre-release
releases of XB 2.1. I'm currently trying to work out ways to make it
easier for people to test upcoming XB releases so that we can find these
gaps in our testing sooner rather than later. It appears that the vast
majority of people do *not* test any of the alpha, beta or release
candidate releases and instead just wait for GA.

Before any code hits the source trees for XtraBackup, we test across an
array of roughly 130 MySQL version and Operating System combinations
with now what numbers about 120 regression tests. There are about 8x
more regression tests now than there were when XtraBackup 1.6 was new
(and many of those have been improved since then too). This is in
addition to stringent code review.

I often joke that doing a build of XtraBackup is a distributed denial of
service attack against build infrastructure for the number of jobs it
kicks off... and we've had to actively optimize the execution of the
test suite as to not be drowned by it completely.

I *completely* understand your frustration. We see each release with a
regression as a failure on our part. We are actively working on both
what caused these specific regressions and in the general case. Stay
tuned :)

--
Stewart Smith