Laurent MINOST <
lol...@gmail.com> writes:
> Totally agree on Sanket's opinion and moreover I think that :
> non-regression tests and release cycle should be IMO strongly reviewed on
> Percona's side to avoid such problems and above all to show a better
> outside image of the company ...
> This is only from my point of view but with all the problems we know from
> previous weeks for different releases, this strongly affects the
> professional/serious and quality image I had of Percona.
There's been a couple of interesting things here:
1) We had a bug late in the cycle with testing against current PXC which
went unnoticed, which is how bug #1180672 sneaked in. We've fixed
this problem and we're trying to come up with better was (both human
and machine) to ensure this doesn't happen again.
2) Bug #1180905 relating to innodb_plugin in 5.1 is a bit mystereous as
this is a combination we actively test... so we obviously have a bug
in our testing system we need to fix.
3) The SST hang (bug #1182698) has shown a deficiency in our automated
test coverage. We were not automatically using the latest XB as an
SST method in PXC as part of our testing before a XB release. We will
obviously need to fix this.
One of these three we should have caught with our existing tests,
another is a bug in the test system itself and the other is a gap in our
other test coverage. As is more often the case then not, all the bad
things happen at once and we got bitten by all three :(
At this point in time we have more automated regression tests, code
review and development processes to ensure quality than we have ever had
in the past (and I do plan to write up some blog posts on
this). However... we still have things that slip through. While some of
these bugs could have been avoided with more automated testing, some
could possibly also have been caught in one of the many pre-release
releases of XB 2.1. I'm currently trying to work out ways to make it
easier for people to test upcoming XB releases so that we can find these
gaps in our testing sooner rather than later. It appears that the vast
majority of people do *not* test any of the alpha, beta or release
candidate releases and instead just wait for GA.
Before any code hits the source trees for XtraBackup, we test across an
array of roughly 130 MySQL version and Operating System combinations
with now what numbers about 120 regression tests. There are about 8x
more regression tests now than there were when XtraBackup 1.6 was new
(and many of those have been improved since then too). This is in
addition to stringent code review.
I often joke that doing a build of XtraBackup is a distributed denial of
service attack against build infrastructure for the number of jobs it
kicks off... and we've had to actively optimize the execution of the
test suite as to not be drowned by it completely.
I *completely* understand your frustration. We see each release with a
regression as a failure on our part. We are actively working on both
what caused these specific regressions and in the general case. Stay
tuned :)
--
Stewart Smith