[Lustre-discuss] Lustre 1.6.3 - where are the bug fixes?
Niklas Edmundsson
OK, I know that there is supposedly some QA before lustre releases and
that it might be the reason for fixes taking such a long time to
propagate, but still: It takes too long for fixes to end up in a
released version...
During our rather limited testing on Ubuntu Dapper (using the Debian
2.6.18 kernel on servers and pkg-lustre packaging) we've run into
a couple of bugs, most of them with the typical "fix in bugzilla".
The pkg-lustre packaging has six fixes from bugzilla applied, they
seem to have munged the bug numbers but it seems that only three of
them are in the 1.6.3 changelog.
We have locally applied fixes from bug 13438 (lustre is totally
useless without it due to servers OOPS:ing) and 13614. None of them
seems to be in the 1.6.3 changelog.
So, I'd suggest that CFS gets their act together and starts releasing
versions more often, if they'd done this during 1.6 development we
wouldn't be installing production releases that you can crash after a
day of testing now.
If QA is the argument for not doing releases more often, consider the
fact that known broken releases that you have to patch yourself with
patches hidden in bugzilla isn't much better.
In reality, I think that doing non-QA'd snapshot releases might be the
way to go. That is, releases with the useful more-or-less trivial
fixes that avoids crashes etc. and that will be included in the next
QA'd release. They would not be suitable for production, but at least
you can rather easily download the latest snapshot and try on your
test cluster and see if it fixes the problem(s) you've encountered.
And if it does, we can bug CFS until they get their act together and
gets a release out with the fix included.
In the end, you have to realise that when you have a production system
you don't want to wait for weeks and months for a new release that
might fix a crash-inducing bug you're hitting. I say might here,
because obviously having a fix hidden in bugzilla is no guarantee that
it's included in a released version.
In our case we're not at production yet because of these problems with
getting fixes out quickly enough. So far we've always been able to
crash lustre 1.6 within days, and that's after waiting for 1.6 for
well over a year.
So, I'd like to challenge CFS to get a version of lustre 1.6 (or 1.8,
whatever) out that proves stable on our small lustre test setup.
Without patches. In the year of 2007.
Since the "internal QA only" approach obviously isn't working, I'd
suggest that you embrace "release early, release often" to get there.
That means one release per week as long as you have fixes pending to
get a decent churn on things.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@hpc2n.umu.se
---------------------------------------------------------------------------
Short cut... the longest distance between two points.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
_______________________________________________
Lustre-discuss mailing list
Lustre-...@clusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Kevin Canady
Are you customer of Lustre Support? I don't have you listed as a supported
customer. Maybe we should arrange a discussion about how we could assist
you more effectively.
Best regards,
Kevin
--
P. Kevin Canady
Director, Business Development
Lustre Group (Formerly CFS)
Sun Microsystems, Inc.
O: 415.928.3633
C: 415.505.7701
On 10/11/07 11:53 PM, "Niklas Edmundsson" <Niklas.E...@hpc2n.umu.se>
wrote:
Charles Taylor
Hmmm. This is the approach that has always frightened us away from
Lustre. Is there one version of Lustre for paying support customers
and another for those who, for whatever reason, can't or don't want
to pay for support. What do you mean when you say "assist you more
effectively"? Does that mean you will apply the patches in the
bugzilla database for him? How does that help if he can do it
himself? The point Nikke is making is that a release labeled
"production" should run for more than a few hours to a day without
crashing. And, if you have a "production" release that crashes that
readily AND there are known fixes, it seems like they should be
readily available in a format that is easy to apply (as opposed to
digging through bug reports and applying code fixes by hand).
For a lot of very good reasons, we would like to go to lustre.
There is much to like about it. However, we run OFED 1.2 on our
cluster and need Lustre 1.6.2+ for OFED support. So far, our
attempts to test this version of lustre on our 400+ node IB cluster
have resulted in impressive performance and scalability and...lots of
crashes (mballoc) and a corrupt file system that neither e2fsck nor
lfsck could fix. It is too bad because it seems that lustre is just
a few fixes away from having one of the most amazing open source
packages in the Linux universe.
We wish you the best and hope that we will be able to use Lustre in
the near future.
Charlie Taylor
UF HPC Center
Tru Huynh
>
> Hmmm. This is the approach that has always frightened us away from
> Lustre. Is there one version of Lustre for paying support customers
> and another for those who, for whatever reason, can't or don't want
> to pay for support.
That's my personnal opinion too: it's fine if clusterfs.com charges for support
but it's pretty frigthening if the OSS version if different from the paying version.
Cheers,
Tru
--
Dr Tru Huynh | http://www.pasteur.fr/recherche/unites/Binfs/
mailto:t...@pasteur.fr | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France
Niklas Edmundsson
> Nikke,
> Are you customer of Lustre Support? I don't have you listed as a supported
> customer. Maybe we should arrange a discussion about how we could assist
> you more effectively.
Not anymore. Back when we had support we didn't get much useful
answers to our support cases, they entered into bugzilla and if we
were lucky we got some kind of response. And given that we didn't have
lustre in production (partly due to hardware issues, so that's not
entirely due to lustre) we didn't see any reason to continue paying
for support.
If we get to the point that lustre 1.6 (or 1.8, or...) seems to be
production ready we might use it. If we use it we might purchase
support again.
But, support or not this doesn't answer the fundamental issue: It's a
long time between lustre releases, a fact that hasn't changed since
the public CVS access was revoked.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@hpc2n.umu.se
---------------------------------------------------------------------------
"He will not be permanently damaged." - Vader
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Andreas Dilger
> OK, I know that there is supposedly some QA before lustre releases and
> that it might be the reason for fixes taking such a long time to
> propagate, but still: It takes too long for fixes to end up in a
> released version...
I tend to agree that it does take a long time between releases, and
we are definitely trying to improve this.
> The pkg-lustre packaging has six fixes from bugzilla applied, they
> seem to have munged the bug numbers but it seems that only three of
> them are in the 1.6.3 changelog.
Could you please indicate which bugs are missing?
> We have locally applied fixes from bug 13438 (lustre is totally
> useless without it due to servers OOPS:ing) and 13614. None of them
> seems to be in the 1.6.3 changelog.
The 13614 patch is actually in 1.6.3 under the original bug number
where the patch came from (bug 13596).
As for 13438, I cannot explain why this didn't make it into the 1.6.3
release, since it was definitely ready in time.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
Andreas Dilger
> On Fri, Oct 12, 2007 at 05:49:44AM -0400, Charles Taylor wrote:
> > Hmmm. This is the approach that has always frightened us away from
> > Lustre. Is there one version of Lustre for paying support customers
> > and another for those who, for whatever reason, can't or don't want
> > to pay for support.
>
> That's my personnal opinion too: it's fine if clusterfs.com charges for
> support but it's pretty frigthening if the OSS version if different from
> the paying version.
I'm not sure where you got this opinion, but it is incorrect. There is
a single version of Lustre that we release. Why would we ever want to
have the overhead of maintaining two separate releases?
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
_______________________________________________
Andreas Dilger
> Hmmm. This is the approach that has always frightened us away from
> Lustre. Is there one version of Lustre for paying support customers
> and another for those who, for whatever reason, can't or don't want
> to pay for support.
No, there is a single version that we release for everyone.
> What do you mean when you say "assist you more
> effectively"? Does that mean you will apply the patches in the
> bugzilla database for him? How does that help if he can do it
> himself?
The issue is that we are a company that is paying a lot of people to
maintain and develop Lustre, and without paying customers none of this
would get very far. It's great that we can give Lustre away freely for
anyone, but the truth is that customers who are paying for support have
priority in terms of getting attention to their bugs.
In terms of how it can help Nikke to have support with CFS - it means
we can afford to have someone smart like Alex on our staff to fix this
problem in the first place. The fact that the fix in question didn't make
it into 1.6.3 is an oversight on our part, as the patch was definitely
ready in time for the release. We will of course include it in the next
release, possibly making a point release with this fix sooner.
> For a lot of very good reasons, we would like to go to lustre.
> There is much to like about it. However, we run OFED 1.2 on our
> cluster and need Lustre 1.6.2+ for OFED support. So far, our
> attempts to test this version of lustre on our 400+ node IB cluster
> have resulted in impressive performance and scalability and...lots of
> crashes (mballoc) and a corrupt file system that neither e2fsck nor
> lfsck could fix. It is too bad because it seems that lustre is just
> a few fixes away from having one of the most amazing open source
> packages in the Linux universe.
I'm sorry to hear you are having problems, possibly the same ones being
discussed here. As for OFED 1.2, this will also be available in the
1.4.12 release, if that is of interest to you, and of course we are
working to address issues with 1.6 as well.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
_______________________________________________
Patrick Winnertz
> On Oct 12, 2007 08:53 +0200, Niklas Edmundsson wrote:
> > OK, I know that there is supposedly some QA before lustre releases and
> > that it might be the reason for fixes taking such a long time to
> > propagate, but still: It takes too long for fixes to end up in a
> > released version...
>
> I tend to agree that it does take a long time between releases, and
> we are definitely trying to improve this.
>
> > The pkg-lustre packaging has six fixes from bugzilla applied, they
> > seem to have munged the bug numbers but it seems that only three of
> > them are in the 1.6.3 changelog.
>
> Could you please indicate which bugs are missing?
I'm one of the debian maintainers and I started today to package 1.6.3 and
look through your bugzilla in order to get all patches which are necessary
to build a stable package.
While doing so i detect that your changelog on your website is not
completly correct. (It's missing some bugs which are closed in the 1.6.3
release):
This bugs are:
- 13610
- 12475
- 5491
- 11880
I guess someone has forgotten to mention them there. I would suggest to
update this information.
Greetings
Patrick Winnertz
--
Patrick Winnertz
Tel.: +49 (0) 2161 / 4643 - 0
credativ GmbH, HRB Mönchengladbach 12080
Hohenzollernstr. 133, 41061 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Jörg Folz
Jim Garlick
> In reality, I think that doing non-QA'd snapshot releases might be the
> way to go. That is, releases with the useful more-or-less trivial
> fixes that avoids crashes etc. and that will be included in the next
> QA'd release.
Yes, we find ourselves carrying a lot of bugfix patches in our
internal Lustre releases (50+ in our last production release based
on 1.4.8, already 30+ in our release planned for january based on 1.6.2).
I think it would be useful if CFS would freeze features in a release at
some point and do bugfix-only releases. For example, we're trying to
get 1.6.2 stablized for production use and we know that many bugfixes
that we need are in 1.6.3, but we have to backport them because we don't
want to destablize our effort by taking the features in 1.6.3.
I will say in CFS's defense that we abuse Lustre in ways that not all
customers do, and so in some cases a bugfix to us might be controversial
to apply to a stable release series. Not true of all the patches
wer are carrying though.
Jim
Andreas Dilger
> Am Freitag, 12. Oktober 2007 12:29:08 schrieb Andreas Dilger:
> > On Oct 12, 2007 08:53 +0200, Niklas Edmundsson wrote:
> > > OK, I know that there is supposedly some QA before lustre releases and
> > > that it might be the reason for fixes taking such a long time to
> > > propagate, but still: It takes too long for fixes to end up in a
> > > released version...
> >
> > I tend to agree that it does take a long time between releases, and
> > we are definitely trying to improve this.
> >
> > > The pkg-lustre packaging has six fixes from bugzilla applied, they
> > > seem to have munged the bug numbers but it seems that only three of
> > > them are in the 1.6.3 changelog.
> >
> > Could you please indicate which bugs are missing?
>
> look through your bugzilla in order to get all patches which are necessary
> to build a stable package.
Actually, what I'M interested in is which patches you have in the Debian
packages that are not in the CFS release. We obviously want to include
such fixes into our release, and I agree that our patch-tracking process
can miss fixes on occasion, so having feedback from the Debian maintainers
would definitely help avoid such issues.
> While doing so i detect that your changelog on your website is not
> completly correct. (It's missing some bugs which are closed in the 1.6.3
> release):
> This bugs are:
> - 13610
This relates to a fix made to a patch that was itself landed before 1.6.3
was released, so our policy is to not mention this as a separate fix in
the ChangeLog, to avoid clutter.
> - 12475
This one is only an error message fix. We don't necessarily add every
change into the ChangeLog, as this makes it too noisy to see the important
fixes going in.
> - 5491
> - 11880
These two were landed 1.6.3, but the proper processes weren't being followed.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
_______________________________________________
Kilian CAVALOTTI
First, I'd like to say that I tend to agree with Niklas' remarks in
general. I think that having stable, QA'd releases is important, but
considering that most of the Lustre audience is made of highly
technical people running large installations and needing to run their
own tests, having more frequent (not fully QA'd) releases would be a
great deal of help.
On Friday 12 October 2007 03:29:08 am Andreas Dilger wrote:
> Could you please indicate which bugs are missing?
We're also missing 13438, which is really a stopper for this release.
> The 13614 patch is actually in 1.6.3 under the original bug number
> where the patch came from (bug 13596).
Good to know. It wasn't clear since 13596 is not public, and the
discussion in 13614 didn't indicate any progress on 13596 resolution,
neither gave any confirmation that both bugs were resolved by the same
fix.
> As for 13438, I cannot explain why this didn't make it into the 1.6.3
> release, since it was definitely ready in time.
What about a 1.6.3.1 including 13438? ;)
Cheers,
--
Kilian
Kilian CAVALOTTI
On Friday 12 October 2007 09:31:46 am Jim Garlick wrote:
> I will say in CFS's defense that we abuse Lustre in ways that not all
> customers do,
We all abuse Lustre one way or another. :)
Considering the astonishing performance Lustre gives our users, and
since most of them are not accustomed to that from their past
experience, they sometimes tend to see it as a ressource they can
stress shamelessly. Which at least gives us the opportunity to fill up
some bug reports. :)
Cheers,
--
Kilian
Niklas Edmundsson
>> The 13614 patch is actually in 1.6.3 under the original bug number
>> where the patch came from (bug 13596).
>
> Good to know. It wasn't clear since 13596 is not public, and the
> discussion in 13614 didn't indicate any progress on 13596 resolution,
> neither gave any confirmation that both bugs were resolved by the same
> fix.
I agree. This is why I prefer that all bug numbers are mentioned in
the changelog (be it a separate long-format changelog).
Actually, I'd prefer if the bugs themselves were public in the
bugzilla once identified. The reason for having bugs private should be
that the discussion/testcase/etc needs to be private, not that the
actual bug itself is a secret, right?
>> As for 13438, I cannot explain why this didn't make it into the 1.6.3
>> release, since it was definitely ready in time.
>
> What about a 1.6.3.1 including 13438? ;)
What about a more general policy that pure bugfixes yields a new
point-release? As Jim Garlick points out there are usually new
features interspersed with the bugfixes, which has the potential of
destabilizing the whole thing.
I'd prefer if point releases were produced until the thing was
relatively stable (meaning that people can actually run it without
applying their own batch of fixes), and then new features were
included. Right now the only way of stabilizing lustre seems to be for
each site to run into bugs, and dig in maillists/bugzilla to find the
fixes.
So, get 1.6.3.1 out the door ASAP, and keep producing point-releases
weekly until you've closed all "my server/client goes OOPS" bugs. I
can almost guarantee that if CFS shows that it intends to fix these
bugs swiftly people will deploy new point releases and report the bugs
they find.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@hpc2n.umu.se
---------------------------------------------------------------------------
* <- Tribble *\ <- Tribble Going Skiing
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Troy Benjegerdes
> On Fri, 12 Oct 2007, Niklas Edmundsson wrote:
>
> > In reality, I think that doing non-QA'd snapshot releases might be the
> > way to go. That is, releases with the useful more-or-less trivial
> > fixes that avoids crashes etc. and that will be included in the next
> > QA'd release.
>
> Yes, we find ourselves carrying a lot of bugfix patches in our
> internal Lustre releases (50+ in our last production release based
> on 1.4.8, already 30+ in our release planned for january based on 1.6.2).
>
> I think it would be useful if CFS would freeze features in a release at
> some point and do bugfix-only releases. For example, we're trying to
> get 1.6.2 stablized for production use and we know that many bugfixes
> that we need are in 1.6.3, but we have to backport them because we don't
> want to destablize our effort by taking the features in 1.6.3.
This sounds like a lot of duplicated effort that if CFS did not want to
support could be managed by a decent distributed source control system
(either git or mercurial), and the interested 'power users'.
It's also annoying that a quick web search for 'Lustre CVS' turns up a
bunch of ancient or dead links. If I could get the latest Lustre
development version from a *public* source control server, I'd be using
Lustre for testing remote filesystem access over InfiniBand WAN links.
As it is, the only parallel filesystem I can easily find with a public
source control server is PVFS, so that's what I'm using, and it works
quite nicely.