[Lustre-discuss] lustre interoperability
Jeff Blasius
I've seriously trekked through the lustre documentation and haven't
found an answer regarding this. Is there an official policy regarding
interoperability among different versions of various lustre
components?
For example (and I'm sure other groups are in the same boat here),
it's relatively painless to perform a rolling upgrade to the lustre
clients, but upgrading the OSS or MDS takes more convincing. Is it OK
for me to run a patched but 1.6.0 based OSS with a 1.6.3 client? In
this case all of the lustre components (kernel, lustre, ldiskfs) are
the same version for each host. Similarly, is it OK to run a lustre
kernel version out of sync with the userland tools? For example a
1.6.0 kernel with a 1.6.3 lustre build on the same host?
I understand that many of these combinations do in fact work, I'm more
interested if they're likely to lead to data corruption or client
evictions. I'm not sure how often incompatibilities arise, but if it's
relatively rare, it would be useful if that was announced on the
change log. Of course if there's a serious "Do at your own risk"
policy that would also be useful to know.
Thank You,
jeff
--
Jeff Blasius / jeff.b...@yale.edu
Phone: (203)432-9940 51 Prospect Rm. 011
High Performance Computing (HPC)
Linux Systems Design & Support (LSDS)
Yale University Information Technology Services (ITS)
_______________________________________________
Lustre-discuss mailing list
Lustre-...@clusterfs.com
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Jody McIntyre
On Fri, Oct 12, 2007 at 12:01:04PM -0400, Jeff Blasius wrote:
> I've seriously trekked through the lustre documentation and haven't
> found an answer regarding this. Is there an official policy regarding
> interoperability among different versions of various lustre
> components?
By coincidence, I just sent information about this to our documentation
team. It should eventually reach the manual. Here it is:
---
Our supported upgrades are from one minor version to another, for
example 1.4.10 to 1.4.11 or 1.6.2 to 1.6.3, and also from the latest
1.4.x version to the latest 1.6.x version, so 1.4.11 to 1.6.3 is
supported.
We also support downgrades within the same ranges. For example, if you
upgrade from 1.6.2 to 1.6.3, you can also downgrade to 1.6.2 (but a
fresh install of 1.6.3 is _not_ guaranteed to be downgradeable.)
Note that other combinations will work and we support them in specific
cases when requested by customers, but the ranges above will always
work.
---
> For example (and I'm sure other groups are in the same boat here),
> it's relatively painless to perform a rolling upgrade to the lustre
> clients, but upgrading the OSS or MDS takes more convincing. Is it OK
> for me to run a patched but 1.6.0 based OSS with a 1.6.3 client? In
> this case all of the lustre components (kernel, lustre, ldiskfs) are
> the same version for each host. Similarly, is it OK to run a lustre
> kernel version out of sync with the userland tools? For example a
> 1.6.0 kernel with a 1.6.3 lustre build on the same host?
Not necessarily. You should do a rolling upgrade to 1.6.1, then 1.6.2,
then 1.6.3. Upgrading will be easier if you stay more current - 1.6.0
is fairly old at this point.
Having said that, I believe 1.6.0 and 1.6.3 is actually something that
will work, but I'm not 100% certain of this, so I'll allow others to
correct me.
Cheers,
Jody
> I understand that many of these combinations do in fact work, I'm more
> interested if they're likely to lead to data corruption or client
> evictions. I'm not sure how often incompatibilities arise, but if it's
> relatively rare, it would be useful if that was announced on the
> change log. Of course if there's a serious "Do at your own risk"
> policy that would also be useful to know.
>
> Thank You,
> jeff
>
> --
> Jeff Blasius / jeff.b...@yale.edu
> Phone: (203)432-9940 51 Prospect Rm. 011
> High Performance Computing (HPC)
> Linux Systems Design & Support (LSDS)
> Yale University Information Technology Services (ITS)
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-...@clusterfs.com
> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
>
--
Klaus Steden
Hello,
I am working on incorporating failover and high availability functionality with Lustre into one of our new research clusters.
I am also a bit of a lazy sysadmin, and prefer to not have to modify much on a day-to-day basis. I am using the ROCKS cluster framework in conjunction with Lustre, so a good deal of the work is handled in an automated fashion.
I have a question, though ...
I have redundant MDS/MDT servers, and when generating their /etc/fstab files, I have them both list the MDT volume under their mounted drives (so that they both mount the metadata volume). Being a bit of a neophyte, I’m not sure if this is the right thing to do. Is this going to cause a problem in future? What are the possible consequences of both hosts mounting the same metadata volume simultaneously? How do the OSTs handle this situation?
thanks,
Klaus
Kilian CAVALOTTI
On Friday 12 October 2007 10:18:49 am Klaus Steden wrote:
> I have redundant MDS/MDT servers, and when generating their
> /etc/fstab files, I have them both list the MDT volume under their
> mounted drives (so that they both mount the metadata volume). Being a
> bit of a neophyte, I¹m not sure if this is the right thing to do. Is
> this going to cause a problem in future? What are the possible
> consequences of both hosts mounting the same metadata volume
> simultaneously? How do the OSTs handle this situation?
I'm afraid mounting the MDT device on both hosts is a very bad idea. Not
only for Lustre, but in a general fashion, unless you're using some
specific filesystem which has been designed in that goal (like OCFS2 or
GFS). A 'standard' filesystem like ext3 can't handle simultaneous
writes from different hosts at the same time. It's likely to cause
heavy data corruption in the MDT filesystem, and the OSTs are not
supposed to deal with anything like that.
You can still reference the MDT device in both MDS' fstab, but be sure
you add the "noauto" option so the volume won't be mounted at boot. It
will be the high-availability provider's job (likely heartbeat) to deal
with mounting the MDT device on the active MDS.
See
http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-08-1.html#wp1035482
for more details.
Cheers,
--
Kilian
Klaus Steden
Heheheh, I figured.
What about mounting OST targets from multiple hosts, i.e. OST0000 and OST0001 on both oss-0-0 and oss-0-1? Just as bad?
Klaus
On 10/12/07 10:38 AM, "Kilian CAVALOTTI" <kil...@stanford.edu>did etch on stone tablets:
Kilian CAVALOTTI
> Heheheh, I figured.
>
> What about mounting OST targets from multiple hosts, i.e. OST0000 and
> OST0001 on both oss-0-0 and oss-0-1? Just as bad?
Yep, for the same reasons.
If you want redundancy for your OSTs, you can have a kind of
active/active setup where oss-0-0 would be active for OST0000 and
passive for OST0001, and oss-0-1 would be passive for OST0000 and
active for OST0001.
Sheila Barthel
Not a problem to add a section in the Lustre manual re: supported upgrades. However, as we do not currently sync Lustre releases with releases of the Lustre manual, this wouldn't help us to address the issue that Jeff poses.
For future Lustre releases, could we add a supported upgrade/interop statement to the change log? I'll also add this content to the manual (which will have more value once Lustre s/w and manual releases are synchronized).
Sheila
Klaus Steden
Neither of these is something that can be easily accomplished from /etc/fstab, I would have to implement the failover functionality described in the link you sent, correct?
Klaus
On 10/12/07 11:21 AM, "Kilian CAVALOTTI" <kil...@stanford.edu>did etch on stone tablets:
Andreas Dilger
> Neither of these is something that can be easily accomplished from
> /etc/fstab, I would have to implement the failover functionality described
> in the link you sent, correct?
Correct. If you are using failover then you also need to have the HA
software manage the filesystem mounting. With 1.6 this is relatively
easy, as filesystem failover is a fairly standard configuration (ala
NFS server failover also).
> On 10/12/07 11:21 AM, "Kilian CAVALOTTI" <kil...@stanford.edu>did etch on
> stone tablets:
> > On Friday 12 October 2007 11:15:16 am Klaus Steden wrote:
> >> > Heheheh, I figured.
> >> >
> >> > What about mounting OST targets from multiple hosts, i.e. OST0000 and
> >> > OST0001 on both oss-0-0 and oss-0-1? Just as bad?
> >
> > Yep, for the same reasons.
> >
> > If you want redundancy for your OSTs, you can have a kind of
> > active/active setup where oss-0-0 would be active for OST0000 and
> > passive for OST0001, and oss-0-1 would be passive for OST0000 and
> > active for OST0001.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
Klaus Steden
Hello again,
Another (hopefully) quick question:
When designing a cluster with high availability using Lustre ... is there a recommended location for the MGS? Are there advantages to housing the MGS information on the same volume as metadata?
thanks,
Klaus
Jeff Blasius
Thank you for this information. Can or someone else comment on what it
means to upgrade? I assume it's not enough to simply build the
software or update via rpm. After updating, you should go through the
process of having all of the OSS's join the MGS/MDS and verify a
successful client connection?
Just for my curiosity, is there a standard process the server
components undergo during the upgrade?
Thank You,
jeff
On 10/12/07, Jody McIntyre <scj...@clusterfs.com> wrote:
UNIX Systems Administrator, Linux Systems Design & Support (LSDS)
Jody McIntyre
On Fri, Oct 12, 2007 at 12:27:23PM -0600, Sheila Barthel wrote:
> Not a problem to add a section in the Lustre manual re: supported upgrades.
> However, as we do not currently sync Lustre releases with releases of the
> Lustre manual, this wouldn't help us to address the issue that Jeff poses.
I don't think we need to list specific versions or anything. Just
include the general policies I have outlined below as well as some
examples - that way the information won't need to be revised regularly.
Cheers,
Jody
Jody McIntyre
On Sun, Oct 14, 2007 at 11:31:45PM -0400, Jeff Blasius wrote:
> Thank you for this information. Can or someone else comment on what it
> means to upgrade? I assume it's not enough to simply build the
> software or update via rpm. After updating, you should go through the
> process of having all of the OSS's join the MGS/MDS and verify a
> successful client connection?
This part is in the manual :) See:
http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-13-1.html
> Just for my curiosity, is there a standard process the server
> components undergo during the upgrade?
I don't understand what you mean here. Can you explain your question?
Cheers,
Jody
--
Jerome, Ron
> > Having said that, I believe 1.6.0 and 1.6.3 is actually something
that
> > will work, but I'm not 100% certain of this, so I'll allow others to
> > correct me.
... as I would like to do just this, go from 1.6.0.1 to 1.6.3 and would
rather not have to go through the intermediate versions if I don't have
to.
Thanks in advance.
Ron Jerome
National Research Council Canada
Andreas Dilger
> Could somebody (Andreas maybe :) give a definitive answer on this...
>
> > > Having said that, I believe 1.6.0 and 1.6.3 is actually something
> that
> > > will work, but I'm not 100% certain of this, so I'll allow others to
> > > correct me.
>
> ... as I would like to do just this, go from 1.6.0.1 to 1.6.3 and would
> rather not have to go through the intermediate versions if I don't have
> to.
I'm not aware of any incompatibilities between 1.6.0 and 1.6.3, and we
have a good mechanism to handle protocol negotiation when we add features
to Lustre (since 1.4.2). That said, we can't possibly do an all-to-all
test matrix, so we only test against the previous minor version (1.6.3
against 1.6.2) and the upgrade from the most recent major version (1.4.12).
There have been no changes to the on-disk format since 1.4 days (excluding
the config logs) that I'm aware of.
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.
_______________________________________________
Jerome, Ron
From: lustre-disc...@clusterfs.com on behalf of Andreas Dilger
Sent: Thu 10/18/2007 5:56 PM
To: Jerome, Ron
Cliff White
> Thanks Andreas,
>
> So I guess that means I can give it a try and hope for the best :-) or would I be wiser to go step by step given this is a production system serving 32 TB?
>
> Ron.
>
As Andreas said, we do not test more than one minor version, but there
are no changes in any protocol, configuration, or on-disk data.
We try very hard to make great big announcements/release notes when we
_do make major changes.
cliffw
> ------------------------------------------------------------------------
Jerome, Ron
From: lustre-disc...@clusterfs.com on behalf of Jerome, Ron
Sent: Thu 10/18/2007 9:33 PM
To: lustre-...@clusterfs.com
Nathan Rutman
Well, let me rephrase: there is no reason I can think of why it wouldn't
work. But we don't generally test interoperability -- the only regular
testing we do is with 1.4.x to the latest release. Let me also say that
we almost never change on-disk formats, so if you're upgrading your
entire cluster from 1.6.0.1 to 1.6.3 then there is in fact no
"interoperability" issue at all; it's all 1.6.3. Indeed, it is just
updating the RPMs. You don't have to --writeconf or anything like that.