[Rocks-Discuss] restore frontend ?

554 views
Skip to first unread message

Ben Ransom

unread,
May 16, 2011, 12:44:39 PM5/16/11
to npaci-rocks...@sdsc.edu
We have a Rocks 5.2 cluster that unfortunately suffered file corruption on the frontend root partition. I can get to the system with a rocks 5.4 dvd, via build rescue, and see the root partition in read-only. All of my /export partition is fine.

I'm wondering about a recovery possiblity whereby I essentially use the method prescribed for earlier version upgrades ( http://www.rocksclusters.org/roll-documentation/base/5.0/upgrade-frontend.html ) . In other words, I believe I can still do:

# cd /export/site-roll/rocks/src/roll/restore
# make roll

Would this allow me to boot with a rocks 5.2 dvd and insert the ISO made from the above command to get my system back with node info preserved? (Note, I don't see a way to use follow-on "upgrade" methods, as they prescribe installing an rpm — which I can't do because of the corrupt root partitions.)

Thanks in adv,

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20110516/6ff8def7/attachment.html

Bart Brashers

unread,
May 16, 2011, 1:06:00 PM5/16/11
to Discussion of Rocks Clusters
If I were in your shoes, I would go buy a new disk and install 5.4 fresh
(no restore roll). Then mount the old disk any way you can (e.g. a USB
enclosure, or a spare SATA port, whatever). Then copy over your data
and users by hand. You won't have the stuff in the Rocks database, but
you can probably pretty quickly re-create that anyway by re-inserting
your nodes after you've imported your old extend-compute.xml and checked
it for 5.4 compatibility.

Recall that skipping a version of Rocks is not supported when using a
restore roll. It may work, but it's not "guaranteed" to work
(guaranteed in the sense of all free software is "use at your own
risk").

And of course once you get up and running and happy again, make a
restore roll and save it on another machine. Repeat periodically, like
quarterly or bi-annually.

My $0.02...

Bart Brashers

-----Original Message-----
From: npaci-rocks-dis...@sdsc.edu
[mailto:npaci-rocks-dis...@sdsc.edu] On Behalf Of Ben Ransom
Sent: Monday, May 16, 2011 9:45 AM
To: npaci-rocks...@sdsc.edu
Subject: [Rocks-Discuss] restore frontend ?

We have a Rocks 5.2 cluster that unfortunately suffered file corruption
on the frontend root partition. I can get to the system with a rocks
5.4 dvd, via build rescue, and see the root partition in read-only. All
of my /export partition is fine.

I'm wondering about a recovery possiblity whereby I essentially use the
method prescribed for earlier version upgrades (
http://www.rocksclusters.org/roll-documentation/base/5.0/upgrade-fronten
d.html ) . In other words, I believe I can still do:

# cd /export/site-roll/rocks/src/roll/restore
# make roll

Would this allow me to boot with a rocks 5.2 dvd and insert the ISO made
from the above command to get my system back with node info preserved?
(Note, I don't see a way to use follow-on "upgrade" methods, as they

prescribe installing an rpm - which I can't do because of the corrupt
root partitions.)

Thanks in adv,

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/2011
0516/6ff8def7/attachment.html


This message contains information that may be confidential, privileged or otherwise protected by law from disclosure. It is intended for the exclusive use of the Addressee(s). Unless you are the addressee or authorized agent of the addressee, you may not review, copy, distribute or disclose to anyone the message or any information contained within. If you have received this message in error, please contact the sender by electronic reply to em...@environcorp.com and immediately delete all copies of the message.

Philip Papadopoulos

unread,
May 16, 2011, 1:09:10 PM5/16/11
to Discussion of Rocks Clusters
Ben,

I'd try something else --
since you can get to your hard disk, see if you can copy the file
/var/db/mysql-backup-cluster
(also copy the RCS directory underneath it).
(and copy /root/.ssh/* ... and /etc/ <--- especially for passwords,
auto.home, etc.... )

mysql-backup-cluster is a mysql dump of your cluster database -- it is in
the schema used for 5.2 and is not compatible
with 5.3/5.4.

First thing I would do is rebuild your cluster frontend with 5.2.
Then restore the contents of /root/.ssh to your newly rebuilt frontend
To get the users passwords and mounts back, you need to -merge- the copied
versions of /etc/passwd, /etc/group, /etc/shadow, /etc/gshadow,
/etc/auto.home

and finally,

# /opt/rocks/bin/mysql -u root -p < mysql-backup-cluster
# rocks sync config

At this point, you should have your DB back and you should be able to get to
your 5.2 compute nodes. Hopefully,
you will have a working 5.2 cluster, with users defined, etc.

We don't have an automatic process for going directly from 5.2 --> 5.4 so a
restore roll won't really do the job
for you. (you could move 5.2 -> 5.3 and then 5.3 --> 5.4, via restore
rolls).


-P


On Mon, May 16, 2011 at 9:44 AM, Ben Ransom <bra...@ucdavis.edu> wrote:

> We have a Rocks 5.2 cluster that unfortunately suffered file corruption on
> the frontend root partition. I can get to the system with a rocks 5.4 dvd,
> via build rescue, and see the root partition in read-only. All of my
> /export partition is fine.
>
> I'm wondering about a recovery possiblity whereby I essentially use the
> method prescribed for earlier version upgrades (

> http://www.rocksclusters.org/roll-documentation/base/5.0/upgrade-frontend.html) . In other words, I believe I can still do:


>
> # cd /export/site-roll/rocks/src/roll/restore
> # make roll
>
> Would this allow me to boot with a rocks 5.2 dvd and insert the ISO made
> from the above command to get my system back with node info preserved?
> (Note, I don't see a way to use follow-on "upgrade" methods, as they
> prescribe installing an rpm — which I can't do because of the corrupt root
> partitions.)
>
> Thanks in adv,
>
> -Ben
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20110516/6ff8def7/attachment.html
>
>


--
Philip Papadopoulos, PhD
University of California, San Diego
858-822-3628 (Ofc)
619-331-2990 (Fax)


-------------- next part --------------
An HTML attachment was scrubbed...

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20110516/3a8c818e/attachment.html

Ben Ransom

unread,
May 16, 2011, 1:27:26 PM5/16/11
to Discussion of Rocks Clusters
I do have /var/db/mysql-backup-cluster with a timestamp just prior to
corruption ...so yes, will try a combo of this and Bart's suggestions.

Thanks!
-Ben

On 5/16/11 10:09 AM, "Philip Papadopoulos" <philip.pa...@gmail.com>
wrote:

Reply all
Reply to author
Forward
0 new messages