[Rocks-Discuss] IP address change on Frontend
Robert Verstandig
What is the procedure to change the public IP address on the frontend
that doesn't involve a full reinstall? I am moving a cluster to a
different network..
Cheers
Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20100617/e5dfd4b0/attachment.html
Anthony Power
You cannot.
You need to reinstall. There is no way to change the public interface on a
Rocks cluster without completely reinstalling the headnode.
Its pretty much my only complaint about rocks, ... besides that its awesome
:)
Anthony Power
Sr Systems Engineer
Toll Free : (877) 870-AHPC (2472)
Phone : (858) 268-9600
Fax : (858) 268-9605
anthon...@advancedhpc.com
Robert Verstandig
Here we go again...
Marc Beaumont
>
> You cannot.
>
> You need to reinstall. There is no way to change the public interface on a
> Rocks cluster without completely reinstalling the headnode.
I don't think that's true.
I've changed the IP of my public interface many times.
Just run system-config-network and change the IP of the public interface.
Works for me every time.
Marc Beaumont
Senior IT Support Engineer
Aircraft Research Association
**********************************************************************
This email contains information that is private and confidential and is intended only for the addressee.
If you are not the intended recipient please delete it and notify us immediately by e-mailing the sender.
Note: All email sent to or from this address may be accessed by someone other than the recipient, for
system management and security reasons.
Aircraft Research Association Ltd. Registered in England, Registration No 503668 Registered Office:
Manton Lane, Bedford MK41 7PF England VAT No GB 196351245
**********************************************************************
Edsall, William (WJ)
attributes for the public interface.
Do a 'rocks list attr' and change the public kickstart variables to
match your new variables. Probably need to do a rocks sync config at
that point and maybe rebuild your distro?
-----Original Message-----
From: npaci-rocks-dis...@sdsc.edu
[mailto:npaci-rocks-dis...@sdsc.edu] On Behalf Of Marc
Beaumont
Sent: Thursday, June 17, 2010 10:12 AM
To: 'Discussion of Rocks Clusters'
Marc Beaumont
the public interface.
The only thing I use it for is ssh login and copying data to it so why
worry. The nodes don't care what the public interface is.
Marc Beaumont
Senior IT Support Engineer
Aircraft Research Association
> -----Original Message-----
> From: npaci-rocks-dis...@sdsc.edu [mailto:npaci-rocks-
Anthony Power
changing the public interface in that manner?
Because every time ive tried it makes parts of rocks just completely
nonfunctional.
What version are you running as well? ... 5.1 and earlier wasn't that bad to
change the public interface, you could tweak a few things and get it to
work, but with 5.3 its *impossible*
Anthony Power
Sr Systems Engineer
Toll Free : (877) 870-AHPC (2472)
Phone : (858) 268-9600
Fax : (858) 268-9605
anthon...@advancedhpc.com
Anthony Power
worry. The nodes don't care what the public interface is."
This statement is actually false if you are using a queuing system, i.e
Torque's communication from the nodes is done via the public address / FQDN
of the headnode.
If the public address doesn't resolve to the FQDN, and the ip address is
changed from what Torque is used to, your queuing system will fail to work.
But once again like I had stated before, .. 5.1 and previous editions of
Rocks, its not difficult to change the public interface, so maybe I jumped
the gun a bit saying it was "impossible" , it all depends on what version
you are running.
But install 5.3, and try and change the public interface, .. your cluster
will crumble, and the rocks command to change the attribute for the public
interface does *not* work , so you cannot update the database unless you do
it all by hand via the Rocks database...
And most anyone on this list will tell you, if you need to change your
public interface, you might as well reinstall, ...
Anthony Power
Sr Systems Engineer
Toll Free : (877) 870-AHPC (2472)
Phone : (858) 268-9600
Fax : (858) 268-9605
anthon...@advancedhpc.com
Robert Verstandig
I thought I would investigate this further and I did manage to change
the public interface ip address and get everything running again (on 5.3
BTW). It took a few attempts to get the process right but I eventually
got everything working. I don't use the Torque roll but a manual
installation as the software we run requires Torque installed in its
default locations (/usr/local/bin & sbin). This also all works fine.
I basically had to remove all the networks, machines and interfaces from
the database via the rocks command line and recreate them. I didn't have
a problem changing the public interface. The rocks commands worked as
specified.
The only real issue I had was getting dhcp operational again but after a
little perseverance that is now also working. I've reinstalled the nodes
and everything is running which probably saved me a couple of days
reinstalling the frontend (we have lots of specialist software..._).
I did notice that during this process if rocks doesn't recognise the new
frontend name it goes and truncates the /etc/hosts and /etc/dhcpd.conf
files - not very nice. It actually writes the error message into the
files. Maybe there should be a more graceful way of handling the 'host
is not in rocks database' type error...
Cheers
Rob
Joe Kaiser
VMWare. But Anthony used it and promptly hosed his system.
There is at least one mysql call you have to make in order to get it right
as well because there's no way to update the "nodes" table via rocks
commands.
It's harder to change this on 5.3 than on 5.1 and I haven't even looked at
the private interface which is actually harder to do having done this for
5.1
The main reason I see this is as something that's needed, i.e. the ability
to change IP hostname information for public and private, is because people
move their clusters, and vendors create a cluster and provision it and then
frequently need to change at least the public information upon delivery.
This should be easy todo and it just gets harder.
Thanks,
Joe
-------------- next part --------------
An HTML attachment was scrubbed...
Bart Brashers
Would it work to use that ifcfg-eth1:1 trick? Like this:
# cat /etc/sysconfig/network-scripts/ifcfg-eth1:1
DEVICE=eth1:1
IPADDR=12.34.56.78
NETMASK=255.255.255.0
BOOTPROTO=static
ONBOOT=yes
Where ifcfg-eth1 contains the old IP address?
I installed a cluster using the eventual IP, and configured ifcfg-eth1:1
to use a local IP so I could work on it here in my office. The
procedure seemed to work. I can't remember right now if I had to mess
with the routing tables or if it 'just worked' after a reboot.
I can imagine just setting up ifcfg-eth1:1 with the new IP address, and
leaving it like that until the next scheduled upgrade.
Bart
This message contains information that may be confidential, privileged or otherwise protected by law from disclosure. It is intended for the exclusive use of the Addressee(s). Unless you are the addressee or authorized agent of the addressee, you may not review, copy, distribute or disclose to anyone the message or any information contained within. If you have received this message in error, please contact the sender by electronic reply to em...@environcorp.com and immediately delete all copies of the message.
Joe Kaiser
On Fri, Jun 18, 2010 at 10:16 AM, Bart Brashers
<bbra...@environcorp.com>wrote:
Anthony Power
You'd just need to change the hosts file to point the FQDN to the new ip, ..
and just make sure if you run rocks sync config at any time you go back and
fix it. As well as would you want the FQDN to resolve externally to the new
or old IP.. Im assuming the new, .. would this cause issues?
My only concern would be if you moved it to a new network, .. its going to
look for its original IP for its FQDN, ... because that's whats in the
database
If the rocks guys could provide us with every location within the Rocks
database the public ip / netmask / gateway / route / DNS is located, ... I
don't think it would be that difficult to put together a bash or perl script
to replace all that information for you, . instead of running a bunch of
rocks commands, maybe manually scripting something just change all the
locations in the database.
It would need to be done on all the nodes too, because the nodes wouldn't
reimage cause the kickstart file would contradict the attributes on the
headnode, ... so it would probably barf all over the place on a
cluster-kickstart, or just image the nodes with the old information ... but
you could just zero out all the nodes, remove them all the from the database
and then reimage, .. but once again that's a lot of time to spend just to
move the cluster from 1 ip to another.
I'm a vendor, I put rocks on all our clusters, I build the cluster to the
specs of the customers network , .. and "emulate" their network here, ... so
I can verify functionality, .. it's a pain, but its all I can do, .. if I
build the cluster with my network environment once its delivered .. it
basically needs to be reinstalled.
Before 5.3 I had no problem changing the public interface to get things
working on cluster deliveries, .. but now with 5.3, I *HAVE* to build the
cluster EXACTLY to the network its being installed on, .. and if I don't
emulate their network here, .. then I have issues with testing Torque,
adding users, rocks sync users, .. etc etc etc etc, .... Rocks has become
even MORE dependent on its public interface and its ability to resolve its
FQDN, so I think adding the functionality to properly change that would be a
HUGE value add to the Rocks software package.
I know my perl, bash and regex pretty well, so im down to help put something
together, ... id just need to know exactly WHAT needs to be changed on the
headnode and the nodes, for a smooth public ip transition.
Maybe even just doing a mysqldump on the database for the headnode and on
one of the nodes and doing some global search and replace might be a quick
and dirty way of doing it, and then just throw that database into the nodes
and headnode
Bart Brashers
> Wow, .. simple but yet sounds like it would work well.
>
> You'd just need to change the hosts file to point the FQDN to the new
ip, ..
> and just make sure if you run rocks sync config at any time you go
back and
> fix it. As well as would you want the FQDN to resolve externally to
the new
> or old IP.. Im assuming the new, .. would this cause issues?
>
> My only concern would be if you moved it to a new network, .. its
going to
> look for its original IP for its FQDN, ... because that's whats in the
> database
Yes, this would cause issues. So you DO NOT want to change to the new
IP in /etc/hosts. That's the whole point here.
For routing, a machine never needs to know its own IP address, just the
addresses of other machines. Inside the cluster (local network context)
it doesn't know that it has moved to a new location.
So you should enter the new IP in the local (non-rocks) DNS server.
That way all the other boxes can resolve and connect to the frontend --
within their network context. You can still enter the non-rocks DNS
server in the forwarders section of /etc/named.conf.
Rocks guys, is /etc/named.conf regenerated automatically during a "rocks
sync config"? Can you supply the correct rocks command line to change
the DNS forwarders?
> If the rocks guys could provide us with every location within the
Rocks
> database the public ip / netmask / gateway / route / DNS is located,
... I
> don't think it would be that difficult to put together a bash or perl
script
> to replace all that information for you, . instead of running a bunch
of
> rocks commands, maybe manually scripting something just change all the
> locations in the database.
Ah, if only. I believe that _reliably_ finding all the instances, and
finding all the dependencies (what has to be re-run) is a major task.
That's why they haven't done it yet! If it was easy, it would be
already done.
The idea here is that it's not supposed to be that hard to re-install a
frontend. If you make sure you do all your customizations via scripts,
and then include those scripts in the restore roll (or keep them on a
permanent partition) then it should be a half-day (at most) amount of
work.
All your local software should be installed on a permanent partition too
(like /share/apps). It also seems like a really good idea to me to have
/usr/local be on a permanent partition.
How long does it take you to re-install a frontend, without upgrading?
What percentage of your total labor does that amount to? Do you feel
it's not fair to charge your customers for this part of the product?
Bart
Joe Kaiser
I've actually done it for 5.1. I believe I have done it for 5.3 but I have
yet to have time to verify. Again, you're right about the _reliably_ part.
The problem is that in addition to all the places, the changes have to made
at the right time in the right order and synced at the correct moment or it
all goes to hell.
And it doesn't even include work outside of the standard rolls.
Thanks,
Joe
> The idea here is that it's not supposed to be that hard to re-install a
> frontend. If you make sure you do all your customizations via scripts,
> and then include those scripts in the restore roll (or keep them on a
> permanent partition) then it should be a half-day (at most) amount of
> work.
>
>
I agree. Certainty is better than convenience.
-------------- next part --------------
An HTML attachment was scrubbed...
Steve Jones
Joe's right though, if you don't get it in all the correct locations, all does go to hell and you're back to editing again.
I do recall a little frontend IP change script (that worked quite well) someone had at some point... ;-/
I have to say for the rocks group it's near impossible for them to create a script for *anyone* to change it as they have no idea of what is taking place with the distribution once it's downloaded. It's possible in a controlled environment though.
My two cents.
Steve
Anthony Power
What version of Rocks are you running?
And how would it be impossible for a universal script / binary to be created to make these changes? Obviously "its trivial, one only needs to know how to do simple find/replaces, a little sed/awking, vi and change a few sports in the database and you are like the initial installation"
If its that simple a script can be written to make it simpler, ... isn’t that why we write them in the first place?
Besides adding different rolls, Rocks environments are pretty similar, unless you go *way* custom, and even then its usually just application packages, partitioning / appliances.
The thing is there is no documentation on WHERE all this information is in the database for one, ... so unless you want to spend all day digging through the thing, finding all the tables and fields etc etc etc .. unless you have over 50 nodes, id rather just reinstall the damn thing.
Minus install time, I can roll out a cluster, have it up and running and processing jobs in a few hours.
I work with TONS of people that swear by Rocks, .. and it really sucks when this issue comes up and I tell them "well, our best bet is to reinstall it" ... I actually have a complete write up on how to rebuild a cluster step by step just for this case, ... whether its Torque / SGE / GigE / OFED / Qlogic OFED / etc /
All we are doing is changing the public interface, .. a core component of the Rocks system, .. well of any cluster really, ... and anyone who has been an admin for any group of systems knows there are times when you have to move machine(s) to new ip ranges, ... finding a way to simplify this in Rocks instead of playing surgeon with the rocks core database and config files would be very beneficial to everyone, especially hardware vendors like myself who preload Rocks on clusters as a value add for customers, .. I deal with a lot of research / edu / etc who have no idea how to really build / maintain a clustered environment, so, using Rocks, which is an amazing package makes their lives A LOT easier.
Because if you slip up while playing surgeon on your cluster, ... you're going to have to reinstall anyways :P ... and its not that easy to remote into a cluster, ... to change its public interface :) and walking through someone who knows how to run jobs, .. but not a cluster, on how to rebuild it can be somewhat frustrating :)
Maybe even something as simple as creating a script that builds a NEW database (like the one that’s created at install) ... but with the new IP information, ... throws out some changes in some config scripts... dumps the running database, ups the new one and then pushes a cluster-kickstart out to all the nodes so they are reimaged with this information.
It could even query the running database so that it pulls all the node information (hostnames, ip's, mac's, etc) ... and even if you had a super custom extend-computes file with a crazy node config, .. nothing would really change.
But the public ip.
Anthony Power
Sr Systems Engineer
Toll Free : (877) 870-AHPC (2472)
Phone : (858) 268-9600
Fax : (858) 268-9605
anthon...@advancedhpc.com
-----Original Message-----
From: npaci-rocks-dis...@sdsc.edu [mailto:npaci-rocks-dis...@sdsc.edu] On Behalf Of Steve Jones
Sent: Friday, June 18, 2010 2:38 PM
To: Discussion of Rocks Clusters
Subject: Re: [Rocks-Discuss] IP address change on Frontend
Steve Jones
> What version of Rocks are you running?
Versions since 2.x, but I don't think you've been around since then, otherwise you wouldn't be asking me this question.
> And how would it be impossible for a universal script / binary to be
> created to make these changes? Obviously "its trivial, one only needs
> to know how to do simple find/replaces, a little sed/awking, vi and
> change a few sports in the database and you are like the initial
> installation"
>
> If its that simple a script can be written to make it simpler, ...
> isn’t that why we write them in the first place?
Please, go ahead and script it up, show your support as a commercial vendor who gives back to the community. I already develop a high number of rolls for the community and have for years.
> Besides adding different rolls, Rocks environments are pretty similar,
> unless you go *way* custom, and even then its usually just application
> packages, partitioning / appliances.
>
> The thing is there is no documentation on WHERE all this information
> is in the database for one, ... so unless you want to spend all day
> digging through the thing, finding all the tables and fields etc etc
> etc .. unless you have over 50 nodes, id rather just reinstall the
> damn thing.
I'd have to guess you're not managing systems for years, then have to move it or change networks. That's an easy route to take as a vendor building systems, but not practical in the real world.
blah blah blah. Write the script, give back to the community.
Steve
Jacques Menu TvTMail
Début du message réexpédié :
> De : Steve Jones <steve...@stanford.edu>
> Date : 18 juin 2010 23:37:47 HAEC
> À : Discussion of Rocks Clusters <npaci-rocks...@sdsc.edu>
> Objet : Rép : [Rocks-Discuss] IP address change on Frontend
> Répondre à : Discussion of Rocks Clusters <npaci-rocks...@sdsc.edu>
Jacques Menu TvTMail
Début du message réexpédié :
> De : "Anthony Power" <anthon...@advancedhpc.com>
> Date : 19 juin 2010 01:26:20 HAEC
> À : "'Discussion of Rocks Clusters'" <npaci-rocks...@sdsc.edu>
> Objet : Rép : [Rocks-Discuss] IP address change on Frontend
> Répondre à : Discussion of Rocks Clusters <npaci-rocks...@sdsc.edu>
>
Jacques Menu TvTMail
Début du message réexpédié :
> De : Steve Jones <steve...@stanford.edu>
> Date : 19 juin 2010 03:15:47 HAEC
> À : Discussion of Rocks Clusters <npaci-rocks...@sdsc.edu>
> Objet : Rép : [Rocks-Discuss] IP address change on Frontend
> Répondre à : Discussion of Rocks Clusters <npaci-rocks...@sdsc.edu>
>
-------------- next part --------------
An HTML attachment was scrubbed...
Anthony Power
... I've been around for 6-7 years, ... so ya you've using rocks for longer than I have, .. this isn’t a pissing contest, the reason I asked was because I was curious to see if you were even RUNNING 5.3 so maybe you could actually SEE what im talking about.
" Please, go ahead and script it up, show your support as a commercial vendor who gives back to the community. I already develop a high number of rolls for the community and have for years."
I am 100% down to do this, .. but I would need some information from the Rocks group on exactly what needs to be changed where, and the order these changes should be made in, ... im asking for SUGGESTIONS from the group and a little help, .. I never said "YOU GUYS DO IT THIS SUCKS" ... im saying it would be a valuable asset to the Rocks suite.
" I'd have to guess you're not managing systems for years, then have to move it or change networks. That's an easy route to take as a vendor building systems, but not practical in the real world."
How is it not practical? I just went through this with 3 different customers who had new data centers built and had to move their clusters and change their ip information, ... shutting it down, rolling it to a new room, and plugging it into a new network is pretty much what I do when I show up on site to install the thing in the first place. Moving it should be no different.
* blah blah blah. Write the script, give back to the community.*
Actually I have given back to the community plenty of times, .. I know who you are, and I know all about your rocks.stanford.com project, I've worked with Scott on a few things, .. he actually helped me out with an openmpi issue I had, .. and I was the one that actually recompiled your openmpi roll with infiniband support it in using the 5.1 OFED roll, so that openmpi would actually work over IB and play nice with Torque .. I give back to this community as much as I can, im in the process of building an OFED roll right now that will build its openmpi with Torque or SGE support since that’s another common issue with the current rolls out there.
Anthony Power
Sr Systems Engineer
Toll Free : (877) 870-AHPC (2472)
Phone : (858) 268-9600
Fax : (858) 268-9605
anthon...@advancedhpc.com
-----Original Message-----
From: npaci-rocks-dis...@sdsc.edu [mailto:npaci-rocks-dis...@sdsc.edu] On Behalf Of Steve Jones
Sent: Friday, June 18, 2010 6:16 PM
To: Discussion of Rocks Clusters
Subject: Re: [Rocks-Discuss] IP address change on Frontend
Steve Jones
The issue still exists, it's near impossible to create a script for the unknown. It's easy for a vendor to create a script for a controlled environment, say you build and ship clusters to a client with known rolls installed, they then run the script to change the network information. It's near impossible for anyone to write a script to change the IP on any system in the wild as people have custom configurations, custom rolls and site modifications that you'd have to account for.
Let us know when you have something to test.
Steve
Mason J. Katz
each other, and keep this at the correct tone.
Changing the IP of the frontend is a long discussed topic in Rocks.
Further, we are always quick to point out that this is not a Rocks issue,
as this is difficult in any parallel sever deployment. We've seen several
good solutions from the community but none have been general purpose, hence
no support is build into Rocks.
The good news is, you no longer need to know anything about the SQL schema
of Rocks, rather the "rocks dump" command will split out everything you
need. You can then edit the output, and re-run the script to modify the
database. This followed by a "rocks sync config" will re-write most of the
Rocks controlled files on the frontend.
The bad news is, this is going to miss several things like apache and the
queuing system. Eventually all Rocks modified files will be
re-written/re-modified by "rocks sync config" but this is still a few
releases away. Hence, you'll need to do some site-specific sed/awk to clean
things finish the job. If you have a static set of Rolls, you should be
able to create something that will always work for all of your clusters, but
once someone wants to add another Roll that you haven't seen before all bets
are off.
mason katz
+1.240.724.6825
On Mon, Jun 21, 2010 at 11:09 AM, Anthony Power <
anthon...@advancedhpc.com> wrote:
> " Versions since 2.x, but I don't think you've been around since then,
> otherwise you wouldn't be asking me this question."
>
> ... I've been around for 6-7 years, ... so ya you've using rocks for longer
> than I have, .. this isn’t a pissing contest, the reason I asked was because
> I was curious to see if you were even RUNNING 5.3 so maybe you could
> actually SEE what im talking about.
>
> " Please, go ahead and script it up, show your support as a commercial
> vendor who gives back to the community. I already develop a high number of
> rolls for the community and have for years."
>
> I am 100% down to do this, .. but I would need some information from the
> Rocks group on exactly what needs to be changed where, and the order these
> changes should be made in, ... im asking for SUGGESTIONS from the group and
> a little help, .. I never said "YOU GUYS DO IT THIS SUCKS" ... im saying it
> would be a valuable asset to the Rocks suite.
>
> " I'd have to guess you're not managing systems for years, then have to
> move it or change networks. That's an easy route to take as a vendor
> building systems, but not practical in the real world."
>
-------------- next part --------------
An HTML attachment was scrubbed...
Anthony Power
Our profit is made from our hardware sales, we do not charge for O/S installations, I don't even charge professional services to customers to help get their code running on a new system, so your assumption in differences is far from correct. I don't make commissions off cluster sales, ... and I don’t get any pretty bonuses for putting in all the extra time I do working on side projects like this, ... so honestly I'd like to think im the same boat as you, .. im just not EDU .. im a Vendor
Minus this whole Rocks thing ive been apart of the Linux community for over 10 years, and I've spent countless hours helping people on other mailing lists im on... including this one.
" I'd certainly hope I have 5.3 since I develop rolls for it."
Well, when you get a chance, ... try your whole sed / awk , database change, *it’s a cake walk* public ip change, .. and let me know how it works for you :)
I'm done with this conversation, .... I feel like im getting more attitude than anything.
Kewei Li
Kewei
> -----Original Message-----
> From: npaci-rocks-dis...@sdsc.edu [mailto:npaci-rocks-discussion-
> bou...@sdsc.edu] On Behalf Of Anthony Power
> Sent: Monday, June 21, 2010 3:02 PM
> To: 'Discussion of Rocks Clusters'
> Subject: Re: [Rocks-Discuss] IP address change on Frontend
>
Joe Kaiser
Thanks,
Joe
-------------- next part --------------
An HTML attachment was scrubbed...