[Rocks-Discuss] NAS with XFS in Rocks 5.1
Paulo Laranjeira da Cunha Lage
I need some guidelines.
I have just received a 128 nodes Opteron Quadricore cluster with a NAS with 16 Tb.
The NAS server came installed with CentOS 5.2 and a 13Tb hardware (3-ware) RAID6 XFS partition.
The frontend has already been installed with the basic rolls in the DVD + mellanox + PVFS2 + PGI.
What is the best way to integrate the NAS into the Rocks cluster?
- should I use 'insert-ethers' and choose NAS?
- should I manually configurate the automount to mount /home/user.. and /share/app into the exported XFS partition?
What is the best path to proceed?
Thanks,
Paulo
------------------------------------
Prof. Paulo Laranjeira da Cunha Lage
Termofluidodinamica
Programa de Engenharia Quimica
COPPE/UFRJ
http://www.peq.coppe.ufrj.br
http://www.peq.coppe.ufrj.br/Areas/tfd/
++++++++++++++++++++++++++++++++++++++++++++
PEQ/COPPE renova o nivel 7, maximo na CAPES:
45 anos de excelencia no ensino e pesquisa em Engenharia Química
45 years of excellence in teaching and research in Chemical Engineering
++++++++++++++++++++++++++++++++++++++++++++
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20090113/5c095fea/attachment.html
Bart Brashers
> I have just received a 128 nodes Opteron Quadricore cluster with a NAS with 16 Tb.
> The NAS server came installed with CentOS 5.2 and a 13Tb hardware (3-ware) RAID6 XFS partition.
>
> The frontend has already been installed with the basic rolls in the DVD + mellanox + PVFS2 + PGI.
> What is the best way to integrate the NAS into the Rocks cluster?
> - should I use 'insert-ethers' and choose NAS?
> - should I manually configurate the automount to mount /home/user.. and /share/app into the
> exported XFS partition?
> What is the best path to proceed?
I have a NAS with xfs in my Rocks 5.0 system. I chose to integrate it into Rocks, thinking that the benefits of the 411 system of keeping users/password synchronized was worth the effort of inserting it into the cluster. You could try to install just the 411 system, but it's probably just as easy if not easier to insert it into the cluster.
I'll bet that it's not running the standard CentOS 5.2 kernel, but rather the centos.plus kernel. Log on and ask: "uname -a". If it's the centos.plus kernel, then you need to download the kernel RPM and the xfs module RPM and put them in /export/rocks/install/contrib/5.1/x86_64/RPMS. You'll also want the dmapi, xfsdump and xfsprogs RPMs. Here's the list for my Rocks 5.0 system (your version numbers will be different).
dmapi-2.2.8-1.el5.centos.x86_64.rpm
kernel-2.6.18-53.1.14.el5.centos.plus.x86_64.rpm
kmod-xfs-0.4-1.2.6.18_53.1.14.el5.centos.plus.x86_64.rpm
xfsdump-2.2.46-1.el5.centos.x86_64.rpm
xfsprogs-2.9.4-1.el5.centos.x86_64.rpm
Then add these to your extend-nas.xml (follow the example in
http://www.rocksclusters.org/roll-documentation/base/5.1/customization-adding-packages.html).
Here's some snippets from my Rocks 5.0 extend-nas.xml (you can leave out the if-statements if you only have one NAS appliance):
-----------------------------------------------------------------
<!-- xfs filesystem support for NAS-0-2 -->
<eval mode="xml">
if [ "$Node_Hostname" == "nas-0-2" ]; then
echo "<package>kmod-xfs</package>"
echo "<package>dmapi</package>"
echo "<package>xfsdump</package>"
echo "<package>xfsprogs</package>"
echo "<package>kernel</package>"
fi
</eval>
<post>
if [ "<var name="Node_Hostname"/>" == "nas-0-2" ]; then
mkdir -p /state/partition1
<file name="/etc/fstab" mode="append">
/dev/sda1 /state/partition1 auto defaults 0 2
</file>
<file name="/etc/exports">
/state/partition1 10.0.0.0/255.0.0.0(rw,no_root_squash,async)
</file>
sed -i "s/id:5:initdefault/id:3:initdefault/" /etc/inittab
fi
</post>
-----------------------------------------------------------------
The last sed statement keeps the NAS from booting in X-windows mode.
Then do a
# cd /export/rocks/install
# rocks create distro
Then do a "insert-ethers" and pick 'NAS Appliance'. It will force you to do manual partitioning. Don't re-format the XFS partition, just leave it alone. Hopefully they included a separate hard drive for the OS (it's always a good idea to separate user data from your OS -- operating systems come and go, user data is forever). If not, see if there's any place inside you can add a hard drive. In fact, if they did include a separate hard drive, considering buying another similar drive to use with Rocks. If things go really wrong, you can always put the old HD back in and be up and running again.
Lastly, you'll have to edit your /etc/auto.home map. I also store /share/apps on a RAID array, so you'll want to edit /etc/auto.share too. After a chmod, you can edit those by hand, despite what it says in the header. You can try to change the HOME variable in /etc/default/useradd to
HOME=nas-0-0:/state/partition1/home
and see if it works for your next new user. It just might...
Bart
This message contains information that may be confidential, privileged or otherwise protected by law from disclosure. It is intended for the exclusive use of the Addressee(s). Unless you are the addressee or authorized agent of the addressee, you may not review, copy, distribute or disclose to anyone the message or any information contained within. If you have received this message in error, please contact the sender by electronic reply to em...@environcorp.com and immediately delete all copies of the message.
Henry Xu
You are familiar with writing extend.xml. How could you know to use <file> <eval> tabs? Are there any manual/guide on the format of extend.xml?
How many tabs supported and what's the syntax?
Henry
echo "<package>dmapi</package>"
echo "<package>xfsdump</package>"
echo "<package>xfsprogs</package>"
echo "<package>kernel</package>"
An HTML attachment was scrubbed...
Greg Bruno
> Bart,
>
> You are familiar with writing extend.xml. How could you know to use <file> <eval> tabs? Are there any manual/guide on the format of extend.xml?
>
> How many tabs supported and what's the syntax?
http://www.rocksclusters.org/rocks-documentation/reference-guide/4.3/kickstart-xml.html
- gb
Bart Brashers
Any particular reason why this section isn't in the 5.x series of docs?
Also note that this section does not cover if-then statements. As it
turns out, inside the <post> section, you can use bash-like structures.
That is, the default interpreter is bash. The docs don't spell this
out, they only give an example of using python as the interpreter. It
took a long time for me to figure that out, I guess I'm dim...
Paulo Laranjeira da Cunha Lage
I would like to thank Bart Brashers for his excellent hint, but I have one more question (see below) and a curious remark (see the P.S.).
Today I managed to install the NAS into Rocks 5.1 using the path he suggested. In order to register this for future use, there was necessary to install the rpms for XFS file support and the centos.plus kernel. The necessary packages are:
dmapi-2.2.8-1.el5.centos.x86_64.rpm
kernel-2.6.18-92.1.13.el5.centos.plus.x86_64.rpm
kmod-xfs-0.4-2.x86_64.rpm
xfsdump-2.2.46-1.el5.centos.x86_64.rpm
xfsprogs-2.9.4-1.el5.centos.x86_64.rpm
I followed exactly Bart's instructions, using the 2 internal HDs for the OS installation in software RAID 1 and leaving the XFS partition alone in the manual partition of the NAS. After installation, NAS booted seeing the XFS partition which was corrected exported.
One curious thing: the Rocks installation kernel identified the XFS partition as /dev/sda1 while for installed centos.plus kernel this is the partition /dev/sdc1 (each kernel have seen the HDs in different order). Thus in the first installation, the extend-nas.xml was configured using the /dev/sda1 in /etc/fstab and, of course, it did not work. Then, I changed the /etc/fstab line to use /etc/sdc1 and everything worked fine.
At this point I stopped with just the frontend and the NAS installed: no compute node installed no user added.
Then, I changed:
- HOME variable in /etc/default/useraddto
HOME=nas-0-0:/state/partition1/home
and created a home directory under /state/partition1 of nas-0-0
- do NOT edit /etc/auto.home map because it was empty
- edit /etc/auto.shareto:
apps nas-0-0.local:/state/partition1/&
and created a apps directory under /state/partition1 of nas-0-0
Then, I managed to easy create a file in the frontend in /share/apps that was correctly stored in nas-0-0.local:/state/partition1/apps/. This part works perfectly.
Finally, I have tried to add a user and get:
# useradd test
useradd: cannot create directory nas-0-0.local:/state/partition1/home/test
and the user was not completed created. I then deleted it.
It seems that nas-0-0.local:/state/partition1 is not being automounted by the frontend. I mounted it explicitly to verify if it is avaialble and it was. Then, I include the following line in the /etc/auto.home to:
test nas-0-0.local:/state/partition1/home/&
and tried again:
#useradd test
useradd: cannot create directory nas-0-0.local:/state/partition1/home/test
but with the same problem. If I changed back the HOME variable in /etc/default/useradd to /export/home, the user was created without any problem.
Do you have any idea of what the problem is and how to correct it?
Thanks,
Paulo
P.S. Curious remark: after installing the NAS, I installed the first compute node. Curiously, without any extend-compute.xml, the compute-0-0 was installed using the centos.plus kernel that was set to be installed in the extend-nas.xml. Then, I simply copy the skeleton.xml to the extend-compute.xml and reinstall the compute-0-0 from scratch. It still gets the centos.plus kernel. How can it be possible? It is getting an extra kernel package thta it was not informed to pick. Has anyone an explanation?
------------------------------------
Prof. Paulo Laranjeira da Cunha Lage
Termofluidodinamica
Programa de Engenharia Quimica
COPPE/UFRJ
http://www.peq.coppe.ufrj.br
http://www.peq.coppe.ufrj.br/Areas/tfd/
++++++++++++++++++++++++++++++++++++++++++++
PEQ/COPPE renova o nivel 7, maximo na CAPES:
45 anos de excelencia no ensino e pesquisa em Engenharia Química
45 years of excellence in teaching and research in Chemical Engineering
++++++++++++++++++++++++++++++++++++++++++++
---------- Original Message -----------
From: "Bart Brashers" <bbra...@Environcorp.com>
To: "Discussion of Rocks Clusters" <npaci-rocks...@sdsc.edu>
Sent: Tue, 13 Jan 2009 15:06:48 -0800
Subject: Re: [Rocks-Discuss] NAS with XFS in Rocks 5.1
------- End of Original Message -------
-------------- next part --------------
An HTML attachment was scrubbed...
Bart Brashers
> (see below) and a curious remark (see the P.S.).
>
> Today I managed to install the NAS into Rocks 5.1 using the path he suggested. In order to register
> this for future use, there was necessary to install the rpms for XFS file support and the
> centos.plus kernel. The necessary packages are:
>
> dmapi-2.2.8-1.el5.centos.x86_64.rpm
> kernel-2.6.18-92.1.13.el5.centos.plus.x86_64.rpm
> kmod-xfs-0.4-2.x86_64.rpm
> xfsdump-2.2.46-1.el5.centos.x86_64.rpm
> xfsprogs-2.9.4-1.el5.centos.x86_64.rpm
Good, these appear to be correct for the newer version of CentOS.
> I followed exactly Bart's instructions, using the 2 internal HDs for the OS installation in
> software RAID 1 and leaving the XFS partition alone in the manual partition of the NAS.
> After installation, NAS booted seeing the XFS partition which was corrected exported.
>
> One curious thing: the Rocks installation kernel identified the XFS partition as /dev/sda1
> while for installed centos.plus kernel this is the partition /dev/sdc1 (each kernel have seen
> the HDs in different order). Thus in the first installation, the extend-nas.xml was configured
> using the /dev/sda1 in /etc/fstab and, of course, it did not work. Then, I changed the
> /etc/fstab line to use /etc/sdc1 and everything worked fine.
My OS hard drive is an old ATA drive, so /dev/hda. Your two are most likely SATA drives, so got mapped to /dev/sda and /dev/sdb. So the RAID array was assigned /dev/sdc. Perfectly logical.
> At this point I stopped with just the frontend and the NAS installed: no compute node installed no
> user added.
> Then, I changed:
>
> - HOME variable in /etc/default/useraddto
> HOME=nas-0-0:/state/partition1/home
> and created a home directory under /state/partition1 of nas-0-0
This is where you went wrong, I think. Normally, you should not create the home directory before you run "useradd". It creates it for you, as /export/home/$USER. Then you should run "rocks sync users", which creates an entry in /etc/auto.home for $USER.
Now that I think some more, I don't think that "HOME=nas-0-0:/state/partition1/home" will work. It would have to ssh to nas-0-0 to create the home directory and populate it with the contents of /etc/skel. That's not going to happen.
> - do NOT edit /etc/auto.home map because it was empty
It gets populated by running "rocks sync users".
> - edit /etc/auto.shareto:
> apps nas-0-0.local:/state/partition1/&
> and created a apps directory under /state/partition1 of nas-0-0
>
> Then, I managed to easy create a file in the frontend in /share/apps that was correctly stored in
> nas-0-0.local:/state/partition1/apps/. This part works perfectly.
Rejoice. Something actually worked!
> Finally, I have tried to add a user and get:
>
> # useradd test
> useradd: cannot create directory nas-0-0.local:/state/partition1/home/test
>
> and the user was not completed created. I then deleted it.
>
> It seems that nas-0-0.local:/state/partition1 is not being automounted by the frontend. I
> mounted it explicitly to verify if it is avaialble and it was. Then, I include the following
> line in the /etc/auto.home to:
>
> test nas-0-0.local:/state/partition1/home/&
>
> and tried again:
>
> #useradd test
> useradd: cannot create directory nas-0-0.local:/state/partition1/home/test
>
> but with the same problem. If I changed back the HOME variable in /etc/default/useradd to
> /export/home, the user was created without any problem.
>
> Do you have any idea of what the problem is and how to correct it?
I think you'll have to create them on the frontend (using the standard /etc/default/useradd) then move them to the NAS by hand. Directions have been written up at
https://wiki.rocksclusters.org/wiki/index.php/Home_directories
Use the "By hand" method, as I don't think the "Changing the defaults" method can handle having them on a separate server. Take a look at plugin_fixnewusers.py and see if it's easy.
> P.S. Curious remark: after installing the NAS, I installed the first compute node. Curiously,
> without any extend-compute.xml, the compute-0-0 was installed using the centos.plus kernel that
> was set to be installed in the extend-nas.xml. Then, I simply copy the skeleton.xml to the
> extend-compute.xml and reinstall the compute-0-0 from scratch. It still gets the centos.plus
> kernel. How can it be possible?It is getting an extra kernel package thta it was not informed
> to pick. Has anyone an explanation?
In my case, I got the centos.plus kernel on ALL my machines after a re-install where I included that RPM in the restore roll. Frontend, all NAS nodes, and compute nodes are all running the centos.plus kernel. It doesn't seem to be a problem, though. It's basically the same kernel, but with a few extra modules.
I think this happens because Anaconda always takes the latest (has the newest time-stamp) of all the kernel*.rpm packages it finds anywhere on the system. So perhaps if you re-set the timestamp of the centos.plus kernel to last year, it would only use it for the NAS.
Or, you could move it out of /export/rocks/contrib and re-build your distro, then install the compute nodes. You'll just have to remember to move it back when you need to re-install your NAS node -- but that seldom happens, as by default NAS nodes are NOT re-installed after a power failure or other ungraceful shutdown.
Peter Kjellstrom
> > I would like to thank Bart Brashers for his excellent hint, but I have
> > one more question (see below) and a curious remark (see the P.S.).
> >
> > Today I managed to install the NAS into Rocks 5.1 using the path he
> > suggested. In order to register this for future use, there was necessary
> > to install the rpms for XFS file support and the centos.plus kernel. The
> > necessary packages are:
> >
> > dmapi-2.2.8-1.el5.centos.x86_64.rpm
> > kernel-2.6.18-92.1.13.el5.centos.plus.x86_64.rpm
> > kmod-xfs-0.4-2.x86_64.rpm
> > xfsdump-2.2.46-1.el5.centos.x86_64.rpm
> > xfsprogs-2.9.4-1.el5.centos.x86_64.rpm
>
> Good, these appear to be correct for the newer version of CentOS.
I'm confused here. Why are you using the centosplus kernel? If all you want is
xfs then install kmod-xfs+xfsprogs with the _normal kernel_.
/Peter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20090115/2c1cd96d/attachment.bin
Hamilton, Scott L.
Hamilton, Scott L.
Add the following line to you /etc/exports on the file server.
/state/partition1/home x.x.x.x/255.255.255.255(rw,async,no_root_squash)
Where x.x.x.x is the ip address of your head node.
This allows root on the head node to be root on the share, by default nfs shares mount the folder as "nobody" when accessed with the root account and "nobody" will not have rights to create the home folders, so rocks sync users will fail without the no_root_squash option.
Thanks,
Scott
-----Original Message-----
From: npaci-rocks-dis...@sdsc.edu [mailto:npaci-rocks-dis...@sdsc.edu] On Behalf Of Bart Brashers
Sent: Wednesday, January 14, 2009 6:30 PM
To: Discussion of Rocks Clusters
Subject: Re: [Rocks-Discuss] NAS with XFS in Rocks 5.1
Bart Brashers
>
> The comment about the creation of home directories is false. If you
enable no_root_squash on the
> export for the home directories.
>
> Add the following line to you /etc/exports on the file server.
> /state/partition1/home
x.x.x.x/255.255.255.255(rw,async,no_root_squash)
> Where x.x.x.x is the ip address of your head node.
>
> This allows root on the head node to be root on the share, by default
nfs shares mount the folder
> as "nobody" when accessed with the root account and "nobody" will not
have rights to create the
> home folders, so rocks sync users will fail without the no_root_squash
option.
What you write is true. But there's no way for <bold>
/usr/sbin/useradd </bold> to create the home directory for a new user
directly on the NAS, since it's not nas-0-0:/state/partition1/home
that's being mounted on the frontend, it's
nas-0-0:/state/partition1/home/$USER. Perhaps you are suggesting to
mount nas-0-0:/state/partition1/home on /export/home? You couldn't do
that with autofs, it would have to be a static mount set in /etc/fstab.
Rocks theology believes that static mount eventually go stale, and we
should always use autofs.
One could hack up plugin_fixnewusers.py to move the home dir to the NAS,
I suppose. But this would likely be more trouble than it's worth,
IMHYCO. It's pretty easy to make yourself a script like this (untested,
just off the top of my head) which would be portable between Rocks
versions:
# cat /usr/local/sbin/useradd
#!/bin/csh -f
if ($#argv == 0 | $username =~ -*h*) then
echo "Usage: $0:t username"
echo "Creates a user, and moves the home directory to the NAS."
exit
endif
set username = $1
/usr/sbin/useradd $username
rocks sync users
sed -i
"s/\/export\/home\/$username/nas-0-0:\/state\/partition1\/home\/$usernam
e/" /etc/auto.home
rsync -a /export/home/$username nas-0-0:/state/partition1/home
rm -rf /export/home/$username
Steven Dick
<bbra...@environcorp.com>wrote:
> What you write is true. But there's no way for <bold>
> /usr/sbin/useradd </bold> to create the home directory for a new user
> directly on the NAS, since it's not nas-0-0:/state/partition1/home
> that's being mounted on the frontend, it's
> nas-0-0:/state/partition1/home/$USER.
Easy to fix. Add a direct mount for the NAS home base to /etc/auto.misc
(and add auto.misc to auto.master), edit the defaults file to create the
home directory there, and then follow the instructions in the wiki to update
fixnewusers.py so that it changes the base directory to the automount
directory.
Of course, it would be better if we could modify fixnewusers.py so that it
read these locations out of the rocks database as a variable (or even
better, a table listing multiple possible base home directories).
-------------- next part --------------
An HTML attachment was scrubbed...
Paulo Laranjeira da Cunha Lage
> > and created a home directory under /state/partition1 of nas-0-0
Let's clarify: I tried to say that I created /state/partition1/home in nas-0-0 and not the user's home. I tried to use the modified /etc/default/useradd to get it created but it did not work. By the way, the /state/partition1 in nas-0-0 is exported for all internal network as:
# ssh -x nas-0-0 cat /etc/exports
/state/partition1 10.10.0.0/255.255.0.0(rw,no_root_squash,async)
The Bart second hint was also correct: the "By hand method" in https://wiki.rocksclusters.org/wiki/index.php/Home_directories worked as it should. On the other hand, the instructions for the change of plugin_fixnewusers.py seems to be incomplete. However, my knowledge of python is almost none. I will try to make the "by hand" method automatic using a script. Interestingly, the python code seems to try to verify the existence of a file server in the database (see lines below). I wonder if there is not a better way of doing this. Maybe one of the developers has the answer.
--------from /opt/rocks/lib/python2.4/site-packages/rocks/commands/sync/users/plugin_fixnewusers.py
# if there is a file server specified in the database
# use it. otherwise, use the default.
hostname = self.db.getGlobalVar('Info', 'HomeDirSrv')
if not hostname:
hostname = '%s.%s' % (self.db.getGlobalVar('Kickstart',
'PrivateHostname'),
self.db.getGlobalVar('Kickstart',
'PrivateDNSDomain'))
--------------------------------------------
Finally, it was really the newer time satmp of the centos.plus kernel that get it into the compute configuration without my request (due to anaconda). Bart was right again.
Thanks,
Paulo
------------------------------------
Prof. Paulo Laranjeira da Cunha Lage
Termofluidodinamica
Programa de Engenharia Quimica
COPPE/UFRJ
http://www.peq.coppe.ufrj.br
http://www.peq.coppe.ufrj.br/Areas/tfd/
++++++++++++++++++++++++++++++++++++++++++++
PEQ/COPPE renova o nivel 7, maximo na CAPES:
45 anos de excelencia no ensino e pesquisa em Engenharia Química
45 years of excellence in teaching and research in Chemical Engineering
++++++++++++++++++++++++++++++++++++++++++++
---------- Original Message -----------
From: "Bart Brashers" <bbra...@Environcorp.com>
To: "Discussion of Rocks Clusters" <npaci-rocks...@sdsc.edu>
Sent: Wed, 14 Jan 2009 16:29:39 -0800
Subject: Re: [Rocks-Discuss] NAS with XFS in Rocks 5.1
------- End of Original Message -------
-------------- next part --------------
An HTML attachment was scrubbed...
Bart Brashers
The wiki hint for changing plugin_fixnewusers.py is probably for Rocks 5.0, so the line numbers will change. I've just updated that page with some code snippets.
Bart
Paulo Laranjeira da Cunha Lage
Well, I assumed the method "by hand" was working after testing by adding 1 user. Then a tried to add another user I could not login to the correct directory. I then delete and add some users and finally I have got stucked with problems with user directory access with the message "Stale NFS file handle".
Then, I removed all users, all corresponding directories in both frontend:/export/home (original ones) and nas-0-0:/state/partition1/home (copied ones), umount the /home/<user> that existed in the /etc/mtab and tried to create again the accounts.
Then, I added a user, used the method "by hand" to relocate the directory and this works. Then used exactly the same procedure to a second user and it did not work. At login, I have got the error: "Could not chdir to home directory /home/lfernando: No such file or directory".
So, my question is: in nas-0-0 I have to export each user's directory in a different line or the /state/partition/home suffices?
I have also tried the method described in hanging the defaults" but it did not work either.
I really need some help.
Paulo
Bart Brashers
Hi Paulo,
This is very odd. If you can do it once and it works, something must have gone wrong during the 2nd time you did it. Were there any error or warning messages printed to the screen during the 2nd time you ran the script?
Perhaps you can post the script you used to do the "by hand" method. Or the exact command lines, for both user #1 and user #2.
Also, investigate a little more the root cause behind the "could not chdir to home directory" error. Look in /var/log/messages. Check that the 2nd call to "rocks sync users" actually re-distributed the new /etc/auto.home files, and did a "service autofs reload".
By the way, did the "could not chdir to home directory" error come from the frontend, or from a compute node?
Bart
Steven Dick
pa...@peq.coppe.ufrj.br> wrote:
> So, my question is: in nas-0-0 I have to export each user's directory in a
> different line or the /state/partition/home suffices?
Exporting /state/partition/home should be enough.
The trick is creating the directory correctly (not in /home/ ) and then
adjusting the python script to translate it to /home/ afterwards.
-------------- next part --------------
An HTML attachment was scrubbed...
Paulo Laranjeira da Cunha Lage
Well, I was too tired yesterday. I was trying the alternatives using commands and I should have made something wrong in the creation of the second user. Today, I decided to use scripts to test everything. I managed to put the users accounts into the NAS using two methods, one that comes from the "By hands method" and the other from the "Changing the defaults" in https://wiki.rocksclusters.org/wiki/index.php/Home_directories , but using a idea that Steven gave.
The two methods are listed below for reference and for improvement suggestions.
First method: let Rocks create the account and them move it and correct the mouting points
-----------------
Need only to add the following line in the /etc/exports of nas-0-0 to export the large partition /state/partition1:
/state/partition1 10.10.0.0/255.255.0.0(rw,no_root_squash,async)
Then, these two scripts do the job (tested in a frontend named ares with only one NAS server: nas-0-0):
-------------------------------------------------------------------------------------------------------------
# cat ./useradd
#!/bin/csh -f
if ($#argv == 0) then
echo "Usage: $0:t username"
echo "Creates a user, and moves the home directory to the NAS."
exit
endif
set username = $1
/usr/sbin/useradd $username
/usr/bin/passwd $username
rocks sync users
sed -i "s/ares.local:\/export\/home\/$username/nas-0-0.local:\/state\/partition1\/home\/$username/" /etc/auto.home
rsync -a /export/home/$username nas-0-0:/state/partition1/home
rm -rf /export/home/$username
------------------------------
# cat ./userdel
#!/bin/csh -f
if ($#argv == 0) then
echo "Usage: $0:t username"
echo "Deletes a user and its home directory in NAS."
exit
endif
set username = $1
/usr/sbin/userdel $username
sed -i "/$username/ D" /etc/auto.home
rocks sync users
umount /home/$username
ssh nas-0-0.local rm -rf /state/partition1/home/$username
rm -f /var/spool/mail/$username
------------------------------
Second method: Try to make Rocks to use the NAS storage
---------------------
1) Need only to add the following line in the /etc/exports of nas-0-0 to enable direct $HOME creation in NAS (the IP address belongs to the the fronted):
/state/partition1/home 10.10.1.1/255.255.255.255(rw,no_root_squash,async)
In the frontend, create a /etc/auto.export file with:
home nas-0-0.local:/state/partition1/&
and include it in the /etc/auto.master:
/share /etc/auto.share --timeout=1200
/home /etc/auto.home --timeout=1200
/export /etc/auto.export --timeout=1200
2) No change is made in /etc/default/useradd => it keeps HOME=/export/home (now automounted from NAS)
3) Edit rocks sync users script in (Rocks 5.1) /opt/rocks/lib/python2.4/site-packages/rocks/commands/sync/users/plugin_fixnewusers.py
- from line 125 on to look like this, definig hostname and hostdirloc explicitly:
# if there is a file server specified in the database
# use it. otherwise, use the default.
# hostname = self.db.getGlobalVar('Info', 'HomeDirSrv')
# if not hostname:
# hostname = '%s.%s' % (self.db.getGlobalVar('Kickstart',
# 'PrivateHostname'),
# self.db.getGlobalVar('Kickstart',
# 'PrivateDNSDomain'))
# if there is a home directory specified in the database
# use it. otherwise, use the default.
# homedirloc = self.db.getGlobalVar('Info', 'HomeDirLoc')
# if not homedirloc:
# homedirloc = '/export/home'
hostname = 'nas-0-0.local'
homedirloc = '/state/partition1/home'
---------------------------------------
OBS: users cannot log on nas-0-0 with this setup (maybe a good thing)
Problems with second setup:
--------------------------------------
- usedel -r <user> do not remove home user. Must use 'rm -rf /export/home/<user>
- rocks sync users do not remove corresponding line from auto.home. Must be done manually.
- <user> directory remains NFS mounted in frontend and in computer nodes (if user has just logoff) => mounting problems if the same user is created again.
Comment: First solution is more robust: you can cycle creating and deleting the same users and the scripts do all that is need. Second solution needs improvement to make it as robust as the first one.
Any suggestions?
Thank very much for the help provided,
Jonas Baltrusaitis
[jbaltrus@mr9540 pwgui_4.1CVS]$ ./pwgui
no display name and no $DISPLAY environment variable
while executing
"load /share/apps/pwgui_4.1CVS/pwgui/lib/tk8.4/libtk8.4.so Tk"
("package ifneeded" script)
invoked from within
"package require Tk 8.4 "
(file "/share/apps/pwgui_4.1CVS/pwgui/main.tcl" line 9)
how do I properly setup $DISPLAY for that? I tried several things but none worked.
[jbaltrus@mr9540 pwgui_4.1CVS]$ export DISPLAY=:0.0
[jbaltrus@mr9540 pwgui_4.1CVS]$ ./pwgui
Xlib: connection to ":0.0" refused by server
Xlib: No protocol specified
couldn't connect to display ":0.0"
while executing
"load /share/apps/pwgui_4.1CVS/pwgui/lib/tk8.4/libtk8.4.so Tk"
("package ifneeded" script)
invoked from within
"package require Tk 8.4 "
(file "/share/apps/pwgui_4.1CVS/pwgui/main.tcl" line 9)
[jbaltrus@mr9540 pwgui_4.1CVS]$ export DISPLAY=localhost:0.0
[jbaltrus@mr9540 pwgui_4.1CVS]$ ./pwgui
couldn't connect to display "localhost:0.0"
while executing
"load /share/apps/pwgui_4.1CVS/pwgui/lib/tk8.4/libtk8.4.so Tk"
("package ifneeded" script)
invoked from within
"package require Tk 8.4 "
(file "/share/apps/pwgui_4.1CVS/pwgui/main.tcl" line 9)
Bart Brashers
Thanks for posting the scripts for the "By Hand" method. I edited them little (made them more general, and moved the "rocks sync users" call to last) and added them to the wiki:
https://wiki.rocksclusters.org/wiki/index.php/Home_directories
I also added a section about an un-documented feature of Rocks 5.1, that can query the database for the default server and home directory location inside plugin_fixnewusers.py. My cluster is still running Rocks 5.0, so I don't have this version. I saw the database calls in your post. NOTE: the same drawbacks about userdel still apply!
> Second method: Try to make Rocks to use the NAS storage
> ---------------------
>
> 1) Need only to add the following line in the /etc/exports of nas-0-0 to enable direct
> $HOME creation in NAS (the IP address belongs to the the fronted):
>
> /state/partition1/home 10.10.1.1/255.255.255.255(rw,no_root_squash,async)
>
> In the frontend, create a /etc/auto.export file with:
>
> home nas-0-0.local:/state/partition1/&
>
> and include it in the /etc/auto.master:
>
> /share /etc/auto.share --timeout=1200
> /home /etc/auto.home --timeout=1200
> /export /etc/auto.export --timeout=1200
**** WARNING! ****
Doing the above causes autofs to take over complete control over /export on the frontend. Any other directories you used to have in /export will now appear to be missing. You may not have tried to access /export/rocks/install during your testing, so you may not have noticed. I think this method will break Rocks 5.1! Here's proof:
[root ~]# mkdir /testme
[root ~]# cd /testme/
[root /testme]# touch testfile
[root /testme]# mkdir testdir
[root /testme]# ls
total 4.0K
drwxr-xr-x 2 root root 4.0K Jan 18 10:39 testdir/
-rw-r--r-- 1 root root 0 Jan 18 10:39 testfile
[root /testme]# cd
[root ~]# ssh nas-0-0
Last login: Thu Jan 15 15:39:40 2009 from frontend.local
Rocks 5.0 (V)
Profile built 13:35 05-Sep-2008
Kickstarted 13:47 05-Sep-2008
[root@nas-0-0 ~]# cd /state/partition1/
[root@nas-0-0 /state/partition1]# mkdir testnas
[root@nas-0-0 /state/partition1]# cd testnas
[root@nas-0-0 partition1/testnas]# touch testfile.nas
[root@nas-0-0 partition1/testnas]# mkdir testdir.nas
[root@nas-0-0 partition1/testnas]# exit
logout
Connection to nas-0-0 closed.
[root ~]# cat /etc/auto.testme
testnas -rw,soft,intr,rsize=32768,wsize=32768 nas-0-0.local:/state/partition1/&
[root ~]# cat /etc/auto.master
/share /etc/auto.share --ghost --timeout=1200
/home /etc/auto.home --ghost --timeout=1200
/testme /etc/auto.testme --timeout=1200
[root ~]# service autofs reload
Reloading maps
[root ~]# ls /testme/
total 0
[root ~]# ls /testme/testnas
total 4.0K
drwxr-xr-x 2 root root 4.0K Jan 18 10:42 testdir.nas/
-rw-r--r-- 1 root root 0 Jan 18 10:41 testfile.nas
Note that the local directory and file I created are now "gone". I can get them back by taking /testme out of autofs's control (by un-doing what I just did).
You can still use something similar to this method, if you pick some other (new) directory to let autofs take over, and then remove /export/home and make it a link to that directory. Something like this:
# mkdir /tmphome
# echo 'home nas-0-0.local:/state/partition1/&' >! /etc/auto.tmphome
# echo '/tmphome /etc/auto.tmphome --timeout=1200' >> /etc/auto.master
# service autofs reload
# rsync -a /export/home /tmphome
# rm -rf /export/home
# ln -s /tmphome/home /export/home
This should work, though I haven't tested it.
Of course, the drawbacks regarding userdel still apply. I would suggest you stick with the "By Hand" scripts, now that they are tested and robust, and don't have any identified drawbacks.
Bart Brashers
installed on rocks from my windows
> pc. when i try that i get an error:
You cannot run an executable that was compiled for Linux on a CPU that's
running Windows.
Perhaps you meant you want to run a program on your Linux host and have
the output displayed on your Windows box. Are you running an X-windows
server on your Windows machine? How do you currently "access" (get a
command line) on your cluster from your Windows box?
Generally speaking, you need to install some software on your Windows
box that speaks the X-windows protocol. There are some free ones
available, but most I've tried have been cumbersome to install/maintain
and slow. I use Netsarang's Xmanager, which is a reasonably cheap
version. It uses ssh to establish the connections between the Windows
box and the Linux box, and passes all the X-windows stuff through that
pipe. I don't have to set any $DISPLAY variables, it just works.
If you are running some sort of X-windows server on your Windows box,
then you need to tell it to allow connections from the Linux box. Use
"xhost +<frontend>" to allow the host <frontend> in (substitute your
real frontend name). Then you can set the display variable on the Linux
command line like this:
# setenv DISPLAY <winboxname>:0.0
Of course, your frontend needs to be able to resolve winboxname to its
IP address. This is generally accomplished by providing the frontend
with the IP address of your local Windows domain controller (which is
generally also a DNS server) as a forwarder. In /etc/named.conf, add
the IP address of your local DNS server (192.168.35.10 in my example
below) that knows about your Windows PC name and IP address:
options {
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
forwarders {192.168.35.10; };
};
and do a "service named restart".
Paulo Laranjeira da Cunha Lage
Regarding the scripts for the "By Hand" method, the useradd script was written with the intention that it behaved similarly to the /usr/sbin/useradd, that is, "rocks sync users" still needs to be run after it. The way you changed the useradd script, it became wrong. You can add a final line with "rocks sync users", but CANNOT remove the "rocks sync users" line after "/usr/sbin/useradd $username" or the /etc/auto.home line that the sed line changes will not exist at the time sed is executed. You should correct this in the wikki.
Regarding you observation, I agree with you. I noticed the problem a little after I sent the e-mail. But I agree that the method will work if a different mounting point (besides /export) is used in the frontend, as you pointed out. I did not test, but your suggestion reproduced below should work:
> # mkdir /tmphome
> # echo 'home nas-0-0.local:/state/partition1/&' >! /etc/auto.tmphome
> # echo '/tmphome /etc/auto.tmphome --timeout=1200' >> /etc/auto.master
> # service autofs reload
> # rsync -a /export/home /tmphome
> # rm -rf /export/home
> # ln -s /tmphome/home /export/home
Thanks,
Paulo
------------------------------------
Prof. Paulo Laranjeira da Cunha Lage
Termofluidodinamica
Programa de Engenharia Quimica
COPPE/UFRJ
http://www.peq.coppe.ufrj.br
http://www.peq.coppe.ufrj.br/Areas/tfd/
++++++++++++++++++++++++++++++++++++++++++++
PEQ/COPPE renova o nivel 7, maximo na CAPES:
45 anos de excelencia no ensino e pesquisa em Engenharia Química
45 years of excellence in teaching and research in Chemical Engineering
++++++++++++++++++++++++++++++++++++++++++++
-------------- next part --------------
An HTML attachment was scrubbed...
Philip Papadopoulos
Client.
You will also need an X11 Server on your Windows machine (X uses the term
Server for the machine the runs the Physical Display, and client for any
application the displays output on a Display).
If you don't have an X server for your Windows box, look at
http://x.cygwin.com/
for free software.
Forwarding X connections is pretty bandwidth intensive -- if you are on your
campus, it will work pretty well. If you are tunneling over
your cable modem, it will be slow.
-P
On Sun, Jan 18, 2009 at 11:12 AM, Bart Brashers
<bbra...@environcorp.com>wrote:
--
Philip Papadopoulos, PhD
University of California, San Diego
858-822-3628 (Ofc)
619-331-2990 (Fax)
-------------- next part --------------
An HTML attachment was scrubbed...
Bart Brashers
> Regarding the scripts for the "By Hand" method, the useradd script was
written with the intention
> that it behaved similarly to the /usr/sbin/useradd, that is, "rocks
sync users" still needs to be
> run after it. The way you changed the useradd script, it became wrong.
You can add a final line
> with "rocks sync users", but CANNOT remove the "rocks sync users" line
after
> "/usr/sbin/useradd $username" or the /etc/auto.home line that the sed
line changes will not exist
> at the time sed is executed. You should correct this in the wikki.
Good catch, fixed.