[Rocks-Discuss] NFS write problems
Renate Mittelmann
rocks list roll:
NAME VERSION ARCH ENABLED
area51: 5.1 x86_64 yes
base: 5.1 x86_64 yes
bio: 5.1 x86_64 yes
ganglia: 5.1 x86_64 yes
hpc: 5.1 x86_64 yes
java: 5.1 x86_64 yes
kernel: 5.1 x86_64 yes
os: 5.1 x86_64 yes
sge: 5.1 x86_64 no
web-server: 5.1 x86_64 yes
mlnx-ofed: 5.1 x86_64 yes
intel-11: 5.1 x86_64 yes
modules: 5.1 x86_64 yes
cuda: 5.1 x86_64 yes
pvfs2: 5.1 x86_64 yes
torque: 5.1.0 x86_64 yes
We are able to submit and run a 40 processor job until it came time to write our output data. Here's the error I got:
File locking failed in ADIOI_Set_lock. If the file system is NFS, you need to us
e NFS version 3, ensure that the lockd daemon is running on all the machines, an
d mount the directory with the 'noac' option (no attribute caching).
[36] [MPI Abort by user] Aborting Program!
Has anyone an idea what is wrong?
Renate
Joe Landman
> We are able to submit and run a 40 processor job until it came time
> to write our output data. Here's the error I got:
>
> File locking failed in ADIOI_Set_lock. If the file system is NFS, you
> need to us e NFS version 3, ensure that the lockd daemon is running
> on all the machines, an d mount the directory with the 'noac' option
> (no attribute caching). [36] [MPI Abort by user] Aborting Program!
>
>
> Has anyone an idea what is wrong?
Yes. This is an MPI-IO problem, and for MPI-IO to work reliably over
NFS, they want you to use the noac mount option.
If you value your NFS performance, you won't turn on noac on your mount.
It drops IO performance by a huge amount.
You have a few options, most not integrated into the Rocks stack.
1) make sure nfslockd is running (usually /etc/init.d/nfs-lock or
similar), and have the noac option in the mount. You don't want to do
this ... or better ... you want to do this once so you see why this is a
bad choice.
2) install PVFS2 on your compute nodes, and do your writes to that for
this run. It knows about MPI-IO, thats what it is designed for. This
could work, depending upon how many writers you have. If you have one
writer, this won't work so well as compared to other options.
3) install PVFS2 on the compute and head nodes, export a portion of the
head node file system as a PVFS2 mount point, and have the compute nodes
mount that. This could work, depending upon how much you write, though
it is possible you could overload the network into the head node (which
is not good for the cluster). This is in part why we usually recommend
using separate IO nodes.
4) install another MPI-IO aware cluster file system alongside NFS.
PVFS2 was a specific simple example. There are others as well.
5) see if the program has an POSIX IO option. MPI-IO isn't
intrinsically faster, it is supposed to make performing parallel IO
easier. However, many people still do all their IO from one node. In
which case, there really isn't much difference between MPI-IO and POSIX IO.
> Renate
Regards,
Joe
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: lan...@scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/jackrabbit
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
Renate Mittelmann
many thanks for the info. So I look at option 2 as a start.
The pvfs2 roll is installed, but when I start the server I get (grafix is the the hostname of this cluster)
Starting PVFS2 server: [S 12/27 10:15] PVFS2 Server on node grafix version 2.7.1 starting...
[E 10:15:54.298525] Configuration file error. No host ID specified for alias grafix.
[E 10:15:54.298544] Error: Invalid configuration; aborting.
[D 10:15:54.298553] *** server shutdown in progress ***
[D 10:15:54.298559] [*] halting logging interface
[FAILED]
The default host config file:
/opt/pvfs2/etc/pvfs2-fs.conf
is empty. What would be the settings for the default .conf file?
Renate
Mason J. Katz
need to edit the /etc/auto.home file on the frontend to insert
whatever options you need, and then run "rocks sync users" to trigger
the 411 updates to the compute nodes.
The below faq shows how to insert options into these files. The nfs
manpage also describes exactly what the noac option does, and why it
will certainly slow things down. But since you can do this on a per
home directory basis you can fix things just for one user and not slow
down everything else.
http://www.faqs.org/docs/Linux-mini/Automount.html#s4
You can also start the lock daemon as Joe says on the frontend.
Unless you want to dive into the pvfs2 pool you it's just two quick
changes on your frontend.
mason katz
+1.240.724.6825
Joe Landman
> Joe many thanks for the info. So I look at option 2 as a start. The
> pvfs2 roll is installed, but when I start the server I get (grafix is
> the the hostname of this cluster)
>
>
> Starting PVFS2 server: [S 12/27 10:15] PVFS2 Server on node grafix
> version 2.7.1 starting... [E 10:15:54.298525] Configuration file
> error. No host ID specified for alias grafix. [E 10:15:54.298544]
> Error: Invalid configuration; aborting. [D 10:15:54.298553] ***
> server shutdown in progress *** [D 10:15:54.298559] [*] halting
> logging interface [FAILED]
>
> The default host config file: /opt/pvfs2/etc/pvfs2-fs.conf is empty.
> What would be the settings for the default .conf file?
You'd have to ask the Roll developer specifically, but you could follow
the directions here: