Troubles with parallel mode

[Kris Harris]

Hello, I hope someone can help me out with getting spinevolution to
run in parallel on a cluster.

Starting with an example file (the hccn example input file), it runs
fine in serial mode

defined cluster.cfg in directory defined by $SPINEV with one entry:
krill@mb3 1.0 /home/krill/work /home/krill/spinev-3.3.3-Linux-i686

but it fails when I try to run it:
harris@madsbak:~/spinevolution_examples$ spinev hccn -cluster
SPINEVOLUTION 3.3.3
Experiment loaded
maxdt = 2 usec
Powder averaging will be split as follows:
0 mb3 168
Error: error -3 occured while communicating with node 0

Running the cluster test seems to provide a positive result:
harris@madsbak:~/spinevolution_examples$ spinev -sclust
mb3:22 OK

Also, if I try and start the program manually with the info. from cluster.cfg :
ssh krill@mb3 /home/krill/spinev-3.3.3-Linux-i686/spinev
seems to work out fine as well

Trying to track down the problem, I replaced the spinev executable on
the called-on node with a simple script to dump a message to a text
file
(just #!/bin/bash echo "this program ran" > /home/krill/output.txt )
If I run manually with the above ssh krill@mb3 command, the message is
output to the file, but if I run as spinev hccn -cluster, no message
is produced, and the error message is the same as above (the error -3
business).

So, it seems that the spinev command on the second node is not even
getting called upon. What have I done wrong in the config. here?

thanks in advance,
Kris Harris

[Mikhail Veshtort]

Hi Kris,

Let's try a quick fix first: run your cluster simulation with the -
fastssh option.

Here is why. If this option is NOT used, ssh is started with the -tt
option. This is supposed to tell ssh to allocate a pseudo-terminal, so
that ^C could be used if the cluster simulation needs to be aborted.
(It also makes the exchange of data between the "master" and the
"slave" processes a little slower). However, I recently discovered
that on some systems this -tt option prevents things from working at
all (I don't know how).
Note that if -fastssh is specified, one has to use kill to abort the
simulation.

Please let us know if this helps.

Best,
Mikhail

[kris]

Hi, and thanks for the help.
It does indeed solve the initial problem when I use the -fastssh
option.


There's a new hurdle showing up now though. The parallel runs work
if I run one or more jobs on the same node, but not if I split them up
onto different nodes.

That is, it runs fine with the cluster.cfg file:
mb3 1.0 /scr/spinev/harris
mb3 1.0 /scr/spinev/harris
or with:
mb4 1.0 /scr/spinev/harris
mb4 1.0 /scr/spinev/harris
but it fails when two differing nodes are used:
mb3 1.0 /scr/spinev/harris
mb4 1.0 /scr/spinev/harris





The outputs for the last two cases (without any verbose modes turned
on) are:
---------------------------------------------------------------------------------------------------------------------------
harris@madsbak:~/spinevolution_examples$ spinev hccn -fastssh -

cluster
SPINEVOLUTION 3.3.3
Experiment loaded
maxdt = 2 usec
Powder averaging will be split as follows:

0 mb4 84
1 mb4 84
0:
1:
0: Powder averaging over:
1: Powder averaging over:
0: 84 (alpha,beta) pairs
1: 84 (alpha,beta) pairs
0: 16 uniformly sampled gamma angles
1: 16 uniformly sampled gamma angles
0:
1:
0: phi_1 = -130
1: phi_1 = -130
0: phi_1 = -140
1: phi_1 = -140
0: phi_1 = -150
1: phi_1 = -150
0: phi_1 = -160
1: phi_1 = -160
0: phi_1 = -170
1: phi_1 = -170
0: phi_1 = -180
1: phi_1 = -180

clock 0.01
0:
0: clock 12.69
1:
1: clock 12.69
harris@madsbak:~/spinevolution_examples$
---------------------------------------------------------------------------------------------------------------------------


versus:
---------------------------------------------------------------------------------------------------------------------------
harris@madsbak:~/spinevolution_examples$ spinev hccn -fastssh -cluster

SPINEVOLUTION 3.3.3
Experiment loaded
maxdt = 2 usec
Powder averaging will be split as follows:

0 mb3 84
1 mb4 84
0:
0: Powder averaging over:
0: 84 (alpha,beta) pairs
0: 16 uniformly sampled gamma angles
0:
0: phi_1 = -130
0: phi_1 = -140
0: phi_1 = -150
0: phi_1 = -160
0: phi_1 = -170
0: phi_1 = -180
Error:
harris@madsbak:~/spinevolution_examples$
---------------------------------------------------------------------------------------------------------------------------


It looks like in the second case, only the first node is really being
contacted. I retried
with verbose mode -v4, but grepping the output for the word error
showed no results.


I tried logging into the nodes that are involved, and the spinev
process and temp files
only show up on the first node (whichever node of mb3 or mb4 that is
listed first in the
cluster.cfg file)

It seems to be a different problem than before, as the spinev
executable is actually called
on the second machine this time (as probed by replacing the spinev
executable on the second
node with a script)

thanks in advance for any help,
Kris

[Mikhail Veshtort]

Hi Chris,

It seems that your nodes have different file systems (I am guessing
this from what you say about the temp files not showing up). In that
case, their lines in the cluster configuration file should be
separated by a comment line that starts with an asterisk. Please see
if this resolves the issue.

Mikhail

[kris]

Yes, that does indeed work everything out and it works fine now.
I guess I was too distracted by getting one node working that I
didn't do enough checking when adding in a second node.

thanks,
Kris