More troubles in parallel mode

[Andreas Brinkmann]

Hi,

I have some more troubles using the -cluster option together with PBS.
My PBS script generates the following cluster.cfg file (just a short
example):
e02-02 1.0 /euler/scratch2/anbr
e02-07 1.0 /euler/scratch2/anbr
and spinevolution is then called inside the PBS script from the
directory where the input file and cluster files can be found:
spinev ${input_file} -cluster -fastssh
However, I get the same error reported before by Kris Harris:
Error: error -3 occured while communicating with node 0

But when I instead start spinevolution on the command line (not in the
PBS script) the calculation is starting without any problem, where I
always have to use -fastssh otherwise I get the problem again.

What could be the reason for the different behavior inside/outside the
PBS script

Thanks,

Andreas

[Mikhail Veshtort]

Hi Andreas,

I suspect this has to do with different environments in these two
cases. Could you please see if there are any differences? For bash
shell, use env to print out all the variables, save them in files and
then compare the files. If you find anything suspicious, try changing
those variables.

If this does not resolve the problem, is there any way I can get a
login on your system to try to debug this?

Mikhail

[Andreas Brinkmann]

Hi Mikhail,

I played around a bit further and found out that the spinev master
process is also started on a cluster slave node and not on the master
node. This is the setup of the cluster here: even master processes
start on slave nodes.

Spinev consistently works fine with "-cluster -fastssh" when I start
it from the master node or a slave node >command line< when I
connected to it by ssh. A tty is set when the master process is
started.

However, the PBS script connects to the first slave node in non-
interactive mode and then starts the spinev master process. No tty is
set when the master process is started.

Summary:

tty assigned "/dev/pts/..." (command line) -> spinev -cluster -fastssh
works on master/slaves and can connect to other slaves
tty not assigned "not a tty" (PBS script) -> spinev -cluster -fastssh
does not connect properly

I hope this might give you some hints on how to solve this problem.
Could I convince PBS to set a tty?

Best wishes,

Andreas