mpi job runs fine when submitted via scheduler, but fails when run by hand

[Faraz H]

I have a simple job submission script which runs fine. However, when I try and run the same script by hand ( outside the scheduler), it fails with the error below. By hand I mean that I take the nodes offline, login to the first node, load the cp2k/open mpi modules, keep same machine file, then run the command:

mpirun -machinefile machines -np 40 --mca btl openib,self cp2k.popt test.inp


mpirun was unable to find the specified executable file, and therefore
did not launch the job.  This error was first reported for process
rank 20; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command
      line parameter option (remember that mpirun interprets the first
      unrecognized command line token as the executable).

Node:       lustwzb5
Executable: /gpfs/app/software/cp2k/3.0.0/exe/Linux_x86-64-intel/cp2k.popt

[Faraz H]

I figured it out by using the full path to the cp2k executable and passing the LD_LIBRARY_PATH environment variable. I had to do mpirun -x LD_LIBRARY_PATH . For some reason I do not have to do this in my submission script. Perhaps the scheduler takes care of exporting paths and env. variables for you?

[Christian Leitold]

On Wednesday, August 2, 2017 at 8:58:09 AM UTC-7, Faraz H wrote:

I figured it out by using the full path to the cp2k executable and passing the LD_LIBRARY_PATH environment variable. I had to do mpirun -x LD_LIBRARY_PATH . For some reason I do not have to do this in my submission script. Perhaps the scheduler takes care of exporting paths and env. variables for you?

That is most certainly the case. What scheduler are you using? In general, I would say the problem you have described is more of a general MPI / scheduler problem, and not specific to cp2k, as the cp2k executable was not even found by your MPI system.

Best,

Christian 

[Faraz H]

Thanks, we are using : Son of Grid Engine:  https://arc.liv.ac.uk/trac/SGE