simultaneous accessing of a fortran code by number of jobs

[habzo...@gmail.com]

hello,

Is it possible to compile a fortran code simultaneously by more than
one job ? For example, I have a shell script that repeatedly calls a
fortran program in a loop:

**********
e=0
while [ $e -le 60 ]
do

mv big_$e big.in
ifort -o user user.f90
mv user user_$e
e=`expr $e + 1`

done
************

the file 'big.in' is used by user.f90.

I want to divide this loop into two jobs (e = 0 to 30, and e = 31 to
60) and run them simultaneosuly, making sure that the output data
files are separate. But since they will be accessing and try to
compile the same code (user.f90) at the same time, will it be a
problem ?

Thanks
Ravi

[Dave Allured]

Louis Krupp wrote:

> There's something missing here. Your script compiles the program, but
> doesn't actually run it.
>
> If the program reads big.in and writes to standard output, you could do
> something like this:
>
> **********
> ifort -o user user.f90 # Just do this once!

> e=0
> while [ $e -le 60 ]
> do
>
> mv big_$e big.in

> user

> e=`expr $e + 1`
>
> done
> ************
>

> Note that this will destroy all big_* files except for the last one,
> big_60, which will be named big.in after you're done.
>
> And, no, you can't split this into two scripts and run them
> simultaneously using the same big.in.
>
> If your program reads from standard input instead of opening a file
> called big.in, you might be able to do this:
>
> **********
> ifort -o user user.f90

> e=0
> while [ $e -le 60 ]
> do
>

> user < big_$e

> e=`expr $e + 1`
>
> done
> ************
>

> This will leave all of your big_* files intact, *and* you can split the
> script and run the bits simultaneously as long as the program doesn't
> open and update other files.
>
> In answer to your original question, having two scripts simultaneously
> creating an executable program 'user' in the same directory is not a
> good idea.
>
> (Note that this isn't strictly a Fortran question.)
>
> Louis

One minor change:

**********
ifort -o user user.f90

e=0
while [ $e -le 60 ]
do

echo big_$e | user

e=`expr $e + 1`
done
************

This version passes only the *name* of the input file to the user
program. Then the program reads the specific input file the same way it
did before, rather than standard in. This works for all file modes, not
just ordinary sequential text.

Get the file name like this:

character*200 inname
read (*,'(a)') inname

This still doesn't get to the OP's original request to run multiple
copies simultaneously. As written above, the "user" program would just
run sequentially, one at a time, in the loop for 0 to 60.

For two simultaneous runs, just make two copies of the *script*, one for
0-30, the other for 31-60. Run them in two different windows, and you
should be all set. As Louis indicated, you will have more to fix if the
program copies want to write to any output files that are the same.
HTH.

--Dave

[habzo...@gmail.com]

> For two simultaneous runs, just make two copies of the *script*, one for
> 0-30, the other for 31-60.  Run them in two different windows, and you
> should be all set.  As Louis indicated, you will have more to fix if the
> program copies want to write to any output files that are the same.
> HTH.

---------------

Thanks. To avoid any overwriting, I created separate directories for
the two sets to run the program. The output files then will be written
in their respective directories, then. This was my last option. In any
case, if I run in a *separate* window in the *same* directory, does it
work ?

Thanks
Ravi

[Dave Allured]

It *might* work. You have not given us any information about how you
write output files. We need to know your output file strategy to be
able to give a fair answer to that question.

Does the user program write to output or temporary files with the same
names for different inputs, or are the output file names always
different for each run? Separate output files is the conventional
strategy for parallel runs.

Some Fortran compilers *may* have non-standard extensions that allow
simultaneous file writes to different parts of the same output file.
This is rather tricky if allowed at all. It may have value for some
applications. I have never tried it myself. If I needed to, I would
only try it with one of the record-oriented formats, or else look for
some kind of file support library.

--Dave

[Steven Correll]

Assuming you are using the "bash" shell on Linux or some other Unix-
like operating system (the syntax of your "while" loop suggests that),
you don't need to use multiple windows. If you put "&" at the end of a
command, then the shell will continue without waiting for the command
to finish, so (for example) the following would run 60 compilations in
parallel:

e=0
while [ $e -le 60 ]
do

mkdir ./$e
ln big_$e ./$e/big.in
(cd $e; ifort -o user_$e ../user.f90) &

e=`expr $e + 1`

done

While the compilations are running in parallel, you need to keep them
from interfering with each other, so I made additional changes:

1. Create a different subdirectory for each compilation
2. Use "ln" rather than "mv" to create big.in so the original file
isn't destroyed in case you need to repeat the process
3. Inside a subshell (created by "()"), change to the subdirectory and
run the compiler (the "cd" affects only the commands following it
inside the "()".)

The same technique will work for executing compiled programs in
parallel. Of course, you probably realize that there are diminishing
returns to running many tasks in parallel on a single machine. And for
more complicated compilations you may want to investigate the parallel
features of GNU "make".