Problem of compiling DIRAC12.6 using MPI compiler mpiifort and mpiicc with 64bit option

[Malaya Kumar Nayak]

Dear Dirac users,

I first compiled the DIRAC12.6 in sequential mode using the command

./setup --fc=ifort --cc=icc --int64

It compiled successfully, and I also did all the tests in sequential mode.
 
Later, I tried to compile the same DIRAC12.6 in parallel mode using the
following command line:

./setup --fc=mpiifort --cc=mpiicc --int64

In this case, I could not compile succesfully, and got the following error (pasted below)


Your support and help in this regard will be highly useful for me

Best regards,
Malaya K. Nayak

[malayak@ycn1 DIRAC-12.6-Source]$ ./setup --fc=mpiifort --cc=mpiicc --int64

 FC=mpiifort CC=mpiicc CXX=mpicxx cmake -DENABLE_MPI=ON -DENABLE_SGI_MPT=OFF -DENABLE_BLAS=ON -DENABLE_LAPACK=ON -DENABLE_TESTS=OFF -DENABLE_64BIT_INTEGERS=ON -DCMAKE_BUILD_TYPE=Release /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source

-- No external LAPACK library found
-- Using own LAPACK implementation (slow)
-- No external BLAS library found
-- Using own BLAS implementation (slow)
-- mpi.mod matches current compiler, setting -DUSE_MPI_MOD_F90
-- MPI-2 support found
-- The Fortran compiler identification is Intel
-- The C compiler identification is Intel 13.0.0.20121010
-- The CXX compiler identification is GNU 4.4.6
-- Check for working Fortran compiler: /opt/intel/impi/4.1.0.024/intel64/bin/mpiifort
-- Check for working Fortran compiler: /opt/intel/impi/4.1.0.024/intel64/bin/mpiifort  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/intel/impi/4.1.0.024/intel64/bin/mpiifort supports Fortran 90
-- Checking whether /opt/intel/impi/4.1.0.024/intel64/bin/mpiifort supports Fortran 90 -- yes
-- Check for working C compiler: /opt/intel/impi/4.1.0.024/intel64/bin/mpiicc
-- Check for working C compiler: /opt/intel/impi/4.1.0.024/intel64/bin/mpiicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /opt/intel/impi/4.1.0.024/intel64/bin/mpicxx
-- Check for working CXX compiler: /opt/intel/impi/4.1.0.024/intel64/bin/mpicxx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found Git: /usr/bin/git 
-- Found MPI_C: /usr/lib64/libdl.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpi.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpigf.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpigi.a;/usr/lib64/librt.so;/usr/lib64/libpthread.so 
-- Found MPI_CXX: /usr/lib64/libdl.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpigc4.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpi.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpigf.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpigi.a;/usr/lib64/librt.so;/usr/lib64/libpthread.so 
-- Found MPI_Fortran: /usr/lib64/libdl.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpi.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpigf.so;/opt/intel/impi/4.1.0.024/intel64/lib/libmpigi.a;/usr/lib64/librt.so;/usr/lib64/libpthread.so 
-- Performing Test MPI_COMPATIBLE
-- Performing Test MPI_COMPATIBLE - Success
-- Performing Test MPI_COMPILER_MATCHES
-- Performing Test MPI_COMPILER_MATCHES - Success
-- Performing Test MPI_ITYPE_MATCHES
-- Performing Test MPI_ITYPE_MATCHES - Success
-- Performing Test MPI_2_COMPATIBLE
-- Performing Test MPI_2_COMPATIBLE - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/build

   configure step is done
   now you need to compile the sources

   to compile with configured parameters (recommended):
   $ cd build
   $ make

   to modify configured parameters and then compile:
   $ cd build
   $ ccmake ..
   $ make
[malayak@ycn1 DIRAC-12.6-Source]$ cd build
[malayak@ycn1 build]$ make
Scanning dependencies of target gen_fortran_interface_h
[  0%] Building C object CMakeFiles/gen_fortran_interface_h.dir/utils/gen_fortran_interface_h.c.o
Linking C executable gen_fortran_interface_h
[  0%] Built target gen_fortran_interface_h
[  0%] Generating fortran_interface.h
Scanning dependencies of target dirac
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/embedding/emb_types.F90.o
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_parameters.F90.o                                                  
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_control.F90.o                                                     
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_tracker.F90.o                                                     
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_track_if.F90.o                                                    
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_errorhandler.F90.o                                                
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_internal_init.F90.o                                               
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_internal_ada_1d.F90.o                                             
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_internal_ada_2d.F90.o                                             
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_internal_ada_3d.F90.o                                             
[  0%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_internal_ada_4d.F90.o                                             
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_legacy_dirac.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/gp/memory_allocator.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/gp/quaternion_algebra.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/x2c/x2c_cfg.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/x2c/x2c_common_block_interface.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/x2c/x2c_io_interface.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/x2c/x2c_utility_routines.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/x2c/x2c_import_export_C1_matrices_ao.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/x2c/x2c_setup_defining_h1_4c_matrix_ao2onmo.F90.o
[  1%] Building Fortran object CMakeFiles/dirac.dir/src/interface_mpi/integer_kind_mpilib.F90.o
[  2%] Building Fortran object CMakeFiles/dirac.dir/src/interface_mpi/interface_to_mpi.F90.o
/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/src/interface_mpi/interface_to_mpi.F90(607): error #6633: The type of the actual argument differs from the type of the dummy argument.   [STATUS_CONTAINER]
    call mpi_probe(rank   , message_id   , communicator   , status_container   , ierr   )
------------------------------------------------------------^
/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/src/interface_mpi/interface_to_mpi.F90(643): error #6633: The type of the actual argument differs from the type of the dummy argument.   [STATUS_CONTAINER]
    call mpi_iprobe(rank   , message_id   , communicator   , x   , status_container   , ierr   )
-------------------------------------------------------------------^
/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/src/interface_mpi/interface_to_mpi.F90(673): error #6633: The type of the actual argument differs from the type of the dummy argument.   [STATUS_CONTAINER]
    call mpi_test(request   , x   , status_container   , ierr   )
------------------------------------^
/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/src/interface_mpi/interface_to_mpi.F90(737): error #6633: The type of the actual argument differs from the type of the dummy argument.   [STATUS_CONTAINER]
    call mpi_wait(request   , status_container   , ierr   )
------------------------------^
/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/src/interface_mpi/interface_to_mpi.F90(2944): error #6633: The type of the actual argument differs from the type of the dummy argument.   [STATUS_ARRAY]
    call mpi_get_count(status_array   , datatype   , elements   , ierr   )
-----------------------^
compilation aborted for /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/src/interface_mpi/interface_to_mpi.F90 (code 1)
make[3]: *** [CMakeFiles/dirac.dir/src/interface_mpi/interface_to_mpi.F90.o] Error 1
make[2]: *** [CMakeFiles/dirac.dir/src/interface_mpi/interface_to_mpi.F90.o.provides] Error 2
make[1]: *** [CMakeFiles/dirac.dir/all] Error 2
make: *** [all] Error 2
[malayak@ycn1 build]$

--
Malaya K. Nayak, Ph.D.
Bhabha Atomic Research Centre,
Trombay, Mumbai - 400085, India.

[Stefan Knecht]

dear Malaya,

two comments/tips:

a. you are using ifort/IntelMPI, so i am pretty sure you also have Intel MKL math libraries that you want to use. ;)
check your MKL installation, i assume you will find it at: /opt/intel/mkl/XXX-your-version-number/lib/intel64
apparently, our adavnced MATH library detection did not work in your case and you can help it by using

$ export MATH_ROOT=/opt/intel/mkl/XXX--your-version-number/lib/intel64

which will tell cmake where to find the MKL libs.

b. i have seen this problem with IntelMPI before and what has helped in my case was to do the following:

- edit cmake/parallel-environment/ConfigParallelEnvironment.cmake in the Dirac directory
- change the following lines:

    if(MPI_COMPILER_MATCHES)
        message("-- mpi.mod matches current compiler, setting -DUSE_MPI_MOD_F90")
        add_definitions(-DUSE_MPI_MOD_F90)
    else()
        message("-- WARNING: mpi.mod compiled with different compiler")
    endif()

to

#    if(MPI_COMPILER_MATCHES)
#        message("-- mpi.mod matches current compiler, setting -DUSE_MPI_MOD_F90")
#        add_definitions(-DUSE_MPI_MOD_F90)
#    else()
#        message("-- WARNING: mpi.mod compiled with different compiler")
#    endif()

- rerun "setup"

- compile happily. you will now use the F77 interface of the (Intel) MPI library instead of the F90 one but that's equally fine.

with best regards,

stefan

p.s.: address Dirac-user questions ONLY (!) o "dirac...@googlegroups.com". all other groups/lists are out-of-use respectively not supported anymore!

--
You received this message because you are subscribed to the Google Groups "dirac-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dirac-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Stefan Knecht
Laboratory of Physical Chemistry - The Reiher Group -
ETH Zuerich
HCIG 230
Wolfgang-Pauli-Str. 10
CH-8093 Zuerich
Switzerland

phone: +41 44 633 46 46   fax: +41 44 633 15 94
email: stefan...@phys.chem.ethz.ch
web: http://www.theochem.uni-duesseldorf.de/users/stefan/index.htm
     http://www.reiher.ethz.ch/people/knechste

[André Luiz Fassone Canova]

removel all packages *mpi* for your distro.

compile

and

install

check if lib are installed in directory /...../....../lib64

in last realese of intel suite, ifort, icc and mkl are present.

one link are created after install, /opt/intel/mkl

:)

André Luiz Fassone Canova
alfc...@gmail.com
alfc...@ig.com.br

"Existem 10 tipos de pessoas: as que entendem código binário e as que não entendem!"
"There are 10 kinds of people: those who understand binary code and those who do not understand!"

2013/4/18 Stefan Knecht <stefan...@phys.chem.ethz.ch>

[Stefan Knecht]

dear Andre,

On 18/04/13 14.56, André Luiz Fassone Canova wrote:

removel all packages *mpi* for your distro.

sorry but what you write doesn't make any sense... :( why would you do that? i assume Malaya has paid extra money for the IntelMPI libraries!
they are nowadays available by default with 32-bit INTEGER and 64-bit INTEGER in one installation.

with best regards,

stefan

[André Luiz Fassone Canova]

Stefan,

Sorry if I not totaly clear im my previous message :)

Resources for research in Brazil are scarce ... very scarce.
In particular our group seeks to invest in hardware and use the maximum possible free tools.

Therefore we use quite the GNU tools, in this case I opted for the Intel.

There is a free version of the suite of compilers, see:
http://software.intel.com/en-us/non-commercial-software-development

however intel mpi is not free, but we OpenMPI! Best choice ....

LJR said it download the openmpi 1.6.4 and compile it, but the file mpd.info gave some kind problem with the build version.

The procedure to solve this is to remove any MPI system, to avoid the problem version ...

check the installation directory (the default openmpi is /usr/local/lib and not /usr/local/lib64)

recompile the DIRAC and ready.

Regarding the Intel suite for free, see:

What's Included:

     Intel ® C, C + + and FORTRAN Compilers>
     Intel ® Threading Building Blocks 4.1>
     Intel ® Math Kernel Library 11.0>
     Intel ® Integrated Performance Primitives 7.1>
     Intel ® Advisor XE 2013>
     Intel ® VTune ™ Amplifier XE 2013>
     Intel ® Inspector XE 2013>
     Static Analysis>

In fact, within the installation directory of the intel suite has several files automatically to set environment variables for Fortran, C, C + + and others. The names of the files are of type: compilevars.sh and the like.

In my case I created intel.sh the file in the directory /etc/profile.d with:
#! / bin / sh
source / opt / intel / bin / compilervars.sh Intel64
source / opt/intel/composer_xe_2013/mkl/bin/mklvars.sh Intel64
source / opt / intel / advisor_xe / advixe-vars.sh
source / opt / intel / inspector_xe / inspxe-vars.sh
source / opt / intel / vtune_amplifier_xe / amplxe-vars.sh

After this is log into the system and have fun :)

PS:
1. Procedure for compile MPI with Intel (http://wiki.chem.vu.nl/dirac/index.php/How_to_build_MPI_libraries_for_64-bit_integers)

./configure --prefix=/path/to/openmpi CXX=icpc CC=icc F77=ifort FC=ifort FFLAGS=-i8 FCFLAGS=-i8

2. Procedure for compile Lapack, Blas, Atlas and other (not necessary)

export MATH_ROOT=/opt/intel/mkl

Dirac setup, atomically, recognizes the 64-bit directory.


I did some sense now?

Thanks for all

[Radovan Bast]

hi Andre,

you don't have to remove conflicting MPI versions

if you set environment variables correctly.

it is no problem to have many MPI installations

on the same machine and switch between them

by carefully setting PATH and LD_LIBRARY_PATH.

  radovan

[André Luiz Fassone Canova]

yes, Radovan has full reason.

But for a personal system, why two intalações MPI?
One for GNU and one for INTEL ... remove the GNU version is more practical.
Previously I had created a lot of scprits to be loaded manually (type load module intelcompiler (or intelcompiler) intelmpi load module (or gnumpi).

But again, in a personal system, why waste time?

IMHO....

Thanks for all

André Luiz Fassone Canova
alfc...@gmail.com
alfc...@ig.com.br

"Existem 10 tipos de pessoas: as que entendem código binário e as que não entendem!"
"There are 10 kinds of people: those who understand binary code and those who do not understand!"

2013/4/18 Radovan Bast <radova...@gmail.com>

[Malaya Kumar Nayak]

Dear Stefan,

Thank you very much for your suggestion!!!

On Thu, Apr 18, 2013 at 6:20 PM, Stefan Knecht <stefan...@phys.chem.ethz.ch> wrote:

dear Malaya,

two comments/tips:

a. you are using ifort/IntelMPI, so i am pretty sure you also have Intel MKL math libraries that you want to use. ;)
check your MKL installation, i assume you will find it at: /opt/intel/mkl/XXX-your-version-number/lib/intel64
apparently, our adavnced MATH library detection did not work in your case and you can help it by using

$ export MATH_ROOT=/opt/intel/mkl/XXX--your-version-number/lib/intel64

which will tell cmake where to find the MKL libs.

The option "a" is not helpful in my case. I mean whether I use the Dirac inbuilt library or the MKL library, the
problem remains same.

b. i have seen this problem with IntelMPI before and what has helped in my case was to do the following:

- edit cmake/parallel-environment/ConfigParallelEnvironment.cmake in the Dirac directory
- change the following lines:

    if(MPI_COMPILER_MATCHES)
        message("-- mpi.mod matches current compiler, setting -DUSE_MPI_MOD_F90")
        add_definitions(-DUSE_MPI_MOD_F90)
    else()
        message("-- WARNING: mpi.mod compiled with different compiler")
    endif()

to

#    if(MPI_COMPILER_MATCHES)
#        message("-- mpi.mod matches current compiler, setting -DUSE_MPI_MOD_F90")
#        add_definitions(-DUSE_MPI_MOD_F90)
#    else()
#        message("-- WARNING: mpi.mod compiled with different compiler")
#    endif()

However, when I made the changes as per option "b" it is compiling smoothly and some tests are
already passed. I will let you know if some tests fails.

- rerun "setup"

- compile happily. you will now use the F77 interface of the (Intel) MPI library instead of the F90 one but that's equally fine.

with best regards,

stefan

Thank you once again

p.s.: address Dirac-user questions ONLY (!) o "dirac...@googlegroups.com". all other groups/lists are out-of-use respectively not supported anymore!

Ok, I will to "googlegroups" only from now onwards.

[Stefan Knecht]

dear Malaya,

regarding tip "a". of course it doesn't matter for your MPI problem but using MKLs or in-house Dirac math libraries DOES matter when it comes to timings
for some serious production calculations. ;)

with best regards,

stefan

[Malaya Kumar Nayak]

Dear Stefan,

Of course, you are perfectly right. I noticed that some tests are getting failed when I run the command

make test all

inside the build directory.

I am pasting below the output of one of the failed test case.

Best,
Malaya

Those tests are:

 6/84 Test  #6: bss_energy .................................***Failed   25.12 sec
      Start  7: cc_energy_and_mp2_dipole
 7/84 Test  #7: cc_energy_and_mp2_dipole ...................***Failed  201.94 sec
      Start  8: cc_linear
 8/84 Test  #8: cc_linear ..................................***Failed   26.46 sec

If I see the LastTest.log.tmp file, it is saying like this:

6/84 Testing: bss_energy
6/84 Test: bss_energy
Command: "/usr/bin/python" "/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/runtest" "--showdiff" "--builddir=/home/e
xternal/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/build" "--verbose" "--tests=bss_energy"
Directory: /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/build
"bss_energy" start time: Apr 19 15:52 IST
Output:
----------------------------------------------------------

 **** dirac-executable stderr console output : ****
Attempting to use an MPI routine before initializing MPI

directory: /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/test/bss_energy
   inputs: Z80H.lsym.dir.mol  &  HF.bss.cc.inp

 **** dirac-executable stderr console output : ****
Attempting to use an MPI routine before initializing MPI

directory: /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/test/bss_energy
   inputs: Z80H.lsym.dir.mol  &  HF.bss_sfb.cc_cisd.inp

 **** dirac-executable stderr console output : ****
Attempting to use an MPI routine before initializing MPI

directory: /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/test/bss_energy
   inputs: Z80H.lsym.dir.mol  &  HF.dk2_sfb.pnuc.cc_cisd.inp

           result                           test                                 inp                   mol/xyz   time

[ 33%]      CRASH                     bss_energy                           HF.bss.cc             Z80H.lsym.dir 00m14s
[ 66%]      CRASH                     bss_energy                  HF.bss_sfb.cc_cisd             Z80H.lsym.dir 00m04s
[100%]      CRASH                     bss_energy             HF.dk2_sfb.pnuc.cc_cisd             Z80H.lsym.dir 00m05s

summary:     0 ok
             3 crashed

total time: 00h00m24s

<end of output>
Test time =  25.12 sec
----------------------------------------------------------
Test Failed.
"bss_energy" end time: Apr 19 15:53 IST
"bss_energy" time elapsed: 00:00:25
----------------------------------------------------------

[Malaya Kumar Nayak]

Dear Stefan,

In the test directory, the corresponding output file has the following error.

Best Malaya,
 Transformation started at : Fri Apr 19 15:52:45 2013

* REACMO: Coefficients read from file DFCOEF - Total energy: -9511.28255377852838
* Heading :  LiH                                             Fri Apr 19 15:52:45 2013
SCR        scr.thr.    Step1    Step2  Coulomb  Exchange    CPU-time
SOfock:LL  1.00D-12   15.59%   20.70%    0.75%    2.21%   0.00699902s

* REAFCK: Fock matrix read from file /home/external/iiap/malayak/scratch/malayak/DIRAC_HF.bss.cc_
* Heading :  LiH                                             Fri Apr 19 15:52:44 2013


 Core energy (includes nuclear repulsion) :          -6389.1828036209
 - Electronic part :                                 -6410.3498919529
   - One-electron terms :                            -6470.3270694225
   - Two-electron terms :                               59.9771774696


 MOLFDIR file MRCONEE is written
 - Integral class 1 : (LL|??)
   - Beginning task   1 of   4 after      0. seconds and      0. CPU-seconds
   - Beginning task   2 of   4 after      0. seconds and      0. CPU-seconds
   - Beginning task   3 of   4 after      0. seconds and      0. CPU-seconds
   - Beginning task   4 of   4 after      0. seconds and      0. CPU-seconds
 - Integral class 2 : (SS|??)

 Node    0 finished first half transformation           75457 HT integrals written ( 74.56%,      0.00 GB)

 <<< Starting 2HT on node                      0


 >>> Finished 2HT on node                      0
 >>> Time used in 2HT_all is   3.34 seconds

 - Binary  file MDCINT was written.

 ====  below this line is the stderr stream  ====

Attempting to use an MPI routine before initializing MPI

[Stefan Knecht]

dear Malaya,

"make test" performs only sequential runtests even when you compiled the code in parallel that's why some of the tests may fail if they enter MPI routines.
you can check whether the MPI implementation works for that particular test case with the following command in your top Dirac directory:

$ /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/runtest --showdiff --builddir=/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/build --verbose --tests=bss_energy --mpi=2

note the "--mpi=2" to run the test in parallel with 2 processes.

with best regards,

stefan

[Malaya Kumar Nayak]

Dear stefan,

Yes, it is passing the test!

By the way, what is the meaning of --verbose?
Is it necessary to use verbose?

Even if I use the command

./runtest --mpi=2 --all

inside the directory "/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/"
without specifying the --builddir=....

it is passing the earlier failed tests.

Is this ok?

Best,
Malaya

[Malaya Kumar Nayak]

Dear Stefan,

Another thing is that, most of the DFT related jobs are getting crashed along with the test "densit_at_nuclei, for example:

Here is the output of density_at_nuclei test:

[malayak@ycn1 DIRAC-12.6-Source]$  /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/runtest --showdiff --builddir=/home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/build --verbose --tests=density_at_nuclei --mpi=2

           result                           test                                 inp                   mol/xyz   time

 **** dirac-executable stderr console output : ****

  Master node : --- SEVERE ERROR, PROGRAM WILL BE ABORTED ---        
 programming error in distribution of points in parallel DFT
application called MPI_Abort(MPI_COMM_WORLD, -9999) - process 0

directory: /home/external/iiap/malayak/DIRAC_12_PARALLEL/DIRAC-12.6-Source/test/density_at_nuclei
   inputs: CO.mol  &  Rhonuc.inp
[100%]      CRASH              density_at_nuclei                              Rhonuc                        CO 00m02s

summary:     0 ok
             1 crashed

total time: 00h00m02s

[malayak@ycn1 DIRAC-12.6-Source]$

[Malaya Kumar Nayak]

p.s. similar thing is happening for the test case "dft_ac", "dft_alda", "dft_geo_opt and many other.

[Radovan Bast]

dear Malaya,

  to explain the DFT error: the DFT code divides the grid into batches

and sends to each MPI worker a portion of the grid. at the end each worker

reports the number of grid points that it used and the sum of the grid points

is compared to the initial number of grid points. this sanity test fails on your installation

which means that grid points are not correctly distributed. at this moment i cannot

say why this happens. i will try to reproduce your installation.

i will write as soon as i know more about it.

best regards,

  radovan

[Radovan Bast]

dear Malaya,

i can reproduce the error that you observe. we are working on a solution.

best regards,

  radovan

[Malaya Kumar Nayak]

Dear Radovan,

Thanks for your cooperation. Pl. let me know if you find some solution.
Other than that, it would be nice if the compilation error without editing
the following file as suggested by Stefan (in earlier mail)

cmake/parallel-environment/ConfigParallelEnvironment.cmake

can be fixed for 64 bit option, i.e. commenting some lines in that file.

Looking forward to the solutions.

Thanks & regards,
Malaya

[Radovan Bast]

dear Malaya,

the compilation problem can be fixed easily (i have a local fix).

with Stefan we are working on a solution for the DFT problem.

as soon as we have it we will package both into the 12.7 patch.

best regards,

  radovan

[Stefan Knecht]

dear Malaya,

in case you need the parallel 64-bit integer really urgent, please use OpenMPI
which works out-of-the-box once you installed OpenMPI with 64-bit integer support.
Please follow our guideline (http://diracprogram.org/doc/release-12/installation/int64/mpi.html).
May i ask why you need 64-bit integers?
if you can answer one of the question on this webpage with "yes" then you will need them.

http://diracprogram.org/doc/release-12/installation/int64/faq.html

the IntelMPI libraries apparently have a BUG concerning the use of a special MPI function when
running with 64-bit integers. we use this particular function mainly in the DFT code.
We will provide a fix with the 12.7 patch but in the long run we hope Intel
will actually correct their MPI library.

with best regards,

stefan

[Malaya Kumar Nayak]

Dear Stefan,

I am not a in very urgent need of installing 64-bit. Even if it takes some time to fix the bug then
also ok. Other than that, I really don't use much the DFT routines. I mainly use CI and CC part
of Dirac program package, and its interface with Kallay's MRCC.

Actually, I install the MRCC with 64-bit integer type, and hence the data generated from Dirac
should also be 64-bit integer type for compatibility point of view. This is the main reason I tried
tried for the 64-bit integer, and whatever problem I faced in installing/running, I brought into the
notice of the users and developers of Dirac.

Anyway, if it takes some time to fix these issues then also it's fine with me. I will continue with
the 32 bit-integer type at present and wait till the release of Dirac 12.7 version.

Thanks & regards,
Malaya

[Stefan Knecht]

dear Malaya,

thanks for the details. you can safely use the CI and CC parts as they are with your present 64-bit integer installation, no need to wait. :) 

as I mentioned before, it's an Intel BUG you are hitting in the DFT part. 

we have to think whether we are going to "fix" our OK code or wait for Intel to fix their WRONG/BUGGY code...

with best regards,

stefan

[Malaya Kumar Nayak]

Dear Stefan,

It's fine!

Thanks for your and Radovan's support.

Best regards,
Malaya