MPI problem with petsc-3.7.4

[seyed...@googlemail.com]

Hi,

I just started to use deal.II for my Ph.D. recently, so I am still not very familiar with the deal.II structure.

That is why I am learning by using Prof. Bangerth's amazing video lectures, which is why I would like now to run step-40 using MPI and p4est.

I have the following problem:

step-40 works perfectly, if I use mpirun with only 1 core, namely -np 1.

But for 8 cores it gets stuck at cycle 6 without any message. It runs forever. 

I assume it is related to PETSC since I am using version 3.7.4 and according to the deal.II Readme:

Note: The most recent version of PETSc that has been reported to be compatible with deal.II is version 3.6.0. If you use a later version than this and encounter problems, let us know.

Hence, I liked to let you know about this problem with PETSC 3.7.4.

Unfortunately, using PETSC 3.6.x results in a compilation error with my current Intel MPI library:

...

          FC x86_64/obj/src/sys/f90-mod/petscsysmod.o

          FC x86_64/obj/src/sys/classes/bag/f2003-src/fsrc/bagenum.o

          FC x86_64/obj/src/sys/f90-src/fsrc/f90_fwrap.o

          CC x86_64/obj/src/vec/is/utils/isio.o

f951: Fatal Error: Reading module ‘mpi’ at line 1 column 2: Unexpected EOF

compilation terminated.

gmakefile:174: die Regel für Ziel „x86_64/obj/src/sys/f90-mod/petscsysmod.o“ scheiterte

make[2]: *** [x86_64/obj/src/sys/f90-mod/petscsysmod.o] Fehler 1

make[2]: *** Auf noch nicht beendete Prozesse wird gewartet …

make[2]: Verzeichnis „/media/truecrypt1/programming/c++/libraries/petsc-3.6.0“ wird verlassen

/home/seyedali/programming/c++/libraries/petsc-3.6.0/lib/petsc/conf/rules:105: die Regel für Ziel „gnumake“ scheiterte

make[1]: *** [gnumake] Fehler 2

make[1]: Verzeichnis „/media/truecrypt1/programming/c++/libraries/petsc-3.6.0“ wird verlassen

**************************ERROR*************************************

  Error during compile, check x86_64/lib/petsc/conf/make.log

  Send it and x86_64/lib/petsc/conf/configure.log to petsc...@mcs.anl.gov

********************************************************************

makefile:24: die Regel für Ziel „all“ scheiterte

make: *** [all] Fehler 1

I would appreciate it, if anyone could give me a hint on how to solve this dilemma :)

Kind regards,

Seyed Ali Mohseni

[Bruno Turcksin]

Seyed,

On Monday, November 21, 2016 at 6:10:49 AM UTC-5, seyed...@googlemail.com wrote:

step-40 works perfectly, if I use mpirun with only 1 core, namely -np 1.

But for 8 cores it gets stuck at cycle 6 without any message. It runs forever. 

I assume it is related to PETSC since I am using version 3.7.4 and according to the deal.II Readme:

Note: The most recent version of PETSc that has been reported to be compatible with deal.II is version 3.6.0. If you use a later version than this and encounter problems, let us know.

Hence, I liked to let you know about this problem with PETSC 3.7.4.

You need to use the development version of deal.II to use the latest version of PETSc.

Best,

Bruno

[seyed...@googlemail.com]

I checked out  the latest development version of deal.II from the git repository. It should be compatible.

With regard to my second issue with installing PETSC 3.6.0, I have found out that with the GNU compiler and MPICH it works fine, but if I use Intel compiler and the Intel MPI library it gives me an error regarding f90 Fortran problems. I already checked mpif90 using test.f90 from Intel examples. Works like a charm. So I am wondering what could be the problem that PETSC 3.7.4 compilation works fine, but has issues with deal.II and PETSC 3.6.0 would work with deal.II, but cannot compile. 

[seyed...@googlemail.com]

This problem is giving me a headache. PETSC is one of the buggiest libraries ever...

If I don't touch anything, the compiler is detected as mpicc (I assume it is some wrapper based on gcc compiler?) and PETSC compiles fine.

The problem though is that I think mpicc is wrong since I use mpiicc from Intel for deal.II compilation. Now that is why I try to tell PETSC the following to do:

 

export PETSC_DIR=/path/petsc-3.7.4

export PETSC_ARCH=intel_mpi

./config/configure.py \

--with-cc=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiicc \

--with-cxx=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiicpc \

--with-fc=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiifort \

--with-shared-libraries=0 \

--known-mpi-shared-libraries=1 \

--with-x=0 --download-hypre=1 \

--with-debugging=1 \

--with-c++-support=1 \

--with-c-support=1 \

--with-python \

--with-pic=fPIC \

--LIBS='-ldl -lstdc++' \

But I receive this error message:

===============================================================================

             Configuring PETSc to compile on your system                       

===============================================================================

TESTING: checkCLibraries from config.compilers(config/BuildSystem/config/compilers.py:168)                                                                                                                   *******************************************************************************

                    UNABLE to EXECUTE BINARIES for ./configure 

-------------------------------------------------------------------------------

Cannot run executables created with FC. If this machine uses a batch system 

to submit jobs you will need to configure using ./configure with the additional option  --with-batch.

 Otherwise there is problem with the compilers. Can you compile and run code with your compiler '/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiifort'?

See http://www.mcs.anl.gov/petsc/documentation/faq.html#libimf

*******************************************************************************

I think without mpiicc the step-40 in deal.II will not work properly. I also tried to use MPICH and OPENMPI where both didnt'work since PETSC cannot detect the compilers correctly altthough I set PATH and LD_LIBRARY_PATH...

The PETSC version I am now using is 3.7.4 since I want to use the latest version. It should work and I hope to find the solution with your help.

Thanks in advance.

Kind regards,

S. A. Mohseni  

[Wolfgang Bangerth]

It is correct that you will not be happy with the result if you use different
MPI installations for PETSc and deal.II. Lots of strange error messages will
happen downstream.

What you ought to do is either:
1/ Figure out why PETSc believes that the fortran compiler specified with FC
does not work. You could, for example, look into the files that ./configure.py
creates and into which it posts the error messages it gets for the tests it
runs. There should also be the error message for the test that makes it think
that the fortran compiler doesn't work.

2/ Disable Fortran. PETSc is written in C. You should be able to just disable
those parts for which it wants to use Fortran. I'm sure there's a configure
switch for that. (--with-fortran-support=0 ?)

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[seyed...@googlemail.com]

No of course I have used the same MPI installations both for PETSc and deal.II. Since I used mpicc it uses the MPICH wrapper or the OPENMPI wrapper. Unfortunately, the mpicc always works with the GNU compiler gcc and I haven't succeeded using the Intel compiler for them.

Thank you for point 1/. I will check it out right now.

Regarding 2/: I already tried a build using --with-fc=0 which shuts off Fortran, but then the nightmare continues:

===============================================================================

             Configuring PETSc to compile on your system                       

===============================================================================

TESTING: checkCxxLibraries from config.compilers(config/BuildSystem/config/compilers.py:426)                                                                                                                 *******************************************************************************

                    UNABLE to EXECUTE BINARIES for ./configure 

-------------------------------------------------------------------------------

Cannot run executables created with C. If this machine uses a batch system 

to submit jobs you will need to configure using ./configure with the additional option  --with-batch.

 Otherwise there is problem with the compilers. Can you compile and run code with your compiler '/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiicc'?

See http://www.mcs.anl.gov/petsc/documentation/faq.html#libimf

*******************************************************************************

Now he complains about mpiicc...

Today I figured out how to compile by specifying exactly the lib and include folders of Intel's MPI and it suceeded, but it still uses the mpicc wrapper as compiler which is based on gcc and not on Intel's mpiicc.

The problem is I see no error message in step-40 at cycle 6, I just assume PETSc is not configured correctly. I even tried it outside of Eclipse. Same error.

But deal.II works fine and compiles like a charm. Only MPI has some issues...

[Wolfgang Bangerth]

On 11/24/2016 08:55 AM, seyedali88 via deal.II User Group wrote:
> *******************************************************************************
> UNABLE to EXECUTE BINARIES for ./configure
> -------------------------------------------------------------------------------
> Cannot run executables created with C. If this machine uses a batch system
> to submit jobs you will need to configure using ./configure with the
> additional option --with-batch.
> Otherwise there is problem with the compilers. Can you compile and run code
> with your compiler
> '/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiicc'?
> See http://www.mcs.anl.gov/petsc/documentation/faq.html#libimf
> *******************************************************************************

Well, so then you need to check why things don't seem to work with mpiicc
either. My suspicion is that your installation of the intel MPI libraries is
not complete, or that some paths are not set up as required.

[seyed...@googlemail.com]

Surprisingly, my MPI installation from Intel works. I tried some examples with mpirun and mpiicc to validate it. Even deal.II was compiled completely without errors using the same Intel compiler. I assume there has to be a library specified since PETSc cannot detect the compiler, it's PETSc problem since other software detect it.

To 1/: I checked it out right now. it seems there is some linker problem. Configure.log tells me it is the "checkCxxLibraries" TEST.

I attached the configure.log file. 

Maybe someone can help and figure out where the error is hidden.

[Wolfgang Bangerth]

On 11/24/2016 09:36 AM, seyedali88 via deal.II User Group wrote:

I don't see the error in there. Is there a separate file that only lists the
error messages from failing tests?

[seyed...@googlemail.com]

Good morning,

Sorry for the delayed answer, yesterday I was just too busy and sleepy :)

I was only able to find the RDict.log and a make.log, but the make.log is a corrupt file it won't open correctly.

Additionally, there is a scaling.log which is from a previous successful compilation I assume.

I attached both.

Maybe someone knows where PETSc saves its error files. I already googled it, but no success.

[Wolfgang Bangerth]

On 11/25/2016 12:48 AM, seyedali88 via deal.II User Group wrote:
>
> Sorry for the delayed answer, yesterday I was just too busy and sleepy :)
> I was only able to find the RDict.log and a make.log, but the make.log is a
> corrupt file it won't open correctly.
> Additionally, there is a scaling.log which is from a previous successful
> compilation I assume.
> I attached both.

Well, only RDict.log is attached, and that's not it. See if you can attach
make.log.

That said, you'll need to learn to look for these sorts of errors. PETSc is a
package that is written by people who know what they're doing. If the
configure step finds a problem, there likely is a real problem in your system,
whatever it may be. Play with their configure options, starting with the
simplest setup you can think of, and try to find out which step it is that
fails and why.

[seyed...@googlemail.com]

The make.log is from a previous successful compilation. First PETSc configures and checks, than it builds the files, so I assume make.log will not help in that case. Unfortunately, there is a linking issue with shared libraries that prevents PETSc from detecting the mpiicc correctly. I will try now to run deal.II on our cluster and also on my laptop maybe. Maybe there is a bug with Intel, but if it has the same issue there: I must assume that the Intel compiler is not the problem and merely admit that PETSc cannot find it. However, I already tried many configurations with PETSc, read a lot in google forums etc. There is a small stupid detail I oversee probably :)

Thank you Prof. Bangerth and also Bruno Turcksin.

I will keep you updated whenever I figure out the solution ;)

Maybe another fellow could have similar problems in future.    

[Toby D. Young]

You have to try this piece-by-piece to get a result. Certainly I am
confused as to which configurations you have tried and which work and
which don't, and if configure or linking fails... :-)

Try as Wolfgang suggested and start with a minimal configuration. Try
this, for example,

./configure
--with-cc=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiicc
--with-cxx=/opt/intel/compilers_and_libraries_2016.3.210/linux/mpi/intel64/bin/mpiicpc
--with-x=0
--with-c++-support=1

That may be the minimal configuration to get PETSc worable with most of
deal.II. Of course, when you try to reconfigure PETSc with something
new, remember to *remove the old configuration*, or *archive it
somewhere*!

By-the-way. Are you exporting PETSC_ARCH and PETSC_DIR every time you
configure? It is safe to put these variables in your ~/.bashrc file (or
equivalent). Something similar to,

export PETSC_DIR=/path/to/petsc ;
export PETSC_ARCH=petsc_arch ;
export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib: ${LD_LIBRARY_PATH}:

Configure and compile PETSc and see if that wors with deal.II. Then you
can extend to add Hypre and friends...

Best,
Toby


Dnia 2016-11-25, o godz. 04:32:05
"seyedali88 via deal.II User Group" <dea...@googlegroups.com>
napisał(a):

[seyed...@googlemail.com]

I found the solution. The compilation on the cluster worked due to an older Intel compiler version. From my judgement, the newest update of Intel compiler 2016 is not compatible with the PETSc 3.7.4 package. Maybe someone can confirm that?

Now I will try to compile deal.II and see how it affects the step-40 example.

@Tobi Young: Thanks, but I already tried your ideas out and they weren't successful. But still, thanks.

[seyed...@googlemail.com]

I reported the problem to the PETSc team. Now I have setup my desktop system with Intel compiler 2017 which compiled now without problems and the step-40 works better now, but still gives an error. On the cluster it worked perfectly. Is MPI only suited for cluster systems or am I able to run step-40 also with my i7 desktop with 8 cores. The problem is it gives me an error in cycle 6 for 8 cores and for 4 cores it gives an error in cycle 7. What is wrong?

[Bruno Turcksin]

On Tuesday, November 29, 2016 at 11:13:07 AM UTC-5, seyed...@googlemail.com wrote:

I reported the problem to the PETSc team. Now I have setup my desktop system with Intel compiler 2017 which compiled now without problems and the step-40 works better now, but still gives an error. On the cluster it worked perfectly. Is MPI only suited for cluster systems or am I able to run step-40 also with my i7 desktop with 8 cores. The problem is it gives me an error in cycle 6 for 8 cores and for 4 cores it gives an error in cycle 7. What is wrong?

It should work on your desktop. Do you have multiple versions of MPI installed on your machine?

Best,

Bruno

[seyed...@googlemail.com]

I have only Intel MPI installed at the moment. It's strange that it works for 2 or 4 cores. Single core works flawlessly.

Maybe I should try the older compiler and see, if it works...

Best regards,

S. A. Mohseni

[Bruno Turcksin]

2016-11-29 16:25 GMT-05:00 seyedali88 via deal.II User Group
<dea...@googlegroups.com>:

> I have only Intel MPI installed at the moment. It's strange that it works
> for 2 or 4 cores. Single core works flawlessly.
> Maybe I should try the older compiler and see, if it works...

I don't think it's a compiler problem. However, I have had problems
with Intel MPI on some other projects. Can you uninstall Intel MPI and
try OpenMPI or MPICH instead. You can use the intel compiler with
OpenMPI and MPICH.

Best,

Bruno

[seyed...@googlemail.com]

Can you tell me which one is faster, MPICH or OpenMPI? I choose Intel MPI, because I assumed it is the fastest available. Googled about it, but the opinions about it are random.

Best Regards,

S. A. Mohseni

[Bruno Turcksin]

2016-11-30 2:40 GMT-05:00 seyedali88 via deal.II User Group
<dea...@googlegroups.com>:

> Can you tell me which one is faster, MPICH or OpenMPI? I choose Intel MPI,
> because I assumed it is the fastest available. Googled about it, but the
> opinions about it are random.

There are no difference. That's not where your code spend most of its
time (or else something is very wrong) so even if one is faster, you
won't see the difference.

Best,

Bruno

[seyed...@googlemail.com]

I'm now using MPICH and Intel compiler 2017. Everything works perfectly fine. Thanks to everyone ;)