PETSC_HAVE_MUMPS not set correctly ?

[Daniel Jodlbauer]

Dear all!

I'm trying to use the MUMPS direct solver via PetscWrappers::SparseDirectMUMPS but I keep running into a minor issue. When calling the solve function, it always throws an error, i.e takes the wrong path

in petsc_solver.cc:680

void SparseDirectMUMPS::solve()
{
#ifdef PETSC_HAVE_MUMPS
... do solving stuff
... is never going to be called when inside the cc file :(
#else
   Assert(mumps not found)
#endif

It seems the PETSC_HAVE_MUMPS flag was not recognized during compilation of deal.II. When I copy the solving code into the header "petsc_solver.h", everything works, so PETSc seems to be configured correctly with MUMPS.

Do I have to set a specific flag during installation to enable petsc+mumps in deal.II? (btw it works through TrilinosWrappers::SolverDirect and selecting MUMPS)

Since I am not too familiar with the whole cmake stuff, I have no idea how to fix / find the error in the installation (except for copying the code to the header file).

I also tried including "petscconf.h" (where PETSC_HAVE_MUMPS is defined) inside the cc, but it did not resolve the issue.

Best regards

Daniel

[Jean-Paul Pelteret]

Dear Daniel,

Looking at the CMake configuration module for PETSc, it would seem that this is a flag that is picked up from the PETSc installation itself. Have you configured PETSc with MUMPS by pointing it towards the MUMPS installation? If you look in the detailed.log log file, you should also see in the entry for PETSC_LIBRARIES that PETSc links against the MUMPS library.

If you need some further help, then its probably best to post the two configuration logs that deal.II outputs so that we have a better idea of how the set of libraries is configured.

Regards,

Jean-Paul

[Jean-Paul Pelteret]

Now that I think about it, the preprocessor directive PETSC_HAVE_MUMPS should probably be DEAL_II_PETSC_WITH_MUMPS, which is set here in another cmake configuration script. Can you change this line accordingly and see if that works? If so, would you like to open a pull request with the fix?

[Daniel Jodlbauer]

Changing PETSC_HAVE_MUMPS to DEAL_II_PETSC_WITH_MUMPS in the petsc_solver.cc file did not solve the issue either.

I attached the log files, if those are of any help (make.log and configure.log are from the petsc build).

From what I can see, PETSC_LIBRARIES includes all the mumps libraries and also the petscconf.h seems to define the flag correctly.

[Jean-Paul Pelteret]

Dear Daniel,

 

Yes, I see that making this change will actually definitely make it fail. I’ve set to correct this with PR #4994 - you could try to patch your version of deal.II with this fix, but I’m not sure that it’d make any difference to you. For my setup (and without the aforementioned patch) I found that PETSc + Mumps was detected perfectly fine. So I’m not sure what the problem can be... I’m going to have to defer to someone more knowledgable on our build system + PETSc to help you work this out.

I hope that you can get this resolved soon!

J-P

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/19snsi6qJ_k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<detailed.log><make.log><configure.log>

[Uwe Köcher]

I think the problematic line is to be found here in the cmake build system:

[koecheru@lillehammer deal.II-v8.5.1]$ find . -type f -exec grep -H 'DEAL_II_PETSC_WITH_MUMPS' {} \;
./cmake/configure/configure_3_petsc.cmake:SET(DEAL_II_PETSC_WITH_MUMPS ${PETSC_WITH_MUMPS})

it should be

./cmake/configure/configure_3_petsc.cmake:SET(DEAL_II_PETSC_WITH_MUMPS ${PETSC_HAVE_MUMPS})

A simple test for  PETSC_HAVE_MUMPS, DEAL_II_PETSC_WITH_MUMPS, PETSC_WITH_MUMPS

results currently (deal.II v8.5.1 via candi) in true, false, false.

Petsc with mumps is installed, if the package "Parmetis" and "Petsc" is enabled in the candi configuration.

Kind regards

  Uwe

[Wolfgang Bangerth]

On 08/30/2017 03:17 AM, Daniel Jodlbauer wrote:
>
> I also tried including "petscconf.h" (where PETSC_HAVE_MUMPS is defined)
> inside the cc, but it did not resolve the issue.

This does not sound right, though. If petscconf.h #defines the
preprocessor variable, and if you #include this file into your own .cc
file, then the flag must be available. Can you double check that this
doesn't work? For example, if you *know* that the variable is #defined
there, then do

#ifdef PETSC_HAVE_MUMPS
std::cout << "OK" << std::endl;
#else
# error We shouldn't get here!
#endif

in your code. If you get the error, double check that you've spelled
things correctly. If you still get the error, are you sure that you're
#including the correct file? Do you have multiple petscconf.h files on
your system?

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Daniel Jodlbauer]

Indeed, there was a previous installation on my system configured without Mumps which I have not noticed so far.

It seems that during installation of deal.II, it compiled against this version instead of the one installed via candi, despite showing the correct include/library paths during cmake. When compiling my code (using eclipse), it choses the correct version.

Since I have no root access, I cannot remove the old version of petsc. Is it possible to somehow force deal.II to use the correct Petsc installation?

(PETSC_DIR/INCLUDE/LIB ... are all pointing towards the correct installation, but still it prefers the one in /usr/include/)

Thanks for pointing me into the right direction!

Best regards,

Daniel

[Timo Heister]

> When compiling my code (using eclipse),
> it choses the correct version.

That doesn't make sense. Eclipse will just run "make". Or are you
referring to the syntax highlighting?

> Since I have no root access, I cannot remove the old version of petsc. Is it
> possible to somehow force deal.II to use the correct Petsc installation?

Weird, it should already do that. Can you provide us with your
detailed.log from candi/tmp/build/deal.*/ ? It should list the correct
PETSc paths.


--
Timo Heister
http://www.math.clemson.edu/~heister/

[Jean-Paul Pelteret]

@timo

Can you provide us with your
detailed.log from candi/tmp/build/deal.*/ ?

Daniel already did so in the 4th post in this thread:

#        DEAL_II_WITH_PETSC set up with external dependencies
#            PETSC_VERSION = 3.7.6.0
#            PETSC_DIR = /home/djodlbauer/deal.ii-candi/petsc-3.7.6
#            PETSC_INCLUDE_DIRS = /home/djodlbauer/deal.ii-candi/petsc-3.7.6/include;/home/djodlbauer/deal.ii-candi/parmetis-4.0.3/include
#            PETSC_USER_INCLUDE_DIRS = /home/djodlbauer/deal.ii-candi/petsc-3.7.6/include;/home/djodlbauer/deal.ii-candi/parmetis-4.0.3/include
#            PETSC_LIBRARIES = /home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libpetsc.so;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libcmumps.a;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libdmumps.a;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libsmumps.a;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libzmumps.a;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libmumps_common.a;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libpord.a;/home/djodlbauer/deal.ii-candi/parmetis-4.0.3/lib/libparmetis.so;/home/djodlbauer/deal.ii-candi/parmetis-4.0.3/lib/libmetis.so;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libHYPRE.a;/home/djodlbauer/deal.ii-candi/petsc-3.7.6/lib/libscalapack.a;/usr/lib64/liblapack.so;/usr/lib64/libblas.so;/usr/lib64/libssl.so;/usr/lib64/libcrypto.so;/usr/lib64/mpi/gcc/openmpi/lib64/libmpi_usempi.so;/usr/lib64/mpi/gcc/openmpi/lib64/libmpi_mpifh.so;gfortran;quadmath;/usr/lib64/mpi/gcc/openmpi/lib64/libmpi_cxx.so;m;/usr/lib64/mpi/gcc/openmpi/lib64/libmpi.so;pthread;dl

I didn't see anything wrong, but maybe I overlooked something.

J-P 

[Jean-Paul Pelteret]

Maybe the order of the includes is a problem, since there are some libraries being picked up from standard locations:

#            BZIP2_INCLUDE_DIRS = /usr/include
#            BZIP2_LIBRARIES = /usr/lib64/libbz2.so

Daniel, where is this other installation of PETSc located on your system? I presume from what you said earlier its in /usr/[include/lib]?

but still it prefers the one in /usr/include/)

[Daniel Jodlbauer]

I think the old petsc headers are in /usr/include/, libs in /usr/lib64/ as far as I remember (will check tomorrow).

That doesn't make sense. Eclipse will just run "make". Or are you 
referring to the syntax highlighting? 

I use the internal builder from Eclipse, and entered all library & include paths for deal/petsc/trilinos/... in the settings. My projects compile and also use code inside the PETSC_HAVE_MUMPS flags.

[Timo Heister]

> Maybe the order of the includes is a problem, since there are some libraries
> being picked up from standard locations:
>
>> # BZIP2_INCLUDE_DIRS = /usr/include
>> # BZIP2_LIBRARIES = /usr/lib64/libbz2.so

Huh, that makes sense. How would we protect against that though? Two
suggestions:
1. if a path is /usr/include or any other default search path, don't add them
2. Sort the include paths and make sure the system path comes after
all the other ones.

[Timo Heister]

Interesting note: apparently cmake should already filter out this
default include path:
https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.kitware.com_cmake_cmake_issues_16291&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=4k7iKXbjGC8LfYxVJJXiaYVu6FRWmEjX38S7JmlS9Vw&m=SLZp46kKykCBXKB5NsgAsszQFrf1t38lshuFDhzaVJ0&s=nuF_OPH4xZbpa7HceXxhfC-4AlCc5Ymv9tfaIJHPLH4&e=
So maybe this is not caused by -I /usr/include (from BZIP2) happening
before the PETSc path, but maybe because we now use -isystem for
includes and these are treated differently (as in not overwriting
/usr/include)?

[Timo Heister]

and I take that back, we are not using -isystem:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dealii_dealii_pull_3897&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=4k7iKXbjGC8LfYxVJJXiaYVu6FRWmEjX38S7JmlS9Vw&m=dIL3wBEPhKcZ12-PL_OQDXRtmH6p_Vzgxdj4W-7qBsE&s=fVxmbdfreL4p1rCxGSP3IVICyedU6jM1ssJGagsPdZM&e=

Something else must be wrong in his configuration.

On Thu, Aug 31, 2017 at 1:06 PM, Timo Heister <hei...@clemson.edu> wrote:
> Interesting note: apparently cmake should already filter out this
> default include path:

> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.kitware.com_cmake_cmake_issues_16291&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=4k7iKXbjGC8LfYxVJJXiaYVu6FRWmEjX38S7JmlS9Vw&m=dIL3wBEPhKcZ12-PL_OQDXRtmH6p_Vzgxdj4W-7qBsE&s=eTejcjX6cyC0eg5obs1whEfIGxO-yIIYIs6o4ZxVVJg&e=

[Wolfgang Bangerth]

On 08/31/2017 10:49 AM, Timo Heister wrote:
> Huh, that makes sense. How would we protect against that though? Two
> suggestions:
> 1. if a path is /usr/include or any other default search path, don't add them
> 2. Sort the include paths and make sure the system path comes after
> all the other ones.

I don't think there is anything you can do about this situation at some
fundamental level. If some of our dependencies are in /usr/include, you
need to have that path accessible. The fact that you may find other
dependencies there as well if they have been installed there is unavoidable.

I would say that this setup (where there is a package in a default path,
but you want to use one from a different path) is just fundamentally
broken and can not reasonably be supported. That's awkward for Daniel,
but hopefully he has access to a sysadmin who can either (i) remove the
offending package, or (ii) update the offending package to include MUMPS
support.

[Uwe Köcher]

Dear Wolfgang

I don't think there is anything you can do about this situation at some
fundamental level. If some of our dependencies are in /usr/include, you
need to have that path accessible. The fact that you may find other
dependencies there as well if they have been installed there is unavoidable. 

I'm not that sure about this last point.

I'm using the environment modules packages

(which is implemented for most of the linux and also macOS).

But correctly sourcing of additional pathes (as we do this also in candi for

a correct build) is more or less the same technique.

In any case, the given strings in search pathes have an order

(not sure in which direction), but you can easily "overwrite" a system path like

/usr/bin by adding it in front or at the back.

Can't we do something similar in the cmake detection of external packages?

I'm not that expert in cmake, but I think that should be possible.

Best

  Uwe

[Wolfgang Bangerth]

> In any case, the given strings in search pathes have an order
> (not sure in which direction), but you can easily "overwrite" a system
> path like
> /usr/bin by adding it in front or at the back.

Imagine you had two different versions of PETSc installed in /usr/ and
/usr/local/, and you had two different versions of Trilinos installed in
/usr/ and /usr/local/. If you want the version of PETSc in /usr and the
version of Trilinos from /usr/local/ to be used, what path do you want
to list first?

The problem is that we just do
#include <petscconf.h>
#include <trilinosconf.h>
for example. There is no way to provide further feedback at this point
where to find each of these files -- the compiler just goes through its
list of search paths. In some sense, that's just a broken system, but
that's just how it is -- if you want multiple versions of the same
package on your system, *all* of them have to be in separate directories
outside where anything else lives.

[Uwe Köcher]

Dear Wolfgang,

the case you describe:

Imagine you had two different versions of PETSc installed in /usr/ and
/usr/local/, and you had two different versions of Trilinos installed in
/usr/ and /usr/local/. If you want the version of PETSc in /usr and the
version of Trilinos from /usr/local/ to be used, what path do you want
to list first?

was not in my mind - since both are usually system pathes, where you typically need to have root-access.

I thought on the case like PATH=/usr:/home/myuser/petsc

(or the other way around).

This case should be correctly detected by deal.II to pick up the user defined petsc lib and headers and not anything from the system pathes because of the order.

I'm also not perfectly know, if it really is a deal.II-cmake detection problem (which I intended from the post of Timo) - or if it is "simple" wrong user environment (the former discussion in this thread).

After the successful build of deal.II with candi,

we state that the user have to either source all installed dependencies with the "enable.sh"

bash script written by Timo

or loading the runtime configuration with the (maybe out of the box imperfect) modulefile,

before cmake / make / running a deal.II based application.

This must be done at any time.

Maybe that is the reason for the eclipse explanation of Daniel, which I do not really understand in the same way as Timo has answered (1st of the 3 above).

Best

  Uwe

[Daniel Jodlbauer]

Ok, I'm going to talk to our admin to remove the old petsc (seems less troublesome than configuring it now and not being able to change that later easily).

Thanks for your help,

Daniel

[Timo Heister]

> Imagine you had two different versions of PETSc installed in /usr/ and
> /usr/local/, and you had two different versions of Trilinos installed in
> /usr/ and /usr/local/. If you want the version of PETSc in /usr and the
> version of Trilinos from /usr/local/ to be used, what path do you want to
> list first?

Well, that might be an impossible thing to get right, but we are
talking about a much simpler problem:
PETSc installed in /usr/ and in some other path that you specify as PETSC_DIR
BZIP2 is only detected in /usr/

We want the PETSC_DIR to take preference and this is supported by gcc
(-I takes precedence over "standard system directories" as explained
here https://urldefense.proofpoint.com/v2/url?u=https-3A__gcc.gnu.org_onlinedocs_gcc-2D7.2.0_gcc_Directory-2DOptions.html-23Directory-2DOptions&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=4k7iKXbjGC8LfYxVJJXiaYVu6FRWmEjX38S7JmlS9Vw&m=8qlsJQq8ko89yICX5ajvQtc1eG6eMp8sCIhF6y6nU68&s=qPwswoHG7CP_1FRZthSQA4-_IkRfHht2AupUoVhqq3w&e=
). It sounds to me like this is not working correctly from the
discussion in this thread. I am not sure if this is caused by the
BZIP2 include path or not. If we add that path with -I the order in
which those appear matters and we might end up including PETSc from
/usr/. I think cmake shouldn't do that (it should filter the include
paths), but maybe we need to do this manually.

I think this is an important enough problem (that you want to ignore
dependencies in system directories) and we should try to fix it.

[Wolfgang Bangerth]

On 09/01/2017 10:44 AM, Timo Heister wrote:
> Well, that might be an impossible thing to get right, but we are
> talking about a much simpler problem:

> PETSc installed in/usr/ and in some other path that you specify as PETSC_DIR

> BZIP2 is only detected in/usr/
>
> We want the PETSC_DIR to take preference and this is supported by gcc
> (-I takes precedence over "standard system directories" as explained

> herehttps://urldefense.proofpoint.com/v2/url?u=https-3A__gcc.gnu.org_onlinedocs_gcc-2D7.2.0_gcc_Directory-2DOptions.html-23Directory-2DOptions&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=4k7iKXbjGC8LfYxVJJXiaYVu6FRWmEjX38S7JmlS9Vw&m=8qlsJQq8ko89yICX5ajvQtc1eG6eMp8sCIhF6y6nU68&s=qPwswoHG7CP_1FRZthSQA4-_IkRfHht2AupUoVhqq3w&e=

> ). It sounds to me like this is not working correctly from the
> discussion in this thread. I am not sure if this is caused by the
> BZIP2 include path or not. If we add that path with -I the order in
> which those appear matters and we might end up including PETSc from
> /usr/. I think cmake shouldn't do that (it should filter the include
> paths), but maybe we need to do this manually.
>
> I think this is an important enough problem (that you want to ignore
> dependencies in system directories) and we should try to fix it.

I don't disagree, but I think it's going to be hard to get right. What
you're saying is that you want to remove everything that's already in
$INCLUDE (or whatever other place it is that the system include dirs are
taken from) from the list of include directories we build in cmake. That
should not be very hard to do. It shouldn't have any effect in regular
setups because there, the order does not matter and whether it's in our
own list of include directories or in $INCLUDE doesn't matter.

All I'm saying is that there will be other cases and setups where it's
still not going to work. These may be so rare as to not care, though.