Error loading petsc shared library in step-40 when distributing cell matrix and rhs separately

[Lucas Myers]

Hi folks,

For debugging purposes, I need to use the `distribute_local_to_global` function which can distribute vectors and matrices separately (this guy in the docs). However, when I try to use this one instead of the function which simultaneously distributes a vector and a matrix (this guy in the docs) it throws an `error while loading shared libraries: libpetsc.so.3.16: cannot open shared object file: No such file or directory` error.

For an example of the behavior, I just replaced the following on line 416 of step-40:

// constraints.distribute_local_to_global(cell_matrix,

//                                                                     cell_rhs,
 //                                                                    local_dof_indices,
//                                                                    system_matrix,
//                                                                    system_rhs);
constraints.distribute_local_to_global(cell_matrix,
                                                                    local_dof_indices,
                                                                    system_matrix);
 constraints.distribute_local_to_global(cell_rhs,
                                                                     local_dof_indices,
                                                                     system_rhs);

This is sort of bizarre to me because (1) it doesn't throw the error in the original tutorial case and (2) CMake ought to take care of the PETSc dependency through the deal.II target. I've attached my detailed.log file in case it's relevant.

Any help figuring out the problem is appreciated!

- Lucas

[Bruno Turcksin]

Lucas,

Just to make sure, you get the error message at runtime right? Are you using the same terminal in both cases? Can you show the output of ldd on your executable and liddealii.so.

Best,

Bruno

[Lucas Myers]

Bruno,

Thanks for the quick response!

Yes, I do get the error at runtime and I am running from the terminal. The call in both cases is `mpirun -np 6 ./install/bin/examples/step-40`. The output from ldd is in the attached file.

- Lucas

[Bruno Turcksin]

Lucas,

There is something strange. When you separate the function somehow your executable depends on PETSc twice and for one of these two dependencies it cannot find PETSc. For the other PETSc object, the code is using  /home/lucas/Application-Data/petsc/x86_64/lib/libpetsc.so.3.16 This usually happens if you have two different versions of a library installed and CMake tries to use both. Did you install PETSc twice? Since you only care about debugging the rest of your code, an easy and dirty way to make the code run is to type in your terminal

export LD_LIBRARY_PATH=home/lucas/Application-Data/petsc/x86_64/lib:$LD_LIBRARY_PATH

This doesn't fix the problem that there are two PETSc objects but at least it should run.

Best,

Brun

[Lucas Myers]

Bruno,

You're saying I just need to export the path before running the executable? If that is the case, it is still not working for whatever reason (still giving the linker error).

As for two PETSc installations: I vaguely recall trying to install it to root and then quickly trying to undo it after reading the warning in the PETSc manual. I'm not sure if that caused the problem, but do you have any tips on sussing it out? That is, how do I find where in the compilation process the executable is being linked to PETSc twice? And how does cmake know that there could be two (possibly deleted) versions of PETSc? It seems like the deal.II installation knows where to find the PETSc shared library given the output of ldd (and the variables set in the detailed.log).

A final bit of information: the linker error only happens when I call the `distribute_local_to_global` function that has calling signature `(local_vector, local_dof_indices, global_vector)`. Both the mixed and matrix equivalent work fine, even when called together.

Let me know if you have any ideas about where to get started.

- Lucas

[Bruno Turcksin]

Lucas,

On Thursday, January 20, 2022 at 3:04:04 PM UTC-5 lucasm...@gmail.com wrote:

Bruno,

You're saying I just need to export the path before running the executable? If that is the case, it is still not working for whatever reason (still giving the linker error).

 

Are you using bash or do you use a different shell?

 

As for two PETSc installations: I vaguely recall trying to install it to root and then quickly trying to undo it after reading the warning in the PETSc manual. I'm not sure if that caused the problem, but do you have any tips on sussing it out? That is, how do I find where in the compilation process the executable is being linked to PETSc twice? And how does cmake know that there could be two (possibly deleted) versions of PETSc? It seems like the deal.II installation knows where to find the PETSc shared library given the output of ldd (and the variables set in the detailed.log).

 

Honestly I am not sure how to debug that. Usually it's pretty obvious that there are two different versions of the library but here it looks like the same library is present twice. CMake has a bunch of default paths that it will search. If you install it to root, it's probably in a path that CMake will search. Try `make VERBOSE=1` maybe we'll see something useful.

Best,

Bruno

[Lucas Myers]

Bruno,

Yes, I'm using bash -- just the default terminal on Ubuntu 20.04.

I've attached the output from `make VERBOSE=1` in the `gcc_linker_debug.txt`.

As an added twist, when I try to compile with clang 13 (just step-40, not deal.II -- that's still compiled with gcc) it tries to look for a bunch of libraries twice: hypre, p4est, and petsc. I've also attached the output of `make VERBOSE=1` from the clang compilation. Note that there's a bunch of extraneous project stuff unrelated to the bug in both these make files.

Thanks for being patient.

- Lucas

[Wolfgang Bangerth]

On 1/20/22 1:43 PM, Lucas Myers wrote:
>
> As an added twist, when I try to compile with clang 13 (just step-40, not
> deal.II -- that's still compiled with gcc) it tries to look for a bunch of
> libraries twice: hypre, p4est, and petsc. I've also attached the output of
> `make VERBOSE=1` from the clang compilation. Note that there's a bunch of
> extraneous project stuff unrelated to the bug in both these make files.

As a general, your life will be unhappy if you have two versions (or just two
copies) of the same library accessible via LD_LIBRARY_PATH. That seems to be
the case for you. It leads to great confusion and error messages that, as you
found, seem to make no sense at all.

My suggestion is to remove one or the other copy of PETSc (or both) on your
system, remove deal.II and everything that was compiled along with it and
might depend on PETSc, and start the whole installation process from scratch.
Root out the problem by starting from scratch. It is by far the easier way
than trying to somehow work around it by setting paths in certain ways or
telling the compiler this or that. Start from scratch.

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Matthias Maier]

Can you find any petsc library under /usr or /usr/local ?

$ find /usr -iname "libpetsc*"

Similarly, would you mind checking how many petsc installation you have
in your home directory?

$ find $HOME -iname "libpetsc*"

Best,
Matthias

[Lucas Myers]

Matthias,

Under /usr the find command doesn't get anything. In fact, if I search for "*petsc*" it only finds the dealii header files.

In $HOME it finds:

/home/lucas/Application-Data/petsc/src/binding/petsc4py/src/libpetsc4py.h
/home/lucas/Application-Data/petsc/src/binding/petsc4py/src/libpetsc4py.c
/home/lucas/Application-Data/petsc/src/binding/petsc4py/src/libpetsc4py
/home/lucas/Application-Data/petsc/src/binding/petsc4py/src/libpetsc4py/libpetsc4py.pyx
/home/lucas/Application-Data/petsc/x86_64/lib/libpetsc.so
/home/lucas/Application-Data/petsc/x86_64/lib/libpetsc.so.3.16
/home/lucas/Application-Data/petsc/x86_64/lib/libpetsc.so.3.16.3

and then an old petsc directory in my trash (~/.local/share/Trash/). I deleted that and recompiled + relinked the file and the problem persists.

Wolfgang,

I would want to do that, but I cannot seem to find a second installation of PETSc. Additionally, I'm almost certain that I only have one installation of p4est and Hypre. Further, my LD_LIBRARY_PATH is empty except for the singular path to PETSc that Bruno recommended. So in the case that I want to clear everything I'm not sure how to reliably uninstall all of these packages, including any sort of cache that cmake contains on where to find these shared libraries.

So sorry for the trouble everyone, and thanks for the help.

- Lucas