Understanding spack compiler wrappers, RPATH and "version `GLIBCXX_3.4.15' not found" error

[Pramod Kumbhar]

Hello,

This may not be specific to Spack but I would like to understand below error. It will be helpful if someone could explain what I am missing below:

On some new cluster I am trying to setup my spack development environment. I have configured external gcc compiler with spack. I build open source package that links as below (using libtool):

mpicxx -O1 -o exe ….  /somepath/gcc-5.2.0/install/lib/../lib64/libstdc++.so    -Wl,-rpath -Wl,/somepath/gcc-5.2.0/install/lib/../lib64 -Wl,-rpath -Wl,/usr/lib64

That libstdc++.so is from gcc-5.2.0 module that I have configured externally. Everything build fine. When I want to use this package, I see following:

ldd exe
exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found
....
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003e8c400000)
....

The /usr/lib64/libstdc++.so has following : 

strings /usr/lib64/libstdc++.so.6 | grep GLIBCXXGLIBCXX_3.4

GLIBCXX_3.4.1

GLIBCXX_3.4.2

GLIBCXX_3.4.3

...
GLIBCXX_3.4.11

GLIBCXX_3.4.12

GLIBCXX_3.4.13

GLIBCXX_FORCE_NEW

GLIBCXX_DEBUG_MESSAGE_LENGTH

And /somepath/gcc-5.2.0/install/lib/../lib64/libstdc++.so :

strings /somepath/gcc-5.2.0/install/lib64/libstdc++.so | grep GLIBCXX

GLIBCXX_3.4

GLIBCXX_3.4.1
....

GLIBCXX_3.4.21

GLIBCXX_FORCE_NEW

GLIBCXX_DEBUG_MESSAGE_LENGTH

So I expected having LD_LIBRARY_PATH=/somepath/gcc-5.2.0/install/lib64:$LD_LIBRARY_PATH (and LD_RUN_PATH) should be sufficient to find correct library but I get : 

/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by...some libexe.so..)

If I look at RPATH header in binary file I see :

readelf -d exe
.....
0x000000000000000f (RPATH)              Library path: [.............:/somepath/gcc-5.2.0/install/lib/../lib64]
....

If I do LD_PRELOAD=/somepath/gcc-5.2.0/install/lib64/libstdc++.so then executable runs fine.

Could someone throw some light on why LD_PRELOAD is necessary here?

-Pramod

[Massimiliano Culpo]

Wild guess: you have /usr/lib64 before /somepath/gcc-5.2.0/install/lib/../lib64 in RPATH.

Note that usually MPI wrappers normally use RUNPATH (which gives precedence to paths in LD_LIBRARY_PATH), but spack modifies them on purpose to use RPATH (which does not).

M.