Shared vs. Static Libraries

[Elizabeth Fischer]

Hello Spackers,

I've been encountering quite a lot of trouble with Spack packages that do not build shared libraries by default.  Some do not even provide a +shared variant.  Others do (sort of), but by a different name (eg, +fpic for netlib-blas).

When libraries are not built shared, in many cases I've found they cannot even link with shared libraries --- because -fPIC was not included on the command line.  For example, hypre/package.py.

I think this is an area where we could improve Spack if we agree on some standards (for which I'm soliciting feedback).  I might suggest, in order of increasing potential for generating controversy:

1. Spack compiler wrappers should always provide -fPIC, just as they always provide an Rpath currently.  This would allow shared libraries to link to static libraries.

2. We standardize on variants for ALL packages that control shared / non-shared generation:

   a) These variants get implemented in the core Spack code, so the package.py files don't have to provide variant() declarations.

   b) There should be a way to set these variants by default for ALL packages (eg: I'd set "+shared" as a general Spack configuration, and then all packages would be built with "+shared").  If this capability is not already in Spack, it would be provided as a follow-on to PR #120.

   b) I would suggest we follow existing convention used by some packages: "+shared" means build shared libraries, "+static" means build static libraries.  
~shared" means don't build shared libraries, etc.

   c) If a package is incapable of building libraries in the ways requested, it can silently override Spack-wide defaults.  But it should throw an error if the default was set per-package.  (Eg: suppose Spack-wide defaults are set to "+shared" and mylib/package.py can't do shared libraries.  Then I can silently just build static libraries.  But if a user asks for mylib+shared, then I need to throw an error).

   d) If the user does nothing, Spack-wide defaults should be configured by default to "+shared+static".

3. If we don't want to implement (2), what if we just keep it simple and make sure that all packages generate shared libraries by default?  Rationale for this would be that static libraries are diminishing in use anyway, so why go to extra effort to keep a dying horse alive?  Shared libraries have many advantages, but are harder to get right at link time and runtime.  Spack solves a lot of these problems with its dependency management and Rpath support.  Meanwhile, static libraries are increasingly untenable in today's world of deep and multiply-branching dependency DAGs.

4. The one shared library problem Spack does not yet solve is this... suppose I've built and installed a binary.  Now I want to "pickle" it to a folder of my choosing, so it can run for three months on a supercomputer.  I want the freedom to recompile code for the next three months while it's running.  For this, I need a "pickle" procedure that works on a set of binaries.  It  would: (a) run ldd on those binaries to determine which shared libraries they load, (b) copy the binaries and all their .so dependencies into a folder, (c) build a lanucher bash script that sets LD_LIBRARY_PATH to the pickle folder and then launches the binaries.

Anyone have any thoughts/objections to these proposals?  If nothing else, this will guide me on the changes I'm making to packages that currently provide no shared library support at all.

-- Elizabeth

[Ben Boeckel]

On Tue, Mar 08, 2016 at 11:05:33 -0500, Elizabeth Fischer wrote:
> 1. Spack compiler wrappers should always provide -fPIC, just as they always
> provide an Rpath currently. This would allow shared libraries to link to
> static libraries.

Well, it's not always spelled '-fPIC', but yes, this makes sense. Here's
CMake's "table" of -fPIC flags:

Modules/Compiler/GNU.cmake: set(CMAKE_${lang}_COMPILE_OPTIONS_PIC "-fPIC")
Modules/Compiler/NAG-Fortran.cmake:set(CMAKE_Fortran_COMPILE_OPTIONS_PIC "-PIC")
Modules/Compiler/SCO.cmake: set(CMAKE_${lang}_COMPILE_OPTIONS_PIC -Kpic)
Modules/Compiler/SunPro-C.cmake:set(CMAKE_C_COMPILE_OPTIONS_PIC -KPIC)
Modules/Compiler/SunPro-CXX.cmake:set(CMAKE_CXX_COMPILE_OPTIONS_PIC -KPIC)
Modules/Compiler/SunPro-Fortran.cmake:set(CMAKE_Fortran_COMPILE_OPTIONS_PIC "-KPIC")
Modules/Compiler/XL.cmake: set(CMAKE_${lang}_COMPILE_OPTIONS_PIC "-qpic")

> 2. We standardize on variants for ALL packages that control shared /
> non-shared generation:
> a) These variants get implemented in the core Spack code, so the
> package.py files don't have to provide variant() declarations.
> b) There should be a way to set these variants by default for ALL
> packages (eg: I'd set "+shared" as a general Spack configuration, and then
> all packages would be built with "+shared"). If this capability is not
> already in Spack, it would be provided as a follow-on to PR #120.
> b) I would suggest we follow existing convention used by some packages:
> "+shared" means build shared libraries, "+static" means build static
> libraries.
> ~shared" means don't build shared libraries, etc.
> c) If a package is incapable of building libraries in the ways
> requested, it can silently override Spack-wide defaults. But it should
> throw an error if the default was set per-package. (Eg: suppose Spack-wide
> defaults are set to "+shared" and mylib/package.py can't do shared
> libraries. Then I can silently just build static libraries. But if a user
> asks for mylib+shared, then I need to throw an error).
> d) If the user does nothing, Spack-wide defaults should be configured by
> default to "+shared+static".

I agree that it should always be spelled "+shared" and "+static" (and
not "+fpic" or "+dynamic"), but I don't know about spack providing it by
default without any mention in package.py. What does `py-six+shared`
even *mean*? What about `py-numpy+static`?

Also, with spack knowing about shared versus static, it means that spack
can drop LD_LIBRARY_PATH and rpath additions for packages which depend
on static libraries.

> 3. If we don't want to implement (2), what if we just keep it simple and
> make sure that all packages generate shared libraries by default?
> Rationale for this would be that static libraries are diminishing in use
> anyway, so why go to extra effort to keep a dying horse alive? Shared
> libraries have many advantages, but are harder to get right at link time
> and runtime. Spack solves a lot of these problems with its dependency
> management and Rpath support. Meanwhile, static libraries are increasingly
> untenable in today's world of deep and multiply-branching dependency DAGs.

Fine with me, but some supercomputers work better with static libraries
due to filesystem performance problems (mainly to do with 4000 nodes
trying to all read the same set of files at the same time) or lack of

shared library support at all.

> 4. The one shared library problem Spack does not yet solve is this...
> suppose I've built and installed a binary. Now I want to "pickle" it to a
> folder of my choosing, so it can run for three months on a supercomputer.
> I want the freedom to recompile code for the next three months while it's
> running. For this, I need a "pickle" procedure that works on a set of
> binaries. It would: (a) run ldd on those binaries to determine which
> shared libraries they load, (b) copy the binaries and all their .so
> dependencies into a folder, (c) build a lanucher bash script that sets
> LD_LIBRARY_PATH to the pickle folder and then launches the binaries.

CMake has something like this which it uses to find what libraries are
necessary when installing binaries for CPack. It's certainly something
that's possible, but probably better by just rewriting the rpaths to be
nothing so that LD_LIBRARY_PATH does all the work. I don't think a
pickled library should use libraries outside of the pickle by accident
(e.g., if you run it on the machine you happened to build it on).

--Ben

[Matthew LeGendre]

(It looks like this discussion bounced between between this thread and
github issue #511. I'd appreciate it if we could keep discussions to one
communication mechanism--they're easier to follow that way).

Adding -fPIC automatically via compiler wrappers will cause problems with
many packages. Most autotool-based build systems assume libtool controls
the presence of -fPIC and make different linking decisions based on
whether they used it or not. Changing that on them under the hood could
cause it to be incorrectly flipped on/off. If an individual package's
build system mis-uses -fPIC, then it's up to that package to fix it.

I agree about standardizing package variants on share/static, and I think
those are a better level of variant names than +fpic/~fpic. Though it
should be up to the individual packages what their default is.

If you did want to turn on -fPIC always and automatically, then PR #120
has much of the infrastructure you need, but not all of it. The
package.yaml config file's format would allow you to specify a variant
that could be applied to all packages, though I didn't test that and don't
think it would quite work without further tweaking.

-Matt

> --
> You received this message because you are subscribed to the Google Groups "Spack" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
> To post to this group, send email to sp...@googlegroups.com.
> Visit this group athttps://secure-web.cisco.com/1llKNXJtHLnPDHlw2o6Ks4XUqi78z0CzvFrqiS_CVzSyrsZJFBq8Jyq3GTxCMBuyeGDjnyQvcEygn4-0UV7hLO6Tng2CYI0l2Rlr
> f_5r7EVYy4IyOqdkdSRR_9mmSt6ahc62md3N9cFwp-Wbp7RRiKs8fz1ov9ZZi_s_affkoHhw58z4eyFt4Hnhf44tECUmY33delgHi_1_LmnJHG57x4fPVcOv3bYL-b2ID
> PqFTR7mAf5U40PELa2NTmyXRpGnLzcAtV7A4thOWlwEXet2JKRcNXmJVz1FG0W8bs8UDzVB-rKTjFuh5NXbyazP1iezKh2Rvjjo06ehSSy2yvcgLXpD81tvJtX7dFvtoV
> hPkClwC8Z_fWBfetXPAoiaqUdm8/https%3A%2F%2Fgroups.google.com%2Fgroup%2Fspack.
> For more options, visithttps://secure-web.cisco.com/1Tmj-i--qtG8jzePS3Vbufp2HqDFNs3MHZfy3fskkstSsmRckNXeIP8nJK2OiQ9AWjOfYYjexrLcytoZ4uNAdHtcykLt7wuNLHza
> WSsa-nidSLSlcn_qcCEwrS28niuP4cNuvd-PKg78a4fMoLpjKU-lIEQvtCO9xu8qUsb_fgA1KW46gfxDdRrBPTkGzqz2FLAG_iNEA8-hjNbQlvb1ZRJiRsA7Ws59fyb11
> A_l7k_b3TeNbXNcg4Gwc2oxrePolTfdFS2Rguc3-KcKtwPi9N8UK7XtPJGgQerMAFXvfzBBw6sGdOp8pm2SsrEpfA9SXvWOLYw1PXdJvUkfhNDq8C3SfzcAk9jdDWDfDs
> Lw6kSaOOwz8brEJ9YoUZfFrt_-b/https%3A%2F%2Fgroups.google.com%2Fd%2Foptout.
>
>