Static libraries considered harmful

[Volker Braun]

It was suggested on http://trac.sagemath.org/14699 that we rehash the old arguments why static libraries have no place in a modular software system. So here we go:

  * No easy way to tell which static library (much less its version) is used by a binary

  * Space (both disk and memory) bloat, code will be copied into each binary.

  * Inability to benefit from updates: Once statically linked, your binary will never use a newer (and potentially bug-fixed) version of the library

  * Various debug tools (LD_PRELOAD, LD_PROFILE, LD_AUDIT) don't work

  * No ASLR => security hole

The counterargument is, basically, since you already had to compile stuff for a shared library you might just as well install a static version, too. And there might still be platforms that can't build shared libraries, though I'm not aware of one that also compiles a non-trivial part of Sage; You certainly can't use Cython without shared libraries.

[Jean-Pierre Flori]

Speed?

[Volker Braun]

There is a speed penalty for calling position-independent code (PIC). For a function that does nothing, it is in the 15% range. In real-world applications it is more in the < 3% range since library functions tend to perform some computational task. If you really need that, you have to compile the code separately non-PIC. Which is what libtool is doing, if you use it for a static+shared library then everything will be compiled twice (and take twice as long to compile). Note that our ATLAS spkg has always compiled atlas only once (and with PIC enabled), so there is no speed benefit to installing the PIC code as static library.

There is a penalty for dynamically loading the code the first time, but you also save load time subsequently since you don't have to load a static copy multiple times.

[Jean-Pierre Flori]

On Saturday, June 8, 2013 5:14:03 PM UTC+2, Volker Braun wrote:

The way we use libtool to produce shared libraries of ATLAS is not portable.
It issues warnings on every system and errors out on some (at least Cygwin) because it plays with non libtool objects.
I did not find an option to get warnings only on Cygwin.

[Jean-Pierre Flori]

On Saturday, June 8, 2013 5:33:43 PM UTC+2, Volker Braun wrote:

There is a speed penalty for calling position-independent code (PIC). For a function that does nothing, it is in the 15% range. In real-world applications it is more in the < 3% range since library functions tend to perform some computational task. If you really need that, you have to compile the code separately non-PIC. Which is what libtool is doing, if you use it for a static+shared library then everything will be compiled twice (and take twice as long to compile). Note that our ATLAS spkg has always compiled atlas only once (and with PIC enabled), so there is no speed benefit to installing the PIC code as static library.

Yeah, Im aware we only build ATLAS once configured to produce shared code, but I thought the point had to be raised.

[leif]

Volker Braun wrote:
> It was suggested on http://trac.sagemath.org/14699 that we rehash the
> old arguments why static libraries have no place in a modular software
> system. So here we go:
> * No easy way to tell which static library (much less its version) is
> used by a binary
> * Space (both disk and memory) bloat, code will be copied into each
> binary.
> * Inability to benefit from updates: Once statically linked, your
> binary will never use a newer (and potentially bug-fixed) version of the
> library
> * Various debug tools (LD_PRELOAD, LD_PROFILE, LD_AUDIT) don't work
> * No ASLR => security hole

That's totally not the point; we weren't discussing whether *Sage*
should *use* static (ATLAS) libraries rather than shared ones, but
whether -- since ATLAS builds static libraries (only) anyway by default
-- we should *keep* them too, or discard them.

It's trivial to delete them from the Sage installation if you don't need
(or want) them.

But if you happen to need them later, for whatever reason, you'd have to
rebuild ATLAS (with SAGE_KEEP_BUILT_SPKGS=yes or 'sage -f -s ...'),
search them in the build tree, and manually copy them from various
places. Inconvenient, tedious, error-prone and a waste of time
(especially if building ATLAS takes hours or nearly a day on your machine).

[And while it is relatively easy to (re)build a shared library from a
static one, the opposite is not.]


Btw., there's still not even a script to save the once built [shared]
ATLAS libraries for use by another Sage installation.


-leif

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Volker Braun]

On Saturday, June 8, 2013 5:14:30 PM UTC+1, Jean-Pierre Flori wrote:

The way we use libtool to produce shared libraries of ATLAS is not portable.

I agree that this is ugly as sin. At one point we will have to switch Sage to the new-style shared libraries that upstream introduced, which package the blas code in a different way. The current libtool hack is just a stopgap for that. In particular, I would rather work on abstracting the blas choice into pkgconfig-style scripts and push that into sage than hardcode slightly different names for atlas. Then we'll also be able to use different vendors like OpenBLAS and MKL.

Until then Cygwin can just use the Cygwin ATLAS libraries IMHO.

[Volker Braun]

On Saturday, June 8, 2013 5:23:57 PM UTC+1, leif wrote:

Btw., there's still not even a script to save the once built [shared]
ATLAS libraries for use by another Sage installation.

And such a script would be totally pointless if there were static libraries floating about.

[Jean-Pierre Flori]

On Saturday, June 8, 2013 6:35:49 PM UTC+2, Volker Braun wrote:

On Saturday, June 8, 2013 5:14:30 PM UTC+1, Jean-Pierre Flori wrote:

The way we use libtool to produce shared libraries of ATLAS is not portable.

I agree that this is ugly as sin. At one point we will have to switch Sage to the new-style shared libraries that upstream introduced, which package the blas code in a different way.

I know, and i know it is not trivial to do so.
I don't criticize what you've done, it's great to be able to use a shared version of ATLAS easily, rather than struggling to use the one packaged by upstream (not even sure the upstream ones would build on Cygwin).

The current libtool hack is just a stopgap for that. In particular, I would rather work on abstracting the blas choice into pkgconfig-style scripts and push that into sage than hardcode slightly different names for atlas. Then we'll also be able to use different vendors like OpenBLAS and MKL.

That would be awesome.
Not really important for me, but i definitely had such requests at Sage's meetings.
 

Until then Cygwin can just use the Cygwin ATLAS libraries IMHO.

I'm less inclined to accept that :)

[leif]

Nope. Copying / saving ATLAS libraries (in order to later avoid useless
rebuilding / tuning) is orthogonal to whether they are static, shared,
or both.

[Volker Braun]

Ok, maybe if you have an old computer that doesn't recompile ATLAS in 10 mins... But honestly, if you can't wait then just use your distribution's ATLAS.

Being able to save / reuse shared libraries would be really nice to benchmark and compare different implementations. And there we have to make sure that no component is linking a static blas library.

[leif]

Volker Braun wrote:
> Ok, maybe if you have an old computer that doesn't recompile ATLAS in 10
> mins... But honestly, if you can't wait then just use your
> distribution's ATLAS.

... which isn't tuned for the specific machine.

And tuning doesn't only take ages on some old, but also some newer CPUs.

> Being able to save / reuse shared libraries would be really nice to
> benchmark and compare different implementations. And there we have to
> make sure that no component is linking a static blas library.

XD Incidentally, upstream's tools / scripts to compare different
implementations / libraries only work (or worked, tried that a while
ago) with *static* ATLAS libraries.

The ATLAS documentation (or website at least) is a mess as well, so I'm
unable to provide links here.

[Volker Braun]

On Saturday, June 8, 2013 6:31:40 PM UTC+1, leif wrote:

... which isn't tuned for the specific machine.

au contraire, full tuned. Of course using some pre-tuned values, for which you need a recent enough Core i7.

XD  Incidentally, upstream's tools / scripts to compare different
implementations / libraries only work (or worked, tried that a while
ago) with *static* ATLAS libraries.

Since it doesn't work with OpenBLAS/MKL its not really the kind of benchmark suite that we should be looking at.

Incidentally, if anybody has a recommendation for a decent BLAS/LAPACK benchmark then please do share.

[leif]

Volker Braun wrote:
> On Saturday, June 8, 2013 6:31:40 PM UTC+1, leif wrote:
>
> ... which isn't tuned for the specific machine.
>
>
> au contraire, full tuned. Of course using some pre-tuned values, for
> which you need a recent enough Core i7.

So you're saying they're shipping ATLAS libraries which dynamically
choose the particularly best implementation, depending on the flavour of
the present Nehalem / SandyBridge / SandyBridge-E / IvyBridge (/
Haswell?) Core i3 / i5 / i7 CPU and its cache sizes etc., and probably
for all kinds of (newer) AMD CPUs as well?

The specific memory latency and bandwidth of the machine matters as
well, as does whether you're using a dedicated GPU for some chips.

[Volker Braun]

Sorry, I was replying to the other sentence: I'm saying that if you compile ATLAS using most sandy / ivy bridge core i7 then the build system will use pre-tuned values that eliminate the need for most of the search, resulting in about 10 mins of build time.