A Cython module requiring -librt. But not on Mac computers

[Nathann Cohen]

Hello everybody !!!!

While attempting to upgrade the optional cbc package we met an issue I have no idea how to solve. For some mysterious reason (for which I did not try to find an explanation -- this package is exhausting me), Coin requires some optional libraries to be run. On my computer, I need to add the libraries librt, libz, and libbz2 to the modules_list.py file :

- libraries = ["csage", "stdc++", "Cbc", "CbcSolver", "Cgl", "Clp", "CoinUtils", "OsiCbc", "OsiClp", "Osi", "OsiVol", "Vol"])
+ libraries = ["csage", "stdc++", "Cbc", "Cgl", "CbcSolver", "Clp","CoinUtils", "OsiClp", "Osi", "z", "bz2", "rt"])

The thing is that when John tests it on his own computer the compilation fails, because it looks like there is no librt on Mac. Actually, it just isn't needed for everything runs fine without the -lrt flag on his computer. Well. Is there any way to detect the architecture and use this rt flag on non-mac computers ?

It may be easier to fix the problem from inside the SPKG, though. Would anybody know how to write something like that in a bash script :

if <I am not running mac>:
  export LDFLAGS="$LDFLAGS -lrt"

For it may be easier to check it there than from modules_list.py....

Thank you very much for your help ! :-)

Nathann

[Nathann Cohen]

Oops. I forgot to add that the upgrade was actually ticket #12220

http://trac.sagemath.org/sage_trac/ticket/12220

Nathann

[kcrisman]

Yes, in the spkg is the appropriate place, I would assume (?) though I
assume this isn't actually in installation of the spkg. Hmm. Can we
modify module_list.py while running an spkg?

Usually it's something like

if [ "$UNAME" = "Darwin" ]; then
(stuff)
fi

Looks like the shell script for not equal (strings) is !=, so


if [ "$UNAME" != "Darwin" ]; then
(stuff)
fi

but I'd check about whether Solaris requires this.

[john_perry_usm]

Hi

Some more information which I find fascinating; apologies if it's not
helpful:

. all this is needed because the library CoinUtils wants to call
gzopen;
. g++ compiles the patch for Nathann even though librt isn't linked;
. Nathann can start sage just fine; but
. when he tries to create a MixedIntegerLinearProgram, he gets an
error
even though nothing we do invokes either the function or even
CoinUtils.

On Mac OSX, by contrast, g++ fails if I ask it to link to librt.

john

[john_perry_usm]

Oh, I forgot -- the patch I have works just fine on 32-bit Ubuntu and
32-bit Mac OSX. As far as we know, it fails only on 64-bit Debian.

john perry

[Volker Braun]

Sounds like some library that you are linking to uses librt, even though your own program doesn't. At the linker command line you must specify all libraries being used. Older versions of ld will automatically try to find implicit dependencies, but this is bad and deprecated. 

[john_perry_usm]

How old a version of ld are we talking about here? The PowerBook is
running 10.6.8 and the workstation is running Ubuntu 10.04.3 LTS.

john perry

[Volker Braun]

Well OSX doesn't support librt and I'm pretty sure Ubuntu 10.04.3 ld is using old-style DSO linking.

I don't see why you cbc needs librt to start with. Does it do any real-time stuff?

[john_perry_usm]

> Well OSX doesn't support librt and I'm pretty sure Ubuntu 10.04.3 ld is
> using old-style DSO linking.

OK. So basically we just have to fix modules_list.py.

> I don't see why you cbc needs librt to start with. Does it do any real-time
> stuff?

I think the problem is that zlib requires rtlib. Cbc doesn't require
rt directly. Nathann, correct me if I'm wrong.

john perry

[Jeroen Demeyer]

On 2012-01-25 22:22, john_perry_usm wrote:
>> Well OSX doesn't support librt and I'm pretty sure Ubuntu 10.04.3 ld is
>> using old-style DSO linking.
>
> OK. So basically we just have to fix modules_list.py.
>
>> I don't see why you cbc needs librt to start with. Does it do any real-time
>> stuff?
>
> I think the problem is that zlib requires rtlib.

This shouldn't be the case, especially not with the zlib which is
distributed within Sage.

[john_perry_usm]

> Well OSX doesn't support librt and I'm pretty sure Ubuntu 10.04.3 ld is
> using old-style DSO linking.

OK. So basically we just have to fix modules_list.py.

> I don't see why you cbc needs librt to start with. Does it do any real-time
> stuff?

[john_perry_usm]

You're right; we were doing it before without zlib, too. Okay, I don't
know why it's calling librt, either. :-)

john perry

[Volker Braun]

From src/Cbc/configure.ac:

  AC_CHECK_LIB([rt],[clock_gettime],

               [CBCLIB_LIBS="-lrt $CBCLIB_LIBS"

                CBCLIB_PCLIBS="-lrt $CBCLIB_PCLIBS"

                AC_DEFINE([HAVE_CLOCK_GETTIME],[1],[Define if clock_gettime and rt library is available])

So cbc uses librt to get a high resolution timer and properly handle the absence of it on OSX. So you'll either have to patch it to not use librt ever or pass different libraries in modules_list.py depending on whether librt is available or not.

[john_perry_usm]

On Jan 25, 3:51 pm, Jeroen Demeyer <jdeme...@cage.ugent.be> wrote:

[Volker Braun]

Unless there is any marked benefit to using the high-resolution timer its probably best to just #define HAVE_CLOCK_GETTIME 1 in config.h after running configure.

[Volker Braun]

On Wednesday, January 25, 2012 3:33:58 PM UTC-8, Volker Braun wrote:

Unless there is any marked benefit to using the high-resolution timer its probably best to just #define HAVE_CLOCK_GETTIME 1 in config.h after running configure.

Should be #define HAVE_CLOCK_GETTIME 0

[Nathann Cohen]

>> Unless there is any marked benefit to using the high-resolution timer its
>> probably best to just #define HAVE_CLOCK_GETTIME 1 in config.h after running
>> configure.

I thought I had found a nice way to do that. I updated the
spkg-install file with :

export CPPFLAGS="-DHAVE_CLOCK_GETTIME=0"

just before configure/make/make install.

The result :

[...]
checking for COIN-OR package Msk... not given: No package 'osi-mosek' found
checking for COIN-OR package Spx... not given: No package 'osi-soplex' found
checking for COIN-OR package Xpr... not given: No package 'osi-xpress' found
checking for COIN-OR package ASL... not given: No package 'coinasl' found
checking for cbc-generic default solver... clp
checking for clock_gettime in -lrt... yes
checking for pthread_create in -lpthread... no
configure: error: --enable-cbc-parallel selected, but -lpthreads unavailable
configure: error: /bin/bash './configure' failed for Cbc

real 0m27.033s
user 0m5.628s
sys 0m3.692s
************************************************************************
Error installing package cbc-2.7.5
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
/home/ncohen/.Sage/spkg/logs/cbc-2.7.5.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/home/ncohen/.Sage/spkg/build/cbc-2.7.5 and type 'make' or whatever is
appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
(cd '/home/ncohen/.Sage/spkg/build/cbc-2.7.5' &&
'/home/ncohen/.Sage/sage' -sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************
Error: Failed to install package 'cbc-2.7.5'.
~$

So now we know it uses this to handle threads. I should not touch this
solver again, each time I do I end up wanting to kill people O_O;;;

Nathann

[Nathann Cohen]

Oh, I understand, it was a mistake. Because the way I wanted to do it
the "configure" scripts plans for everything to be compiled with -lrt,
and I prevent that with this flag... What I need is a way to force
"configure" to understand that it should *NOT* use librt, or to make
it believe it actually does not exist O_o

Anyway why isn't the library linked with librt when it is compiled ? I
do not understand why we should do it ourselves when compiling the
Sage code anyway O_O;;;

Nathann

[Volker Braun]

The cbc library is linked with librt if it is available. The problem is that any other library (like any Cython extension class) then also must explicitly link against librt. 

You can't do it by setting some CFLAGS and friends. Ideally configure would have a switch to turn it off; since it doesn't you have to monkey-patch the configure output (config.h)

[Nathann Cohen]

> The cbc library is linked with librt if it is available. The problem is that
> any other library (like any Cython extension class) then also must
> explicitly link against librt.

I am trying to compile it with the --ensable-static to see whether it
would solve the problem cheaply.... But God it is long O_O

Nathann

[Nathann Cohen]

> I am trying to compile it with the --ensable-static to see whether it
> would solve the problem cheaply.... But God it is long O_O

sage: MixedIntegerLinearProgram(solver="Coin")
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)

/home/ncohen/.Sage/devel/sage-2/<ipython console> in <module>()

/home/ncohen/.Sage/local/lib/python2.7/site-packages/sage/numerical/mip.so
in sage.numerical.mip.MixedIntegerLinearProgram.__init__
(sage/numerical/mip.c:1480)()

/home/ncohen/.Sage/local/lib/python2.7/site-packages/sage/numerical/backends/generic_backend.so
in sage.numerical.backends.generic_backend.get_solver
(sage/numerical/backends/generic_backend.c:6139)()

/home/ncohen/.Sage/local/lib/python2.7/site-packages/sage/numerical/backends/generic_backend.so
in sage.numerical.backends.generic_backend.get_solver
(sage/numerical/backends/generic_backend.c:5857)()

ImportError: /home/ncohen/.Sage/local/lib/libCbc.so.3: undefined
symbol: clock_gettime

I need simething heavy... And sharp....

Nathann

[Volker Braun]

E.g. Fedora only has librt.so and no librt.a. So even the best makefile will not be able to statically link with librt in that case.