bob.ip.base fails to pass the tests when built on conda

[Amir Mohammadi]

Please take a look at the log here:
https://travis-ci.org/bioidiap/bob.conda/jobs/111792718

The error is:

import: 'bob.ip.base'

/home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.extension-2.0.11-py2.7.egg/bob/extension/__init__.py:12: UserWarning: Module bob was already imported from /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob/__init__.pyc, but /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.blitz-2.0.8-py2.7-linux-x86_64.egg is being added to sys.path

 import pkg_resources

/home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.extension-2.0.11-py2.7.egg/bob/extension/__init__.py:12: UserWarning: Module bob was already imported from /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob/__init__.pyc, but /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.core-2.1.0-py2.7-linux-x86_64.egg is being added to sys.path

 import pkg_resources

/home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.extension-2.0.11-py2.7.egg/bob/extension/__init__.py:12: UserWarning: Module bob was already imported from /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob/__init__.pyc, but /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.extension-2.0.11-py2.7.egg is being added to sys.path

 import pkg_resources

/home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.extension-2.0.11-py2.7.egg/bob/extension/__init__.py:12: UserWarning: Module bob was already imported from /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob/__init__.pyc, but /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.ip.base-2.0.8-py2.7-linux-x86_64.egg is being added to sys.path

 import pkg_resources

/home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.extension-2.0.11-py2.7.egg/bob/extension/__init__.py:12: UserWarning: Module bob was already imported from /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob/__init__.pyc, but /home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.math-2.0.3-py2.7-linux-x86_64.egg is being added to sys.path

 import pkg_resources

Traceback (most recent call last):

 File "/home/travis/miniconda/conda-bld/test-tmp_dir/run_test.py", line 31, in <module>

   import bob.ip.base

 File "/home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.ip.base-2.0.8-py2.7-linux-x86_64.egg/bob/ip/base/__init__.py", line 7, in <module>

   bob.extension.load_bob_library('bob.ip.base', __file__)

 File "/home/travis/miniconda/envs/_test/lib/python2.7/site-packages/bob.extension-2.0.11-py2.7.egg/bob/extension/__init__.py", line 234, in load_bob_library

   ctypes.cdll.LoadLibrary(full_libname)

 File "/home/travis/miniconda/envs/_test/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary

   return self._dlltype(name)

 File "/home/travis/miniconda/envs/_test/lib/python2.7/ctypes/__init__.py", line 365, in __init__

   self._handle = _dlopen(self._name, mode)

OSError: libbob_math.so: cannot open shared object file: No such file or directory

TESTS FAILED: bob.ip.base-2.0.8-py27_0

[Manuel Günther]

Hmm... this is weired. bob.ip.base should not be dependent on bob.math and, thus, it should not try to load the libbob_math.so library in the first place.

If you have created a dependency, then you have to "import bob.math" in "bob/ip/base/__init__.py" prior to "bob.extension.load_bob_library('bob.ip.base', __file__)"

[Manuel Günther]

Correction: bob.ip.base **is** dependent on bob.math. Particularly, the HOG functionality requires bob::math::gradient.

I wonder how this package was working so far... There is clearly an import "bob.math" missing in the __init__.py file. I will fix that.

[Manuel Günther]

I have fixed bob.ip.base (see https://github.com/bioidiap/bob.ip.base/commit/ef83584ddca5bb0b4a643f6fba5153f4ef6d3b5e). 

I now also see, why it has been working before, though I don't know, why it didn't work on conda.

I start to remember, what I did to make the build system working. Let me try to start from the beginning, maybe then I will remember all the details:

The bob::math::gradient function is a header-only function (as it is a C++ template function). Hence, it is not compiled into the libbob_math.so library.

However, we have a C++ dependency on the bob.math package (more specifically, the header files).

During the implementation of the build system (which is located in bob.extension), I did not distinguish between a header-only dependency and a library dependency. Hence, I added a library dependency to the bob.math package.

Now, during the linking process, the compilers on Ubuntu and MacOS find out that there is no actual library dependency between libbob_ip_base.so and libbob_math.so and, thus, do not require to load libbob_math.so in order to load libbob_ip_base.so.

However, the conda compiler (or linker, to be more specific) does not seem to be smart enough to see that there is no actual dependency. Hence it requires to load libbob_math.so in order to load libbob_ip_base.so.

Now comes the issue:

In order to be able to handle multiple versions of the same library, I load the .so file exactly from that library that we actually import in the python file.

Hence, the location of the libraries is only known at import time. So, each python package will load its own library from its current directory (i.e., using the bob.extension.load_bob_library('bob.ip.base', __file__) instruction). 

When there is a library dependency between the packages, the dependent package must load its library first. In python, this is done by simply importing the library before loading the library of the own package.

I hope that the explanation of the problem was understandable.

Anyways, I am pretty sure that there will be more instances in other bob packages, where there is a header-only dependency and, thus, the dependent library is not imported.

However, I am not sure if we need to change all these instances by importing the libraries, or if you could try to get another compiler (linker) that actually removes the library-dependencies automatically.

[Amir Mohammadi]

Conda is not a compiler. It is a python distribution/package manager. GCC 4.8.5 and CMAKE 3.3.1 was used to compile this package as you can see in the log.

[Amir Mohammadi]

Did this one fail because of a similar reason too?
https://travis-ci.org/bioidiap/bob.conda/jobs/111792726

[Manuel Günther]

No, this one fails because libsvm.so is not found. Note that it does not fail because libbob_learn_libsvm.so is missing.

[Manuel Günther]

So, I still don't understand, why this is working for me, and not for you (using conda). When compiling bob.ip.base, during the linking stage you can see that the -lbob_math flag is enabled and, hence, libbob_ip_base.so is linked against libbob_math.so, e.g.:

https://travis-ci.org/bioidiap/bob.ip.base/jobs/111864673#L809

Now, when I check the dependencies of the package using:

$ ldd libbob_ip_base.so | sort

I find three bob library dependencies, which are libbob_core.so, libbob_io_base.so and libbob_sp.so, but no libbob_math.so.

The compiler used in here is gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, see, an even older version, see:

https://travis-ci.org/bioidiap/bob.ip.base/jobs/111864673#L31

which is even older as the one used by conda, and I don't think that they have disabled any features like the ones that we need.

So besides that there might be some difference in the environment, I don't see anything that would trigger this issue. Maybe you can have a look, if the ldd command from above in your case shows a dependency to libbob_math.so. If it does, can you figure out, which functionality of libbob_math.so gets imported to libbob_ip_base.so?

[Amir Mohammadi]

Hi Manuel,

With your latest commit, bob.ip.base did compile and test fine.

    libblitz.so.0 => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../libblitz.so.0 (0x00007f6f9f803000)
    libbob_core.so => not found
    libbob_core.so => not found
    libbob_io_base.so => not found
    libbob_math.so => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../libbob_math.so (0x00007f6f9fe59000)
    libbob_sp.so => not found
    libboost_system.so.1.60.0 => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../libboost_system.so.1.60.0 (0x00007f6f9fa06000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f6f9ea12000)
    libgcc_s.so.1 => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../libgcc_s.so.1 (0x00007f6f9efd1000)
    libgfortran.so.3 => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../.././libgfortran.so.3 (0x00007f6f9c196000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007f6f9f1e7000)
    libopenblas.so.0 => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../libopenblas.so.0 (0x00007f6f9c6c5000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f6f9edb3000)
    libquadmath.so.0 => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../././libquadmath.so.0 (0x00007f6f9bf59000)
    librt.so.1 => /usr/lib/librt.so.1 (0x00007f6f9c4bc000)
    libstdc++.so.6 => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../libstdc++.so.6 (0x00007f6f9f4ec000)
    libvl.so => /home/amir/miniconda3/lib/python3.5/site-packages/bob.ip.base-2.0.9b0-py3.5-linux-x86_64.egg/bob/ip/base/./../../../../../../libvl.so (0x00007f6f9fc09000)
    linux-vdso.so.1 (0x00007fff6e1ec000)
    /usr/lib64/ld-linux-x86-64.so.2 (0x0000558c1402a000)

[Amir Mohammadi]

Here is the log for the succesful build:
https://travis-ci.org/bioidiap/bob.conda/jobs/112018645

[Manuel Günther]

Yes, I was pretty sure it would work now, but it doesn't explain, what is the reason for the issue. In your case, libbob_ip_base.so does link against libbob_math.so (and it even finds a path for it, which shouldn't be the case). Also, it links twice against libbob_core.so, and I don't see, why this is the case.

I know that there are some tools in Linux to find out, which functions are imported from which shared library. Do you think it is possible for you to see, which of the functions from libbob_core.so are actually imported into libbob_ip_base.so?

I don't know, how easily this is possible (if at all), but can you try to use exactly the same compiler that we were using to compile our packages, i.e., the default compiler on Travis? Maybe with the new version they have introduced a feature that links header dependencies...