cannot import name flapack

[P Purkayastha]

(I sent this email 12h ago, but it hasn't appeared, so I am reposting this.)

I have been having this weird error ever since I moved to a system installed atlas (in /usr/lib) and I am no longer sure how to fix this. Every time I try to doctest certain files which require scipy, they fail spectacularly like this:

**********************************************************************
File "/home/punarbasu/Installations/sage-5.5.rc0/devel/sage-main/sage/plot/plot3d/plot3d.py", line 228:
sage: import scipy.interpolate
Exception raised:
Traceback (most recent call last):
File "/home/punarbasu/Installations/sage-5.5.rc0/local/bin/ncadoctest.py", line 1231, in run_one_test
self.run_one_example(test, example, filename, compileflags)
File "/home/punarbasu/Installations/sage-5.5.rc0/local/bin/sagedoctest.py", line 38, in run_one_example
OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
File "/home/punarbasu/Installations/sage-5.5.rc0/local/bin/ncadoctest.py", line 1172, in run_one_example
compileflags, 1) in test.globs
File "<doctest __main__.example_5[34]>", line 1, in <module>
import scipy.interpolate###line 228:
sage: import scipy.interpolate
File "/home/punarbasu/Installations/sage-5.5.rc0/local/lib/python/site-packages/scipy/interpolate/__init__.py", line 154, in <module>
from rbf import Rbf
File "/home/punarbasu/Installations/sage-5.5.rc0/local/lib/python/site-packages/scipy/interpolate/rbf.py", line 49, in <module>
from scipy import linalg
File "/home/punarbasu/Installations/sage-5.5.rc0/local/lib/python/site-packages/scipy/linalg/__init__.py", line 133, in <module>
from basic import *
File "/home/punarbasu/Installations/sage-5.5.rc0/local/lib/python/site-packages/scipy/linalg/basic.py", line 12, in <module>
from lapack import get_lapack_funcs
File "/home/punarbasu/Installations/sage-5.5.rc0/local/lib/python/site-packages/scipy/linalg/lapack.py", line 14, in <module>
from scipy.linalg import flapack
ImportError: cannot import name flapack



I also get other errors like:

ImportError: /home/punarbasu/Installations/sage-5.5.rc0/local/lib/python2.7/site-packages/scipy/linalg/clapack.so: undefined symbol: clapack_sgesv


Reinstalling scipy doesn't help. What could be the fix for this error? Searching in Google isn't particularly illuminating either.

[François Bissey]

Try re-installing numpy first. It is a classic example of misconfiguration of
blas/lapack in numpy and/or scipy.
You can also try to import numpy from the sage-python shell to see if it
suffers from it too.

Francois

[P Purkayastha]

I reinstalled numpy, and before reinstalling I checked that I could
import it.

Could ccache affect this? I had installed sage with ccache before moving
to a system atlas (it is sci-libs/atlas-3.10.0 from science overlay).
And now I am recompiled sage with ccache enabled - this+system atlas
brought down my compile time to less than 1.5 hours (from 4h+).

That said, the reinstallation of numpy and scipy were done without
ccache enabled.

[François Bissey]

On Thu, 22 Nov 2012 08:31:34 P Purkayastha wrote:
>
> I reinstalled numpy, and before reinstalling I checked that I could
> import it.
>
> Could ccache affect this? I had installed sage with ccache before moving
> to a system atlas (it is sci-libs/atlas-3.10.0 from science overlay).
> And now I am recompiled sage with ccache enabled - this+system atlas
> brought down my compile time to less than 1.5 hours (from 4h+).
>
> That said, the reinstallation of numpy and scipy were done without
> ccache enabled.

It's been a while since I compiled sage with ccache (used to do it fairly
often in the early days of sage-on-gentoo).
If you are using ATLAS from the science overlay it may be that it is not
picked up properly by numpy/scipy.
Do you have
local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
in your sage install? If so what does
ldd -r local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
say within a sage shell and outside of a sage shell.

Because we allow people to install multiple implementations of
blas/cblas/lapack at the same time ATLAS doesn't install the
following
libcblas
liblapack
libclapack
what it installs is:
libatlas
libf77blas
libatlcblas
libatllapack
libatlclapack
and if you threads enabled
libptf77blas
libptcblas
libptlapack
libptclapack
Of these only libatalas and libf77blas are looked for in a regular sage
install. So unless you have created some links you probably picked a wrong
combination of stuff. Your missing symbol should come from libclapack.
numpy/scipy almost certainly recognized you have ATLAS installed from the
presence of libatlas after that things may have gone wrong in the linking
step.

I don't think ccache is at fault.

Francois

[P Purkayastha]

Hmm.. I don't see any missing library links, but I don't see it linked
against libclapack either.

...llations/sage-5.5.rc0.old» ldd -r
local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
linux-vdso.so.1 (0x00007fffdd3ff000)
liblapack.so.0 => /usr/lib64/liblapack.so.0 (0x00007f4834707000)
libptf77blas.so.3 => /usr/lib64/libptf77blas.so.3 (0x00007f48344e4000)
libptcblas.so.3 => /usr/lib64/libptcblas.so.3 (0x00007f48342bf000)
libatlas.so.3 => /usr/lib64/libatlas.so.3 (0x00007f48339e1000)
libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007f4833620000)
libgfortran.so.3 =>
/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/libgfortran.so.3 (0x00007f483333a000)
libm.so.6 => /lib64/libm.so.6 (0x00007f4833044000)
libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/libgcc_s.so.1
(0x00007f4832e2e000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4832a83000)
libblas.so.0 => /usr/lib64/libblas.so.0 (0x00007f48327f5000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f48325d8000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f48323d3000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f48321d0000)
/lib64/ld-linux-x86-64.so.2 (0x000000339ac00000)



libclapack is present in my system:

...llations/sage-5.5.rc0.old» equery files atlas | grep clapack
/usr/include/atlas/clapack.h
/usr/lib64/libatlclapack.so
/usr/lib64/libatlclapack.so.3
/usr/lib64/libptclapack.so
/usr/lib64/libptclapack.so.3
/usr/lib64/pkgconfig/atlas-clapack-threads.pc
/usr/lib64/pkgconfig/atlas-clapack.pc

...llations/sage-5.5.rc0.old» equery files atlas | grep 'so.3$'
/usr/lib64/libatlas.so.3
/usr/lib64/libatlcblas.so.3
/usr/lib64/libatlclapack.so.3
/usr/lib64/libatllapack.so.3
/usr/lib64/libf77blas.so.3
/usr/lib64/libptcblas.so.3
/usr/lib64/libptclapack.so.3
/usr/lib64/libptf77blas.so.3
/usr/lib64/libptlapack.so.3

[François Bissey]

You are using the threaded version of ATLAS. Or at least that's what is being
picked up by numpy. What does
ldd -r local/lib/python/site-packages/scipy/linalg/clapack.so
it may be that you have to create a link to libptclapack.so in local/lib of
your sage install and that would fix it.

Francois

[P Purkayastha]

I see. I did both

$ cd local/lib
$ ln -s /usr/lib/libpt* .

$ cd python/site-packages/scipy/linalg
$ ln -s /usr/lib/libpt* .

Even after this, the clapack does have missing symbols. Here is the output:


~/Installations/sage-5.5.rc0» ldd -r
local/lib/python/site-packages/scipy/linalg/clapack.so
linux-vdso.so.1 (0x00007fff65df5000)
liblapack.so.0 => /usr/lib64/liblapack.so.0 (0x00007fc97bbe4000)
libptf77blas.so.3 => /usr/lib64/libptf77blas.so.3 (0x00007fc97b9c1000)
libptcblas.so.3 => /usr/lib64/libptcblas.so.3 (0x00007fc97b79c000)
libatlas.so.3 => /usr/lib64/libatlas.so.3 (0x00007fc97aebe000)
libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007fc97aafd000)
libgfortran.so.3 =>
/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/libgfortran.so.3 (0x00007fc97a817000)
libm.so.6 => /lib64/libm.so.6 (0x00007fc97a521000)
libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/libgcc_s.so.1
(0x00007fc97a30b000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc979f60000)
libblas.so.0 => /usr/lib64/libblas.so.0 (0x00007fc979cd2000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc979ab5000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc9798b0000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fc9796ad000)
/lib64/ld-linux-x86-64.so.2 (0x000000339ac00000)
undefined symbol: clapack_sgesv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_dgesv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_cgesv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_zgesv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_sposv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_dposv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_cposv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_zposv
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_spotrf
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_dpotrf
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_cpotrf
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_zpotrf
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_spotrs
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_dpotrs
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_cpotrs
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_zpotrs
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_spotri
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_dpotri
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_cpotri
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_zpotri
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_slauum
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_dlauum
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_clauum
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_zlauum
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_strtri
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_dtrtri
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_ctrtri
(local/lib/python/site-packages/scipy/linalg/clapack.so)
undefined symbol: clapack_ztrtri
(local/lib/python/site-packages/scipy/linalg/clapack.so)

[Francois Bissey]

On 23/11/12 14:01, P Purkayastha wrote:

> On 11/23/2012 04:30 AM, Fran�ois Bissey wrote:
>> On Thu, 22 Nov 2012 22:24:06 P Purkayastha wrote:

Should have started there. You are not even linked to lib(pt)clapack
I need the build logs for numpy and scipy and to look at what we do
in these spkg again. Which ATLAS spkg do you use by the way? 3.8.4 or
3.10? I know you want to use system but these will still be called and
do some magic but I think it is different magic depending on which one.

Francois

[P Purkayastha]

On 11/24/2012 02:07 AM, Francois Bissey wrote:
> Should have started there. You are not even linked to lib(pt)clapack
> I need the build logs for numpy and scipy and to look at what we do
> in these spkg again. Which ATLAS spkg do you use by the way? 3.8.4 or
> 3.10? I know you want to use system but these will still be called and
> do some magic but I think it is different magic depending on which one.
>
> Francois
>

Ok. I am using sage-5.5.rc0, and whatever atlas comes with that -
atlas-3.8.4.p1.spkg. Of course, this is not installed. System atlas is
3.10.0. I faced the same problem with sage-5.4.rc4.

The build logs from 5.5rc0 are attached.

Thanks for your help!