libkaldi-matrix.so: undefined symbol: clapack_dgetrf

503 views
Skip to first unread message

Tony Robinson

unread,
Sep 18, 2018, 10:58:55 PM9/18/18
to kaldi-help

I'm doing what I thought should be a standard install of Kaldi.   What's changed is that this is first time I've done this on Ubuntu 18.4 and g++-6 (latest GPU stuff installed with lambda-stack-cuda which is great)

I'm getting:

code7 tonyr: /home/tonyr/kaldi/src/bin/est-lda --write-full-matrix=exp/tri2/full.mat --dim=40 exp/tri2/0.mat exp/tri2/lda.*acc
/home/tonyr/kaldi/src/bin/est-lda --write-full-matrix=exp/tri2/full.mat --dim=40 exp/tri2/0.mat exp/tri2/lda.10.acc exp/tri2/lda.11.acc exp/tri2/lda.12.acc exp/tri2/lda.13.acc exp/tri2/lda.14.acc exp/tri2/lda.15.acc exp/tri2/lda.16.acc exp/tri2/lda.17.acc exp/tri2/lda.18.acc exp/tri2/lda.19.acc exp/tri2/lda.1.acc exp/tri2/lda.20.acc exp/tri2/lda.21.acc exp/tri2/lda.22.acc exp/tri2/lda.23.acc exp/tri2/lda.24.acc exp/tri2/lda.25.acc exp/tri2/lda.26.acc exp/tri2/lda.27.acc exp/tri2/lda.28.acc exp/tri2/lda.29.acc exp/tri2/lda.2.acc exp/tri2/lda.30.acc exp/tri2/lda.31.acc exp/tri2/lda.32.acc exp/tri2/lda.33.acc exp/tri2/lda.34.acc exp/tri2/lda.35.acc exp/tri2/lda.3.acc exp/tri2/lda.4.acc exp/tri2/lda.5.acc exp/tri2/lda.6.acc exp/tri2/lda.7.acc exp/tri2/lda.8.acc exp/tri2/lda.9.acc
/home/tonyr/kaldi/src/bin/est-lda: symbol lookup error: /home/tonyr/kaldi/src/lib/libkaldi-matrix.so: undefined symbol: clapack_dgetrf

From where I Read The Fine Documentation at http://kaldi-asr.org/doc/matrixwrap.html under "Missing the ATLAS implementation of (parts of) CLAPACK"

I have atlas:

root@code7# apt install libatlas-base-dev libatlas3-base
Reading package lists... Done
Building dependency tree      
Reading state information... Done
libatlas-base-dev is already the newest version (3.10.3-5).
libatlas3-base is already the newest version (3.10.3-5).

I can see the dependencies:

code7 tonyr: make
g++-6  -Wl,-rpath=/home/tonyr/kaldi/tools/openfst/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -rdynamic -Wl,-rpath=/home/tonyr/kaldi/src/lib  est-lda.o   ../decoder/libkaldi-decoder.so  ../lat/libkaldi-lat.so  ../lm/libkaldi-lm.so  ../fstext/libkaldi-fstext.so  ../hmm/libkaldi-hmm.so  ../transform/libkaldi-transform.so  ../gmm/libkaldi-gmm.so  ../tree/libkaldi-tree.so  ../util/libkaldi-util.so  ../matrix/libkaldi-matrix.so  ../base/libkaldi-base.so /home/tonyr/kaldi/tools/openfst/lib/libfst.so /usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libcblas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libf77blas.so -lm -lpthread -ldl -o est-lda

The dynamic libraries seems to be installed fine:

code7 tonyr: ls -lta /usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libcblas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libf77blas.so
lrwxrwxrwx 1 root root 47 Sep 18 15:41 /usr/lib/x86_64-linux-gnu/liblapack.so -> /etc/alternatives/liblapack.so-x86_64-linux-gnu
lrwxrwxrwx 1 root root 18 Sep 13  2017 /usr/lib/x86_64-linux-gnu/libatlas.so -> libatlas.so.3.10.3
lrwxrwxrwx 1 root root 18 Sep 13  2017 /usr/lib/x86_64-linux-gnu/libcblas.so -> libcblas.so.3.10.3
lrwxrwxrwx 1 root root 20 Sep 13  2017 /usr/lib/x86_64-linux-gnu/libf77blas.so -> libf77blas.so.3.10.3

and the debug in http://kaldi-asr.org/doc/matrixwrap.html also seems to be fine:

root@code7# strings /usr/lib/x86_64-linux-gnu/liblapack.so | egrep ATL_[cd]getrf$
ATL_cgetrf
ATL_dgetrf

root@code7:# strings /usr/lib/x86_64-linux-gnu/liblapack.so | egrep clapack_dgetrf
clapack_dgetrf

That's as far as I've got.   All of this would seem to be the latest Ubuntu and CUDA and so perhaps someone else has had this problem and fixed it.   I see nothing in the googlegroup archives except problems with manually compiling ATLAS.

Anyone else seen this?   No doubt I'll fix it as soon as I post....


Tony

P.S. full install instructions:

git clone https://github.com/kaldi-asr/kaldi.git kaldi --origin upstream
cd kaldi

( cd tools ; make -j 12 CXX=g++-6 ; extras/install_pocolm.sh )

( cd src; CXX=g++-6 ./configure --shared ; make depend -j 12 ; make -j 12 )


--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com

Private & Confidential
Speechmatics is the trading name of Cantab Research Limited (Reg. in UK: 05697423). This email is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Speechmatics immediately on the sender's email address, then delete it from your system.

Daniel Povey

unread,
Sep 18, 2018, 11:21:30 PM9/18/18
to kaldi-help
What happens when you do "ldd" on the binary?


--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/db341fc0-1e93-9766-1a34-9db5bb538158%40speechmatics.com.
For more options, visit https://groups.google.com/d/optout.

Tony Robinson

unread,
Sep 19, 2018, 3:33:47 AM9/19/18
to kaldi...@googlegroups.com
Hi Dan,

ldd looks like this:

code7 tonyr: ldd /home/tonyr/kaldi/src/bin/est-lda
    linux-vdso.so.1 (0x00007fffd5bcb000)
    libkaldi-transform.so => /home/tonyr/kaldi/src/lib/libkaldi-transform.so (0x00007f2843709000)
    libkaldi-util.so => /home/tonyr/kaldi/src/lib/libkaldi-util.so (0x00007f28434d2000)
    libkaldi-matrix.so => /home/tonyr/kaldi/src/lib/libkaldi-matrix.so (0x00007f2843235000)
    libkaldi-base.so => /home/tonyr/kaldi/src/lib/libkaldi-base.so (0x00007f284302c000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2842c9e000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2842a86000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2842695000)
    libkaldi-gmm.so => /home/tonyr/kaldi/src/lib/libkaldi-gmm.so (0x00007f284245d000)
    libkaldi-tree.so => /home/tonyr/kaldi/src/lib/libkaldi-tree.so (0x00007f284220b000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2841fec000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2841c4e000)
    liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f28413c8000)
    libcblas.so.3 => /usr/lib/x86_64-linux-gnu/libcblas.so.3 (0x00007f28411a6000)
    libf77blas.so.3 => /usr/lib/x86_64-linux-gnu/libf77blas.so.3 (0x00007f2840f84000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f2843b48000)
    libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f283ecde000)
    libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f283e8ff000)
    libatlas.so.3 => /usr/lib/x86_64-linux-gnu/libatlas.so.3 (0x00007f283e376000)
    libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f283e136000)

Now that I think about it, one thing that could be going wrong is that lambda-stack-cuda is installing some conflicting blas library (openblas?) that is messing up the Kaldi configure.   lambda-stack-cuda is a really impressive "one liner" to get started with CUDA, tensorflow, etc, and if it looks too good to be true it probably is.


Tony


--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com

Tony Robinson

unread,
Sep 19, 2018, 6:00:55 AM9/19/18
to kaldi...@googlegroups.com
Thank you Dan, as ever you ask the right questions.

Based on the use of both openblas and atlas below I did a lot more digging.   I haven't as yet found out why libopenblas is being pulled in (but if anyone wants me to try stuff I can).   I did find the instructions to compile using openblas, which for me were:

( cd tools ; ./extras/install_openblas.sh )
( cd src; CXX=g++-6 ./configure --shared --mathlib=OPENBLAS --openblas-root=$PWD/../tools/OpenBLAS/install ; make depend -j 16 ; make -j 16 )

I know I've both got openblas installed and I've compiled it - so that's a bit yukky but it works.

I did have to remove a -lgfortran from ./configure

code7 tonyr: diff kaldi1/src/configure kaldi/src/configure
1339c1339
<       OPENBLASLIBS="-L$OPENBLASLIBDIR -lopenblas -Wl,-rpath=$OPENBLASLIBDIR"
---
>       OPENBLASLIBS="-L$OPENBLASLIBDIR -lopenblas -lgfortran -Wl,-rpath=$OPENBLASLIBDIR"

If I can try out any configurations to help improve Kaldi then please do ask now whilst I can play about with stuff.

Out of curiosity, is there a consensus as to whether OpenBLAS or ATLAS is fastest on a modern Xeon processor?



Tony

--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com

Daniel Povey

unread,
Sep 19, 2018, 12:47:43 PM9/19/18
to kaldi-help
Hm.  You shouldn't have had to change the configure script.
What did "configure" say when you ran it, and what are the contents of kaldi.mk?
It's not supposed to ever pull in both OpenBLAS and ATLAS, because they both provide BLAS.

Dan

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/48886be7-da4f-35dd-395c-dbd7adeaa894%40speechmatics.com.

Tony Robinson

unread,
Sep 20, 2018, 1:44:00 AM9/20/18
to kaldi...@googlegroups.com
Hi Dan,

lambda-stack-cuda aims to be a simple way to get any Ubuntu system installed so people can do ML, so let's assume that it become popular and therefore it's worth time getting a clean install for others.

I agree I shouldn't have had to change ./configure, it was a quick hack as I was quite close.   There has to be a better solution.   To be clear, I only hacked ./configure for -lgfortran when trying to use openblas and it's the standard ATLAS install I assume you are interested in.   If I got this wrong I can start again, it would be clean to autodetect a lambda-stack-cuda installation with openblas libraries and use that without having to install ATLAS or download openblas.

Here is what I get from `CXX=g++-6 ./configure --shared`:

Configuring ...
Checking compiler g++-6 ...
Checking OpenFst library in /home/tonyr/kaldi1/src/matrix/kaldi-debug/tools/openfst ...
Doing OS specific configurations ...
On Linux: Checking for linear algebra header files ...
Using ATLAS as the linear algebra library.
Atlas found in /usr/lib/x86_64-linux-gnu
Validating presence of ATLAS libs in /usr/lib/x86_64-linux-gnu
Using library /usr/lib/x86_64-linux-gnu/liblapack.so as ATLAS's CLAPACK library.
Using CUDA toolkit /usr/ (nvcc compiler and runtime libraries)
Info: configuring Kaldi not to link with Speex (don't worry, it's only needed if you
intend to use 'compress-uncompress-speex', which is very unlikely)
Successfully configured for Linux [dynamic libraries] with ATLASLIBS =/usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libcblas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libf77blas.so
SUCCESS
To compile: make clean -j; make depend -j; make -j
 ... or e.g. -j 10, instead of -j, to use a specified number of CPUs

which looks to me as though it went through linux_check_dynamic() as that's where the "Atlas found in " comes from - so all good so far.

I've attached kaldi.mk.  You'll see I made a mistake in my parent directory, that's not relevant to the discussion.

In general, if there are bite-size chunks I can do to help Kaldi then I can do so.   Kaldi in Indian languages seems to come up a lot, maybe that's a good bite sized chunk for me.  Or maybe a speed test between openblas and ATLAS given that's the thread right now.


Tony

P.S. for the record this run was with:
sudo apt install lambda-stack-cuda liblapack3 libatlas3-base libatlas-base-dev
git clone https://github.com/kaldi-asr/kaldi.git kaldi-debug --origin upstream
( cd tools ; make -j 16 )
( cd src; CXX=g++-6 ./configure --shared ; make depend -j 16 ; make -j 16 )


--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com


For more options, visit https://groups.google.com/d/optout.
kaldi.mk

Daniel Povey

unread,
Sep 20, 2018, 12:58:19 PM9/20/18
to kaldi-help
Hm.  There's nothing in the flags kaldi.mk that refers to linking in openblas, so I'm a little surprised that your "ldd" showed up openblas.  I don't think .so files can somehow pull in other .so files ...   See if "make clean" and "make" resolves the problem. 

.Dan


To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/f2737d7a-d14b-314e-8de2-4c255fbc6995%40speechmatics.com.

joseph.an...@gmail.com

unread,
Sep 20, 2018, 4:53:20 PM9/20/18
to kaldi-help
I've not faced any issues with Kaldi and Cuda on Ubuntu 18.04 both desktop and server editions. I haven't used lamda-stack-cuda and instead used the debian package  from Nvidia's website. 

Regards,
Anand

Tony Robinson

unread,
Sep 21, 2018, 5:23:18 AM9/21/18
to kaldi...@googlegroups.com
It does seem that a .so can pull in another .so.   I traced the problem back and libkaldi-matrix.so is being built using

code7 tonyr: g++-6 -shared -o libkaldi-matrix.so -Wl,--no-undefined -Wl,--as-needed  -Wl,-soname=libkaldi-matrix.so,--whole-archive kaldi-matrix.a -Wl,--no-whole-archive  -Wl,-rpath=/home/tonyr/exp0/kaldi2/tools/openfst/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -rdynamic -Wl,-rpath=/home/exp0/exp/tonyr/kaldi2/src/lib  ../base/libkaldi-base.so /home/tonyr/exp0/kaldi2/tools/openfst/lib/libfst.so /usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libcblas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libf77blas.so -lm -lpthread -ldl

code7 tonyr: ldd libkaldi-matrix.so | egrep openblas
    libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007fcef7ff6000)

code7 tonyr: ldd /usr/lib/x86_64-linux-gnu/liblapack.so | egrep openblas
    libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f8d308c2000)

So it's my liblapack that pulls in libopenblas, even though it seems to be pointing to an ATLAS version:


code7 tonyr: ls -lta /usr/lib/x86_64-linux-gnu/liblapack.so
lrwxrwxrwx 1 root root 47 Sep 18 15:41 /usr/lib/x86_64-linux-gnu/liblapack.so -> /etc/alternatives/liblapack.so-x86_64-linux-gnu
code7 tonyr: ls -lta etc/alternatives/liblapack.so-x86_64-linux-gnu
ls: cannot access 'etc/alternatives/liblapack.so-x86_64-linux-gnu': No such file or directory
code7 tonyr: ls -lta /etc/alternatives/liblapack.so-x86_64-linux-gnu
lrwxrwxrwx 1 root root 44 Sep 18 15:41 /etc/alternatives/liblapack.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/atlas/liblapack.so
code7 tonyr: ldd /usr/lib/x86_64-linux-gnu/atlas/liblapack.so | egrep openblas
    libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f01748fb000)

We are of course assuming that the inclusion of libopenblas.so is causing the failure to resolve clapack_dgetrf.

I really don't see this as a big issue for Kaldi unless lambda-stack-cuda becomes popular - seems like one to document the work around...



Tony

--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com

Tony Robinson

unread,
Sep 21, 2018, 6:12:10 AM9/21/18
to kaldi...@googlegroups.com
Thanks Anand.   Having everything ML related install (and automatically update) with one line really appealed to me at the time, but now I've spent more time than if I'd gone down the route of install the individual packages, as I have done before.


Tony


On 20/09/18 21:53, an...@sayint.ai wrote:
I've not faced any issues with Kaldi and Cuda on Ubuntu 18.04 both desktop and server editions. I haven't used lamda-stack-cuda and instead used the debian package  from Nvidia's website. 

Regards,
Anand


For more options, visit https://groups.google.com/d/optout.

Daniel Povey

unread,
Sep 21, 2018, 12:16:35 PM9/21/18
to kaldi-help
Hm.  OpenBLAS and ATLAS both provide a library by the name of liblapack.  I suspect that what happened is that OpenBLAS overwrote ATLAS's version of liblapack there, and the symbol names are different.  Maybe if you use that package you have to configure as for OpenBLAS, e.g. with --openblas-root=/usr.   It might not work right if both ATLAS and OpenBLAS are installed there though, as they have some of the same header names.  It also looks to me like the configure script doesn't yet support the 'lib' subdirectory being named 'x86_64-linux-gnu' though.

You aren't really supposed to install both ATLAS and OpenBLAS on the same machine.

Dan


To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/ef20d5cd-b6ec-026b-2da1-55648c0b436a%40speechmatics.com.

Tony Robinson

unread,
Sep 22, 2018, 3:51:02 AM9/22/18
to kaldi...@googlegroups.com
I'm very happy with your last statement.   So lambda-stack-cuda users will have to go the OpenBLAS route.  That seems perfectly fine provided they know they have to do it from the start (which they should find out by searching this group and finding this thread).



Tony

--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com

Reply all
Reply to author
Forward
0 new messages