libkaldi-matrix.so: undefined symbol: clapack_dgetrf
Tony Robinson
I'm doing what I thought should be a standard install of Kaldi.
What's changed is that this is first time I've done this on Ubuntu
18.4 and g++-6 (latest GPU stuff installed with lambda-stack-cuda
which is great)
I'm getting:
code7 tonyr: /home/tonyr/kaldi/src/bin/est-lda
--write-full-matrix=exp/tri2/full.mat --dim=40 exp/tri2/0.mat
exp/tri2/lda.*acc
/home/tonyr/kaldi/src/bin/est-lda
--write-full-matrix=exp/tri2/full.mat --dim=40 exp/tri2/0.mat
exp/tri2/lda.10.acc exp/tri2/lda.11.acc exp/tri2/lda.12.acc
exp/tri2/lda.13.acc exp/tri2/lda.14.acc exp/tri2/lda.15.acc
exp/tri2/lda.16.acc exp/tri2/lda.17.acc exp/tri2/lda.18.acc
exp/tri2/lda.19.acc exp/tri2/lda.1.acc exp/tri2/lda.20.acc
exp/tri2/lda.21.acc exp/tri2/lda.22.acc exp/tri2/lda.23.acc
exp/tri2/lda.24.acc exp/tri2/lda.25.acc exp/tri2/lda.26.acc
exp/tri2/lda.27.acc exp/tri2/lda.28.acc exp/tri2/lda.29.acc
exp/tri2/lda.2.acc exp/tri2/lda.30.acc exp/tri2/lda.31.acc
exp/tri2/lda.32.acc exp/tri2/lda.33.acc exp/tri2/lda.34.acc
exp/tri2/lda.35.acc exp/tri2/lda.3.acc exp/tri2/lda.4.acc
exp/tri2/lda.5.acc exp/tri2/lda.6.acc exp/tri2/lda.7.acc
exp/tri2/lda.8.acc exp/tri2/lda.9.acc
/home/tonyr/kaldi/src/bin/est-lda: symbol lookup error:
/home/tonyr/kaldi/src/lib/libkaldi-matrix.so: undefined symbol:
clapack_dgetrf
From where I Read The Fine Documentation at http://kaldi-asr.org/doc/matrixwrap.html under "Missing the ATLAS implementation of (parts of) CLAPACK"
I have atlas:
root@code7# apt install libatlas-base-dev libatlas3-base
Reading package lists... Done
Building dependency tree
Reading state information... Done
libatlas-base-dev is already the newest version (3.10.3-5).
libatlas3-base is already the newest version (3.10.3-5).
I can see the dependencies:
code7 tonyr: make
g++-6 -Wl,-rpath=/home/tonyr/kaldi/tools/openfst/lib
-Wl,-rpath,/usr/lib/x86_64-linux-gnu -rdynamic
-Wl,-rpath=/home/tonyr/kaldi/src/lib est-lda.o
../decoder/libkaldi-decoder.so ../lat/libkaldi-lat.so
../lm/libkaldi-lm.so ../fstext/libkaldi-fstext.so
../hmm/libkaldi-hmm.so ../transform/libkaldi-transform.so
../gmm/libkaldi-gmm.so ../tree/libkaldi-tree.so
../util/libkaldi-util.so ../matrix/libkaldi-matrix.so
../base/libkaldi-base.so
/home/tonyr/kaldi/tools/openfst/lib/libfst.so
/usr/lib/x86_64-linux-gnu/liblapack.so
/usr/lib/x86_64-linux-gnu/libcblas.so
/usr/lib/x86_64-linux-gnu/libatlas.so
/usr/lib/x86_64-linux-gnu/libf77blas.so -lm -lpthread -ldl -o
est-lda
The dynamic libraries seems to be installed fine:
code7 tonyr: ls -lta /usr/lib/x86_64-linux-gnu/liblapack.so
/usr/lib/x86_64-linux-gnu/libcblas.so
/usr/lib/x86_64-linux-gnu/libatlas.so
/usr/lib/x86_64-linux-gnu/libf77blas.so
lrwxrwxrwx 1 root root 47 Sep 18 15:41
/usr/lib/x86_64-linux-gnu/liblapack.so ->
/etc/alternatives/liblapack.so-x86_64-linux-gnu
lrwxrwxrwx 1 root root 18 Sep 13 2017
/usr/lib/x86_64-linux-gnu/libatlas.so -> libatlas.so.3.10.3
lrwxrwxrwx 1 root root 18 Sep 13 2017
/usr/lib/x86_64-linux-gnu/libcblas.so -> libcblas.so.3.10.3
lrwxrwxrwx 1 root root 20 Sep 13 2017
/usr/lib/x86_64-linux-gnu/libf77blas.so -> libf77blas.so.3.10.3
and the debug in http://kaldi-asr.org/doc/matrixwrap.html also seems to be fine:
root@code7# strings /usr/lib/x86_64-linux-gnu/liblapack.so |
egrep ATL_[cd]getrf$
ATL_cgetrf
ATL_dgetrf
root@code7:# strings /usr/lib/x86_64-linux-gnu/liblapack.so |
egrep clapack_dgetrf
clapack_dgetrf
That's as far as I've got. All of this would seem to be the latest Ubuntu and CUDA and so perhaps someone else has had this problem and fixed it. I see nothing in the googlegroup archives except problems with manually compiling ATLAS.
Anyone else seen this? No doubt I'll fix it as soon as I post....
Tony
P.S. full install instructions:
git clone https://github.com/kaldi-asr/kaldi.git kaldi --origin
upstream
cd kaldi
( cd tools ; make -j 12 CXX=g++-6 ; extras/install_pocolm.sh )
( cd src; CXX=g++-6 ./configure --shared ; make depend -j 12 ;
make -j 12 )
--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com
Speechmatics is the trading name of Cantab Research Limited (Reg. in UK: 05697423). This email is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Speechmatics immediately on the sender's email address, then delete it from your system.
Daniel Povey
--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/db341fc0-1e93-9766-1a34-9db5bb538158%40speechmatics.com.
For more options, visit https://groups.google.com/d/optout.
Tony Robinson
ldd looks like this:
code7 tonyr: ldd /home/tonyr/kaldi/src/bin/est-lda
linux-vdso.so.1 (0x00007fffd5bcb000)
libkaldi-transform.so => /home/tonyr/kaldi/src/lib/libkaldi-transform.so (0x00007f2843709000)
libkaldi-util.so => /home/tonyr/kaldi/src/lib/libkaldi-util.so (0x00007f28434d2000)
libkaldi-matrix.so => /home/tonyr/kaldi/src/lib/libkaldi-matrix.so (0x00007f2843235000)
libkaldi-base.so => /home/tonyr/kaldi/src/lib/libkaldi-base.so (0x00007f284302c000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2842c9e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2842a86000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2842695000)
libkaldi-gmm.so => /home/tonyr/kaldi/src/lib/libkaldi-gmm.so (0x00007f284245d000)
libkaldi-tree.so => /home/tonyr/kaldi/src/lib/libkaldi-tree.so (0x00007f284220b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2841fec000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2841c4e000)
liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f28413c8000)
libcblas.so.3 => /usr/lib/x86_64-linux-gnu/libcblas.so.3 (0x00007f28411a6000)
libf77blas.so.3 => /usr/lib/x86_64-linux-gnu/libf77blas.so.3 (0x00007f2840f84000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2843b48000)
libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f283ecde000)
libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f283e8ff000)
libatlas.so.3 => /usr/lib/x86_64-linux-gnu/libatlas.so.3 (0x00007f283e376000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f283e136000)
Now that I think about it, one thing that could be going wrong is that lambda-stack-cuda is installing some conflicting blas library (openblas?) that is messing up the Kaldi configure. lambda-stack-cuda is a really impressive "one liner" to get started with CUDA, tensorflow, etc, and if it looks too good to be true it probably is.
Tony
--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAEWAuyTKsNfVUiY16BK%3Dzehej5QLvsPSvLm2RwV6NY%3DZ9daALg%40mail.gmail.com.
Tony Robinson
Based on the use of both openblas and atlas below I did a lot more digging. I haven't as yet found out why libopenblas is being pulled in (but if anyone wants me to try stuff I can). I did find the instructions to compile using openblas, which for me were:
( cd tools ; ./extras/install_openblas.sh )
( cd src; CXX=g++-6 ./configure --shared --mathlib=OPENBLAS --openblas-root=$PWD/../tools/OpenBLAS/install ; make depend -j 16 ; make -j 16 )
I know I've both got openblas installed and I've compiled it - so that's a bit yukky but it works.
I did have to remove a -lgfortran from ./configure
code7 tonyr: diff kaldi1/src/configure kaldi/src/configure
1339c1339
< OPENBLASLIBS="-L$OPENBLASLIBDIR -lopenblas -Wl,-rpath=$OPENBLASLIBDIR"
---
> OPENBLASLIBS="-L$OPENBLASLIBDIR -lopenblas -lgfortran -Wl,-rpath=$OPENBLASLIBDIR"
If I can try out any configurations to help improve Kaldi then please do ask now whilst I can play about with stuff.
Out of curiosity, is there a consensus as to whether OpenBLAS or ATLAS is fastest on a modern Xeon processor?
Tony
--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com
Daniel Povey
Dan
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/48886be7-da4f-35dd-395c-dbd7adeaa894%40speechmatics.com.
Tony Robinson
lambda-stack-cuda aims to be a simple way to get any Ubuntu system installed so people can do ML, so let's assume that it become popular and therefore it's worth time getting a clean install for others.
I agree I shouldn't have had to change ./configure, it was a quick hack as I was quite close. There has to be a better solution. To be clear, I only hacked ./configure for -lgfortran when trying to use openblas and it's the standard ATLAS install I assume you are interested in. If I got this wrong I can start again, it would be clean to autodetect a lambda-stack-cuda installation with openblas libraries and use that without having to install ATLAS or download openblas.
Here is what I get from `CXX=g++-6 ./configure --shared`:
Configuring ...
Checking compiler g++-6 ...
Checking OpenFst library in /home/tonyr/kaldi1/src/matrix/kaldi-debug/tools/openfst ...
Doing OS specific configurations ...
On Linux: Checking for linear algebra header files ...
Using ATLAS as the linear algebra library.
Atlas found in /usr/lib/x86_64-linux-gnu
Validating presence of ATLAS libs in /usr/lib/x86_64-linux-gnu
Using library /usr/lib/x86_64-linux-gnu/liblapack.so as ATLAS's CLAPACK library.
Using CUDA toolkit /usr/ (nvcc compiler and runtime libraries)
Info: configuring Kaldi not to link with Speex (don't worry, it's only needed if you
intend to use 'compress-uncompress-speex', which is very unlikely)
Successfully configured for Linux [dynamic libraries] with ATLASLIBS =/usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libcblas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libf77blas.so
SUCCESS
To compile: make clean -j; make depend -j; make -j
... or e.g. -j 10, instead of -j, to use a specified number of CPUs
which looks to me as though it went through linux_check_dynamic() as that's where the "Atlas found in " comes from - so all good so far.
I've attached kaldi.mk. You'll see I made a mistake in my parent directory, that's not relevant to the discussion.
In general, if there are bite-size chunks I can do to help Kaldi then I can do so. Kaldi in Indian languages seems to come up a lot, maybe that's a good bite sized chunk for me. Or maybe a speed test between openblas and ATLAS given that's the thread right now.
Tony
P.S. for the record this run was with:
sudo apt install lambda-stack-cuda liblapack3 libatlas3-base libatlas-base-dev
git clone https://github.com/kaldi-asr/kaldi.git kaldi-debug --origin upstream
( cd tools ; make -j 16 )
( cd src; CXX=g++-6 ./configure --shared ; make depend -j 16 ; make -j 16 )
--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAEWAuyTF7TL7D-6uNGHxuFz%3DtcvJ%3DP4%2BaJkmM8YG%2ByRY_WfduQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Daniel Povey
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/f2737d7a-d14b-314e-8de2-4c255fbc6995%40speechmatics.com.
joseph.an...@gmail.com
Tony Robinson
code7 tonyr: g++-6 -shared -o libkaldi-matrix.so -Wl,--no-undefined -Wl,--as-needed -Wl,-soname=libkaldi-matrix.so,--whole-archive kaldi-matrix.a -Wl,--no-whole-archive -Wl,-rpath=/home/tonyr/exp0/kaldi2/tools/openfst/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -rdynamic -Wl,-rpath=/home/exp0/exp/tonyr/kaldi2/src/lib ../base/libkaldi-base.so /home/tonyr/exp0/kaldi2/tools/openfst/lib/libfst.so /usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libcblas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libf77blas.so -lm -lpthread -ldl
code7 tonyr: ldd libkaldi-matrix.so | egrep openblas
libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007fcef7ff6000)
code7 tonyr: ldd /usr/lib/x86_64-linux-gnu/liblapack.so | egrep openblas
libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f8d308c2000)
So it's my liblapack that pulls in libopenblas, even though it seems to be pointing to an ATLAS version:
code7 tonyr: ls -lta /usr/lib/x86_64-linux-gnu/liblapack.so
ls: cannot access 'etc/alternatives/liblapack.so-x86_64-linux-gnu': No such file or directory
code7 tonyr: ls -lta /etc/alternatives/liblapack.so-x86_64-linux-gnu
lrwxrwxrwx 1 root root 44 Sep 18 15:41 /etc/alternatives/liblapack.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/atlas/liblapack.so
code7 tonyr: ldd /usr/lib/x86_64-linux-gnu/atlas/liblapack.so | egrep openblas
libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f01748fb000)
We are of course assuming that the inclusion of libopenblas.so is causing the failure to resolve clapack_dgetrf.
I really don't see this as a big issue for Kaldi unless lambda-stack-cuda becomes popular - seems like one to document the work around...
Tony
--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAEWAuyS64iAc-51aSuYEZcpbcutpvd%3DDjqz0NpwD1Af6Zf-kug%40mail.gmail.com.
Tony Robinson
Tony
I've not faced any issues with Kaldi and Cuda on Ubuntu 18.04 both desktop and server editions. I haven't used lamda-stack-cuda and instead used the debian package from Nvidia's website.
Regards,
Anand
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/57e21f1e-a0bf-47dc-a6e1-c02ac336e9e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Daniel Povey
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/ef20d5cd-b6ec-026b-2da1-55648c0b436a%40speechmatics.com.
Tony Robinson
Tony
--
Tony Robinson
CTO, Speechmatics
Mobile: +44 (0)7808 165099 | LinkedIn | Twitter
www.speechmatics.com
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAEWAuyTv62iZn%3DTmwZWpU5_Moz9JWUd3%2BE_yTkbUg8wnyai_XA%40mail.gmail.com.