cuda error when i make src

1,256 views
Skip to first unread message

wangj...@gmail.com

unread,
Apr 26, 2019, 8:19:37 AM4/26/19
to kaldi-help
make[1]: Leaving directory `/home/wjs/tools/kaldi/src/rnnlm'
make -C bin
make[1]: Entering directory `/home/wjs/tools/kaldi/src/bin'
rm -f arpa2fst
make[1]: Leaving directory `/home/wjs/tools/kaldi/src/bin'
make -C fstbin
make[1]: Entering directory `/home/wjs/tools/kaldi/src/fstbin'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/wjs/tools/kaldi/src/fstbin'
make -C gmmbin
make[1]: Entering directory `/home/wjs/tools/kaldi/src/gmmbin'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/wjs/tools/kaldi/src/gmmbin'
make -C fgmmbin
make[1]: Entering directory `/home/wjs/tools/kaldi/src/fgmmbin'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/wjs/tools/kaldi/src/fgmmbin'
make -C featbin
make[1]: Entering directory `/home/wjs/tools/kaldi/src/featbin'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/wjs/tools/kaldi/src/featbin'
make -C nnetbin
make[1]: Entering directory `/home/wjs/tools/kaldi/src/nnetbin'
g++  -Wl,-rpath=/home/wjs/tools/kaldi/tools/openfst-1.6.7/lib -rdynamic -L/home/wjs/local/lib -L/home/wjs/local/lib64 -L/home/wjs/local/lib/gstreamer-1.0 -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/local/cuda/lib64 -Wl,-rpath=/home/wjs/tools/kaldi/src/lib  nnet-train-frmshuff.o   ../nnet/libkaldi-nnet.so  ../cudamatrix/libkaldi-cudamatrix.so  ../lat/libkaldi-lat.so  ../hmm/libkaldi-hmm.so  ../tree/libkaldi-tree.so  ../util/libkaldi-util.so  ../matrix/libkaldi-matrix.so  ../base/libkaldi-base.so /home/wjs/tools/kaldi/tools/openfst-1.6.7/lib/libfst.so -L/home/wjs/tools/kaldi/tools/OpenBLAS/install/lib -lopenblas -lgfortran -Wl,-rpath=/home/wjs/tools/kaldi/tools/OpenBLAS/install/lib -lm -lpthread -ldl -lcublas -lcusparse -lcusolver -lcudart -lcurand -lnvToolsExt  -o nnet-train-frmshuff
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_parallel_start'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_critical_start'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_loop_dynamic_start'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_critical_end'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `omp_get_max_threads'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_loop_end_nowait'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `omp_get_thread_num'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_loop_dynamic_next'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_parallel_end'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `omp_get_num_threads'
collect2: error: ld returned 1 exit status
make[1]: *** [nnet-train-frmshuff] Error 1
make[1]: Leaving directory `/home/wjs/tools/kaldi/src/nnetbin'
make: *** [nnetbin] Error 2

wangj...@gmail.com

unread,
Apr 26, 2019, 10:16:24 AM4/26/19
to kaldi-help
i just checkout to the commit half a mouth before, and compile success

the commit i checkout is：

commit b96cab759caf411559d37d2c76b2af4008af61ed (HEAD)
Author: Tien-Hong Lo <teinh...@gmail.com>
Date:   Wed Apr 10 08:48:18 2019 +0800

    [build] Add missing line continuation '\' in tfrnnlmbin/Makefile (#3218)

Jan Trmal

unread,
Apr 26, 2019, 10:26:47 AM4/26/19
to kaldi-help
it's missing OMP library, but I'm not sure why. Did you try to do it from scratch, i.e. make clean && make depends && make ?
y

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/f876c82a-d7be-48ad-b775-fac29ee7fa24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Daniel Povey

unread,
Apr 26, 2019, 12:21:18 PM4/26/19
to kaldi-help, Justin Luitjens, Ruizhe Huang
We added cusolver in a very recent commit but it wasn't actually currently needed for anything right now (only something Justin planned to do later).
It's part of this commit.

I assumed it was always present when cublas is present, but it seems it has an extra dependency.  I think the easiest fix would be to just remove cusolver.  I am asking Ruizhe to prepare a PR to do this (although if Justin says there is another fix, I am open).

Dan


To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAFReZQY9gxjP9U388bRAyY8paGkZ_Dse1HSH8_%3D-ja4apYJY5A%40mail.gmail.com.

Daniel Povey

unread,
Apr 26, 2019, 12:36:02 PM4/26/19
to Justin Luitjens, kaldi-help, Ruizhe Huang, Ryan Leary
Yes I think it is a dealbreaker, at least until we figure out how easy it is to install and what value it brings (which would, currently, be zero since the cusolver stuff is not used).

On Fri, Apr 26, 2019 at 12:33 PM Justin Luitjens <jlui...@nvidia.com> wrote:

Looking more closely it seems the error is that cusolver has an omp dependency.  Is adding -lomp a dealbreaker?

 

From: Justin Luitjens
Sent: Friday, April 26, 2019 10:30 AM
To: dpo...@gmail.com; kaldi-help <kaldi...@googlegroups.com>; Ruizhe Huang <huangr...@gmail.com>
Cc: Ryan Leary <rle...@nvidia.com>
Subject: RE: [kaldi-help] Re: cuda error when i make src

 

Instead of reverting what if I add compiler guards that prevent it from being compiled with older toolkits?

This email message is for the sole use of the intended recipient(s) and may contain confidential information.  Any unauthorized review, use, disclosure or distribution is prohibited.  If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

Daniel Povey

unread,
Apr 26, 2019, 1:48:03 PM4/26/19
to Justin Luitjens, kaldi-help, Ruizhe Huang, Ryan Leary
What toolkit version did you have?
You could also try running `configure` again and see what happens (back up your kaldi.mk first though).  It could be your toolkit version is unsupported.

wangj...@gmail.com

unread,
Apr 26, 2019, 11:49:00 PM4/26/19
to kaldi-help
when i run configure,everything is ok

在 2019年4月27日星期六 UTC+8上午1:48:03，Dan Povey写道：

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.
To post to this group, send email to kald...@googlegroups.com.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.
To post to this group, send email to kald...@googlegroups.com.

Daniel Povey

unread,
Apr 27, 2019, 12:52:00 AM4/27/19
to kaldi-help
So what cuda toolkit version are you using?


To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/35acc1a2-d012-4375-93bb-ce7acecb860b%40googlegroups.com.

wangj...@gmail.com

unread,
Apr 27, 2019, 8:25:31 AM4/27/19
to kaldi-help
cuda8.0

在 2019年4月27日星期六 UTC+8下午12:52:00，Dan Povey写道：

wangj...@gmail.com

unread,
Apr 27, 2019, 8:26:05 AM4/27/19
to kaldi-help
should i upgrade cuda?


在 2019年4月27日星期六 UTC+8下午12:52:00，Dan Povey写道：
So what cuda toolkit version are you using?


Justin Luitjens

unread,
Apr 27, 2019, 9:53:21 AM4/27/19
to kaldi...@googlegroups.com
so when you ran configure the error went away? 

From: kaldi...@googlegroups.com <kaldi...@googlegroups.com> on behalf of wangj...@gmail.com <wangj...@gmail.com>
Sent: Saturday, April 27, 2019 6:26:05 AM
To: kaldi-help
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

wangj...@gmail.com

unread,
Apr 27, 2019, 11:04:22 AM4/27/19
to kaldi-help
error appeard again....
what i run:

cd kaldi/tools
extra/check_dependencies.sh
make
make openblas
cd ../src
./configure --openblas-root=../tools/OpenBLAS/install --shared
make clean
make depend -j 40
make -j 40

expect for the last step(make -j 40), there are no errors


在 2019年4月27日星期六 UTC+8下午9:53:21，Justin Luitjens写道：

Daniel Povey

unread,
Apr 27, 2019, 11:09:34 AM4/27/19
to kaldi-help
Justin, the following might indicate that the requirement was dropped in 9.1.  

root@b09:~# ldd /opt/NVIDIA/cuda-8.0/lib64/libcusolver.so
        linux-vdso.so.1 (0x00007ffc885ed000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007faae6fc9000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007faae6dac000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007faae6ba4000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007faae6822000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007faae651e000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007faae6307000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faae5f68000)
        /lib64/ld-linux-x86-64.so.2 (0x00007faaea73d000)
root@b09:~# objdump -TC /opt/NVIDIA/cuda-8.0/lib64/libcusolver.so | grep -i omp
0000000000000000      D  *UND*  0000000000000000              GOMP_critical_end
0000000000000000      D  *UND*  0000000000000000              omp_get_thread_num
0000000000000000      D  *UND*  0000000000000000              omp_get_num_threads
0000000000000000      D  *UND*  0000000000000000              GOMP_parallel_start
0000000000000000      D  *UND*  0000000000000000              omp_get_max_threads
0000000000000000      D  *UND*  0000000000000000              GOMP_loop_end_nowait
0000000000000000      D  *UND*  0000000000000000              GOMP_critical_start
0000000000000000      D  *UND*  0000000000000000              GOMP_parallel_end
0000000000000000      D  *UND*  0000000000000000              GOMP_loop_dynamic_next
0000000000000000      D  *UND*  0000000000000000              GOMP_loop_dynamic_start
root@b09:~# objdump -TC /opt/NVIDIA/cuda-9.0/lib64/libcusolver.so | grep -i omp
0000000000000000      D  *UND*  0000000000000000              GOMP_critical_end
0000000000000000      D  *UND*  0000000000000000              omp_get_thread_num
0000000000000000      D  *UND*  0000000000000000              omp_get_num_threads
0000000000000000      D  *UND*  0000000000000000              GOMP_parallel_start
0000000000000000      D  *UND*  0000000000000000              omp_get_max_threads
0000000000000000      D  *UND*  0000000000000000              GOMP_loop_end_nowait
0000000000000000      D  *UND*  0000000000000000              GOMP_critical_start
0000000000000000      D  *UND*  0000000000000000              GOMP_parallel_end
0000000000000000      D  *UND*  0000000000000000              GOMP_loop_dynamic_next
0000000000000000      D  *UND*  0000000000000000              GOMP_loop_dynamic_start
root@b09:~# objdump -TC /opt/NVIDIA/cuda-9.1/lib64/libcusolver.so | grep -i omp
root@b09:~# 


To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/dd152938-afb2-4d98-9901-e34ea27bedf1%40googlegroups.com.

Justin Luitjens

unread,
Apr 27, 2019, 11:26:48 AM4/27/19
to kaldi...@googlegroups.com
right that is what my library guys told me.  I'm confused on how a reconfigure fixed the issue for older toolkits.  maybe it was statically linked in the library?

From: kaldi...@googlegroups.com <kaldi...@googlegroups.com> on behalf of Daniel Povey <dpo...@gmail.com>
Sent: Saturday, April 27, 2019 9:09:18 AM

Daniel Povey

unread,
Apr 27, 2019, 11:31:58 AM4/27/19
to kaldi-help
I think he's saying it didn't fix the issue.
I'm asking Ruizhe to revert that part of the patch for now.  You can just take the diff from that commit and reverse it and apply to your own branch, it shouldn't cause major merge conflicts.  We can always include it later on; anyway, usage of older toolkits will decline with time (we can also include the omp stuff for older toolkits).

If the proper fix for this is going to require changes to kaldi.mk, it would require bumping the configure version to force people to re-run configure, and I don't want to do that unless there is a good reason.


To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/BYAPR12MB2679E51CF58D3B3117FBAB5BD43F0%40BYAPR12MB2679.namprd12.prod.outlook.com.

Justin Luitjens

unread,
Apr 27, 2019, 1:46:53 PM4/27/19
to kaldi...@googlegroups.com
ok, eventually we will want this library.  we could actually use it for choleskeys right now and speed up that routine.   certainly the users should upgrade to a newer version of cuda when possible but I understand we don't want to force this.  

in the future we should discuss adding a minimum cuda version to configure.  perhaps at the same time that we readd cusolver. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAEWAuySMxkymw%2BwNhBWdUDfZi_eSt78mCT3FxWGFs8H%3DVeRB4w%40mail.gmail.com.

wangj...@gmail.com

unread,
Apr 27, 2019, 11:20:44 PM4/27/19
to kaldi-help
so...the minimum cuda version is cuda 10?

在 2019年4月28日星期日 UTC+8上午1:46:53，Justin Luitjens写道：

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.
To post to this group, send email to kald...@googlegroups.com.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.

Daniel Povey

unread,
Apr 27, 2019, 11:55:34 PM4/27/19
to kaldi-help
No, it's somethin.g like 6 or 7.  If you do "git pull" and reconfigure, the problem should be resolved now.


To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/93f49356-4fbb-4fb6-9657-46ffd00579a1%40googlegroups.com.

wangj...@gmail.com

unread,
Apr 28, 2019, 1:32:04 AM4/28/19
to kaldi-help
thank you for your pattient~

在 2019年4月28日星期日 UTC+8上午11:55:34，Dan Povey写道：

panjinsong

unread,
Jun 27, 2019, 10:11:39 AM6/27/19
to kaldi-help
Dan

the version of my machine's CUDA is 9.0, and the the version of g++ is 6.3.  I also encountered similar problem when I compiled the makefile in kaldi/src. it reminded me:

/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_parallel_start'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_critical_start'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_loop_dynamic_start'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_critical_end'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `omp_get_max_threads'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_loop_end_nowait'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `omp_get_thread_num'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_loop_dynamic_next'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `GOMP_parallel_end'
/usr/local/cuda/lib64/libcusolver.so: undefined reference to `omp_get_num_threads'
collect2: error: ld returned 1 exit status
<builtin>: recipe target 'lattice-best-path' failed
make[1]: *** [nnet-train-frmshuff] Error 1

how can I solve it?
thanks

pan

在 2019年4月26日星期五 UTC+8下午8:19:37，wangj...@gmail.com写道：

Daniel Povey

unread,
Jun 27, 2019, 1:39:26 PM6/27/19
to kaldi-help
Try updating your repo and reconfiguring, we fixed some `configure` bugs about this yesterday.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/0a42d748-e2ad-4579-9f23-623cd2fd4a4c%40googlegroups.com.

panjinsong

unread,
Jun 28, 2019, 2:11:18 AM6/28/19
to kaldi-help
Dan
Thanks for your reply.
I updated my repo and reconfiguring as you said. But when I finished configuring and executing 'make depend -j8', it reminded me:

kaldi ../cudafeat/feature-mfcc-cuda.h:22:19: fatal error:cufft.h: No such file or directory
#include <cufft.h>
compilation terminated.
online-ivector-feature-cuda.cc:19:24: fatal error:no toolsExt.h:No such file or directory
#include < toolsExt.h>
compilation terminated.
../makefiles/defult_rules.mk:141. recipe for target 'depend' failed
make[1]: ***[depend] Error 1
make[1]: leaving directory '/home/panjinsong/dl_projects/ASR/kaldi/src/cudafeat'
Makefile:127: recipe for target 'cudafeat/depend'; failed
make[1]: ***['cudafeat/depend] Error 2

what's wrong with it?
pan

在 2019年6月28日星期五 UTC+8上午1:39:26，Dan Povey写道：
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.

Daniel Povey

unread,
Jun 28, 2019, 11:28:17 AM6/28/19
to kaldi-help
Show me what CUDA version you have (e.g. nvcc --version) and what CUDA-related things are in your kaldi.mk.


To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/2f370e23-e5ef-49b5-84df-e70d6fcce38d%40googlegroups.com.

Daniel Povey

unread,
Jun 28, 2019, 11:33:08 AM6/28/19
to kaldi-help
Also I would need to see more Makefile output than that, I can't see what it was trying to do.

panjinsong

unread,
Jul 1, 2019, 3:18:40 AM7/1/19
to kaldi-help
Dan

I manually included the path to my cuda folder :
export CPLUS_INCLUDE_PATH=/usr/local/cuda-9.0/include/:$CPLUS_INCLUDE_PATH
then:

the output of 'find / -name cublas_v2.h':
/usr/local/cuda-9.0/include/cublas_v2.h


the output of 'nvcc --version':
cuda compilation tools, release 9.0, V9.0.176


the output of 'grep -i cuda kaldi.mk':
CUDA = true
CUDATKDIR = /usr/
CUDA_ARCH = -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50, code=sm_50,arch=compute_52, code=sm_52 \ 
-gencode arch=compute_60, code=sm_60, -gencode arch=compute_61, code=sm_61 -gencode arch=compute_70, code=sm_70
ifnedef CUDATKDIR
$(error CUDATKDIR not defined.)
CXXFLAGS += -DHAVE_CUDA -I$(CUDATKDIR)/include -fPIC -pthread -isystem $(OPENFSTINC)
CUDA_INCLUDE= -I$(CUDATKDIR)/include -I$(CUBROOT)
CUDA_FLAGS = --machine 64 -DHAVE_CUDA \
                            -std=c++11 -DCUDA_API_PER_THREAD_DEFAULT_STREAM -lineinfo \
CUDA_LDFLAGS += -L$(CUDATKDIR).lib64 -Wl, -rpath, $(CUDATHDIR)/lib64
CUDA_LDLIBS += -lcublas -lcusparse -cudart -lcurand -lcufft -lnvToolsExt # LDLIBS : The .so libs are loaded later than static libs in implicit rule


the output of 'make -j12'(extract):
nvcc fatal : Path to libdevice library not specified
Makefile:32: recope for target 'cu-kernels.o' failed
make[1]: *** [cu-kernels.o] Error 1
...
make[1]: leaving directory '/home/panjinsong/dl_projects/ASR/kaldi/src/cudamatrix'
Makefile:136: recope for target 'cudamatrix' failed
make: *** [cudamatrix] Error 2

Because of the permission problem, I can't copy the output to the network.so I manual output of key information of 'make -j12'.

thanks
pan


在 2019年6月28日星期五 UTC+8下午11:33:08，Dan Povey写道：

Daniel Povey

unread,
Jul 1, 2019, 2:27:18 PM7/1/19
to kaldi-help
Are you saying you manually edited kaldi.mk ?   That doesn't sound right.  You should probably have done something like
./configure --cudatk-dir=/usr/local/cuda-9.0

The configure command and full output of configure would help too.



To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/0407afd4-f5e2-4268-8fb1-fb22e8978831%40googlegroups.com.

panjinsong

unread,
Jul 1, 2019, 10:17:21 PM7/1/19
to kaldi-help
Dan

 the full output when'./configure'is executed in the folder'kaldi/src/'  :

configuringKALDI to use MKL.
Backinbg up kaldi.mk to kaidi.mk.nak ...
Checking compiler g++ ...
Checking OpenFst library in /home/panjinsong/dl_projects/ASR/kaldi/tools/openfst-1.6.7 ...
Checking cub library in /home/panjinsong/dl_projects/ASR/kaldi/tools/cub-1.8.0 ...
Doing OS specific configurations ...
On Linux: Checking for linear algebra header files ...
Configuring MKL library directory: Found: /opt/intel/mkl/lib/intel64
MKL configured with threading: sequential, libs: -L/opt/intel/mkl/lib/intel64 -Wl,-rpath=/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lm
kl_core -lmkl_sequential
MKL include directory configured as: /opt/intel/mkl/include
Configuring MKL threading as sequential
MKL threading libraries configured as -ldl -lpthread -lm
Using Intel MKL as th linear algebra library.
Inter(R) Math kernel Library Version 2019.0.1 Product Build 2018928 for Inter(R) 64 architecture applications
Successfully configured for Linux  with MKL libs from /opt/intel/mkl
Using CUDA toolkit /usr/ (nvcc compiler and runtime librariers)
INFO: Configuring Kaldi not to link with Speex. Don't worry, it's only needed if
           you intend to use 'compress0uncompress-speex', which is very unlikely.
Kaldi has been successfully configured. To compile:

  make -j clean depend; make -j <NCPU>

where <NCPU> is the number of paraller builds you can afford to do. If unsure,
use the smaller of the number of CPUs or the amount of RAM in GB divided by 2,
to stay within safe limits. 'make -j' without the numeric value may not limit 
the number of paraller jobs at all, and overwhelm even a powerful workstation,
since Kaldi build is highly parallelized.

sorry, you misunderstood me. I mean I have to manually transfer the output from the server to the internet for some permission problems, I can't show you the full output of 'make -j12', it's too long.
在 2019年7月2日星期二 UTC+8上午2:27:18，Dan Povey写道：

Justin Luitjens

unread,
Jul 1, 2019, 10:20:57 PM7/1/19
to kaldi...@googlegroups.com
Cusolver is only supposed to be linked if cuda is greater than 9.0.  There must be something that is not detected right here.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

Justin Luitjens

unread,
Jul 1, 2019, 10:54:02 PM7/1/19
to kaldi...@googlegroups.com
Looking at configure I wonder if the 9_* is the problem in the case.  I never noticed that before and it seems out of place.  Dan do you agree 9_* should be removed?  I'm not certain how bash behaves when it can match to multiple cases.  I'll write a test program to find out.

      case $CUDA_VERSION in
        7_*)
          MIN_UNSUPPORTED_GCC_VER="5.0"
          MIN_UNSUPPORTED_GCC_VER_NUM=50000;
          CUSOLVER=false
        ;;
        8_*)
          MIN_UNSUPPORTED_GCC_VER="6.0"
          MIN_UNSUPPORTED_GCC_VER_NUM=60000;
          CUSOLVER=false
        ;;
        9_0)
          MIN_UNSUPPORTED_GCC_VER="7.0"
          MIN_UNSUPPORTED_GCC_VER_NUM=70000;
          CUSOLVER=false
        ;;
        9_1)
          MIN_UNSUPPORTED_GCC_VER="7.0"
          MIN_UNSUPPORTED_GCC_VER_NUM=70000;
          CUSOLVER=true
        ;;
        9_2 | 9_* | 10_0)
          MIN_UNSUPPORTED_GCC_VER="8.0"
          MIN_UNSUPPORTED_GCC_VER_NUM=80000;
          CUSOLVER=true
        ;;
        10_1 | 10_*)
          MIN_UNSUPPORTED_GCC_VER="9.0"
          MIN_UNSUPPORTED_GCC_VER_NUM=90000;
          CUSOLVER=true
        ;;
        *)
          echo "Unsupported CUDA_VERSION (CUDA_VERSION=$CUDA_VERSION), please report it to Kaldi mailing list, together with 'nvcc -h' or 'ptxas -h' which lists allowed -gencode values..."; exit 1;
        ;;
      esac

Justin Luitjens

unread,
Jul 1, 2019, 10:56:49 PM7/1/19
to kaldi...@googlegroups.com
hmm nope bash only matches to a single case (sensibly).

Can you add echo statements to that part of the case statement and see which path you are taking?

Justin Luitjens

unread,
Jul 1, 2019, 11:02:48 PM7/1/19
to kaldi...@googlegroups.com
looking more closely:

Using CUDA toolkit /usr/ (nvcc compiler and runtime librariers) 

This doesn't seem right.  As far as I know CUDA doesn't install by default to /usr/

Have you manually installed it there?  

If not maybe you set --cudatk-dir wrong on the configure line.  You may want to send us your entire kaldi.mk file.


panjinsong

unread,
Jul 2, 2019, 1:38:56 AM7/2/19
to kaldi-help
Justin
Thans for your relpy.
The path of CUDA is indeed /usr/, and I manually added the the CPLUS_INCLUDE_PATH to '~/.bashrc' like this:
export CPLUS_INCLUDE_PATH=/usr/local/cuda-9.0/include/:$CPLUS_INCLUDE_PATH

在 2019年7月2日星期二 UTC+8上午11:02:48，Justin Luitjens写道：

Justin Luitjens

unread,
Jul 2, 2019, 8:24:06 AM7/2/19
to kaldi...@googlegroups.com
Not sure where that path is used.  It is not used in the make files and in Linux you should be using CPATH as the variable.

and note your second description contradicts your first.  Is the path /usr or /usr/local/cuda-9.0?

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

panjinsong

unread,
Jul 2, 2019, 10:10:33 PM7/2/19
to kaldi-help
Justin:

I maybe know where the conflict is: the 'kaldi.mk' thingks the path of 'CUDA' is '/usr', but it is actually ' /usr/local/cuda-9.0'.
how can I solve the confict by modifying the Makefile?

Thanks
pan

在 2019年7月2日星期二 UTC+8下午8:24:06，Justin Luitjens写道：

Daniel Povey

unread,
Jul 2, 2019, 10:42:28 PM7/2/19
to kaldi-help
Looks like it's a configure script issue, probably.  It looks in this rather random sequence of directories:

    for base in /Developer/NVIDIA/CUDA-6.0 /usr/local/share/cuda /usr/local/cuda /pkgs_local/cuda-3.2/ /opt/nvidia_cuda/cuda-6.0/ /usr/; do
 ...

We should probably bring some sense to this.  I'm not sure if there is a canonical way to find where CUDA lives.
Or maybe better to leave it to the user.



To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/9fa22bc2-26e4-49e5-9211-0fd880da7d9b%40googlegroups.com.

Justin Luitjens

unread,
Jul 2, 2019, 10:58:13 PM7/2/19
to kaldi...@googlegroups.com
As mentioned earlier set the parameter in the configure call.  It is something like —cuda-tkdir=/usr/local/cuda-9.0

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

panjinsong

unread,
Jul 11, 2019, 4:27:56 AM7/11/19
to kaldi-help
Thanks for your generous help.

As Dan said, I perfectly solved the problem by modifying the part code : /Developer/NVIDIA/CUDA-6.0 /usr/local/share/cuda /usr/local/cuda /pkgs_local/cuda-3.2/ /opt/nvidia_cuda/cuda-6.0/ /usr/; do ...
So it could confirm that the previous problems were caused by the cuba path recognition error.
I don't know if this is a common problem. Hope my experience can inspire people with similar problems.

Pan

在 2019年7月3日星期三 UTC+8上午10:42:28，Dan Povey写道：

Justin Luitjens

unread,
Jul 11, 2019, 4:52:04 AM7/11/19
to kaldi...@googlegroups.com
I would vote for the user to select its path as it can vary so much based on system configuration.

Sent from my iPhone
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages