GPU usage
Jon Nichols
I believe the GPU is only required for neural net training, but when is the GPU used when running Kaldi on a machine with the GPU? Specifically, when is the GPU used in the librispeech recipe? I’m up to the local/run_cleanup_segmentation.sh step in the script and I’ve seen 0% usage of the GPU so far. I know the next step, local/chain/run_tdnn.sh should use it, but thought other parts would have taken advantage of it.
When looking into this I think I might have an issue from when I compiled Kaldi. http://kaldi-asr.org/doc/cudamatrix.html says that kaldi.mk should include “nvcc” for a GPU setup, but it’s not there and cu-array-test didn’t compile. But cuda-compiled returns 0, meaning it was compiled with cuda support. So, I’m a little confused.
Daniel Povey
I believe the GPU is only required for neural net training, but when is the GPU used when running Kaldi on a machine with the GPU? Specifically, when is the GPU used in the librispeech recipe? I’m up to the local/run_cleanup_segmentation.sh step in the script and I’ve seen 0% usage of the GPU so far. I know the next step, local/chain/run_tdnn.sh should use it, but thought other parts would have taken advantage of it.
When looking into this I think I might have an issue from when I compiled Kaldi. http://kaldi-asr.org/doc/cudamatrix.html says that kaldi.mk should include “nvcc” for a GPU setup, but it’s not there and cu-array-test didn’t compile. But cuda-compiled returns 0, meaning it was compiled with cuda support. So, I’m a little confused.
--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Jon Nichols
When I run, ./configure it shows its using CUDA(see below), but the http://kaldi-asr.org/doc/cudamatrix.html page says “ If you want to tell whether Kaldi has been configured to use CUDA, you can grep for nvcc in kaldi.mk” and nvcc isn’t in kaldi.mk. Looking at ./configure I don’t see where it would add “nvcc” into kaldi.mk, is greping for nvcc maybe old/out-of-date documentation?
if that is the case then, i think i have the Kaldi compiled for cuda correctly and cu-array-test didn't compile for a different reason, i didn't notice an error when compiling but might have missed it.
Output from ./configure
./configure
Configuring ...
Backing up kaldi.mk to kaldi.mk.bak ...
Checking compiler g++ ...
Checking OpenFst library in /usr/local/src/kaldi/tools/openfst ...
Doing OS specific configurations ...
On Linux: Checking for linear algebra header files ...
Using ATLAS as the linear algebra library.
Could not find {libatlas,libsatlas}.so in any of the obvious places, will most likely try static:
Could not find libatlas.a in any of the generic-Linux places, but we'll try other stuff...
Successfully configured for red hat [dynamic libraries] with ATLASLIBS =/usr/lib64/atlas/libatlas.so.3 /usr/lib64/atlas/libf77blas.so.3 /usr/lib64/atlas/libcblas.so.3 /usr/lib64/atlas/libclapack.so.3
Using CUDA toolkit /usr/local/cuda (nvcc compiler and runtime libraries)
SUCCESS
To compile: make clean -j; make depend -j; make -j
... or e.g. -j 10, instead of -j, to use a specified number of CPUs
My kaldi.mk file contains
...
# CUDA configuration
CUDA = true
CUDATKDIR = /usr/local/cuda
CUDA_ARCH = -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_53,code=sm_53
ifndef DOUBLE_PRECISION
$(error DOUBLE_PRECISION not defined.)
endif
ifndef CUDATKDIR
$(error CUDATKDIR not defined.)
endif
CUDA_INCLUDE= -I$(CUDATKDIR)/include
CUDA_FLAGS = -g -Xcompiler -fPIC --verbose --machine 64 -DHAVE_CUDA \
-DKALDI_DOUBLEPRECISION=$(DOUBLE_PRECISION)
CXXFLAGS += -DHAVE_CUDA -I$(CUDATKDIR)/include
CUDA_LDFLAGS += -L$(CUDATKDIR)/lib64 -Wl,-rpath,$(CUDATKDIR)/lib64
CUDA_LDLIBS += -lcublas -lcudart -lcurand #LDLIBS : The libs are loaded later than static libs in implicit rule
...
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.