Re: [kaldi-help] CUDA error: no kernel image is available for execution on the device

[Daniel Povey]

You shouldn't have had to edit configure. you may have made a change to the configure script that broke things.

I think the "no kernel image is available for execution on the device" message is about the compute capability being unsupported.

The compute capability of A100 is 8.0.  However, it should be supported if you are using toolkit 11.- or above, see the compute_80,sm_80 options below in the

configure script.  The configure script writes to kaldi.mk, and options like sm_80 should show up in that file if that compute  capability is supported.

The configure script should also print what it is doing.

After compiling Kaldi the CUDA toolkit should not matter, apart from nvidia-smi.  Any 11_* or 12_* toolkit should work in principle.

```

           10_*) CUDA_ARCH="-gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 

-gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75" ;;

            11_0) CUDA_ARCH="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80" ;;

            11_*) CUDA_ARCH="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" ;;

            12_*) CUDA_ARCH="-gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_90,code=sm_90" ;;

            *) failure \

```

On Sat, Nov 18, 2023 at 2:13 PM Shreya Singhal <shreyas...@gmail.com> wrote:

Stuck with cuda compilation

This is not a new problem and I've searched and brainstormed for potential problems online.

While trying to train a tdnn, I am being plagued with "CUDA error: no kernel image is available for execution on the device"

I am trying to use kaldi on a Nvidia A100 GPU, previously I had cuda 11.7 soft linked to /usr/local/cuda and nvidia-smi showed driver 470, and cuda version 11.4,

So I deleted the soft link and relinked /usr/local/cuda to cuda-11.3 installation.


Then in the src dir I :

1.) Edit configure to set use-cuda true

2.) ./configure --cudatk-dir /usr/local/cuda

3.) make clean

4.) make depend -j 24

5.) make -j 24


After installation in src/nnet3bin/cuda-gpu-available shows this:
(base) azureuser@trainingServer:~/kaldi/src/nnet3bin$ ./cuda-gpu-available
LOG ([5.5.1124~1-21ae4]:main():cuda-gpu-available.cc:61)

### IS CUDA GPU AVAILABLE? 'trainingServer' ###
### CUDA WAS NOT COMPILED IN! ###
To support CUDA, you must run 'configure' on a machine that has the CUDA compiler 'nvcc' available.
(base) azureuser@trainingServer:~/kaldi/src/nnet3bin$


I am able to see cuda from pytorch but have run out of ideas how to fix this. Previously I had no issues with cuda 10.1.

Should I update my nvidia driver or any other things I can try out next?

--
Go to http://kaldi-asr.org/forums.html to find out how to join the kaldi-help group
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/1127f65b-82e7-4c97-a1a2-2f60172b0f60n%40googlegroups.com.