Problem while training chain based TDNN model on GPUs

[Sougata Mukherjee]

I am using this script to run chain based tdnn model on my dataset. But it is showing the following error:-

Failed to allocate a memory region of 24902631424 bytes.  Possibly this is due to sharing the GPU.  Try switching the GPUs to exclusive mode (nvidia-smi -c 3) and using the option --use-gpu=wait to scripts like steps/nnet3/chain/train.py.  Memory info: free:47496M, used:1088M, total:48585M, free/total:0.97759 CUDA error: 'out of memory'

So I have switched the GPUs to the exclusive mode by using the command 'sudo nvidia-smi -c 3' and again have run the script, then it is showing a different error which is as follows:-

'Failed to create CUDA context, no more unused GPUs?'

Kindly please help me in running this script on GPUs.

[Sougata Mukherjee]

@Daniel Povey can you kindly please answer this.

[Astik Biswas]

Please checkout number of initial and final GPUs. I guess there is a mismatch between the number of GPU declared in the script and the physical gpu available.

--
Go to http://kaldi-asr.org/forums.html to find out how to join the kaldi-help group
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/1a9da130-edaa-4021-a473-755b3c21aee8n%40googlegroups.com.

[Sougata Mukherjee]

How to check the number of initial and final GPUs? I couldn't find any such thing in this script.

[Astik Biswas]

Please check line 232 and 233. It is hard coded. 

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/94585e7a-22d5-4255-b07d-2ad6a77fd81en%40googlegroups.com.

[Sougata Mukherjee]

But here it is mentioned that the results will get changed. Isn't it possible to carry out the experiment where the actual result will not get changed? 

[Astik Biswas]

In this case, you need to have 5 physical GPUs if you are using exclusive mode. However, I don't think there will be a major drop in performance if you use less GPUs. You can try

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/d222d184-cdab-4385-9b1c-7ef50cafed2en%40googlegroups.com.