batched-wav-nnet3-cuda2 | RAM Memory exceeding

86 views
Skip to first unread message

CM

unread,
Apr 16, 2021, 9:19:18 AM4/16/21
to kaldi...@googlegroups.com
Hi,

While running the following command, the system RAM gets full and the run is killed. [RAM: 16 GB, 1 GPU Card: 8GB]

What parameter do I need to tune to overcome this error? I tried changing --max-batch-size, --num-channels, --cuda-worker-thread, -frames-per-chunk. But none of this helped. I tried to lower the numbers to the bare minimum, still the 16GB system RAM is getting overwhelmed within seconds.

__________________________________
batched-wav-nnet3-cuda2 --cuda-use-tensor-cores=false --iterations=1 --acoustic-scale=1.0 --feature-type=mfcc --mfcc-config=exp/model/conf/mfcc.conf --add-pitch=false --ivector-extraction-config=exp/model/conf/ivector_extractor.conf --frame-subsampling-factor=3 --frames-per-chunk=384 --cuda-worker-threads=2 --gpu-feature-extract=false --use-online-features=true --max-batch-size=2 --num-channels=4 --beam=15 --lattice-beam=6.0 --word-symbol-table=exp/graph_combined/words.txt exp/model/final.mdl exp/graph_combined/HCLG.fst scp:test/conversational/wav.scp ark:exp/decode/lat.ark 
...
...
LOG (batched-wav-nnet3-cuda2[5.5.907~1-cbed4]:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 0 orphan nodes.
LOG (batched-wav-nnet3-cuda2[5.5.907~1-cbed4]:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 0 orphan components.
LOG (batched-wav-nnet3-cuda2[5.5.907~1-cbed4]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda2[5.5.907~1-cbed4]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda2[5.5.907~1-cbed4]:main():batched-wav-nnet3-cuda2.cc:172) Inferencing...
gpu_decoding.sh: line 20: 90584 Killed                  batched-wav-nnet3-cuda2 --cuda-use-tensor-cores=false --iterations=1 --acoustic-scale=1.0 --feature-type=mfcc --mfcc-config=exp/model/conf/mfcc.conf --add-pitch=false --ivector-extraction-config=exp/model/conf/ivector_extractor.conf
_________________________________________

Thanks,
CM

CM

unread,
Apr 18, 2021, 2:10:12 AM4/18/21
to kaldi...@googlegroups.com
Kindly help me on the below mentioned issue.

CM

unread,
Apr 18, 2021, 3:02:32 AM4/18/21
to kaldi...@googlegroups.com
Resolved by giving "--segmentation=true" and related options.

Thanks.
Reply all
Reply to author
Forward
0 new messages