Is this an error? bash:line1: 4594 killed

1,432 views
Skip to first unread message

mirfan.ms...@seecs.edu.pk

unread,
Feb 13, 2017, 10:21:02 AM2/13/17
to kaldi-help
I'm running librispeach on gridengine and the output file is showing this the following. Is this an error? 

decode.sh: feature type is delta
bash: line 1:  4594 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/13/utt2spk scp:data/test_clean/split20/13/cmvn.scp scp:data/test_clean/split20/13/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.13.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.13.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.13.log
bash: line 1:  4601 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/19/utt2spk scp:data/test_clean/split20/19/cmvn.scp scp:data/test_clean/split20/19/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.19.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.19.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.19.log
bash: line 1:  4607 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/8/utt2spk scp:data/test_clean/split20/8/cmvn.scp scp:data/test_clean/split20/8/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.8.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.8.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.8.log
bash: line 1:  4623 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/12/utt2spk scp:data/test_clean/split20/12/cmvn.scp scp:data/test_clean/split20/12/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.12.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.12.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.12.logEnter code here...

decode.sh: feature type is delta
bash: line 1:  4594 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/13/utt2spk scp:data/test_clean/split20/13/cmvn.scp scp:data/test_clean/split20/13/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.13.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.13.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.13.log
bash: line 1:  4601 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/19/utt2spk scp:data/test_clean/split20/19/cmvn.scp scp:data/test_clean/split20/19/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.19.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.19.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.19.log
bash: line 1:  4607 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/8/utt2spk scp:data/test_clean/split20/8/cmvn.scp scp:data/test_clean/split20/8/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.8.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.8.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.8.log
bash: line 1:  4623 Killed                  ( gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/mono/graph_nosp_tgsmall/words.txt exp/mono/final.mdl exp/mono/graph_nosp_tgsmall/HCLG.fst "ark,s,cs:apply-cmvn  --utt2spk=ark:data/test_clean/split20/12/utt2spk scp:data/test_clean/split20/12/cmvn.scp scp:data/test_clean/split20/12/feats.scp ark:- | add-deltas  ark:- ark:- |" "ark:|gzip -c > exp/mono/decode_nosp_tgsmall_test_clean/lat.12.gz" ) 2>> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.12.log >> exp/mono/decode_nosp_tgsmall_test_clean/log/decode.12.log

Daniel Povey

unread,
Feb 13, 2017, 2:17:04 PM2/13/17
to kaldi-help
They are being killed by the Linux OOM killer because it's using too much memory for your machine.
It looks to me like, while you may have installed GridEngine, you are not using it because your $cmd (or $decode_cmd, etc.) variables are set to run.pl not queue.pl.  Read http://kaldi-asr.org/doc/queue.html to understand how Kaldi interacts with GridEngine and similar tools.

Dan


--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mirfan.ms...@seecs.edu.pk

unread,
Feb 13, 2017, 3:05:16 PM2/13/17
to kaldi-help
I'm currently using run.pl because compute nodes on a cluster are not allowed to run the gridentine job. Is there a way to run the job with available 24 GB RAM?

Daniel Povey

unread,
Feb 13, 2017, 3:24:57 PM2/13/17
to kaldi-help
You will have to either reduce the number of decoding jobs (--nj), or use e.g. "run.pl --max-jobs-run 10" as the $cmd variable.  Also if things are backgrounded (&) in run.sh, run them in sequence.

Reply all
Reply to author
Forward
0 new messages