--online-ivector-dir exp/nnet3/ivectors_train_clean_100_sp_sp_sp
CUDA not compiled. Proceeding with CPU mode (slower training)...
local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
fix_data_dir.sh: kept all 428085 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: making sure the utt2dur and the reco2dur files are present
... in data/train_clean_100_sp_sp_sp, because obtaining it after speed-perturbing
... would be very slow, and you might need them.
utils/data/get_utt2dur.sh: data/train_clean_100_sp_sp_sp/utt2dur already exists with the expected length. We won't recompute it.
utils/data/get_reco2dur.sh: data/train_clean_100_sp_sp_sp/reco2dur already exists with the expected length. We won't recompute it.
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean_100_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_speed0.9
fix_data_dir.sh: kept all 428085 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_speed0.9/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_speed0.9
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean_100_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_speed1.1
fix_data_dir.sh: kept all 428085 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_speed1.1/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_speed1.1
utils/data/combine_data.sh data/train_clean_100_sp_sp_sp_sp data/train_clean_100_sp_sp_sp data/train_clean_100_sp_sp_sp_sp_speed0.9 data/train_clean_100_sp_sp_sp_sp_speed1.1
utils/data/combine_data.sh: combined utt2uniq
utils/data/combine_data.sh [info]: not combining segments as it does not exist
utils/data/combine_data.sh: combined utt2spk
utils/data/combine_data.sh [info]: not combining utt2lang as it does not exist
utils/data/combine_data.sh: combined utt2dur
utils/data/combine_data.sh [info]: **not combining utt2num_frames as it does not exist everywhere**
utils/data/combine_data.sh: combined reco2dur
utils/data/combine_data.sh [info]: **not combining feats.scp as it does not exist everywhere**
utils/data/combine_data.sh: combined text
utils/data/combine_data.sh [info]: **not combining cmvn.scp as it does not exist everywhere**
utils/data/combine_data.sh [info]: not combining vad.scp as it does not exist
utils/data/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist
utils/data/combine_data.sh: combined wav.scp
utils/data/combine_data.sh [info]: not combining spk2gender as it does not exist
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp/text is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp/wav.scp is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp/utt2uniq is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp/utt2dur is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp/reco2dur is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: generated 3-way speed-perturbed version of data in data/train_clean_100_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp
local/nnet3/run_ivector_common.sh: making MFCC features for low-resolution speed-perturbed data
steps/make_mfcc.sh --cmd
run.pl --nj 50 data/train_clean_100_sp_sp_sp_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp
steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
steps/make_mfcc.sh: Succeeded creating MFCC features for train_clean_100_sp_sp_sp_sp
steps/compute_cmvn_stats.sh data/train_clean_100_sp_sp_sp_sp
Succeeded creating CMVN stats for train_clean_100_sp_sp_sp_sp
local/nnet3/run_ivector_common.sh: fixing input data-dir to remove nonexistent features, in case some
.. speed-perturbed segments were too short.
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp/.backup
local/nnet3/run_ivector_common.sh: aligning with the perturbed low-resolution data
steps/align_fmllr.sh --nj 100 --cmd
run.pl data/train_clean_100_sp_sp_sp_sp data/lang exp/tri3 exp/tri3_ali_train_clean_100_sp_sp_sp_sp
steps/align_fmllr.sh: feature type is lda
steps/align_fmllr.sh: compiling training graphs
steps/align_fmllr.sh: aligning data in data/train_clean_100_sp_sp_sp_sp using exp/tri3/final.mdl and speaker-independent features.
steps/align_fmllr.sh: computing fMLLR transforms
steps/align_fmllr.sh: doing final alignment.
steps/align_fmllr.sh: done aligning data.
steps/diagnostic/analyze_alignments.sh --cmd
run.pl data/lang exp/tri3_ali_train_clean_100_sp_sp_sp_sp
steps/diagnostic/analyze_alignments.sh: see stats in exp/tri3_ali_train_clean_100_sp_sp_sp_sp/log/analyze_alignments.log
22386 warnings in exp/tri3_ali_train_clean_100_sp_sp_sp_sp/log/align_pass2.*.log
2314 warnings in exp/tri3_ali_train_clean_100_sp_sp_sp_sp/log/fmllr.*.log
66419 warnings in exp/tri3_ali_train_clean_100_sp_sp_sp_sp/log/align_pass1.*.log
local/nnet3/run_ivector_common.sh: creating high-resolution MFCC features
utils/copy_data_dir.sh: copied data from data/train_clean_100_sp_sp_sp_sp to data/train_clean_100_sp_sp_sp_sp_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_hires
utils/copy_data_dir.sh: copied data from data/test_clean to data/test_clean_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/test_clean_hires
copy_data_dir.sh: no such file data/test_other/utt2spk
(kaldi_py38) shubham@csis-ML10Gen9:~/kaldi/egs/librispeech/s5$
(kaldi_py38) shubham@csis-ML10Gen9:~/kaldi/egs/librispeech/s5$ local/nnet3/run_tdnn.sh \
--stage 0 \
--train-set train_clean_100_sp_sp_sp_sp \
--gmm tri3 \
--nnet3-affix "" \
--online-ivector-dir exp/nnet3/ivectors_train_clean_100_sp_sp_sp_sp
CUDA not compiled. Proceeding with CPU mode (slower training)...
local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: making sure the utt2dur and the reco2dur files are present
... in data/train_clean_100_sp_sp_sp_sp, because obtaining it after speed-perturbing
... would be very slow, and you might need them.
utils/data/get_utt2dur.sh: data/train_clean_100_sp_sp_sp_sp/utt2dur already exists with the expected length. We won't recompute it.
utils/data/get_reco2dur.sh: data/train_clean_100_sp_sp_sp_sp/reco2dur already exists with the expected length. We won't recompute it.
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean_100_sp_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_sp_speed0.9
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_sp_speed0.9/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp_speed0.9
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean_100_sp_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_sp_speed1.1
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_sp_speed1.1/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp_speed1.1
utils/data/combine_data.sh data/train_clean_100_sp_sp_sp_sp_sp data/train_clean_100_sp_sp_sp_sp data/train_clean_100_sp_sp_sp_sp_sp_speed0.9 data/train_clean_100_sp_sp_sp_sp_sp_speed1.1
utils/data/combine_data.sh: combined utt2uniq
utils/data/combine_data.sh [info]: not combining segments as it does not exist
utils/data/combine_data.sh: combined utt2spk
utils/data/combine_data.sh [info]: not combining utt2lang as it does not exist
utils/data/combine_data.sh: combined utt2dur
utils/data/combine_data.sh [info]: **not combining utt2num_frames as it does not exist everywhere**
utils/data/combine_data.sh: combined reco2dur
utils/data/combine_data.sh [info]: **not combining feats.scp as it does not exist everywhere**
utils/data/combine_data.sh: combined text
utils/data/combine_data.sh [info]: **not combining cmvn.scp as it does not exist everywhere**
utils/data/combine_data.sh [info]: not combining vad.scp as it does not exist
utils/data/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist
utils/data/combine_data.sh: combined wav.scp
utils/data/combine_data.sh [info]: not combining spk2gender as it does not exist
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/text is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/wav.scp is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/utt2uniq is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/utt2dur is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/reco2dur is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 1797957 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: generated 3-way speed-perturbed version of data in data/train_clean_100_sp_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp
local/nnet3/run_ivector_common.sh: making MFCC features for low-resolution speed-perturbed data
steps/make_mfcc.sh --cmd
run.pl --nj 50 data/train_clean_100_sp_sp_sp_sp_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp
steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
steps/make_mfcc.sh: Succeeded creating MFCC features for train_clean_100_sp_sp_sp_sp_sp
steps/compute_cmvn_stats.sh data/train_clean_100_sp_sp_sp_sp_sp
Succeeded creating CMVN stats for train_clean_100_sp_sp_sp_sp_sp
local/nnet3/run_ivector_common.sh: fixing input data-dir to remove nonexistent features, in case some
.. speed-perturbed segments were too short.
fix_data_dir.sh: kept all 1797957 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_sp/.backup
local/nnet3/run_ivector_common.sh: aligning with the perturbed low-resolution data
steps/align_fmllr.sh --nj 100 --cmd
run.pl data/train_clean_100_sp_sp_sp_sp_sp data/lang exp/tri3 exp/tri3_ali_train_clean_100_sp_sp_sp_sp_sp
steps/align_fmllr.sh: feature type is lda
steps/align_fmllr.sh: compiling training graphs
run.pl: 40 / 100 failed, log is in exp/tri3_ali_train_clean_100_sp_sp_sp_sp_sp/log/compile_graphs.*.log
(kaldi_py38) shubham@csis-ML10Gen9:~/kaldi/egs/librispeech/s5$ rm -rf data/train_clean_100_sp_sp_sp_sp_sp*
rm -rf exp/tri3_ali_train_clean_100_sp_sp_sp_sp_sp
(kaldi_py38) shubham@csis-ML10Gen9:~/kaldi/egs/librispeech/s5$ [ -f $train_data_dir/feats.scp ]
[ -f $train_ivector_dir/ivector_online.scp ]
[ -f $ali_dir/ali.1.gz ]
(kaldi_py38) shubham@csis-ML10Gen9:~/kaldi/egs/librispeech/s5$ du -sh data/ exp/
130G data/
68G exp/
(kaldi_py38) shubham@csis-ML10Gen9:~/kaldi/egs/librispeech/s5$ local/nnet3/run_tdnn.sh \
--stage 0 \
--train-set train_clean_100_sp_sp_sp_sp \
--gmm tri3 \
--nnet3-affix "" \
--online-ivector-dir exp/nnet3/ivectors_train_clean_100_sp_sp_sp_sp
CUDA not compiled. Proceeding with CPU mode (slower training)...
local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: making sure the utt2dur and the reco2dur files are present
... in data/train_clean_100_sp_sp_sp_sp, because obtaining it after speed-perturbing
... would be very slow, and you might need them.
utils/data/get_utt2dur.sh: data/train_clean_100_sp_sp_sp_sp/utt2dur already exists with the expected length. We won't recompute it.
utils/data/get_reco2dur.sh: data/train_clean_100_sp_sp_sp_sp/reco2dur already exists with the expected length. We won't recompute it.
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean_100_sp_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_sp_speed0.9
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_sp_speed0.9/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp_speed0.9
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train_clean_100_sp_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_sp_speed1.1
fix_data_dir.sh: kept all 884709 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_sp_speed1.1/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp_speed1.1
utils/data/combine_data.sh data/train_clean_100_sp_sp_sp_sp_sp data/train_clean_100_sp_sp_sp_sp data/train_clean_100_sp_sp_sp_sp_sp_speed0.9 data/train_clean_100_sp_sp_sp_sp_sp_speed1.1
utils/data/combine_data.sh: combined utt2uniq
utils/data/combine_data.sh [info]: not combining segments as it does not exist
utils/data/combine_data.sh: combined utt2spk
utils/data/combine_data.sh [info]: not combining utt2lang as it does not exist
utils/data/combine_data.sh: combined utt2dur
utils/data/combine_data.sh [info]: **not combining utt2num_frames as it does not exist everywhere**
utils/data/combine_data.sh: combined reco2dur
utils/data/combine_data.sh [info]: **not combining feats.scp as it does not exist everywhere**
utils/data/combine_data.sh: combined text
utils/data/combine_data.sh [info]: **not combining cmvn.scp as it does not exist everywhere**
utils/data/combine_data.sh [info]: not combining vad.scp as it does not exist
utils/data/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist
utils/data/combine_data.sh: combined wav.scp
utils/data/combine_data.sh [info]: not combining spk2gender as it does not exist
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/text is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/wav.scp is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/utt2uniq is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/utt2dur is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/train_clean_100_sp_sp_sp_sp_sp/reco2dur is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 1797957 utterances.
fix_data_dir.sh: old files are kept in data/train_clean_100_sp_sp_sp_sp_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: generated 3-way speed-perturbed version of data in data/train_clean_100_sp_sp_sp_sp, in data/train_clean_100_sp_sp_sp_sp_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp
local/nnet3/run_ivector_common.sh: making MFCC features for low-resolution speed-perturbed data
steps/make_mfcc.sh --cmd
run.pl --nj 50 data/train_clean_100_sp_sp_sp_sp_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_clean_100_sp_sp_sp_sp_sp
steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
i am trying to use TDNN but again and again it is failing I don't know why so