Allocating training subset examples take too much time

[Brij Mohan Lal Srivastava]

Hello,

I am trying to train a x-vector model using custom 256-dim features. The command to create egs for training looks like this:

sid/nnet3/xvector/get_egs.sh --cmd "$train_cmd" \

  --nj 8 \

  --stage 0 \

  --frames-per-iter 50000000 \

  --frames-per-iter-diagnostic 100000 \

  --min-frames-per-chunk 200 \

  --max-frames-per-chunk 400 \

  --num-diagnostic-archives 3 \

  --num-repeats 35 \

  "$data" $egs_dir

The command proceeds to produce the following log on console:

sid/nnet3/xvector/get_egs.sh: Preparing train and validation lists

sid/nnet3/xvector/get_egs.sh: Producing 92 archives for training

sid/nnet3/xvector/get_egs.sh: Allocating training examples

sid/nnet3/xvector/get_egs.sh: Allocating training subset examples

And then its stuck there for more than 20h. Even the log file says nothing. Basically it is stuck at this line: 

https://github.com/kaldi-asr/kaldi/blob/master/egs/sre08/v1/sid/nnet3/xvector/get_egs.sh#L161

Can anybody please let me know what could be the issue and how to debug this?

Thanks,

Brij

[Brij Mohan Lal Srivastava]

The program is going into an infinite loop here:

https://github.com/kaldi-asr/kaldi/blob/master/egs/sre08/v1/sid/nnet3/xvector/allocate_egs.py#L261

The utt_len is always smaller than the length in the while loop specified in the above link.

This is the command that is executed:

$ sid/nnet3/xvector/allocate_egs.py --prefix train_subset --num-repeats=1 --min-frame

s-per-chunk=200 --max-frames-per-chunk=400 --randomize-chunk-length false --frames-per-iter=100000 --num-archives=3 --num-jobs=1 --utt2len-filename=exp/asv_models/asv_n

oise_baseline/egs/temp/utt2num_frames.train_subset --utt2int-filename=exp/asv_models/asv_noise_baseline/egs/temp/utt2int.train_subset --egs-dir=exp/asv_models/asv_noise

_baseline/egs

Any ideas why this is happening?

Thanks,

Brij

[Desh Raj]

I think we may need to revert this PR: https://github.com/kaldi-asr/kaldi/pull/4320. It was originally intended to fix https://github.com/kaldi-asr/kaldi/issues/4319 but it seems it often leads to infinite looping, as Dan had predicted.

- Desh

--
Go to http://kaldi-asr.org/forums.html to find out how to join the kaldi-help group
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/e8b8201a-0cbf-45e3-8db1-c390fee63e6dn%40googlegroups.com.

[Brij Mohan Lal Srivastava]

Thanks for pointing out the exact issue, Desh! I will make changes locally to proceed with my experiment for now.

One question: I see that there is no change in the body of get_random_utt function based on min_length param. How was it used in the function?

Thanks,

Brij

[Desh Raj]

It wasn't being used at all.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/ea482ff2-7630-4234-931d-a65f7f831591n%40googlegroups.com.

[Daniel Povey]

Mm, I'd rather merge a fix that actually fixes the original issue as well as the looping, but I don't have time to create one myself.

Shouldn't be too hard though.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAHRYyT5vBbUVK7gt_aT_CJQ0BQ%3Db%3DR0noWK%2BDWBU1qO4PCrXCA%40mail.gmail.com.

[Brij Mohan Lal Srivastava]

Dear Desh and Dan,

I made local changes to allocate_egs.sh to revert to the old code but I face the same issue as this: https://github.com/kaldi-asr/kaldi/issues/4319

So I guess I need to find a better solution to continue the training.

Can you give me some idea what has to be done here so that I can implement it and bypass this issue?

Thanks,

Brij

[Daniel Povey]

It should be a question of thoroughly reading that script and figuring out its logic, and implementing a fix.  I don't have time to figure it out right now though.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/48498aef-357c-41e7-9df2-03eb2bd0aa71n%40googlegroups.com.

[Brij Mohan Lal Srivastava]

Added a fix to break out of infinite loop when the utterances are exhausted.

https://github.com/brijmohan/kaldi/commit/62b8ed90e261a6bb5088dfe506b7972dd052743f

Thanks,

Brij