[kaldi-help]Lre07 train_ivector_extractor: How to debug the failed token?

[郭震]

Hi all

I have questions while running the lre07 script locally to do the language recognition.
I followed the original code to run the lid/train_ivector_extractor.sh and the warning is:
  run.pl: Warning: ignoring options "-l mem_free=8G,ram_free=8G "
  run.pl: Warning: ignoring options "-l mem_free=8G,ram_free=8G -pe smp 16 "
exp/extractor_2048/log/acc.0.1.log produce following WARNING message
 ERROR (ivector-extractor-sum-accs:ExpectToken():io-funcs.cc:193) Failed to read token [started at file position -1], expected <IvectorExtractorStats>

My computer has 32G RAM and I have 17G train set to run, was my option wrong? 

What  is the expected <IvectorExtractorStats> ? How do I know the correct format of the token? Why rum.pl always ignore the option?

Best wishes!

Yours,
Zhen Guo

 

 

[Jan Trmal]

ad run.pl -- that is harmless. run.pl simply ignores any switches which doesn't know about. It won't fail as we want run.pl be more or less command-line compatible with queue.pl. The newer versions actually do not print the warning to avoid confusion.

Add the token -- that could probably mean that you mistakenly switched the arguments

y.

 

--
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Povey]

It could be a bug in that version of the code, but obviously you have
an out-of-date version of Kaldi. Bear in mind that Kaldi lives on
github how. See kaldi-asr.org/doc/.
Dan

[郭震]

I will try to update Kaldi, thanks for your help!

[David Snyder]

Once you get updated copy of Kaldi you'll find the most recent language ID examples in egs/lre07/v1 (instead of egs/lre/v1).

I think the error you saw was due to the older version of the ivector extractor code. I ran the language ID script recently, without any problems.

Feel free to ask me if you have any other questions about this.

[郭震]

Dear David Snyder

I updated Kaldi and ran the new version of lre07. But there are some new errors displayed in console:
>lid/train_ivector_extractor.sh --cmd run.pl -l mem_free=30G,ram_free=30G --stage 3 --use-weights true --num-iters 5
  exp/full_ubm_2048/final.ubm data/train exp/extractor_2048
>cat: exp/extractor_2048/num_jobs: No such file or directory
>lid/train_ivector_extractor.sh: line 113: [: 40: unary operator expected
>cat: exp/extractor_2048/num_jobs: No such file or directory
>Num-jobs mismatch 40 versus

Then I checked lid/train_ivector_extractor.sh and found that exp/extractor_2048 is newly made, the judgement
"if ! [ $nj_full -eq $(cat $dir/num_jobs) ]" (lid/train train_ivector_extractor.sh: line 113) will fail because extractor_2048 is empty and there is
no document called num_jobs.

According to the code of lid/train_ivector_extractor.sh, the document  "num_jobs" will be made if "stage" is less or equal than -1. So I guess the value of "stage" may be -3. I changed stage to -3 and deleted the parameter "--parallel=true"  in lid/train_ivector_extractor.sh line 133
(original code : ivector-extractor-sum-accs --parallel=true "${Args[@]:$start:$num_processes}" \) because I run lre07 on single machine.

Here are the errors produced in the log file:

LOG (ivector-extractor-acc-stats:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
Killed
ERROR (ivector-extractor-sum-accs:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>
WARNING (ivector-extractor-sum-accs:Close():kaldi-io.cc:465) Pipe ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/37/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/37/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.37.gz|' -| had nonzero return status 35072
ERROR (ivector-extractor-sum-accs:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>

[stack trace: ]
kaldi::KaldiGetStackTrace()
kaldi::KaldiErrorMessage::~KaldiErrorMessage()
kaldi::ExpectToken(std::istream&, bool, char const*)
kaldi::IvectorExtractorStats::Read(std::istream&, bool, bool)

These errors look like those encountered in the previous version. My training data is from lre15. Is there any possibility that the data preparation which is different from lre07 causes these errors? Or should I must run the project parallelly?
The attachments are the part of the log files.

Best withes,
GuoZhen

 

从网易126邮箱发来的云附件

acc.0.1.log (5.47K, 2015年10月25日 22:20 到期)

下载

acc.0.2.log (6K, 2015年10月25日 22:21 到期)

下载

acc.0.3.log (6.02K, 2015年10月25日 22:21 到期)

下载

 

[David Snyder]

It looks like the main issue is the --stage 3 argument. I must have accidentally committed that last time I was working on this example. You should just delete that argument altogether. Thanks for noticing this. I sent a merge request to fix this, but it won't clear right away.  

If this doesn't fix the issue, could you confirm that you have the posteriors in exp/extractor_2048/post*gz and a UBM in exp/full_ubm_2048 ?

Are you training on lre15 or are you going to use that for your test data? Bear in that you may need to add a few lines to local/lang_map.txt to account for any languages (or spellings of those languages) that lre15 has but our current setup does not. 

[李健]

Dear David Snyder,

I am also using the lre07 recipe to deal with lre15 data. I have changed the content of local/lang_map.txt and make it fit for lre15, but I have just met the problems like GuoZhen. Hope that you can fix these problems soon.

One more thing is that, could you please tell me whether there are some references about your lre07 recipe? I want to know more about the details but can't find related papers.

Thanks a lot for all you have done.

Best Wishes,

Jian Li

 

在2015-10-11，David Snyder <david.ry...@gmail.com> 写道:

-----原始邮件-----
发件人: David Snyder <david.ry...@gmail.com>
发送时间: 2015年10月11日 星期日
收件人: kaldi-help <kaldi...@googlegroups.com>
抄送: david.ry...@gmail.com, guo19...@126.com
主题: Re: Re: Re: [kaldi-help]Lre07 train_ivector_extractor: How to debug the failed token?

[David Snyder]

In run.sh remove the --stage 3 argument to train_ivector_extractor.sh and rerun just that script. What happens now?

I believe that the LRE07 example is typical of some of the ivector-based systems from around 2010-2012. We use multiclass logistic regression to classify ivectors to one of the languages. The ivectors are generated in a similar way as GMM-based ivectors for speaker ID, but for language ID we use shifted delta cepstra for our intput features. VTLN helps to reduce speaker variability. Try Googling for these keywords and you'll find similar systems, especially from the BUT group.

[李健]

Dear Snyder,

Thanks for your advice about how to find related references, I have tried to remove the argument --stage 3 but I still got the errors. There is the detailed information below:

The information in the console is that:
lid/train_ivector_extractor.sh --cmd run.pl -l mem_free=35G,ram_free=35G --use-weights true --num-iters 5 exp/full_ubm_2048/final.ubm data/train exp/extractor_2048
lid/train_ivector_extractor.sh: doing Gaussian selection and posterior computation
Accumulating stats (pass 0)
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.6.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.8.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.9.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.3.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.10.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.7.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.5.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.1.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.4.log
run.pl: job failed, log is in exp/extractor_2048/log/acc.0.2.log
Error accumulating stats on iteration 0

The content of acc.0.1.log is that:

# Started at Sun Oct 11 17:38:28 CST 2015
#
ivector-extractor-sum-accs --parallel=true "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/1/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/1/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.1.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/2/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/2/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.2.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/3/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/3/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.3.gz|' -|" "ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/4/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/4/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.4.gz|' -|" exp/extractor_2048/acc.0.1
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/4/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/4/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.4.gz|' -
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/1/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/1/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.1.gz|' -
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/2/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/2/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.2.gz|' -
ivector-extractor-acc-stats --num-threads=4 --num-samples-for-weights=3 exp/extractor_2048/0.ie 'ark,s,cs:apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/3/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/3/vad.scp ark:- |' 'ark,s,cs:gunzip -c exp/extractor_2048/post.3.gz|' -
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/1/feats.scp ark:-
select-voiced-frames ark:- scp,s,cs:data/train/split40/4/vad.scp ark:-
apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/3/feats.scp ark:-

add-deltas-sdc ark:- ark:-

apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/4/feats.scp ark:-

add-deltas-sdc ark:- ark:-

apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/2/feats.scp ark:-
select-voiced-frames ark:- scp,s,cs:data/train/split40/2/vad.scp ark:-
select-voiced-frames ark:- scp,s,cs:data/train/split40/3/vad.scp ark:-

add-deltas-sdc ark:- ark:-

select-voiced-frames ark:- scp,s,cs:data/train/split40/1/vad.scp ark:-

add-deltas-sdc ark:- ark:-

LOG (ivector-extractor-acc-stats:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (ivector-extractor-acc-stats:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor

LOG (ivector-extractor-acc-stats:ComputeDerivedVars():ivector-extractor.cc:204) Done.
WARNING (ivector-extractor-acc-stats:Close():kaldi-io.cc:465) Pipe gunzip -c exp/extractor_2048/post.1.gz| had nonzero return status 13
WARNING (ivector-extractor-acc-stats:Close():kaldi-io.cc:465) Pipe apply-cmvn-sliding --norm-vars=false --center=true --cmn-window=300 scp:data/train/split40/1/feats.scp ark:- | add-deltas-sdc ark:- ark:- | select-voiced-frames ark:- scp,s,cs:data/train/split40/1/vad.scp ark:- | had nonzero return status 36096
std::bad_allocERROR (ivector-extractor-sum-accs:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>

ERROR (ivector-extractor-sum-accs:ExpectToken():io-funcs.cc:197) Failed to read token [started at file position -1], expected <IvectorExtractorStats>

[stack trace: ]
kaldi::KaldiGetStackTrace()
kaldi::KaldiErrorMessage::~KaldiErrorMessage()
kaldi::ExpectToken(std::istream&, bool, char const*)
kaldi::IvectorExtractorStats::Read(std::istream&, bool, bool)

ivector-extractor-sum-accs(main+0x406) [0x473448]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f5cc19d0ec5]
ivector-extractor-sum-accs() [0x472fa9]


# Accounting: time=2619 threads=1
# Ended (code 255) at Sun Oct 11 18:22:08 CST 2015, elapsed time 2619 seconds
LOG (ivector-extractor-acc-stats:ComputeDerivedVars():ivector-extractor.cc:204) Done.

The other nine files are just like this file with slight difference. I have confirmed that I did have the posteriors in exp/extractor_2048/post*gz and a final.ubm in exp/full_ubm_2048. Is that possible that I don't have enough memory to do this job? My machine has 32GiB memories.

Thanks,

Jian Li


在2015-10-11，David Snyder <david.ry...@gmail.com> 写道:

-----原始邮件-----
发件人: David Snyder <david.ry...@gmail.com>
发送时间: 2015年10月11日 星期日
收件人: kaldi-help <kaldi...@googlegroups.com>

抄送: david.ry...@gmail.com, guo19...@126.com, li_...@tju.edu.cn
主题: Re: Re: Re: Re: [kaldi-help]Lre07 train_ivector_extractor: How to debug the failed token?

[Daniel Povey]

That script is designed to be used with a cluster -- each of those
jobs individually probably uses something like 32G of memory or
thereabouts.
You could edit run.pl to add --max-jobs-run 1 (or 2 or 3 if there is
enough memory), so they will run sequentially. Of course it will be
much slower.
Dan

[郭震]

Hi

I run lre07 locally and serially, so I deleted & in lid/train_extract_ivector.sh line 134 to prevent the script from running background. It works.

Yours,
GuoZhen

[Malo Grisard]

Hi all,

I am trying to run the lre07 v1 run.sh file. I am having the same trouble with the log error "failed to read token" for i-vector extractor training.

After reading all the posts from this blog I have tried deleting the "--parallel=true" parameter and removing the "&" line 134.

However now I get :

...

run.pl: job failed, log is in exp/extractor_512/log/acc.0.19.log

run.pl: job failed, log is in exp/extractor_512/log/acc.0.20.log

Error accumulating stats on iteration 0

the logs look like whats printed below. Hence not showing any error. 

-----------------------------------acc.0.6.log------------------------------------------------------

# ivector-extractor-sum-accs --verbose=1 exp/extractor_512/acc.0.6 

# Started at Tue Mar  6 14:10:17 UTC 2018

#

ivector-extractor-sum-accs --verbose=1 exp/extractor_512/acc.0.6 

Sum accumulators for training of iVector extractor

Usage: ivector-extractor-sum-accs [options] <stats-in1> <stats-in2> ... <stats-inN> <stats-out>

Options:

  --binary                    : Write output in binary mode (bool, default = true)

  --parallel                  : If true, the program makes sure to open all filehandles before reading for any (useful when summing accs from long processes) (bool, default = false)

Standard options:

  --config                    : Configuration file to read (this option may be repeated) (string, default = "")

  --help                      : Print out usage message (bool, default = false)

  --print-args                : Print the command line arguments (to stderr) (bool, default = true)

  --verbose                   : Verbose level (higher->more logging) (int, default = 0)

# Accounting: time=0 threads=1

# Ended (code 1) at Tue Mar  6 14:10:17 UTC 2018, elapsed time 0 seconds

---------------------------------------------------------------------------------------------------------------------

Anyone knows what could be done ?

[Daniel Povey]

What's in the log is a usage message, meaning the num-args was wrong.

I think you made a mistake when editing the script, such as removing the backslash (\) from the

end of a line, or inserting a space after it.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---

You received this message because you are subscribed to the Google Groups "kaldi-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+unsubscribe@googlegroups.com.
To post to this group, send email to kaldi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/1f3316d2-7fad-49ac-90a8-bfddf243aae5%40googlegroups.com.

[Malo Grisard]

All good thank you. I had mistakenly removed "${Args[@]:$start:$num_processes}".

What a fast answer !

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

[yuejia...@gmail.com]

i also ran into the same error.

here is the console information.

and here is the log info:

# ivector-extractor-sum-accs --verbose=1 exp/nnet3/extractor/acc.0.2
# Started at Mon Sep  3 11:07:12 CST 2018
#
ivector-extractor-sum-accs --verbose=1 exp/nnet3/extractor/acc.0.2

Sum accumulators for training of iVector extractor
Usage: ivector-extractor-sum-accs [options] <stats-in1> <stats-in2> ... <stats-inN> <stats-out>

Options:
  --binary                    : Write output in binary mode (bool, default = true)
  --parallel                  : If true, the program makes sure to open all filehandles before reading for any (useful when summing accs from long processes) (bool, default = false)

Standard options:
  --config                    : Configuration file to read (this option may be repeated) (string, default = "")
  --help                      : Print out usage message (bool, default = false)
  --print-args                : Print the command line arguments (to stderr) (bool, default = true)
  --verbose                   : Verbose level (higher->more logging) (int, default = 0)

# Accounting: time=0 threads=1

# Ended (code 1) at Mon Sep  3 11:07:12 CST 2018, elapsed time 0 seconds

and this is my code:

      $cmd --num-threads $[$num_threads*$num_processes] $dir/log/acc.$x.$g.log \
        ivector-extractor-sum-accs --verbose=1 "${Args[@]:$start:$num_processes}" \
          $dir/acc.$x.$g || touch $dir/.error

i altered the code and removed the parallel flag and & but the error is same.

Any thought will be appreciated.

在 2018年3月7日星期三 UTC+8下午4:07:30，Malo Grisard写道：

[Daniel Povey]

I suspect that you might have changed the training script somehow.  Make sure it's exactly the same as what is checked into Kaldi (you can use "git diff" on that file).

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/ace42605-d083-4ec8-9053-1bcc9b48f9e2%40googlegroups.com.

[yuejia...@gmail.com]

Thank you for fast answer,Dan.

i ran the command "git diff"

here is the result:

and i replaced the options "--verbose=1" with "--parallel=true",so this is the code i ran:

      $cmd --num-threads $[$num_threads*$num_processes] $dir/log/acc.$x.$g.log \

        ivector-extractor-sum-accs --parallel=true "${Args[@]:$start:$num_processes}" \
          $dir/acc.$x.$g || touch $dir/.error

i also checked the extra whitespace or backslash,but seems did't work.

在 2018年9月3日星期一 UTC+8上午11:19:19，Dan Povey写道：

[Daniel Povey]

You need to revert all changes you made to the script, and it will work.  If you want to use fewer jobs, use an option like --nj 2.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/06af6d32-890b-48dc-abe1-9458c4890d0e%40googlegroups.com.