failed steps/nnet3/chain/get_egs.sh

[keiya]

I try librispeech recipe.

When I run Generating training examples on disk, it fails.

What should I do?

Error is as follows.

steps/nnet3/chain/get_egs.sh: Generating training examples on disk

run.pl: 99 / 100 failed, log is in exp/chain_cleaned/tdnn_1d_sp/egs/log/get_egs.*.log

Traceback (most recent call last):

  File "steps/nnet3/chain/train.py", line 637, in main

    train(args, run_opts)

  File "steps/nnet3/chain/train.py", line 396, in train

    stage=args.egs_stage

  File "steps/libs/nnet3/train/chain_objf/acoustic_model.py", line 118, in generate_chain_egs

    egs_opts=egs_opts if egs_opts is not None else ''))

  File "steps/libs/common.py", line 158, in execute_command

    p.returncode, command))

Exception: Command exited with status 1: steps/nnet3/chain/get_egs.sh --frames-overlap-per-eg 0 --constrained false                 --cmd "run.pl"                 --cmvn-opts "--norm-means=false --norm-vars=false"                 --online-ivector-dir "exp/nnet3_cleaned/ivectors_train_clean_100_sp_hires"                 --left-context 41                 --right-context 41                 --left-context-initial -1                 --right-context-final -1                 --left-tolerance '5'                 --right-tolerance '5'                 --frame-subsampling-factor 3                 --alignment-subsampling-factor 3                 --stage -10                 --frames-per-iter 2500000                 --frames-per-eg 150,110,100                 --srand 0                 /nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires exp/chain_cleaned/tdnn_1d_sp exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats exp/chain_cleaned/tdnn_1d_sp/egs

steps/nnet3/chain/train.py --stage -10 --cmd run.pl --feat.online-ivector-dir exp/nnet3_cleaned/ivectors_train_clean_100_sp_hires --feat.cmvn-opts --norm-means=false --norm-vars=false --chain.xent-regularize 0.1 --chain.leaky-hmm-coefficient 0.1 --chain.l2-regularize 0.0 --chain.apply-deriv-weights false --chain.lm-opts=--num-extra-lm-states=2000 --egs.dir  --egs.stage -10 --egs.opts --frames-overlap-per-eg 0 --constrained false --egs.chunk-width 150,110,100 --trainer.dropout-schedule 0,0...@0.20,0...@0.50,0 --trainer.add-option=--optimization.memory-compression-level=2 --trainer.num-chunk-per-minibatch 64 --trainer.frames-per-iter 2500000 --trainer.num-epochs 4 --trainer.optimization.num-jobs-initial 3 --trainer.optimization.num-jobs-final 16 --trainer.optimization.initial-effective-lrate 0.00015 --trainer.optimization.final-effective-lrate 0.000015 --trainer.max-param-change 2.0 --cleanup.remove-egs true --feat-dir /nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires --tree-dir exp/chain_cleaned/tree_sp --lat-dir exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats --dir exp/chain_cleaned/tdnn_1d_sp

['steps/nnet3/chain/train.py', '--stage', '-10', '--cmd', 'run.pl', '--feat.online-ivector-dir', 'exp/nnet3_cleaned/ivectors_train_clean_100_sp_hires', '--feat.cmvn-opts', '--norm-means=false --norm-vars=false', '--chain.xent-regularize', '0.1', '--chain.leaky-hmm-coefficient', '0.1', '--chain.l2-regularize', '0.0', '--chain.apply-deriv-weights', 'false', '--chain.lm-opts=--num-extra-lm-states=2000', '--egs.dir', '', '--egs.stage', '-10', '--egs.opts', '--frames-overlap-per-eg 0 --constrained false', '--egs.chunk-width', '150,110,100', '--trainer.dropout-schedule', '0,0...@0.20,0...@0.50,0', '--trainer.add-option=--optimization.memory-compression-level=2', '--trainer.num-chunk-per-minibatch', '64', '--trainer.frames-per-iter', '2500000', '--trainer.num-epochs', '4', '--trainer.optimization.num-jobs-initial', '3', '--trainer.optimization.num-jobs-final', '16', '--trainer.optimization.initial-effective-lrate', '0.00015', '--trainer.optimization.final-effective-lrate', '0.000015', '--trainer.max-param-change', '2.0', '--cleanup.remove-egs', 'true', '--feat-dir', '/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires', '--tree-dir', 'exp/chain_cleaned/tree_sp', '--lat-dir', 'exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats', '--dir', 'exp/chain_cleaned/tdnn_1d_sp']

steps/nnet3/chain/get_egs.sh: Getting subsets of validation examples for diagnostics and combination.

exp/chain_cleaned/tdnn_1d_sp/egs/log/get_egs.1.log:

# lattice-align-phones --replace-output-symbols=true exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/final.mdl "ark:gunzip -c exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/lat.1.gz |" ark:- | chain-get-supervision --lattice-input=true --frame-subsampling-factor=3 --right-tolerance=5 --left-tolerance=5 --convert-to-pdfs=false exp/chain_cleaned/tdnn_1d_sp/tree exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl ark:- ark:- | nnet3-chain-get-egs --online-ivectors=scp:exp/nnet3_cleaned/ivectors_train_clean_100_sp_hires/ivector_online.scp --online-ivector-period=10 --srand=$[1+0] --left-context=41 --right-context=41 --num-frames=150,110,100 --frame-subsampling-factor=3 --compress=true --num-frames-overlap=0 --transition-model=exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl "ark,s,cs:utils/filter_scp.pl --exclude exp/chain_cleaned/tdnn_1d_sp/egs/valid_uttlist /nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/feats.scp | apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/utt2spk scp:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/cmvn.scp scp:- ark:- |" ark,s,cs:- ark:- | nnet3-chain-copy-egs --random=true --srand=$[1+0] ark:- ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.1.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.2.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.3.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.4.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.5.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.6.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.7.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.8.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.9.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.10.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.11.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.12.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.13.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.14.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.15.ark

# Started at Wed Aug 28 20:26:28 JST 2019

#

lattice-align-phones --replace-output-symbols=true exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/final.mdl 'ark:gunzip -c exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/lat.1.gz |' ark:-

nnet3-chain-copy-egs --random=true --srand=1 ark:- ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.1.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.2.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.3.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.4.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.5.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.6.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.7.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.8.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.9.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.10.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.11.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.12.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.13.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.14.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.1.15.ark

nnet3-chain-get-egs --online-ivectors=scp:exp/nnet3_cleaned/ivectors_train_clean_100_sp_hires/ivector_online.scp --online-ivector-period=10 --srand=1 --left-context=41 --right-context=41 --num-frames=150,110,100 --frame-subsampling-factor=3 --compress=true --num-frames-overlap=0 --transition-model=exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl 'ark,s,cs:utils/filter_scp.pl --exclude exp/chain_cleaned/tdnn_1d_sp/egs/valid_uttlist /nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/feats.scp | apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/utt2spk scp:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/cmvn.scp scp:- ark:- |' ark,s,cs:- ark:-

LOG (nnet3-chain-get-egs[5.5.356~1-8706]:ComputeDerived():nnet-example-utils.cc:335) Rounding up --num-frames=150,110,100 to multiples of --frame-subsampling-factor=3, to: 150,111,102

chain-get-supervision --lattice-input=true --frame-subsampling-factor=3 --right-tolerance=5 --left-tolerance=5 --convert-to-pdfs=false exp/chain_cleaned/tdnn_1d_sp/tree exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl ark:- ark:-

apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/utt2spk scp:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/1/cmvn.scp scp:- ark:-

LOG (apply-cmvn[5.5.356~1-8706]:main():apply-cmvn.cc:81) Copied 277 utterances.

LOG (nnet3-chain-get-egs[5.5.356~1-8706]:~UtteranceSplitter():nnet-example-utils.cc:357) Split 277 utts, with total length 334071 frames (0.927975 hours assuming 100 frames per second)

LOG (nnet3-chain-get-egs[5.5.356~1-8706]:~UtteranceSplitter():nnet-example-utils.cc:366) Average chunk length was 144.227 frames; overlap between adjacent chunks was 0.609751% of input length; length of output was 100.462% of input length (minus overlap = 99.8527%).

LOG (nnet3-chain-get-egs[5.5.356~1-8706]:~UtteranceSplitter():nnet-example-utils.cc:382) Output frames are distributed among chunk-sizes as follows: 102 = 6.382%, 111 = 2.844%, 150 = 90.77%

ERROR: CompactFst write failed: <unknown>

ERROR (chain-get-supervision[5.5.356~1-8706]:WriteToken():io-funcs.cc:141) Write failure in WriteToken.

[ Stack-Trace: ]

kaldi::MessageLogger::LogMessage() const

kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)

kaldi::WriteToken(std::ostream&, bool, char const*)

kaldi::chain::Supervision::Write(std::ostream&, bool) const

kaldi::KaldiObjectHolder<kaldi::chain::Supervision>::Write(std::ostream&, bool, kaldi::chain::Supervision const&)

kaldi::TableWriterArchiveImpl<kaldi::KaldiObjectHolder<kaldi::chain::Supervision> >::Write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kaldi::chain::Supervision const&)

chain-get-supervision() [0x546df1]

main

__libc_start_main

_start

WARNING (chain-get-supervision[5.5.356~1-8706]:Write():util/kaldi-holder-inl.h:57) Exception caught writing Table object. kaldi::KaldiFatalError

WARNING (chain-get-supervision[5.5.356~1-8706]:Write():util/kaldi-table-inl.h:1057) Write failure to standard output

ERROR (chain-get-supervision[5.5.356~1-8706]:Write():util/kaldi-table-inl.h:1515) Error in TableWriter::Write

[ Stack-Trace: ]

kaldi::MessageLogger::LogMessage() const

kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)

chain-get-supervision() [0x546e43]

main

__libc_start_main

_start

WARNING (chain-get-supervision[5.5.356~1-8706]:Close():util/kaldi-table-inl.h:1089) Error closing stream: wspecifier is ark:-

ERROR (chain-get-supervision[5.5.356~1-8706]:~TableWriter():util/kaldi-table-inl.h:1539) Error closing TableWriter [in destructor].

[ Stack-Trace: ]

kaldi::MessageLogger::LogMessage() const

kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)

kaldi::TableWriter<kaldi::KaldiObjectHolder<kaldi::chain::Supervision> >::~TableWriter()

main

__libc_start_main

_start

terminate called after throwing an instance of 'kaldi::KaldiFatalError'

  what():  kaldi::KaldiFatalError

ERROR: VectorFst::Write: Write failed: <unspecifed>

WARNING (lattice-align-phones[5.5.356~1-8706]:Write():util/kaldi-table-inl.h:1057) Write failure to standard output

ERROR (lattice-align-phones[5.5.356~1-8706]:Write():util/kaldi-table-inl.h:1515) Error in TableWriter::Write

[ Stack-Trace: ]

kaldi::MessageLogger::LogMessage() const

kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)

main

__libc_start_main

_start

WARNING (lattice-align-phones[5.5.356~1-8706]:Close():util/kaldi-table-inl.h:1089) Error closing stream: wspecifier is ark:-

ERROR (lattice-align-phones[5.5.356~1-8706]:~TableWriter():util/kaldi-table-inl.h:1539) Error closing TableWriter [in destructor].

[ Stack-Trace: ]

kaldi::MessageLogger::LogMessage() const

kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)

kaldi::TableWriter<kaldi::CompactLatticeHolder>::~TableWriter()

main

__libc_start_main

_start

terminate called after throwing an instance of 'kaldi::KaldiFatalError'

  what():  kaldi::KaldiFatalError

gzip: stdout: Broken pipe

LOG (nnet3-chain-copy-egs[5.5.356~1-8706]:main():nnet3-chain-copy-egs.cc:395) Read 2327 neural-network training examples, wrote 2327

# Accounting: time=11 threads=1

# Ended (code 0) at Wed Aug 28 20:26:39 JST 2019, elapsed time 11 seconds

[Cemil Demir]

you can check left disk space size. Moreover, you can check the number of open files limit in your system.

Regards.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/67db850a-cb7b-417a-944d-54aa18698e22%40googlegroups.com.

[keiya]

Thank you Cemil.

disk space :270GB

$cat /proc/sys/fs/file-max

13171618

I use trian-clean-100 only, but is that much capacity required?

2019年8月28日水曜日 21時08分00秒 UTC+9 Cemil Demir:

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.

[Daniel Povey]

You can check if it's full with

df

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/5ba01233-dbcb-4515-bc92-c7dca58816fe%40googlegroups.com.

[keiya]

by df
270GB

[Daniel Povey]

I suspect you might be misinterpreting the output of `df`.  But if not, it could be a memory-full problem, with get_egs.sh using up too much memory.  To train.py's --egs.opts option, you could add the extra option --max-jobs-run 4 or something like that.

On Wed, Aug 28, 2019 at 11:30 AM keiya <keiy...@gmail.com> wrote:

by df
270GB

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/d97509a9-3599-4e59-bdf1-20ec0066e23c%40googlegroups.com.

[keiya]

Isn't it free space?

I tried --max-jobs-run 4 and the results were the same.

In addition, it is as follows after the second log.

# lattice-align-phones --replace-output-symbols=true exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/final.mdl "ark:gunzip -c exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/lat.2.gz |" ark:- | chain-get-supervision --lattice-input=true --frame-subsampling-factor=3 --right-tolerance=5 --left-tolerance=5 --convert-to-pdfs=false exp/chain_cleaned/tdnn_1d_sp/tree exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl ark:- ark:- | nnet3-chain-get-egs --online-ivectors=scp:exp/nnet3_cleaned/ivectors_train_clean_100_sp_hires/ivector_online.scp --online-ivector-period=10 --srand=$[2+0] --left-context=41 --right-context=41 --num-frames=150,110,100 --frame-subsampling-factor=3 --compress=true --num-frames-overlap=0 --transition-model=exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl "ark,s,cs:utils/filter_scp.pl --exclude exp/chain_cleaned/tdnn_1d_sp/egs/valid_uttlist /nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/feats.scp | apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/utt2spk scp:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/cmvn.scp scp:- ark:- |" ark,s,cs:- ark:- | nnet3-chain-copy-egs --random=true --srand=$[2+0] ark:- ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.1.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.2.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.3.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.4.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.5.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.6.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.7.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.8.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.9.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.10.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.11.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.12.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.13.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.14.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.15.ark

# Started at Thu Aug 29 03:46:06 JST 2019

#

chain-get-supervision --lattice-input=true --frame-subsampling-factor=3 --right-tolerance=5 --left-tolerance=5 --convert-to-pdfs=false exp/chain_cleaned/tdnn_1d_sp/tree exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl ark:- ark:-

nnet3-chain-copy-egs --random=true --srand=2 ark:- ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.1.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.2.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.3.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.4.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.5.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.6.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.7.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.8.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.9.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.10.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.11.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.12.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.13.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.14.ark ark:exp/chain_cleaned/tdnn_1d_sp/egs/cegs_orig.2.15.ark

lattice-align-phones --replace-output-symbols=true exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/final.mdl 'ark:gunzip -c exp/chain_cleaned/tri4b_cleaned_train_clean_100_sp_lats/lat.2.gz |' ark:-

nnet3-chain-get-egs --online-ivectors=scp:exp/nnet3_cleaned/ivectors_train_clean_100_sp_hires/ivector_online.scp --online-ivector-period=10 --srand=2 --left-context=41 --right-context=41 --num-frames=150,110,100 --frame-subsampling-factor=3 --compress=true --num-frames-overlap=0 --transition-model=exp/chain_cleaned/tdnn_1d_sp/0.trans_mdl 'ark,s,cs:utils/filter_scp.pl --exclude exp/chain_cleaned/tdnn_1d_sp/egs/valid_uttlist /nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/feats.scp | apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/utt2spk scp:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/cmvn.scp scp:- ark:- |' ark,s,cs:- ark:-

LOG (nnet3-chain-get-egs[5.5.356~1-8706]:ComputeDerived():nnet-example-utils.cc:335) Rounding up --num-frames=150,110,100 to multiples of --frame-subsampling-factor=3, to: 150,111,102

apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/utt2spk scp:/nas206ce8_homes/takagi/kaldi/egs/librispeech/s5/data/train_clean_100_sp_hires/split100/2/cmvn.scp scp:- ark:-

WARNING (nnet3-chain-get-egs[5.5.356~1-8706]:main():nnet3-chain-get-egs.cc:386) No pdf-level posterior for key 1069-133699-0000

WARNING (nnet3-chain-get-egs[5.5.356~1-8706]:main():nnet3-chain-get-egs.cc:386) No pdf-level posterior for key 1069-133699-0001

Does this affect anything?

I forgot to write. The feature value was changed from mfcc to fbank_pitch.

2019年8月29日木曜日 3時35分26秒 UTC+9 Dan Povey:

I suspect you might be misinterpreting the output of `df`.  But if not, it could be a memory-full problem, with get_egs.sh using up too much memory.  To train.py's --egs.opts option, you could add the extra option --max-jobs-run 4 or something like that.

On Wed, Aug 28, 2019 at 11:30 AM keiya <keiy...@gmail.com> wrote:

by df
270GB

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi...@googlegroups.com.

[Daniel Povey]

RE the output of `df`, it would usually print both free and total, so I can't tell without context.

RE the second log: it looks to me like either it hadn't finished, or the disk filled up while it was running, because the log is partial.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/f475fc80-04c9-4183-a408-5f980aa6d537%40googlegroups.com.

[keiya]

I see.

The result by df is as follows.

Filesystem                      Size  Used Avail Use% Mounted on

udev                             63G     0   63G   0% /dev

tmpfs                            13G   18M   13G   1% /run

/dev/sda3                       886G  105G  737G  13% /

tmpfs                            63G  308K   63G   1% /dev/shm

tmpfs                           5.0M  4.0K  5.0M   1% /run/lock

tmpfs                            63G     0   63G   0% /sys/fs/cgroup

/dev/sdb1                       1.8T   68M  1.7T   1% /mnt/data1

/dev/sda1                       945M  220M  660M  25% /boot

tmpfs                            13G   52K   13G   1% /run/user/1000

192.168.100.87:/tank_main/home  6.3T  6.0T  269G  96% /tank_home

192.168.100.88:/homes            10T  2.8T  7.2T  29% /nas206ce8_homes

tmpfs                            13G     0   13G   0% /run/user/1077

data (corpus etc.) is in nas206ce8_home, exp etc is in tank_home.

If there is not enough space, it is tank_home.

2019年8月29日木曜日 4時08分03秒 UTC+9 Dan Povey:

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/f475fc80-04c9-4183-a408-5f980aa6d537%40googlegroups.com.

[Daniel Povey]

269G  is the amount of free space.

However, it's quite possible that even before  that disk is full, it would impose some kind of per-user quota.

That's something you should ask your local sysadmin about.

You could attempt to create a large file and see if it succeeds, e.g.

head -c 1000000000 /dev/urandom > foo

echo $?

will return 0 if it succeeds.

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/2c27b5d8-173a-40fa-9b73-8c218f9f5b1e%40googlegroups.com.

[keiya]

I tried it.

and I got 0.

local sysadmin said there was no capacity limit.

2019年8月30日金曜日 9時11分50秒 UTC+9 Dan Povey:

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/2c27b5d8-173a-40fa-9b73-8c218f9f5b1e%40googlegroups.com.

[keiya]

Error: Isn't "gzip: stdout: Broken pipe" relevant?

Somewhere big output seems to be connected by pipeline.

2019年8月30日金曜日 9時11分50秒 UTC+9 Dan Povey:

269G  is the amount of free space.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/2c27b5d8-173a-40fa-9b73-8c218f9f5b1e%40googlegroups.com.

[Daniel Povey]

The broken pipe could mean one of the programs died, but for what reason I don't know.. could be memory limits or many other things.  Running with fewer jobs should fix it.  The 2nd time you ran I saw no errors in the log, only a truncated log.  

I'd ask someone local who is more experienced to try to figure it out.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/ac8b317a-38d0-4f93-885c-921161809a75%40googlegroups.com.

[keiya]

Noted with thanks.

2019年8月31日土曜日 5時34分22秒 UTC+9 Dan Povey:

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/ac8b317a-38d0-4f93-885c-921161809a75%40googlegroups.com.