Cholesky decomposition failed. Maybe matrix is not positive definite.

1,325 views

Skip to first unread message

laine...@gmail.com

unread,

Jun 23, 2020, 6:59:59 AM6/23/20

to kaldi-help

Hi, I met the "Cholesky decomposition failed" error when I do chain training, the error as follows. I used sox tool to cut portions out of the audio, and do feature extraction normally, but cmvn is computed from the original audio. If I just run one epoch, training seems ok, but if the epoch is large than one, it will meet the error.

# nnet3-chain-train --use-gpu=yes --apply-deriv-weights=False --l2-regularize=5e-05 --leaky-hmm-coefficient=0.1 --read-cache=exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/cache.1 --write-cache=exp/chainno_ivector/tdnn_1b1

_1195-130164_sup_data_regularize_sp/cache.2 --xent-regularize=0.1 --print-interval=10 --momentum=0.0 --max-param-change=2.0 --backstitch-training-scale=0.0 --backstitch-training-interval=1 --l2-regularize-factor=1.0 --srand=1 "nnet3-am-copy --raw=true --learning-rate=0.000857695898591 --edits='' --scale=1.0 exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/1.mdl - |" exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/den.fst "ark,bg:nnet3-chain-copy-egs --outputs=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.output.1.ark --weights=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.weight.1.ark --frame-shift=2 scp:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.1.scp ark:- | nnet3-chain-shuffle-egs --buffer-size=5000 --srand=1 ark:- ark:- | nnet3-chain-merge-egs --minibatch-size=2:32,64,128 ark:- ark:- |" exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/2.1.raw

# Started at Tue Jun 23 18:46:44 CST 2020

nnet3-chain-train --use-gpu=yes --apply-deriv-weights=False --l2-regularize=5e-05 --leaky-hmm-coefficient=0.1 --read-cache=exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/cache.1 --write-cache=exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/cache.2 --xent-regularize=0.1 --print-interval=10 --momentum=0.0 --max-param-change=2.0 --backstitch-training-scale=0.0 --backstitch-training-interval=1 --l2-regularize-factor=1.0 --srand=1 "nnet3-am-copy --raw=true --learning-rate=0.000857695898591 --edits='' --scale=1.0 exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/1.mdl - |" exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/den.fst 'ark,bg:nnet3-chain-copy-egs --outputs=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.output.1.ark --weights=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.weight.1.ark --frame-shift=2 scp:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.1.scp ark:- | nnet3-chain-shuffle-egs --buffer-size=5000 --srand=1 ark:- ark:- | nnet3-chain-merge-egs --minibatch-size=2:32,64,128 ark:- ark:- |' exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/2.1.raw

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuId():cu-device.cc:228) Not in compute-exclusive mode. Suggestion: use 'nvidia-smi -c 3' to set compute exclusive mode

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuIdAuto():cu-device.cc:408) Selecting from 4 GPUs

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuIdAuto():cu-device.cc:423) cudaSetDevice(0): Tesla K40m free:10370M, used:1069M, total:11439M, free/total:0.906489

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuIdAuto():cu-device.cc:423) cudaSetDevice(1): Tesla K40m free:5517M, used:5922M, total:11439M, free/total:0.482271

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuIdAuto():cu-device.cc:423) cudaSetDevice(2): Tesla K40m free:2935M, used:8504M, total:11439M, free/total:0.256575

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuIdAuto():cu-device.cc:423) cudaSetDevice(3): Tesla K40m free:8256M, used:3183M, total:11439M, free/total:0.721719

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuIdAuto():cu-device.cc:471) Device: 0, mem_ratio: 0.906489

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuId():cu-device.cc:352) Trying to select device: 0

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:SelectGpuIdAuto():cu-device.cc:481) Success selecting device 0 free mem ratio: 0.906489

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:FinalizeActiveGpu():cu-device.cc:308) The active GPU is [0]: Tesla K40m free:10328M, used:1111M, total:11439M, free/total:0.902818 version 3.5

nnet3-am-copy --raw=true --learning-rate=0.000857695898591 --edits= --scale=1.0 exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/1.mdl -

LOG (nnet3-am-copy[5.5.726~1-ffc4f]:main():nnet3-am-copy.cc:153) Copied neural net from exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/1.mdl to raw format as -

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:NnetChainTrainer():nnet-chain-training.cc:51) Read computation cache from exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/cache.1

nnet3-chain-copy-egs --outputs=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.output.1.ark --weights=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.weight.1.ark --frame-shift=2 scp:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.1.scp ark:-

nnet3-chain-shuffle-egs --buffer-size=5000 --srand=1 ark:- ark:-

nnet3-chain-merge-egs --minibatch-size=2:32,64,128 ark:- ark:-

LOG (nnet3-chain-copy-egs[5.5.726~1-ffc4f]:main():nnet3-chain-copy-egs.cc:395) Read 762 neural-network training examples, wrote 762

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:439) On time 0, alpha-beta product 184.252 != 128 alpha-dash-sum = 140.8, beta-dash-sum = 1.92353e+06

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:444) Excessive error detected, will abandon this minibatch

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:452) On time 0, log-prob-deriv sum 184.25 != 128

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:455) Excessive error detected, will abandon this minibatch

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:ComputeChainObjfAndDeriv():chain-training.cc:286) Objective function is -34260.4 and denominator computation (if done) returned false, setting objective function to -10 per frame.

ERROR (nnet3-chain-train[5.5.726~1-ffc4f]:Cholesky():tp-matrix.cc:110) Cholesky decomposition failed. Maybe matrix is not positive definite.

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7f337053f2da]

nnet3-chain-train(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x4115e3]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(kaldi::TpMatrix<float>::Cholesky(kaldi::SpMatrix<float> const&)+0x1b1) [0x7f33707a5ce3]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::OnlineNaturalGradient::ReorthogonalizeRt1(kaldi::VectorBase<float> const&, float, kaldi::CuMatrixBase<float>*, kaldi::CuMatrixBase<float>*, kaldi::CuMatrixBase<float>*)+0x3a4) [0x7f3371e70964]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::OnlineNaturalGradient::PreconditionDirectionsInternal(float, float, bool, kaldi::Vector<float> const&, kaldi::CuMatrixBase<float>*, kaldi::CuMatrixBase<float>*)+0xfb2) [0x7f3371e71e7e]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::OnlineNaturalGradient::PreconditionDirections(kaldi::CuMatrixBase<float>*, float*)+0x1c2) [0x7f3371e72bd4]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::OnlineNaturalGradient::Init(kaldi::CuMatrixBase<float> const&)+0xde) [0x7f3371e72950]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::OnlineNaturalGradient::PreconditionDirections(kaldi::CuMatrixBase<float>*, float*)+0x4a) [0x7f3371e72a5c]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NaturalGradientAffineComponent::Update(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kaldi::CuMatrixBase<float> const&, kaldi::CuMatrixBase<float> const&)+0x214) [0x7f3371e2c1f6]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::AffineComponent::Backprop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kaldi::nnet3::ComponentPrecomputedIndexes const*, kaldi::CuMatrixBase<float> const&, kaldi::CuMatrixBase<float> const&, kaldi::CuMatrixBase<float> const&, void*, kaldi::nnet3::Component*, kaldi::CuMatrixBase<float>*) const+0xa2) [0x7f3371e298ac]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetComputer::ExecuteCommand()+0x87c) [0x7f3371ec12de]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetComputer::Run()+0x18a) [0x7f3371ec2052]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetChainTrainer::TrainInternal(kaldi::nnet3::NnetChainExample const&, kaldi::nnet3::NnetComputation const&)+0x76) [0x7f3371f140ce]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetChainTrainer::Train(kaldi::nnet3::NnetChainExample const&)+0x19d) [0x7f3371f1450b]

nnet3-chain-train(main+0x84d) [0x410483]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f336f692830]

nnet3-chain-train(_start+0x29) [0x40fb69]

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:ReorthogonalizeRt1():natural-gradient-online.cc:248) Cholesky or Invert() failed while re-orthogonalizing R_t. Re-orthogonalizing on CPU.

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:ReorthogonalizeRt1():natural-gradient-online.cc:241) Cholesky out of expected range, reorthogonalizing with Gram-Schmidt

LOG (nnet3-chain-train[5.5.726~1-ffc4f]:UpdateNnetWithMaxChange():nnet-utils.cc:2189) Per-component max-change active on 7 / 10 Updatable Components. (Smallest factor=3.27101e-06 on tdnn3.affine with max-change=0.75).

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:439) On time 0, alpha-beta product 132.565 != 128 alpha-dash-sum = 140.8, beta-dash-sum = 2.4947e+06

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:444) Excessive error detected, will abandon this minibatch

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:452) On time 0, log-prob-deriv sum 132.537 != 128

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:BetaGeneralFrameDebug():chain-denominator.cc:455) Excessive error detected, will abandon this minibatch

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:ComputeChainObjfAndDeriv():chain-training.cc:286) Objective function is -7494.12 and denominator computation (if done) returned false, setting objective function to -10 per frame.

WARNING (nnet3-chain-train[5.5.726~1-ffc4f]:ReorthogonalizeRt1():natural-gradient-online.cc:241) Cholesky out of expected range, reorthogonalizing with Gram-Schmidt

ASSERTION_FAILED (nnet3-chain-train[5.5.726~1-ffc4f]:HouseBackward():qr.cc:124) Assertion failed: (KALDI_ISFINITE(sigma) && "Tridiagonalizing matrix that is too large or has NaNs.")

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7f337053f2da]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::KaldiAssertFailure_(char const*, char const*, int, char const*)+0x6c) [0x7f337053fd48]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(void kaldi::HouseBackward<float>(int, float const*, float*, float*)+0x131) [0x7f33707aa4f7]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(kaldi::SpMatrix<float>::Tridiagonalize(kaldi::MatrixBase<float>*)+0x147) [0x7f33707aa84b]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(kaldi::SpMatrix<float>::Eig(kaldi::VectorBase<float>*, kaldi::MatrixBase<float>*) const+0xa7) [0x7f33707ac0c7]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::OnlineNaturalGradient::PreconditionDirections(kaldi::CuMatrixBase<float>*, float*)+0x1c2) [0x7f3371e72bd4]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetComputer::ExecuteCommand()+0x87c) [0x7f3371ec12de]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetComputer::Run()+0x18a) [0x7f3371ec2052]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetChainTrainer::Train(kaldi::nnet3::NnetChainExample const&)+0x19d) [0x7f3371f1450b]

nnet3-chain-train(main+0x84d) [0x410483]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f336f692830]

nnet3-chain-train(_start+0x29) [0x40fb69]

ERROR (nnet3-chain-merge-egs[5.5.726~1-ffc4f]:Write():kaldi-matrix.cc:1404) Failed to write matrix to stream

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7ff9b96c52da]

nnet3-chain-merge-egs(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x412aed]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(kaldi::MatrixBase<float>::Write(std::ostream&, bool) const+0x1c3) [0x7ff9b990b8bf]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(kaldi::GeneralMatrix::Write(std::ostream&, bool) const+0x63) [0x7ff9b993c2e5]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetIo::Write(std::ostream&, bool) const+0x90) [0x7ff9b9f61d36]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetChainExample::Write(std::ostream&, bool) const+0xd6) [0x7ff9b9fcb40a]

nnet3-chain-merge-egs(kaldi::KaldiObjectHolder<kaldi::nnet3::NnetChainExample>::Write(std::ostream&, bool, kaldi::nnet3::NnetChainExample const&)+0x6b) [0x41a0e9]

nnet3-chain-merge-egs(kaldi::TableWriterArchiveImpl<kaldi::KaldiObjectHolder<kaldi::nnet3::NnetChainExample> >::Write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kaldi::nnet3::NnetChainExample const&)+0x182) [0x41a346]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::ChainExampleMerger::WriteMinibatch(std::vector<kaldi::nnet3::NnetChainExample, std::allocator<kaldi::nnet3::NnetChainExample> >*)+0x3be) [0x7ff9b9fcdf84]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::ChainExampleMerger::AcceptExample(kaldi::nnet3::NnetChainExample*)+0x450) [0x7ff9b9fce560]

nnet3-chain-merge-egs(main+0x230) [0x4117f6]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff9b8818830]

nnet3-chain-merge-egs(_start+0x29) [0x4114f9]

WARNING (nnet3-chain-merge-egs[5.5.726~1-ffc4f]:Write():util/kaldi-holder-inl.h:57) Exception caught writing Table object. kaldi::KaldiFatalError

WARNING (nnet3-chain-merge-egs[5.5.726~1-ffc4f]:Write():util/kaldi-table-inl.h:1057) Write failure to standard output

ERROR (nnet3-chain-merge-egs[5.5.726~1-ffc4f]:Write():util/kaldi-table-inl.h:1515) Error in TableWriter::Write

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7ff9b96c52da]

nnet3-chain-merge-egs(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x412aed]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::ChainExampleMerger::AcceptExample(kaldi::nnet3::NnetChainExample*)+0x450) [0x7ff9b9fce560]

nnet3-chain-merge-egs(main+0x230) [0x4117f6]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff9b8818830]

nnet3-chain-merge-egs(_start+0x29) [0x4114f9]

WARNING (nnet3-chain-merge-egs[5.5.726~1-ffc4f]:Write():util/kaldi-table-inl.h:1047) Attempting to write to invalid stream.

ERROR (nnet3-chain-merge-egs[5.5.726~1-ffc4f]:Write():util/kaldi-table-inl.h:1515) Error in TableWriter::Write

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7ff9b96c52da]

nnet3-chain-merge-egs(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x412aed]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::ChainExampleMerger::Finish()+0x5f3) [0x7ff9b9fcec05]

nnet3-chain-merge-egs(kaldi::nnet3::ChainExampleMerger::~ChainExampleMerger()+0x12) [0x414896]

nnet3-chain-merge-egs(main+0x591) [0x411b57]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff9b8818830]

nnet3-chain-merge-egs(_start+0x29) [0x4114f9]

terminate called after throwing an instance of 'kaldi::KaldiFatalError'

what(): kaldi::KaldiFatalError

ERROR (nnet3-chain-shuffle-egs[5.5.726~1-ffc4f]:Write():compressed-matrix.cc:563) Error writing compressed matrix to stream.

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7fe3c8c002da]

nnet3-chain-shuffle-egs(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x413687]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(kaldi::CompressedMatrix::Write(std::ostream&, bool) const+0x19e) [0x7fe3c8e71a70]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-matrix.so(kaldi::GeneralMatrix::Write(std::ostream&, bool) const+0x52) [0x7fe3c8e772d4]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetIo::Write(std::ostream&, bool) const+0x90) [0x7fe3c949cd36]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-nnet3.so(kaldi::nnet3::NnetChainExample::Write(std::ostream&, bool) const+0xd6) [0x7fe3c950640a]

nnet3-chain-shuffle-egs(kaldi::KaldiObjectHolder<kaldi::nnet3::NnetChainExample>::Write(std::ostream&, bool, kaldi::nnet3::NnetChainExample const&)+0x6b) [0x41c557]

nnet3-chain-shuffle-egs(kaldi::TableWriterArchiveImpl<kaldi::KaldiObjectHolder<kaldi::nnet3::NnetChainExample> >::Write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kaldi::nnet3::NnetChainExample const&)+0x182) [0x41c7b4]

nnet3-chain-shuffle-egs(main+0x866) [0x4124ec]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fe3c7d53830]

nnet3-chain-shuffle-egs(_start+0x29) [0x411bb9]

WARNING (nnet3-chain-shuffle-egs[5.5.726~1-ffc4f]:Write():util/kaldi-holder-inl.h:57) Exception caught writing Table object. kaldi::KaldiFatalError

WARNING (nnet3-chain-shuffle-egs[5.5.726~1-ffc4f]:Write():util/kaldi-table-inl.h:1057) Write failure to standard output

ERROR (nnet3-chain-shuffle-egs[5.5.726~1-ffc4f]:Write():util/kaldi-table-inl.h:1515) Error in TableWriter::Write

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7fe3c8c002da]

nnet3-chain-shuffle-egs(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x413687]

nnet3-chain-shuffle-egs(main+0x8b6) [0x41253c]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fe3c7d53830]

nnet3-chain-shuffle-egs(_start+0x29) [0x411bb9]

WARNING (nnet3-chain-shuffle-egs[5.5.726~1-ffc4f]:Close():util/kaldi-table-inl.h:1089) Error closing stream: wspecifier is ark:-

ERROR (nnet3-chain-shuffle-egs[5.5.726~1-ffc4f]:~TableWriter():util/kaldi-table-inl.h:1539) Error closing TableWriter [in destructor].

[ Stack-Trace: ]

/home/cca01/work2019/luoxj/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x82c) [0x7fe3c8c002da]

nnet3-chain-shuffle-egs(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x413687]

nnet3-chain-shuffle-egs(kaldi::TableWriter<kaldi::KaldiObjectHolder<kaldi::nnet3::NnetChainExample> >::~TableWriter()+0x6d) [0x416bdd]

nnet3-chain-shuffle-egs(main+0xd59) [0x4129df]

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fe3c7d53830]

nnet3-chain-shuffle-egs(_start+0x29) [0x411bb9]

terminate called after throwing an instance of 'kaldi::KaldiFatalError'

what(): kaldi::KaldiFatalError

sh: line 1: 19521 Done nnet3-chain-copy-egs --outputs=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.output.1.ark --weights=ark:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.weight.1.ark --frame-shift=2 scp:exp/chainno_ivector/tdnn_1b1_1195-130164_sup_data_regularize_sp/egs_comb/cegs.1.scp ark:-

19522 Aborted (core dumped) | nnet3-chain-shuffle-egs --buffer-size=5000 --srand=1 ark:- ark:-

19523 Aborted (core dumped) | nnet3-chain-merge-egs --minibatch-size=2:32,64,128 ark:- ark:-

# Accounting: time=8 threads=1

# Ended (code 134) at Tue Jun 23 18:46:52 CST 2020, elapsed time 8 seconds

Daniel Povey

unread,

Jun 23, 2020, 11:26:41 PM6/23/20

to kaldi-help

Instability. Would be an issue with your xconfig or with your training configuration (e.g. learning rate or l2 too high)

--
Go to http://kaldi-asr.org/forums.html find out how to join
---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/e637cc86-4cf6-4a49-8859-6ca014ec0904n%40googlegroups.com.

mura...@gmail.com

unread,

Oct 21, 2020, 8:08:24 PM10/21/20

to kaldi-help

I can confirm reducing learning rate might lead to a successful training, previously I was training a multilang a CNN-TDNN-f (same script as run_tdnn_multilingual, just changed the architecture) and first, when using lr 0.0015 - 0.00015 the net was giving this Cholelsky error in the second epoch, after reducing to 0.001 - 0.0001 the error now happens on the 4th epoch. I guess if I keep decreasing lr, the training will converge

dragon stone

unread,

Apr 25, 2021, 7:15:17 AM4/25/21

to kaldi-help

I met the same problem as you. Now do you know the reason of "Cholesky decomposition failed. Maybe matrix is not positive definite?"

What configuration do I need to change

Sarah

unread,

Aug 29, 2022, 7:00:31 PM8/29/22

to kaldi-help

I came cross the same problem. It happens from the 1st iteration in my case. However, the training could still going on with error. Does this a fatal error? Any idea of the possible reason?

Thanks!

Daniel Povey

unread,

Aug 29, 2022, 11:39:38 PM8/29/22

to kaldi...@googlegroups.com

That exception is usually caught; if training continues normally it is not a problem.

Go to http://kaldi-asr.org/forums.html to find out how to join the kaldi-help group

---
You received this message because you are subscribed to the Google Groups "kaldi-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/b0c04c9e-fa39-4216-89be-060c085a7619n%40googlegroups.com.

Reply all

Reply to author

Forward

0 new messages