LFMMI, CE and L2 loss in Kaldi...

[marc....@protonmail.com]

Hello,

I was wondering about the optimization strategies in Kaldi for CE and output L2 regularization, say for a Librispeech recipe. In principle, CE loss should encourage the numerator of the LF-MMI loss, and output L2 regularization should prevent extremely peaky posterior distributions, so avoid overfitting? Could you share what strategy you followed for optimizing the Librispeech recipe? Can the Librispeech recipe be further optimized by tuning the regularization CE and L2 scaling factors?

Out of curiosity, could the LF-MMI loss become negative when the numerator (likelihood) becomes extremely large? The numerator hypothesis is included in the denominator but with much smaller probability along the n-gram LM, I assume...

Best,

- Marc

[Daniel Povey]

With CE there are issues with how to normalize the softmax, you'd probably need to include some division-by-the-prior after the log-softmax.

We probably tuned the Librispeech recipe pretty well already.

We prevent the LF-MMI loss from becoming negative by composing the numerator paths with the `normalization FST` which has the 

same costs as the denominator; this won't really affect training but makes the objective more interpretable.

--
visit http://kaldi-asr.org/forums.html to find out how to join.
---
You received this message because you are subscribed to the Google Groups "kaldi-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-develope...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-developers/e3814164-e7ac-438d-b614-6cd82de064a4n%40googlegroups.com.

[marc....@protonmail.com]

It makes a lot of sense. Thank you so much, Dan!