Separate Affine Layer for Chain Training Xent Regularization
al...@i2x.ai
From the configs for the tdnn model in the swbd recipe it looks like there are separate affine layers for each head, one for the nnet outputs that are fed to `ComputeChainObjfAndDeriv` and another for the xent outputs. As I understand it, the xent regularization is performed by driving a scaled xent gradient (computed as part of the chain loss op) back through the network. This makes sense to me, but I don't understand what the advantage of heaving separate affine layers for each head. Wouldn't it make more sense to share the parameters for each head? Or is this what it is actually doing underneath?
Daniel Povey
Empirically having different outputs is better. Possibly because the
regular output does not want values that are close to what the xent
output needs, so to avoid damaging it you'd have to set the xent
regularization scale so low as to be useless.
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups
> "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kaldi-help+...@googlegroups.com.
> To post to this group, send email to kaldi...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kaldi-help/edfb6281-8cd4-4e3e-b7b0-d58cc8de6645%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
al...@i2x.ai
Daniel Povey
The posterior distribution over states is determined by the numerator
lattice or supervision FST (which reflects the transcript) and the
chain output (which affects the alignment). But the gradient depends
more strongly on the xent output because that determines the
posterior... if it matches the distribution from the soft-alignment of
the transcript given the chain output, there would be no gradient.
The chain output won't affect the distribution super strongly because
it's highly constrained by the transcript.
Dan
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups
> "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kaldi-help+...@googlegroups.com.
> To post to this group, send email to kaldi...@googlegroups.com.
> To view this discussion on the web visit
Ilya Edrenkin
Frankly, I still fail to understand what is going on with the xent regularization. Apparently the function that computes the chain loss and both chain and xent derivatives only has one input, nnet_output. I assume it to be the output of the chain head of the network. https://github.com/kaldi-asr/kaldi/blob/f8b678a61e932f4858115dbe2d11caed48a7dbac/src/chain/chain-training.h#L118
However, if xent regularization is used, xent gradients are pushed through another head of the network, if I get it correctly: https://github.com/kaldi-asr/kaldi/blob/f8b678a61e932f4858115dbe2d11caed48a7dbac/src/nnet3/nnet-chain-training.cc#L343
I don't get why it could work. Given that the supervision head is different from the xent head, I don't understand how we can compute the gradient of xent loss wrt supervision output. Probably my reasoning is wrong at some earlier point?
Best,
Ilya
Daniel Povey
normalized log-probs, and the derivatives w.r.t. those do not depend
on the log-probs themselves, only on the supervision.
Dan
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
> To post to this group, send email to kaldi...@googlegroups.com.
Ilya Edrenkin
The xent output seems to be fetched at this line: https://github.com/kaldi-asr/kaldi/blob/f8b678a61e932f4858115dbe2d11caed48a7dbac/src/nnet3/nnet-chain-training.cc#L316 . As it doesn't seem to influence xent_deriv, it looks like the xent head gets updated with a great that might be not correct.
Consider an extreme case where the computation performed by the xent head is the negation of the computation performed by the supervision head. Then, if we estimate gradients for the xent head using supervision chain output, we are actually increasing the loss.
Realistically in most cases I would expect xent and supervision heads to be uncorrelated, turning this update into random gradient noise.
I believe I am missing something trivial early on that would explain the rationale?
Daniel Povey
depend strongly on the xent output values, but it happens that the
gradients after the LogSoftmax are invariant to them.
That's just the way the cross-entopy objective works.
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
> To post to this group, send email to kaldi...@googlegroups.com.
Ilya Edrenkin
My current understanding is that in swbd nnet3-chain config we see a network with common stem, which splits into two branches after certain layer, close to the top. These both branches are parameterized and don't share parameters. One of them ends with "supervision" or "chain" output (linear), another ends with xent output (last transformation on xent branch is LogSoftmax).
When we call ComputeChainObjfAndDeriv we pass the output of the first head as an argument, nnet_output matrix. We get back two matrices with gradients, nnet_output_deriv and xent_deriv. Later xent_deriv is not modified (except for scaling with weights).
xent_output is used to compute the xent_objf, but doesn't affect the xent_deriv: https://github.com/kaldi-asr/kaldi/blob/f8b678a61e932f4858115dbe2d11caed48a7dbac/src/nnet3/nnet-chain-training.cc#L320
I assume that at this point xent_deriv is not really the derivative, but soft alignment (per-frame posteriors wrt the output of supervision head?). The comment seems to be also saying these are posteriors, not gradients.
It looks plausible that backprop of logsoftmax handles these posteriors correctly as its input. But: I still find it strange that we update the xent head with the gradient whose computation didn't use the parameters of the xent head (e.g. affine transforms in it) in the forward pass.
Could you please correct my understanding?
Thanks for your patience!
Ilya
Ilya Edrenkin
It was just the naming xent_deriv that confused me, but now I see why the procedure makes sense.
Thanks a lot Dan!
Daniel Povey
branch *are* the soft posteriors. That's how the cross-entropy
objective works. Write down the equations and you'll see it.
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-help+...@googlegroups.com.
> To post to this group, send email to kaldi...@googlegroups.com.
Ilya Edrenkin
(The source of confusion for me was that I for some reason erroneously thought that xent_deriv contains log-posteriors, not properly normalized probabilities.)