The skip connections in the paper "Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks"

[付嘉懿]

Hi all,

  I'm trying to look into the skip connections which was introduced in "Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks" and also had been implemented in nnet3.

  

  I trained a factorized tdnn model in nnet3 and looked into the final.mdl with the command nnet-copy. I felt confused about the line: 

component-node name=tdnn7.affine component=tdnn7.affine input=Append(tdnn7l, Offset(tdnn7l, 3), tdnn6l, tdnn4l, tdnn2l) input-dim=1280 output-dim=1280

  In my understanding, the number of frames of tdnn7l, tdnn6l, tdnn4l and tdnn2l are different because the sub-sampling and the operation like Append(Offset(tdnn6.dropout, -3), tdnn6.dropout).

  If I want to implement the append operation between tdnn7l and tdnn4l，I should know the mapping from the frame index of tdnn7l to tdnn6l.

  eg. The index mapping is: the i-th frame of tdnn7l is concatenated with the (3*i)+5 -th frame of tdnn2l because there is a 3-fold sub-sampling in tdnn4 and 5 Append(-1, 0) operations between tdnn2l and tdnn7l. 

  But when I implement the inference process of this TDNN with Python, I found the output matrix of tdnn7.affine is different from the result of kaldi. 

  I have got the right result on all layers before the tdnn7.affine so I think the mistake should happen in input=Append(tdnn7l, Offset(tdnn7l, 3), tdnn6l, tdnn4l, tdnn2l).

  

  I guess I wrote the wrong mapping between the tdnn7l with tdnn6l, tdnn4l and tdnn2l.

  

  Is anybody can tell me where I did wrong? 

  I look forward to getting some advice as soon as possible.

  Thank you! 

[Daniel Povey]

Your logic generally seems plausible, but there may be something in
there that is not quite exact, maybe something to do with which frames
at each layer are required. To be sure you'd have to figure out
exactly what set of 't' values were required at each layer and what
the mapping to row indexes of the matrix was, in order to properly
work that out. Just writing that out carefully may reveal some error
in your logic.

If you run, say, decoding with a high enough verbose level it will
print out the compiled computation, and part of that printout will be
a list of matrices (search for "m1") and which parts of the
computation they correspond to, with which 't' values. If you look
through that carefully enough you may find out where your assumptions
diverge from the reality of what it is doing. But first try what I
said in the previous paragraph. I certainly don't have time to do an
in-depth debugging of what you are doing.


Dan

> --
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups
> "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kaldi-help+...@googlegroups.com.
> To post to this group, send email to kaldi...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kaldi-help/fb4cfecc-237b-4760-b055-77364dca7ab6%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[付嘉懿]

Dear Dan,

  Thanks! 

  I will try what you said.

  If I want to print the matrix after the Append operation or the Offset operation, which .cpp file should I look into ?  "nnet-compute.cc" or "nnet-descriptor.cc" or any other .cpp files ?

> email to kaldi-help+unsubscribe@googlegroups.com.

> To post to this group, send email to kaldi...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kaldi-help/fb4cfecc-237b-4760-b055-77364dca7ab6%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Go to http://kaldi-asr.org/forums.html find out how to join
---

You received this message because you are subscribed to a topic in the Google Groups "kaldi-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kaldi-help/g_sgGhmehDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kaldi-help+unsubscribe@googlegroups.com.

To post to this group, send email to kaldi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kaldi-help/CAEWAuyR%3D3HyAxqtsLVo31NjEA_tYOFS7JcBwfn_KJrbNc_OeOA%40mail.gmail.com.

[Daniel Povey]

You should look in nnet-compute.cc. It already prints the change in
norm of the matrices involved before and after each command, if you
are at high enough debug level; in principle you could change that to
print the actual matrix, although the output would be very large.

But you won't be able to break out the individual Append() and
Offset() operations, they are 'compiled out' and are not identifiable
individually.

>> > email to kaldi-help+...@googlegroups.com.

>> > To post to this group, send email to kaldi...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/kaldi-help/fb4cfecc-237b-4760-b055-77364dca7ab6%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> Go to http://kaldi-asr.org/forums.html find out how to join
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "kaldi-help" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kaldi-help/g_sgGhmehDU/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> kaldi-help+...@googlegroups.com.

>> To post to this group, send email to kaldi...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kaldi-help/CAEWAuyR%3D3HyAxqtsLVo31NjEA_tYOFS7JcBwfn_KJrbNc_OeOA%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> Go to http://kaldi-asr.org/forums.html find out how to join
> ---
> You received this message because you are subscribed to the Google Groups
> "kaldi-help" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kaldi-help+...@googlegroups.com.

> To post to this group, send email to kaldi...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kaldi-help/CAGinpHTJ4ZCk08PG%3Dsua15BSdMbdFXxrTF8y0E44GfDpBPC%2BKQ%40mail.gmail.com.