weight initialization for nn.LookupTable

[Peng Sun]

Hi dear Torch 7 users/developers,

What kind of weight initialization should nn.LookupTable use? I know the Gaussian weight initialization with the std involving fan-in/fan-out is popular for Linear layer or Convolutional layer, but a simple answer for that of nn.LookupTable cannot be found by a JFGI. Also, the default way of the lua code is to use a Gaussian with std = 1, does that make sense? 

I understand this topic might slightly divert from pure Torch 7, but it would be great if any one who has applied Torch 7 to NLP could talk about the experience :)  Any links to Torch 7 code for NLP is welcome either!

[soumith]

The default way of the lua code to use gaussian with std of 1 makes sense.

https://github.com/torch/nn/blob/master/LookupTable.lua#L35

Good weight initialization's only consideration seems to be that the resultant responses of a layer are zero-mean and scale invariant. So, this default seems good.

You could alternatively initialize with any priors that you have, for example starting from word2vec embeddings (maybe noisy word2vec) or glove embeddings.

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+un...@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at http://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[Peng Sun]

Hi @smth, the zero mean and scale invariant explanation seems convincing, thanks!

--
You received this message because you are subscribed to a topic in the Google Groups "torch7" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/torch7/-NBrFw8Q6_s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to torch7+un...@googlegroups.com.