how does nematus support pre-trained word embeddings?

[sam]

Is it available for nematus to support loading google pre-trained word embeddings? If no, where is the best location to enhance code? I think it would be 

emb = get_layer_constr('embedding')(tparams, x, suffix='', factors= options['factors'])

 

[Rico Sennrich]

Hello Sam,

the best way to start is to look at how Nematus will override randomly initialized parameters if you use the "--reload" option. You can modify this function to load pre-trained word embeddings.

You could also achieve your goal without any changes to Nematus, by first training a model for 0 minibatches (or 1 if that doesn't work), then taking the saved model and replacing the word embeddings with your pretrained ones, then training again with the "--reload" option.

best wishes,
Rico

On 28/04/17 09:09, sam wrote:

Is it available for nematus to support loading google pre-trained word embeddings? If no, where is the best location to enhance code? I think it would be 

emb = get_layer_constr('embedding')(tparams, x, suffix='', factors= options['factors'])

 

--
You received this message because you are subscribed to the Google Groups "Nematus Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nematus-suppo...@googlegroups.com.
To post to this group, send email to nematus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nematus-support/3bab07ca-7239-46db-bc66-aa89b3fc6878%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sam]

I think the correct space would be the param_init_embedding_layer in layer.py, which uses norm_weight to initialize the parameter of embedding(Wemb)

[sam]

Another thing is that the embedding [vocab,emb_dim] is not actually a theano tensor object but a numpy ndarray. So, no need to set trainable to false and it will not be updated when training, right?

On Friday, April 28, 2017 at 10:09:18 AM UTC+2, sam wrote:

[Haruo Con]

Hi Sam,

Were you able to include pre-trained embedding in nematus? 

Thanks,

Haru