Using embeddings in autoencoders

[Ajay Mandlekar]

Hi everyone,

I wanted to train an autoencoder, but I want the first layer to be an embedding layer. The training examples are all going to be 2-sparse vectors that are part of a large vocabulary of around 60,000 words, so it is convenient for me to use the embedding layer to handle the input, but I'm confused on how I would make the decoder stage of the autoencoder, since it would need to be the opposite of an embedding layer. 

Does anyone know how to do this? Thanks!

[François Chollet]

It is not possible to "decode" embeddings in a learnable/differentiable way. The best you can do is a type of nearest-neighbor search in a pre-trained embedding space (e.g. your input Embedding layer). That's not a Keras issue, that's an algorithmic issue.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/f04530ca-9a10-46d5-91e3-bde47cf717f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ajay Mandlekar]

I thought that the only difference between a Dense layer and a Embedding layer is that the Embedding layer takes sparse vectors as input. If this is not the case, what is the difference between the Embedding layer and a Dense layer and what makes the former non-differentiable when you try to decode it?

[François Chollet]

A Dense layer is a matrix multiplication layer (Output = W . Input + b). An Embedding layer is a dictionary lookup (Output = dict[input]). Hope that clears it up.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/94c556d1-3aa3-4c39-b48d-97f7e497686b%40googlegroups.com.

[Ajay Mandlekar]

So how is the dictionary created? Is there any learning that goes into making the embedding or is it an arbitrary assignment? 

For my application, I am now thinking that I'll have to use scipy's sparse matrices with an autoencoder that only uses dense layers. Does keras have support for training and testing matrices coming from scipy's sparse matrices? I'm trying to reduce memory strain - I will have many training examples that are 2-sparse vectors in a 60,000 dimensional vector space and the autoencoder will try to minimize reconstruction error. 

Thank you!

[francesc...@gmail.com]

Don't think it accepts scipy sparse matrices. In any case, you can just use model.train_on_batch if you are memory limited.

Francesco