contractive autoencoder loss

[lvd...@gmail.com]

I'm trying to implement contractive autoencoder using examples from keras and https://github.com/wiseodd/hipsternet/blob/master/adhoc/autoencoders.py

I'm not sure if I am using the layer name correctly for stacked autoencoder when defining loss function. Below is an example similar to what I am trying to do, just an answer relating to if I am doing it wrong vs right would suffice. Thanks so much for your time.

input_img = Input(shape=(784,))
encoded = Dense(128, activation='relu',name='encoded')(input_img)
encoded = Dense(64, activation='relu')(encoded)
encoded = Dense(32, activation='relu')(encoded)

decoded = Dense(64, activation='relu')(encoded)
decoded = Dense(128, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)

autoencoder = Model(input=input_img, output=decoded)
autoencoder.compile(optimizer='adadelta', loss='contractive_loss')

autoencoder.fit(x_train, x_train,
                nb_epoch=100,
                batch_size=256,
                shuffle=True,
                validation_data=(x_test, x_test))


import keras.backend as K
def contractive_loss(y_pred, y_true):
        mse = K.mean(K.square(y_true - y_pred), axis=1)

        W = K.variable(value=autoencoder.get_layer('encoded').get_weights()[0])  # N x N_hidden
        W = K.transpose(W)  # N_hidden x N
        h = model.get_layer('encoded').output
        dh = h * (1 - h)  # N_batch x N_hidden

        # N_batch x N_hidden * N_hidden x 1 = N_batch x 1
        contractive = lam * K.sum(dh**2 * K.sum(W**2, axis=1), axis=1)

        return mse + contractive

[d.zhan...@gmail.com]

To my knowledge, each layer in stacked autoencoder should be pre-trained (layer-wise training), like that the first layer with 128 nodes should be added in stacked autoencoder after pre-trained as usual. If three or more layers connected together may be difficult to train directly. Hope helpful for you. 

在 2017年1月12日星期四 UTC+8上午1:04:37，lvd...@gmail.com写道：

[lvd...@gmail.com]

Thanks. I'm a little confused, so when we build a multi-layer autoencoder similar to one in keras examples, we only pass loss once when compiling it. How does that accomplishes layer wise training compared to what I am trying.

[Daπid]

On 11 January 2017 at 18:04, <lvd...@gmail.com> wrote:
> contractive = lam * K.sum(dh**2 * K.sum(W**2, axis=1), axis=1)

That looks like a L2 regularisation on the encoded layer. It is easier
to add it like that than as a loss.

[Lovedeep Singh]

Please forgive me for naive questions. Going back to my original post, is it the right way to use contractive loss? If not, can somebody please point me to a resource or a similar example using keras.

Thank you so much.

[su...@quant-one.com]

It fundamentally doesn't make sense because you are not calculating gradients with respect to parameters.