Autoencoder not converging - dimensioanlity reduction

[piotrch...@gmail.com]

Hey,

    I am trying to use autoencoder for dimensionality reduction of small images I have (34x34). Images are scaled to 0,1 range and are binary so only values are 0,1. After training I want to extract middle layer with smallest amount of neurons to treat it as my dimensionally reduced reprecentation.

    I tried almost every combination of activation functions, loss,optimizer and they all are not converging which means I am doing something fundamentally wrong here.... Loss stays the same from epoch 1 until the end.

 

    X =  np.asarray(data)

    

    

    encoder = containers.Sequential([Dense(512,input_dim=1156,

                                           activation='sigmoid'),

                                     Dense(300,input_dim=512,

                                            activation='sigmoid'),

                                     Dense(100,input_dim=300, 

                                           activation='sigmoid')])

    

    decoder = containers.Sequential([Dense(300,input_dim=100),

                                     Dense(512,input_dim=300),

                                     Dense(1156,input_dim=512)])

    

    autoencoder = Sequential()

    

    autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder,

                    output_reconstruction=False))

    

    sgd = SGD(lr=0.1, decay=1e-6, momentum=0.0, nesterov=True)

    autoencoder.compile(loss = 'mse', 

                        optimizer = sgd)

    

    autoencoder.fit(X,X,batch_size=20,nb_epoch=1000,

                    show_accuracy=False,verbose=2)

    

    (trainscore, trainaccuracy) = autoencoder.evaluate(X, X, 

                                batch_size=500, show_accuracy=True)

    

    print("Training Score: " + str(trainscore))

    print("Training Accuracy: " + str(trainaccuracy))

Please help,

P

[notthesha...@gmail.com]

I'm having the same issue with the Autoencoder model that previously worked until I updated keras. Loss remains the same through training in each epoch. I've tried changing all the hyper-parameters I can think of with no luck. Any Idea what is going on?

[Greg Hayes]

Admittedly, I'm new to Keras, but I have been trying to build an autoencoder, and having the same problem.  I've gone so far as to trying to construct an autoencoder that would learn the identity function, but its still not converging.  

encoder = containers.Sequential([Dense(5, input_dim=5, activation='softmax'),

                                 Dense(5)

                                ]

                               )

decoder = containers.Sequential([Dense(5, input_dim=5, activation='softmax'),

                                ]

                               )

autoencoder=Sequential()

autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False))

sgd = SGD(lr=0.01, momentum=1e-6, decay=0.0, nesterov=True)

autoencoder.compile(loss='rmse', optimizer=sgd)
autoencoder.fit(f, f, nb_epoch = 200, show_accuracy=True, verbose=1, shuffle=True)

The accuracy and loss function never change, regardless of how many epochs I train on.  Sample set is ~1200.

Any help or guidance would be appreciated.

[Kris Cao]

I think I've found the issue:

the AutoEncoder class grabs the parameters to update off the encoder and decoder layers when AutoEncoder.build() is called. Normally, this is called at two times:
1) by set_previous when you add a layer to a container with one or more layers already
2) by set_input_shape when you specify the input dimension of the first layer of the network.

However, when you call autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False)), neither is happening, so it doesn't initialise the parameters of the AutoEncoder layer properly.

To work around this, you'll have to add the keyword argument input_dim when calling autoencoder.add, and set it equal to the input dimension of the autoencoder (i.e. call autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False), input_dim=1156) ).

Hope that helps
Kris

[haoq...@gmail.com]

This does not work for me, it shows like:
TypeError: add() got an unexpected keyword argument 'input_dim'

[Kris Cao]

Sorry, input_dim is only recognized by some layers. Try input_shape=(1156,) (or the equivalent for you).

[haoq...@gmail.com]

Hi I did lay by layer pretraining followed by example in keras github, this is my code, it does not work, when I add input_shape(438,)

# layer-wise pre-training

trained_encoders = []

X_train_temp = X_train

#para

batch_size = 64

nb_epoch = 5

nb_hidden_layers = [438, 200, 120 ]

for n_in, n_out in zip(nb_hidden_layers[:-1], nb_hidden_layers[1:]):

    print ('Pre-training the layer: Input {} -> Output {}'.format(n_in, n_out))

    # Creat AutoEncoder and training

    ae = Sequential()

    encoder = containers.Sequential([Dense(n_out,input_dim = n_in, activation = 'sigmoid')])

    decoder = containers.Sequential([Dense(n_in,input_dim = n_out, activation = 'sigmoid')])

    ae.add(AutoEncoder(encoder=encoder, decoder=decoder, output_reconstruction=False),input_shape(438,))

    ae.compile(loss='mean_squared_error', optimizer='sgd')

    ae.fit(X_train_temp, X_train_temp, batch_size=batch_size, nb_epoch=nb_epoch)

    # Store trainined weight

    trained_encoders.append((ae.layers[0].encoder,ae.layers[0].encoder.get_weights()))

    # Update training data

    X_train_temp = ae.predict(X_train_temp)

[Kris Cao]

You have to make sure that input_shape is correct on each loop iteration. Try input_shape=(n_in,) instead.

[Kris Cao]

Actually, scratch this suggestion: AutoEncoder calls super.__init__() before sets its encoder and decoder attributes, and so the call to self.build won't pick up the encoder and decoder parameters. I can raise an issue on the Keras GitHub if you'd like.

Kris

[Greg Hayes]

That would be great.  When I attempt to use the 'input_shape' workaround, I'm getting a message that reads '.add' is getting an unexpected keyword argument.

[Kris Cao]

As a workaround, you can call build() manually after you've added the layer to your network. Add the line ae.layers[0].build() after ae.add(AutoEncoder...) and see if that works.

[haoq...@gmail.com]

Cool, it worked. Thanks a lot

[Greg Hayes]

This worked for me as well.  Thanks!