L1 and L2 regularization with using the models.mlp.PretrainedLayer class?

[cle...@carnegiescience.edu]

Hi,

I'm using Pylearn2 to solve a regression problem in Earth science. I've been training a neural network with Tanh units with 2 hidden layers of 12 and 4 units, connected to 12 inputs and 1 output. I want to use this network for doing regression on real experimental data values: I have 4200 points for the training dataset, and 900 points for the independent validation dataset.

Up to now, I've been initializing my two hidden layers with using models.mlp.Tanh and setting irange to 0.05 and ibias to 1.0. It's working good, particularly with a fine-tuning of the SGD algorithm and with playing with the L1 and L2 regularization terms for avoiding over-training and improving generalization.

I wanted to try to initialize the weights of my two layers with an unsupervised pre-training procedure, so I implemented a way to do so using the DenoisingAutoEncoder model, with Tanh encoders and Linear decoders. I don't think it's going to really improve my results since my network is quite shallow, but just for the sack of consistency, I had to try that.

I'm using the pylearn2.models.mlp.PretrainedLayer class for constructing my final network in my final yaml file for the supervised training part. Right now, I've been successful in fitting my data, but I can't use the L1 and L2 terms with this pylearn2.models.mlp.PretrainedLayer class because it is not implemented. As a result, the network overfits the training dataset and gives not-so-good results on the validation dataset.

So here my question: Does anybody knows a way to implement the L1 and L2 regularization with the pylearn2.models.mlp.PretrainedLayer class? I didn't find anything on the internet or group discussion about that...

Thanks in advance!

Best regards,

Charles Le Losq.

[Jesse Livezey]

If it's not implemented, it would probably be fairly easy to look at how it is implemented in mlp.Linear [1] and add it to the pretrained layer[2]. You'll need to change how the weights are found from the pretrained layer, but it should otherwise be very similar.

[1] https://github.com/lisa-lab/pylearn2/blob/09075120d6d8c8b6d63a6db73090e7f2fe2535a8/pylearn2/models/mlp.py#L2055

[2] https://github.com/lisa-lab/pylearn2/blob/09075120d6d8c8b6d63a6db73090e7f2fe2535a8/pylearn2/models/mlp.py#L4006

[cle...@carnegiescience.edu]

Thanks, yes I tried to add two wrappers in the PreTrainedLayer class as

    @wraps(Layer.get_weight_decay)

    def get_weight_decay(self, coeff):

        if isinstance(coeff, str):

            coeff = float(coeff)

        assert isinstance(coeff, float) or hasattr(coeff, 'dtype')

        W = self.layer_content.get_weights()

        return coeff * T.sqr(W).sum()

    @wraps(Layer.get_l1_weight_decay)

    def get_l1_weight_decay(self, coeff):

        if isinstance(coeff, str):

            coeff = float(coeff)

        assert isinstance(coeff, float) or hasattr(coeff, 'dtype')

        W = self.layer_content.get_weights()

        return coeff * abs(W).sum()

Here I get the weights directly from the model.autoencoder class, contained in layer_content from what I understood. So it should be ok, but I get an error trying to run this code:

Traceback (most recent call last):

  File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/scripts/train.py", line 261, in <module>

    args.verbose_logging, args.debug)

  File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/scripts/train.py", line 251, in train

    train_obj.main_loop(time_budget=time_budget)

  File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/train.py", line 141, in main_loop

    self.setup()

  File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/train.py", line 121, in setup

    self.algorithm.setup(model=self.model, dataset=self.dataset)

  File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/training_algorithms/sgd.py", line 316, in setup

    ** fixed_var_descr.fixed_vars)

  File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/costs/cost.py", line 343, in expr

    costs.append(cost.expr(model, cost_data, **kwargs))

  File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/costs/mlp/__init__.py", line 206, in expr

    total_cost.name = 'MLP_L1Penalty'

Something went wrong in the initialisation of the cost function? Are the weights returned by the autoencoders.get_weights() function in the good format?

[Jesse Livezey]

Is that the full error trace? It looks like there should be more below.

If you look at the get_weights() method, it looks like it returns the value of the weights rather than the theano variable.

https://github.com/lisa-lab/pylearn2/blob/09075120d6d8c8b6d63a6db73090e7f2fe2535a8/pylearn2/models/autoencoder.py#L423

So, I think you can just do

self.layer_content.weights

[cle...@carnegiescience.edu]

Yes, it is the full error trace as it appears in my terminal...

I tried to use self.layer_content.weights as you suggested, but I got another error message:

File "/Users/closq/Dropbox/NeuralNetwork/Viscosity/pylearn2/pylearn2/models/mlp.py", line 4110, in get_l1_weight_decay

    W, = self.layer_content.weights()

TypeError: 'TensorSharedVariable' object is not callable

Here it seems that the layer_content.weights() cannot be assigned to W... Do you have any idea to avoid this issue?

Despite those code issues, I solved my problem by avoiding using the PreTrainedLayer class and directly reading the weights of the autoencoders in a python script, and assigning them to the MLP after creating the latter. However, it will be nice to find a way to be able to directly use the PreTrainedLayer class with L1 and L2 regularizations in a yaml script.

[Jesse Livezey]

The weights variable isn't a function, which is why it's saying it's not callable. So this should work:

W = self.layer_content.weights

[cle...@carnegiescience.edu]

My mistake! Yes, this works, thanks a lot for your help!