How to fix weights of a neural network layer?

[Pedro Tabacof]

Hello,

I have a 2-layer neural network and I want to fix the topmost layer (which is a linear one) and train only the first layer (which is a tanh).

Is there a direct and simple way to do this? I tried using the "evaluate" method, but it didn't seem to change anything when I extracted the gradient of the whole network.

The only way I see to do this is to get the whole gradient, and discard the second layer part, but this does not seem elegant (especially if I had multiple fixed layers, which would be my next development).

Thank you very much.
Pedro.

[soumith]

Override the layer's accGradParameters function to be an empty function.

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+un...@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at http://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[soumith]

If you want to use this kind of layer a lot, just derive a layer with nn.Linear as parent, make accGradParameters and empty function and use that everywhere.

[Pedro Tabacof]

Thank you, this is exactly what I was looking for!

Pedro.

You received this message because you are subscribed to a topic in the Google Groups "torch7" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/torch7/S8hWQtEIkxg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to torch7+un...@googlegroups.com.

To post to this group, send email to tor...@googlegroups.com.
Visit this group at http://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

--

Pedro Tabacof

[dai...@gmail.com]

hi  friends..

i think u should review backpropagation algorithm.

have a dififirent bettween us...

[David Belanger]

This technique works if you want to permanently turn off gradient updates on certain modules, but it doesn't give you a means to turn the updates back on. This might be important in various applications where you train the model stagewise. Does anyone know an extension of the trick described in this thread such that you can reverse it?

[Soumava Roy]

Can You please specify how to use it along with the other nn modules?? How do i compile this layer along with other layers?

Thank You

Soumava Kumar Roy

[David Belanger]

Overall, I recommend not using any call to getParameters() and instead to use separate optimizers for each block of weights. These can be accessed using the :parameters() method. 

Visit this group at https://groups.google.com/group/torch7.

[Soumava Roy]

Thank You for your reply.

Is there a way to integrate the custom made layers with the nn package??

Soumava Kumar Roy

On Tuesday, March 28, 2017 at 5:46:52 AM UTC+5:30, David Belanger wrote:

Overall, I recommend not using any call to getParameters() and instead to use separate optimizers for each block of weights. These can be accessed using the :parameters() method. 

[David Belanger]

You could make a wrapper container that takes a module and implements its same forward-backward functionality, but optionally can execute a noop for accumulating parameter gradients. This would be analogous to the stop_gradient op in tensorflow. 

On Mon, Mar 27, 2017 at 10:01 PM Soumava Roy via torch7 <torch7+APn2wQdawQFGgpwwRSm1SovGh...@googlegroups.com> wrote:

Thank You for your reply.

Is there a way to integrate the custom made layers with the nn package??

Soumava Kumar Roy

On Tuesday, March 28, 2017 at 5:46:52 AM UTC+5:30, David Belanger wrote:

Overall, I recommend not using any call to getParameters() and instead to use separate optimizers for each block of weights. These can be accessed using the :parameters() method.