What about max_col_norm?

[boni...@gmail.com]

Hi,
probably another dumb question, but in mnist.yaml there's max_col_norm that has a value of 1.9365. How that value comes out?
I have my own dataset, so I tried without it but the training fails so I leave it there, but with that value! :S

Any suggestions?
Sorry for bad english ;)

Marco

[David Warde-Farley]

It's a hyperparameter, like any other. For maxout on MNIST we set it by randomly sampling hyperparameters and keeping the configuration with the lowest validation error.

--
You received this message because you are subscribed to the Google Groups "pylearn-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylearn-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ian Goodfellow]

2014-02-26 10:09 GMT-05:00 <boni...@gmail.com>:

Hi,
probably another dumb question, but in mnist.yaml there's max_col_norm that has a value of 1.9365. How that value comes out?
I have my own dataset, so I tried without it but the training fails so I leave it there, but with that value! :S

Any suggestions?

You can look in the monitor at the size of the norms that happen during training. Try running without max_col_norm, then set max_col_norm to about 80% of the mean norm you get when training without it.

 

Sorry for bad english ;)

Puoi scrivere in italiano se vuoi :) 

[Alex Rothberg]

I have a CNN with 2 convolutional layers, then one fully connected layer (using RectifiedLinear) and a final SoftMax layer.

I have "max_col_norm: 2." set for the fully connected layer.

I am seeing numbers like this pretty consistently:

	valid_h2_col_norms_max: 2.00000166893
	valid_h2_col_norms_mean: 1.99946582317
	valid_h2_col_norms_min: 1.99480116367

It seems bad to me that the min/mean col_norm is sitting at the max allowed value. Should I try increasing the weight decay? Raise the allowed col norm?

[David Warde-Farley]

On Mon, Aug 25, 2014 at 10:46 AM, Alex Rothberg <agrot...@gmail.com> wrote:

I have a CNN with 2 convolutional layers, then one fully connected layer (using RectifiedLinear) and a final SoftMax layer.

I have "max_col_norm: 2." set for the fully connected layer.

I am seeing numbers like this pretty consistently:

	valid_h2_col_norms_max: 2.00000166893
	valid_h2_col_norms_mean: 1.99946582317
	valid_h2_col_norms_min: 1.99480116367

It seems bad to me that the min/mean col_norm is sitting at the max allowed value. Should I try increasing the weight decay? Raise the allowed col norm?

Not necessarily. It could still be learning, but it's being projected back onto the hypersphere after every step. To know for sure you'd probably need to monitor (abs(W_old * W)).sum(axis=0).mean() or something like that.

[Alex Rothberg]

Even if it is still learning (I believe it is), should I be content with the min of norm sitting at the max allowed value? Or should I try tuning something. 

[David Warde-Farley]

These things are too problem-specific to say for certain. I'd probably explore a little bit.

--

You received this message because you are subscribed to the Google Groups "pylearn-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylearn-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sara]

Hi Ian,

I am trying to use your hint on initializing the max_kernal_norm of my mlp.ConvRectifiedLinear layers. 

Should I look at the value of "kernel_norms_mean" at epoch 0 or epoch 1?

It can't run epoch 1 completely and stopped with this error:

 raise Exception("NaN in " + param.name)

Exception: NaN in h0_W

In this case I don't have any other choice other than using %80 of kernel_norms_mean  in the epoch 0. But then at epoch 0 the algorithm hasn't seen any examples yet. Is it correct? 

Thanks,

Sara.