softmax layer init failed with INF params

[Bing Hsu]

I use a data set of 16GB, and no INF 

But it seems the softmax layer was not initialized correctly. Here is the message:

mlp.yaml

/usr/local/lib/python2.7/site-packages/pylearn2-0.1dev-py2.7.egg/pylearn2/models/mlp.py:40: UserWarning: MLP changing the recursion limit.

  warnings.warn("MLP changing the recursion limit.")

/usr/local/lib/python2.7/site-packages/pylearn2-0.1dev-py2.7.egg/pylearn2/utils/call_check.py:98: UserWarning: training_algorithms.sgd.EpochCounter has been moved to termination_criteria.EpochCounter. This link may be removed on or after October 3, 2013.

  return to_call(**kwargs)

Traceback (most recent call last):

  File "/data/pylearn2/pylearn2/scripts/train.py", line 142, in <module>

    train_obj.main_loop()

  File "/usr/local/lib/python2.7/site-packages/pylearn2-0.1dev-py2.7.egg/pylearn2/train.py", line 112, in main_loop

    self.algorithm.setup(model=self.model, dataset=self.dataset)

  File "/usr/local/lib/python2.7/site-packages/pylearn2-0.1dev-py2.7.egg/pylearn2/training_algorithms/sgd.py", line 131, in setup

    raise ValueError("These params are Inf: "+str(inf_params))

ValueError: These params are Inf: [softmax_b]

And here is my YAML file:

!obj:pylearn2.train.Train {

    dataset: &train !obj:pylearn2.scripts.mock.io {

        start: 0,

        stop: 200000

    },

    model: !obj:pylearn2.models.mlp.MLP {

        batch_size: 100,

        layers : [

            !obj:pylearn2.models.mlp.RectifiedLinear {

                layer_name: 'h1',

                dim: 400,

                irange: .05,

                max_col_norm: 2.

            },

            !obj:pylearn2.models.mlp.RectifiedLinear {

                layer_name: 'h3',

                dim: 400,

                irange: .05,

                max_col_norm: 2.

            },

            !obj:pylearn2.models.mlp.Softmax {

                layer_name: 'y',

                init_bias_target_marginals: *train,

                irange: .0,

                n_classes: 421

            }

        ],

        nvis: 19654,

    },

    algorithm: !obj:pylearn2.training_algorithms.sgd.SGD {

        learning_rate: .0015,

        init_momentum: .5,

        termination_criterion: !obj:pylearn2.training_algorithms.sgd.EpochCounter {

            "max_epochs": 1500

        },

        cost: !obj:pylearn2.costs.mlp.dropout.Dropout {},

    },

    save_path: "daemlp.pkl",

    save_freq: 1000

}

Is it a bug of pylearn2?

[Ian Goodfellow]

Probably one of your classes never appears in the training data, so when you initialize the softmax biases it sets the bias for that class to -Inf.

If that is indeed what's happening its not exactly a bug per se since it's doing exactly what you asked it to do. You might want to add a Laplace smoothing feature to the softmax bias initializer, so you can ask it to avoid deterministic probabilities by adding pseudocounts to the training data. 

--
 
---
You received this message because you are subscribed to the Google Groups "pylearn-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylearn-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Sent from Gmail Mobile

[Bing Hsu]

Hi Ian,

Thanks for your help. I will check the data again. It should not happen, or it may be a bug when I generate numpy array.

Thank you.

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "pylearn-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylearn-dev/qfLrCPHDzzo/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to pylearn-dev...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Follow Your Interests. Discover Your World.

Bing XU (许冰)

School of Computer Science

Beijing University of Posts and Telecommunications


Tel: +86-152-1080-7047
Skype: antinucleon