Which loss Function should i use?

[André Lopes]

I changed my neural network structure!

Im not classifying the output, i just want the accuracy to the real output from the yLabel.

I read over the docs but its still very confusing to me!

Would anyone point me to the right path?

name            size        total    cap.Y    cap.X    cov.Y    cov.X

--------------  --------  -------  -------  -------  -------  -------

InputLayer      1x19x19       361   100.00   100.00   100.00   100.00

Conv2DLayer     20x23x23    10580   100.00   100.00    15.79    15.79

Conv2DLayer     20x23x23    10580    60.00    60.00    26.32    26.32

MaxPool2DLayer  20x11x11     2420    60.00    60.00    26.32    26.32

Conv2DLayer     40x13x13     6760    66.67    66.67    47.37    47.37

Conv2DLayer     40x13x13     6760    46.15    46.15    68.42    68.42

MaxPool2DLayer  40x6x6       1440    46.15    46.15    68.42    68.42

Conv2DLayer     80x6x6       2880    57.14    57.14   110.53   110.53

Conv2DLayer     80x6x6       2880    41.38    41.38   152.63   152.63

MaxPool2DLayer  80x3x3        720    41.38    41.38   152.63   152.63

Conv2DLayer     160x3x3      1440    53.33    53.33   236.84   236.84

DropoutLayer    160x3x3      1440   100.00   100.00   100.00   100.00

DenseLayer      2056         2056   100.00   100.00   100.00   100.00

DropoutLayer    2056         2056   100.00   100.00   100.00   100.00

DenseLayer      128           128   100.00   100.00   100.00   100.00

DropoutLayer    128           128   100.00   100.00   100.00   100.00

DenseLayer      1               1   100.00   100.00   100.00   100.00

The loss function currently is :

self.loss = lasagne.objectives.categorical_crossentropy(self.prediction, self.target_var)

self.loss = self.loss.mean()

self.params = lasagne.layers.get_all_params(self.network, trainable=True)

self.updates = lasagne.updates.nesterov_momentum(self.loss, self.params, learning_rate=learning_rate, momentum=momentum)

self.test_prediction = lasagne.layers.get_output(self.network, deterministic=True)

self.test_loss = lasagne.objectives.categorical_crossentropy(self.test_prediction, self.target_var)
self.test_loss = self.test_loss.mean()

# As a bonus, also create an expression for the classification accuracy:
self.test_acc = T.mean(T.eq(T.argmax(self.test_prediction, axis=1), self.target_var),
                       dtype=theano.config.floatX)

# Compile a function performing a training step on a mini-batch (by giving
# the updates dictionary) and returning the corresponding training loss:
self.train_fn = theano.function([self.input_var, self.target_var], self.loss, updates=self.updates)

# Compile a second function computing the validation loss and accuracy:
self.val_fn = theano.function([self.input_var, self.target_var], [self.test_loss, self.test_acc])

[goo...@jan-schlueter.de]

I don't know what your question is, but maybe lasagne.objectives.squared_error() is the answer.

[André Lopes]

When i do that, i get this error :

Starting training...

Traceback (most recent call last):

  File "ConvNet_Script.py", line 206, in <module>

    main()

  File "ConvNet_Script.py", line 175, in main

    save_weights_every_epoch=save_every_weights, folder=folder, load_weights=load_weights)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/trios/classifiers/ConvNet/ConvNet.py", line 279, in train

    train_err += self.train_fn(inputs, targets)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 871, in __call__

    storage_map=getattr(self.fn, 'storage_map', None))

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/theano/gof/link.py", line 314, in raise_with_op

    reraise(exc_type, exc_value, exc_trace)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 859, in __call__

    outputs = self.fn()

ValueError: Input dimension mis-match. (input[0].shape[1] = 1, input[1].shape[1] = 1000)

Apply node that caused the error: Elemwise{sub,no_inplace}(HostFromGpu.0, InplaceDimShuffle{x,0}.0)

Toposort index: 208

Inputs types: [TensorType(float32, matrix), TensorType(int32, row)]

Inputs shapes: [(1000, 1), (1, 1000)]

Inputs strides: [(4, 4), (4000, 4)]

Inputs values: ['not shown', 'not shown']

Outputs clients: [[Elemwise{Composite{Cast{float32}(((i0 * i1) / i2))}}(TensorConstant{(1, 1) of 2.0}, Elemwise{sub,no_inplace}.0, Elemwise{mul,no_inplace}.0), Elemwise{Sqr}[(0, 0)](Elemwise{sub,no_inplace}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):

  File "ConvNet_Script.py", line 206, in <module>

    main()

  File "ConvNet_Script.py", line 171, in main

    batch_size=batch_size)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/ConvNet/ConvNet.py", line 232, in init

    self.loss = lasagne.objectives.squared_error(self.prediction, self.target_var)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/lasagne/objectives.py", line 169, in squared_error

    return (a - b)**2

-------------

# Lasagne Function
self.prediction = lasagne.layers.get_output(self.network)

# create loss function
# Create a loss expression for training, i.e., a scalar objective we want
# to minimize (for our multi-class problem, it is the cross-entropy loss):
#self.loss = lasagne.objectives.categorical_crossentropy(self.prediction, self.target_var)
self.loss = lasagne.objectives.squared_error(self.prediction, self.target_var)

self.loss = self.loss.mean()

self.params = lasagne.layers.get_all_params(self.network, trainable=True)

self.updates = lasagne.updates.nesterov_momentum(self.loss, self.params, learning_rate=learning_rate, momentum=momentum)

self.test_prediction = lasagne.layers.get_output(self.network, deterministic=True)

self.test_loss = lasagne.objectives.squared_error(self.prediction, self.target_var)

self.test_loss = self.test_loss.mean()

# As a bonus, also create an expression for the classification accuracy:
self.test_acc = T.mean(T.eq(T.argmax(self.test_prediction, axis=1), self.target_var),
                       dtype=theano.config.floatX)

# Compile a function performing a training step on a mini-batch (by giving
# the updates dictionary) and returning the corresponding training loss:
self.train_fn = theano.function([self.input_var, self.target_var], self.loss, updates=self.updates)

# Compile a second function computing the validation loss and accuracy:
self.val_fn = theano.function([self.input_var, self.target_var], [self.test_loss, self.test_acc])

2016-03-24 14:39 GMT-03:00 <goo...@jan-schlueter.de>:

I don't know what your question is, but maybe lasagne.objectives.squared_error() is the answer.

--
You received this message because you are subscribed to a topic in the Google Groups "lasagne-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lasagne-users/fBtOHc33svM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lasagne-user...@googlegroups.com.
To post to this group, send email to lasagn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lasagne-users/650ea84b-9fbf-4a73-b0a7-edf17b0f9dd1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[André Lopes]

I think its almost the same problem that i bothered you before Jan! 

But its very confusing to me, because i replicated the model on Keras and, well , since its automatic, it worked fine there with MSE.

However, here lasagne complains about the Tensors i guess.

Im not sure what are the requisites for each change.

I will build a monument on your honour if you can help me !

I sweaaaaar!

[goo...@jan-schlueter.de]

Inputs shapes: [(1000, 1), (1, 1000)]

From this line you see that some tensor is a column matrix while the other is a row matrix (or, more likely, it was a 1-dimensional vector that is broadcasted to a row matrix). With this information, go through your code and compare the output shape of the network with the dimensionality and shape of your target variable. You can figure it out by yourself! I don't want any monument.

Best, Jan

[André Lopes]

Hi Jan, thanks for the reply, i will leave here my diary_attempt.

Im not sure i can still get help here, but..

The output is 1 sigmoid neuron.

If the variable will receive the y_train with the  shape of (batchsizes) , it would be a icol, however, when i do that it complains.

However when i use :

input_var = T.tensor4('inputs')
target_var = T.ivector('targets')

And use batch_size = 1, it works.

According to the list of tensors :

https://theano.readthedocs.org/en/rel-0.6rc3/library/tensor/basic.html

ivector

int32

1

(?,)

(False,)

Which, in my logic, means it just receives the output value.

So i would need a (?,?) Tensor. For the batch_size and output.

--

Which should be:

imatrix

int32

2

(?,?)

(False, False)

TypeError: ('Bad input argument to theano function with name "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/ConvNet/ConvNet.py:253"  at index 1(0-based)',

 'Wrong number of dimensions: expected 2, got 1 with shape (1,).')

Even with batch_size equals to 100, same error : 'Wrong number of dimensions: expected 2, got 1 with shape (100,).')

So if with the batch > 1 , it receives a shape(?,), then the ivector should be enough.

The Tensor4 is supposed to be correct, since im properly inputing a 4shape data.

But then again, when i use ivector it complains  : ValueError: Input dimension mis-match. (input[0].shape[1] = 1, input[1].shape[1] = 100)

Which means its feeding a 1,batchsize shape array.

..Error of when i use ivector : 

ValueError: Input dimension mis-match. (input[0].shape[1] = 1, input[1].shape[1] = 100)

Apply node that caused the error: Elemwise{sub,no_inplace}(HostFromGpu.0, InplaceDimShuffle{x,0}.0)

Toposort index: 208

Inputs types: [TensorType(float32, matrix), TensorType(int32, row)]

Inputs shapes: [(100, 1), (1, 100)]

Inputs strides: [(4, 4), (400, 4)]

Inputs values: ['not shown', 'not shown']

Outputs clients: [[Elemwise{Composite{Cast{float32}(((i0 * i1) / i2))}}(TensorConstant{(1, 1) of 2.0}, Elemwise{sub,no_inplace}.0, Elemwise{mul,no_inplace}.0), Elemwise{Sqr}[(0, 0)](Elemwise{sub,no_inplace}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):

  File "ConvNet_Script.py", line 207, in <module>

    main()

  File "ConvNet_Script.py", line 175, in main

    batch_size=batch_size)

  File "/home/andrelopes/private/Projeto/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/ConvNet/ConvNet.py", line 235, in init

    self.loss = lasagne.objectives.squared_error(self.prediction, self.target_var)

  File "/home/andrelopes/private/Projeto/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/lasagne/objectives.py", line 169, in squared_error

    return (a - b)**2

--

You received this message because you are subscribed to a topic in the Google Groups "lasagne-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lasagne-users/fBtOHc33svM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lasagne-user...@googlegroups.com.
To post to this group, send email to lasagn...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lasagne-users/7de75f3d-2f91-4e6f-a65d-87a388377415%40googlegroups.com.

[goo...@jan-schlueter.de]

Please read my email again and then answer these simple questions:
What is the output shape of your network?
What is the shape of your target variable?
Does squared_error() require them to be the same, according to the documentation?

[André Lopes]

Hi Jan, thanks for the patience! 😇

What is the output shape of your network?

Output (None, 1)

So i suppose (batch_size,1) ?

What is the shape of your target variable?

ivector  int32    1   (?,)     (False,)

1 dimension

Does squared_error() require them to be the same?

Its not explicit on the documentation but , yes it has to be the same. Or it wouldnt make sense.

-----

So the tensor for target should be ?,1 ...

icol int32 2 (?,1) (False, True)

But when i do that the error changes, which is a good thing?

--------------------------

Traceback (most recent call last):

  File "ConvNet_Script.py", line 200, in <module>

    main()

  File "ConvNet_Script.py", line 173, in main

    save_weights_every_epoch=save_every_weights, folder=folder, load_weights=load_weights)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/ConvNet/ConvNet.py", line 275, in train

    train_err += self.train_fn(inputs, targets)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/theano/compile/function_module.py", line 786, in __call__

    allow_downcast=s.allow_downcast)

  File "/home/andrelopes/private/ProjetoMestrado/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/theano/tensor/type.py", line 177, in filter

    data.shape))

TypeError: ('Bad input argument to theano function with name "/home/andrelopes/private/Projeto/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/ConvNet/ConvNet.py:247"  at index 1(0-based)', 

'Wrong number of dimensions: expected 2, got 1 with shape (10,).')

10 is the batch_size i used.

Do i have to change anything on the rest of the code? Seems like it, but i cant see what i could change that it wouldnt affect the tensors

--

You received this message because you are subscribed to a topic in the Google Groups "lasagne-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lasagne-users/fBtOHc33svM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lasagne-user...@googlegroups.com.
To post to this group, send email to lasagn...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lasagne-users/1b7c114a-75cd-48c4-b086-409ea3b2c2dc%40googlegroups.com.

[goo...@jan-schlueter.de]

Carefully read the error message and understand what each part means. Do not try to guess what to do just because you get an error, the message actually tells you everything you need to know when you learn how to read it.

[André Lopes]

This makes no sense to me. If its an icol, its supposed to already be 2 shapes. ? for the batch_size and 1 because it is a COL !

Im not trying to guess, or well, i am, but i am trying to understand.

2016-03-25 12:43 GMT-03:00 <goo...@jan-schlueter.de>:

Carefully read the error message and understand what each part means. Do not try to guess what to do just because you get an error, the message actually tells you everything you need to know when you learn how to read it.

--

You received this message because you are subscribed to a topic in the Google Groups "lasagne-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lasagne-users/fBtOHc33svM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lasagne-user...@googlegroups.com.
To post to this group, send email to lasagn...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lasagne-users/d60843bf-1b43-4869-9e2a-8e76933026ef%40googlegroups.com.

[André Lopes]

Sorry jan, but im definetly stuck. I have no idea of what to do or which logic-thought i should have to solve this.

I really thought it would be the tensor problem, but it seems... well i dont even know anymore.

But thanks for the help anyway.

And sorry for the trouble.

[André Lopes]

Believe it or not, i woke up at 3am and had this weird idea...

# We iterate over epochs:
for epoch in range(0, max_epochs):
    # In each epoch, we do a full pass over the training data:
    train_err = 0
    train_batches = 0
    epoch_start_time = time.time()

    for batch in self.iterate_minibatches(self.x_train, self.y_train, self.batch_size, shuffle=False):
        inputs, targets = batch
        targets = targets.reshape(-1,1)
        train_err += self.train_fn(inputs, targets)
        train_batches += 1

    # And a full pass over the validation data:
    val_err = 0
    val_acc = 0
    val_batches = 0
    for batch in self.iterate_minibatches(self.x_validation, self.y_validation, self.batch_size, shuffle=False):
        inputs, targets = batch
        targets = targets.reshape(-1,1)
        err, acc = self.val_fn(inputs, targets)
        val_err += err
        val_acc += acc
        val_batches += 1

Seems it worked.
Weird.

PS: Im very sleepy so forgive if i miss-pasted the answer

[André Lopes]

It seems i just need to change this to get the accuracy correctly

# As a bonus, also create an expression for the classification accuracy:
self.test_acc = T.mean(T.eq(T.argmax(self.test_prediction, axis=1), self.target_var), dtype=theano.config.floatX)

Does anything fit for this task, an output that does MSE, regarding accuracy? 

[goo...@jan-schlueter.de]

Seems it worked.

Yes, this was the correct fix. Sorry, I wanted you to learn how to read the error message and figure it out. If you want, we can go through the error message again so you understand. Let's break the error message down a bit. I want you to give the meaning of each of the following parts:

1. Bad input argument to theano function
2. with name "/home/andrelopes/private/Projeto/virtualenv-13.1.2/myVE/local/lib/python2.7/site-packages/ConvNet/ConvNet.py:247"
3. at index 1(0-based)',
4. 'Wrong number of dimensions: expected 2, got 1
5. with shape (10,).')

Please try and write a sentence for each of these parts explaining what it means. Taken together, they actually directly lead to your solution of targets = targets.reshape(-1,1).

Does anything fit for this task, an output that does MSE, regarding accuracy?

Not really. Directly monitoring mean squared error on the validation set is usually enough.

Cheers, Jan

[André Lopes]

Thanks Jan! It really helped to understand!

I feel i know more about lasagne now.

Right now im trying to figure another issue with another convnet structure but, with this knowledge im being able to debug it!

Im not so lost as before.