Negetive loss for cross entropy autoencoder

Shaunak De

unread,

Apr 30, 2015, 1:07:30 PM4/30/15

to caffe...@googlegroups.com

I am trying to implement an auto encoder in caffe with a Sigmoid Cross Entropy loss layer. I keep getting negative loss values and the network doesn’t converge. Any suggestions?

I0430 10:06:59.373145 14116 net.cpp:217] Network initialization done.
I0430 10:06:59.373167 14116 net.cpp:218] Memory required for data: 330964
I0430 10:06:59.373275 14116 solver.cpp:42] Solver scaffolding done.
I0430 10:06:59.373358 14116 solver.cpp:222] Solving WineNetAE
I0430 10:06:59.373390 14116 solver.cpp:223] Learning Rate Policy: fixed
I0430 10:06:59.373455 14116 solver.cpp:266] Iteration 0, Testing net (#0)
I0430 10:07:00.328730 14116 solver.cpp:302] Test loss: -806.359
I0430 10:07:00.328963 14116 solver.cpp:315]     Test net output #0: cross_entropy_loss = -806.359 (* 1 = -806.359 loss)
I0430 10:07:00.329018 14116 solver.cpp:315]     Test net output #1: l2_error = 349954
I0430 10:07:00.543859 14116 solver.cpp:189] Iteration 0, loss = -677.362
I0430 10:07:00.544093 14116 solver.cpp:204]     Train net output #0: cross_entropy_loss = -677.362 (* 1 = -677.362 loss)
I0430 10:07:00.544142 14116 solver.cpp:204]     Train net output #1: l2_error = 331259
I0430 10:07:00.544180 14116 solver.cpp:697] Iteration 0, lr = 0.01

Genevieve Patterson

unread,

Feb 19, 2016, 9:03:53 PM2/19/16

to Caffe Users

Did you ever figure out this problem? I am having the same issue right now.

Dolev Raviv

unread,

Jun 2, 2016, 9:05:18 AM6/2/16

to Caffe Users

Same problem...

Any suggestions?

Clint Sebastian

unread,

Jun 2, 2016, 10:44:55 AM6/2/16

to Caffe Users

Hi Shaunak,

The Sigmoid Cross Entropy loss layer calculates the Cross-Entropy between the output pixel values and the groundtruth pixel values.

Check this.

https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/

Note that natural log of values greater than 1 is positive and values less than 1 is negative.

The net output layer most likely produces a value greater than 1. Putting this value in the Cross-Entropy formula yields a negative error.

Hence using Cross Entropy loss layer with softmax will work well as it produces probabilities as the output (less than 1, hence positive error)