strange logging: different loss values at the same iteration

[escorciav]

Hi,

any help about the cause of the following suspicious* log.

I0815 02:28:03.118463 25940 solver.cpp:220] Iteration 38, loss = -18179.8

I0815 02:28:03.118500 25940 solver.cpp:235]     Train net output #0: loss = 23418.4 (* 1 = 23418.4 loss)

The most strange is that the first loss is negative while my loss layer shouldn't produce negative values.

* This is the first time that I saw that loss are different for the same iteration.

[escorciav]

In case that someone has the same problem, I was debugging my network and I found that there is an overflow computing smoothed_loss (first reported loss in the log) at solver:step.

Even if you don't want an average loss, this step will execute. In my case, my loss exploited at some iteration and overflow happened.