Calculating Cross Entropy without Softmax?

[philip....@gmail.com]

Hi there,

Maybe I am missing something, but,

is there a way to calculate cross_entropy without softmax or sigmoid?

Suppose that I want to calculate softmax of the last layer first, get the probability distribution of the words in vocabulary and I want to interpolate that probability with another probability.

After that I want to calculate the loss of those probability.

Thanks!

[Kenta Oono]

Hi

As far as I remember, we do not have cross entropy loss Function, whose inputs

can be interpreted as a probability distribution.

Although it has an overhead slightly, we can calculate independently the probability distribution

and the cross entropy from the output of the last layer with F.softmax() and F.softmax_cross_entropy(), respectively.

Best

Kenta

2016年4月7日木曜日 14時53分22秒 UTC+9 philip....@gmail.com:

[Philip Arthur]

Thanks for the reply.

Yes I think it makes sense. Do you think I should do F.log first so that the number being calculated in softmax_cross_entropy function is still in the log probability space?

I hope that in the future, chainer will support the standard cross entropy function :)

--

Philip Arthur, Ph.D student,
 Augmented Human Communication lab,
 Nara Institute of Science and Technology, Japan
 Email: philip.arthur.om0[at]is.naist.jp