Initialization of weights (Ex22)

[leap...@gmail.com]

Hello,

we have some problems evaluating the objective function. We already tried different initialization approaches (from the lecture), but our objective is always Inf or NaN and our train- and testloss improves at first but then becomes 0.9 (and that's just as good as a constant classification). It seems that some values together with log,exp or division cause Inf,-Inf or 0-values. Do you have some suggestions about the initialization of w and u?

Thank you!

[Maksym Andriushchenko]

hi,

if your weight initialization is fine, then I guess the problem is a numerically unstable implementation of softmax + cross entropy

please check section 4.1 in https://www.deeplearningbook.org/contents/numerical.html which recommends 2 tricks to prevent underflows and overflow.

note that this is exactly the reason why all deep learning frameworks always implement softmax and cross entropy together as a single function, e.g. as in https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits

I hope that helps.

best,

maksym

--
You received this message because you are subscribed to the Google Groups "Machine Learning WS18/19" group.
To unsubscribe from this group and stop receiving emails from it, send an email to machine-learning-...@googlegroups.com.
To post to this group, send email to machine-lea...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/machine-learning-ws1819/2eae688c-c784-44b5-9cd0-562b8e14ef2d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.