Hi, I have recently read the paper "Improving Deep Neural Network Acoustic Models Using Generalized Maxout Networks" which proposed a few variants of maxout activation. It also suggested a normalization layer as in Eq (5) to constraint the stand deviation of the unbounded activation values, as follows:
Input: xi
Output: yi
sigma = \sqrt{1/K * \sum_i (x_i)^2}
if sigma <= 1: yi = xi
else : yi = xi / sigma
I am interested in implementing this in Caffe using the existing layer or activation. For sigma, I can use the POWER and ELTWISE with sum. But I don't quite figure out how to compute yi given xi.
Any suggestion will be appreciated.