Understanding Local Response Normalization (LRN)

[Gil Levi]

Hi,

As an exercise, I'm trying to implement a toy version of LRN in python. 

However, the results I'm getting are very different from Caffe's network results.

Here is my code (which is based on Krizhevsky's paper)

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
caffe_root = '../'  # this file is expected to be in {caffe_root}/examples
import sys
sys.path.insert(0, caffe_root + 'python')
import caffe
# Set the right path to your model definition file, pretrained model weights,
# and the image you would like to classify.
MODEL_FILE = 'imagenet/imagenet_deploy.prototxt'
PRETRAINED = 'imagenet/caffe_reference_imagenet_model'
IMAGE_FILE = 'images/cat.jpg'
net = caffe.Classifier(MODEL_FILE, PRETRAINED,
                       mean=np.load(caffe_root + 'python/caffe/imagenet/ilsvrc_2012_mean.npy'),
                       channel_swap=(2,1,0),
                       raw_scale=255,
                       image_dims=(256, 256))
net.set_phase_test()
net.set_mode_cpu()
input_image = caffe.io.load_image(IMAGE_FILE)

prediction,out_res = net.predict([input_image], oversample=False)

data = net.blobs['pool1'].data[0]
print 'shape of data is {0}'.format(data.shape)
out = net.blobs['norm1'].data[0]
print 'shape of pool1 output is {0}'.format(out.shape)
#trying to reproduce the norm response results
out_e=zeros(shape=out.shape)
data_shape=data.shape
k=2
alpha=0.0001
beta=0.75
n=5

for filter_ind in range(data_shape[0]):
    for y in range(data_shape[1]):
        for x in range(data_shape[2]):
            
            filter_start=max(0,filter_ind-n/2)
            filter_end=min(data_shape[0]-1,filter_ind+n/2)
            
            
            cur_sum=0;
            for j in range(filter_start,filter_end):
                cur_sum+=(data[j,y,x]**2)
                
            final_res= data[filter_ind,y,x] /((k + alpha*cur_sum)**beta)
            
            out_e[filter_ind,y,x]=final_res

The correlation between my out (the LRN layer's result) and out_e (my estimation) is only about 91%.

I tried to play with the code a bit, especially with the parameters, but without any luck.

Can someone please point out where am I wrong?

Thanks,

Gil

[Gil Levi]

OK, I figured it out.

It should be :

   final_res= data[filter_ind,y,x] /((1.0 + alpha/n*cur_sum)**beta

Also, I think I found a bug in lrn_layer.cpp:

https://github.com/BVLC/caffe/blob/master/src/caffe/layers/lrn_layer.cpp

Line 120:  scale_data[i] = 1.;

Shouldn't it be scale_data[i]=k ? it seems as though the normalization code does not use the "k" parameter at all and just replaces it with 1.0.

Should I post this as an issue on Caffe's github?

Thanks,

Gil

[Evan Shelhamer]

If after spotting and double-checking what might be a bug and still being convinced it is worth filing an issue on github to report it for confirming and fixing if need be.

Thanks for taking a close look.

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/8372b805-5714-4920-85f2-c2343f8c8d62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Evan Shelhamer

[Gil Levi]

Hi Evan,

When taking a closer look, I noticed that the "k" parameter also does not appear in the net's prototxt file, so perhaps this is just an implementation decision.

Gil. 

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/8372b805-5714-4920-85f2-c2343f8c8d62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Evan Shelhamer