Is there a python interface for layer-wise backward?

[Min Li]

Hi,

I'm trying to implement the guided backpropagation (http://arxiv.org/pdf/1412.6806v3.pdf, https://github.com/Aysegul/torch-visbox/blob/master/display.lua#L85). In order to reconstruct a input image w.r.t a certain filter, I need to perform the backward step by step for each layer and zero-out some value at the same time. I'm relatively new to Caffe and trying with python. However, I cannot find this layer-wise backward api. Could some one give me some suggestion on how to do this?

Thanks!

Min

[Etienne Perot]

Hello!

I have done guided backprop in a very hacky way, but it works :

def guided_backprop(net,end):

    #threshold gradients

    keys = net.blobs.keys()

    nkeys = []

    for key in keys:

        nkeys.append(key)

        if(key == end):

            break

    for l in reversed(range(len(nkeys)-1)):

        layer = keys[l]

        next_layer = keys[l+1]

        if(layer == 'data'):

            imdiff = net.backward(start=next_layer)['data']

           

        else:

            relu_diff = net.blobs[next_layer].diff

            guidance = (relu_diff>0)*1

            if('relu' not in next_layer):

                net.backward(start=next_layer,end=layer)

                continue

            net.backward(start=next_layer,end=layer)[layer]

            

            net.blobs[layer].diff[...] = net.blobs[layer].diff * guidance

            

    return imdiff

[volo...@missinglink.ai]

Hello, Etienne Perot.

Could you please explain what is the purpose of this line: net.backward(start=next_layer,end=layer)[layer]? because it looks useless. Also if I am not mistaken

net.backward returns gradients but you are not storing them.

Thanks in advance.

Volodymyr

[Przemek D]

This line performs the backpropagation itself. The [layer] part is useless though, because indeed it extracts some key from net.backward() output without having it stored.
Not storing gradients is not a problem, since they are accessible via net.blobs[key].diff.

[volo...@missinglink.ai]

Thank you for making things more clear. I am trying to implement this algorithm in PyCaffe as well on ResNet-152 and there are issues.
I posted a separate question here . This implementation is not working BTW. 
Also i do not like how the author determines ReLU layers. "relu" in name is not very robust from my point of view.  I tried the following 

relu_layers = [(i, v) for i, v in enumerate(net.layers) if v.type.lower() == "relu"]

but i have issues with net.blobs[layer_name] . Not all layers are present in net.blobs.
I was also thinking about making custom ReLU layer with modified backward() method ( a typical way to implement this algorithm in other frameworks like theano ) but i need assistance on how to implement setup() and forward() as well since i cannot inherit from ReLU layer, only from generic Layer. 

[maay]

Hello Etienne Perot,

some questions to ask

1. How to plot the "imdiff image"(a four dimension numpy.ndarray) like your example.  

    The value of net.backward is all "zero" ...

2. Does the "net" here refer to the "deploy.prototxt" file (for inference)?

Thank you for your time~

Etienne Perot於 2016年4月28日星期四 UTC+8下午4時58分27秒寫道：

[maay]

Hello , 

I set the diff value of the last layer to be one. net.blobs['last_layer'].diff = 1. Now, I can get the diff value (not be all "zero").

But... I have the other questions that my output of "keys = net.blobs.keys()" does not have "ReLU Layer" (just have "data", "conv", "pool" layers). But, there are really "ReLU Layers" in my deploy.prototxt. What's the problem?

and I am confused that what is different between net.backward & net.blobs[layer].diff. Can someone tell me about that?!

Thanks!!

maay於 2017年11月7日星期二 UTC+8下午6時59分26秒寫道：

[Przemek D]

net.blobs does not list layers, but data blobs. ReLU layers are often run in-place (that is, top is the same as bottom) - you will not see any blobs corresponding to them.
A quick cheatsheet:
  net.blobs - OrderedDict containing blob names (as keys) and blob objects (values), equivalent to net._blobs_dict
  net.layer_dict - as above but for layers. Note that this is different than net.layers (LayerVec containing layer objects - compare with net._blobs) and net._layer_names (StringVec containing layer names - compare with net._blob_names) and should be used instead
  net.top_names - OrderedDict binding layer with blobs they produce; layer names are keys and blob names are values
  net.bottom_names - as above but for blobs the layers take as input
  net.params - contains layer parameters (weights, biases etc.) as an OrderedDict with layer names as keys and parameter blobs as values (note: parameters are packed into vectors, since layers typically have more than one parameter blob: weights and biases, for example)

To answer your second question: net.backward is a function that executes backpropagation, and net.blobs[name].diff is a blob containing gradients (note how I rephrased the latter - you access the blobs dict via blob name, not layer name - while it's common to name tops the same as layers that produce them, it is not always the case and can lead to errors).

[maay]

@Przemek D  

Thank for your quick and carefully reply.(Sorry, I just saw it)  It's very helpful for me!

After reading your reply, I renamed the top name the same as layer name on ReLU layers, BatchNorm layers and Scale layers. 

Now, the guided_backprop function can read and do backward of ReLU, BatchNorm and Scale layers. The output image of guided_backprop function seems like successful.

But I wonder if @Etienne Perot's guided_backprop function can also be used correctly on BatchNorm layers and Scale layers.

and where do you find these net access method (net.blobs, net.layer_dict, ...) , does it have official document list for that?

Thanks very much.

Przemek D於 2017年11月8日星期三 UTC+8下午10時13分40秒寫道：

[Przemek D]

pycaffe.py file contains all the descriptions as it binds the Python interface with the C++ backend.

[maay]

Przemek D, Thank you so much. 

Przemek D於 2017年11月16日星期四 UTC+8下午10時13分06秒寫道：