How can I run tied weight AutoEncoder ?

Erogol

unread,

Jan 27, 2015, 2:41:06 PM1/27/15

to caffe...@googlegroups.com

I want to run AE with tied weights where encoder has weight matrix W and decoder has its transpose W^t. Is it viable by caffe?

Philip H

unread,

Jan 28, 2015, 5:53:54 AM1/28/15

to caffe...@googlegroups.com

I'd be interested in this problem, too!

Youssef Kashef

unread,

Sep 10, 2015, 1:07:39 PM9/10/15

to Caffe Users

Caffe does allow sharing weights. But it don't think it transposes the weights at the decoder.

Weight sharing is described briefly in the Siamese network tutorial. It does this by assigning a name to the parameter and referencing in all the layers that share it.

Example (Inner product layers ip1 and ip1_p share the same weights ip1_w and biases ip1_b):

...layer definitions...
layer {
name: "ip1"
type: "InnerProduct"
bottom: "pool2"
top: "ip1"
param {
name: "ip1_w"
lr_mult: 1
}
param {
name: "ip1_b"
lr_mult: 2
}
inner_product_param {
num_output: 500
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
...layer definitions...
layer {
name: "ip1_p"
type: "InnerProduct"
bottom: "pool2_p"
top: "ip1_p"
param {
name: "ip1_w"
lr_mult: 1
}
param {
name: "ip1_b"
lr_mult: 2
}
inner_product_param {
num_output: 500
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
...layer definitions...

An idea for transposing the weights in the decoder:

Derive a new layer class that inherits from the inner product layer. You can call it DecodeInnerProductLayer.

DecodeInnerProductLayer can continue doing everyhing the same way as the parent class. It only needs to implement the forward and backward methods.

Both methods start by transposing the weights and then call the parent class implementation of forward() and backward().