triplet loss and finetuning shape mismatch

[Peter Neher]

Hi everyone. I want to create a triplet network N3 from a previously trained normal image classification network N1. N1 takes 1 channel inputs and N3 takes 3 channel inputs. The 3 channel input of N3 is converted into three 1 channel inputs using the Slice layer. After that, each of the three subnetworks has the same structure as N1 (with removed classification layer). One of the subnetworks has the same layer names as N1 to copy the weights from the trained N1 model and the wights are shared between all three subnets. The outputs of the three subnetworks are fed into the triplet loss layer.
If I start training, I get the following error:

I1119 13:17:03.685039 17617 caffe.cpp:121] Finetuning from model.caffemodel
F1119 13:17:03.740715 17617 blob.cpp:455] Check failed: ShapeEquals(proto) shape mismatch (reshape not set)
*** Check failure stack trace: ***
@     0x7f786196bdaa  (unknown)
@     0x7f786196bce4  (unknown)
@     0x7f786196b6e6  (unknown)
@     0x7f786196e687  (unknown)
@     0x7f7861debc72  caffe::Blob<>::FromProto()
@     0x7f7861cf35de  caffe::Net<>::CopyTrainedLayersFrom()
@     0x7f7861cfbcd2  caffe::Net<>::CopyTrainedLayersFrom()
@           0x40a374  CopyLayers()
@           0x40ab50  train()
@           0x4086d1  main
@     0x7f7860c1fec5  (unknown)
@           0x408cfb  (unknown)
@              (nil)  (unknown)

Any hints what might go wrong here?

Maybe the slicing? The slice layer looks like this:

layer {
  name: "slice_triplet"
  type: "Slice"
  bottom: "triplet_data"
  top: "data_A"
  top: "data_P"
  top: "data_N"
  slice_param {
    slice_dim: 1
    slice_point: 1
    slice_point: 2
  }
}

The outputs data_A, data_P and data_N should have 1 channel each and serve as input to each of the subnetworks that have the same structure as N1.

Cheers,
Peter

[chenfsjz]

I think your slice method is correct, you must be sure that your bottom data has 3 channels,
then in the slice layer, triplet_data be sliced along the channel.I havThe input data to your tri_data layer should be like this: num:64 channels:3 height:224 width:224, and the lmdb like data input your data_layer
should be constructed with 3 channels,then you can slice this triplet_data into three 1channel separate  e converted the cifar10 data set into 9 channels tri_data,
because every image in cifar10 has 3 channels. you can check your trip_data whether has a correct channels numbers.

[Peter Neher]

I think I found the issue. I am using nvidia digits for training and for some reason digits replaced my "num_output: 256" in the fully connected layers by "num_output: 2". I will verify this by using caffe directly to finetune.

[chenfsjz]

I just encounter with another problem,In my net,I also use the triplet to training, I have converted the cifar10 dataset to two file,one includes the anchor file and other includes the sim/dis images for the anchor. Then I convert them to the lmdb data .

When I begin to training, the initialization  of the net seems ok, but when it start do do testing iteration before training,there comes the error, it is suspending, how is your net going on.I think that may be there is something wrong in my lmdb file,but I just can't find the problem.

    I1119 18:29:24.176877  7445 net.cpp:80] Setting up loss_p
    I1119 18:29:24.176889  7445 net.cpp:83] Top shape: 1 1 1 1 (1)
    I1119 18:29:24.176899  7445 net.cpp:90] Memory required for data: 25171972
    I1119 18:29:24.176909  7445 net.cpp:126] loss_p needs backward computation.
    I1119 18:29:24.176919  7445 net.cpp:157] This network produces output loss_p
    I1119 18:29:24.177197  7445 net.cpp:418] Collecting Learning Rate and Weight Decay.
    I1119 18:29:24.177270  7445 net.cpp:168] Network initialization done.
    I1119 18:29:24.177276  7445 net.cpp:169] Memory required for data: 25171972
    I1119 18:29:24.178000  7445 solver.cpp:53] Solver scaffolding done.
    I1119 18:29:24.178010  7445 solver.cpp:173] Solving 
    I1119 18:29:24.178155  7445 solver.cpp:243] test_net_id: 1
    I1119 18:29:24.178163  7445 solver.cpp:303] Iteration 0, Testing net (#0)
    I1119 18:29:24.178176  7445 net.cpp:612] Copying source layer cifar10
    I1119 18:29:24.178181  7445 net.cpp:613] Ignoring source layer cifar10
    I1119 18:29:24.179338  7445 solver.cpp:314] 10
    I1119 18:29:24.179355  7445 net.cpp:495] FORWARD
    I1119 18:29:24.179361  7445 net.cpp:482] LAYERS: 195
    I1119 18:29:24.179368  7445 net.cpp:455] end: 194
    E1119 18:29:24.179376  7445 net.cpp:458] Forwarding nuswide
    I1119 18:29:24.179574  7445 net.cpp:460] bottomvector:0
    I1119 18:29:24.179587  7445 data_layer_tri.cpp:459] prefetch_data_.count: 

[Peter Neher]

Just to let you know, Digits automatically sets the number of outputs of the last fully connected layers to the number of classes. Usually this makes sense, but since triplet networks don't use classes this approach fails.