Save output for euclidean loss debugging for regression

[Yoann]

Hi all,

I'm tring to learn a very simple model composed of only two inner product layers. However, when I train the model the loss is very low (~0.01) even during the first testing iteration which is very surprising since I'm trying to infer a high-level concept with values ranging from 0 to 1.

Thus, is it possible to save in a file the output of the prediction during the test iteration in order to understand where is the problem?

Something like:

'data1 label1 predictValue1

data2 label2 predictValue2...'

Anyway, maybe something is wrong with my very simple solver or train_val.prototxt. This problem could also be due to the fact that I use my own dataset that I converted into LMDB style, maybe it can't read the label or something like that.

Thanks in advance for your help! When this issue will be solved it will hopefully allow me to build much more complicated models.

solver.prototxt

net: "models/liris-accede_baseline/train_val.prototxt"

test_iter: 500

test_interval: 500

# The base learning rate, momentum and the weight decay of the network.

base_lr: 0.0001

momentum: 0.9

weight_decay: 0.0005

# The learning rate policy

lr_policy: "step"

gamma: 0.1

stepsize: 100000

# Display every 100 iterations

display: 100

# The maximum number of iterations

max_iter: 500000

# solver mode: CPU or GPU

solver_mode: CPU

train_val.prototxt

name: "lirisbaseline"

layers {

  name: "data"

  type: DATA

  top: "data"

  data_param {

    source: "data/liris-accede/train_data_lmdb"

    backend: LMDB

    batch_size: 32

  }

  transform_param {

    mean_file: "data/liris-accede/train_data_mean.binaryproto"

  }

  include: { phase: TRAIN }

}

layers {

  name: "label_valence_train"

  type: DATA

  top: "label"

  data_param {

    source: "data/liris-accede/train_valence_score_lmdb"

    backend: LMDB

    batch_size: 32

  }

  include: { phase: TRAIN }

}

layers {

  name: "data"

  type: DATA

  top: "data"

  data_param {

    source: "data/liris-accede/test_data_lmdb"

    backend: LMDB

    batch_size: 32

  }

  transform_param {

    mean_file: "data/liris-accede/train_data_mean.binaryproto"

  }

  include: { phase: TEST }

}

layers {

  name: "label_valence_test"

  type: DATA

  top: "label"

  data_param {

    source: "data/liris-accede/test_valence_score_lmdb"

    backend: LMDB

    batch_size: 32

  }

  include: { phase: TEST }

}

layers {

  name: "fc1"

  type: INNER_PRODUCT

  bottom: "data"

  top: "fc1"

  blobs_lr: 1

  blobs_lr: 2

  weight_decay: 1

  weight_decay: 0

  inner_product_param {

    num_output: 32

    weight_filler {

      type: "gaussian"

      std: 0.005

    }

    bias_filler {

      type: "constant"

      value: 0.1

    }

  }

}

layers {

  name: "relu1"

  type: RELU

  bottom: "fc1"

  top: "fc1"

}

layers {

  name: "drop1"

  type: DROPOUT

  bottom: "fc1"

  top: "fc1"

  dropout_param {

    dropout_ratio: 0.5

  }

}

layers {

  name: "fc2"

  type: INNER_PRODUCT

  bottom: "fc1"

  top: "fc2"

  blobs_lr: 1

  blobs_lr: 2

  weight_decay: 1

  weight_decay: 0

  inner_product_param {

    num_output: 1 # number of output from regression task

    weight_filler {

      type: "gaussian"

      std: 0.01

    }

    bias_filler {

      type: "constant"

      value: 0

    }

  }

}

layers {

  name: "mse"

  type: EUCLIDEAN_LOSS

  bottom: "fc2"

  bottom: "label"

  top: "mse"

  include: { phase: TEST }

}

layers {

  name: "loss"

  type: EUCLIDEAN_LOSS

  bottom: "fc2"

  bottom: "label"

  top: "loss"

  include: { phase: TRAIN }

}

[PatWie]

I wrote a layer which simply prints out the values:

https://github.com/BVLC/caffe/issues/1391

The output is

// true_label prob_0 label_0 prob_1 label_1 prob_2 label_2 prob_3 label_3

in your case:

// true_value loss

[Yoann]

Thanks so much! That's what I was looking for!

However, I'm using the master branch and I faced some compilation issues.

For those interested, to make it compile I modified some lines in loss_layers.hpp (const vector<Blob<Dtype>*>& top becomes vector<Blob<Dtype>*>* top) and I commended the line "REGISTER_LAYER_CLASS(TESTSTATISTIC, TestStatisticLayer);" in statistic_layer.cpp since it is not recognized in the master branch. To compensate I've added in layer_factory.cpp these two lines:

  case LayerParameter_LayerType_TESTSTATISTIC:

    return new TestStatisticLayer<Dtype>(param);

[alameen...@gmail.com]

I noticed that you are using a loss layer WITH A TOP as an accuracy layer. Is that right? Can you explain your motivation behind that?

I have similar regression application so any elaboration will be highly appreciated.

thanx 

[Viktor]

Hey can someone explain how to use teststatistic layer?

I tryed Yoanns compiling tips:

For those interested, to make it compile I modified some lines in loss_layers.hpp (const vector<Blob<Dtype>*>& top becomes vector<Blob<Dtype>*>* top) and I commended the line "REGISTER_LAYER_CLASS(TESTSTATISTIC, TestStatisticLayer);" in statistic_layer.cpp since it is not recognized in the master branch. To compensate I've added in layer_factory.cpp these two lines:

  case LayerParameter_LayerType_TESTSTATISTIC:

    return new TestStatisticLayer<Dtype>(param);

but I always get this error:

net: "examples/MyFirstTest/HDF5/60kmh/MinimalOutput/train.prototxt"

I0812 01:27:07.587349  9518 solver.cpp:91] Creating training net from net file: examples/MyFirstTest/HDF5/60kmh/MinimalOutput/train.prototxt

[libprotobuf ERROR google/protobuf/text_format.cc:245] Error parsing text-format caffe.NetParameter: 173:9: Expected integer or identifier.

F0812 01:27:07.588017  9518 upgrade_proto.cpp:79] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: examples/MyFirstTest/HDF5/60kmh/MinimalOutput/train.prototxt

*** Check failure stack trace: ***

    @     0x7f1747c6edaa  (unknown)

    @     0x7f1747c6ece4  (unknown)

    @     0x7f1747c6e6e6  (unknown)

    @     0x7f1747c71687  (unknown)

    @     0x7f1748011c7e  caffe::ReadNetParamsFromTextFileOrDie()

    @     0x7f1748065f1b  caffe::Solver<>::InitTrainNet()

    @     0x7f1748066fec  caffe::Solver<>::Init()

    @     0x7f174806731a  caffe::Solver<>::Solver()

    @     0x7f1748032713  caffe::Creator_SGDSolver<>()

    @           0x40e67e  caffe::SolverRegistry<>::CreateSolver()

    @           0x40794b  train()

    @           0x40590c  main

    @     0x7f1746f7cf45  (unknown)

    @           0x406041  (unknown)

    @              (nil)  (unknown)

Aborted (core dumped)

can someone help me please?

[Deshana Desai]

Hey. It looks like there is an error encountered while parsing your train.prototxt.

More specifically, look at row 173, column 9. It expected some sort of integer or identifier but could not find the same.

If you could attach the train.prototxt here, that would help!