Where is the input of "caffe test" come from?

[Cheng Lin]

As mentioned in caffe tutorial, we can test our trained model with the following command

caffe test -model examples/mnist/lenet_train_test.prototxt -weights examples/mnist/lenet_iter_10000.caffemodel -gpu 0

It does not specify where the input images are.

Some people's tran_val.prototxt may include source of lmdb. Initially I guess that might be the case.

However, even when using deploy.prototxt, 

caffe test -model examples/mnist/deploy.prototxt -weights examples/mnist/lenet_iter_10000.caffemodel -gpu 0

which has nothing pointing to any dataset or image, caffe test still works successfully.

I tried to look into caffe.cpp but did not see any default input path specified.

Could anyone please tell what is going on? Where is the input caffe testing with?

[Przemek D]

Deployment is different from testing, and train_test.prototxt will be different from deploy.prototxt. Deployment means querying the network to get answers about entirely new data (perhaps online) - testing means verifying how well does your net work by querying it on a known test set. caffe test lets you do the second task, while examples/cpp_classification/classification.cpp is an example on how to approach the deployment.

The difference is also in how you supply data for both tasks. For deployment you typically use the input fields, while for testing the approach is much like for training/validation - by a DataLayer and an LMDB database, for example. You could test the example network with something like:

caffe test -model=examples/mnist/lenet_train_test.prototxt -weights=examples/mnist/lenet_iter_10000.caffemodel -gpu=0 -stage="val"

note the last argument which makes Caffe run the validation version of the model (which loads the test database instead of the training one).

Hope this clears things up a bit?

[Cheng Lin]

Thanks for the answer! Seems like the "caffe test" is actually doing validation rather than test, whose input is unseen before. I am probably going to move on to classification.cpp for my further test.

However, I am still curious about why caffe.bin is able to test the network without input specified. I expect some error when assigning deploy.prototxt as the model, since there is no input specified in deploy.prototxt, but caffe.bin runs successfully. Even when I removed all layers but the input layer in deploy.prototxt, caffe test still works.

Any ideas about what is going on?

[Przemek D]

Why do you say that "there is no input specified in deploy.prototxt" when you have the input field? All inputs are equally valid - it does not matter for the rest of the network where does the data come from; be it a DataLayer, DummyDataLayer, input field or whatever - all that matters is that the net has some blob to take data from. Another thing is that Caffe makes no assumptions about what you want to do, i.e. whether it makes sense or not. It only does what you ask it to do. So if you test a network that only has an input field (so its input is at the same time its output, with no operations performed) it does exactly what you asked it to do - personally, I see nothing wrong with that :)

[Cheng Lin]

Well, there is input field in deploy.prototxt, but there is no image/data fed into the input field. 

It seems like I am asking: "Hey, Caffe, could you please test the model for me?", and hand over my model to Caffe.

Since I did not provide any image, I expect Caffe to complain with the lack of test data, or at least end up with anything but success.

We need both model and data to train/test the model, thus my question is where are the images come from when we do caffe test -model ... -weight ... ? We did not assign images.

The sentence in my previous "there is no input specified in deploy.prototxt" is probably misleading, sorry about that. It was talking about train_val.prototxt may have a field of data_param:( source: "...lmdb" ) which assigns the lmdb file as inputs, but deploy.prototxt does not.

Przemek D於 2018年3月1日星期四 UTC+8下午4時26分55秒寫道：

[Przemek D]

It seems like I am asking: "Hey, Caffe, could you please test the model for me?", and hand over my model to Caffe.

Yes exactly - except Caffe does not validate your model, it just runs it as is, assuming you know what you're doing. It's up to the user to make sure that the prototxt does what they want it to.

Best is to remember that (deploy != test) and caffe test requires a prototxt more like the train_val one rather than deploy.

I hope this cleared your doubts?

[Cheng Lin]

Hmmm... not actually.

I understand your point that caffe does not "test" the model, it just runs the model as is. And also caffe test is for train_val not deploy.

However, running the model always needs some input data, otherwise the forward propagation cannot be calculated. For train_val, input data is the lmdb dataset, assigned in the Data layer in train_val.prototxt. Something like:

layer {
  type: "Data"
  ...
  data_param {
    source: "examples/mnist/mnist_train_lmdb"
    ...
  }
}

 There is no question here so far.

A deploy.prototxt probably looks like:

layer {
    name: "data"
    type: "Input"
    top: "data"
    input_param { shape: { dim: 1 dim: 1 dim: 28 dim: 28 } }
}

There is no param pointing to lmdb or other sources that can be fed into the network.

The question comes when I intentionally use deploy.prototxt instead of train_val.prototxt.

Even though I understand that caffe test should be using train_val.prototxt and has nothing to do with deploy.prototxt after your previous explains, it did not explain why caffe test is able to generate some mysterious loss values if I intentionally use deploy.prototxt, which specifies no source in it. How can the forward propagation compute loss without input data specified? If there are actually some inputs, where are they?

Przemek D於 2018年3月1日星期四 UTC+8下午7時52分31秒寫道：

[Przemek D]

The question is: how can your deploy.prototxt even calculate loss? At deployment, by definition, you don't know the label (ground truth) - so there is nothing to compute the loss against. caffe test takes whatever network you give it and returns whatever output it produces... so what does your deploy.prototxt ouput? Would you mind attaching it?

[Cheng Lin]

Sure, please find as attached. deploy.prototxt and caffemodel is the file I used, while caffe_test.log is the output of command.

The command I am using is

./build/tools/caffe.bin test -model=examples/mnist/deploy.prototxt -weights=examples/mnist/lenet_iter_10000.caffemodel -gpu=0

"loss" comes from the name of top of the last layer, really a bad naming though. I should have renamed it to something different.
Nonetheless, whatever it is called, you can see there is nothing related to input data either in file or command, thus I have no idea where those batches and floating points come from.



Przemek D於 2018年3月2日星期五 UTC+8下午3時22分36秒寫道：

[Przemek D]

It's exactly how I'm saying. You test a network with only an input field - there is place for data, but you don't actually load any. You have a blob which gets initialized to 0s and stays the same throughout the test - imagine a batch of images containing nothing but black. Look at the output - for each batch there are 10 numbers, but they are always the same 10. You propagated an empty net 50 times! The outputs are not all zero because of the biases your net has, but they repeat (in batches) because the input is the same all the time - just zeros.

[Cheng Lin]

Thank you so much! Very detailed explanation.Now I know what is going on :)

Wish you a good day!

Przemek D於 2018年3月2日星期五 UTC+8下午8時52分04秒寫道：