Confusion with deploy and validation

[Kağan İncetan]

Hi,

I am following PoseNet example which is a GoogleNet network consisted of several layers. (train_posenet.prototxt) By using this network I aim to predict positions for input images. I have trained the model with my custom dataset and this example has a testing procedure in which I get the error from validation dataset. (test_posenet.py) My trained network gives promising results from this test script.

However, after that, I troed to deploy for the images that unknown to system but I got higher errors than I got for validation (test_posenet.py) then I decided to deploy the validation images that I input to train_posenet.prototxt (im lmdb format) and I compared the errors from deploy and test_posenet.py and I got very different results. Does that mean there is something wrong in my deployment way (pre-processing) or this sounds reasonable ? Please take a look at my deploy script. (deploy.py) (I use deploy.prototxt from the alexkendall's github)

Besides the question above I have other confusions :

1)

While I am training, I get snapshots for every 5000 iterations and I am realizing that the median error from test_posenet.py is sometimes higher for caffemodels that are iterated more. Does that make sense? Because I expect the more the iterations the less the error. Also corresponding observation is that while training the cost is not always decreasing it usually jumps up then decrease then jumps up again. Is it normal ? I tried to reduce the learning rate but it did not help. Now I am planning to use even bigger dataset. What do you recommend ?

2)

Once I have trained a model with around 400 images and tested for 200 images then I trained it with around 800 images (taken from the same place) and tested with same 200 images and the error was higher with the model that trained with more images.This really confused me. How come can it be possible?

Thanks in advance for any kind help.

Regards

[Kağan İncetan]

Any idea ?

24 Ekim 2017 Salı 10:22:15 UTC+2 tarihinde Kağan İncetan yazdı: