Hi,
I am following PoseNet example which is a GoogleNet network consisted of several layers. (train_posenet.prototxt) By using this network I aim to predict positions for input images. I have trained the model with my custom dataset and this example has a testing procedure in which I get the error from validation dataset. (test_posenet.py) My trained network gives promising results from this test script.
However, after that, I troed to deploy for
the images that unknown to system but I got higher errors than I got for validation (test_posenet.py) then I
decided to deploy the validation images that I input to train_posenet.prototxt (im lmdb format)
and I compared the errors from deploy and test_posenet.py and I got
very different results. Does that mean there is something wrong in my
deployment way (pre-processing) or this sounds reasonable ? Please take a look at my deploy script. (deploy.py) (I
use deploy.prototxt from the alexkendall's github)
Besides the question above I have other confusions :
1)
While
I am training, I get snapshots for every 5000 iterations and I am
realizing that the median error from test_posenet.py is sometimes higher
for caffemodels that are iterated more. Does that make sense? Because I
expect the more the iterations the less the error. Also corresponding
observation is that while training the cost is not always decreasing it
usually jumps up then decrease then jumps up again. Is it normal ? I
tried to reduce the learning rate but it did not help. Now I am planning
to use even bigger dataset. What do you recommend ?
2)
Once
I have trained a model with around 400 images and tested for 200 images
then I trained it with around 800 images (taken from the same place)
and tested with same 200 images and the error was higher with the model
that trained with more images.This really confused me. How come can it
be possible?
Thanks in advance for any kind help.
Regards