Achieving great test accuracy during training, but solver is bad

[James Li]

I'm trying to train a binary classifier for detecting whether something is in an image, and the testing accuracy is very good (>99% and loss ~0.01 by the end). However, when I was trying to use the trained model for predictions (using pycaffe), it consistently miscalssified one of the types.

It would correctly classify one type with high probability (0.90~1.00) and misclassify the other, but with a lower probability (0.55~0.80) so it somehow knows that there are images with a lower chance of containing the object but won't classify it as such.

I appreciate any help on this, so let me know of any possible leads.