Trying to use LogReg to classify images

[Viktor Österberg]

I am implementing the LogReg model to try to classify between two types of symbols.
I have 60 images in total that i refer to in a textfile where i assign each image either an 0 or 1. I use the following net to train a model


-----------------------------------------------------------------
name: "myModel"
layer {
    name: "ImgData"
    type: "ImageData"
    top: "data"
    top: "label"

    image_data_param {
        source: "/home/user/images/test/label.txt"
        batch_size: 1
    }

}
layer {
  name: "ip"
  type: "InnerProduct"
  bottom: "data"
  top: "ip"
  inner_product_param {
    num_output: 2
  }
}
layer {
  name: "prob"
  type: "Softmax"
  bottom: "ip"
  top: "prob"
}
--------------------------------------------------------------------

I train the net using the following solver

net: "myModel.prototxt"
test_iter: 58
test_interval: 58
base_lr: 0.01
lr_policy: "step"
gamma: 0.1
stepsize: 1
display: 10
max_iter: 58
momentum: 0.9
weight_decay: 0.0005
#snapshot: 1
snapshot_prefix: "LogReg"
solver_mode: GPU

However, if I use pycaffe to replace the data with another image thats not in my training set
and run a forward pass, I get the following probabilities REGARDLESS OF MY INPUT.

[[ 0.5  0.5]]

Where am I going wrong?