Softmax loss starts and remains on 0 using lmdb data

[Peter Milani]

Hi all,

I'm trying to train a network for image segmentation. I've preprocessed the images and labels, with the labels resized to allow for pooling in the net. The images and labels are saved in lmdb databases for both the training and test sets. I am applying a softmax layer between my label image (with data points 0 to 1, in a 37 x 27 image) with the output of the last convolutional layer.

However the output from the solver is:

 

I0704 12:49:08.439554  1451 solver.cpp:60] Solver scaffolding done.
I0704 12:49:08.440820  1451 solver.cpp:338] Iteration 0, Testing net (#0)
I0704 12:51:25.642366  1451 solver.cpp:406]     Test net output #0: loss = 0 (* 1 = 0 loss)
I0704 12:51:27.548197  1451 solver.cpp:229] Iteration 0, loss = 0
I0704 12:51:27.548260  1451 solver.cpp:245]     Train net output #0: loss = 0 (* 1 = 0 loss)
I0704 12:51:27.548282  1451 sgd_solver.cpp:106] Iteration 0, lr = 0.001
I0704 12:51:41.429247  1451 solver.cpp:229] Iteration 10, loss = 0
I0704 12:51:41.429291  1451 solver.cpp:245]     Train net output #0: loss = 0 (* 1 = 0 loss)
I0704 12:51:41.429306  1451 sgd_solver.cpp:106] Iteration 10, lr = 0.000999251
I0704 12:51:54.940322  1451 solver.cpp:229] Iteration 20, loss = 0
I0704 12:51:54.940362  1451 solver.cpp:245]     Train net output #0: loss = 0 (* 1 = 0 loss)
I0704 12:51:54.940374  1451 sgd_solver.cpp:106] Iteration 20, lr = 0.000998503

which means there must be a problem with the data coming from the lmdb (in that they are both zero arrays) or a problem with the loss layer. The python code I've used to make the lmdb data is as follows:

    mapsize = data_blob.nbytes * 10
    train_data_env = lmdb.open('train_data', map_size= mapsize)
    print("Making Learning examples save in LMDB format")

    with train_data_env.begin(write=True) as txn:
   
        for i in range(0, np.size(trainnp,0)-1):
            train_filename = MODE +"/" + String(trainnp[i][0]) + "_" + String(trainnp[i][1]) + ".tif";
            #print(train_filename)
            with open(FOLDER + "trainexamples.txt", 'a') as myfile:
                myfile.write("{}\n".format(train_filename))
           
            img = cv2.imread(FOLDER + train_filename)
            imggrey = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
            data = np.zeros((1, 420, 580))
            data[0] = imggrey
                 
            datum = caffe.proto.caffe_pb2.Datum()
            datum.channels = 1
            datum.height = data.shape[1]
            datum.width = data.shape[2]
            datum.data = data.tostring()
                       
            str_id = '{:08}'.format(i)
           
            txn.put(str_id.encode('ascii'), datum.SerializeToString())

Can anyone suggest any issues with how I've encoded my data in lmdb? Its mostly a cut and paste from examples online

cheers

Peter

[Peter Milani]

Edit:

I've been able to display the blobs from the lmdb and they are correct. I'm not sure how optimised softmax with loss to compare two images pixelwise and output a valid loss value.