Training CNN to convert Grey scale image to CIElab/RGB, problem while testing!

[Kshitija Hande]

Hi again,

I'm working on training a ConvNet to automatically colorize grey-scale image to RGB-image, But instead of directly mapping with RGB values, I'm using an intermediate CIELab color space. Using libraries Lasagne+nolearn. Unlike training over MNIST dataset, i have x_train as grey valued image, and y_train as corresponding CIELab image.  The problem is while training: 

Dumping and loading pickle file:

flab = open('mypicklelab.pickle','wb')

with open('mypicklegrey.pickle','wb') as fgrey:

    num=1

    for filename in glob.glob("*.jpg"):

        im=Image.open(filename)

        #data = data.reshape(-1, 1, 28, 28)

        im = im.resize((224,224), PIL.Image.ANTIALIAS)

        rgb=im

        im=im.convert('L')

        imsave('C:/Users/Sonu/Project Codes/grey/grey'+str(num)+'.jpg',im)

        imnp=lasagne.utils.floatX(im)

        pickle.dump(imnp, fgrey)

        lab = color.rgb2lab(rgb)

        imsave('C:/Users/Sonu/Project Codes/lab/lab'+str(num)+'.jpg',lab)

        labnp=lasagne.utils.floatX(lab)

        pickle.dump(labnp, flab)

        num+=1

  

        

flab.close()

 

with open('mypicklegrey.pickle','rb') as fgrey:
   X_train = pickle.load(fgrey)
   print ('data_grey')



with open('mypicklelab.pickle','rb') as flab:
   Y_train = pickle.load(flab)
   print ('data_lab')

Creating NN and fitting:

net1 = NeuralNet(
layers=[('input', layers.InputLayer),
        ('conv2d1', layers.Conv2DLayer),
        ('maxpool1', layers.MaxPool2DLayer),
        ('conv2d2', layers.Conv2DLayer),
        ('maxpool2', layers.MaxPool2DLayer),
        ('dropout1', layers.DropoutLayer),
        ('dense', layers.DenseLayer),
        ('dropout2', layers.DropoutLayer),
        ('output', layers.DenseLayer),

        ],
# input layer
input_shape=(None, 1, 224,224),
# layer conv2d1
conv2d1_num_filters=32,
conv2d1_filter_size=(3, 3),
conv2d1_nonlinearity=lasagne.nonlinearities.rectify,
conv2d1_W=lasagne.init.GlorotUniform(),  
# layer maxpool1
maxpool1_pool_size=(2, 2),    
# layer conv2d2
conv2d2_num_filters=32,
conv2d2_filter_size=(3, 3),
conv2d2_nonlinearity=lasagne.nonlinearities.rectify,
# layer maxpool2
maxpool2_pool_size=(2, 2),
# dropout1
dropout1_p=0.5,  

# dense
dense_num_units=256,
dense_nonlinearity=lasagne.nonlinearities.rectify,    
# dropout2
dropout2_p=0.5,

# output
output_nonlinearity=lasagne.nonlinearities.softmax,
output_num_units=10,
# optimization method params
update=nesterov_momentum,
update_learning_rate=0.01,
update_momentum=0.9,
max_epochs=10,

verbose=1, )

net1.fit(X_train, Y_train)

I'm getting an error while fitting------------------------------------------------>

File "C:\Users\Sonu\Anaconda3\lib\site-packages\nolearn\lasagne\base.py", line 544, in fit

    self.train_loop(X, y, epochs=epochs)

  File "C:\Users\Sonu\Anaconda3\lib\site-packages\nolearn\lasagne\base.py", line 554, in train_loop

    X_train, X_valid, y_train, y_valid = self.train_split(X, y, self)

  File "C:\Users\Sonu\Anaconda3\lib\site-packages\nolearn\lasagne\base.py", line 140, in __call__

    kf = StratifiedKFold(y, round(1. / self.eval_size))

  File "C:\Users\Sonu\Anaconda3\lib\site-packages\sklearn\cross_validation.py", line 533, in __init__

    label_test_folds = test_folds[y == label]

IndexError: too many indices for array

I get that this is because there is a problem while conversion on Image object in numpy array, maybe instead of 2D, 1D array is loaded or something like that, but i dont know if that assumption is correct or how to fix that . Please Help!

 

[Jan Schlüter]

You seem to treat this as a classification task. Your network output layer has 10 softmax units, and the train/validation split by nolearn tries to split by class. Make sure you treat this as a regression task. Take your time, read every line of code in the script you copied and look up what it does exactly. Do not try to rush and guess your way through it, this will most likely not work.

[Jan Schlüter]

Make sure you treat this as a regression task.

And also make sure the network output shape matches the shape of your targets. You want the network to output an image of three channels and the same size as the input, not a vector of 10 values. Have a look at some image colorization papers to get an idea of how to design the network architecture.

[Kshitija Hande]

I'll look into it. Thank You.