image colorization

[Никита Закревский]

I try to do a neural network for image colorization. I get the output from vgg16 and learn CNN to get UV channels. 

train prototxt

name: "color_net"

layer {
  name: "data_layer"
  type: "Data"
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "train_x_lmdb"
    backend: LMDB
    batch_size: 1
  }
  top: "data"
}

layer {
  name: "data_layer1"
  type: "Data"
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "train_y_lmdb"
    backend: LMDB
    batch_size: 1
  }
  top: "data1"
}

layer {
    bottom: "data"
    top: "conv1"
    name: "conv1"
    type: "Convolution"
    convolution_param {
        num_output: 128
        pad: 1
        kernel_size: 3
    }
}

layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}

layer {
    bottom: "conv1"
    top: "conv2"
    name: "conv2"
    type: "Convolution"
    convolution_param {
        num_output: 64
        pad: 1
        kernel_size: 3
    }
}

layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}

layer {
    bottom: "conv2"
    top: "conv3"
    name: "conv3"
    type: "Convolution"
    convolution_param {
        num_output: 2
        pad: 1
        kernel_size: 3
    }
}

layer {
  name: "loss"
  type: "EuclideanLoss"
  bottom: 'data1'
  bottom: 'conv3'
  top: 'loss'
  loss_weight: 1
}

In train_x_lmdb I have hypercolumns. And in train_y_lmdb original UV channels.

solver prototxt

net: "color_net.prototxt"
test_iter: 1
test_interval: 10000000
test_initialization: False
base_lr: 0.0000001
lr_policy: "step"
gamma: 0.316
stepsize: 215000
display: 1
max_iter: 500000
momentum: 0.9
weight_decay: 0.001
snapshot: 1000
snapshot_prefix: ""
solver_mode: CPU
average_loss: 1000

When I try to learn this model it's start to minimize loss, but after 50 iterations loss value stop changing.

What's wrong I do?