U-Net Image segmentation won't converge, loss doesn't change significantly

[Marie Nachname]

Hello,
I have an issue with training the U-Net, a fully convolutional network for image segmentation. I updated the net.prototxt used by the authors (http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/) to the latest caffe. I simply can't make it converge no matter how much I play around with the learning rate. At lower learning rate the loss fluctuates somewhere between 0.65 and 0.72. At higher learning rates like 0.1 it just goes up. For momentum and batch size I have usually used batch size 1 and momentum 0.99 as suggested by the U-Net authors. My attempts with making the images smaller and increasing batch size a little didn't affect the loss. I've had the network go through sometimes as many as 50.000 iterations with no cange in loss except incredibly minor twitches when the learning rate was lowered by the step function. The loss changes looked very regular, so I shuffled the hdf5 input, it's more random now but the numbers don't change. Removing "loss_param { ignore_label: 2 }" does not change the loss either.

name: '2dUnet'
force_backward: true
layer {                 top: 'data' top: 'label'  name: 'loaddata'       type: 'HDF5Data'  hdf5_data_param { source: 'trainfileList.txt' batch_size: 1 } include: { phase: TRAIN }}

layer { bottom: 'data'              top: 'd0b'   name: 'conv_d0a-b'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 64 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd0b'               top: 'd0b'   name: 'relu_d0b'       type: 'ReLU' }
layer { bottom: 'd0b'               top: 'd0c'   name: 'conv_d0b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 64 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd0c'               top: 'd0c'   name: 'relu_d0c'       type: 'ReLU' }

layer { bottom: 'd0c'               top: 'd1a'   name: 'pool_d0c-1a'    type: 'Pooling'  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layer { bottom: 'd1a'               top: 'd1b'   name: 'conv_d1a-b'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 128 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd1b'               top: 'd1b'   name: 'relu_d1b'       type: 'ReLU' }
layer { bottom: 'd1b'               top: 'd1c'   name: 'conv_d1b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 128 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd1c'               top: 'd1c'   name: 'relu_d1c'       type: 'ReLU' }

layer { bottom: 'd1c'               top: 'd2a'   name: 'pool_d1c-2a'    type: 'Pooling'  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layer { bottom: 'd2a'               top: 'd2b'   name: 'conv_d2a-b'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 256 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd2b'               top: 'd2b'   name: 'relu_d2b'       type: 'ReLU' }
layer { bottom: 'd2b'               top: 'd2c'   name: 'conv_d2b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 256 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd2c'               top: 'd2c'   name: 'relu_d2c'       type: 'ReLU' }

layer { bottom: 'd2c'               top: 'd3a'   name: 'pool_d2c-3a'    type: 'Pooling'  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layer { bottom: 'd3a'               top: 'd3b'   name: 'conv_d3a-b'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 512 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd3b'               top: 'd3b'   name: 'relu_d3b'       type: 'ReLU' }
layer { bottom: 'd3b'               top: 'd3c'   name: 'conv_d3b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 512 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd3c'               top: 'd3c'   name: 'relu_d3c'       type: 'ReLU' }
layer { bottom: 'd3c'               top: 'd3c'   name: 'dropout_d3c'    type: 'Dropout' dropout_param { dropout_ratio: 0.5 } include: { phase: TRAIN }}

layer { bottom: 'd3c'               top: 'd4a'   name: 'pool_d3c-4a'    type: 'Pooling'  pooling_param { pool: MAX kernel_size: 2 stride: 2 } }
layer { bottom: 'd4a'               top: 'd4b'   name: 'conv_d4a-b'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 1024 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd4b'               top: 'd4b'   name: 'relu_d4b'       type: 'ReLU' }
layer { bottom: 'd4b'               top: 'd4c'   name: 'conv_d4b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 1024 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'd4c'               top: 'd4c'   name: 'relu_d4c'       type: 'ReLU' }
layer { bottom: 'd4c'               top: 'd4c'   name: 'dropout_d4c'    type: 'Dropout' dropout_param { dropout_ratio: 0.5 } include: { phase: TRAIN }}

layer { bottom: 'd4c'               top: 'u3a'   name: 'upconv_d4c_u3a' type: 'Deconvolution' param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 512 pad: 0 kernel_size: 2 stride: 2 weight_filler { type: 'xavier' }} }
layer { bottom: 'u3a'               top: 'u3a'   name: 'relu_u3a'       type: 'ReLU' }
layer { bottom: 'd3c' bottom: 'u3a' top: 'd3cc'  name: 'crop_d3c-d3cc'  type: 'Crop' }
layer { bottom: 'u3a' bottom: 'd3cc' top: 'u3b'  name: 'concat_d3cc_u3a-b'  type: 'Concat' }
layer { bottom: 'u3b'               top: 'u3c'   name: 'conv_u3b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 512 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u3c'               top: 'u3c'   name: 'relu_u3c'       type: 'ReLU' }
layer { bottom: 'u3c'               top: 'u3d'   name: 'conv_u3c-d'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 512 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u3d'               top: 'u3d'   name: 'relu_u3d'       type: 'ReLU' }

layer { bottom: 'u3d'               top: 'u2a'   name: 'upconv_u3d_u2a' type: 'Deconvolution' param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 256 pad: 0 kernel_size: 2 stride: 2 weight_filler { type: 'xavier' }} }
layer { bottom: 'u2a'               top: 'u2a'   name: 'relu_u2a'       type: 'ReLU' }
layer { bottom: 'd2c' bottom: 'u2a' top: 'd2cc'  name: 'crop_d2c-d2cc'  type: 'Crop' }
layer { bottom: 'u2a' bottom: 'd2cc' top: 'u2b'   name: 'concat_d2cc_u2a-b'  type: 'Concat' }
layer { bottom: 'u2b'               top: 'u2c'   name: 'conv_u2b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 256 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u2c'               top: 'u2c'   name: 'relu_u2c'       type: 'ReLU' }
layer { bottom: 'u2c'               top: 'u2d'   name: 'conv_u2c-d'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 256 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u2d'               top: 'u2d'   name: 'relu_u2d'       type: 'ReLU' }

layer { bottom: 'u2d'               top: 'u1a'   name: 'upconv_u2d_u1a' type: 'Deconvolution' param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 128 pad: 0 kernel_size: 2 stride: 2 weight_filler { type: 'xavier' }} }
layer { bottom: 'u1a'               top: 'u1a'   name: 'relu_u1a'       type: 'ReLU' }
layer { bottom: 'd1c' bottom: 'u1a' top: 'd1cc'  name: 'crop_d1c-d1cc'  type: 'Crop' }
layer { bottom: 'u1a' bottom: 'd1cc' top: 'u1b'   name: 'concat_d1cc_u1a-b'  type: 'Concat' }
layer { bottom: 'u1b'               top: 'u1c'   name: 'conv_u1b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 128 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u1c'               top: 'u1c'   name: 'relu_u1c'       type: 'ReLU' }
layer { bottom: 'u1c'               top: 'u1d'   name: 'conv_u1c-d'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 128 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u1d'               top: 'u1d'   name: 'relu_u1d'       type: 'ReLU' }

layer { bottom: 'u1d'               top: 'u0a'   name: 'upconv_u1d_u0a' type: 'Deconvolution' param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 128 pad: 0 kernel_size: 2 stride: 2 weight_filler { type: 'xavier' }} }
layer { bottom: 'u0a'               top: 'u0a'   name: 'relu_u0a'       type: 'ReLU' }
layer { bottom: 'd0c' bottom: 'u0a' top: 'd0cc'  name: 'crop_d0c-d0cc'  type: 'Crop' }
layer { bottom: 'u0a' bottom: 'd0cc' top: 'u0b'   name: 'concat_d0cc_u0a-b'  type: 'Concat' }
layer { bottom: 'u0b'               top: 'u0c'   name: 'conv_u0b-c'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 64 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u0c'               top: 'u0c'   name: 'relu_u0c'       type: 'ReLU' }
layer { bottom: 'u0c'               top: 'u0d'   name: 'conv_u0c-d'     type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { num_output: 64 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layer { bottom: 'u0d'               top: 'u0d'   name: 'relu_u0d'       type: 'ReLU' }

layer { bottom: 'u0d'               top: 'score' name: 'conv_u0d-score' type: 'Convolution'  param {lr_mult: 1 decay_mult: 1} param {lr_mult: 2 decay_mult: 0}  convolution_param { engine: CAFFE num_output: 2 pad: 0 kernel_size: 1 weight_filler { type: 'xavier' }} }

layer { bottom: 'score' bottom: 'label' top: 'loss'  name: 'loss' type: 'SoftmaxWithLoss' loss_param { ignore_label: 2 } include: { phase: TRAIN }}

As dataset I use 85 rgb (purple HE coloration) 430x430 sized images. I have tried a variety of different approaches of preparing the data, most recently I turned them to greyscale and increased my dataset size to 255 by also mirroring the images horizontally and vertically. The label is 260x260 sized as the network architecture does not use padding and the output segmentation should be for middle section of the input. There are two classes background (0) and segmentation (1). I use a hdf5 container for the data and labels. Maybe there is an error in my transformations?

train_dir = 'train/'
data_dir = 'data/'
label_dir = 'label/'

f = h5py.File('train_430_rgm.h5', 'w')
f.create_dataset('data', (85*3, 1, 430, 430), dtype='f8')
f.create_dataset('label', (85*3, 1, 260, 260), dtype='f8')


def gland(x):
    return (0.0 if x == 0 else 1.0)

vgland = np.vectorize(gland)

i = 0
for filename in os.listdir(train_dir+data_dir):
    data = misc.imread(train_dir+data_dir+filename, flatten=True)
    img = data.astype(float) / 255
    img = img[0:430,0:430]
    f['data'][i] = img.transpose()
    f['data'][i+1] = np.fliplr(img.transpose())
    f['data'][i+2] = np.flipud(img.transpose())
    i+=3

i = 0
for filename in os.listdir(train_dir+label_dir):
    img = misc.imread(train_dir+label_dir+filename)
    img = img[85:345,85:345]
    f['label'][i] = vgland(img.transpose())
    f['label'][i+1] = vgland(np.fliplr(img.transpose()))
    f['label'][i+2] = vgland(np.flipud(img.transpose()))
    i+=3

f.close()

I realize it's a small dataset but I thought for a sanity check this is okay. I just want to make sure the network can fit the data at all, no matter how much it will be overfitted. Right now I'm just wondering if training isn't working at all.

One thing I have not done as descripted by the U-Net paper is calculate weights. I don't really see how this is reflected in the network.prototxt.

[Marie Nachname]

up

[Tianyi Miao]

Got the same issue, have you solve the problem? I am dealing with a small dataset and do some augmentation to train the U-NET with update it to new version caffe, but the loss is not decreasing much as you mentioned as well.

在 2017年2月13日星期一 UTC-5上午9:05:54，Marie Nachname写道：

up

[Peter Neher]

I usually set the batch size as large as possible. Also I use padding to get an output the same size as the input. My deconvolution is different since I want it to behave like  simple upsampling without learning weights. This avoids checkerboard patterns. There are maybe some other differences. Check it out here: https://github.com/peterneher/peters-stuff/tree/master/CaffeScripts

Cheers,
Peter

[Daile Osorio]

I have the same problem but in my case I visualized the intermediate layers and these layers are correctly learning my images. However, the Deconvolution layes I see that these are not learning, I am obtaining in theses layers checkerboard patterns. Now, Peter you say that I do not initialize the deconvolucion layers with any weight filler parameters? what you want say when said: "I want it to behave like  simple upsampling without learning weights, This avoids checkerboard patterns"?. In my case I had initialized the deconvolution layers with weight_filler {

      type: "gaussian"

      std: 0.01

    }

Can it be my problem that my loss does not decrease?  

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/da14957a-0a99-40e7-9251-b241c64972d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Neher]

I also tried to train the deconvolution and that also worked. I used an mrsa filler though. Basically the same setup as for the convolution. For a simple upsampling, my deconvolution layers look like this:
      

layer {

name: "upscale_4"

type: "Deconvolution"

bottom: "encode_2"

top: "upscale_4"

param {

lr_mult: 0

decay_mult: 0

}

convolution_param {

num_output: 512

bias_term: false

kernel_size: 2

group: 512

stride: 2

weight_filler {

type: "constant"

value: 1

}

}

}

As you can see I set the learning rate to zero and the weights to one. The group parameter tells the layer to treat the channels independently, meaning that the first input channel is only mapped to the first output channel. This realizes a channel-wise upsampling.

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.

[Przemek D]

Using gaussian initialization for upsampling deconvolution is not a good idea. Caffe has a dedicated solution for that, it's a bilinear filler. See the "detailed description" section, it contains a reference on how to create a deconvolution layer so that it upsamples the input factor times.

[Tianyi Miao]

Hi Daile,

May I know how do you visualize the intermediate layers?

在 2017年7月4日星期二 UTC-4上午10:55:32，Daile Osorio写道：

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.

[Daile Osorio]

For visualizing intermediate  layers you can doing this:

out = solver.net.blobs['name of the layer'].data[0].argmax(0)
img = np.array(out).astype(np.uint8) #obtaining the numpy array in python for visualizing image
img = img > 0
img = img * 255
cv2.imwrite("name.png", img) # save image in a directory

Now, also you can obtaining the weights of these intermediate  layers, 

it is good because you know if your network is learning, i.e you can see if the weights are not zero. The

way to know this is doing:

weights = solver.net.blobs['name layer'].data[0]
print 'name layer'
print np.unique(weights)

Remenber before doing all this you have that load you .caffemodel, i.e you can doing you script.py in the case of pycaffe. For instance:
import sys
import numpy as np
import os
from PIL import Image
import setproctitle
v=setproctitle.setproctitle(os.path.basename(os.getcwd()))


weights = 'path/weights.caffemodel'
print weights
weights = 'path/weights.caffemodel'
caffe.set_mode_cpu()

solver = caffe.SGDSolver('solver.prototxt')
solver.net.copy_from(weights)
val = np.loadtxt('path/namesvalid.txt', dtype=str)
DEBUG_WEIGHTS = 1
if DEBUG_WEIGHTS:
    solver.step(1)

weights = solver.net.blobs['name layer'].data[0]
print 'name layer'
print np.unique(weights)

out = solver.net.blobs['name of the layer'].data[0].argmax(0)
img = np.array(out).astype(np.uint8) #obtaining thne numpy array in python for visualizing image
img = img > 0
img = img * 255
cv2.imwrite("name.png", img) # save image in the directpry

I hope that it help you!!!!

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/dcfa27d3-cc30-4211-bd70-1f7955d409fb%40googlegroups.com.

[Daile Osorio]

Przemek D, to use bilinear filler to initialize the deconvolution layers was perfect !!!! The loss is decreasing I am 

very happy for this, thank for your help, but I selected a generated .caffemodel during the training and the file deploy.prototxt and tried 

doing inference on one of the images of validation (for this I used the infer.py of fcn8s) and I am obtaining this error: 

" Cannot copy param 0 weights from layer 'upscore2_2class'; shape mismatch.  

Source param shape is 2 1 4 4 (32); target param shape is 2 2 4 4 (64). To learn this layer's 

parameters from scratch rather than copying from a saved net, rename the layer."

I think that this is relationed with the definition of the "upscore2_2class" layer (a deconvolution layer) in the deploy.prototxt.

I will mention some aspects that I think are important for you to know on I am doing in the network:

1- I am doing finetune on the Fcn8s network for segmenting images which only I need two output, so, on the fcn8s train.prototxt

I changed the num output of the score and the deconvolution layers, I renamed the names of these layers by others names and I added

the  weight filler bilinear, the group and "decay_mult: 0" into param to the deconvolution layers.

2-In the deploy. prototxt only update the names of the layers that I had renamed.

Here I attached my train. prototxt and my deploy.prototxt for  you  see as I defined each layer for doing finetune

Do you can said me what is the problem ? or What can I do ?

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/54a1d6f7-0c15-439e-9ddb-a32a26137013%40googlegroups.com.

[Tianyi Miao]

Thanks a lot Daile, that helps a lot! Truly Appreciated! 

So after checking upconv and conv layers, it seems network is learning something (it is not just zero). But the loss and accuracy is fluctuating in a small range.

One more thing that I am curious, did they mentioned how many input images are using for training in total? I know the original images are 30, and they did some deformation to increase the training set, but how many do they generated at the end? 

I my case, I just use 150(original+augmentation), does this make sense?

Again, great appreciated!

在 2017年7月5日星期三 UTC-4下午4:26:43，Daile Osorio写道：

[Daile Osorio]

Tianyi  I do not remember how many images they used to train but if you are using for training 150 images in  total with data-augmentation, I think it is ok but I think you should look  other caffe-user responses like making data augmentation on caffe with FCN because I haven´t experience doing data augmentation in CNNs by using Caffe. On the idea of that "the loss and accuracy is fluctuating in a small range ", you can try varying the rate learning and you can analyze the loss in each case.

I hope it help you!!!!!

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/7b30139b-35e9-4e7f-87b6-c7263ebd0213%40googlegroups.com.

[Tianyi Miao]

Thank you Daile! The last thing confuses me is, the output of U-net has two channels (foreground and background as they mentioned), whereas the compared label just has one channel (contain background(0) and foreground(1)), do they mismatch? How should I understand that U-net prediction two channel? and how two reconstruct them into an image?

And I have tried written label into two channels as well(e.g. #1:background(1)+foreground(0),#2:background(0)+foreground(1)), but when put them into network it says mismatch this time...So I am confusing...

How is that possible to compare two channels with on channel..

Thanks a lot!

在 2017年7月6日星期四 UTC-4下午12:23:40，Daile Osorio写道：

[Przemek D]

Classification networks output as many channels as the number of classes in your dataset. So if you have foreground and background, your output will be two channel. The reason for this is that each channel contains probability of this class being assigned to the pixel - they should add up to 1.
The label contains only ground truth. Only one label is the right one, so there is no need for multiple channels - all of them would contain zeros except the right one, which would be a 1; instead, the label blob is a single channel "image" which contains the index of the correct class.