Usage of MVN layer? (parameters)

1,088 views
Skip to first unread message

Hyo Jin Kim

unread,
Oct 7, 2015, 3:09:44 AM10/7/15
to Caffe Users
Could someone show me a usage of MVN layer?
What parameters does MVN take and what are their default values? 

Hyo Jin Kim

unread,
Oct 7, 2015, 3:16:11 AM10/7/15
to Caffe Users
I found this:

mvn_param{across_channels: true}
mvn_param{normalize_variance: false} 

I guess I shouldn't put anything for a default setting? (across_channels: false & normalize_variance: true)

Suraj Sheth

unread,
Oct 29, 2015, 3:27:35 PM10/29/15
to Caffe Users
Did you figure out the usage. I tried all combinations, but, all the outputs of MVn layer are ZEROs. Any idean on how to fix this will be helpful.

Dai Yan

unread,
Oct 30, 2015, 10:43:12 AM10/30/15
to Caffe Users
Jim, it seems that I also encounter similar issue, mvn always ouput zero.

Have you figured out what's the problem?

在 2015年10月7日星期三 UTC+8下午3:16:11，Hyo Jin Kim写道：

SURAJ SHETH

unread,
Oct 30, 2015, 12:12:17 PM10/30/15
to Dai Yan, Caffe Users
An update :
Since the input has only two dimensions, with the second dimension representing the number of channels and the first dimension representing the batch size, setting across_channels : true produces non-zero output.

But, I am facing a few new problems :
1) The mean normalization works properly only on train iterations and the test iteration output of the layer is not      Zero-mean
2) Most of the times, the layer produces nan (s) as the output while the input is valid. Something wrong with the layer

Any help will be appreciated.

Here are the logs. The net is present below.
I1030 16:08:10.961153 110931 solver.cpp:226] Learning Rate Policy: step
I1030 16:08:10.961176 110931 solver.cpp:269] Iteration 0, Testing net (#0)
I1030 16:08:11.151047 110931 solver.cpp:318]     Test net output #0: input_mvn = -1.91535
I1030 16:08:11.151131 110931 solver.cpp:318]     Test net output #1: input_mvn = 0.237822
I1030 16:08:11.151146 110931 solver.cpp:318]     Test net output #2: input_mvn = 1.41848
I1030 16:08:11.151160 110931 solver.cpp:318]     Test net output #3: label = 0.1
I1030 16:08:11.151181 110931 solver.cpp:318]     Test net output #4: output_mvn = nan
I1030 16:08:11.151216 110931 solver.cpp:318]     Test net output #5: output_mvn = nan
I1030 16:08:11.151229 110931 solver.cpp:318]     Test net output #6: output_mvn = nan
I1030 16:08:11.164382 110931 solver.cpp:189] Iteration 0, loss = 0
I1030 16:08:11.164433 110931 solver.cpp:204]     Train net output #0: input_mvn = -1.93058
I1030 16:08:11.164470 110931 solver.cpp:204]     Train net output #1: input_mvn = -2.16308
I1030 16:08:11.164485 110931 solver.cpp:204]     Train net output #2: input_mvn = 3.98105
I1030 16:08:11.164523 110931 solver.cpp:204]     Train net output #3: label = 0
I1030 16:08:11.164563 110931 solver.cpp:204]     Train net output #4: output_mvn = -0.665824
I1030 16:08:11.164583 110931 solver.cpp:204]     Train net output #5: output_mvn = -0.747604
I1030 16:08:11.164609 110931 solver.cpp:204]     Train net output #6: output_mvn = 1.41343
I1030 16:08:11.164635 110931 solver.cpp:461] Iteration 0, lr = 0
I1030 16:08:11.200248 110931 solver.cpp:189] Iteration 1, loss = 0
I1030 16:08:11.200299 110931 solver.cpp:204]     Train net output #0: input_mvn = -2.17866
I1030 16:08:11.200325 110931 solver.cpp:204]     Train net output #1: input_mvn = 1.52306
I1030 16:08:11.200340 110931 solver.cpp:204]     Train net output #2: input_mvn = 5.1902
I1030 16:08:11.200378 110931 solver.cpp:204]     Train net output #3: label = 0
I1030 16:08:11.200414 110931 solver.cpp:204]     Train net output #4: output_mvn = -1.22665
I1030 16:08:11.200436 110931 solver.cpp:204]     Train net output #5: output_mvn = 0.00383358
I1030 16:08:11.200453 110931 solver.cpp:204]     Train net output #6: output_mvn = 1.22283
I1030 16:08:11.200466 110931 solver.cpp:461] Iteration 1, lr = 0
I1030 16:08:11.200829 110931 solver.cpp:269] Iteration 2, Testing net (#0)
I1030 16:08:11.335741 110931 solver.cpp:318]     Test net output #0: input_mvn = -4.33491
I1030 16:08:11.335808 110931 solver.cpp:318]     Test net output #1: input_mvn = -0.370263
I1030 16:08:11.335824 110931 solver.cpp:318]     Test net output #2: input_mvn = 0.434494
I1030 16:08:11.335835 110931 solver.cpp:318]     Test net output #3: label = 0.3
I1030 16:08:11.335855 110931 solver.cpp:318]     Test net output #4: output_mvn = nan
I1030 16:08:11.335881 110931 solver.cpp:318]     Test net output #5: output_mvn = nan
I1030 16:08:11.335893 110931 solver.cpp:318]     Test net output #6: output_mvn = nan
I1030 16:08:11.347332 110931 solver.cpp:189] Iteration 2, loss = 0
I1030 16:08:11.347378 110931 solver.cpp:204]     Train net output #0: input_mvn = -2.9707
I1030 16:08:11.347403 110931 solver.cpp:204]     Train net output #1: input_mvn = 2.57361
I1030 16:08:11.347415 110931 solver.cpp:204]     Train net output #2: input_mvn = 8.95418
I1030 16:08:11.347446 110931 solver.cpp:204]     Train net output #3: label = 0
I1030 16:08:11.347481 110931 solver.cpp:204]     Train net output #4: output_mvn = -1.19513
I1030 16:08:11.347503 110931 solver.cpp:204]     Train net output #5: output_mvn = -0.0572087
I1030 16:08:11.347519 110931 solver.cpp:204]     Train net output #6: output_mvn = 1.25235
I1030 16:08:11.347532 110931 solver.cpp:461] Iteration 2, lr = 0
I1030 16:08:11.379030 110931 solver.cpp:189] Iteration 3, loss = 0
I1030 16:08:11.379076 110931 solver.cpp:204]     Train net output #0: input_mvn = -2.62701
I1030 16:08:11.379106 110931 solver.cpp:204]     Train net output #1: input_mvn = -1.24267
I1030 16:08:11.379122 110931 solver.cpp:204]     Train net output #2: input_mvn = -1.95568
I1030 16:08:11.379148 110931 solver.cpp:204]     Train net output #3: label = 0
I1030 16:08:11.379184 110931 solver.cpp:204]     Train net output #4: output_mvn = -1.21226
I1030 16:08:11.379204 110931 solver.cpp:204]     Train net output #5: output_mvn = 1.23683
I1030 16:08:11.379216 110931 solver.cpp:204]     Train net output #6: output_mvn = -0.0245948
I1030 16:08:11.379230 110931 solver.cpp:461] Iteration 3, lr = 0
I1030 16:08:11.379573 110931 solver.cpp:269] Iteration 4, Testing net (#0)
I1030 16:08:11.514329 110931 solver.cpp:318]     Test net output #0: input_mvn = -3.92829
I1030 16:08:11.514360 110931 solver.cpp:318]     Test net output #1: input_mvn = 0.767336
I1030 16:08:11.514374 110931 solver.cpp:318]     Test net output #2: input_mvn = 1.60613
I1030 16:08:11.514387 110931 solver.cpp:318]     Test net output #3: label = 0.4
I1030 16:08:11.514402 110931 solver.cpp:318]     Test net output #4: output_mvn = -1.06733
I1030 16:08:11.514435 110931 solver.cpp:318]     Test net output #5: output_mvn = 0.304378
I1030 16:08:11.514447 110931 solver.cpp:318]     Test net output #6: output_mvn = 0.13991
I1030 16:08:11.525923 110931 solver.cpp:189] Iteration 4, loss = 0
I1030 16:08:11.525970 110931 solver.cpp:204]     Train net output #0: input_mvn = -1.23216
I1030 16:08:11.526005 110931 solver.cpp:204]     Train net output #1: input_mvn = -0.737749
I1030 16:08:11.526020 110931 solver.cpp:204]     Train net output #2: input_mvn = 0.925414
I1030 16:08:11.526057 110931 solver.cpp:204]     Train net output #3: label = 0
I1030 16:08:11.526095 110931 solver.cpp:204]     Train net output #4: output_mvn = -0.957845
I1030 16:08:11.526129 110931 solver.cpp:204]     Train net output #5: output_mvn = -0.422135
I1030 16:08:11.526144 110931 solver.cpp:204]     Train net output #6: output_mvn = 1.37998
I1030 16:08:11.526165 110931 solver.cpp:461] Iteration 4, lr = 0
I1030 16:08:11.557688 110931 solver.cpp:189] Iteration 5, loss = 0
I1030 16:08:11.557736 110931 solver.cpp:204]     Train net output #0: input_mvn = -1.38162
I1030 16:08:11.557761 110931 solver.cpp:204]     Train net output #1: input_mvn = -2.60718
I1030 16:08:11.557776 110931 solver.cpp:204]     Train net output #2: input_mvn = 1.03692
I1030 16:08:11.557816 110931 solver.cpp:204]     Train net output #3: label = 0
I1030 16:08:11.557853 110931 solver.cpp:204]     Train net output #4: output_mvn = -0.262649
I1030 16:08:11.557875 110931 solver.cpp:204]     Train net output #5: output_mvn = -1.07212
I1030 16:08:11.557909 110931 solver.cpp:204]     Train net output #6: output_mvn = 1.33476
I1030 16:08:11.557924 110931 solver.cpp:461] Iteration 5, lr = 0
I1030 16:08:11.558264 110931 solver.cpp:269] Iteration 6, Testing net (#0)
I1030 16:08:11.693305 110931 solver.cpp:318]     Test net output #0: input_mvn = -3.04445
I1030 16:08:11.693339 110931 solver.cpp:318]     Test net output #1: input_mvn = 0.536213
I1030 16:08:11.693356 110931 solver.cpp:318]     Test net output #2: input_mvn = 2.08053
I1030 16:08:11.693372 110931 solver.cpp:318]     Test net output #3: label = 0.4
I1030 16:08:11.693392 110931 solver.cpp:318]     Test net output #4: output_mvn = nan
I1030 16:08:11.693408 110931 solver.cpp:318]     Test net output #5: output_mvn = nan
I1030 16:08:11.693434 110931 solver.cpp:318]     Test net output #6: output_mvn = nan




Network Definition :
name: "CaffeNet"
state {
  phase: TEST
}
layer {
  name: "data"
  type: "ImageData"
  top: "data"
  top: "label"
  transform_param {
    mirror: false
    crop_size: 227
    mean_file: "/home/ubuntu/caffe/data/ilsvrc12/imagenet_mean.binaryproto"
  }
  image_data_param {
    source: "/image/vis_sim/watch/sortedTotal_first_2k.txt"
    batch_size: 1
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    name: "conv1_w"
    lr_mult: 1
  }
  param {
    name: "conv1_b"
    lr_mult: 2
  }
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm1"
  type: "LRN"
  bottom: "pool1"
  top: "norm1"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "norm1"
  top: "conv2"
  param {
    name: "conv2_w"
    lr_mult: 1
  }
  param {
    name: "conv2_b"
    lr_mult: 2
  }
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm2"
  type: "LRN"
  bottom: "pool2"
  top: "norm2"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "norm2"
  top: "conv3"
  param {
    name: "conv3_w"
    lr_mult: 1
  }
  param {
    name: "conv3_b"
    lr_mult: 2
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "conv4"
  type: "Convolution"
  bottom: "conv3"
  top: "conv4"
  param {
    name: "conv4_w"
    lr_mult: 1
  }
  param {
    name: "conv4_b"
    lr_mult: 2
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layer {
  name: "relu4"
  type: "ReLU"
  bottom: "conv4"
  top: "conv4"
}
layer {
  name: "conv5"
  type: "Convolution"
  bottom: "conv4"
  top: "conv5"
  param {
    name: "conv5_w"
    lr_mult: 1
  }
  param {
    name: "conv5_b"
    lr_mult: 2
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
  }
}
layer {
  name: "relu5"
  type: "ReLU"
  bottom: "conv5"
  top: "conv5"
}
layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {
    name: "ip1_w"
    lr_mult: 1
  }
  param {
    name: "ip1_b"
    lr_mult: 2
  }
  inner_product_param {
    num_output: 4096
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    name: "ip2_w"
    lr_mult: 1
  }
  param {
    name: "ip2_b"
    lr_mult: 2
  }
  inner_product_param {
    num_output: 4096
  }
}
layer {
  name: "fc9"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc9"
  param {
    name: "ip3_w"
    lr_mult: 1
  }
  param {
    name: "ip3_b"
    lr_mult: 2
  }
  inner_product_param {
    num_output: 3
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "output_mvn"
  type: "MVN"
  bottom: "fc9"
  top: "output_mvn"
  mvn_param {
    normalize_variance: true
    across_channels: true
    eps: 1e-09
  }
}
layer {
  name: "input_mvn"
  type: "Power"
  bottom: "fc9"
  top: "input_mvn"
  power_param {
    power: 1
    scale: 1
    shift: 0
  }
}


Thanks and Regards,
Suraj Sheth


--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/1SPxngz9bR8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/ebc08079-9675-4b4b-a5d0-b70c975485b7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jie Yang

unread,
Mar 28, 2016, 8:54:41 PM3/28/16
to Caffe Users
Has anyone figured out how to use MVN layer? I encountered the same problem.

在 2015年10月7日星期三 UTC+10:30下午5:39:44，Hyo Jin Kim写道：
Reply all
Reply to author
Forward
0 new messages