Error when running caffenet on imagenet data (error == cudaSuccess (2 vs. 0 out of memory)

305 views
Skip to first unread message

Gedditoffme

unread,
Jul 24, 2016, 10:19:52 AM7/24/16
to Caffe Users
Hello,

Trying to test Caffenet - I can get it working on CPU only, but it is slow. When I try to use GPU however, it starts to cause problems.

I'm running on windows, and have run on both Git Bash (in Windows) and CMD (default windows prompt). In both cases I get output similar to the below, ending in "*** Check failure stack trace: ***". Is there a separate failure stack I should be seeing, because the below doesn't show a clear failure mechanism.

The key line seems to be only in windows CMD
F0724 22:26:54.063722 10780 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
Does anyone know what this means or what the resolution is?


Final few lines on CMD
I0724 22:17:08.556980 10780 net.cpp:217] conv2 needs backward computation.
I0724 22:17:08.556980 10780 net.cpp:217] norm1 needs backward computation.
I0724 22:17:08.556980 10780 net.cpp:217] pool1 needs backward computation.
I0724 22:17:08.556980 10780 net.cpp:217] relu1 needs backward computation.
I0724 22:17:08.556980 10780 net.cpp:217] conv1 needs backward computation.
I0724 22:17:08.556980 10780 net.cpp:219] label_data_1_split does not need backwa
rd computation.
I0724 22:17:08.556980 10780 net.cpp:219] data does not need backward computation
.
I0724 22:17:08.557981 10780 net.cpp:261] This network produces output accuracy
I0724 22:17:08.557981 10780 net.cpp:261] This network produces output loss
I0724 22:17:08.557981 10780 net.cpp:274] Network initialization done.
I0724 22:17:08.557981 10780 solver.cpp:60] Solver scaffolding done.
I0724 22:17:08.559983 10780 caffe.cpp:220] Starting Optimization
I0724 22:17:08.559983 10780 solver.cpp:279] Solving CaffeNet
I0724 22:17:08.559983 10780 solver.cpp:280] Learning Rate Policy: step
I0724 22:17:08.605013 10780 solver.cpp:337] Iteration 0, Testing net (#0)
I0724 22:26:50.424646 10780 solver.cpp:404]     Test net output #0: accuracy = 0

I0724 22:26:50.424646 10780 solver.cpp:404]     Test net output #1: loss = 6.833
8 (* 1 = 6.8338 loss)
F0724 22:26:54.063722 10780 syncedmem.cpp:56] Check failed: error == cudaSuccess
 (2 vs. 0)  out of memory
*** Check failure stack trace: ***




Full output from Bash
./Build/x64/Release/caffe.exe train --solver=models/bvlc_reference_caffenet/solver.prototxt -gpu 0 >TesT_Caffe_output_24July.txt
I0724 20:55:29.871520  6520 caffe.cpp:186] Using GPUs 0
I0724 20:55:30.593121  6520 caffe.cpp:191] GPU 0: GeForce GT 650M
I0724 20:55:30.786249  6520 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.
I0724 20:55:30.786249  6520 solver.cpp:48] Initializing solver from parameters:
test_iter: 1000
test_interval: 1000
base_lr: 0.01
display: 20
max_iter: 450000
lr_policy: "step"
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
stepsize: 100000
snapshot: 10000
snapshot_prefix: "models/bvlc_reference_caffenet/caffenet_train"
solver_mode: CPU
device_id: 0
net: "models/bvlc_reference_caffenet/train_val.prototxt"
I0724 20:55:30.786249  6520 solver.cpp:91] Creating training net from net file: models/bvlc_reference_caffenet/train_val.prototxt
I0724 20:55:30.787250  6520 net.cpp:313] The NetState phase (0) differed from the phase (1) specified by a rule in layer data
I0724 20:55:30.787250  6520 net.cpp:313] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
I0724 20:55:30.787250  6520 net.cpp:49] Initializing net from parameters:
name: "CaffeNet"
state {
  phase: TRAIN
}
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    mirror: true
    crop_size: 227
    mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"
  }
  data_param {
    source: "examples/imagenet/ilsvrc12_train_lmdb"
    batch_size: 128
    backend: LMDB
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm1"
  type: "LRN"
  bottom: "pool1"
  top: "norm1"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "norm1"
  top: "conv2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm2"
  type: "LRN"
  bottom: "pool2"
  top: "norm2"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "norm2"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "conv4"
  type: "Convolution"
  bottom: "conv3"
  top: "conv4"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu4"
  type: "ReLU"
  bottom: "conv4"
  top: "conv4"
}
layer {
  name: "conv5"
  type: "Convolution"
  bottom: "conv4"
  top: "conv5"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu5"
  type: "ReLU"
  bottom: "conv5"
  top: "conv5"
}
layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "drop6"
  type: "Dropout"
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu7"
  type: "ReLU"
  bottom: "fc7"
  top: "fc7"
}
layer {
  name: "drop7"
  type: "Dropout"
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc8"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc8"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 1000
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "fc8"
  bottom: "label"
  top: "loss"
}
I0724 20:55:30.787250  6520 layer_factory.hpp:77] Creating layer data
I0724 20:55:30.788250  6520 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.
I0724 20:55:30.788250  6520 net.cpp:91] Creating Layer data
I0724 20:55:30.788250  6520 net.cpp:399] data -> data
I0724 20:55:30.788250  6520 net.cpp:399] data -> label
I0724 20:55:30.788250  6520 data_transformer.cpp:25] Loading mean file from: data/ilsvrc12/imagenet_mean.binaryproto
I0724 20:55:30.790253 10648 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.
I0724 20:55:30.790253 10648 db_lmdb.cpp:40] Opened lmdb examples/imagenet/ilsvrc12_train_lmdb
I0724 20:55:30.824275  6520 data_layer.cpp:41] output data size: 128,3,227,227
I0724 20:55:30.930346  6520 net.cpp:141] Setting up data
I0724 20:55:30.930346  6520 net.cpp:148] Top shape: 128 3 227 227 (19787136)
I0724 20:55:30.930346  6520 net.cpp:148] Top shape: 128 (128)
I0724 20:55:30.930346  6520 net.cpp:156] Memory required for data: 79149056
I0724 20:55:30.930346  6520 layer_factory.hpp:77] Creating layer conv1
I0724 20:55:30.930346  6520 net.cpp:91] Creating Layer conv1
I0724 20:55:30.930346  6520 net.cpp:425] conv1 <- data
I0724 20:55:30.930346  6520 net.cpp:399] conv1 -> conv1
I0724 20:55:30.933347  8788 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.
I0724 20:55:31.171507  6520 net.cpp:141] Setting up conv1
I0724 20:55:31.171507  6520 net.cpp:148] Top shape: 128 96 55 55 (37171200)
I0724 20:55:31.171507  6520 net.cpp:156] Memory required for data: 227833856
I0724 20:55:31.171507  6520 layer_factory.hpp:77] Creating layer relu1
I0724 20:55:31.171507  6520 net.cpp:91] Creating Layer relu1
I0724 20:55:31.171507  6520 net.cpp:425] relu1 <- conv1
I0724 20:55:31.171507  6520 net.cpp:386] relu1 -> conv1 (in-place)
I0724 20:55:31.171507  6520 net.cpp:141] Setting up relu1
I0724 20:55:31.171507  6520 net.cpp:148] Top shape: 128 96 55 55 (37171200)
I0724 20:55:31.171507  6520 net.cpp:156] Memory required for data: 376518656
I0724 20:55:31.171507  6520 layer_factory.hpp:77] Creating layer pool1
I0724 20:55:31.171507  6520 net.cpp:91] Creating Layer pool1
I0724 20:55:31.171507  6520 net.cpp:425] pool1 <- conv1
I0724 20:55:31.171507  6520 net.cpp:399] pool1 -> pool1
I0724 20:55:31.171507  6520 net.cpp:141] Setting up pool1
I0724 20:55:31.171507  6520 net.cpp:148] Top shape: 128 96 27 27 (8957952)
I0724 20:55:31.171507  6520 net.cpp:156] Memory required for data: 412350464
I0724 20:55:31.171507  6520 layer_factory.hpp:77] Creating layer norm1
I0724 20:55:31.171507  6520 net.cpp:91] Creating Layer norm1
I0724 20:55:31.171507  6520 net.cpp:425] norm1 <- pool1
I0724 20:55:31.171507  6520 net.cpp:399] norm1 -> norm1
I0724 20:55:31.172508  6520 net.cpp:141] Setting up norm1
I0724 20:55:31.172508  6520 net.cpp:148] Top shape: 128 96 27 27 (8957952)
I0724 20:55:31.172508  6520 net.cpp:156] Memory required for data: 448182272
I0724 20:55:31.172508  6520 layer_factory.hpp:77] Creating layer conv2
I0724 20:55:31.172508  6520 net.cpp:91] Creating Layer conv2
I0724 20:55:31.172508  6520 net.cpp:425] conv2 <- norm1
I0724 20:55:31.172508  6520 net.cpp:399] conv2 -> conv2
I0724 20:55:31.179512  6520 net.cpp:141] Setting up conv2
I0724 20:55:31.179512  6520 net.cpp:148] Top shape: 128 256 27 27 (23887872)
I0724 20:55:31.179512  6520 net.cpp:156] Memory required for data: 543733760
I0724 20:55:31.179512  6520 layer_factory.hpp:77] Creating layer relu2
I0724 20:55:31.179512  6520 net.cpp:91] Creating Layer relu2
I0724 20:55:31.179512  6520 net.cpp:425] relu2 <- conv2
I0724 20:55:31.179512  6520 net.cpp:386] relu2 -> conv2 (in-place)
I0724 20:55:31.180512  6520 net.cpp:141] Setting up relu2
I0724 20:55:31.180512  6520 net.cpp:148] Top shape: 128 256 27 27 (23887872)
I0724 20:55:31.180512  6520 net.cpp:156] Memory required for data: 639285248
I0724 20:55:31.180512  6520 layer_factory.hpp:77] Creating layer pool2
I0724 20:55:31.180512  6520 net.cpp:91] Creating Layer pool2
I0724 20:55:31.180512  6520 net.cpp:425] pool2 <- conv2
I0724 20:55:31.180512  6520 net.cpp:399] pool2 -> pool2
I0724 20:55:31.180512  6520 net.cpp:141] Setting up pool2
I0724 20:55:31.180512  6520 net.cpp:148] Top shape: 128 256 13 13 (5537792)
I0724 20:55:31.180512  6520 net.cpp:156] Memory required for data: 661436416
I0724 20:55:31.180512  6520 layer_factory.hpp:77] Creating layer norm2
I0724 20:55:31.180512  6520 net.cpp:91] Creating Layer norm2
I0724 20:55:31.180512  6520 net.cpp:425] norm2 <- pool2
I0724 20:55:31.180512  6520 net.cpp:399] norm2 -> norm2
I0724 20:55:31.180512  6520 net.cpp:141] Setting up norm2
I0724 20:55:31.180512  6520 net.cpp:148] Top shape: 128 256 13 13 (5537792)
I0724 20:55:31.180512  6520 net.cpp:156] Memory required for data: 683587584
I0724 20:55:31.180512  6520 layer_factory.hpp:77] Creating layer conv3
I0724 20:55:31.180512  6520 net.cpp:91] Creating Layer conv3
I0724 20:55:31.180512  6520 net.cpp:425] conv3 <- norm2
I0724 20:55:31.180512  6520 net.cpp:399] conv3 -> conv3
I0724 20:55:31.200525  6520 net.cpp:141] Setting up conv3
I0724 20:55:31.200525  6520 net.cpp:148] Top shape: 128 384 13 13 (8306688)
I0724 20:55:31.200525  6520 net.cpp:156] Memory required for data: 716814336
I0724 20:55:31.200525  6520 layer_factory.hpp:77] Creating layer relu3
I0724 20:55:31.200525  6520 net.cpp:91] Creating Layer relu3
I0724 20:55:31.200525  6520 net.cpp:425] relu3 <- conv3
I0724 20:55:31.200525  6520 net.cpp:386] relu3 -> conv3 (in-place)
I0724 20:55:31.200525  6520 net.cpp:141] Setting up relu3
I0724 20:55:31.201527  6520 net.cpp:148] Top shape: 128 384 13 13 (8306688)
I0724 20:55:31.201527  6520 net.cpp:156] Memory required for data: 750041088
I0724 20:55:31.201527  6520 layer_factory.hpp:77] Creating layer conv4
I0724 20:55:31.201527  6520 net.cpp:91] Creating Layer conv4
I0724 20:55:31.201527  6520 net.cpp:425] conv4 <- conv3
I0724 20:55:31.201527  6520 net.cpp:399] conv4 -> conv4
I0724 20:55:31.221539  6520 net.cpp:141] Setting up conv4
I0724 20:55:31.221539  6520 net.cpp:148] Top shape: 128 384 13 13 (8306688)
I0724 20:55:31.221539  6520 net.cpp:156] Memory required for data: 783267840
I0724 20:55:31.221539  6520 layer_factory.hpp:77] Creating layer relu4
I0724 20:55:31.221539  6520 net.cpp:91] Creating Layer relu4
I0724 20:55:31.221539  6520 net.cpp:425] relu4 <- conv4
I0724 20:55:31.221539  6520 net.cpp:386] relu4 -> conv4 (in-place)
I0724 20:55:31.222542  6520 net.cpp:141] Setting up relu4
I0724 20:55:31.222542  6520 net.cpp:148] Top shape: 128 384 13 13 (8306688)
I0724 20:55:31.222542  6520 net.cpp:156] Memory required for data: 816494592
I0724 20:55:31.222542  6520 layer_factory.hpp:77] Creating layer conv5
I0724 20:55:31.222542  6520 net.cpp:91] Creating Layer conv5
I0724 20:55:31.222542  6520 net.cpp:425] conv5 <- conv4
I0724 20:55:31.222542  6520 net.cpp:399] conv5 -> conv5
I0724 20:55:31.234547  6520 net.cpp:141] Setting up conv5
I0724 20:55:31.234547  6520 net.cpp:148] Top shape: 128 256 13 13 (5537792)
I0724 20:55:31.234547  6520 net.cpp:156] Memory required for data: 838645760
I0724 20:55:31.234547  6520 layer_factory.hpp:77] Creating layer relu5
I0724 20:55:31.234547  6520 net.cpp:91] Creating Layer relu5
I0724 20:55:31.234547  6520 net.cpp:425] relu5 <- conv5
I0724 20:55:31.234547  6520 net.cpp:386] relu5 -> conv5 (in-place)
I0724 20:55:31.234547  6520 net.cpp:141] Setting up relu5
I0724 20:55:31.234547  6520 net.cpp:148] Top shape: 128 256 13 13 (5537792)
I0724 20:55:31.234547  6520 net.cpp:156] Memory required for data: 860796928
I0724 20:55:31.234547  6520 layer_factory.hpp:77] Creating layer pool5
I0724 20:55:31.234547  6520 net.cpp:91] Creating Layer pool5
I0724 20:55:31.234547  6520 net.cpp:425] pool5 <- conv5
I0724 20:55:31.234547  6520 net.cpp:399] pool5 -> pool5
I0724 20:55:31.234547  6520 net.cpp:141] Setting up pool5
I0724 20:55:31.234547  6520 net.cpp:148] Top shape: 128 256 6 6 (1179648)
I0724 20:55:31.234547  6520 net.cpp:156] Memory required for data: 865515520
I0724 20:55:31.234547  6520 layer_factory.hpp:77] Creating layer fc6
I0724 20:55:31.234547  6520 net.cpp:91] Creating Layer fc6
I0724 20:55:31.234547  6520 net.cpp:425] fc6 <- pool5
I0724 20:55:31.235548  6520 net.cpp:399] fc6 -> fc6
I0724 20:55:31.744889  6520 net.cpp:141] Setting up fc6
I0724 20:55:31.744889  6520 net.cpp:148] Top shape: 128 4096 (524288)
I0724 20:55:31.744889  6520 net.cpp:156] Memory required for data: 867612672
I0724 20:55:31.744889  6520 layer_factory.hpp:77] Creating layer relu6
I0724 20:55:31.744889  6520 net.cpp:91] Creating Layer relu6
I0724 20:55:31.744889  6520 net.cpp:425] relu6 <- fc6
I0724 20:55:31.744889  6520 net.cpp:386] relu6 -> fc6 (in-place)
I0724 20:55:31.744889  6520 net.cpp:141] Setting up relu6
I0724 20:55:31.744889  6520 net.cpp:148] Top shape: 128 4096 (524288)
I0724 20:55:31.744889  6520 net.cpp:156] Memory required for data: 869709824
I0724 20:55:31.744889  6520 layer_factory.hpp:77] Creating layer drop6
I0724 20:55:31.744889  6520 net.cpp:91] Creating Layer drop6
I0724 20:55:31.744889  6520 net.cpp:425] drop6 <- fc6
I0724 20:55:31.744889  6520 net.cpp:386] drop6 -> fc6 (in-place)
I0724 20:55:31.744889  6520 net.cpp:141] Setting up drop6
I0724 20:55:31.744889  6520 net.cpp:148] Top shape: 128 4096 (524288)
I0724 20:55:31.744889  6520 net.cpp:156] Memory required for data: 871806976
I0724 20:55:31.744889  6520 layer_factory.hpp:77] Creating layer fc7
I0724 20:55:31.744889  6520 net.cpp:91] Creating Layer fc7
I0724 20:55:31.744889  6520 net.cpp:425] fc7 <- fc6
I0724 20:55:31.744889  6520 net.cpp:399] fc7 -> fc7
I0724 20:55:31.971040  6520 net.cpp:141] Setting up fc7
I0724 20:55:31.971040  6520 net.cpp:148] Top shape: 128 4096 (524288)
I0724 20:55:31.971040  6520 net.cpp:156] Memory required for data: 873904128
I0724 20:55:31.971040  6520 layer_factory.hpp:77] Creating layer relu7
I0724 20:55:31.971040  6520 net.cpp:91] Creating Layer relu7
I0724 20:55:31.971040  6520 net.cpp:425] relu7 <- fc7
I0724 20:55:31.971040  6520 net.cpp:386] relu7 -> fc7 (in-place)
I0724 20:55:31.972039  6520 net.cpp:141] Setting up relu7
I0724 20:55:31.972039  6520 net.cpp:148] Top shape: 128 4096 (524288)
I0724 20:55:31.972039  6520 net.cpp:156] Memory required for data: 876001280
I0724 20:55:31.972039  6520 layer_factory.hpp:77] Creating layer drop7
I0724 20:55:31.972039  6520 net.cpp:91] Creating Layer drop7
I0724 20:55:31.972039  6520 net.cpp:425] drop7 <- fc7
I0724 20:55:31.972039  6520 net.cpp:386] drop7 -> fc7 (in-place)
I0724 20:55:31.972039  6520 net.cpp:141] Setting up drop7
I0724 20:55:31.972039  6520 net.cpp:148] Top shape: 128 4096 (524288)
I0724 20:55:31.972039  6520 net.cpp:156] Memory required for data: 878098432
I0724 20:55:31.972039  6520 layer_factory.hpp:77] Creating layer fc8
I0724 20:55:31.972039  6520 net.cpp:91] Creating Layer fc8
I0724 20:55:31.972039  6520 net.cpp:425] fc8 <- fc7
I0724 20:55:31.972039  6520 net.cpp:399] fc8 -> fc8
I0724 20:55:32.030079  6520 net.cpp:141] Setting up fc8
I0724 20:55:32.030079  6520 net.cpp:148] Top shape: 128 1000 (128000)
I0724 20:55:32.030079  6520 net.cpp:156] Memory required for data: 878610432
I0724 20:55:32.030079  6520 layer_factory.hpp:77] Creating layer loss
I0724 20:55:32.033082  6520 net.cpp:91] Creating Layer loss
I0724 20:55:32.033082  6520 net.cpp:425] loss <- fc8
I0724 20:55:32.033082  6520 net.cpp:425] loss <- label
I0724 20:55:32.033082  6520 net.cpp:399] loss -> loss
I0724 20:55:32.033082  6520 layer_factory.hpp:77] Creating layer loss
I0724 20:55:32.034081  6520 net.cpp:141] Setting up loss
I0724 20:55:32.034081  6520 net.cpp:148] Top shape: (1)
I0724 20:55:32.034081  6520 net.cpp:151]     with loss weight 1
I0724 20:55:32.034081  6520 net.cpp:156] Memory required for data: 878610436
I0724 20:55:32.034081  6520 net.cpp:217] loss needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] fc8 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] drop7 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] relu7 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] fc7 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] drop6 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] relu6 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] fc6 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] pool5 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] relu5 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] conv5 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] relu4 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] conv4 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] relu3 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] conv3 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] norm2 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] pool2 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] relu2 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] conv2 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] norm1 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] pool1 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] relu1 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:217] conv1 needs backward computation.
I0724 20:55:32.034081  6520 net.cpp:219] data does not need backward computation.
I0724 20:55:32.034081  6520 net.cpp:261] This network produces output loss
I0724 20:55:32.034081  6520 net.cpp:274] Network initialization done.
I0724 20:55:32.035082  6520 solver.cpp:181] Creating test net (#0) specified by net file: models/bvlc_reference_caffenet/train_val.prototxt
I0724 20:55:32.035082  6520 net.cpp:313] The NetState phase (1) differed from the phase (0) specified by a rule in layer data
I0724 20:55:32.035082  6520 net.cpp:49] Initializing net from parameters:
name: "CaffeNet"
state {
  phase: TEST
}
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    mirror: false
    crop_size: 227
    mean_file: "data/ilsvrc12/imagenet_mean.binaryproto"
  }
  data_param {
    source: "examples/imagenet/ilsvrc12_val_lmdb"
    batch_size: 50
    backend: LMDB
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 96
    kernel_size: 11
    stride: 4
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm1"
  type: "LRN"
  bottom: "pool1"
  top: "norm1"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "norm1"
  top: "conv2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "norm2"
  type: "LRN"
  bottom: "pool2"
  top: "norm2"
  lrn_param {
    local_size: 5
    alpha: 0.0001
    beta: 0.75
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "norm2"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "conv4"
  type: "Convolution"
  bottom: "conv3"
  top: "conv4"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 384
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu4"
  type: "ReLU"
  bottom: "conv4"
  top: "conv4"
}
layer {
  name: "conv5"
  type: "Convolution"
  bottom: "conv4"
  top: "conv5"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 256
    pad: 1
    kernel_size: 3
    group: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu5"
  type: "ReLU"
  bottom: "conv5"
  top: "conv5"
}
layer {
  name: "pool5"
  type: "Pooling"
  bottom: "conv5"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "fc6"
  type: "InnerProduct"
  bottom: "pool5"
  top: "fc6"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu6"
  type: "ReLU"
  bottom: "fc6"
  top: "fc6"
}
layer {
  name: "drop6"
  type: "Dropout"
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc7"
  type: "InnerProduct"
  bottom: "fc6"
  top: "fc7"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 4096
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layer {
  name: "relu7"
  type: "ReLU"
  bottom: "fc7"
  top: "fc7"
}
layer {
  name: "drop7"
  type: "Dropout"
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "fc8"
  type: "InnerProduct"
  bottom: "fc7"
  top: "fc8"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 1000
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "fc8"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "fc8"
  bottom: "label"
  top: "loss"
}
I0724 20:55:32.035082  6520 layer_factory.hpp:77] Creating layer data
I0724 20:55:32.035082  6520 net.cpp:91] Creating Layer data
I0724 20:55:32.035082  6520 net.cpp:399] data -> data
I0724 20:55:32.035082  6520 net.cpp:399] data -> label
I0724 20:55:32.035082  6520 data_transformer.cpp:25] Loading mean file from: data/ilsvrc12/imagenet_mean.binaryproto
I0724 20:55:32.037084  9108 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.
I0724 20:55:32.038084  9108 db_lmdb.cpp:40] Opened lmdb examples/imagenet/ilsvrc12_val_lmdb
I0724 20:55:32.041086  6520 data_layer.cpp:41] output data size: 50,3,227,227
I0724 20:55:32.086115  6520 net.cpp:141] Setting up data
I0724 20:55:32.086115  6520 net.cpp:148] Top shape: 50 3 227 227 (7729350)
I0724 20:55:32.086115  6520 net.cpp:148] Top shape: 50 (50)
I0724 20:55:32.086115  6520 net.cpp:156] Memory required for data: 30917600
I0724 20:55:32.086115  6520 layer_factory.hpp:77] Creating layer label_data_1_split
I0724 20:55:32.086115  6520 net.cpp:91] Creating Layer label_data_1_split
I0724 20:55:32.086115  6520 net.cpp:425] label_data_1_split <- label
I0724 20:55:32.086115  6520 net.cpp:399] label_data_1_split -> label_data_1_split_0
I0724 20:55:32.086115  6520 net.cpp:399] label_data_1_split -> label_data_1_split_1
I0724 20:55:32.086115  6520 net.cpp:141] Setting up label_data_1_split
I0724 20:55:32.086115  6520 net.cpp:148] Top shape: 50 (50)
I0724 20:55:32.086115  6520 net.cpp:148] Top shape: 50 (50)
I0724 20:55:32.086115  6520 net.cpp:156] Memory required for data: 30918000
I0724 20:55:32.086115  6520 layer_factory.hpp:77] Creating layer conv1
I0724 20:55:32.086115  6520 net.cpp:91] Creating Layer conv1
I0724 20:55:32.086115  6520 net.cpp:425] conv1 <- data
I0724 20:55:32.086115  6520 net.cpp:399] conv1 -> conv1
I0724 20:55:32.088117  4652 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.
I0724 20:55:32.094121  6520 net.cpp:141] Setting up conv1
I0724 20:55:32.094121  6520 net.cpp:148] Top shape: 50 96 55 55 (14520000)
I0724 20:55:32.094121  6520 net.cpp:156] Memory required for data: 88998000
I0724 20:55:32.094121  6520 layer_factory.hpp:77] Creating layer relu1
I0724 20:55:32.094121  6520 net.cpp:91] Creating Layer relu1
I0724 20:55:32.094121  6520 net.cpp:425] relu1 <- conv1
I0724 20:55:32.094121  6520 net.cpp:386] relu1 -> conv1 (in-place)
I0724 20:55:32.095124  6520 net.cpp:141] Setting up relu1
I0724 20:55:32.095124  6520 net.cpp:148] Top shape: 50 96 55 55 (14520000)
I0724 20:55:32.095124  6520 net.cpp:156] Memory required for data: 147078000
I0724 20:55:32.095124  6520 layer_factory.hpp:77] Creating layer pool1
I0724 20:55:32.095124  6520 net.cpp:91] Creating Layer pool1
I0724 20:55:32.095124  6520 net.cpp:425] pool1 <- conv1
I0724 20:55:32.095124  6520 net.cpp:399] pool1 -> pool1
I0724 20:55:32.095124  6520 net.cpp:141] Setting up pool1
I0724 20:55:32.095124  6520 net.cpp:148] Top shape: 50 96 27 27 (3499200)
I0724 20:55:32.095124  6520 net.cpp:156] Memory required for data: 161074800
I0724 20:55:32.095124  6520 layer_factory.hpp:77] Creating layer norm1
I0724 20:55:32.095124  6520 net.cpp:91] Creating Layer norm1
I0724 20:55:32.095124  6520 net.cpp:425] norm1 <- pool1
I0724 20:55:32.095124  6520 net.cpp:399] norm1 -> norm1
I0724 20:55:32.095124  6520 net.cpp:141] Setting up norm1
I0724 20:55:32.095124  6520 net.cpp:148] Top shape: 50 96 27 27 (3499200)
I0724 20:55:32.095124  6520 net.cpp:156] Memory required for data: 175071600
I0724 20:55:32.095124  6520 layer_factory.hpp:77] Creating layer conv2
I0724 20:55:32.096125  6520 net.cpp:91] Creating Layer conv2
I0724 20:55:32.096125  6520 net.cpp:425] conv2 <- norm1
I0724 20:55:32.096125  6520 net.cpp:399] conv2 -> conv2
I0724 20:55:32.104131  6520 net.cpp:141] Setting up conv2
I0724 20:55:32.104131  6520 net.cpp:148] Top shape: 50 256 27 27 (9331200)
I0724 20:55:32.104131  6520 net.cpp:156] Memory required for data: 212396400
I0724 20:55:32.104131  6520 layer_factory.hpp:77] Creating layer relu2
I0724 20:55:32.104131  6520 net.cpp:91] Creating Layer relu2
I0724 20:55:32.104131  6520 net.cpp:425] relu2 <- conv2
I0724 20:55:32.104131  6520 net.cpp:386] relu2 -> conv2 (in-place)
I0724 20:55:32.105129  6520 net.cpp:141] Setting up relu2
I0724 20:55:32.105129  6520 net.cpp:148] Top shape: 50 256 27 27 (9331200)
I0724 20:55:32.105129  6520 net.cpp:156] Memory required for data: 249721200
I0724 20:55:32.105129  6520 layer_factory.hpp:77] Creating layer pool2
I0724 20:55:32.105129  6520 net.cpp:91] Creating Layer pool2
I0724 20:55:32.105129  6520 net.cpp:425] pool2 <- conv2
I0724 20:55:32.105129  6520 net.cpp:399] pool2 -> pool2
I0724 20:55:32.105129  6520 net.cpp:141] Setting up pool2
I0724 20:55:32.105129  6520 net.cpp:148] Top shape: 50 256 13 13 (2163200)
I0724 20:55:32.105129  6520 net.cpp:156] Memory required for data: 258374000
I0724 20:55:32.105129  6520 layer_factory.hpp:77] Creating layer norm2
I0724 20:55:32.105129  6520 net.cpp:91] Creating Layer norm2
I0724 20:55:32.105129  6520 net.cpp:425] norm2 <- pool2
I0724 20:55:32.105129  6520 net.cpp:399] norm2 -> norm2
I0724 20:55:32.106132  6520 net.cpp:141] Setting up norm2
I0724 20:55:32.106132  6520 net.cpp:148] Top shape: 50 256 13 13 (2163200)
I0724 20:55:32.106132  6520 net.cpp:156] Memory required for data: 267026800
I0724 20:55:32.106132  6520 layer_factory.hpp:77] Creating layer conv3
I0724 20:55:32.106132  6520 net.cpp:91] Creating Layer conv3
I0724 20:55:32.106132  6520 net.cpp:425] conv3 <- norm2
I0724 20:55:32.106132  6520 net.cpp:399] conv3 -> conv3
I0724 20:55:32.127143  6520 net.cpp:141] Setting up conv3
I0724 20:55:32.127143  6520 net.cpp:148] Top shape: 50 384 13 13 (3244800)
I0724 20:55:32.127143  6520 net.cpp:156] Memory required for data: 280006000
I0724 20:55:32.127143  6520 layer_factory.hpp:77] Creating layer relu3
I0724 20:55:32.127143  6520 net.cpp:91] Creating Layer relu3
I0724 20:55:32.127143  6520 net.cpp:425] relu3 <- conv3
I0724 20:55:32.128298  6520 net.cpp:386] relu3 -> conv3 (in-place)
I0724 20:55:32.128298  6520 net.cpp:141] Setting up relu3
I0724 20:55:32.128298  6520 net.cpp:148] Top shape: 50 384 13 13 (3244800)
I0724 20:55:32.128298  6520 net.cpp:156] Memory required for data: 292985200
I0724 20:55:32.128298  6520 layer_factory.hpp:77] Creating layer conv4
I0724 20:55:32.128298  6520 net.cpp:91] Creating Layer conv4
I0724 20:55:32.128298  6520 net.cpp:425] conv4 <- conv3
I0724 20:55:32.128298  6520 net.cpp:399] conv4 -> conv4
I0724 20:55:32.143156  6520 net.cpp:141] Setting up conv4
I0724 20:55:32.143156  6520 net.cpp:148] Top shape: 50 384 13 13 (3244800)
I0724 20:55:32.143156  6520 net.cpp:156] Memory required for data: 305964400
I0724 20:55:32.143156  6520 layer_factory.hpp:77] Creating layer relu4
I0724 20:55:32.143156  6520 net.cpp:91] Creating Layer relu4
I0724 20:55:32.143156  6520 net.cpp:425] relu4 <- conv4
I0724 20:55:32.143156  6520 net.cpp:386] relu4 -> conv4 (in-place)
I0724 20:55:32.143156  6520 net.cpp:141] Setting up relu4
I0724 20:55:32.143156  6520 net.cpp:148] Top shape: 50 384 13 13 (3244800)
I0724 20:55:32.143156  6520 net.cpp:156] Memory required for data: 318943600
I0724 20:55:32.143156  6520 layer_factory.hpp:77] Creating layer conv5
I0724 20:55:32.143156  6520 net.cpp:91] Creating Layer conv5
I0724 20:55:32.143156  6520 net.cpp:425] conv5 <- conv4
I0724 20:55:32.143156  6520 net.cpp:399] conv5 -> conv5
I0724 20:55:32.152161  6520 net.cpp:141] Setting up conv5
I0724 20:55:32.152161  6520 net.cpp:148] Top shape: 50 256 13 13 (2163200)
I0724 20:55:32.152161  6520 net.cpp:156] Memory required for data: 327596400
I0724 20:55:32.153162  6520 layer_factory.hpp:77] Creating layer relu5
I0724 20:55:32.153162  6520 net.cpp:91] Creating Layer relu5
I0724 20:55:32.153162  6520 net.cpp:425] relu5 <- conv5
I0724 20:55:32.153162  6520 net.cpp:386] relu5 -> conv5 (in-place)
I0724 20:55:32.153162  6520 net.cpp:141] Setting up relu5
I0724 20:55:32.153162  6520 net.cpp:148] Top shape: 50 256 13 13 (2163200)
I0724 20:55:32.153162  6520 net.cpp:156] Memory required for data: 336249200
I0724 20:55:32.153162  6520 layer_factory.hpp:77] Creating layer pool5
I0724 20:55:32.153162  6520 net.cpp:91] Creating Layer pool5
I0724 20:55:32.153162  6520 net.cpp:425] pool5 <- conv5
I0724 20:55:32.153162  6520 net.cpp:399] pool5 -> pool5
I0724 20:55:32.153162  6520 net.cpp:141] Setting up pool5
I0724 20:55:32.153162  6520 net.cpp:148] Top shape: 50 256 6 6 (460800)
I0724 20:55:32.153162  6520 net.cpp:156] Memory required for data: 338092400
I0724 20:55:32.154163  6520 layer_factory.hpp:77] Creating layer fc6
I0724 20:55:32.154163  6520 net.cpp:91] Creating Layer fc6
I0724 20:55:32.154163  6520 net.cpp:425] fc6 <- pool5
I0724 20:55:32.154163  6520 net.cpp:399] fc6 -> fc6
I0724 20:55:32.678510  6520 net.cpp:141] Setting up fc6
I0724 20:55:32.678510  6520 net.cpp:148] Top shape: 50 4096 (204800)
I0724 20:55:32.678510  6520 net.cpp:156] Memory required for data: 338911600
I0724 20:55:32.678510  6520 layer_factory.hpp:77] Creating layer relu6
I0724 20:55:32.678510  6520 net.cpp:91] Creating Layer relu6
I0724 20:55:32.678510  6520 net.cpp:425] relu6 <- fc6
I0724 20:55:32.678510  6520 net.cpp:386] relu6 -> fc6 (in-place)
I0724 20:55:32.678510  6520 net.cpp:141] Setting up relu6
I0724 20:55:32.678510  6520 net.cpp:148] Top shape: 50 4096 (204800)
I0724 20:55:32.678510  6520 net.cpp:156] Memory required for data: 339730800
I0724 20:55:32.678510  6520 layer_factory.hpp:77] Creating layer drop6
I0724 20:55:32.678510  6520 net.cpp:91] Creating Layer drop6
I0724 20:55:32.678510  6520 net.cpp:425] drop6 <- fc6
I0724 20:55:32.678510  6520 net.cpp:386] drop6 -> fc6 (in-place)
I0724 20:55:32.678510  6520 net.cpp:141] Setting up drop6
I0724 20:55:32.678510  6520 net.cpp:148] Top shape: 50 4096 (204800)
I0724 20:55:32.678510  6520 net.cpp:156] Memory required for data: 340550000
I0724 20:55:32.678510  6520 layer_factory.hpp:77] Creating layer fc7
I0724 20:55:32.678510  6520 net.cpp:91] Creating Layer fc7
I0724 20:55:32.678510  6520 net.cpp:425] fc7 <- fc6
I0724 20:55:32.679513  6520 net.cpp:399] fc7 -> fc7
I0724 20:55:32.906666  6520 net.cpp:141] Setting up fc7
I0724 20:55:32.906666  6520 net.cpp:148] Top shape: 50 4096 (204800)
I0724 20:55:32.906666  6520 net.cpp:156] Memory required for data: 341369200
I0724 20:55:32.906666  6520 layer_factory.hpp:77] Creating layer relu7
I0724 20:55:32.906666  6520 net.cpp:91] Creating Layer relu7
I0724 20:55:32.906666  6520 net.cpp:425] relu7 <- fc7
I0724 20:55:32.906666  6520 net.cpp:386] relu7 -> fc7 (in-place)
I0724 20:55:32.907663  6520 net.cpp:141] Setting up relu7
I0724 20:55:32.907663  6520 net.cpp:148] Top shape: 50 4096 (204800)
I0724 20:55:32.907663  6520 net.cpp:156] Memory required for data: 342188400
I0724 20:55:32.907663  6520 layer_factory.hpp:77] Creating layer drop7
I0724 20:55:32.907663  6520 net.cpp:91] Creating Layer drop7
I0724 20:55:32.907663  6520 net.cpp:425] drop7 <- fc7
I0724 20:55:32.907663  6520 net.cpp:386] drop7 -> fc7 (in-place)
I0724 20:55:32.907663  6520 net.cpp:141] Setting up drop7
I0724 20:55:32.907663  6520 net.cpp:148] Top shape: 50 4096 (204800)
I0724 20:55:32.907663  6520 net.cpp:156] Memory required for data: 343007600
I0724 20:55:32.907663  6520 layer_factory.hpp:77] Creating layer fc8
I0724 20:55:32.907663  6520 net.cpp:91] Creating Layer fc8
I0724 20:55:32.907663  6520 net.cpp:425] fc8 <- fc7
I0724 20:55:32.907663  6520 net.cpp:399] fc8 -> fc8
I0724 20:55:32.964702  6520 net.cpp:141] Setting up fc8
I0724 20:55:32.964702  6520 net.cpp:148] Top shape: 50 1000 (50000)
I0724 20:55:32.964702  6520 net.cpp:156] Memory required for data: 343207600
I0724 20:55:32.964702  6520 layer_factory.hpp:77] Creating layer fc8_fc8_0_split
I0724 20:55:32.964702  6520 net.cpp:91] Creating Layer fc8_fc8_0_split
I0724 20:55:32.964702  6520 net.cpp:425] fc8_fc8_0_split <- fc8
I0724 20:55:32.964702  6520 net.cpp:399] fc8_fc8_0_split -> fc8_fc8_0_split_0
I0724 20:55:32.964702  6520 net.cpp:399] fc8_fc8_0_split -> fc8_fc8_0_split_1
I0724 20:55:32.964702  6520 net.cpp:141] Setting up fc8_fc8_0_split
I0724 20:55:32.964702  6520 net.cpp:148] Top shape: 50 1000 (50000)
I0724 20:55:32.964702  6520 net.cpp:148] Top shape: 50 1000 (50000)
I0724 20:55:32.964702  6520 net.cpp:156] Memory required for data: 343607600
I0724 20:55:32.964702  6520 layer_factory.hpp:77] Creating layer accuracy
I0724 20:55:32.964702  6520 net.cpp:91] Creating Layer accuracy
I0724 20:55:32.964702  6520 net.cpp:425] accuracy <- fc8_fc8_0_split_0
I0724 20:55:32.964702  6520 net.cpp:425] accuracy <- label_data_1_split_0
I0724 20:55:32.964702  6520 net.cpp:399] accuracy -> accuracy
I0724 20:55:32.964702  6520 net.cpp:141] Setting up accuracy
I0724 20:55:32.965703  6520 net.cpp:148] Top shape: (1)
I0724 20:55:32.965703  6520 net.cpp:156] Memory required for data: 343607604
I0724 20:55:32.965703  6520 layer_factory.hpp:77] Creating layer loss
I0724 20:55:32.965703  6520 net.cpp:91] Creating Layer loss
I0724 20:55:32.965703  6520 net.cpp:425] loss <- fc8_fc8_0_split_1
I0724 20:55:32.965703  6520 net.cpp:425] loss <- label_data_1_split_1
I0724 20:55:32.965703  6520 net.cpp:399] loss -> loss
I0724 20:55:32.965703  6520 layer_factory.hpp:77] Creating layer loss
I0724 20:55:32.965703  6520 net.cpp:141] Setting up loss
I0724 20:55:32.965703  6520 net.cpp:148] Top shape: (1)
I0724 20:55:32.965703  6520 net.cpp:151]     with loss weight 1
I0724 20:55:32.965703  6520 net.cpp:156] Memory required for data: 343607608
I0724 20:55:32.965703  6520 net.cpp:217] loss needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:219] accuracy does not need backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] fc8_fc8_0_split needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] fc8 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] drop7 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] relu7 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] fc7 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] drop6 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] relu6 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] fc6 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] pool5 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] relu5 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] conv5 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] relu4 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] conv4 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] relu3 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] conv3 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] norm2 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] pool2 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] relu2 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] conv2 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] norm1 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] pool1 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] relu1 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:217] conv1 needs backward computation.
I0724 20:55:32.965703  6520 net.cpp:219] label_data_1_split does not need backward computation.
I0724 20:55:32.965703  6520 net.cpp:219] data does not need backward computation.
I0724 20:55:32.965703  6520 net.cpp:261] This network produces output accuracy
I0724 20:55:32.965703  6520 net.cpp:261] This network produces output loss
I0724 20:55:32.965703  6520 *** Check failure stack trace: ***

Hossein Hasanpour

unread,
Jul 25, 2016, 12:19:21 AM7/25/16
to Caffe Users
try smaller batch sizes, currently the network does not fit in your GPU memory along with the test batches.
make sure when you do change the batchsize, reflect the necessary changes in the solver file as well (test iter,max_iter, test_interval)
...

par...@unlv.nevada.edu

unread,
Jul 25, 2016, 12:22:40 AM7/25/16
to Caffe Users
With a 650 GT, I think you need smaller batches.
...

Gedditoffme

unread,
Jul 28, 2016, 5:45:03 AM7/28/16
to Caffe Users
Thanks - I reduced the batch size to 64 and it worked fine.

Been training on and off for a couple of days, got to 65,000 iterations and accuracy is showing approx. 60% test accuracy, and doesn't seem to be improving much. Training loss is still improving, so possibly overfitting.to training data.

Is this sort of performance (plateau at 60% accuracy) normal for Caffenet & Imagenet data?


...

Kağan İncetan

unread,
Sep 22, 2017, 8:50:45 AM9/22/17
to Caffe Users
Hi, what modifications do I need after changing batch size?
Thanks

28 Temmuz 2016 Perşembe 11:45:03 UTC+2 tarihinde Gedditoffme yazdı:
Reply all
Reply to author
Forward
0 new messages