Can I turn off some of the Intel Caffe Resnet_50 logging?

62 views

ImageNetResNetcaffedatafinetuneimage_datalogprototxtsavetesting

Skip to first unread message

Matthew Brown

unread,

Mar 22, 2018, 10:50:16 AM3/22/18

to Caffe Users

Hello all,

This is my first post here so I hope I am giving all the needed info and asking this the right way.

I have been testing Caffe using the Imagenet dataset and Resnet_50 but am getting huge log files due to all the data output from the run.

My system is a dual processor Intel 6148s with 384GB of memory running RHEL 7.4 and IntelCaffe 1.1.0

build/tools/caffe --version
caffe version 1.1.0
Debug build (NDEBUG not #defined)

[root@ caffe]# uname -a
Linux  3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@ caffe]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    1
Core(s) per socket:    20
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping:              4
CPU MHz:               3099.937
CPU max MHz:           3700.0000
CPU min MHz:           1000.0000
BogoMIPS:              4800.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0-19
NUMA node1 CPU(s):     20-39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req
[root@ caffe]# free
              total        used        free      shared  buff/cache   available
Mem:      395706544     4442436   104654772       50712   286609336   389896192
Swap:       4194300           0     4194300
[root@aist01 caffe]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0 894.3G  0 disk 
â””â”€sda1            8:1    0 894.3G  0 part /local_data
sdb               8:16   0 894.3G  0 disk 
â””â”€sdb1            8:17   0 894.3G  0 part /local_logs
sdc               8:32   0 119.2G  0 disk 
â”œâ”€sdc1            8:33   0    50M  0 part /boot/efi
â”œâ”€sdc2            8:34   0   512M  0 part /boot
â”œâ”€sdc3            8:35   0     4G  0 part [SWAP]
â””â”€sdc4            8:36   0 114.6G  0 part 
  â””â”€system-root 253:0    0 114.6G  0 lvm  /
[root@ caffe]#

I run the training using the command:

./scripts/run_intelcaffe.sh --solver models/intel_optimized_models/resnet_50/skx/solver_dummydata.prototxt

I have the solver_dummydata.prototxt edited as so:

net: "models/intel_optimized_models/resnet_50/skx/train_val.prototxt" (<- Edited prototxt with imagenet data)
test_iter: 1000
test_interval: 1000
base_lr: 0.007
lr_policy: "poly"
power: 0.6
display: 100
max_iter: 1000
momentum: 0.9
weight_decay: 0.0005
snapshot: 10000
snapshot_prefix: "/local_logs/resnet_50/skx/resnet_50_train"    (<- I moved the snapshots to my larger storage drive so that helped some)
solver_mode: CPU

My train_val.prototxt looks like this at the top and was copied from caffe/models/intel_optimized_models/resnet_50/train_val.prototxt :

name: "ResNet-50"
bn_stats_batch_size: 50
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    mirror: true
    crop_size: 224
    scale: 0.0078125
    mean_value: 104
    mean_value: 117
    mean_value: 123
    random_aspect_ratio_param {
      min_area_ratio: 0.08
      max_area_ratio: 1
      aspect_ratio_change: 0.75
      resize_param {
        interp_mode: CUBIC
      }
    }
  }
  data_param {
    source: "/local_data/imagenet/resized/ilsvrc12_train_lmdb"
    batch_size: 50
    backend: LMDB
    prefetch: 2
    shuffle: true
  }
}
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    mirror: false
    crop_size: 224
    scale: 0.0078125
    mean_value: 104
    mean_value: 117
    mean_value: 123
    random_resize_param {
      min_size: 256
      max_size: 256
      resize_param {
        interp_mode: CUBIC
      }
    }
  }
  data_param {
    source: "/local_data/imagenet/resized/ilsvrc12_val_lmdb"
    batch_size: 50
    backend: LMDB
  }
}

I have edited the run_intelcaffe.sh script to also move the log files to my local_logs drive:

#result_dir=${root_dir}/"result-`date +%Y%m%d%H%M%S`"  #Changing directory to move logs to larger drive
result_dir="/local_logs/result-`date +%Y%m%d%H%M%S`"

My remaining issue is that when I run the training even though I change the solver to display every 100 iterations or more for every iteration I get:

W0322 04:40:39.925824 348930 net.cpp:2198] DELTA TIME (Computation + Communication): 0.126519 sec
W0322 04:40:39.925829 348930 net.cpp:2206] ####################################################
W0322 04:40:39.925833 348930 net.cpp:2207] 
I0322 04:40:39.925848 348930 solver.cpp:363] iter 22, forward_backward_update_time: 126 ms
W0322 04:40:40.054692 348930 net.cpp:2119] 
W0322 04:40:40.054731 348930 net.cpp:2120] ####################################################
W0322 04:40:40.054738 348930 net.cpp:2140] DELTA FORWARD TIME: 41.628 ms
W0322 04:40:40.054744 348930 net.cpp:2142] LAYER-0 Data: forward_time: 1.636 ms
W0322 04:40:40.054750 348930 net.cpp:2142] LAYER-1 Split: forward_time: 0.002 ms
W0322 04:40:40.054755 348930 net.cpp:2142] LAYER-2 Convolution: forward_time: 0.588 ms
W0322 04:40:40.054760 348930 net.cpp:2142] LAYER-3 BatchNorm: forward_time: 0.726 ms
W0322 04:40:40.054765 348930 net.cpp:2142] LAYER-4 ReLU: forward_time: 0.404 ms
W0322 04:40:40.054770 348930 net.cpp:2142] LAYER-5 Pooling: forward_time: 0.272 ms
W0322 04:40:40.054775 348930 net.cpp:2142] LAYER-6 Split: forward_time: 0.004 ms
W0322 04:40:40.054780 348930 net.cpp:2142] LAYER-7 Convolution: forward_time: 0.596 ms
W0322 04:40:40.054785 348930 net.cpp:2142] LAYER-8 BatchNorm: forward_time: 0.866 ms
W0322 04:40:40.054796 348930 net.cpp:2142] LAYER-9 Convolution: forward_time: 0.178 ms
W0322 04:40:40.054801 348930 net.cpp:2142] LAYER-10 BatchNorm: forward_time: 0.222 ms
W0322 04:40:40.054806 348930 net.cpp:2142] LAYER-11 ReLU: forward_time: 0.09 ms
W0322 04:40:40.054811 348930 net.cpp:2142] LAYER-12 Convolution: forward_time: 0.422 ms
W0322 04:40:40.054826 348930 net.cpp:2142] LAYER-13 BatchNorm: forward_time: 0.234 ms
W0322 04:40:40.054831 348930 net.cpp:2142] LAYER-14 ReLU: forward_time: 0.092 ms
W0322 04:40:40.054836 348930 net.cpp:2142] LAYER-15 Convolution: forward_time: 0.59 ms
W0322 04:40:40.054841 348930 net.cpp:2142] LAYER-16 BatchNorm: forward_time: 0.786 ms
W0322 04:40:40.054846 348930 net.cpp:2142] LAYER-17 Eltwise: forward_time: 0.888 ms
W0322 04:40:40.054850 348930 net.cpp:2142] LAYER-18 ReLU: forward_time: 0.152 ms
W0322 04:40:40.054854 348930 net.cpp:2142] LAYER-19 Split: forward_time: 0.004 ms
W0322 04:40:40.054859 348930 net.cpp:2142] LAYER-20 Convolution: forward_time: 0.388 ms
W0322 04:40:40.054863 348930 net.cpp:2142] LAYER-21 BatchNorm: forward_time: 0.218 ms
W0322 04:40:40.054868 348930 net.cpp:2142] LAYER-22 ReLU: forward_time: 0.08 ms
W0322 04:40:40.054873 348930 net.cpp:2142] LAYER-23 Convolution: forward_time: 0.416 ms
W0322 04:40:40.054877 348930 net.cpp:2142] LAYER-24 BatchNorm: forward_time: 0.232 ms
W0322 04:40:40.054883 348930 net.cpp:2142] LAYER-25 ReLU: forward_time: 0.098 ms
W0322 04:40:40.054888 348930 net.cpp:2142] LAYER-26 Convolution: forward_time: 0.336 ms
W0322 04:40:40.054891 348930 net.cpp:2142] LAYER-27 BatchNorm: forward_time: 0.74 ms
W0322 04:40:40.054896 348930 net.cpp:2142] LAYER-28 Eltwise: forward_time: 0.524 ms
W0322 04:40:40.054900 348930 net.cpp:2142] LAYER-29 ReLU: forward_time: 0.136 ms
W0322 04:40:40.054905 348930 net.cpp:2142] LAYER-30 Split: forward_time: 0.002 ms
W0322 04:40:40.054909 348930 net.cpp:2142] LAYER-31 Convolution: forward_time: 0.334 ms
W0322 04:40:40.054914 348930 net.cpp:2142] LAYER-32 BatchNorm: forward_time: 0.224 ms
W0322 04:40:40.054919 348930 net.cpp:2142] LAYER-33 ReLU: forward_time: 0.09 ms
W0322 04:40:40.054924 348930 net.cpp:2142] LAYER-34 Convolution: forward_time: 0.404 ms
W0322 04:40:40.054929 348930 net.cpp:2142] LAYER-35 BatchNorm: forward_time: 0.18 ms
W0322 04:40:40.054932 348930 net.cpp:2142] LAYER-36 ReLU: forward_time: 0.056 ms
W0322 04:40:40.054937 348930 net.cpp:2142] LAYER-37 Convolution: forward_time: 0.394 ms
W0322 04:40:40.054942 348930 net.cpp:2142] LAYER-38 BatchNorm: forward_time: 0.674 ms
W0322 04:40:40.054946 348930 net.cpp:2142] LAYER-39 Eltwise: forward_time: 0.508 ms
W0322 04:40:40.054951 348930 net.cpp:2142] LAYER-40 ReLU: forward_time: 0.122 ms
W0322 04:40:40.054955 348930 net.cpp:2142] LAYER-41 Split: forward_time: 0.002 ms
W0322 04:40:40.054960 348930 net.cpp:2142] LAYER-42 Convolution: forward_time: 0.492 ms
W0322 04:40:40.054965 348930 net.cpp:2142] LAYER-43 BatchNorm: forward_time: 0.368 ms
W0322 04:40:40.054970 348930 net.cpp:2142] LAYER-44 Convolution: forward_time: 0.406 ms
W0322 04:40:40.054975 348930 net.cpp:2142] LAYER-45 BatchNorm: forward_time: 0.382 ms
W0322 04:40:40.055012 348930 net.cpp:2142] LAYER-46 ReLU: forward_time: 0.152 ms
W0322 04:40:40.055018 348930 net.cpp:2142] LAYER-47 Convolution: forward_time: 0.624 ms
W0322 04:40:40.055023 348930 net.cpp:2142] LAYER-48 BatchNorm: forward_time: 0.076 ms
W0322 04:40:40.055028 348930 net.cpp:2142] LAYER-49 ReLU: forward_time: 0.028 ms
W0322 04:40:40.055033 348930 net.cpp:2142] LAYER-50 Convolution: forward_time: 0.248 ms
W0322 04:40:40.055037 348930 net.cpp:2142] LAYER-51 BatchNorm: forward_time: 0.312 ms
W0322 04:40:40.055042 348930 net.cpp:2142] LAYER-52 Eltwise: forward_time: 0.406 ms
W0322 04:40:40.055047 348930 net.cpp:2142] LAYER-53 ReLU: forward_time: 0.044 ms
W0322 04:40:40.055052 348930 net.cpp:2142] LAYER-54 Split: forward_time: 0 ms
W0322 04:40:40.055058 348930 net.cpp:2142] LAYER-55 Convolution: forward_time: 0.312 ms
W0322 04:40:40.055083 348930 net.cpp:2142] LAYER-56 BatchNorm: forward_time: 0.084 ms
W0322 04:40:40.055093 348930 net.cpp:2142] LAYER-57 ReLU: forward_time: 0.03 ms
W0322 04:40:40.055099 348930 net.cpp:2142] LAYER-58 Convolution: forward_time: 0.412 ms
W0322 04:40:40.055104 348930 net.cpp:2142] LAYER-59 BatchNorm: forward_time: 0.082 ms
W0322 04:40:40.055109 348930 net.cpp:2142] LAYER-60 ReLU: forward_time: 0.028 ms
W0322 04:40:40.055112 348930 net.cpp:2142] LAYER-61 Convolution: forward_time: 0.276 ms
W0322 04:40:40.055117 348930 net.cpp:2142] LAYER-62 BatchNorm: forward_time: 0.344 ms
W0322 04:40:40.055122 348930 net.cpp:2142] LAYER-63 Eltwise: forward_time: 0.23 ms
W0322 04:40:40.055127 348930 net.cpp:2142] LAYER-64 ReLU: forward_time: 0.052 ms
W0322 04:40:40.055132 348930 net.cpp:2142] LAYER-65 Split: forward_time: 0.002 ms
W0322 04:40:40.055137 348930 net.cpp:2142] LAYER-66 Convolution: forward_time: 0.298 ms
W0322 04:40:40.055142 348930 net.cpp:2142] LAYER-67 BatchNorm: forward_time: 0.128 ms
W0322 04:40:40.055146 348930 net.cpp:2142] LAYER-68 ReLU: forward_time: 0.064 ms
W0322 04:40:40.055151 348930 net.cpp:2142] LAYER-69 Convolution: forward_time: 0.404 ms
W0322 04:40:40.055156 348930 net.cpp:2142] LAYER-70 BatchNorm: forward_time: 0.114 ms
W0322 04:40:40.055161 348930 net.cpp:2142] LAYER-71 ReLU: forward_time: 0.054 ms
W0322 04:40:40.055166 348930 net.cpp:2142] LAYER-72 Convolution: forward_time: 0.268 ms
W0322 04:40:40.055171 348930 net.cpp:2142] LAYER-73 BatchNorm: forward_time: 0.348 ms
W0322 04:40:40.055176 348930 net.cpp:2142] LAYER-74 Eltwise: forward_time: 0.27 ms
W0322 04:40:40.055181 348930 net.cpp:2142] LAYER-75 ReLU: forward_time: 0.052 ms
W0322 04:40:40.055186 348930 net.cpp:2142] LAYER-76 Split: forward_time: 0.002 ms
W0322 04:40:40.055191 348930 net.cpp:2142] LAYER-77 Convolution: forward_time: 0.306 ms
W0322 04:40:40.055196 348930 net.cpp:2142] LAYER-78 BatchNorm: forward_time: 0.128 ms
W0322 04:40:40.055200 348930 net.cpp:2142] LAYER-79 ReLU: forward_time: 0.048 ms
W0322 04:40:40.055205 348930 net.cpp:2142] LAYER-80 Convolution: forward_time: 0.406 ms
W0322 04:40:40.055210 348930 net.cpp:2142] LAYER-81 BatchNorm: forward_time: 0.122 ms
W0322 04:40:40.055215 348930 net.cpp:2142] LAYER-82 ReLU: forward_time: 0.048 ms
W0322 04:40:40.055219 348930 net.cpp:2142] LAYER-83 Convolution: forward_time: 0.268 ms
W0322 04:40:40.055224 348930 net.cpp:2142] LAYER-84 BatchNorm: forward_time: 0.362 ms
W0322 04:40:40.055229 348930 net.cpp:2142] LAYER-85 Eltwise: forward_time: 0.272 ms
W0322 04:40:40.055234 348930 net.cpp:2142] LAYER-86 ReLU: forward_time: 0.05 ms
W0322 04:40:40.055239 348930 net.cpp:2142] LAYER-87 Split: forward_time: 0.002 ms
W0322 04:40:40.055244 348930 net.cpp:2142] LAYER-88 Convolution: forward_time: 0.626 ms
W0322 04:40:40.055249 348930 net.cpp:2142] LAYER-89 BatchNorm: forward_time: 0.192 ms
W0322 04:40:40.055253 348930 net.cpp:2142] LAYER-90 Convolution: forward_time: 0.408 ms
W0322 04:40:40.055258 348930 net.cpp:2142] LAYER-91 BatchNorm: forward_time: 0.232 ms
W0322 04:40:40.055263 348930 net.cpp:2142] LAYER-92 ReLU: forward_time: 0.088 ms
W0322 04:40:40.055269 348930 net.cpp:2142] LAYER-93 Convolution: forward_time: 0.692 ms
W0322 04:40:40.055274 348930 net.cpp:2142] LAYER-94 BatchNorm: forward_time: 0.044 ms
W0322 04:40:40.055286 348930 net.cpp:2142] LAYER-95 ReLU: forward_time: 0.016 ms
W0322 04:40:40.055291 348930 net.cpp:2142] LAYER-96 Convolution: forward_time: 0.338 ms
W0322 04:40:40.055296 348930 net.cpp:2142] LAYER-97 BatchNorm: forward_time: 0.204 ms
W0322 04:40:40.055302 348930 net.cpp:2142] LAYER-98 Eltwise: forward_time: 0.216 ms
W0322 04:40:40.055307 348930 net.cpp:2142] LAYER-99 ReLU: forward_time: 0.06 ms
W0322 04:40:40.055312 348930 net.cpp:2142] LAYER-100 Split: forward_time: 0 ms
W0322 04:40:40.055317 348930 net.cpp:2142] LAYER-101 Convolution: forward_time: 0.332 ms
W0322 04:40:40.055322 348930 net.cpp:2142] LAYER-102 BatchNorm: forward_time: 0.044 ms
W0322 04:40:40.055327 348930 net.cpp:2142] LAYER-103 ReLU: forward_time: 0.018 ms
W0322 04:40:40.055332 348930 net.cpp:2142] LAYER-104 Convolution: forward_time: 0.42 ms
W0322 04:40:40.055346 348930 net.cpp:2142] LAYER-105 BatchNorm: forward_time: 0.052 ms
W0322 04:40:40.055354 348930 net.cpp:2142] LAYER-106 ReLU: forward_time: 0.02 ms
W0322 04:40:40.055361 348930 net.cpp:2142] LAYER-107 Convolution: forward_time: 0.34 ms
W0322 04:40:40.055366 348930 net.cpp:2142] LAYER-108 BatchNorm: forward_time: 0.204 ms
W0322 04:40:40.055371 348930 net.cpp:2142] LAYER-109 Eltwise: forward_time: 0.164 ms
W0322 04:40:40.055375 348930 net.cpp:2142] LAYER-110 ReLU: forward_time: 0.056 ms
W0322 04:40:40.055380 348930 net.cpp:2142] LAYER-111 Split: forward_time: 0 ms
W0322 04:40:40.055385 348930 net.cpp:2142] LAYER-112 Convolution: forward_time: 0.334 ms
W0322 04:40:40.055390 348930 net.cpp:2142] LAYER-113 BatchNorm: forward_time: 0.042 ms
W0322 04:40:40.055395 348930 net.cpp:2142] LAYER-114 ReLU: forward_time: 0.018 ms
W0322 04:40:40.055400 348930 net.cpp:2142] LAYER-115 Convolution: forward_time: 0.432 ms
W0322 04:40:40.055405 348930 net.cpp:2142] LAYER-116 BatchNorm: forward_time: 0.056 ms
W0322 04:40:40.055410 348930 net.cpp:2142] LAYER-117 ReLU: forward_time: 0.018 ms
W0322 04:40:40.055415 348930 net.cpp:2142] LAYER-118 Convolution: forward_time: 0.334 ms
W0322 04:40:40.055420 348930 net.cpp:2142] LAYER-119 BatchNorm: forward_time: 0.196 ms
W0322 04:40:40.055425 348930 net.cpp:2142] LAYER-120 Eltwise: forward_time: 0.148 ms
W0322 04:40:40.055430 348930 net.cpp:2142] LAYER-121 ReLU: forward_time: 0.046 ms
W0322 04:40:40.055435 348930 net.cpp:2142] LAYER-122 Split: forward_time: 0.004 ms
W0322 04:40:40.055440 348930 net.cpp:2142] LAYER-123 Convolution: forward_time: 0.332 ms
W0322 04:40:40.055445 348930 net.cpp:2142] LAYER-124 BatchNorm: forward_time: 0.042 ms
W0322 04:40:40.055450 348930 net.cpp:2142] LAYER-125 ReLU: forward_time: 0.02 ms
W0322 04:40:40.055455 348930 net.cpp:2142] LAYER-126 Convolution: forward_time: 0.424 ms
W0322 04:40:40.055460 348930 net.cpp:2142] LAYER-127 BatchNorm: forward_time: 0.05 ms
W0322 04:40:40.055465 348930 net.cpp:2142] LAYER-128 ReLU: forward_time: 0.018 ms
W0322 04:40:40.055471 348930 net.cpp:2142] LAYER-129 Convolution: forward_time: 0.344 ms
W0322 04:40:40.055476 348930 net.cpp:2142] LAYER-130 BatchNorm: forward_time: 0.22 ms
W0322 04:40:40.055481 348930 net.cpp:2142] LAYER-131 Eltwise: forward_time: 0.212 ms
W0322 04:40:40.055486 348930 net.cpp:2142] LAYER-132 ReLU: forward_time: 0.056 ms
W0322 04:40:40.055491 348930 net.cpp:2142] LAYER-133 Split: forward_time: 0 ms
W0322 04:40:40.055496 348930 net.cpp:2142] LAYER-134 Convolution: forward_time: 0.342 ms
W0322 04:40:40.055501 348930 net.cpp:2142] LAYER-135 BatchNorm: forward_time: 0.042 ms
W0322 04:40:40.055506 348930 net.cpp:2142] LAYER-136 ReLU: forward_time: 0.018 ms
W0322 04:40:40.055511 348930 net.cpp:2142] LAYER-137 Convolution: forward_time: 0.42 ms
W0322 04:40:40.055516 348930 net.cpp:2142] LAYER-138 BatchNorm: forward_time: 0.052 ms
W0322 04:40:40.055521 348930 net.cpp:2142] LAYER-139 ReLU: forward_time: 0.016 ms
W0322 04:40:40.055526 348930 net.cpp:2142] LAYER-140 Convolution: forward_time: 0.344 ms
W0322 04:40:40.055531 348930 net.cpp:2142] LAYER-141 BatchNorm: forward_time: 0.188 ms
W0322 04:40:40.055536 348930 net.cpp:2142] LAYER-142 Eltwise: forward_time: 0.164 ms
W0322 04:40:40.055549 348930 net.cpp:2142] LAYER-143 ReLU: forward_time: 0.048 ms
W0322 04:40:40.055555 348930 net.cpp:2142] LAYER-144 Split: forward_time: 0.002 ms
W0322 04:40:40.055560 348930 net.cpp:2142] LAYER-145 Convolution: forward_time: 0.338 ms
W0322 04:40:40.055565 348930 net.cpp:2142] LAYER-146 BatchNorm: forward_time: 0.042 ms
W0322 04:40:40.055572 348930 net.cpp:2142] LAYER-147 ReLU: forward_time: 0.018 ms
W0322 04:40:40.055577 348930 net.cpp:2142] LAYER-148 Convolution: forward_time: 0.426 ms
W0322 04:40:40.055582 348930 net.cpp:2142] LAYER-149 BatchNorm: forward_time: 0.052 ms
W0322 04:40:40.055588 348930 net.cpp:2142] LAYER-150 ReLU: forward_time: 0.018 ms
W0322 04:40:40.055593 348930 net.cpp:2142] LAYER-151 Convolution: forward_time: 0.332 ms
W0322 04:40:40.055604 348930 net.cpp:2142] LAYER-152 BatchNorm: forward_time: 0.182 ms
W0322 04:40:40.055610 348930 net.cpp:2142] LAYER-153 Eltwise: forward_time: 0.162 ms
W0322 04:40:40.055619 348930 net.cpp:2142] LAYER-154 ReLU: forward_time: 0.048 ms
W0322 04:40:40.055624 348930 net.cpp:2142] LAYER-155 Split: forward_time: 0.002 ms
W0322 04:40:40.055629 348930 net.cpp:2142] LAYER-156 Convolution: forward_time: 0.56 ms
W0322 04:40:40.055634 348930 net.cpp:2142] LAYER-157 BatchNorm: forward_time: 0.094 ms
W0322 04:40:40.055640 348930 net.cpp:2142] LAYER-158 Convolution: forward_time: 0.538 ms
W0322 04:40:40.055645 348930 net.cpp:2142] LAYER-159 BatchNorm: forward_time: 0.066 ms
W0322 04:40:40.055650 348930 net.cpp:2142] LAYER-160 ReLU: forward_time: 0.028 ms
W0322 04:40:40.055656 348930 net.cpp:2142] LAYER-161 Convolution: forward_time: 0.614 ms
W0322 04:40:40.055661 348930 net.cpp:2142] LAYER-162 BatchNorm: forward_time: 0.032 ms
W0322 04:40:40.055666 348930 net.cpp:2142] LAYER-163 ReLU: forward_time: 0.012 ms
W0322 04:40:40.055671 348930 net.cpp:2142] LAYER-164 Convolution: forward_time: 0.304 ms
W0322 04:40:40.055677 348930 net.cpp:2142] LAYER-165 BatchNorm: forward_time: 0.062 ms
W0322 04:40:40.055682 348930 net.cpp:2142] LAYER-166 Eltwise: forward_time: 0.158 ms
W0322 04:40:40.055687 348930 net.cpp:2142] LAYER-167 ReLU: forward_time: 0.04 ms
W0322 04:40:40.055693 348930 net.cpp:2142] LAYER-168 Split: forward_time: 0 ms
W0322 04:40:40.055698 348930 net.cpp:2142] LAYER-169 Convolution: forward_time: 0.362 ms
W0322 04:40:40.055703 348930 net.cpp:2142] LAYER-170 BatchNorm: forward_time: 0.032 ms
W0322 04:40:40.055708 348930 net.cpp:2142] LAYER-171 ReLU: forward_time: 0.014 ms
W0322 04:40:40.055714 348930 net.cpp:2142] LAYER-172 Convolution: forward_time: 0.55 ms
W0322 04:40:40.055719 348930 net.cpp:2142] LAYER-173 BatchNorm: forward_time: 0.032 ms
W0322 04:40:40.055724 348930 net.cpp:2142] LAYER-174 ReLU: forward_time: 0.012 ms
W0322 04:40:40.055730 348930 net.cpp:2142] LAYER-175 Convolution: forward_time: 0.302 ms
W0322 04:40:40.055735 348930 net.cpp:2142] LAYER-176 BatchNorm: forward_time: 0.068 ms
W0322 04:40:40.055740 348930 net.cpp:2142] LAYER-177 Eltwise: forward_time: 0.098 ms
W0322 04:40:40.055745 348930 net.cpp:2142] LAYER-178 ReLU: forward_time: 0.04 ms
W0322 04:40:40.055752 348930 net.cpp:2142] LAYER-179 Split: forward_time: 0 ms
W0322 04:40:40.055757 348930 net.cpp:2142] LAYER-180 Convolution: forward_time: 0.328 ms
W0322 04:40:40.055761 348930 net.cpp:2142] LAYER-181 BatchNorm: forward_time: 0.03 ms
W0322 04:40:40.055768 348930 net.cpp:2142] LAYER-182 ReLU: forward_time: 0.014 ms
W0322 04:40:40.055773 348930 net.cpp:2142] LAYER-183 Convolution: forward_time: 0.532 ms
W0322 04:40:40.055778 348930 net.cpp:2142] LAYER-184 BatchNorm: forward_time: 0.034 ms
W0322 04:40:40.055783 348930 net.cpp:2142] LAYER-185 ReLU: forward_time: 0.012 ms
W0322 04:40:40.055788 348930 net.cpp:2142] LAYER-186 Convolution: forward_time: 0.296 ms
W0322 04:40:40.055794 348930 net.cpp:2142] LAYER-187 BatchNorm: forward_time: 0.07 ms
W0322 04:40:40.055799 348930 net.cpp:2142] LAYER-188 Eltwise: forward_time: 0.084 ms
W0322 04:40:40.055804 348930 net.cpp:2142] LAYER-189 ReLU: forward_time: 0.038 ms
W0322 04:40:40.055809 348930 net.cpp:2142] LAYER-190 Pooling: forward_time: 0.022 ms
W0322 04:40:40.055826 348930 net.cpp:2142] LAYER-191 InnerProduct: forward_time: 0.21 ms
W0322 04:40:40.055835 348930 net.cpp:2142] LAYER-192 Split: forward_time: 0.002 ms
W0322 04:40:40.055840 348930 net.cpp:2142] LAYER-193 SoftmaxWithLoss: forward_time: 0.052 ms
W0322 04:40:40.055845 348930 net.cpp:2142] LAYER-194 Accuracy: forward_time: 0.108 ms
W0322 04:40:40.055850 348930 net.cpp:2142] LAYER-195 Accuracy: forward_time: 0.096 ms
W0322 04:40:40.055856 348930 net.cpp:2147] 
W0322 04:40:40.055861 348930 net.cpp:2151] DELTA BACKWARD TIME: 67.866 ms
W0322 04:40:40.055868 348930 net.cpp:2153] LAYER-0 Data: backward_time: 0 ms
W0322 04:40:40.055873 348930 net.cpp:2153] LAYER-1 Split: backward_time: 0 ms
W0322 04:40:40.055886 348930 net.cpp:2153] LAYER-2 Convolution: backward_time: 0.95 ms
W0322 04:40:40.055892 348930 net.cpp:2153] LAYER-3 BatchNorm: backward_time: 1.242 ms
W0322 04:40:40.055897 348930 net.cpp:2153] LAYER-4 ReLU: backward_time: 0.288 ms
W0322 04:40:40.055908 348930 net.cpp:2153] LAYER-5 Pooling: backward_time: 0.3 ms
W0322 04:40:40.055914 348930 net.cpp:2153] LAYER-6 Split: backward_time: 0.116 ms
W0322 04:40:40.055919 348930 net.cpp:2153] LAYER-7 Convolution: backward_time: 0.692 ms
W0322 04:40:40.055925 348930 net.cpp:2153] LAYER-8 BatchNorm: backward_time: 0.74 ms
W0322 04:40:40.055930 348930 net.cpp:2153] LAYER-9 Convolution: backward_time: 0.518 ms
W0322 04:40:40.055936 348930 net.cpp:2153] LAYER-10 BatchNorm: backward_time: 0.32 ms
W0322 04:40:40.055941 348930 net.cpp:2153] LAYER-11 ReLU: backward_time: 0.166 ms
W0322 04:40:40.055948 348930 net.cpp:2153] LAYER-12 Convolution: backward_time: 1.12 ms
W0322 04:40:40.055953 348930 net.cpp:2153] LAYER-13 BatchNorm: backward_time: 0.268 ms
W0322 04:40:40.055958 348930 net.cpp:2153] LAYER-14 ReLU: backward_time: 0.058 ms
W0322 04:40:40.055963 348930 net.cpp:2153] LAYER-15 Convolution: backward_time: 0.86 ms
W0322 04:40:40.055969 348930 net.cpp:2153] LAYER-16 BatchNorm: backward_time: 0.86 ms
W0322 04:40:40.055974 348930 net.cpp:2153] LAYER-17 Eltwise: backward_time: 0.004 ms
W0322 04:40:40.055980 348930 net.cpp:2153] LAYER-18 ReLU: backward_time: 0.184 ms
W0322 04:40:40.055985 348930 net.cpp:2153] LAYER-19 Split: backward_time: 0.504 ms
W0322 04:40:40.055990 348930 net.cpp:2153] LAYER-20 Convolution: backward_time: 0.792 ms
W0322 04:40:40.055996 348930 net.cpp:2153] LAYER-21 BatchNorm: backward_time: 0.252 ms
W0322 04:40:40.056001 348930 net.cpp:2153] LAYER-22 ReLU: backward_time: 0.154 ms
W0322 04:40:40.056007 348930 net.cpp:2153] LAYER-23 Convolution: backward_time: 1.112 ms
W0322 04:40:40.056012 348930 net.cpp:2153] LAYER-24 BatchNorm: backward_time: 0.234 ms
W0322 04:40:40.056018 348930 net.cpp:2153] LAYER-25 ReLU: backward_time: 0.046 ms
W0322 04:40:40.056023 348930 net.cpp:2153] LAYER-26 Convolution: backward_time: 0.864 ms
W0322 04:40:40.056030 348930 net.cpp:2153] LAYER-27 BatchNorm: backward_time: 0.914 ms
W0322 04:40:40.056035 348930 net.cpp:2153] LAYER-28 Eltwise: backward_time: 0 ms
W0322 04:40:40.056040 348930 net.cpp:2153] LAYER-29 ReLU: backward_time: 0.192 ms
W0322 04:40:40.056046 348930 net.cpp:2153] LAYER-30 Split: backward_time: 0.464 ms
W0322 04:40:40.056051 348930 net.cpp:2153] LAYER-31 Convolution: backward_time: 0.862 ms
W0322 04:40:40.056056 348930 net.cpp:2153] LAYER-32 BatchNorm: backward_time: 0.216 ms
W0322 04:40:40.056061 348930 net.cpp:2153] LAYER-33 ReLU: backward_time: 0.14 ms
W0322 04:40:40.056067 348930 net.cpp:2153] LAYER-34 Convolution: backward_time: 1.13 ms
W0322 04:40:40.056072 348930 net.cpp:2153] LAYER-35 BatchNorm: backward_time: 0.274 ms
W0322 04:40:40.056078 348930 net.cpp:2153] LAYER-36 ReLU: backward_time: 0.056 ms
W0322 04:40:40.056083 348930 net.cpp:2153] LAYER-37 Convolution: backward_time: 0.966 ms
W0322 04:40:40.056089 348930 net.cpp:2153] LAYER-38 BatchNorm: backward_time: 0.956 ms
W0322 04:40:40.056094 348930 net.cpp:2153] LAYER-39 Eltwise: backward_time: 0 ms
W0322 04:40:40.056100 348930 net.cpp:2153] LAYER-40 ReLU: backward_time: 0.158 ms
W0322 04:40:40.056105 348930 net.cpp:2153] LAYER-41 Split: backward_time: 0.502 ms
W0322 04:40:40.056118 348930 net.cpp:2153] LAYER-42 Convolution: backward_time: 1.3 ms
W0322 04:40:40.056125 348930 net.cpp:2153] LAYER-43 BatchNorm: backward_time: 0.364 ms
W0322 04:40:40.056130 348930 net.cpp:2153] LAYER-44 Convolution: backward_time: 1.008 ms
W0322 04:40:40.056136 348930 net.cpp:2153] LAYER-45 BatchNorm: backward_time: 0.368 ms
W0322 04:40:40.056143 348930 net.cpp:2153] LAYER-46 ReLU: backward_time: 0.096 ms
W0322 04:40:40.056149 348930 net.cpp:2153] LAYER-47 Convolution: backward_time: 1.336 ms
W0322 04:40:40.056154 348930 net.cpp:2153] LAYER-48 BatchNorm: backward_time: 0.128 ms
W0322 04:40:40.056159 348930 net.cpp:2153] LAYER-49 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056165 348930 net.cpp:2153] LAYER-50 Convolution: backward_time: 0.726 ms
W0322 04:40:40.056177 348930 net.cpp:2153] LAYER-51 BatchNorm: backward_time: 0.404 ms
W0322 04:40:40.056183 348930 net.cpp:2153] LAYER-52 Eltwise: backward_time: 0.002 ms
W0322 04:40:40.056192 348930 net.cpp:2153] LAYER-53 ReLU: backward_time: 0.042 ms
W0322 04:40:40.056198 348930 net.cpp:2153] LAYER-54 Split: backward_time: 0.236 ms
W0322 04:40:40.056205 348930 net.cpp:2153] LAYER-55 Convolution: backward_time: 0.78 ms
W0322 04:40:40.056210 348930 net.cpp:2153] LAYER-56 BatchNorm: backward_time: 0.112 ms
W0322 04:40:40.056216 348930 net.cpp:2153] LAYER-57 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056221 348930 net.cpp:2153] LAYER-58 Convolution: backward_time: 0.946 ms
W0322 04:40:40.056227 348930 net.cpp:2153] LAYER-59 BatchNorm: backward_time: 0.114 ms
W0322 04:40:40.056233 348930 net.cpp:2153] LAYER-60 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056238 348930 net.cpp:2153] LAYER-61 Convolution: backward_time: 0.694 ms
W0322 04:40:40.056244 348930 net.cpp:2153] LAYER-62 BatchNorm: backward_time: 0.378 ms
W0322 04:40:40.056251 348930 net.cpp:2153] LAYER-63 Eltwise: backward_time: 0.002 ms
W0322 04:40:40.056255 348930 net.cpp:2153] LAYER-64 ReLU: backward_time: 0.042 ms
W0322 04:40:40.056262 348930 net.cpp:2153] LAYER-65 Split: backward_time: 0.204 ms
W0322 04:40:40.056267 348930 net.cpp:2153] LAYER-66 Convolution: backward_time: 0.774 ms
W0322 04:40:40.056272 348930 net.cpp:2153] LAYER-67 BatchNorm: backward_time: 0.088 ms
W0322 04:40:40.056278 348930 net.cpp:2153] LAYER-68 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056284 348930 net.cpp:2153] LAYER-69 Convolution: backward_time: 0.928 ms
W0322 04:40:40.056289 348930 net.cpp:2153] LAYER-70 BatchNorm: backward_time: 0.104 ms
W0322 04:40:40.056295 348930 net.cpp:2153] LAYER-71 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056301 348930 net.cpp:2153] LAYER-72 Convolution: backward_time: 0.756 ms
W0322 04:40:40.056308 348930 net.cpp:2153] LAYER-73 BatchNorm: backward_time: 0.468 ms
W0322 04:40:40.056313 348930 net.cpp:2153] LAYER-74 Eltwise: backward_time: 0.004 ms
W0322 04:40:40.056318 348930 net.cpp:2153] LAYER-75 ReLU: backward_time: 0.044 ms
W0322 04:40:40.056324 348930 net.cpp:2153] LAYER-76 Split: backward_time: 0.22 ms
W0322 04:40:40.056329 348930 net.cpp:2153] LAYER-77 Convolution: backward_time: 0.76 ms
W0322 04:40:40.056335 348930 net.cpp:2153] LAYER-78 BatchNorm: backward_time: 0.094 ms
W0322 04:40:40.056341 348930 net.cpp:2153] LAYER-79 ReLU: backward_time: 0.018 ms
W0322 04:40:40.056347 348930 net.cpp:2153] LAYER-80 Convolution: backward_time: 0.938 ms
W0322 04:40:40.056352 348930 net.cpp:2153] LAYER-81 BatchNorm: backward_time: 0.086 ms
W0322 04:40:40.056358 348930 net.cpp:2153] LAYER-82 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056365 348930 net.cpp:2153] LAYER-83 Convolution: backward_time: 0.774 ms
W0322 04:40:40.056370 348930 net.cpp:2153] LAYER-84 BatchNorm: backward_time: 0.424 ms
W0322 04:40:40.056375 348930 net.cpp:2153] LAYER-85 Eltwise: backward_time: 0.002 ms
W0322 04:40:40.056381 348930 net.cpp:2153] LAYER-86 ReLU: backward_time: 0.064 ms
W0322 04:40:40.056386 348930 net.cpp:2153] LAYER-87 Split: backward_time: 0.236 ms
W0322 04:40:40.056393 348930 net.cpp:2153] LAYER-88 Convolution: backward_time: 1.392 ms
W0322 04:40:40.056398 348930 net.cpp:2153] LAYER-89 BatchNorm: backward_time: 0.218 ms
W0322 04:40:40.056411 348930 net.cpp:2153] LAYER-90 Convolution: backward_time: 1.05 ms
W0322 04:40:40.056418 348930 net.cpp:2153] LAYER-91 BatchNorm: backward_time: 0.188 ms
W0322 04:40:40.056424 348930 net.cpp:2153] LAYER-92 ReLU: backward_time: 0.05 ms
W0322 04:40:40.056430 348930 net.cpp:2153] LAYER-93 Convolution: backward_time: 1.212 ms
W0322 04:40:40.056437 348930 net.cpp:2153] LAYER-94 BatchNorm: backward_time: 0.04 ms
W0322 04:40:40.056442 348930 net.cpp:2153] LAYER-95 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056448 348930 net.cpp:2153] LAYER-96 Convolution: backward_time: 0.684 ms
W0322 04:40:40.056454 348930 net.cpp:2153] LAYER-97 BatchNorm: backward_time: 0.272 ms
W0322 04:40:40.056466 348930 net.cpp:2153] LAYER-98 Eltwise: backward_time: 0.004 ms
W0322 04:40:40.056473 348930 net.cpp:2153] LAYER-99 ReLU: backward_time: 0.034 ms
W0322 04:40:40.056483 348930 net.cpp:2153] LAYER-100 Split: backward_time: 0.15 ms
W0322 04:40:40.056488 348930 net.cpp:2153] LAYER-101 Convolution: backward_time: 0.74 ms
W0322 04:40:40.056494 348930 net.cpp:2153] LAYER-102 BatchNorm: backward_time: 0.052 ms
W0322 04:40:40.056500 348930 net.cpp:2153] LAYER-103 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056506 348930 net.cpp:2153] LAYER-104 Convolution: backward_time: 0.984 ms
W0322 04:40:40.056512 348930 net.cpp:2153] LAYER-105 BatchNorm: backward_time: 0.04 ms
W0322 04:40:40.056519 348930 net.cpp:2153] LAYER-106 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056524 348930 net.cpp:2153] LAYER-107 Convolution: backward_time: 0.676 ms
W0322 04:40:40.056530 348930 net.cpp:2153] LAYER-108 BatchNorm: backward_time: 0.27 ms
W0322 04:40:40.056536 348930 net.cpp:2153] LAYER-109 Eltwise: backward_time: 0.002 ms
W0322 04:40:40.056542 348930 net.cpp:2153] LAYER-110 ReLU: backward_time: 0.036 ms
W0322 04:40:40.056548 348930 net.cpp:2153] LAYER-111 Split: backward_time: 0.144 ms
W0322 04:40:40.056555 348930 net.cpp:2153] LAYER-112 Convolution: backward_time: 0.76 ms
W0322 04:40:40.056560 348930 net.cpp:2153] LAYER-113 BatchNorm: backward_time: 0.052 ms
W0322 04:40:40.056565 348930 net.cpp:2153] LAYER-114 ReLU: backward_time: 0.022 ms
W0322 04:40:40.056571 348930 net.cpp:2153] LAYER-115 Convolution: backward_time: 0.978 ms
W0322 04:40:40.056577 348930 net.cpp:2153] LAYER-116 BatchNorm: backward_time: 0.044 ms
W0322 04:40:40.056583 348930 net.cpp:2153] LAYER-117 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056589 348930 net.cpp:2153] LAYER-118 Convolution: backward_time: 0.676 ms
W0322 04:40:40.056596 348930 net.cpp:2153] LAYER-119 BatchNorm: backward_time: 0.264 ms
W0322 04:40:40.056602 348930 net.cpp:2153] LAYER-120 Eltwise: backward_time: 0.002 ms
W0322 04:40:40.056607 348930 net.cpp:2153] LAYER-121 ReLU: backward_time: 0.04 ms
W0322 04:40:40.056613 348930 net.cpp:2153] LAYER-122 Split: backward_time: 0.122 ms
W0322 04:40:40.056619 348930 net.cpp:2153] LAYER-123 Convolution: backward_time: 0.722 ms
W0322 04:40:40.056625 348930 net.cpp:2153] LAYER-124 BatchNorm: backward_time: 0.052 ms
W0322 04:40:40.056632 348930 net.cpp:2153] LAYER-125 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056638 348930 net.cpp:2153] LAYER-126 Convolution: backward_time: 0.984 ms
W0322 04:40:40.056643 348930 net.cpp:2153] LAYER-127 BatchNorm: backward_time: 0.042 ms
W0322 04:40:40.056649 348930 net.cpp:2153] LAYER-128 ReLU: backward_time: 0.012 ms
W0322 04:40:40.056655 348930 net.cpp:2153] LAYER-129 Convolution: backward_time: 0.668 ms
W0322 04:40:40.056661 348930 net.cpp:2153] LAYER-130 BatchNorm: backward_time: 0.248 ms
W0322 04:40:40.056668 348930 net.cpp:2153] LAYER-131 Eltwise: backward_time: 0.002 ms
W0322 04:40:40.056674 348930 net.cpp:2153] LAYER-132 ReLU: backward_time: 0.038 ms
W0322 04:40:40.056679 348930 net.cpp:2153] LAYER-133 Split: backward_time: 0.148 ms
W0322 04:40:40.056684 348930 net.cpp:2153] LAYER-134 Convolution: backward_time: 0.728 ms
W0322 04:40:40.056691 348930 net.cpp:2153] LAYER-135 BatchNorm: backward_time: 0.052 ms
W0322 04:40:40.056697 348930 net.cpp:2153] LAYER-136 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056710 348930 net.cpp:2153] LAYER-137 Convolution: backward_time: 0.988 ms
W0322 04:40:40.056717 348930 net.cpp:2153] LAYER-138 BatchNorm: backward_time: 0.04 ms
W0322 04:40:40.056723 348930 net.cpp:2153] LAYER-139 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056730 348930 net.cpp:2153] LAYER-140 Convolution: backward_time: 0.68 ms
W0322 04:40:40.056735 348930 net.cpp:2153] LAYER-141 BatchNorm: backward_time: 0.242 ms
W0322 04:40:40.056741 348930 net.cpp:2153] LAYER-142 Eltwise: backward_time: 0 ms
W0322 04:40:40.056747 348930 net.cpp:2153] LAYER-143 ReLU: backward_time: 0.04 ms
W0322 04:40:40.056754 348930 net.cpp:2153] LAYER-144 Split: backward_time: 0.164 ms
W0322 04:40:40.056766 348930 net.cpp:2153] LAYER-145 Convolution: backward_time: 0.764 ms
W0322 04:40:40.056773 348930 net.cpp:2153] LAYER-146 BatchNorm: backward_time: 0.052 ms
W0322 04:40:40.056779 348930 net.cpp:2153] LAYER-147 ReLU: backward_time: 0.02 ms
W0322 04:40:40.056789 348930 net.cpp:2153] LAYER-148 Convolution: backward_time: 0.98 ms
W0322 04:40:40.056795 348930 net.cpp:2153] LAYER-149 BatchNorm: backward_time: 0.042 ms
W0322 04:40:40.056802 348930 net.cpp:2153] LAYER-150 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056807 348930 net.cpp:2153] LAYER-151 Convolution: backward_time: 0.688 ms
W0322 04:40:40.056814 348930 net.cpp:2153] LAYER-152 BatchNorm: backward_time: 0.264 ms
W0322 04:40:40.056828 348930 net.cpp:2153] LAYER-153 Eltwise: backward_time: 0.004 ms
W0322 04:40:40.056833 348930 net.cpp:2153] LAYER-154 ReLU: backward_time: 0.034 ms
W0322 04:40:40.056840 348930 net.cpp:2153] LAYER-155 Split: backward_time: 0.154 ms
W0322 04:40:40.056846 348930 net.cpp:2153] LAYER-156 Convolution: backward_time: 1.334 ms
W0322 04:40:40.056852 348930 net.cpp:2153] LAYER-157 BatchNorm: backward_time: 0.076 ms
W0322 04:40:40.056859 348930 net.cpp:2153] LAYER-158 Convolution: backward_time: 1.168 ms
W0322 04:40:40.056864 348930 net.cpp:2153] LAYER-159 BatchNorm: backward_time: 0.094 ms
W0322 04:40:40.056870 348930 net.cpp:2153] LAYER-160 ReLU: backward_time: 0.03 ms
W0322 04:40:40.056876 348930 net.cpp:2153] LAYER-161 Convolution: backward_time: 1.68 ms
W0322 04:40:40.056883 348930 net.cpp:2153] LAYER-162 BatchNorm: backward_time: 0.028 ms
W0322 04:40:40.056890 348930 net.cpp:2153] LAYER-163 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056895 348930 net.cpp:2153] LAYER-164 Convolution: backward_time: 0.654 ms
W0322 04:40:40.056901 348930 net.cpp:2153] LAYER-165 BatchNorm: backward_time: 0.086 ms
W0322 04:40:40.056908 348930 net.cpp:2153] LAYER-166 Eltwise: backward_time: 0.004 ms
W0322 04:40:40.056915 348930 net.cpp:2153] LAYER-167 ReLU: backward_time: 0.028 ms
W0322 04:40:40.056921 348930 net.cpp:2153] LAYER-168 Split: backward_time: 0.096 ms
W0322 04:40:40.056926 348930 net.cpp:2153] LAYER-169 Convolution: backward_time: 0.746 ms
W0322 04:40:40.056932 348930 net.cpp:2153] LAYER-170 BatchNorm: backward_time: 0.028 ms
W0322 04:40:40.056938 348930 net.cpp:2153] LAYER-171 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056946 348930 net.cpp:2153] LAYER-172 Convolution: backward_time: 1.26 ms
W0322 04:40:40.056951 348930 net.cpp:2153] LAYER-173 BatchNorm: backward_time: 0.028 ms
W0322 04:40:40.056957 348930 net.cpp:2153] LAYER-174 ReLU: backward_time: 0.016 ms
W0322 04:40:40.056963 348930 net.cpp:2153] LAYER-175 Convolution: backward_time: 0.68 ms
W0322 04:40:40.056969 348930 net.cpp:2153] LAYER-176 BatchNorm: backward_time: 0.08 ms
W0322 04:40:40.056975 348930 net.cpp:2153] LAYER-177 Eltwise: backward_time: 0.002 ms
W0322 04:40:40.056982 348930 net.cpp:2153] LAYER-178 ReLU: backward_time: 0.034 ms
W0322 04:40:40.056988 348930 net.cpp:2153] LAYER-179 Split: backward_time: 0.096 ms
W0322 04:40:40.056994 348930 net.cpp:2153] LAYER-180 Convolution: backward_time: 0.668 ms
W0322 04:40:40.057000 348930 net.cpp:2153] LAYER-181 BatchNorm: backward_time: 0.028 ms
W0322 04:40:40.057008 348930 net.cpp:2153] LAYER-182 ReLU: backward_time: 0.014 ms
W0322 04:40:40.057013 348930 net.cpp:2153] LAYER-183 Convolution: backward_time: 1.258 ms
W0322 04:40:40.057027 348930 net.cpp:2153] LAYER-184 BatchNorm: backward_time: 0.024 ms
W0322 04:40:40.057034 348930 net.cpp:2153] LAYER-185 ReLU: backward_time: 0.016 ms
W0322 04:40:40.057041 348930 net.cpp:2153] LAYER-186 Convolution: backward_time: 0.712 ms
W0322 04:40:40.057047 348930 net.cpp:2153] LAYER-187 BatchNorm: backward_time: 0.088 ms
W0322 04:40:40.057054 348930 net.cpp:2153] LAYER-188 Eltwise: backward_time: 0.004 ms
W0322 04:40:40.057060 348930 net.cpp:2153] LAYER-189 ReLU: backward_time: 0.026 ms
W0322 04:40:40.057066 348930 net.cpp:2153] LAYER-190 Pooling: backward_time: 0.098 ms
W0322 04:40:40.057072 348930 net.cpp:2153] LAYER-191 InnerProduct: backward_time: 0.774 ms
W0322 04:40:40.057086 348930 net.cpp:2153] LAYER-192 Split: backward_time: 0.02 ms
W0322 04:40:40.057092 348930 net.cpp:2153] LAYER-193 SoftmaxWithLoss: backward_time: 0.008 ms
W0322 04:40:40.057099 348930 net.cpp:2153] LAYER-194 Accuracy: backward_time: 0 ms
W0322 04:40:40.057111 348930 net.cpp:2153] LAYER-195 Accuracy: backward_time: 0 ms
W0322 04:40:40.057117 348930 net.cpp:2158] 
W0322 04:40:40.057123 348930 net.cpp:2162] DELTA UPDATE TIME: 16.256 ms
W0322 04:40:40.057130 348930 net.cpp:2164] LAYER-0 Data: update_time: 0.002 ms
W0322 04:40:40.057137 348930 net.cpp:2164] LAYER-1 Split: update_time: 0 ms
W0322 04:40:40.057143 348930 net.cpp:2164] LAYER-2 Convolution: update_time: 0.054 ms
W0322 04:40:40.057149 348930 net.cpp:2164] LAYER-3 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057157 348930 net.cpp:2164] LAYER-4 ReLU: update_time: 0 ms
W0322 04:40:40.057163 348930 net.cpp:2164] LAYER-5 Pooling: update_time: 0.002 ms
W0322 04:40:40.057169 348930 net.cpp:2164] LAYER-6 Split: update_time: 0 ms
W0322 04:40:40.057175 348930 net.cpp:2164] LAYER-7 Convolution: update_time: 0.04 ms
W0322 04:40:40.057181 348930 net.cpp:2164] LAYER-8 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057188 348930 net.cpp:2164] LAYER-9 Convolution: update_time: 0.01 ms
W0322 04:40:40.057194 348930 net.cpp:2164] LAYER-10 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057200 348930 net.cpp:2164] LAYER-11 ReLU: update_time: 0 ms
W0322 04:40:40.057207 348930 net.cpp:2164] LAYER-12 Convolution: update_time: 0.172 ms
W0322 04:40:40.057214 348930 net.cpp:2164] LAYER-13 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057220 348930 net.cpp:2164] LAYER-14 ReLU: update_time: 0 ms
W0322 04:40:40.057226 348930 net.cpp:2164] LAYER-15 Convolution: update_time: 0.032 ms
W0322 04:40:40.057232 348930 net.cpp:2164] LAYER-16 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057240 348930 net.cpp:2164] LAYER-17 Eltwise: update_time: 0 ms
W0322 04:40:40.057245 348930 net.cpp:2164] LAYER-18 ReLU: update_time: 0 ms
W0322 04:40:40.057251 348930 net.cpp:2164] LAYER-19 Split: update_time: 0 ms
W0322 04:40:40.057258 348930 net.cpp:2164] LAYER-20 Convolution: update_time: 0.028 ms
W0322 04:40:40.057265 348930 net.cpp:2164] LAYER-21 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057271 348930 net.cpp:2164] LAYER-22 ReLU: update_time: 0 ms
W0322 04:40:40.057277 348930 net.cpp:2164] LAYER-23 Convolution: update_time: 0.166 ms
W0322 04:40:40.057283 348930 net.cpp:2164] LAYER-24 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057289 348930 net.cpp:2164] LAYER-25 ReLU: update_time: 0 ms
W0322 04:40:40.057296 348930 net.cpp:2164] LAYER-26 Convolution: update_time: 0.036 ms
W0322 04:40:40.057302 348930 net.cpp:2164] LAYER-27 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057309 348930 net.cpp:2164] LAYER-28 Eltwise: update_time: 0 ms
W0322 04:40:40.057315 348930 net.cpp:2164] LAYER-29 ReLU: update_time: 0 ms
W0322 04:40:40.057322 348930 net.cpp:2164] LAYER-30 Split: update_time: 0 ms
W0322 04:40:40.057328 348930 net.cpp:2164] LAYER-31 Convolution: update_time: 0.032 ms
W0322 04:40:40.057334 348930 net.cpp:2164] LAYER-32 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057340 348930 net.cpp:2164] LAYER-33 ReLU: update_time: 0 ms
W0322 04:40:40.057348 348930 net.cpp:2164] LAYER-34 Convolution: update_time: 0.178 ms
W0322 04:40:40.057353 348930 net.cpp:2164] LAYER-35 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057368 348930 net.cpp:2164] LAYER-36 ReLU: update_time: 0 ms
W0322 04:40:40.057374 348930 net.cpp:2164] LAYER-37 Convolution: update_time: 0.032 ms
W0322 04:40:40.057381 348930 net.cpp:2164] LAYER-38 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057389 348930 net.cpp:2164] LAYER-39 Eltwise: update_time: 0.002 ms
W0322 04:40:40.057394 348930 net.cpp:2164] LAYER-40 ReLU: update_time: 0 ms
W0322 04:40:40.057401 348930 net.cpp:2164] LAYER-41 Split: update_time: 0 ms
W0322 04:40:40.057407 348930 net.cpp:2164] LAYER-42 Convolution: update_time: 0.304 ms
W0322 04:40:40.057420 348930 net.cpp:2164] LAYER-43 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.057428 348930 net.cpp:2164] LAYER-44 Convolution: update_time: 0.172 ms
W0322 04:40:40.057435 348930 net.cpp:2164] LAYER-45 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057441 348930 net.cpp:2164] LAYER-46 ReLU: update_time: 0 ms
W0322 04:40:40.057447 348930 net.cpp:2164] LAYER-47 Convolution: update_time: 0.282 ms
W0322 04:40:40.057457 348930 net.cpp:2164] LAYER-48 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057464 348930 net.cpp:2164] LAYER-49 ReLU: update_time: 0 ms
W0322 04:40:40.057471 348930 net.cpp:2164] LAYER-50 Convolution: update_time: 0.21 ms
W0322 04:40:40.057477 348930 net.cpp:2164] LAYER-51 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057484 348930 net.cpp:2164] LAYER-52 Eltwise: update_time: 0 ms
W0322 04:40:40.057492 348930 net.cpp:2164] LAYER-53 ReLU: update_time: 0 ms
W0322 04:40:40.057497 348930 net.cpp:2164] LAYER-54 Split: update_time: 0 ms
W0322 04:40:40.057504 348930 net.cpp:2164] LAYER-55 Convolution: update_time: 0.188 ms
W0322 04:40:40.057510 348930 net.cpp:2164] LAYER-56 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057518 348930 net.cpp:2164] LAYER-57 ReLU: update_time: 0 ms
W0322 04:40:40.057523 348930 net.cpp:2164] LAYER-58 Convolution: update_time: 0.274 ms
W0322 04:40:40.057530 348930 net.cpp:2164] LAYER-59 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057536 348930 net.cpp:2164] LAYER-60 ReLU: update_time: 0 ms
W0322 04:40:40.057543 348930 net.cpp:2164] LAYER-61 Convolution: update_time: 0.2 ms
W0322 04:40:40.057550 348930 net.cpp:2164] LAYER-62 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.057556 348930 net.cpp:2164] LAYER-63 Eltwise: update_time: 0 ms
W0322 04:40:40.057564 348930 net.cpp:2164] LAYER-64 ReLU: update_time: 0 ms
W0322 04:40:40.057569 348930 net.cpp:2164] LAYER-65 Split: update_time: 0 ms
W0322 04:40:40.057576 348930 net.cpp:2164] LAYER-66 Convolution: update_time: 0.186 ms
W0322 04:40:40.057582 348930 net.cpp:2164] LAYER-67 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057590 348930 net.cpp:2164] LAYER-68 ReLU: update_time: 0 ms
W0322 04:40:40.057595 348930 net.cpp:2164] LAYER-69 Convolution: update_time: 0.276 ms
W0322 04:40:40.057602 348930 net.cpp:2164] LAYER-70 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057610 348930 net.cpp:2164] LAYER-71 ReLU: update_time: 0 ms
W0322 04:40:40.057615 348930 net.cpp:2164] LAYER-72 Convolution: update_time: 0.206 ms
W0322 04:40:40.057622 348930 net.cpp:2164] LAYER-73 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.057628 348930 net.cpp:2164] LAYER-74 Eltwise: update_time: 0 ms
W0322 04:40:40.057636 348930 net.cpp:2164] LAYER-75 ReLU: update_time: 0 ms
W0322 04:40:40.057641 348930 net.cpp:2164] LAYER-76 Split: update_time: 0 ms
W0322 04:40:40.057648 348930 net.cpp:2164] LAYER-77 Convolution: update_time: 0.196 ms
W0322 04:40:40.057654 348930 net.cpp:2164] LAYER-78 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057662 348930 net.cpp:2164] LAYER-79 ReLU: update_time: 0 ms
W0322 04:40:40.057667 348930 net.cpp:2164] LAYER-80 Convolution: update_time: 0.276 ms
W0322 04:40:40.057674 348930 net.cpp:2164] LAYER-81 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057682 348930 net.cpp:2164] LAYER-82 ReLU: update_time: 0 ms
W0322 04:40:40.057687 348930 net.cpp:2164] LAYER-83 Convolution: update_time: 0.21 ms
W0322 04:40:40.057694 348930 net.cpp:2164] LAYER-84 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057701 348930 net.cpp:2164] LAYER-85 Eltwise: update_time: 0 ms
W0322 04:40:40.057714 348930 net.cpp:2164] LAYER-86 ReLU: update_time: 0 ms
W0322 04:40:40.057721 348930 net.cpp:2164] LAYER-87 Split: update_time: 0 ms
W0322 04:40:40.057729 348930 net.cpp:2164] LAYER-88 Convolution: update_time: 0.38 ms
W0322 04:40:40.057735 348930 net.cpp:2164] LAYER-89 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057742 348930 net.cpp:2164] LAYER-90 Convolution: update_time: 0.232 ms
W0322 04:40:40.057749 348930 net.cpp:2164] LAYER-91 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057756 348930 net.cpp:2164] LAYER-92 ReLU: update_time: 0 ms
W0322 04:40:40.057868 348930 net.cpp:2164] LAYER-93 Convolution: update_time: 0.384 ms
W0322 04:40:40.057879 348930 net.cpp:2164] LAYER-94 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057883 348930 net.cpp:2164] LAYER-95 ReLU: update_time: 0.002 ms
W0322 04:40:40.057888 348930 net.cpp:2164] LAYER-96 Convolution: update_time: 0.216 ms
W0322 04:40:40.057893 348930 net.cpp:2164] LAYER-97 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.057901 348930 net.cpp:2164] LAYER-98 Eltwise: update_time: 0 ms
W0322 04:40:40.057906 348930 net.cpp:2164] LAYER-99 ReLU: update_time: 0 ms
W0322 04:40:40.057910 348930 net.cpp:2164] LAYER-100 Split: update_time: 0 ms
W0322 04:40:40.057914 348930 net.cpp:2164] LAYER-101 Convolution: update_time: 0.208 ms
W0322 04:40:40.057919 348930 net.cpp:2164] LAYER-102 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057924 348930 net.cpp:2164] LAYER-103 ReLU: update_time: 0 ms
W0322 04:40:40.057929 348930 net.cpp:2164] LAYER-104 Convolution: update_time: 0.33 ms
W0322 04:40:40.057932 348930 net.cpp:2164] LAYER-105 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057937 348930 net.cpp:2164] LAYER-106 ReLU: update_time: 0 ms
W0322 04:40:40.057941 348930 net.cpp:2164] LAYER-107 Convolution: update_time: 0.23 ms
W0322 04:40:40.057946 348930 net.cpp:2164] LAYER-108 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.057950 348930 net.cpp:2164] LAYER-109 Eltwise: update_time: 0 ms
W0322 04:40:40.057955 348930 net.cpp:2164] LAYER-110 ReLU: update_time: 0 ms
W0322 04:40:40.057960 348930 net.cpp:2164] LAYER-111 Split: update_time: 0 ms
W0322 04:40:40.057963 348930 net.cpp:2164] LAYER-112 Convolution: update_time: 0.208 ms
W0322 04:40:40.057968 348930 net.cpp:2164] LAYER-113 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057972 348930 net.cpp:2164] LAYER-114 ReLU: update_time: 0 ms
W0322 04:40:40.057977 348930 net.cpp:2164] LAYER-115 Convolution: update_time: 0.328 ms
W0322 04:40:40.057981 348930 net.cpp:2164] LAYER-116 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.057986 348930 net.cpp:2164] LAYER-117 ReLU: update_time: 0 ms
W0322 04:40:40.057991 348930 net.cpp:2164] LAYER-118 Convolution: update_time: 0.206 ms
W0322 04:40:40.057994 348930 net.cpp:2164] LAYER-119 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.057999 348930 net.cpp:2164] LAYER-120 Eltwise: update_time: 0 ms
W0322 04:40:40.058003 348930 net.cpp:2164] LAYER-121 ReLU: update_time: 0.002 ms
W0322 04:40:40.058008 348930 net.cpp:2164] LAYER-122 Split: update_time: 0 ms
W0322 04:40:40.058012 348930 net.cpp:2164] LAYER-123 Convolution: update_time: 0.216 ms
W0322 04:40:40.058017 348930 net.cpp:2164] LAYER-124 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.058022 348930 net.cpp:2164] LAYER-125 ReLU: update_time: 0 ms
W0322 04:40:40.058027 348930 net.cpp:2164] LAYER-126 Convolution: update_time: 0.332 ms
W0322 04:40:40.058030 348930 net.cpp:2164] LAYER-127 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.058035 348930 net.cpp:2164] LAYER-128 ReLU: update_time: 0 ms
W0322 04:40:40.058039 348930 net.cpp:2164] LAYER-129 Convolution: update_time: 0.216 ms
W0322 04:40:40.058043 348930 net.cpp:2164] LAYER-130 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.058048 348930 net.cpp:2164] LAYER-131 Eltwise: update_time: 0 ms
W0322 04:40:40.058053 348930 net.cpp:2164] LAYER-132 ReLU: update_time: 0 ms
W0322 04:40:40.058056 348930 net.cpp:2164] LAYER-133 Split: update_time: 0.002 ms
W0322 04:40:40.058061 348930 net.cpp:2164] LAYER-134 Convolution: update_time: 0.214 ms
W0322 04:40:40.058065 348930 net.cpp:2164] LAYER-135 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.058076 348930 net.cpp:2164] LAYER-136 ReLU: update_time: 0 ms
W0322 04:40:40.058082 348930 net.cpp:2164] LAYER-137 Convolution: update_time: 0.332 ms
W0322 04:40:40.058086 348930 net.cpp:2164] LAYER-138 BatchNorm: update_time: 0.004 ms
W0322 04:40:40.058091 348930 net.cpp:2164] LAYER-139 ReLU: update_time: 0 ms
W0322 04:40:40.058096 348930 net.cpp:2164] LAYER-140 Convolution: update_time: 0.214 ms
W0322 04:40:40.058100 348930 net.cpp:2164] LAYER-141 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.058105 348930 net.cpp:2164] LAYER-142 Eltwise: update_time: 0 ms
W0322 04:40:40.058110 348930 net.cpp:2164] LAYER-143 ReLU: update_time: 0 ms
W0322 04:40:40.058115 348930 net.cpp:2164] LAYER-144 Split: update_time: 0 ms
W0322 04:40:40.058120 348930 net.cpp:2164] LAYER-145 Convolution: update_time: 0.216 ms
W0322 04:40:40.058125 348930 net.cpp:2164] LAYER-146 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.058248 348930 net.cpp:2164] LAYER-147 ReLU: update_time: 0 ms
W0322 04:40:40.058254 348930 net.cpp:2164] LAYER-148 Convolution: update_time: 0.32 ms
W0322 04:40:40.058259 348930 net.cpp:2164] LAYER-149 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.058262 348930 net.cpp:2164] LAYER-150 ReLU: update_time: 0 ms
W0322 04:40:40.058267 348930 net.cpp:2164] LAYER-151 Convolution: update_time: 0.212 ms
W0322 04:40:40.058271 348930 net.cpp:2164] LAYER-152 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.058276 348930 net.cpp:2164] LAYER-153 Eltwise: update_time: 0.002 ms
W0322 04:40:40.058280 348930 net.cpp:2164] LAYER-154 ReLU: update_time: 0 ms
W0322 04:40:40.058284 348930 net.cpp:2164] LAYER-155 Split: update_time: 0 ms
W0322 04:40:40.058290 348930 net.cpp:2164] LAYER-156 Convolution: update_time: 0.672 ms
W0322 04:40:40.058293 348930 net.cpp:2164] LAYER-157 BatchNorm: update_time: 0.01 ms
W0322 04:40:40.058298 348930 net.cpp:2164] LAYER-158 Convolution: update_time: 0.296 ms
W0322 04:40:40.058302 348930 net.cpp:2164] LAYER-159 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.058306 348930 net.cpp:2164] LAYER-160 ReLU: update_time: 0 ms
W0322 04:40:40.058311 348930 net.cpp:2164] LAYER-161 Convolution: update_time: 0.744 ms
W0322 04:40:40.058315 348930 net.cpp:2164] LAYER-162 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.058320 348930 net.cpp:2164] LAYER-163 ReLU: update_time: 0 ms
W0322 04:40:40.058324 348930 net.cpp:2164] LAYER-164 Convolution: update_time: 0.526 ms
W0322 04:40:40.058328 348930 net.cpp:2164] LAYER-165 BatchNorm: update_time: 0.01 ms
W0322 04:40:40.058333 348930 net.cpp:2164] LAYER-166 Eltwise: update_time: 0 ms
W0322 04:40:40.058337 348930 net.cpp:2164] LAYER-167 ReLU: update_time: 0 ms
W0322 04:40:40.058341 348930 net.cpp:2164] LAYER-168 Split: update_time: 0 ms
W0322 04:40:40.058346 348930 net.cpp:2164] LAYER-169 Convolution: update_time: 0.58 ms
W0322 04:40:40.058351 348930 net.cpp:2164] LAYER-170 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.058356 348930 net.cpp:2164] LAYER-171 ReLU: update_time: 0 ms
W0322 04:40:40.058359 348930 net.cpp:2164] LAYER-172 Convolution: update_time: 0.73 ms
W0322 04:40:40.058363 348930 net.cpp:2164] LAYER-173 BatchNorm: update_time: 0.006 ms
W0322 04:40:40.058368 348930 net.cpp:2164] LAYER-174 ReLU: update_time: 0 ms
W0322 04:40:40.058372 348930 net.cpp:2164] LAYER-175 Convolution: update_time: 0.59 ms
W0322 04:40:40.058377 348930 net.cpp:2164] LAYER-176 BatchNorm: update_time: 0.01 ms
W0322 04:40:40.058382 348930 net.cpp:2164] LAYER-177 Eltwise: update_time: 0 ms
W0322 04:40:40.058385 348930 net.cpp:2164] LAYER-178 ReLU: update_time: 0 ms
W0322 04:40:40.058390 348930 net.cpp:2164] LAYER-179 Split: update_time: 0 ms
W0322 04:40:40.058394 348930 net.cpp:2164] LAYER-180 Convolution: update_time: 0.528 ms
W0322 04:40:40.058398 348930 net.cpp:2164] LAYER-181 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.058403 348930 net.cpp:2164] LAYER-182 ReLU: update_time: 0 ms
W0322 04:40:40.058408 348930 net.cpp:2164] LAYER-183 Convolution: update_time: 0.864 ms
W0322 04:40:40.058411 348930 net.cpp:2164] LAYER-184 BatchNorm: update_time: 0.008 ms
W0322 04:40:40.058423 348930 net.cpp:2164] LAYER-185 ReLU: update_time: 0 ms
W0322 04:40:40.058429 348930 net.cpp:2164] LAYER-186 Convolution: update_time: 0.632 ms
W0322 04:40:40.058432 348930 net.cpp:2164] LAYER-187 BatchNorm: update_time: 0.012 ms
W0322 04:40:40.058437 348930 net.cpp:2164] LAYER-188 Eltwise: update_time: 0 ms
W0322 04:40:40.058442 348930 net.cpp:2164] LAYER-189 ReLU: update_time: 0 ms
W0322 04:40:40.058447 348930 net.cpp:2164] LAYER-190 Pooling: update_time: 0 ms
W0322 04:40:40.058451 348930 net.cpp:2164] LAYER-191 InnerProduct: update_time: 1.006 ms
W0322 04:40:40.058456 348930 net.cpp:2164] LAYER-192 Split: update_time: 0 ms
W0322 04:40:40.058461 348930 net.cpp:2164] LAYER-193 SoftmaxWithLoss: update_time: 0 ms
W0322 04:40:40.058465 348930 net.cpp:2164] LAYER-194 Accuracy: update_time: 0 ms
W0322 04:40:40.058470 348930 net.cpp:2164] LAYER-195 Accuracy: update_time: 0 ms
W0322 04:40:40.058482 348930 net.cpp:2169] 
W0322 04:40:40.058490 348930 net.cpp:2172] DELTA CLEAR PARAMETER DIFFS TIME: 1.407 ms
W0322 04:40:40.058495 348930 net.cpp:2173] 
W0322 04:40:40.058501 348930 net.cpp:2178] DELTA START COMMUNICATION TIME: 0 ms
W0322 04:40:40.058504 348930 net.cpp:2180] LAYER-0 Data: startcomm_time: 0 ms
W0322 04:40:40.058509 348930 net.cpp:2180] LAYER-1 Split: startcomm_time: 0 ms
W0322 04:40:40.058514 348930 net.cpp:2180] LAYER-2 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058519 348930 net.cpp:2180] LAYER-3 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058523 348930 net.cpp:2180] LAYER-4 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058528 348930 net.cpp:2180] LAYER-5 Pooling: startcomm_time: 0 ms
W0322 04:40:40.058533 348930 net.cpp:2180] LAYER-6 Split: startcomm_time: 0 ms
W0322 04:40:40.058537 348930 net.cpp:2180] LAYER-7 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058542 348930 net.cpp:2180] LAYER-8 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058547 348930 net.cpp:2180] LAYER-9 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058552 348930 net.cpp:2180] LAYER-10 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058557 348930 net.cpp:2180] LAYER-11 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058560 348930 net.cpp:2180] LAYER-12 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058565 348930 net.cpp:2180] LAYER-13 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058570 348930 net.cpp:2180] LAYER-14 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058574 348930 net.cpp:2180] LAYER-15 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058579 348930 net.cpp:2180] LAYER-16 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058584 348930 net.cpp:2180] LAYER-17 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.058588 348930 net.cpp:2180] LAYER-18 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058593 348930 net.cpp:2180] LAYER-19 Split: startcomm_time: 0 ms
W0322 04:40:40.058598 348930 net.cpp:2180] LAYER-20 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058603 348930 net.cpp:2180] LAYER-21 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058607 348930 net.cpp:2180] LAYER-22 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058611 348930 net.cpp:2180] LAYER-23 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058616 348930 net.cpp:2180] LAYER-24 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058621 348930 net.cpp:2180] LAYER-25 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058625 348930 net.cpp:2180] LAYER-26 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058630 348930 net.cpp:2180] LAYER-27 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058635 348930 net.cpp:2180] LAYER-28 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.058640 348930 net.cpp:2180] LAYER-29 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058645 348930 net.cpp:2180] LAYER-30 Split: startcomm_time: 0 ms
W0322 04:40:40.058650 348930 net.cpp:2180] LAYER-31 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058653 348930 net.cpp:2180] LAYER-32 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058658 348930 net.cpp:2180] LAYER-33 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058662 348930 net.cpp:2180] LAYER-34 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058667 348930 net.cpp:2180] LAYER-35 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058678 348930 net.cpp:2180] LAYER-36 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058683 348930 net.cpp:2180] LAYER-37 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058688 348930 net.cpp:2180] LAYER-38 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058693 348930 net.cpp:2180] LAYER-39 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.058698 348930 net.cpp:2180] LAYER-40 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058703 348930 net.cpp:2180] LAYER-41 Split: startcomm_time: 0 ms
W0322 04:40:40.058707 348930 net.cpp:2180] LAYER-42 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058712 348930 net.cpp:2180] LAYER-43 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058717 348930 net.cpp:2180] LAYER-44 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058722 348930 net.cpp:2180] LAYER-45 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058733 348930 net.cpp:2180] LAYER-46 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058742 348930 net.cpp:2180] LAYER-47 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058748 348930 net.cpp:2180] LAYER-48 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058751 348930 net.cpp:2180] LAYER-49 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058756 348930 net.cpp:2180] LAYER-50 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058761 348930 net.cpp:2180] LAYER-51 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058766 348930 net.cpp:2180] LAYER-52 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.058771 348930 net.cpp:2180] LAYER-53 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058776 348930 net.cpp:2180] LAYER-54 Split: startcomm_time: 0 ms
W0322 04:40:40.058780 348930 net.cpp:2180] LAYER-55 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058785 348930 net.cpp:2180] LAYER-56 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058790 348930 net.cpp:2180] LAYER-57 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058795 348930 net.cpp:2180] LAYER-58 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058800 348930 net.cpp:2180] LAYER-59 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058805 348930 net.cpp:2180] LAYER-60 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058809 348930 net.cpp:2180] LAYER-61 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058820 348930 net.cpp:2180] LAYER-62 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058825 348930 net.cpp:2180] LAYER-63 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.058830 348930 net.cpp:2180] LAYER-64 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058835 348930 net.cpp:2180] LAYER-65 Split: startcomm_time: 0 ms
W0322 04:40:40.058840 348930 net.cpp:2180] LAYER-66 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058845 348930 net.cpp:2180] LAYER-67 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058850 348930 net.cpp:2180] LAYER-68 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058854 348930 net.cpp:2180] LAYER-69 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058858 348930 net.cpp:2180] LAYER-70 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058863 348930 net.cpp:2180] LAYER-71 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058868 348930 net.cpp:2180] LAYER-72 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058873 348930 net.cpp:2180] LAYER-73 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058878 348930 net.cpp:2180] LAYER-74 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.058883 348930 net.cpp:2180] LAYER-75 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058887 348930 net.cpp:2180] LAYER-76 Split: startcomm_time: 0 ms
W0322 04:40:40.058892 348930 net.cpp:2180] LAYER-77 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058897 348930 net.cpp:2180] LAYER-78 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058902 348930 net.cpp:2180] LAYER-79 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058907 348930 net.cpp:2180] LAYER-80 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058912 348930 net.cpp:2180] LAYER-81 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.058917 348930 net.cpp:2180] LAYER-82 ReLU: startcomm_time: 0 ms
W0322 04:40:40.058921 348930 net.cpp:2180] LAYER-83 Convolution: startcomm_time: 0 ms
W0322 04:40:40.058926 348930 net.cpp:2180] LAYER-84 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059069 348930 net.cpp:2180] LAYER-85 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059078 348930 net.cpp:2180] LAYER-86 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059083 348930 net.cpp:2180] LAYER-87 Split: startcomm_time: 0 ms
W0322 04:40:40.059161 348930 net.cpp:2180] LAYER-88 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059167 348930 net.cpp:2180] LAYER-89 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059171 348930 net.cpp:2180] LAYER-90 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059176 348930 net.cpp:2180] LAYER-91 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059180 348930 net.cpp:2180] LAYER-92 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059185 348930 net.cpp:2180] LAYER-93 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059190 348930 net.cpp:2180] LAYER-94 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059238 348930 net.cpp:2180] LAYER-95 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059244 348930 net.cpp:2180] LAYER-96 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059253 348930 net.cpp:2180] LAYER-97 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059259 348930 net.cpp:2180] LAYER-98 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059263 348930 net.cpp:2180] LAYER-99 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059268 348930 net.cpp:2180] LAYER-100 Split: startcomm_time: 0 ms
W0322 04:40:40.059273 348930 net.cpp:2180] LAYER-101 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059276 348930 net.cpp:2180] LAYER-102 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059281 348930 net.cpp:2180] LAYER-103 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059285 348930 net.cpp:2180] LAYER-104 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059290 348930 net.cpp:2180] LAYER-105 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059294 348930 net.cpp:2180] LAYER-106 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059299 348930 net.cpp:2180] LAYER-107 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059303 348930 net.cpp:2180] LAYER-108 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059309 348930 net.cpp:2180] LAYER-109 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059312 348930 net.cpp:2180] LAYER-110 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059316 348930 net.cpp:2180] LAYER-111 Split: startcomm_time: 0 ms
W0322 04:40:40.059321 348930 net.cpp:2180] LAYER-112 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059325 348930 net.cpp:2180] LAYER-113 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059329 348930 net.cpp:2180] LAYER-114 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059334 348930 net.cpp:2180] LAYER-115 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059339 348930 net.cpp:2180] LAYER-116 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059343 348930 net.cpp:2180] LAYER-117 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059347 348930 net.cpp:2180] LAYER-118 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059351 348930 net.cpp:2180] LAYER-119 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059356 348930 net.cpp:2180] LAYER-120 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059360 348930 net.cpp:2180] LAYER-121 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059365 348930 net.cpp:2180] LAYER-122 Split: startcomm_time: 0 ms
W0322 04:40:40.059370 348930 net.cpp:2180] LAYER-123 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059375 348930 net.cpp:2180] LAYER-124 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059378 348930 net.cpp:2180] LAYER-125 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059382 348930 net.cpp:2180] LAYER-126 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059387 348930 net.cpp:2180] LAYER-127 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059391 348930 net.cpp:2180] LAYER-128 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059396 348930 net.cpp:2180] LAYER-129 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059401 348930 net.cpp:2180] LAYER-130 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059404 348930 net.cpp:2180] LAYER-131 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059409 348930 net.cpp:2180] LAYER-132 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059413 348930 net.cpp:2180] LAYER-133 Split: startcomm_time: 0 ms
W0322 04:40:40.059425 348930 net.cpp:2180] LAYER-134 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059432 348930 net.cpp:2180] LAYER-135 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059437 348930 net.cpp:2180] LAYER-136 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059440 348930 net.cpp:2180] LAYER-137 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059445 348930 net.cpp:2180] LAYER-138 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059450 348930 net.cpp:2180] LAYER-139 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059455 348930 net.cpp:2180] LAYER-140 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059459 348930 net.cpp:2180] LAYER-141 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059464 348930 net.cpp:2180] LAYER-142 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059469 348930 net.cpp:2180] LAYER-143 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059473 348930 net.cpp:2180] LAYER-144 Split: startcomm_time: 0 ms
W0322 04:40:40.059486 348930 net.cpp:2180] LAYER-145 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059494 348930 net.cpp:2180] LAYER-146 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059499 348930 net.cpp:2180] LAYER-147 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059504 348930 net.cpp:2180] LAYER-148 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059509 348930 net.cpp:2180] LAYER-149 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059514 348930 net.cpp:2180] LAYER-150 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059517 348930 net.cpp:2180] LAYER-151 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059522 348930 net.cpp:2180] LAYER-152 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059527 348930 net.cpp:2180] LAYER-153 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059531 348930 net.cpp:2180] LAYER-154 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059536 348930 net.cpp:2180] LAYER-155 Split: startcomm_time: 0 ms
W0322 04:40:40.059541 348930 net.cpp:2180] LAYER-156 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059546 348930 net.cpp:2180] LAYER-157 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059551 348930 net.cpp:2180] LAYER-158 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059556 348930 net.cpp:2180] LAYER-159 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059561 348930 net.cpp:2180] LAYER-160 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059564 348930 net.cpp:2180] LAYER-161 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059569 348930 net.cpp:2180] LAYER-162 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059574 348930 net.cpp:2180] LAYER-163 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059578 348930 net.cpp:2180] LAYER-164 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059583 348930 net.cpp:2180] LAYER-165 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059588 348930 net.cpp:2180] LAYER-166 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059593 348930 net.cpp:2180] LAYER-167 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059597 348930 net.cpp:2180] LAYER-168 Split: startcomm_time: 0 ms
W0322 04:40:40.059602 348930 net.cpp:2180] LAYER-169 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059607 348930 net.cpp:2180] LAYER-170 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059612 348930 net.cpp:2180] LAYER-171 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059617 348930 net.cpp:2180] LAYER-172 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059622 348930 net.cpp:2180] LAYER-173 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059625 348930 net.cpp:2180] LAYER-174 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059630 348930 net.cpp:2180] LAYER-175 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059635 348930 net.cpp:2180] LAYER-176 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059640 348930 net.cpp:2180] LAYER-177 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059644 348930 net.cpp:2180] LAYER-178 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059649 348930 net.cpp:2180] LAYER-179 Split: startcomm_time: 0 ms
W0322 04:40:40.059654 348930 net.cpp:2180] LAYER-180 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059659 348930 net.cpp:2180] LAYER-181 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059664 348930 net.cpp:2180] LAYER-182 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059674 348930 net.cpp:2180] LAYER-183 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059680 348930 net.cpp:2180] LAYER-184 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059685 348930 net.cpp:2180] LAYER-185 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059690 348930 net.cpp:2180] LAYER-186 Convolution: startcomm_time: 0 ms
W0322 04:40:40.059695 348930 net.cpp:2180] LAYER-187 BatchNorm: startcomm_time: 0 ms
W0322 04:40:40.059700 348930 net.cpp:2180] LAYER-188 Eltwise: startcomm_time: 0 ms
W0322 04:40:40.059705 348930 net.cpp:2180] LAYER-189 ReLU: startcomm_time: 0 ms
W0322 04:40:40.059710 348930 net.cpp:2180] LAYER-190 Pooling: startcomm_time: 0 ms
W0322 04:40:40.059715 348930 net.cpp:2180] LAYER-191 InnerProduct: startcomm_time: 0 ms
W0322 04:40:40.059720 348930 net.cpp:2180] LAYER-192 Split: startcomm_time: 0 ms
W0322 04:40:40.059731 348930 net.cpp:2180] LAYER-193 SoftmaxWithLoss: startcomm_time: 0 ms
W0322 04:40:40.059739 348930 net.cpp:2180] LAYER-194 Accuracy: startcomm_time: 0 ms
W0322 04:40:40.059744 348930 net.cpp:2180] LAYER-195 Accuracy: startcomm_time: 0 ms
W0322 04:40:40.059749 348930 net.cpp:2185] 
W0322 04:40:40.059754 348930 net.cpp:2189] DELTA WAIT COMMUNICATION TIME: 0 ms
W0322 04:40:40.059759 348930 net.cpp:2191] LAYER-0 Data: waitcomm_time: 0 ms
W0322 04:40:40.059764 348930 net.cpp:2191] LAYER-1 Split: waitcomm_time: 0 ms
W0322 04:40:40.059769 348930 net.cpp:2191] LAYER-2 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059774 348930 net.cpp:2191] LAYER-3 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059779 348930 net.cpp:2191] LAYER-4 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059784 348930 net.cpp:2191] LAYER-5 Pooling: waitcomm_time: 0 ms
W0322 04:40:40.059789 348930 net.cpp:2191] LAYER-6 Split: waitcomm_time: 0 ms
W0322 04:40:40.059794 348930 net.cpp:2191] LAYER-7 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059799 348930 net.cpp:2191] LAYER-8 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059804 348930 net.cpp:2191] LAYER-9 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059809 348930 net.cpp:2191] LAYER-10 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059814 348930 net.cpp:2191] LAYER-11 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059824 348930 net.cpp:2191] LAYER-12 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059829 348930 net.cpp:2191] LAYER-13 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059834 348930 net.cpp:2191] LAYER-14 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059839 348930 net.cpp:2191] LAYER-15 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059844 348930 net.cpp:2191] LAYER-16 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059849 348930 net.cpp:2191] LAYER-17 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.059854 348930 net.cpp:2191] LAYER-18 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059859 348930 net.cpp:2191] LAYER-19 Split: waitcomm_time: 0 ms
W0322 04:40:40.059864 348930 net.cpp:2191] LAYER-20 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059868 348930 net.cpp:2191] LAYER-21 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059873 348930 net.cpp:2191] LAYER-22 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059877 348930 net.cpp:2191] LAYER-23 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059882 348930 net.cpp:2191] LAYER-24 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059887 348930 net.cpp:2191] LAYER-25 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059892 348930 net.cpp:2191] LAYER-26 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059897 348930 net.cpp:2191] LAYER-27 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059902 348930 net.cpp:2191] LAYER-28 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.059907 348930 net.cpp:2191] LAYER-29 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059912 348930 net.cpp:2191] LAYER-30 Split: waitcomm_time: 0 ms
W0322 04:40:40.059917 348930 net.cpp:2191] LAYER-31 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059921 348930 net.cpp:2191] LAYER-32 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059926 348930 net.cpp:2191] LAYER-33 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059931 348930 net.cpp:2191] LAYER-34 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059937 348930 net.cpp:2191] LAYER-35 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059948 348930 net.cpp:2191] LAYER-36 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059954 348930 net.cpp:2191] LAYER-37 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059959 348930 net.cpp:2191] LAYER-38 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059964 348930 net.cpp:2191] LAYER-39 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.059969 348930 net.cpp:2191] LAYER-40 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.059974 348930 net.cpp:2191] LAYER-41 Split: waitcomm_time: 0 ms
W0322 04:40:40.059979 348930 net.cpp:2191] LAYER-42 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059984 348930 net.cpp:2191] LAYER-43 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.059989 348930 net.cpp:2191] LAYER-44 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.059994 348930 net.cpp:2191] LAYER-45 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060006 348930 net.cpp:2191] LAYER-46 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060016 348930 net.cpp:2191] LAYER-47 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060022 348930 net.cpp:2191] LAYER-48 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060027 348930 net.cpp:2191] LAYER-49 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060032 348930 net.cpp:2191] LAYER-50 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060037 348930 net.cpp:2191] LAYER-51 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060042 348930 net.cpp:2191] LAYER-52 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060047 348930 net.cpp:2191] LAYER-53 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060052 348930 net.cpp:2191] LAYER-54 Split: waitcomm_time: 0 ms
W0322 04:40:40.060057 348930 net.cpp:2191] LAYER-55 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060062 348930 net.cpp:2191] LAYER-56 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060067 348930 net.cpp:2191] LAYER-57 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060072 348930 net.cpp:2191] LAYER-58 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060077 348930 net.cpp:2191] LAYER-59 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060082 348930 net.cpp:2191] LAYER-60 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060087 348930 net.cpp:2191] LAYER-61 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060092 348930 net.cpp:2191] LAYER-62 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060097 348930 net.cpp:2191] LAYER-63 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060102 348930 net.cpp:2191] LAYER-64 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060107 348930 net.cpp:2191] LAYER-65 Split: waitcomm_time: 0 ms
W0322 04:40:40.060112 348930 net.cpp:2191] LAYER-66 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060118 348930 net.cpp:2191] LAYER-67 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060123 348930 net.cpp:2191] LAYER-68 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060128 348930 net.cpp:2191] LAYER-69 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060133 348930 net.cpp:2191] LAYER-70 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060138 348930 net.cpp:2191] LAYER-71 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060143 348930 net.cpp:2191] LAYER-72 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060148 348930 net.cpp:2191] LAYER-73 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060153 348930 net.cpp:2191] LAYER-74 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060158 348930 net.cpp:2191] LAYER-75 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060163 348930 net.cpp:2191] LAYER-76 Split: waitcomm_time: 0 ms
W0322 04:40:40.060168 348930 net.cpp:2191] LAYER-77 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060173 348930 net.cpp:2191] LAYER-78 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060179 348930 net.cpp:2191] LAYER-79 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060184 348930 net.cpp:2191] LAYER-80 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060189 348930 net.cpp:2191] LAYER-81 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060194 348930 net.cpp:2191] LAYER-82 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060199 348930 net.cpp:2191] LAYER-83 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060204 348930 net.cpp:2191] LAYER-84 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060209 348930 net.cpp:2191] LAYER-85 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060221 348930 net.cpp:2191] LAYER-86 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060227 348930 net.cpp:2191] LAYER-87 Split: waitcomm_time: 0 ms
W0322 04:40:40.060232 348930 net.cpp:2191] LAYER-88 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060237 348930 net.cpp:2191] LAYER-89 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060243 348930 net.cpp:2191] LAYER-90 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060248 348930 net.cpp:2191] LAYER-91 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060253 348930 net.cpp:2191] LAYER-92 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060259 348930 net.cpp:2191] LAYER-93 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060264 348930 net.cpp:2191] LAYER-94 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060276 348930 net.cpp:2191] LAYER-95 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060282 348930 net.cpp:2191] LAYER-96 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060292 348930 net.cpp:2191] LAYER-97 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060297 348930 net.cpp:2191] LAYER-98 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060302 348930 net.cpp:2191] LAYER-99 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060307 348930 net.cpp:2191] LAYER-100 Split: waitcomm_time: 0 ms
W0322 04:40:40.060312 348930 net.cpp:2191] LAYER-101 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060318 348930 net.cpp:2191] LAYER-102 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060323 348930 net.cpp:2191] LAYER-103 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060328 348930 net.cpp:2191] LAYER-104 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060334 348930 net.cpp:2191] LAYER-105 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060339 348930 net.cpp:2191] LAYER-106 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060344 348930 net.cpp:2191] LAYER-107 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060350 348930 net.cpp:2191] LAYER-108 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060355 348930 net.cpp:2191] LAYER-109 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060361 348930 net.cpp:2191] LAYER-110 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060366 348930 net.cpp:2191] LAYER-111 Split: waitcomm_time: 0 ms
W0322 04:40:40.060371 348930 net.cpp:2191] LAYER-112 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060377 348930 net.cpp:2191] LAYER-113 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060382 348930 net.cpp:2191] LAYER-114 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060387 348930 net.cpp:2191] LAYER-115 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060392 348930 net.cpp:2191] LAYER-116 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060398 348930 net.cpp:2191] LAYER-117 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060403 348930 net.cpp:2191] LAYER-118 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060408 348930 net.cpp:2191] LAYER-119 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060415 348930 net.cpp:2191] LAYER-120 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060420 348930 net.cpp:2191] LAYER-121 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060425 348930 net.cpp:2191] LAYER-122 Split: waitcomm_time: 0 ms
W0322 04:40:40.060431 348930 net.cpp:2191] LAYER-123 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060436 348930 net.cpp:2191] LAYER-124 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060441 348930 net.cpp:2191] LAYER-125 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060446 348930 net.cpp:2191] LAYER-126 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060451 348930 net.cpp:2191] LAYER-127 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060457 348930 net.cpp:2191] LAYER-128 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060462 348930 net.cpp:2191] LAYER-129 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060467 348930 net.cpp:2191] LAYER-130 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060472 348930 net.cpp:2191] LAYER-131 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060478 348930 net.cpp:2191] LAYER-132 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060483 348930 net.cpp:2191] LAYER-133 Split: waitcomm_time: 0 ms
W0322 04:40:40.060488 348930 net.cpp:2191] LAYER-134 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060500 348930 net.cpp:2191] LAYER-135 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060506 348930 net.cpp:2191] LAYER-136 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060513 348930 net.cpp:2191] LAYER-137 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060518 348930 net.cpp:2191] LAYER-138 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060523 348930 net.cpp:2191] LAYER-139 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060529 348930 net.cpp:2191] LAYER-140 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060534 348930 net.cpp:2191] LAYER-141 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060540 348930 net.cpp:2191] LAYER-142 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060546 348930 net.cpp:2191] LAYER-143 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060559 348930 net.cpp:2191] LAYER-144 Split: waitcomm_time: 0 ms
W0322 04:40:40.060564 348930 net.cpp:2191] LAYER-145 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060570 348930 net.cpp:2191] LAYER-146 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060580 348930 net.cpp:2191] LAYER-147 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060585 348930 net.cpp:2191] LAYER-148 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060590 348930 net.cpp:2191] LAYER-149 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060596 348930 net.cpp:2191] LAYER-150 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060601 348930 net.cpp:2191] LAYER-151 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060607 348930 net.cpp:2191] LAYER-152 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060612 348930 net.cpp:2191] LAYER-153 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060618 348930 net.cpp:2191] LAYER-154 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060623 348930 net.cpp:2191] LAYER-155 Split: waitcomm_time: 0 ms
W0322 04:40:40.060629 348930 net.cpp:2191] LAYER-156 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060636 348930 net.cpp:2191] LAYER-157 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060641 348930 net.cpp:2191] LAYER-158 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060647 348930 net.cpp:2191] LAYER-159 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060652 348930 net.cpp:2191] LAYER-160 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060657 348930 net.cpp:2191] LAYER-161 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060662 348930 net.cpp:2191] LAYER-162 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060668 348930 net.cpp:2191] LAYER-163 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060673 348930 net.cpp:2191] LAYER-164 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060679 348930 net.cpp:2191] LAYER-165 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060684 348930 net.cpp:2191] LAYER-166 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060690 348930 net.cpp:2191] LAYER-167 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060695 348930 net.cpp:2191] LAYER-168 Split: waitcomm_time: 0 ms
W0322 04:40:40.060701 348930 net.cpp:2191] LAYER-169 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060708 348930 net.cpp:2191] LAYER-170 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060712 348930 net.cpp:2191] LAYER-171 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060719 348930 net.cpp:2191] LAYER-172 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060724 348930 net.cpp:2191] LAYER-173 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060729 348930 net.cpp:2191] LAYER-174 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060734 348930 net.cpp:2191] LAYER-175 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060740 348930 net.cpp:2191] LAYER-176 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060746 348930 net.cpp:2191] LAYER-177 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060751 348930 net.cpp:2191] LAYER-178 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060757 348930 net.cpp:2191] LAYER-179 Split: waitcomm_time: 0 ms
W0322 04:40:40.060762 348930 net.cpp:2191] LAYER-180 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060768 348930 net.cpp:2191] LAYER-181 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060775 348930 net.cpp:2191] LAYER-182 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060780 348930 net.cpp:2191] LAYER-183 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060786 348930 net.cpp:2191] LAYER-184 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060797 348930 net.cpp:2191] LAYER-185 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060803 348930 net.cpp:2191] LAYER-186 Convolution: waitcomm_time: 0 ms
W0322 04:40:40.060809 348930 net.cpp:2191] LAYER-187 BatchNorm: waitcomm_time: 0 ms
W0322 04:40:40.060819 348930 net.cpp:2191] LAYER-188 Eltwise: waitcomm_time: 0 ms
W0322 04:40:40.060827 348930 net.cpp:2191] LAYER-189 ReLU: waitcomm_time: 0 ms
W0322 04:40:40.060832 348930 net.cpp:2191] LAYER-190 Pooling: waitcomm_time: 0 ms
W0322 04:40:40.060837 348930 net.cpp:2191] LAYER-191 InnerProduct: waitcomm_time: 0 ms
W0322 04:40:40.060843 348930 net.cpp:2191] LAYER-192 Split: waitcomm_time: 0 ms
W0322 04:40:40.060856 348930 net.cpp:2191] LAYER-193 SoftmaxWithLoss: waitcomm_time: 0 ms
W0322 04:40:40.060863 348930 net.cpp:2191] LAYER-194 Accuracy: waitcomm_time: 0 ms
W0322 04:40:40.060868 348930 net.cpp:2191] LAYER-195 Accuracy: waitcomm_time: 0 ms
W0322 04:40:40.060878 348930 net.cpp:2196] 
W0322 04:40:40.060883 348930 net.cpp:2198] DELTA TIME (Computation + Communication): 0.127157 sec
W0322 04:40:40.060890 348930 net.cpp:2206] ####################################################
W0322 04:40:40.060895 348930 net.cpp:2207] 
I0322 04:40:40.060904 348930 solver.cpp:363] iter 23, forward_backward_update_time: 127 ms

My 100,000 iteration run I tried had almost 9GB log files since every iteration is outputting so much data.

Any help to turn off all the reporting would be great!

Przemek D

unread,

Mar 23, 2018, 8:36:40 AM3/23/18

to Caffe Users

I haven't used Intel Caffe before, but master branch has a flag for controlling the log outputs: -minloglevel. It takes and int argument and the default is zero, meaning all messages are logged. Run caffe --help to see if you can use this flag too.

Matthew Brown

unread,

Mar 23, 2018, 9:43:32 AM3/23/18

to Caffe Users

Thank you!

I will give that a try and see if that fixes my issue!

Matt

Reply all

Reply to author

Forward

0 new messages