GP100?

[Joseph-Jonathan Salzano]

I am running tests on multiple GPUs. I am using the windows branch of Caffe.

If I am running my OpenCL version I caffe runs on all GPUs. If I use the Cuda version I can get it working on everything I have tried so far up to a P6000. But when I try and run on a GP100 it fails.

I am using CUDA 8 and cuDNN 6. Has anyone successfully ran Caffe on a GP100 or have insights into what I should be looking for?

Joseph

[Przemek D]

How exactly "it fails"? Are you getting any error messages?

[Joseph-Jonathan Salzano]

I get a "check failure stack trace" when Caffe is setting up. The models and data sets have been read in but the first iteration has not started. I have never been able to find the stack trace even with Debug enabled so I don't know exactly what is going on other than that.

I'll get my hands on the stdout but I haven't seen anywhere about Caffe supporting GP100s so I didn't know if they were just unsupported or if someone has used one before.

[Joseph-Jonathan Salzano]

MNIST using LeNet Solver

I0918 08:44:21.711534  8724 layer_factory.cpp:58] Creating layer mnist

I0918 08:44:21.727166  8724 db_lmdb.cpp:40] Opened lmdb data/mnist/mnist_test_lmdb

I0918 08:44:21.727166  8724 net.cpp:84] Creating Layer mnist

I0918 08:44:21.727166  8724 net.cpp:380] mnist -> data

I0918 08:44:21.727166  8724 net.cpp:380] mnist -> label

I0918 08:44:21.727166  8724 data_layer.cpp:45] output data size: 100,1,28,28

I0918 08:44:21.727166  8724 net.cpp:122] Setting up mnist

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 1 28 28 (78400)

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 (100)

I0918 08:44:21.727166  8724 net.cpp:137] Memory required for data: 314000

I0918 08:44:21.727166  8724 layer_factory.cpp:58] Creating layer label_mnist_1_split

I0918 08:44:21.727166  8724 net.cpp:84] Creating Layer label_mnist_1_split

I0918 08:44:21.727166  8724 net.cpp:406] label_mnist_1_split <- label

I0918 08:44:21.727166  8724 net.cpp:380] label_mnist_1_split -> label_mnist_1_split_0

I0918 08:44:21.727166  8724 net.cpp:380] label_mnist_1_split -> label_mnist_1_split_1

I0918 08:44:21.727166  8724 net.cpp:122] Setting up label_mnist_1_split

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 (100)

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 (100)

I0918 08:44:21.727166  8724 net.cpp:137] Memory required for data: 314800

I0918 08:44:21.727166  8724 layer_factory.cpp:58] Creating layer conv1

I0918 08:44:21.727166  8724 net.cpp:84] Creating Layer conv1

I0918 08:44:21.727166  8724 net.cpp:406] conv1 <- data

I0918 08:44:21.727166  8724 net.cpp:380] conv1 -> conv1

I0918 08:44:21.727166  8724 net.cpp:122] Setting up conv1

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 20 24 24 (1152000)

I0918 08:44:21.727166  8724 net.cpp:137] Memory required for data: 4922800

I0918 08:44:21.727166  8724 layer_factory.cpp:58] Creating layer pool1

I0918 08:44:21.727166  8724 net.cpp:84] Creating Layer pool1

I0918 08:44:21.727166  8724 net.cpp:406] pool1 <- conv1

I0918 08:44:21.727166  8724 net.cpp:380] pool1 -> pool1

I0918 08:44:21.727166  8724 net.cpp:122] Setting up pool1

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 20 12 12 (288000)

I0918 08:44:21.727166  8724 net.cpp:137] Memory required for data: 6074800

I0918 08:44:21.727166  8724 layer_factory.cpp:58] Creating layer conv2

I0918 08:44:21.727166  8724 net.cpp:84] Creating Layer conv2

I0918 08:44:21.727166  8724 net.cpp:406] conv2 <- pool1

I0918 08:44:21.727166  8724 net.cpp:380] conv2 -> conv2

I0918 08:44:21.727166 14540 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.

I0918 08:44:21.727166  8724 net.cpp:122] Setting up conv2

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 50 8 8 (320000)

I0918 08:44:21.727166  8724 net.cpp:137] Memory required for data: 7354800

I0918 08:44:21.727166  8724 layer_factory.cpp:58] Creating layer pool2

I0918 08:44:21.727166  8724 net.cpp:84] Creating Layer pool2

I0918 08:44:21.727166  8724 net.cpp:406] pool2 <- conv2

I0918 08:44:21.727166  8724 net.cpp:380] pool2 -> pool2

I0918 08:44:21.727166  8724 net.cpp:122] Setting up pool2

I0918 08:44:21.727166  8724 net.cpp:129] Top shape: 100 50 4 4 (80000)

I0918 08:44:21.727166  8724 net.cpp:137] Memory required for data: 7674800

I0918 08:44:21.727166  8724 layer_factory.cpp:58] Creating layer ip1

I0918 08:44:21.727166  8724 net.cpp:84] Creating Layer ip1

I0918 08:44:21.727166  8724 net.cpp:406] ip1 <- pool2

I0918 08:44:21.727166  8724 net.cpp:380] ip1 -> ip1

I0918 08:44:21.742789  8724 net.cpp:122] Setting up ip1

I0918 08:44:21.742789  8724 net.cpp:129] Top shape: 100 500 (50000)

I0918 08:44:21.742789  8724 net.cpp:137] Memory required for data: 7874800

I0918 08:44:21.742789  8724 layer_factory.cpp:58] Creating layer relu1

I0918 08:44:21.742789  8724 net.cpp:84] Creating Layer relu1

I0918 08:44:21.742789  8724 net.cpp:406] relu1 <- ip1

I0918 08:44:21.742789  8724 net.cpp:367] relu1 -> ip1 (in-place)

I0918 08:44:21.742789  8724 net.cpp:122] Setting up relu1

I0918 08:44:21.742789  8724 net.cpp:129] Top shape: 100 500 (50000)

I0918 08:44:21.742789  8724 net.cpp:137] Memory required for data: 8074800

I0918 08:44:21.742789  8724 layer_factory.cpp:58] Creating layer ip2

I0918 08:44:21.742789  8724 net.cpp:84] Creating Layer ip2

I0918 08:44:21.742789  8724 net.cpp:406] ip2 <- ip1

I0918 08:44:21.742789  8724 net.cpp:380] ip2 -> ip2

I0918 08:44:21.742789  8724 net.cpp:122] Setting up ip2

I0918 08:44:21.742789  8724 net.cpp:129] Top shape: 100 10 (1000)

I0918 08:44:21.742789  8724 net.cpp:137] Memory required for data: 8078800

I0918 08:44:21.742789  8724 layer_factory.cpp:58] Creating layer ip2_ip2_0_split

I0918 08:44:21.742789  8724 net.cpp:84] Creating Layer ip2_ip2_0_split

I0918 08:44:21.742789  8724 net.cpp:406] ip2_ip2_0_split <- ip2

I0918 08:44:21.742789  8724 net.cpp:380] ip2_ip2_0_split -> ip2_ip2_0_split_0

I0918 08:44:21.742789  8724 net.cpp:380] ip2_ip2_0_split -> ip2_ip2_0_split_1

I0918 08:44:21.742789  8724 net.cpp:122] Setting up ip2_ip2_0_split

I0918 08:44:21.742789  8724 net.cpp:129] Top shape: 100 10 (1000)

I0918 08:44:21.742789  8724 net.cpp:129] Top shape: 100 10 (1000)

I0918 08:44:21.742789  8724 net.cpp:137] Memory required for data: 8086800

I0918 08:44:21.742789  8724 layer_factory.cpp:58] Creating layer accuracy

I0918 08:44:21.742789  8724 net.cpp:84] Creating Layer accuracy

I0918 08:44:21.742789  8724 net.cpp:406] accuracy <- ip2_ip2_0_split_0

I0918 08:44:21.742789  8724 net.cpp:406] ac*** Check failure stack trace: ***

MNIST Autoencoder

I0918 08:44:31.040866 15128 layer_factory.cpp:58] Creating layer data

I0918 08:44:31.040866 15128 db_lmdb.cpp:40] Opened lmdb data/mnist/mnist_test_lmdb

I0918 08:44:31.040866 15128 net.cpp:84] Creating Layer data

I0918 08:44:31.040866 15128 net.cpp:380] data -> data

I0918 08:44:31.040866 15128 data_layer.cpp:45] output data size: 100,1,28,28

I0918 08:44:31.040866 15128 net.cpp:122] Setting up data

I0918 08:44:31.040866 15128 net.cpp:129] Top shape: 100 1 28 28 (78400)

I0918 08:44:31.040866 15128 net.cpp:137] Memory required for data: 313600

I0918 08:44:31.040866 15128 layer_factory.cpp:58] Creating layer data_data_0_split

I0918 08:44:31.040866 15128 net.cpp:84] Creating Layer data_data_0_split

I0918 08:44:31.040866 15128 net.cpp:406] data_data_0_split <- data

I0918 08:44:31.040866 15128 net.cpp:380] data_data_0_split -> data_data_0_split_0

I0918 08:44:31.040866 15128 net.cpp:380] data_data_0_split -> data_data_0_split_1

I0918 08:44:31.040866 15128 net.cpp:122] Setting up data_data_0_split

I0918 08:44:31.040866 15128 net.cpp:129] Top shape: 100 1 28 28 (78400)

I0918 08:44:31.040866 15128 net.cpp:129] Top shape: 100 1 28 28 (78400)

I0918 08:44:31.040866 15128 net.cpp:137] Memory required for data: 940800

I0918 08:44:31.040866 15128 layer_factory.cpp:58] Creating layer flatdata

I0918 08:44:31.040866 15128 net.cpp:84] Creating Layer flatdata

I0918 08:44:31.040866 15128 net.cpp:406] flatdata <- data_data_0_split_0

I0918 08:44:31.040866 15128 net.cpp:380] flatdata -> flatdata

I0918 08:44:31.040866 15128 net.cpp:122] Setting up flatdata

I0918 08:44:31.040866 15128 net.cpp:129] Top shape: 100 784 (78400)

I0918 08:44:31.040866 15128 net.cpp:137] Memory required for data: 1254400

I0918 08:44:31.040866 15128 layer_factory.cpp:58] Creating layer flatdata_flatdata_0_split

I0918 08:44:31.040866 15128 net.cpp:84] Creating Layer flatdata_flatdata_0_split

I0918 08:44:31.040866 15128 net.cpp:406] flatdata_flatdata_0_split <- flatdata

I0918 08:44:31.040866 15128 net.cpp:380] flatdata_flatdata_0_split -> flatdata_flatdata_0_split_0

I0918 08:44:31.040866 15128 net.cpp:380] flatdata_flatdata_0_split -> flatdata_flatdata_0_split_1

I0918 08:44:31.040866 15128 net.cpp:122] Setting up flatdata_flatdata_0_split

I0918 08:44:31.040866 15128 net.cpp:129] Top shape: 100 784 (78400)

I0918 08:44:31.040866 15128 net.cpp:129] Top shape: 100 784 (78400)

I0918 08:44:31.040866 15128 net.cpp:137] Memory required for data: 1881600

I0918 08:44:31.040866 15128 layer_factory.cpp:58] Creating layer encode1

I0918 08:44:31.040866 15128 net.cpp:84] Creating Layer encode1

I0918 08:44:31.040866 15128 net.cpp:406] encode1 <- data_data_0_split_1

I0918 08:44:31.040866 15128 net.cpp:380] encode1 -> encode1

I0918 08:44:31.040866 14880 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.

I0918 08:44:31.056491 15128 net.cpp:122] Setting up encode1

I0918 08:44:31.056491 15128 net.cpp:129] Top shape: 100 1000 (100000)

I0918 08:44:31.056491 15128 net.cpp:137] Memory required for data: 2281600

I0918 08:44:31.056491 15128 layer_factory.cpp:58] Creating layer encode1neuron

I0918 08:44:31.056491 15128 net.cpp:84] Creating Layer encode1neuron

I0918 08:44:31.056491 15128 net.cpp:406] encode1neuron <- encode1

I0918 08:44:31.056491 15128 net.cpp:380] encode1neuron -> encode1neuron

I0918 08:44:31.056491 15128 net.cpp:122] Setting up encode1neuron

I0918 08:44:31.056491 15128 net.cpp:129] Top shape: 100 1000 (100000)

I0918 08:44:31.056491 15128 net.cpp:137] Memory required for data: 2681600

I0918 08:44:31.056491 15128 layer_factory.cpp:58] Creating layer encode2

I0918 08:44:31.056491 15128 net.cpp:84] Creating Layer encode2

I0918 08:44:31.056491 15128 net.cpp:406] encode2 <- encode1neuron

I0918 08:44:31.056491 15128 net.cpp:380] encode2 -> encode2

I0918 08:44:31.056491 15128 net.cpp:122] Setting up encode2

I0918 08:44:31.056491 15128 net.cpp:129] Top shape: 100 500 (50000)

I0918 08:44:31.056491 15128 net.cpp:137] Memory required for data: 2881600

I0918 08:44:31.056491 15128 layer_factory.cpp:58] Creating layer encode2neuron

I0918 08:44:31.056491 15128 net.cpp:84] Creating Layer encode2neuron

I0918 08:44:31.056491 15128 net.cpp:406] encode2neuron <- encode2

I0918 08:44:31.056491 15128 net.cpp:380] encode2neuron -> encode2neuron

I0918 08:44:31.056491 15128 net.cpp:122] Setting up encode2neuron

I0918 08:44:31.056491 15128 net.cpp:129] Top shape: 100 500 (50000)

I0918 08:44:31.056491 15128 net.cpp:137] Memory required for data: 3081600

I0918 08:44:31.056491 15128 layer_factory.cpp:58] Creating layer encode3

I0918 08:44:31.056491 15128 net.cpp:84] Creating Layer encode3

I0918 08:44:31.056491 15128 net.cpp:406] encode3 <- encode2neuron

I0918 08:44:31.056491 15128 net.cpp:380] encode3 -> encode3

I0918 08:44:31.072119 15128 net.cpp:122] Setting up encode3

I0918 08:44:31.072119 15128 net.cpp:129] Top shape: 100 250 (25000)

I0918 08:44:31.072119 15128 net.cpp:137] Memory required for data: 3181600

I0918 08:44:31.072119 15128 layer_factory.cpp:58] Creating layer encode3neuron

I0918 08:44:31.072119 15128 net.cpp:84] Creating Layer encode3neuron

I0918 08:44:31.072119 15128 net.cpp:406] encode3neuron <- encode3

I0918 08:44:31.072119 15128 net.cpp:380] encode3neuron -> encode3neuron

I0918 08:44:31.072119 15128 net.cpp:122] Setting up encode3neuron

I0918 08:44:31.072119 15128 net.cpp:129] Top shape: 100 250 (25000)

I0918 08:44:31.072119 15128 net.cpp:137] Memory required for data: 3281600

I0918 08:44:31.072119 15128 layer_factory.cpp:58] Creating layer encode4

I0918 08:44:31.072119 15128 net.cpp:84] Creating Layer encode4

I0918 08:44:31.072119 15128 net.cpp:406] encode4 <- encode3neuron

I0918 08:44:31.072119 15128 net.cpp:380] encode4 -> encode4

I0918 08:44:31.072119 15128 net.cpp:122] Setting up encode4

I0918 08:44:31.072119 15128 net.cpp:129] Top shape: 100 30 (3000)

I0918 08:44:31.072119 15128 net.cpp:137] Memory required for data: 3293600

I0918 08:44:31.072119 15128 layer_factory.cpp:58] Creating layer decode4

I0918 08:44:31.072119 15128 net.cpp:84] Creating Layer decode4

I0918 08:44:31.072119 15128 net.cpp:406] decode4 <- encode4

I0918 08:44:31.072119 15128 net.cpp:380] decode4 -> decode4

I0918 08:44:31.072119 15128 net.cpp:122] Setting up decode4

I0918 08:44:31.072119 15128 net.cpp:129] Top shape: 100 250 (25000)

I0918 08:44:31.072119 15128 net.cpp:137] Memory required for data: 3393600

I0918 08:44:31.072119 15128 layer_factory.cpp:58] Creating layer decode4neuron

I0918 08:44:31.072119 15128 net.cpp:84] Creating Layer decode4neuron

I0918 08:44:31.072119 15128 net.cpp:406] decode4neuron <- decode4

I0918 08:44:31.072119 15128 net.cpp:380] decode4neuron -> decode4neuron

I0918 08:44:31.072119 15128 net.cpp:122] Setting up decode4neuron

I0918 08:44:31.072119 15128 net.cpp:129] Top shape: 100 250 (25000)

I0918 08:44:31.072119 15128 net.cpp:137] Memory required for data: 3493600

I0918 08:44:31.072119 15128 layer_factory.cpp:58] Creating layer decode3

I0918 08:44:31.072119 15128 net.cpp:84] Creating Layer decode3

I0918 08:44:31.072119 15128 net.cpp:406] decode3 <- decode4neuron

I0918 08:44:31.072119 15128 net.cpp:380] decode3 -> decode3

I0918 08:44:31.072119 15128 net.cpp:122] Setting up decode3

I0918 08:44:31.072119 15128 net.cpp:129] Top shape: 100 500 (50000)

I0918 08:44:31.072119 15128 net.cpp:137] Memory required for data: 3693600

I0918 08:44:31.072119 15128 layer_factory.cpp:58] Creating layer decode3neuron

I0918 08:44:31.072119 15128 net.cpp:84] Creating Layer decode3neuron

I0918 08:44:31.072119 15128 net.cpp:406] decode3neuron <- decode3

I0918 08:44:31.072119 15128 net.cpp:380] decode3neuron -> decode3neuron

I0918 08:44:31.072119 15128 net.cpp:122] Setting up decode3neuron

I0918 08:44:31.072119 15128 net.cpp:129] Top shape: 100 500 (50000)

I0918 08:44:31.072119 15128 net.cpp:137] Memory required for data: 3893600

I0918 08:44:31.072119 15128 layer_factory.cpp:58] Creating layer decode2

I0918 08:44:31.072119 15128 net.cpp:84] Creating Layer decode2

I0918 08:44:31.072119 15128 net.cpp:406] decode2 <- decode3neuron

I0918 08:44:31.072119 15128 net.cpp:380] decode2 -> decode2

I0918 08:44:31.087746 15128 net.cpp:122] Setting up decode2

I0918 08:44:31.087746 15128 net.cpp:129] Top shape: 100 1000 (100000)

I0918 08:44:31.087746 15128 net.cpp:137] Memory required for data: 4293600

I0918 08:44:31.087746 15128 layer_factory.cpp:58] Creating layer decode2neuron

I0918 08:44:31.087746 15128 net.cpp:84] Creating Layer decode2neuron

I0918 08:44:31.087746 15128 net.cpp:406] decode2neuron <- decode2

I0918 08:44:31.087746 15128 net.cpp:380] decode2neuron -> decode2neuron

I0918 08:44:31.087746 15128 net.cpp:122] Setting up decode2neuron

I0918 08:44:31.087746 15128 net.cpp:129] Top shape: 100 1000 (100000)

I0918 08:44:31.087746 15128 net.cpp:137] Memory required for data: 4693600

I0918 08:44:31.087746 15128 layer_factory.cpp:58] Creating layer decode1

I0918 08:44:31.087746 15128 net.cpp:84] Creating Layer decode1

I0918 08:44:31.087746 15128 net.cpp:406] decode1 <- decode2neuron

I0918 08:44:31.087746 15128 net.cpp:380] decode1 -> decode1

I0918 08:44:31.103371 15128 net.cpp:122] Setting up decode1

I0918 08:44:31.103371 15128 net.cpp:129] Top shape: 100 784 (78400)

I0918 08:44:31.103371 15128 net.cpp:137] Memory required for data: 5007200

I0918 08:44:31.103371 15128 layer_factory.cpp:58] Creating layer decode1_decode1_0_split

I0918 08:44:31.103371 15128 net.cpp:84] Creating Layer decode1_decode1_0_split

I0918 08:44:31.103371 15128 net.cpp:406] decode1_decode1_0_split <- decode1

I0918 08:44:31.103371 15128 net.cpp:380] decode1_decode1_0_split -> decode1_decode1_0_split_0

I0918 08:44:31.103371 15128 net.cpp:380] decode1_decode1_0_split -> decode1_decode1_0_split_1

I0918 08:44:31.103371 15128 net.cpp:122] Setting up decode1_decode1_0_split

I0918 08:44:31.103371 15128 net.cpp:129] Top shape: 100 784 (78400)

I0918 08:44:31.103371 15128 net.cpp:129] Top shape: 100 784 (78400)

I0918 08:44:31.103371 15128 net.cpp:137] Memory required for data: 5634400

I0918 08:44:31.103371 15128 layer_factory.cpp:58] Creating layer loss

I0918 08:44:31.103371 15128 net.cpp:84] Creating Layer loss

I0918 08:44:31.103371 15128 net.cpp:406] loss <- decode1_decode1_0_split_0

I0918 08:44:31.103371 15128 net.cpp:406] loss <- flatdata_flatdata_0_split_0

I0918 08:44:31.103371 15128 net.cpp:380] loss -> cross_entropy_loss

I0918 08:44:31.103371 15128 net.cpp:122] Setting up loss

I0918 08:44:31.103371 15128 net.cpp:129] Top shape: (1)

I0918 08:44:31.103371 15128 net.cpp:132]     with loss weight 1

I0918 08:44:31.103371 15128 net.cpp:137] Memory required for data: 56344*** Check failure stack trace: ***

CIFAR10 Solver

I0918 08:44:40.401397 13628 layer_factory.cpp:58] Creating layer cifar

I0918 08:44:40.417024 13628 db_lmdb.cpp:40] Opened lmdb data/cifar10/cifar10_test_lmdb

I0918 08:44:40.417024 13628 net.cpp:84] Creating Layer cifar

I0918 08:44:40.417024 13628 net.cpp:380] cifar -> data

I0918 08:44:40.417024 13628 net.cpp:380] cifar -> label

I0918 08:44:40.417024 13628 data_transformer.cpp:25] Loading mean file from: data/cifar10/mean.binaryproto

I0918 08:44:40.417024 13628 data_layer.cpp:45] output data size: 100,3,32,32

I0918 08:44:40.432672 13628 net.cpp:122] Setting up cifar

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 3 32 32 (307200)

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 (100)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 1229200

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer label_cifar_1_split

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer label_cifar_1_split

I0918 08:44:40.432672 13628 net.cpp:406] label_cifar_1_split <- label

I0918 08:44:40.432672 13628 net.cpp:380] label_cifar_1_split -> label_cifar_1_split_0

I0918 08:44:40.432672 13628 net.cpp:380] label_cifar_1_split -> label_cifar_1_split_1

I0918 08:44:40.432672 13628 net.cpp:122] Setting up label_cifar_1_split

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 (100)

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 (100)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 1230000

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer conv1

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer conv1

I0918 08:44:40.432672 13628 net.cpp:406] conv1 <- data

I0918 08:44:40.432672 13628 net.cpp:380] conv1 -> conv1

I0918 08:44:40.432672 10140 common.cpp:36] System entropy source not available, using fallback algorithm to generate seed instead.

I0918 08:44:40.432672 13628 net.cpp:122] Setting up conv1

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 32 32 (3276800)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 14337200

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer pool1

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer pool1

I0918 08:44:40.432672 13628 net.cpp:406] pool1 <- conv1

I0918 08:44:40.432672 13628 net.cpp:380] pool1 -> pool1

I0918 08:44:40.432672 13628 net.cpp:122] Setting up pool1

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 16 16 (819200)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 17614000

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer relu1

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer relu1

I0918 08:44:40.432672 13628 net.cpp:406] relu1 <- pool1

I0918 08:44:40.432672 13628 net.cpp:367] relu1 -> pool1 (in-place)

I0918 08:44:40.432672 13628 net.cpp:122] Setting up relu1

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 16 16 (819200)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 20890800

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer norm1

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer norm1

I0918 08:44:40.432672 13628 net.cpp:406] norm1 <- pool1

I0918 08:44:40.432672 13628 net.cpp:380] norm1 -> norm1

I0918 08:44:40.432672 13628 net.cpp:122] Setting up norm1

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 16 16 (819200)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 24167600

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer conv2

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer conv2

I0918 08:44:40.432672 13628 net.cpp:406] conv2 <- norm1

I0918 08:44:40.432672 13628 net.cpp:380] conv2 -> conv2

I0918 08:44:40.432672 13628 net.cpp:122] Setting up conv2

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 16 16 (819200)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 27444400

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer relu2

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer relu2

I0918 08:44:40.432672 13628 net.cpp:406] relu2 <- conv2

I0918 08:44:40.432672 13628 net.cpp:367] relu2 -> conv2 (in-place)

I0918 08:44:40.432672 13628 net.cpp:122] Setting up relu2

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 16 16 (819200)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 30721200

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer pool2

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer pool2

I0918 08:44:40.432672 13628 net.cpp:406] pool2 <- conv2

I0918 08:44:40.432672 13628 net.cpp:380] pool2 -> pool2

I0918 08:44:40.432672 13628 net.cpp:122] Setting up pool2

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 8 8 (204800)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 31540400

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer norm2

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer norm2

I0918 08:44:40.432672 13628 net.cpp:406] norm2 <- pool2

I0918 08:44:40.432672 13628 net.cpp:380] norm2 -> norm2

I0918 08:44:40.432672 13628 net.cpp:122] Setting up norm2

I0918 08:44:40.432672 13628 net.cpp:129] Top shape: 100 32 8 8 (204800)

I0918 08:44:40.432672 13628 net.cpp:137] Memory required for data: 32359600

I0918 08:44:40.432672 13628 layer_factory.cpp:58] Creating layer conv3

I0918 08:44:40.432672 13628 net.cpp:84] Creating Layer conv3

I0918 08:44:40.432672 13628 net.cpp:406] conv3 <- norm2

I0918 08:44:40.432672 13628 net.cpp:380] conv3 -> conv3

I0918 08:44:40.448310 13628 net.cpp:122] Setting up conv3

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: 100 64 8 8 (409600)

I0918 08:44:40.448310 13628 net.cpp:137] Memory required for data: 33998000

I0918 08:44:40.448310 13628 layer_factory.cpp:58] Creating layer relu3

I0918 08:44:40.448310 13628 net.cpp:84] Creating Layer relu3

I0918 08:44:40.448310 13628 net.cpp:406] relu3 <- conv3

I0918 08:44:40.448310 13628 net.cpp:367] relu3 -> conv3 (in-place)

I0918 08:44:40.448310 13628 net.cpp:122] Setting up relu3

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: 100 64 8 8 (409600)

I0918 08:44:40.448310 13628 net.cpp:137] Memory required for data: 35636400

I0918 08:44:40.448310 13628 layer_factory.cpp:58] Creating layer pool3

I0918 08:44:40.448310 13628 net.cpp:84] Creating Layer pool3

I0918 08:44:40.448310 13628 net.cpp:406] pool3 <- conv3

I0918 08:44:40.448310 13628 net.cpp:380] pool3 -> pool3

I0918 08:44:40.448310 13628 net.cpp:122] Setting up pool3

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: 100 64 4 4 (102400)

I0918 08:44:40.448310 13628 net.cpp:137] Memory required for data: 36046000

I0918 08:44:40.448310 13628 layer_factory.cpp:58] Creating layer ip1

I0918 08:44:40.448310 13628 net.cpp:84] Creating Layer ip1

I0918 08:44:40.448310 13628 net.cpp:406] ip1 <- pool3

I0918 08:44:40.448310 13628 net.cpp:380] ip1 -> ip1

I0918 08:44:40.448310 13628 net.cpp:122] Setting up ip1

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: 100 10 (1000)

I0918 08:44:40.448310 13628 net.cpp:137] Memory required for data: 36050000

I0918 08:44:40.448310 13628 layer_factory.cpp:58] Creating layer ip1_ip1_0_split

I0918 08:44:40.448310 13628 net.cpp:84] Creating Layer ip1_ip1_0_split

I0918 08:44:40.448310 13628 net.cpp:406] ip1_ip1_0_split <- ip1

I0918 08:44:40.448310 13628 net.cpp:380] ip1_ip1_0_split -> ip1_ip1_0_split_0

I0918 08:44:40.448310 13628 net.cpp:380] ip1_ip1_0_split -> ip1_ip1_0_split_1

I0918 08:44:40.448310 13628 net.cpp:122] Setting up ip1_ip1_0_split

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: 100 10 (1000)

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: 100 10 (1000)

I0918 08:44:40.448310 13628 net.cpp:137] Memory required for data: 36058000

I0918 08:44:40.448310 13628 layer_factory.cpp:58] Creating layer accuracy

I0918 08:44:40.448310 13628 net.cpp:84] Creating Layer accuracy

I0918 08:44:40.448310 13628 net.cpp:406] accuracy <- ip1_ip1_0_split_0

I0918 08:44:40.448310 13628 net.cpp:406] accuracy <- label_cifar_1_split_0

I0918 08:44:40.448310 13628 net.cpp:380] accuracy -> accuracy

I0918 08:44:40.448310 13628 net.cpp:122] Setting up accuracy

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: (1)

I0918 08:44:40.448310 13628 net.cpp:137] Memory required for data: 36058004

I0918 08:44:40.448310 13628 layer_factory.cpp:58] Creating layer loss

I0918 08:44:40.448310 13628 net.cpp:84] Creating Layer loss

I0918 08:44:40.448310 13628 net.cpp:406] loss <- ip1_ip1_0_split_1

I0918 08:44:40.448310 13628 net.cpp:406] loss <- label_cifar_1_split_1

I0918 08:44:40.448310 13628 net.cpp:380] loss -> loss

I0918 08:44:40.448310 13628 layer_factory.cpp:58] Creating layer loss

I0918 08:44:40.448310 13628 net.cpp:122] Setting up loss

I0918 08:44:40.448310 13628 net.cpp:129] Top shape: (1)

I0918 08:44:40.448310 13628 net.cpp:132]     with loss weight 1

I0918 08:44:40.448310 13628 net.cpp:137] Memory required for data: 36058008

I0918 08:44:40.448310 13628 net.cpp:198] loss needs backward computation.

I0918 08:44:40.448310 13628 net.cpp:200] accuracy does not need backward computation.

I0918 08:44:40.448310 13628 net.cpp:198] ip1_ip1_0_split needs backward computation.

I0918 08:44:40.448310 13628 net.cpp:198] ip1 needs backward computation.

I0918 08:44:40.448310 13628 net.cpp:198] pool3 needs b*** Check failure stack trace: ***