Check failed: status == CUBLAS_STATUS_SUCCESS (11 vs. 0) CUBLAS_STATUS_MAPPING_ERROR *** Check fail

[章程]

Hi,everyone

I'm running FCN on NYUDv2(RGBD segmantation dataset).  When I run the solve.py, everthing seems OK and then I encounter the error

Check failed: status == CUBLAS_STATUS_SUCCESS (11 vs. 0)  CUBLAS_STATUS_MAPPING_ERROR

*** Check failure stack trace: 

Aborted (core dumped)

I have no idea what caused this error, is there anyone can offer some advice?

[章程]

Here is some information shown before

I1009 16:23:30.312739  5140 layer_factory.hpp:77] Creating layer data
I1009 16:23:30.312798  5140 net.cpp:84] Creating Layer data
I1009 16:23:30.312808  5140 net.cpp:380] data -> data
I1009 16:23:30.312819  5140 net.cpp:380] data -> label
I1009 16:23:30.331507  5140 net.cpp:122] Setting up data
I1009 16:23:30.331557  5140 net.cpp:129] Top shape: 1 3 640 480 (921600)
I1009 16:23:30.331565  5140 net.cpp:129] Top shape: 1 1 640 480 (307200)
I1009 16:23:30.331570  5140 net.cpp:137] Memory required for data: 4915200
I1009 16:23:30.331578  5140 layer_factory.hpp:77] Creating layer data_data_0_split
I1009 16:23:30.331593  5140 net.cpp:84] Creating Layer data_data_0_split
I1009 16:23:30.331598  5140 net.cpp:406] data_data_0_split <- data
I1009 16:23:30.331607  5140 net.cpp:380] data_data_0_split -> data_data_0_split_0
I1009 16:23:30.331619  5140 net.cpp:380] data_data_0_split -> data_data_0_split_1
I1009 16:23:30.331655  5140 net.cpp:122] Setting up data_data_0_split
I1009 16:23:30.331663  5140 net.cpp:129] Top shape: 1 3 640 480 (921600)
I1009 16:23:30.331670  5140 net.cpp:129] Top shape: 1 3 640 480 (921600)
I1009 16:23:30.331676  5140 net.cpp:137] Memory required for data: 12288000
I1009 16:23:30.331681  5140 layer_factory.hpp:77] Creating layer conv1_1
I1009 16:23:30.331691  5140 net.cpp:84] Creating Layer conv1_1
I1009 16:23:30.331696  5140 net.cpp:406] conv1_1 <- data_data_0_split_0
I1009 16:23:30.331703  5140 net.cpp:380] conv1_1 -> conv1_1
I1009 16:23:30.334264  5140 net.cpp:122] Setting up conv1_1
I1009 16:23:30.334300  5140 net.cpp:129] Top shape: 1 64 838 678 (36362496)
I1009 16:23:30.334306  5140 net.cpp:137] Memory required for data: 157737984
I1009 16:23:30.334322  5140 layer_factory.hpp:77] Creating layer relu1_1
I1009 16:23:30.334336  5140 net.cpp:84] Creating Layer relu1_1
I1009 16:23:30.334342  5140 net.cpp:406] relu1_1 <- conv1_1
I1009 16:23:30.334349  5140 net.cpp:367] relu1_1 -> conv1_1 (in-place)
I1009 16:23:30.334556  5140 net.cpp:122] Setting up relu1_1
I1009 16:23:30.334569  5140 net.cpp:129] Top shape: 1 64 838 678 (36362496)
I1009 16:23:30.334574  5140 net.cpp:137] Memory required for data: 303187968
I1009 16:23:30.334580  5140 layer_factory.hpp:77] Creating layer conv1_2
I1009 16:23:30.334592  5140 net.cpp:84] Creating Layer conv1_2
I1009 16:23:30.334599  5140 net

在 2017年10月9日星期一 UTC+8下午4:57:19，章程写道：

.cpp:406] conv1_2 <- conv1_1
I1009 16:23:30.334605  5140 net.cpp:380] conv1_2 -> conv1_2
I1009 16:23:30.336829  5140 net.cpp:122] Setting up conv1_2
I1009 16:23:30.336882  5140 net.cpp:129] Top shape: 1 64 838 678 (36362496)
I1009 16:23:30.336889  5140 net.cpp:137] Memory required for data: 448637952
I1009 16:23:30.336943  5140 layer_factory.hpp:77] Creating layer relu1_2
I1009 16:23:30.336959  5140 net.cpp:84] Creating Layer relu1_2
I1009 16:23:30.336966  5140 net.cpp:406] relu1_2 <- conv1_2
I1009 16:23:30.336977  5140 net.cpp:367] relu1_2 -> conv1_2 (in-place)
I1009 16:23:30.337755  5140 net.cpp:122] Setting up relu1_2
I1009 16:23:30.337777  5140 net.cpp:129] Top shape: 1 64 838 678 (36362496)
I1009 16:23:30.337782  5140 net.cpp:137] Memory required for data: 594087936
I1009 16:23:30.337788  5140 layer_factory.hpp:77] Creating layer pool1
I1009 16:23:30.337800  5140 net.cpp:84] Creating Layer pool1
I1009 16:23:30.337806  5140 net.cpp:406] pool1 <- conv1_2
I1009 16:23:30.337813  5140 net.cpp:380] pool1 -> pool1
I1009 16:23:30.337877  5140 net.cpp:122] Setting up pool1
I1009 16:23:30.337885  5140 net.cpp:129] Top shape: 1 64 419 339 (9090624)
I1009 16:23:30.337890  5140 net.cpp:137] Memory required for data: 630450432
I1009 16:23:30.337896  5140 layer_factory.hpp:77] Creating layer conv2_1
I1009 16:23:30.337906  5140 net.cpp:84] Creating Layer conv2_1
I1009 16:23:30.337913  5140 net.cpp:406] conv2_1 <- pool1
I1009 16:23:30.337920  5140 net.cpp:380] conv2_1 -> conv2_1
I1009 16:23:30.340560  5140 net.cpp:122] Setting up conv2_1
I1009 16:23:30.340607  5140 net.cpp:129] Top shape: 1 128 419 339 (18181248)
I1009 16:23:30.340615  5140 net.cpp:137] Memory required for data: 703175424
I1009 16:23:30.340637  5140 layer_factory.hpp:77] Creating layer relu2_1
I1009 16:23:30.340651  5140 net.cpp:84] Creating Layer relu2_1
I1009 16:23:30.340658  5140 net.cpp:406] relu2_1 <- conv2_1
I1009 16:23:30.340667  5140 net.cpp:367] relu2_1 -> conv2_1 (in-place)
I1009 16:23:30.340857  5140 net.cpp:122] Setting up relu2_1
I1009 16:23:30.340869  5140 net.cpp:129] Top shape: 1 128 419 339 (18181248)
I1009 16:23:30.340874  5140 net.cpp:137] Memory required for data: 775900416
I1009 16:23:30.340880  5140 layer_factory.hpp:77] Creating layer conv2_2
I1009 16:23:30.340889  5140 net.cpp:84] Creating Layer conv2_2
I1009 16:23:30.340895  5140 net.cpp:406] conv2_2 <- conv2_1
I1009 16:23:30.340903  5140 net.cpp:380] conv2_2 -> conv2_2
I1009 16:23:30.342480  5140 net.cpp:122] Setting up conv2_2
I1009 16:23:30.342496  5140 net.cpp:129] Top shape: 1 128 419 339 (18181248)
I1009 16:23:30.342502  5140 net.cpp:137] Memory required for data: 848625408
I1009 16:23:30.342510  5140 layer_factory.hpp:77] Creating layer relu2_2
I1009 16:23:30.342519  5140 net.cpp:84] Creating Layer relu2_2
I1009 16:23:30.342525  5140 net.cpp:406] relu2_2 <- conv2_2
I1009 16:23:30.342530  5140 net.cpp:367] relu2_2 -> conv2_2 (in-place)
I1009 16:23:30.342672  5140 net.cpp:122] Setting up relu2_2
I1009 16:23:30.342682  5140 net.cpp:129] Top shape: 1 128 419 339 (18181248)
I1009 16:23:30.342689  5140 net.cpp:137] Memory required for data: 921350400
I1009 16:23:30.342694  5140 layer_factory.hpp:77] Creating layer pool2
I1009 16:23:30.342701  5140 net.cpp:84] Creating Layer pool2
I1009 16:23:30.342707  5140 net.cpp:406] pool2 <- conv2_2
I1009 16:23:30.342713  5140 net.cpp:380] pool2 -> pool2
I1009 16:23:30.342752  5140 net.cpp:122] Setting up pool2
I1009 16:23:30.342761  5140 net.cpp:129] Top shape: 1 128 210 170 (4569600)
I1009 16:23:30.342766  5140 net.cpp:137] Memory required for data: 939628800
I1009 16:23:30.342770  5140 layer_factory.hpp:77] Creating layer conv3_1
I1009 16:23:30.342778  5140 net.cpp:84] Creating Layer conv3_1
I1009 16:23:30.342783  5140 net.cpp:406] conv3_1 <- pool2
I1009 16:23:30.342790  5140 net.cpp:380] conv3_1 -> conv3_1
I1009 16:23:30.344796  5140 net.cpp:122] Setting up conv3_1
I1009 16:23:30.344813  5140 net.cpp:129] Top shape: 1 256 210 170 (9139200)
I1009 16:23:30.344820  5140 net.cpp:137] Memory required for data: 976185600
I1009 16:23:30.344830  5140 layer_factory.hpp:77] Creating layer relu3_1
I1009 16:23:30.344840  5140 net.cpp:84] Creating Layer relu3_1
I1009 16:23:30.344846  5140 net.cpp:406] relu3_1 <- conv3_1
I1009 16:23:30.344851  5140 net.cpp:367] relu3_1 -> conv3_1 (in-place)
I1009 16:23:30.345386  5140 net.cpp:122] Setting up relu3_1
I1009 16:23:30.345401  5140 net.cpp:129] Top shape: 1 256 210 170 (9139200)
I1009 16:23:30.345407  5140 net.cpp:137] Memory required for data: 1012742400
I1009 16:23:30.345412  5140 layer_factory.hpp:77] Creating layer conv3_2
I1009 16:23:30.345422  5140 net.cpp:84] Creating Layer conv3_2
I1009 16:23:30.345427  5140 net.cpp:406] conv3_2 <- conv3_1
I1009 16:23:30.345434  5140 net.cpp:380] conv3_2 -> conv3_2
I1009 16:23:30.348095  5140 net.cpp:122] Setting up conv3_2
I1009 16:23:30.348139  5140 net.cpp:129] Top shape: 1 256 210 170 (9139200)
I1009 16:23:30.348145  5140 net.cpp:137] Memory required for data: 1049299200
I1009 16:23:30.348155  5140 layer_factory.hpp:77] Creating layer relu3_2
I1009 16:23:30.348166  5140 net.cpp:84] Creating Layer relu3_2
I1009 16:23:30.348173  5140 net.cpp:406] relu3_2 <- conv3_2
I1009 16:23:30.348193  5140 net.cpp:367] relu3_2 -> conv3_2 (in-place)
I1009 16:23:30.348347  5140 net.cpp:122] Setting up relu3_2
I1009 16:23:30.348358  5140 net.cpp:129] Top shape: 1 256 210 170 (9139200)
I1009 16:23:30.348363  5140 net.cpp:137] Memory required for data: 1085856000
I1009 16:23:30.348369  5140 layer_factory.hpp:77] Creating layer conv3_3
I1009 16:23:30.348382  5140 net.cpp:84] Creating Layer conv3_3
I1009 16:23:30.348387  5140 net.cpp:406] conv3_3 <- conv3_2
I1009 16:23:30.348394  5140 net.cpp:380] conv3_3 -> conv3_3
I1009 16:23:30.350980  5140 net.cpp:122] Setting up conv3_3
I1009 16:23:30.351066  5140 net.cpp:129] Top shape: 1 256 210 170 (9139200)
I1009 16:23:30.351073  5140 net.cpp:137] Memory required for data: 1122412800
I1009 16:23:30.351088  5140 layer_factory.hpp:77] Creating layer relu3_3
I1009 16:23:30.351105  5140 net.cpp:84] Creating Layer relu3_3
I1009 16:23:30.351116  5140 net.cpp:406] relu3_3 <- conv3_3
I1009 16:23:30.351126  5140 net.cpp:367] relu3_3 -> conv3_3 (in-place)
I1009 16:23:30.351317  5140 net.cpp:122] Setting up relu3_3
I1009 16:23:30.351344  5140 net.cpp:129] Top shape: 1 256 210 170 (9139200)
I1009 16:23:30.351351  5140 net.cpp:137] Memory required for data: 1158969600
I1009 16:23:30.351356  5140 layer_factory.hpp:77] Creating layer pool3
I1009 16:23:30.351368  5140 net.cpp:84] Creating Layer pool3
I1009 16:23:30.351374  5140 net.cpp:406] pool3 <- conv3_3
I1009 16:23:30.351382  5140 net.cpp:380] pool3 -> pool3
I1009 16:23:30.351449  5140 net.cpp:122] Setting up pool3
I1009 16:23:30.351459  5140 net.cpp:129] Top shape: 1 256 105 85 (2284800)
I1009 16:23:30.351464  5140 net.cpp:137] Memory required for data: 1168108800
I1009 16:23:30.351469  5140 layer_factory.hpp:77] Creating layer conv4_1
I1009 16:23:30.351480  5140 net.cpp:84] Creating Layer conv4_1
I1009 16:23:30.351486  5140 net.cpp:406] conv4_1 <- pool3
I1009 16:23:30.351492  5140 net.cpp:380] conv4_1 -> conv4_1
I1009 16:23:30.356215  5140 net.cpp:122] Setting up conv4_1
I1009 16:23:30.356245  5140 net.cpp:129] Top shape: 1 512 105 85 (4569600)
I1009 16:23:30.356252  5140 net.cpp:137] Memory required for data: 1186387200
I1009 16:23:30.356262  5140 layer_factory.hpp:77] Creating layer relu4_1
I1009 16:23:30.356274  5140 net.cpp:84] Creating Layer relu4_1
I1009 16:23:30.356281  5140 net.cpp:406] relu4_1 <- conv4_1
I1009 16:23:30.356288  5140 net.cpp:367] relu4_1 -> conv4_1 (in-place)
I1009 16:23:30.356758  5140 net.cpp:122] Setting up relu4_1
I1009 16:23:30.356773  5140 net.cpp:129] Top shape: 1 512 105 85 (4569600)
I1009 16:23:30.356779  5140 net.cpp:137] Memory required for data: 1204665600
I1009 16:23:30.356784  5140 layer_factory.hpp:77] Creating layer conv4_2
I1009 16:23:30.356794  5140 net.cpp:84] Creating Layer conv4_2
I1009 16:23:30.356801  5140 net.cpp:406] conv4_2 <- conv4_1
I1009 16:23:30.356808  5140 net.cpp:380] conv4_2 -> conv4_2
I1009 16:23:30.362445  5140 net.cpp:122] Setting up conv4_2
I1009 16:23:30.362480  5140 net.cpp:129] Top shape: 1 512 105 85 (4569600)
I1009 16:23:30.362488  5140 net.cpp:137] Memory required for data: 1222944000
I1009 16:23:30.362504  5140 layer_factory.hpp:77] Creating layer relu4_2
I1009 16:23:30.362515  5140 net.cpp:84] Creating Layer relu4_2
I1009 16:23:30.362522  5140 net.cpp:406] relu4_2 <- conv4_2
I1009 16:23:30.362529  5140 net.cpp:367] relu4_2 -> conv4_2 (in-place)
I1009 16:23:30.362686  5140 net.cpp:122] Setting up relu4_2
I1009 16:23:30.362697  5140 net.cpp:129] Top shape: 1 512 105 85 (4569600)
I1009 16:23:30.362704  5140 net.cpp:137] Memory required for data: 1241222400
I1009 16:23:30.362709  5140 layer_factory.hpp:77] Creating layer conv4_3
I1009 16:23:30.362718  5140 net.cpp:84] Creating Layer conv4_3
I1009 16:23:30.362725  5140 net.cpp:406] conv4_3 <- conv4_2
I1009 16:23:30.362731  5140 net.cpp:380] conv4_3 -> conv4_3
I1009 16:23:30.369655  5140 net.cpp:122] Setting up conv4_3
I1009 16:23:30.369688  5140 net.cpp:129] Top shape: 1 512 105 85 (4569600)
I1009 16:23:30.369694  5140 net.cpp:137] Memory required for data: 1259500800
I1009 16:23:30.369706  5140 layer_factory.hpp:77] Creating layer relu4_3
I1009 16:23:30.369719  5140 net.cpp:84] Creating Layer relu4_3
I1009 16:23:30.369725  5140 net.cpp:406] relu4_3 <- conv4_3
I1009 16:23:30.369734  5140 net.cpp:367] relu4_3 -> conv4_3 (in-place)
I1009 16:23:30.369890  5140 net.cpp:122] Setting up relu4_3
I1009 16:23:30.369902  5140 net.cpp:129] Top shape: 1 512 105 85 (4569600)
I1009 16:23:30.369909  5140 net.cpp:137] Memory required for data: 1277779200
I1009 16:23:30.369913  5140 layer_factory.hpp:77] Creating layer pool4
I1009 16:23:30.369922  5140 net.cpp:84] Creating Layer pool4
I1009 16:23:30.369928  5140 net.cpp:406] pool4 <- conv4_3
I1009 16:23:30.369935  5140 net.cpp:380] pool4 -> pool4
I1009 16:23:30.369982  5140 net.cpp:122] Setting up pool4
I1009 16:23:30.369990  5140 net.cpp:129] Top shape: 1 512 53 43 (1166848)
I1009 16:23:30.369995  5140 net.cpp:137] Memory required for data: 1282446592
I1009 16:23:30.370000  5140 layer_factory.hpp:77] Creating layer conv5_1
I1009 16:23:30.370010  5140 net.cpp:84] Creating Layer conv5_1
I1009 16:23:30.370016  5140 net.cpp:406] conv5_1 <- pool4
I1009 16:23:30.370023  5140 net.cpp:380] conv5_1 -> conv5_1
I1009 16:23:30.376606  5140 net.cpp:122] Setting up conv5_1
I1009 16:23:30.376636  5140 net.cpp:129] Top shape: 1 512 53 43 (1166848)
I1009 16:23:30.376641  5140 net.cpp:137] Memory required for data: 1287113984
I1009 16:23:30.376652  5140 layer_factory.hpp:77] Creating layer relu5_1
I1009 16:23:30.376662  5140 net.cpp:84] Creating Layer relu5_1
I1009 16:23:30.376668  5140 net.cpp:406] relu5_1 <- conv5_1
I1009 16:23:30.376677  5140 net.cpp:367] relu5_1 -> conv5_1 (in-place)
I1009 16:23:30.376823  5140 net.cpp:122] Setting up relu5_1
I1009 16:23:30.376833  5140 net.cpp:129] Top shape: 1 512 53 43 (1166848)
I1009 16:23:30.376839  5140 net.cpp:137] Memory required for data: 1291781376
I1009 16:23:30.376844  5140 layer_factory.hpp:77] Creating layer conv5_2
I1009 16:23:30.376853  5140 net.cpp:84] Creating Layer conv5_2
I1009 16:23:30.376859  5140 net.cpp:406] conv5_2 <- conv5_1
I1009 16:23:30.376866  5140 net.cpp:380] conv5_2 -> conv5_2
I1009 16:23:30.383059  5140 net.cpp:122] Setting up conv5_2
I1009 16:23:30.383124  5140 net.cpp:129] Top shape: 1 512 53 43 (1166848)
I1009 16:23:30.383132  5140 net.cpp:137] Memory required for data: 1296448768
I1009 16:23:30.383147  5140 layer_factory.hpp:77] Creating layer relu5_2
I1009 16:23:30.383162  5140 net.cpp:84] Creating Layer relu5_2
I1009 16:23:30.383180  5140 net.cpp:406] relu5_2 <- conv5_2
I1009 16:23:30.383190  5140 net.cpp:367] relu5_2 -> conv5_2 (in-place)
I1009 16:23:30.383919  5140 net.cpp:122] Setting up relu5_2
I1009 16:23:30.383960  5140 net.cpp:129] Top shape: 1 512 53 43 (1166848)
I1009 16:23:30.383965  5140 net.cpp:137] Memory required for data: 1301116160
I1009 16:23:30.383972  5140 layer_factory.hpp:77] Creating layer conv5_3
I1009 16:23:30.383996  5140 net.cpp:84] Creating Layer conv5_3
I1009 16:23:30.384002  5140 net.cpp:406] conv5_3 <- conv5_2
I1009 16:23:30.384011  5140 net.cpp:380] conv5_3 -> conv5_3
I1009 16:23:30.389905  5140 net.cpp:122] Setting up conv5_3
I1009 16:23:30.389941  5140 net.cpp:129] Top shape: 1 512 53 43 (1166848)
I1009 16:23:30.389961  5140 net.cpp:137] Memory required for data: 1305783552
I1009 16:23:30.389974  5140 layer_factory.hpp:77] Creating layer relu5_3
I1009 16:23:30.389987  5140 net.cpp:84] Creating Layer relu5_3
I1009 16:23:30.390004  5140 net.cpp:406] relu5_3 <- conv5_3
I1009 16:23:30.390013  5140 net.cpp:367] relu5_3 -> conv5_3 (in-place)
I1009 16:23:30.390177  5140 net.cpp:122] Setting up relu5_3
I1009 16:23:30.390188  5140 net.cpp:129] Top shape: 1 512 53 43 (1166848)
I1009 16:23:30.390193  5140 net.cpp:137] Memory required for data: 1310450944
I1009 16:23:30.390213  5140 layer_factory.hpp:77] Creating layer pool5
I1009 16:23:30.390224  5140 net.cpp:84] Creating Layer pool5
I1009 16:23:30.390230  5140 net.cpp:406] pool5 <- conv5_3
I1009 16:23:30.390238  5140 net.cpp:380] pool5 -> pool5
I1009 16:23:30.390291  5140 net.cpp:122] Setting up pool5
I1009 16:23:30.390300  5140 net.cpp:129] Top shape: 1 512 27 22 (304128)
I1009 16:23:30.390305  5140 net.cpp:137] Memory required for data: 1311667456
I1009 16:23:30.390311  5140 layer_factory.hpp:77] Creating layer fc6
I1009 16:23:30.390333  5140 net.cpp:84] Creating Layer fc6
I1009 16:23:30.390347  5140 net.cpp:406] fc6 <- pool5
I1009 16:23:30.390353  5140 net.cpp:380] fc6 -> fc6
I1009 16:23:30.577654  5140 net.cpp:122] Setting up fc6
I1009 16:23:30.577702  5140 net.cpp:129] Top shape: 1 4096 21 16 (1376256)
I1009 16:23:30.577708  5140 net.cpp:137] Memory required for data: 1317172480
I1009 16:23:30.577718  5140 layer_factory.hpp:77] Creating layer relu6
I1009 16:23:30.577735  5140 net.cpp:84] Creating Layer relu6
I1009 16:23:30.577744  5140 net.cpp:406] relu6 <- fc6
I1009 16:23:30.577754  5140 net.cpp:367] relu6 -> fc6 (in-place)
I1009 16:23:30.577922  5140 net.cpp:122] Setting up relu6
I1009 16:23:30.577949  5140 net.cpp:129] Top shape: 1 4096 21 16 (1376256)
I1009 16:23:30.577955  5140 net.cpp:137] Memory required for data: 1322677504
I1009 16:23:30.577965  5140 layer_factory.hpp:77] Creating layer drop6
I1009 16:23:30.577972  5140 net.cpp:84] Creating Layer drop6
I1009 16:23:30.577978  5140 net.cpp:406] drop6 <- fc6
I1009 16:23:30.577985  5140 net.cpp:367] drop6 -> fc6 (in-place)
I1009 16:23:30.578011  5140 net.cpp:122] Setting up drop6
I1009 16:23:30.578019  5140 net.cpp:129] Top shape: 1 4096 21 16 (1376256)
I1009 16:23:30.578024  5140 net.cpp:137] Memory required for data: 1328182528
I1009 16:23:30.578029  5140 layer_factory.hpp:77] Creating layer fc7
I1009 16:23:30.578038  5140 net.cpp:84] Creating Layer fc7
I1009 16:23:30.578044  5140 net.cpp:406] fc7 <- fc6
I1009 16:23:30.578052  5140 net.cpp:380] fc7 -> fc7
I1009 16:23:30.616230  5140 net.cpp:122] Setting up fc7
I1009 16:23:30.616313  5140 net.cpp:129] Top shape: 1 4096 21 16 (1376256)
I1009 16:23:30.616320  5140 net.cpp:137] Memory required for data: 1333687552
I1009 16:23:30.616336  5140 layer_factory.hpp:77] Creating layer relu7
I1009 16:23:30.616359  5140 net.cpp:84] Creating Layer relu7
I1009 16:23:30.616369  5140 net.cpp:406] relu7 <- fc7
I1009 16:23:30.616379  5140 net.cpp:367] relu7 -> fc7 (in-place)
I1009 16:23:30.617022  5140 net.cpp:122] Setting up relu7
I1009 16:23:30.617038  5140 net.cpp:129] Top shape: 1 4096 21 16 (1376256)
I1009 16:23:30.617044  5140 net.cpp:137] Memory required for data: 1339192576
I1009 16:23:30.617064  5140 layer_factory.hpp:77] Creating layer drop7
I1009 16:23:30.617074  5140 net.cpp:84] Creating Layer drop7
I1009 16:23:30.617079  5140 net.cpp:406] drop7 <- fc7
I1009 16:23:30.617089  5140 net.cpp:367] drop7 -> fc7 (in-place)
I1009 16:23:30.617132  5140 net.cpp:122] Setting up drop7
I1009 16:23:30.617141  5140 net.cpp:129] Top shape: 1 4096 21 16 (1376256)
I1009 16:23:30.617146  5140 net.cpp:137] Memory required for data: 1344697600
I1009 16:23:30.617151  5140 layer_factory.hpp:77] Creating layer score_fr
I1009 16:23:30.617174  5140 net.cpp:84] Creating Layer score_fr
I1009 16:23:30.617182  5140 net.cpp:406] score_fr <- fc7
I1009 16:23:30.617190  5140 net.cpp:380] score_fr -> score_fr
I1009 16:23:30.618475  5140 net.cpp:122] Setting up score_fr
I1009 16:23:30.618491  5140 net.cpp:129] Top shape: 1 40 21 16 (13440)
I1009 16:23:30.618496  5140 net.cpp:137] Memory required for data: 1344751360
I1009 16:23:30.618520  5140 layer_factory.hpp:77] Creating layer upscore
I1009 16:23:30.618530  5140 net.cpp:84] Creating Layer upscore
I1009 16:23:30.618535  5140 net.cpp:406] upscore <- score_fr
I1009 16:23:30.618542  5140 net.cpp:380] upscore -> upscore
I1009 16:23:30.633252  5140 net.cpp:122] Setting up upscore
I1009 16:23:30.633301  5140 net.cpp:129] Top shape: 1 40 704 544 (15319040)
I1009 16:23:30.633307  5140 net.cpp:137] Memory required for data: 1406027520
I1009 16:23:30.633325  5140 layer_factory.hpp:77] Creating layer score
I1009 16:23:30.633342  5140 net.cpp:84] Creating Layer score
I1009 16:23:30.633349  5140 net.cpp:406] score <- upscore
I1009 16:23:30.633358  5140 net.cpp:406] score <- data_data_0_split_1
I1009 16:23:30.633366  5140 net.cpp:380] score -> score
I1009 16:23:30.633479  5140 net.cpp:122] Setting up score
I1009 16:23:30.633491  5140 net.cpp:129] Top shape: 1 40 640 480 (12288000)
I1009 16:23:30.633498  5140 net.cpp:137] Memory required for data: 1455179520
I1009 16:23:30.633504  5140 layer_factory.hpp:77] Creating layer loss
I1009 16:23:30.633513  5140 net.cpp:84] Creating Layer loss
I1009 16:23:30.633520  5140 net.cpp:406] loss <- score
I1009 16:23:30.633527  5140 net.cpp:406] loss <- label
I1009 16:23:30.633534  5140 net.cpp:380] loss -> loss
I1009 16:23:30.633544  5140 layer_factory.hpp:77] Creating layer loss
I1009 16:23:30.662021  5140 net.cpp:122] Setting up loss
I1009 16:23:30.662072  5140 net.cpp:129] Top shape: (1)
I1009 16:23:30.662080  5140 net.cpp:132]     with loss weight 1
I1009 16:23:30.662096  5140 net.cpp:137] Memory required for data: 1455179524
I1009 16:23:30.662103  5140 net.cpp:198] loss needs backward computation.
I1009 16:23:30.662111  5140 net.cpp:198] score needs backward computation.
I1009 16:23:30.662118  5140 net.cpp:198] upscore needs backward computation.
I1009 16:23:30.662124  5140 net.cpp:198] score_fr needs backward computation.
I1009 16:23:30.662130  5140 net.cpp:198] drop7 needs backward computation.
I1009 16:23:30.662139  5140 net.cpp:198] relu7 needs backward computation.
I1009 16:23:30.662145  5140 net.cpp:198] fc7 needs backward computation.
I1009 16:23:30.662150  5140 net.cpp:198] drop6 needs backward computation.
I1009 16:23:30.662156  5140 net.cpp:198] relu6 needs backward computation.
I1009 16:23:30.662161  5140 net.cpp:198] fc6 needs backward computation.
I1009 16:23:30.662168  5140 net.cpp:198] pool5 needs backward computation.
I1009 16:23:30.662174  5140 net.cpp:198] relu5_3 needs backward computation.
I1009 16:23:30.662181  5140 net.cpp:198] conv5_3 needs backward computation.
I1009 16:23:30.662187  5140 net.cpp:198] relu5_2 needs backward computation.
I1009 16:23:30.662194  5140 net.cpp:198] conv5_2 needs backward computation.
I1009 16:23:30.662199  5140 net.cpp:198] relu5_1 needs backward computation.
I1009 16:23:30.662204  5140 net.cpp:198] conv5_1 needs backward computation.
I1009 16:23:30.662210  5140 net.cpp:198] pool4 needs backward computation.
I1009 16:23:30.662216  5140 net.cpp:198] relu4_3 needs backward computation.
I1009 16:23:30.662223  5140 net.cpp:198] conv4_3 needs backward computation.
I1009 16:23:30.662230  5140 net.cpp:198] relu4_2 needs backward computation.
I1009 16:23:30.662237  5140 net.cpp:198] conv4_2 needs backward computation.
I1009 16:23:30.662243  5140 net.cpp:198] relu4_1 needs backward computation.
I1009 16:23:30.662250  5140 net.cpp:198] conv4_1 needs backward computation.
I1009 16:23:30.662256  5140 net.cpp:198] pool3 needs backward computation.
I1009 16:23:30.662261  5140 net.cpp:198] relu3_3 needs backward computation.
I1009 16:23:30.662267  5140 net.cpp:198] conv3_3 needs backward computation.
I1009 16:23:30.662273  5140 net.cpp:198] relu3_2 needs backward computation.
I1009 16:23:30.662279  5140 net.cpp:198] conv3_2 needs backward computation.
I1009 16:23:30.662284  5140 net.cpp:198] relu3_1 needs backward computation.
I1009 16:23:30.662291  5140 net.cpp:198] conv3_1 needs backward computation.
I1009 16:23:30.662297  5140 net.cpp:198] pool2 needs backward computation.
I1009 16:23:30.662302  5140 net.cpp:198] relu2_2 needs backward computation.
I1009 16:23:30.662308  5140 net.cpp:198] conv2_2 needs backward computation.
I1009 16:23:30.662314  5140 net.cpp:198] relu2_1 needs backward computation.
I1009 16:23:30.662320  5140 net.cpp:198] conv2_1 needs backward computation.
I1009 16:23:30.662326  5140 net.cpp:198] pool1 needs backward computation.
I1009 16:23:30.662331  5140 net.cpp:198] relu1_2 needs backward computation.
I1009 16:23:30.662338  5140 net.cpp:198] conv1_2 needs backward computation.
I1009 16:23:30.662343  5140 net.cpp:198] relu1_1 needs backward computation.
I1009 16:23:30.662348  5140 net.cpp:198] conv1_1 needs backward computation.
I1009 16:23:30.662355  5140 net.cpp:200] data_data_0_split does not need backward computation.
I1009 16:23:30.662361  5140 net.cpp:200] data does not need backward computation.
I1009 16:23:30.662367  5140 net.cpp:242] This network produces output loss
I1009 16:23:30.662392  5140 net.cpp:255] Network initialization done.
I1009 16:23:30.662487  5140 solver.cpp:56] Solver scaffolding done.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 563915216
F1009 16:23:35.350841  5140 math_functions.cu:141] Check failed: status == CUBLAS_STATUS_SUCCESS (11 vs. 0)  CUBLAS_STATUS_MAPPING_ERROR
*** Check failure stack trace: ***
Aborted (core dumped)

[Przemek D]

This error happened to me with FCNs when the final output layer produced less output channels than the classes in my dataset. Your network has 40 output channels - are you sure this is the correct number for the NYUDv2 data? I have never used it myself.
On the other hand, this message is sometimes so cryptic that it's hard to pinpoint where exactly does the problem lie...

[章程]

thank you for your prompt reply. I will check the class of outputs to see if I can correct it 

在 2017年10月9日星期一 UTC+8下午10:02:22，Przemek D写道：

[章程]

thank you very much. you are right, the FCN source code doesn't use the official release of nuydv2 but the version from here whose class is 40.

在 2017年10月9日星期一 UTC+8下午10:02:22，Przemek D写道：

This error happened to me with FCNs when the final output layer produced less output channels than the classes in my dataset. Your network has 40 output channels - are you sure this is the correct number for the NYUDv2 data? I have never used it myself.