Matcaffe: Matlab crashes on caffe('forward', inputImage)

[Oksana Meledin]

I have trained a CNN model that takes as input 64 x 64 x 3 images.

The deploy.prototxt network definition file expects images in batches of 5000 and outputs a feature vector. The input is a 1x1 cell array with content 64 x 64 x 3 x 5000 image of type 'single'.

After executing

matcaffe_init(true, 'deploy.prototxt', 'weights.caffemodel')

the 'forward' method is called

feature = caffe('forward', inputImage);

This causes Matlab to crash with "MATLAB has encountered an internal problem and needs to close." See dump below.

Yes, caffe('is_initialized') returns 1. matcaffe_demo works.

Here are some details about my system.
Ubuntu 14.04 LTS
CUDA 7.0
NVIDIA GTX TITAN Z with driver 346.96
gcc 4.8.4
Matlab 2015b
Compiled using GPU mode
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
                -gencode arch=compute_20,code=sm_21 \
                -gencode arch=compute_30,code=sm_30 \
                -gencode arch=compute_35,code=sm_35 \
                -gencode arch=compute_50,code=sm_50 \
                -gencode arch=compute_50,code=compute_

50

---- crash dump -----

------------------------------------------------------------------------
              abort() detected at Wed Oct 28 16:38:19 2015
------------------------------------------------------------------------

Configuration:
  Crash Decoding      : Disabled
  Crash Mode          : continue (default)
  Current Graphics Driver: Unknown hardware
  Current Visual      : 0x21 (class 4, depth 24)
  Default Encoding    : UTF-8
  GNU C Library       : 2.19 stable
  Host Name           : CNN00
  MATLAB Architecture : glnxa64
  MATLAB Root         : /usr/local/MATLAB/R2015b
  MATLAB Version      : 8.6.0.267246 (R2015b)
  OpenGL              : hardware
  Operating System    : Linux 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64
  Processor ID        : x86 Family 6 Model 63 Stepping 2, GenuineIntel
  Virtual Machine     : Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
  Window System       : The X.Org Foundation (11701000), display :0

Fault Count: 1


Abnormal termination:
abort()

Register State (from fault):
  RAX = 0000000000000000  RBX = 00007f6e51f6d620
  RCX = ffffffffffffffff  RDX = 0000000000000006
  RSP = 00007f6f99613b38  RBP = 00007f6f99613c70
  RSI = 0000000000001ece  RDI = 0000000000001ea4

   R8 = 000000000000ff08   R9 = 00007f6fb05d5d40
  R10 = 0000000000000008  R11 = 0000000000000202
  R12 = 0000001402ca0000  R13 = 00000014510c0000
  R14 = 00007f6f0af8a648  R15 = 00007f6e29575e50

  RIP = 00007f6fb04bbcc9  EFL = 0000000000000202

   CS = 0033   FS = 0000   GS = 0000

Stack Trace (from fault):
[  0] 0x00007f6fb04bbcc9                    /lib/x86_64-linux-gnu/libc.so.6+00224457 gsignal+00000057
[  1] 0x00007f6fb04bf0d8                    /lib/x86_64-linux-gnu/libc.so.6+00237784 abort+00000328
[  2] 0x00007f6e51d47d81             /usr/lib/x86_64-linux-gnu/libglog.so.0+00068993 _ZN6google22InstallFailureFunctionEPFvvE+00000000
[  3] 0x00007f6e51d47daa             /usr/lib/x86_64-linux-gnu/libglog.so.0+00069034 _ZN6google10LogMessage10SendToSinkEv+00000000
[  4] 0x00007f6e51d47ce4             /usr/lib/x86_64-linux-gnu/libglog.so.0+00068836 _ZN6google10LogMessage9SendToLogEv+00001224
[  5] 0x00007f6e51d476e6             /usr/lib/x86_64-linux-gnu/libglog.so.0+00067302 _ZN6google10LogMessage5FlushEv+00000414
[  6] 0x00007f6e51d4a687             /usr/lib/x86_64-linux-gnu/libglog.so.0+00079495 _ZN6google15LogMessageFatalD1Ev+00000025
[  7] 0x00007f6e57b6e0cb         /usr/local/caffe/matlab/caffe/caffe.mexa64+01392843
[  8] 0x00007f6e57a958f2         /usr/local/caffe/matlab/caffe/caffe.mexa64+00506098
[  9] 0x00007f6e57b913dc         /usr/local/caffe/matlab/caffe/caffe.mexa64+01536988
[ 10] 0x00007f6e57b7894f         /usr/local/caffe/matlab/caffe/caffe.mexa64+01435983
[ 11] 0x00007f6e57b78d95         /usr/local/caffe/matlab/caffe/caffe.mexa64+01437077
[ 12] 0x00007f6e57a4d13c         /usr/local/caffe/matlab/caffe/caffe.mexa64+00209212
[ 13] 0x00007f6e57a4a3c0         /usr/local/caffe/matlab/caffe/caffe.mexa64+00197568 mexFunction+00000363
[ 14] 0x00007f6fa3055c4a     /usr/local/MATLAB/R2015b/bin/glnxa64/libmex.so+00142410 mexRunMexFile+00000090
[ 15] 0x00007f6fa3052244     /usr/local/MATLAB/R2015b/bin/glnxa64/libmex.so+00127556
[ 16] 0x00007f6fa3052de4     /usr/local/MATLAB/R2015b/bin/glnxa64/libmex.so+00130532
[ 17] 0x00007f6fa75f4dbd /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00724413 _ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_+00001549
[ 18] 0x00007f6fa75f5250 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00725584 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000032
[ 19] 0x00007f6fa1c6f8af /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+08612015
[ 20] 0x00007f6fa1d937ff /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09807871
[ 21] 0x00007f6fa1d89254 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09765460
[ 22] 0x00007f6fa1d56901 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09558273
[ 23] 0x00007f6fa19b8d6c /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05766508
[ 24] 0x00007f6fa19a34c1 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05678273
[ 25] 0x00007f6fa19b2075 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05738613
[ 26] 0x00007f6fa1b83033 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07643187
[ 27] 0x00007f6fa1c613bc /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+08553404
[ 28] 0x00007f6fa75f4b31 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00723761 _ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_+00000897
[ 29] 0x00007f6fa75f5250 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_dispatcher.so+00725584 _ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_+00000032
[ 30] 0x00007f6fa1c6f8af /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+08612015
[ 31] 0x00007f6fa1d937ff /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09807871
[ 32] 0x00007f6fa1d8947f /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09766015
[ 33] 0x00007f6fa1d56981 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+09558401
[ 34] 0x00007f6fa19b8d6c /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05766508
[ 35] 0x00007f6fa19a34c1 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05678273
[ 36] 0x00007f6fa19b2075 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+05738613
[ 37] 0x00007f6fa1b83033 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07643187
[ 38] 0x00007f6fa1b4ac40 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07412800
[ 39] 0x00007f6fa1b4d078 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07422072
[ 40] 0x00007f6fa1b4d140 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07422272
[ 41] 0x00007f6fa1bc46bc /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07911100
[ 42] 0x00007f6fa1bc4b42 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwm_lxe.so+07912258
[ 43] 0x00007f6fa32892bc /usr/local/MATLAB/R2015b/bin/glnxa64/libmwbridge.so+00168636
[ 44] 0x00007f6fa3289ef7 /usr/local/MATLAB/R2015b/bin/glnxa64/libmwbridge.so+00171767 _Z8mnParserv+00000823
[ 45] 0x00007f6fa78dbb4f   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00686927 _ZN11mcrInstance30mnParser_on_interpreter_threadEv+00000031
[ 46] 0x00007f6fa78c8443   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00607299
[ 47] 0x00007f6fa78c8a39   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00608825 _ZN5boost6detail11task_objectIvNS_3_bi6bind_tIvPFvRKNS_8functionIFvvEEEENS2_5list1INS2_5valueIS6_EEEEEEE6do_runEv+00000025
[ 48] 0x00007f6fa78c9f47   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00614215 _ZN5boost6detail9task_baseIvE3runEv+00000071
[ 49] 0x00007f6fa78c9fa7   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00614311
[ 50] 0x00007f6fa78c52fa   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00594682
[ 51] 0x00007f6f9b07e6b6   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwuix.so+00313014
[ 52] 0x00007f6f9b064862   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwuix.so+00206946
[ 53] 0x00007f6fa825b0df /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03002591
[ 54] 0x00007f6fa825b24c /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03002956
[ 55] 0x00007f6fa825cd9f /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03009951
[ 56] 0x00007f6fa825d84c /usr/local/MATLAB/R2015b/bin/glnxa64/libmwservices.so+03012684 _Z25svWS_ProcessPendingEventsiib+00000092
[ 57] 0x00007f6fa78c59b8   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00596408
[ 58] 0x00007f6fa78c5cd4   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00597204
[ 59] 0x00007f6fa78b1fed   /usr/local/MATLAB/R2015b/bin/glnxa64/libmwmcr.so+00516077
[ 60] 0x00007f6fb0852182              /lib/x86_64-linux-gnu/libpthread.so.0+00033154
[ 61] 0x00007f6fb057f47d                    /lib/x86_64-linux-gnu/libc.so.6+01025149 clone+00000109
[ 62] 0x0000000000000000                                   <unknown-module>+00000000


This error was detected while a MEX-file was running. If the MEX-file
is not an official MathWorks function, please examine its source code
for errors. Please consult the External Interfaces Guide for information
on debugging MEX-files.

If this problem is reproducible, please submit a Service Request via:
    http://www.mathworks.com/support/contact_us/

A technical support engineer might contact you with further information.

Any help would be appreciated.

[Silvia Vinyes]

Hi Oksana,

I was having the same problem until I found out that the input data should be of cell{} format with single values in; this solved it for me.

Found the solution here: https://github.com/BVLC/caffe/issues/1544

Silvia

[Developer]

can you give me a tutorial that will help me to run caffe on command line of  matlab