Matcaffe training net produces "Data layer prefetch queue empty"

[Fábio Ferreira]

Hi,

I'm trying to figure out why my MatCaffe implementation cannot pop from my train lmdb, which I've created using X.

What I do is basically just this:

solver = caffe.Solver(solverFile);
solver.step(500);

and when looking at the terminal, the output after the last statement is this:

I0322 11:15:11.830241 **098 net.cpp:228] data does not need backward computation.
I0322 11:15:11.830250 **098 net.cpp:270] This network produces output accuracy
I0322 11:15:11.830257 **098 net.cpp:270] This network produces output loss
I0322 11:15:11.830281 **098 net.cpp:283] Network initialization done.
I0322 11:15:11.830377 **098 solver.cpp:60] Solver scaffolding done.
I0322 11:15:16.625566 **098 solver.cpp:341] Iteration 0, Testing net (#0)
I0322 11:15:19.976579 **098 solver.cpp:409]     Test net output #0: accuracy = 0.445407
I0322 11:15:19.976654 **098 solver.cpp:409]     Test net output #1: loss = 0.693147 (* 1 = 0.693147 loss)
I0322 11:15:20.317916 **098 solver.cpp:237] Iteration 0, loss = 0.693147
I0322 11:15:20.317989 **098 solver.cpp:253]     Train net output #0: loss = 0.693147 (* 1 = 0.693147 loss)
I0322 11:15:20.318009 **098 sgd_solver.cpp:106] Iteration 0, lr = 0.001
I0322 11:15:21.342550 **098 blocking_queue.cpp:50] Data layer prefetch queue empty

I can reproduce this problem even when I delete the locks.mdb to make sure that no locks are left when I restart this procedure. After the message I only can do a hard Matlab shutdown.
I've checked the lmdb with Matlab LMDB and the contents of both, my train and test lmdb seem to be ok. Parameters I've used to generate the lmdb: shuffle.

Note (might be the source of problems here): currently I'm facing MEX problems with this constellation. On the first run of my implementation I get the error message
"Unexpected unknown exception from MEX file.."

for which the terminal output looks like this:

I0322 11:42:09.465801 **875 layer_factory.hpp:77] Creating layer data
I0322 11:42:09.466012 **875 net.cpp:106] Creating Layer data
I0322 11:42:09.466030 **875 net.cpp:411] data -> data
I0322 11:42:09.466053 **875 net.cpp:411] data -> label
I0322 11:42:09.469091 **151 db_lmdb.cpp:38] Opened lmdb /home/user/caffe-master/data/train/lmdbTrain

What I've tried so far:
I implemented a try-catch block so that the pointers, spaces etc. are freed (hopefully) by using 'caffe.reset_all();" so that in ANY case this method is called.
On the second run, I get the above mentioned output. It seems that my first run blocks the lmdb access, what lead me to delete locks.mdb manually between the first and the second run -> same effect unfortunately. A "manual" train by command line does work with the same lmdb's. Only the matcaffe run seems to raise these problems and questions. Note that I want to use Matcaffe for manual initialization of my weights for the layers - "weight_filler" in .prototxt isn't an option.
My MatCaffe implementation is from January 2016 and I've also recompiled the mex-file for caffe_ with the correct gcc version (before it gave me the warning that my gcc version should be "x" -> changed to "x" and recompiled).

Do you have any other ideas, recommendations or inputs please?

Thank you!

[Fábio Ferreira]

X means convert_imageset.bin

[Shu Wang]

I have exactly the same problem using the newest version of Caffe around May, 2016.

I then tried the Caffe version of July, 2015, and it works.

So I suppose some updates with MatCaffe go wrong...

[SAV]

I am trying to use the same command in matlab, could not see the output as the same u provide, test, loss, accuracy  etc.

Can you please  provide solution.