Blocking_queue.cpp - waiting for data

49 views

Skip to first unread message

unread,

Mar 1, 2017, 11:56:35 AM3/1/17

to Caffe Users

Hi all,

I do actually observe following log:

I0301 19:18:52.763780 25761 solver.cpp:219] Iteration 252000 (1.84212 iter/s, 542.854s/1000 iters), loss = 0.667764

I0301 19:18:52.798413 25761 solver.cpp:238] Train net output #0: loss = 0.667764 (* 1 = 0.667764 loss)

I0301 19:18:52.798447 25761 sgd_solver.cpp:105] Iteration 252000, lr = 8.1e-06

I0301 19:19:07.079993 25761 blocking_queue.cpp:49] Waiting for data

I0301 19:28:15.665973 25761 solver.cpp:219] Iteration 253000 (1.7766 iter/s, 562.871s/1000 iters), loss = 0.54343

I0301 19:28:15.666098 25761 solver.cpp:238] Train net output #0: loss = 0.543431 (* 1 = 0.543431 loss)

I0301 19:28:15.666110 25761 sgd_solver.cpp:105] Iteration 253000, lr = 8.1e-06

I0301 19:28:30.009287 25761 blocking_queue.cpp:49] Waiting for data

batch_size is only 24 images, images are on SSD, already resized and cropped.

iostat says following:

$ iostat -dx /dev/sda

Linux 3.19.0-25-generic (Utka) 01.03.2017 _x86_64_ (8 CPU)

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

sda 0,09 88,86 232,98 69,75 26414,37 811,29 179,87 0,47 1,54 0,45 5,18 0,20 6,12

what could be the reason?

Thanks

unread,

Mar 1, 2017, 12:09:18 PM3/1/17

to Caffe Users

Image size ~17kB in average, so we need only 17*24*2~800kB/s to support Caffe with data.