What is the purpose of shuffle?

[Guofeng Hu]

Hey guys, I have a problem. When we create the data of lmdb for the datalayer, we often shuffled our images, why we do that? And my images were the frames from the video, I have a lstm layer in my model,. Do I need to shuffle my images? Thank you

[Steven Liu]

I know little about LSTM.

In general, shuffling images is to increase  randomness and variety in train dataset, and make train result more stable
If your train images is time-dependent( or is time-ordered, you should not shuffle them.

I think, you could split frames into many parts. In each part, images is time-ordered.
then you can shuffle between parts.

Anyway, shuffle is necessary.

Steven

[Jonathan R. Williford]

An example should contain a video, not just a frame. It might be best to write a custom python layer to read in videos. See: http://caffe.help/manual/layers/python.html for general information on writing custom python layers. You could also try HDF5Data layer (http://caffe.help/manual/layers/hdf5data.html), but it might be very inefficient.

You should shuffle your videos / examples.

Jonathan

[Przemek D]

To answer the general question: we shuffle the data, because caffe reads from an LMDB in a sequential way - in the same linear order each epoch. If you don't shuffle, your LMDB will likely contain large groups of very similar images; during training, first batches will all contain class 0, then the next batches class 1 and so on. The idea of SGD is that we mix examples within a batch so that each of them was an estimate of the whole dataset. A practical consequence to not shuffling might be getting your learning process stuck (it's not unlikely that your network will "learn" to recognize all images as the largest class in your dataset).
Note that the above applies to feedforward networks as I haven't worked with LSTM units yet. Listen to Jonathan here - you want to feed video data to your network, not single frames. But you probably want to shuffle anyway(?), however this time you should do that on the sequence level.

[Guofeng Hu]

And how to shuffle the data between parts? When we create the lmdb, we shuffle them between parts? 

在 2016年12月30日星期五 UTC+8下午4:31:56，Steven Liu写道：

[Guofeng Hu]

I have a idea. How about to shuffle the frames in the datalayer? And the lmdb data don't shuffle. Is it work well?

在 2017年1月2日星期一 UTC+8上午11:16:44，Jonathan R. Williford写道：

[Guofeng Hu]

Thank you for your reply, I will try that.

在 2017年1月2日星期一 UTC+8下午7:27:56，Przemek D写道：

[Guofeng Hu]

Thank you for your reply, I will try that.

在 2017年1月2日星期一 UTC+8下午7:27:56，Przemek D写道：

To answer the general question: we shuffle the data, because caffe reads from an LMDB in a sequential way - in the same linear order each epoch. If you don't shuffle, your LMDB will likely contain large groups of very similar images; during training, first batches will all contain class 0, then the next batches class 1 and so on. The idea of SGD is that we mix examples within a batch so that each of them was an estimate of the whole dataset. A practical consequence to not shuffling might be getting your learning process stuck (it's not unlikely that your network will "learn" to recognize all images as the largest class in your dataset).