Shuffle training data every epoch

Henggang Cui

unread,

Jun 16, 2016, 11:30:53 AM6/16/16

to Caffe Users

Hello,

I'm wondering is that possible to shuffle the training data every epoch? I guess shuffling is probably necessary, because most of the deep learning papers say that they shuffle the training data.

Currently I'm using the LMDB format, but I'm willing to switch to other formats if it makes shuffling easier.

Thank you!

Cui

mprl

unread,

Jun 16, 2016, 11:46:58 AM6/16/16

to Caffe Users

The HDF5Data layer has a shuffle: true hdf5_data_param. I think it shuffle the h5 files and the images into the h5 files every epoch.

Henggang Cui

unread,

Jun 16, 2016, 1:03:56 PM6/16/16

to Caffe Users

That's very good! Do you know how to convert the ImageNet data into the HDF5 format? Is there a program or script doing that for us?

Thank you!

Cui

mprl

unread,

Jun 17, 2016, 8:27:21 AM6/17/16

to Caffe Users

I don't know ImageNet data, but i think you can edit a Python or Matlab script to do so. Check these links :
http://stackoverflow.com/questions/31774953/test-labels-for-regression-caffe-float-not-allowed/31808324#31808324
https://github.com/nikogamulin/caffe-utils/blob/master/hdf5/generate_hdf5_dataset.m

Le jeudi 16 juin 2016 17:30:53 UTC+2, Henggang Cui a écrit :

Reply all

Reply to author

Forward