Support for blobs > 2GB

[Tiferet Gazit]

Hello,

I posted issue #3510 on Github a while ago and haven't received a response.

I have some 2,650 features, and when I try to use a training set greater than about 750,000 examples I run into the following error:

Check failed: shape[i] <= 2147483647 / count_ (21 vs. 12) blob size exceeds INT_MAX

The full console printout of the error I am getting plus all my net and solver parameters is attached.

Is there any solution (or workaround) that would allow me to train with a larger training set? Since I use batch training, it seems I should be able to use an unlimited total number of examples, perhaps even by reading them from multiple smaller files.

This problem seems to be referenced in the following issues, but as far as I can tell none of them actually involve a merge to solve the bug (please correct me if I'm wrong):
#2473 (see especially jeffdonahue's comment)
#3159
#1970

Thank you!

[Evan Shelhamer]

Since I use batch training, it seems I should be able to use an unlimited total number of examples, perhaps even by reading them from multiple smaller files.

You're right: computational limits only restrict the batch size. The total number of instances can be unlimited since only a single batch is computed at a time.

However HDF5 data is a special case in Caffe and the whole input HDF5 file is loaded. To control memory usage you can split your HDF5 input into several files and then list these files in your data layer input text file. Each of these files will be loaded and unloaded in turn so the memory is determined by the largest single HDF5 file.

See PR #2892 for details and history.

Happy brewing,

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/e7999d8a-3165-4237-ac26-cbfd84b006ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tiferet Gazit]

Thank you - this is great news! Being able to use more training examples should greatly improve my results!