Caffe "Memory required for data" misleading?

[CaffeStudent]

Hello!

When loading and testing images with my model, for the largest sized image, it will tell me it requires something like 900MB of RAM

When loading the model, Caffe tells me:

"Memory required for data: 895452040"

I've noticed my GPU has about 2GB of RAM, the Nvidia tools will tell me only about 100-200MB is currently being used.

When I go to do a net.forward() in python, it generates the error message "Check failed: error == cudaSuccess (2 vs. 0)  out of memory" when it seems like I have 1.8GB of free gpu RAM.

If I drop the image size to something that generates an estimate of 700-800MB then net.forward() produces the output with no errors.  It seems almost as if the python caffe interface requires twice as much gpu RAM as is estimated.

Has anyone else experienced an issue like this?

[Jan C Peters]

Just a guess: Every blob stores "data" as well as an equally sized "diff" chunk. So maybe only the "data" parted is counted for this estimate and indeed twice the estimated size is needed? I have not looked into the code, just a wild guess.

Jan

[Evan Shelhamer]

See https://github.com/BVLC/caffe/issues/2387 for an explanation of what this is reporting, which is an incomplete measure of memory usage.

Evan Shelhamer

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/dfbafefd-48a8-4b44-8a05-348b086ea5b9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jan C Peters]

Thanks for clearing that up, Evan. For reference, Evan's comment https://github.com/BVLC/caffe/issues/2387#issuecomment-97910200 sums it up pretty concisely.

Jan