Determine memory usage of TensorFlow?

[Mohammed AlQuraishi]

Is there some way to determine how much memory is being used during a tensorflow run? I don't have access to a GPU and would like to determine how much memory my model is using before investing in one, since GPU memory can often be the limiting factor.

[Nick Shah]

Yes! Specifically a layout that shows what op is taking how much memory would be incredibly useful!

[Mike Schuster]

On Tuesday, December 1, 2015 at 6:33:46 AM UTC-8, Mohammed AlQuraishi wrote:

Is there some way to determine how much memory is being used during a tensorflow run? I don't have access to a GPU and would like to determine how much memory my model is using before investing in one, since GPU memory can often be the limiting factor.

Yes, GPU memory is often a limiting factor. We currently don't have a nice way of exactly seeing where and how much memory is used (Nick Shah's point taken...) but for the GPU, basically assume that it will reserve all the available memory (often 12GB) and make the best use of it. If your run takes too much GPU memory, try to lower batch size etc. until it fits. Also of course, you can always chose to put some ops not on the GPU if they happen to take a very much memory.

Mike

 

[Mohammed AlQuraishi]

Can we at least see how much total memory is used by the graph / session? I understand that we can always use smaller batch sizes, but if we're trying to decide on a GPU platform, etc, it would help to know this beforehand.

[Nicholas Shah]

I agree with you Mike, that it would be ideal to take memory-intensive processes and place them on the cpu, but the problem is....is that its hard to diagnose what processes are memory intensive. I appreciate your help though!

--
You received this message because you are subscribed to a topic in the Google Groups "Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/a/tensorflow.org/d/topic/discuss/AWbonlBDzJ8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss+u...@tensorflow.org.
To post to this group, send email to dis...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/discuss/57a3a1db-e3a6-4f71-892c-47c6983166ee%40tensorflow.org.

[Vijay Vasudevan]

On Wed, Dec 2, 2015 at 1:32 AM, Mohammed AlQuraishi <nom...@gmail.com> wrote:

Can we at least see how much total memory is used by the graph / session? I understand that we can always use smaller batch sizes, but if we're trying to decide on a GPU platform, etc, it would help to know this beforehand.

Yes, this type of instrumentation is being worked on, will eventually be made available, though I can't give a timeline.

On Wednesday, December 2, 2015 at 2:08:54 AM UTC-5, Mike Schuster wrote:

On Tuesday, December 1, 2015 at 6:33:46 AM UTC-8, Mohammed AlQuraishi wrote:

Is there some way to determine how much memory is being used during a tensorflow run? I don't have access to a GPU and would like to determine how much memory my model is using before investing in one, since GPU memory can often be the limiting factor.

Yes, GPU memory is often a limiting factor. We currently don't have a nice way of exactly seeing where and how much memory is used (Nick Shah's point taken...) but for the GPU, basically assume that it will reserve all the available memory (often 12GB) and make the best use of it. If your run takes too much GPU memory, try to lower batch size etc. until it fits. Also of course, you can always chose to put some ops not on the GPU if they happen to take a very much memory.

Mike

 

--
You received this message because you are subscribed to the Google Groups "Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@tensorflow.org.

To post to this group, send email to dis...@tensorflow.org.