Re: Memory usage in each loop

[chriss...@gmail.com]

Sorry that code block should be:

  connection.directories.get(fog_directory).files.each do |aws_file|

    aws_file.body
  end

On Monday, April 29, 2013 12:04:32 PM UTC+9, chriss...@gmail.com wrote:

Hi,

I've read some other answers in this group but none seem to apply to my problem.

I have a very simple code block where I iterate over files (photos) in a bucket, resizing and saving each to local storage. The problem is that memory usage grows as ruby iterates over the block, to the point where it crashes with a "Cannot allocate memory" error for large buckets.

I've confirmed that the problem is related to fog only, and not to any of the other processing. Reducing the block to just this:

   directory = connection.directories.get(fog_directory).files.each do |aws_file|
     aws_file.body
   end

still has the same result. It seems that once I access the file body, it puts that in memory and does not free it until the loop completes (i.e. until it has iterated over all files in the directory), which never happens because it hits a memory limit.

Is there something I am doing wrong here? Any help would be much appreciated. Thanks!

Chris

[geemus (Wesley Beary)]

Yeah, I could certainly see how that would be a problem. For convenience it sets the body into an attribute on the file object, and since all the file objects would still be in scope throughout (and after) the loop, it all comes crashing down.

I think you should be able to get what you need if you change the inside of the loop to this:

directory.files.get(aws_file.key).body

That is more or less what the body method would normally do anyway, except you are then accessing it in such a way that it should fall out of scope at the end of the loop, which should allow it to be garbage collected.

Hope that helps, but do let me know if you have further questions.

Thanks!

wes

--
You received this message because you are subscribed to the Google Groups "ruby-fog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-fog+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[chriss...@gmail.com]

Thanks very much! Will try that.