Memory error with large batch process

[David Kysela]

I'm batch processing large (to me) datasets that include a couple of thousand images, which are themselves fairly substantial (4k, 16 bit, 4 channels, ~34 MB each). During processing, I can watch the memory usage climb up to critical values >90%, but then it periodically drops down to lower values when I assume there's some garbage collection going on to free up memory. However, there are several points during processing where memory and CPU usage become very high and ImageJ hangs for a bit; sometimes this eventually clears up and the batch process continues with the subsequent images, but eventually the batch process crashes and reports the error below.

Any ideas what might be happening here?

Thanks!

Exception in thread "Thread-60" java.lang.OutOfMemoryError: Java heap space
 at loci.formats.ImageTools.splitChannels(ImageTools.java:218)
 at loci.formats.ImageTools.splitChannels(ImageTools.java:198)
 at loci.formats.in.NativeND2Reader.openBytes(NativeND2Reader.java:314)
 at loci.formats.DelegateReader.openBytes(DelegateReader.java:227)
 at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156)
 at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:229)
 at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:348)
 at loci.formats.DimensionSwapper.openBytes(DimensionSwapper.java:249)
 at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:348)
 at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:269)
 at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:260)
 at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
 at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:334)
 at loci.plugins.util.ImageProcessorReader.openProcessors(ImageProcessorReader.java:186)
 at loci.plugins.in.ImagePlusReader.readProcessors(ImagePlusReader.java:422)
 at loci.plugins.in.ImagePlusReader.readPlanes(ImagePlusReader.java:387)
 at loci.plugins.in.ImagePlusReader.readImage(ImagePlusReader.java:282)
 at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:243)
 at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:221)
 at loci.plugins.in.ImagePlusReader.openImagePlus(ImagePlusReader.java:116)
 at loci.plugins.BF.openImagePlus(BF.java:98)
 at com.ducret.resultJ.ImPlus.loadImagePlus(ImPlus.java:2083)
 at com.ducret.resultJ.ImPlus.loadImagePlus(ImPlus.java:2056)
 at com.ducret.resultJ.ImPlus.getImages(ImPlus.java:2602)
 at com.ducret.resultJ.ImPlus.getImages(ImPlus.java:2582)
 at com.ducret.resultJ.ImPlus.loadImagePlus(ImPlus.java:2126)
 at com.ducret.resultJ.ImPlus.loadImagePlus(ImPlus.java:2121)
 at com.ducret.resultJ.Im.getImagePlus(Im.java:442)
 at com.ducret.resultJ.Im.getImagePlus(Im.java:390)
 at com.ducret.resultJ.Im.load(Im.java:269)
 at com.ducret.resultJ.ImPlus.load(ImPlus.java:440)
 at com.ducret.microbeJ.Experiment.run(Experiment.java:288)

[Andreas Schick]

Hi,

have you tried to increase the memory in imageJ?

• Edit  › Options  › Memory & Threads
• Change "Maximum Memory" to something larger (at most, 1000 MB less than your computer's total RAM).
• Restart ImageJ for the new memory settings to take effect.
  (taken from here: https://imagej.net/Troubleshooting)

Though I am truly no microbeJ expert (yet) and I don't know whether this will resolve your issue.

All the best!

[Joe Villanueva]

Changing my maximum memory space in ImageJ solved my problem! 

Thank you Andreas! 

Joe Villanueva