Simple program leaking memory - why?

[gre...@gmail.com]

Hi folks,

I'm fighting an issue in which a simple Mathematica program to export a bunch of plots is leaking a huge amount of memory - enough to exhaust the 12G available on my machine. The problem is, it looks like the program should run in bounded memory! A simplified version of my code which demonstrates the problem follows:

====================
n = 10000;
(* just some bogus data *)
makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
data = Array[makeData, n];
ParallelMap[
Export[
"movie-" <> IntegerString[#, 10, 4] <> ".png",
ListVectorPlot[data[[#]]]] &,
Range[1, n]];
====================

The problem occurs whether the loop is Map or Do, Parallel- or not. It occurs in Mathematica 8 and 9, both under Linux.

The amount of memory leaked per frame is considerable - approximately .6 MB in my tests. I don't see where it could be going, unless Export is caching every byte which passes through it!

I've also asked this question on stackexchange [1] but so far nobody has a clue.

Thanks,
-Gabe

[1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory

[Joe Gwinn]

Mathematica keeps all In and Out cells, and this can look like a leak.
Set $HistoryLength to zero and see if that helps.

There was a thread on this in September 2012 titled "Kernel exits (for
lack of memory)" or just "Kernel exits".

Joe Gwinn

[Gabriel Redner]

Hi Joe,

I don't see how this could be the issue. All of the statements have semicolons after them, and in any case I've tried changing $HistoryLength with no effect.

Thanks,
-Gabe

[Joe Gwinn]

Gabe,

In article <kmfks5$bqt$1...@smc.vnet.net>, Gabriel Redner

The semicolons prevent printing, not generation.

If you did $HistoryLength=0 early in the computations, with a clean
start (exit Mathematica and launch Mathematica) and the problem persists, then it is
probably some other cause.

The most general way to figure out what's going on is to divide the
notebook into two notebooks (communicating via files if necessary) and
see which half leaks. Do this recursively. If neither half leaks, the
cause is something in or near the cut point. This approach is modeled
on binary search, and is very fast albeit manual.

Joe Gwinn

[Roland Franzius]

Am 07.05.2013 09:53, schrieb gre...@gmail.com:
> Hi folks,
>
> I'm fighting an issue in which a simple Mathematica program to export a bunch of plots is leaking a huge amount of memory - enough to exhaust the 12G available on my machine. The problem is, it looks like the program should run in bounded memory! A simplified version of my code which demonstrates the problem follows:
>

FileNames["*.png"]
DeleteFiles["*.png"]

<<VectorFieldPlots`

> ====================
> n = 10000;
> (* just some bogus data *)
> makeData[_] := Table[{{x, y} = RandomReal[{-3, 3}, {2}], {y, -x}}, {200}];
> data = Array[makeData, n];
> ParallelMap[
> Export[
> "movie-" <> IntegerString[#, 10, 4] <> ".png",

strg="movie-" <> IntegerString[#, 10, 4] <> ".png";Print[strg];strg,

> ListVectorPlot[data[[#]]]] &,

ListVectorFieldPlot

> Range[1, n]];
> ====================
>
> The problem occurs whether the loop is Map or Do, Parallel- or not. It occurs in Mathematica 8 and 9, both under Linux.
>
> The amount of memory leaked per frame is considerable - approximately .6 MB in my tests. I don't see where it could be going, unless Export is caching every byte which passes through it!
>
> I've also asked this question on stackexchange [1] but so far nobody has a clue.
>
> Thanks,
> -Gabe
>
> [1] http://mathematica.stackexchange.com/questions/24461/why-does-this-simple-program-leak-memory
>

I can't comment on Linux.

In Windows 7 the Windows Explorer interrupts, after generation of about
1000 files, are stalling suddenly. The Task Manager is still working,
but only on free parts of the screen. Overlapping windows will be cut in
two.

You even cannot open the %user%\Document directory in Windows Explorer,
so I suspect the file name writing interface to the NTFS system has
generated an infinite loop on the medium or its map in memory. Possibly
there may exist an 1024-overflow error in the file stack interface
somewhere.

In Windows 7, one hase to delete the whole bunch of .png files using
DeleteFiles[] in Mathematica or a del in a cmd window.

Since the problem seems to occur on two completely different systems
there will be an error in the Link library of MathLink, Jlink or
whatsoever.

--

Roland Franzius