Clearing RAM Memory during evaluation
Aaron Bramson
I am parsing a large data file and naturally this is memory intensive so I
am breaking it into the smallest chunks possible. But I'm still having a
problem with memory because I can't seem to clear up my RAM between chunks.
So far there is NO output except writing a csv file, and I've cleared the
main data-holding variable I use, and all of it is done inside a
module...but still when the module finishes the memory isn't cleared.
Quitting the kernel does clear the memory, but I can't do that in between
calls of this function. Can anybody tell me what I need to change to get
Mathematica to clear the memory at the end of the function?
Here's the function:
GetData[popdynamics_, simultaneous_, learning_] :=
Module[{ThisData, DataLine, DataFile},
ThisData = {};
DataLine = {};
DataFile =
OpenRead[
"Experiment-table.csv"];
Do[DataLine = ReadList[DataFile, Record, 1], {7}];
DataLine =
Map[StringTake[#, {2, StringLength[#] - 1}] &,
ReadList[DataFile, Record, 36,
RecordSeparators -> {",", "\n"}]];
While[DataLine != {},
If[(DataLine[[6]] == popdynamics) && (DataLine[[8]] ==
simultaneous) && (DataLine[[10]] == learning) ,
AppendTo[ThisData, DataLine]];
DataLine =
Map[StringTake[#, {2, StringLength[#] - 1}] &,
ReadList[DataFile, Record, 36, RecordSeparators -> {",", "\n"}]];
];
If[ThisData != {},
Export["Test Data 2e.csv", ThisData]];
Close[DataFile];
ClearAll[ThisData];
ClearSystemCache[];
]
Here's the function call: GetData["true", "false", "true"];
Thanks in advance for your assistance,
Aaron Bramson
Vince Virgilio
> Hello Everybody,
>
> I am parsing a large data file and naturally this is memory intensive so I
> am breaking it into the smallest chunks possible. But I'm still having a
> problem with memory because I can't seem to clear up my RAM between chunks.
> So far there is NO output except writing a csv file, and I've cleared the
> main data-holding variable I use, and all of it is done inside a
> module...but still when the module finishes the memory isn't cleared.
> Quitting the kernel does clear the memory, but I can't do that in between
> calls of this function. Can anybody tell me what I need to change to get
> Mathematica to clear the memory at the end of the function?
Aaron,
Mathematica is a bottomless pit when it comes to memory consumption.
Sooner or later, you will hit this wall. There is no way around it,
except to save intermediate results to file, restart the kernel(s),
then continue the computation.
Vince
David Bailey
smallest possible example of his problem as a working example (complete
with data). Also remember that Mathematica will not return memory when
it does not need it (I imagine), it will just reuse that memory as and
when it is required.
Vince, have you actually run out of memory doing this calculation?
David Bailey
peter
Clear["Global`*"]
Sébastien Roy
One thing that helps a lot in practice is:
$HistoryLength = 0
The fact that it is set to infinity by default probably contributes to
the "bottomless pit"...
Aaron Bramson
@Vince: I fear that you may be correct, and that doing this in even smaller
steps is more trouble than it's worth. I have moved towards a database
approach to the data sorting and gathering and just using Mathematica to run
calculations and plot. However I still need to overcome this problem of
memory not clearing, probably, just to get through the calculations and
plotting that I have to do.
@ Peter: That did not work for me...it doesn't seem to do anything different
than clearing just the variables individually...and remember that they are
all defined INSIDE A MODULE anyway. Hmmmm.
@ Sebastien: I already have the history length set to 0 as you recommend
since that is (as you point out) a common issue, but since the only output
is to an exported file I don't expect that's the issue here.
@David: I don't know if you meant to direct your question at me, but YES, I
have run out of memory doing this calculation...well it's not really a
calculation just a partitioning of data into cases. I read the claims that
Mathematica will reuse memory, and that variables inside modules are
supposed to be local variables (and hence when the module is completed the
memory used in it would be cleared even if I didn't explicitly clear
it...which I do), but all these claims seem totally false in practice. When
run on a small test dataset I can complete the process, but each time it
goes through the module you can see the memory usage steadily climb (I have
6GB of RAM on this computer and that's a little more than enough). When I
tried to run it on an abbreviated version of the full dataset (one that only
contained two of the eight cases, but all of the data for those two cases)
it went like this: at start RAM usage was 23%, after completing the first
run through the module it was at 38%, during the second run of the module
the RAM usage steadily climbed up to 87% over the course of 14 hours. After
that time I just aborted the process since clearly 1) without the RAM
reseting there is no way I'd have enough memory to use this algorithm for 8
cases, and 2) even if I solved the memory issue I don't have a year to wait
for all my results. When I imported a data file (which are csv files ranging
from 250MB to 2.5GB) instead of using the readopen command I very quickly
used up all 6GB of ram before anything useful happened. Mathematica does
seem to be the wrong tool for this sort of job.
But as I said, I still need to figure out how to clear the memory in a case
LIKE THIS where INSIDE A MODULE I'm reading in a file (I'll be getting it
from a database call within Mathematica) running a calculation and making a
plot based on the data, and then exporting the plot as a pdf or jpg.
According to all the documentation I could find, I'm already doing WAY MORE
than I should need to do and none of it is working.
Any more advice or special tricks?
Thanks,
Aaron
----------
p.s. I can provide the test data if you think that would actually help you
find the memory leak.
Here's the function:
On Thu, Aug 12, 2010 at 5:28 AM, David Bailey <da...@removedbailey.co.uk>wrote:
> On 11/08/10 09:44, Vince Virgilio wrote:
> > On Aug 10, 4:01 am, Aaron Bramson<aaronbram...@gmail.com> wrote:
> >> Hello Everybody,
> >>
> >> I am parsing a large data file and naturally this is memory intensive so
> I
> >> am breaking it into the smallest chunks possible. But I'm still having
> a
> >> problem with memory because I can't seem to clear up my RAM between
> chunks.
> >> So far there is NO output except writing a csv file, and I've cleared
> the
> >> main data-holding variable I use, and all of it is done inside a
> >> module...but still when the module finishes the memory isn't cleared.
> >> Quitting the kernel does clear the memory, but I can't do that in
> between
> >> calls of this function. Can anybody tell me what I need to change to
> get
> >> Mathematica to clear the memory at the end of the function?
> >
> > Aaron,
> >
> > Mathematica is a bottomless pit when it comes to memory consumption.
> > Sooner or later, you will hit this wall. There is no way around it,
> > except to save intermediate results to file, restart the kernel(s),
> > then continue the computation.
> >
> > Vince
> >