Memory Blowup Issues

[Kevin J. McCann]

I am dealing with some good sized data sets from satellite data. These
data are stored in hdf format. The problem is that I process a sequence
of these data sets by Importing and then processing; however, it seems
that the memory usage grows with each Import, even if I Remove the data.

Here is a somewhat simplified example:

MemoryInUse[] (* some data already read in *)
2781549920

Data=Import[fileName,"Data"]; (* import a new data set *)
MemoryInUse[]
3302168936

Data=Import[fileName,"Data"];(* import it again *)
MemoryInUse[]
3822781920

Remove[Data] (* this has no effect on the memory usage *)
MemoryInUse[]
3822787840

As you can see, I will shortly run out of memory. Any ideas?

Thanks,

Kevin

[Bill Rowe]

On 6/1/12 at 5:18 AM, k...@KevinMcCann.com (Kevin J. McCann) wrote:

>I am dealing with some good sized data sets from satellite data.
>These data are stored in hdf format. The problem is that I process a
>sequence of these data sets by Importing and then processing;
>however, it seems that the memory usage grows with each Import, even
>if I Remove the data.

Try doing $HistoryLength=0; at the start of a session when
working with large data sets. This will keep Mathematica from
keeping copies on the In/Out stack which can quickly consume memory

Another approach might be to load the package CleanSlate before
importing large data sets. CleanSlate can restore things to the
state they were in immediately after the package was loaded
including clearing the In/Out stack.

[David Reiss]

One thing to try is to set

$HistoryLength = 0.

This way Mathematica does not keep a record of the In[n] and Out[n]
values.

[Szabolcs]

Hello Kevin,

The most likely cause of this behaviour is that the intermediate results are being remembered by Mathematica through the Out symbol. You probably know that you can refer to previous results using %, %%, %%% or Out[10], Out[11], etc. but you may not have realized that Out will get assigned a new value even if the last input ended in a semicolon.

Example:

In[1]:= data=RandomReal[1,10000]; (* output suppressed using semicolon *)

In[2]:= Length[%] (* note that % will have the same value as data *)

The solution is to set $HistoryLength = 0 before you start working, to prevent the system from remembering previous results.

Please see these articles as well:

http://mathematica.stackexchange.com/questions/3376/old-values-are-not-freed-garbage-collected-when-you-re-evaluate-an-assignment

http://mathematica.stackexchange.com/questions/4322/memory-not-freed-after-running-clear-when-using-a-table

Szabolcs

[Ralph Dratman]

Solution: install more memory in your computer?

That answer is sort of a wisecrack, but in real life, my Mac Mini now
has 8 GB of RAM for the benefit of Mathematica -- yet the program
still somehow manages to fill up memory and crash. That is why I keep
glancing at the desktop memory monitor. If memory is almost full, I
can usually stop Mathematica before things get really serious.

That said, my experience has been that disposing of variables (using
myvar=.) does help quite a bit. But I am not reading files very often.
In your case, maybe something about reading a file is making it harder
to free the memory used.

By the way, in case you might be thinking that this whole crash
problem is pretty unacceptable behavior for an app that sells for a
lot of money in 2012.... yeah, you're right, it is.

Ralph

On Fri, Jun 1, 2012 at 5:18 AM, Kevin J. McCann <k...@kevinmccann.com> wrote:
> I am dealing with some good sized data sets from satellite data. These
> data are stored in hdf format. The problem is that I process a sequence
> of these data sets by Importing and then processing; however, it seems
> that the memory usage grows with each Import, even if I Remove the data.
>
> Here is a somewhat simplified example:
>
> MemoryInUse[] (* some data already read in *)
> 2781549920
>
> Data=Import[fileName,"Data"]; (* import a new data set *)
> MemoryInUse[]
> 3302168936
>
> Data=Import[fileName,"Data"];(* import it again *)
> MemoryInUse[]
> 3822781920
>
> Remove[Data] (* this has no effect on the memory usage *)
> MemoryInUse[]
> 3822787840
>
> As you can see, I will shortly run out of memory. Any ideas?
>

> Thanks,
>
> Kevin
>

[Sseziwa Mukasa]

Clear out your history?

[Kevin J. McCann]

I already have 12Gb; so, that is not a solution. Others have suggested
$HistoryLength = 0, which works. On the other hand, mvary = . did not
change the amount of memory in use at all.

I guess acceptability of the memory/crash problem depends on your point
of view. If indeed the ability to access previous outputs is important
to the way you work (it is not to me), then I can see why it needs to be
there even though it could lead to a crash in extreme cases like mine.
However, the ability to limit this with $HistoryLength gives flexibility
to the user.

I do not make use of the whole In[] Out[] thing, and, in fact, turn
those labels off in all my notebooks.

Kevin

[Ralph Dratman]

Kevin,

Thank you for passing along those tips. I usually put $HistoryLength=1
and also turn off "Enable Notebook History Tracking" in the
Preferences dialog -- though I'm not sure the latter makes any
difference.

I just want to re-emphasize how egregious I think the situation with
crashing really is. It is unacceptable. I hope everyone is making that
absolutely clear to WRI. How can we continue to recommend a program
which often crashes? It defies common sense.

Ralph

[Szabolcs Horvát]

On 2012.06.03. 11:03, Ralph Dratman wrote:
> Kevin,
>
> Thank you for passing along those tips. I usually put $HistoryLength=1
> and also turn off "Enable Notebook History Tracking" in the
> Preferences dialog -- though I'm not sure the latter makes any
> difference.
>

Hello Ralph,

"Notebook History Tracking" only affects what is saved with the notebook
you are working with, but it does not affect how the kernel works or
what it keeps in memory (i.e. doesn't appreciably affect memory use).
This option will simply record the times at which who have created or
edited each cell. If you open a notebook file with a text editor,
you'll see this information in the CellChangeTimes options.

Turning it off will slightly reduce notebook sizes and makes version
tracking much more convenient.

--
Szabolcs Horv�t
Visit Mathematica.SE: http://mathematica.stackexchange.com/

[David Bailey]

On 03/06/2012 10:03, Ralph Dratman wrote:
> Kevin,
>
> Thank you for passing along those tips. I usually put $HistoryLength=1
> and also turn off "Enable Notebook History Tracking" in the
> Preferences dialog -- though I'm not sure the latter makes any
> difference.
>
> I just want to re-emphasize how egregious I think the situation with
> crashing really is. It is unacceptable. I hope everyone is making that
> absolutely clear to WRI. How can we continue to recommend a program
> which often crashes? It defies common sense.
>

To be fair to WRI, I think the problem is that you have a very complex
piece of software that is partly programmed by users. The kernel does
put out a message if it runs out of memory, so it does detect the
problem (at least under Windows!), but leaving the software in a well
defined and useful state is another matter. The FrontEnd should remain
running.

It is also possible to run calculations using MemoryConstrained.

Assuming the data you are reading in is an array or real numbers (or of
integers), you should check if Import returns a packed array, and pack
it if necessary - this will save a lot of space.

David Bailey
http://www.dbaileyconsultancy.co.uk

[W Craig Carter]

Hello,

I find the following function to be a useful way to extract information
for memory management:

checkMemory[]:=Grid@Sort[{ByteCount[ToExpression[#]], HoldForm[#]} &
/@ Names["Global`*"], (First[#1] > First[#2]) &]

This returns a list of symbols ranked by their ByteCount. However, I am
not sure if evaluating checkMemory causes a momentary doubling of memory
for each evaluation of ByteCount=85.

I thought I would pass this along for anyone who might be hunting for
such a thing=85


W Craig Carter
Professor of Materials Science, MIT

On Jun 6, , at Wed Jun 6, 12 @4:52 AM, David Bailey wrote:

> On 03/06/2012 10:03, Ralph Dratman wrote:

>> Kevin,
>>
>> Thank you for passing along those tips. I usually put $HistoryLength=1
>> and also turn off "Enable Notebook History Tracking" in the
>> Preferences dialog -- though I'm not sure the latter makes any
>> difference.
>>
>> I just want to re-emphasize how egregious I think the situation with
>> crashing really is. It is unacceptable. I hope everyone is making that
>> absolutely clear to WRI. How can we continue to recommend a program
>> which often crashes? It defies common sense.
>>
>
>

[Ralph Dratman]

David,

Thank you for your comments. I actually am trying to be helpful to
Wolfram, simply because I think Mathematica is an outstanding product.
Taming the experience of using the software, especially for new users,
would be of enormous benefit to the company, since the software might
then reach a far wider audience.

It seems possible to me that Mathematica, in a slightly different form
and possibly under a different name, could become for today's
computers what Basic was for the Apple II and the IBM PC. I think the
little message handler we have been discussing actually brings it
closer to that goal.

messageHandler = If[Last[#], Interrupt[]] &
Internal`AddHandler["Message", messageHandler]

With a different style of documentation and some related changes to
the user interface, Mathematica might shrug off its (currently well
deserved) reputation for being expensive and difficult to use, and
take on an entirely new position in the marketplace.

Ralph

[Bill Rowe]

On 6/7/12 at 5:22 AM, ralph....@gmail.com (Ralph Dratman) wrote:

>Taming the experience of using the software, especially for
>new users, would be of enormous benefit to the company, since the
>software might then reach a far wider audience.

Maybe this would benefit Wolfram. It isn't obvious. If there
were no additional costs associated with reaching a wider
audience (i.e., selling more licenses) there is a clear benefit.
But there are costs such as programming resources to provide the
"taming" you have in mind. An even if this entailed no
additional programming resources, more licenses sold will mean
more people wanting support and increased support costs. I don't
know what tradeoff Wolfram would want in this regard. But I am
sure reaching a wider audience doesn't come without additional cost.

>It seems possible to me that Mathematica, in a slightly different
>form and possibly under a different name, could become for today's
>computers what Basic was for the Apple II and the IBM PC.

This almost certainly won't happen with Mathematica. BASIC was
designed to be a beginner's programming language. Mathematica is
designed to be a very powerful mathematical toolset. And it
seems clear Wolfram intends to increase the power of this
toolset and extend it to additional areas of computation. To
really make effective use of Mathematica you really need a solid
understanding to mathematics and numerical computation, quite a
bit deeper understanding than what is required for something
like BASIC.

True, if all you do with Mathematica is use it as a super
scientific calculator, you don't need any deeper understanding
than what would be required for BASIC. But, I would strongly
argue this isn't making significant use of Mathematica's
capabilities. There are far less expensive solutions for such
calculations that are much easier to master than Mathematica. Of
course, these lack the power Mathematica offers.

[Richard Fateman]

On 6/8/2012 12:37 AM, Bill Rowe wrote:

>
> This almost certainly won't happen with Mathematica. BASIC was
> designed to be a beginner's programming language. Mathematica is
> designed to be a very powerful mathematical toolset.

I think that Wolfram's ambition is/was that Mathematica's underlying
language (without the "Math" stuff) would be recognized as the ideal
blend of programming features and esthetic melding of design for all
purposes, including introductory programming, typesetting, web design,
parallel computation, etc.

It hasn't happened.

And it
> seems clear Wolfram intends to increase the power of this
> toolset and extend it to additional areas of computation. To
> really make effective use of Mathematica you really need a solid
> understanding to mathematics and numerical computation, quite a
> bit deeper understanding than what is required for something
> like BASIC.

Actually, if you look around at the questions here, you see that
many of them have almost nothing to do with advanced math
(e.g. plotting!) or numerical calculation (except to display
ignorance of numerical analysis.)

>
> True, if all you do with Mathematica is use it as a super
> scientific calculator, you don't need any deeper understanding
> than what would be required for BASIC.

And if you use it to draw pictures, or make web pages, ...

But, I would strongly
> argue this isn't making significant use of Mathematica's
> capabilities.

I suspect that if you measured the hours and expense of building
Mathematica's non-math portions -- graphics, front-end, ... and
also included the overall business expenses (marketing, public
relations, ...) you would find that the math part is not
so significant. If Wolfram Research employs 400 people (I assume
this number in the wikipedia article was put there by Wolframites),
how many of them do you think are PhD applied mathematicians?
5 percent?

There are far less expensive solutions for such
> calculations that are much easier to master than Mathematica.

Maybe the secret is for Wolfram Research should offer a version without
the Math, as a $0.99 app download to iphones and droids. Or give it
away, subsidized by the Wolfram|Alpha revenue.

Of
> course, these lack the power Mathematica offers.

With great power comes great responsibility..
http://en.wikiquote.org/wiki/Stan_Lee
>
RJF
>

[David Bailey]

On 10/06/2012 07:16, Richard Fateman wrote:
> On 6/8/2012 12:37 AM, Bill Rowe wrote:
>
>>
>> This almost certainly won't happen with Mathematica. BASIC was
>> designed to be a beginner's programming language. Mathematica is
>> designed to be a very powerful mathematical toolset.
>
> I think that Wolfram's ambition is/was that Mathematica's underlying
> language (without the "Math" stuff) would be recognized as the ideal
> blend of programming features and esthetic melding of design for all
> purposes, including introductory programming, typesetting, web design,
> parallel computation, etc.
>
> It hasn't happened.
>

Algol, Algol68, Pascal, C, C++, Ada Java, Lisp, Prolog - all were touted
as ideal programing languages at one time or another!

After 30+ years in the software industry, I think I can confidently say
that there is no ideal blend of programming features, because
programmers and their projects vary so much.

David Bailey
http://www.dbaileyconsultancy.co.uk

[Szabolcs Horvát]

You might find this interesting:

http://www.reddit.com/r/IAmA/comments/qisot/im_stephen_wolfram_mathematica_nks_wolframalpha/c3xz8by

Quoting the last paragraph:

"It'll probably be related to my goal in the next year or two of making
Mathematica definitively the world's easiest to learn language..."

--
Szabolcs Horvát
Visit Mathematica.SE: http://mathematica.stackexchange.com/

[Ralph Dratman]

Mathematica's language could certainly be easier to learn than it is
now. I don't know whether it will actually become the world's easiest
computer language -- but that is certainly a worthy goal.

There are two problems which form part of the learning difficulty, yet
have nothing to do with the language structure itself:

1) The difficulty of finding out how to accomplish some task. This is
a documentation and search problem. I have less trouble with that
issue now, but it was very hard at the beginning. The documentation
search is frustratingly slow -- surely an unnecessary situation.

2) The difficulty of debugging, as a result of the frequency of
crashes. This simply makes trying things much slower. Fortunately, an
enormously helpful change is given by the following:

messageHandler = (If[Last[#], Interrupt[] ] &); (* Create a message handler. *)
Internal`AddHandler["Message", messageHandler]; (* Install the handler. *)

The above makes Mathematica stop on any error, promptly. Debugging
time is now a fraction of what it was for me only a few weeks ago.

Unfortunately there are still situations in which I have to quit the
kernel or force-quit the application entirely. This becomes necessary
when there has been no message issued during evaluation, and, for
unknown reasons, the Interrupt menu item appears as disabled. I do not
know what makes that happen. There should always be a way out.

Ralph

On Thu, Jun 14, 2012 at 10:11 AM, Szabolcs Horv=E1t <szho...@gmail.com> wrote:
> On 2012.06.08. 9:37, Bill Rowe wrote:
>>

> You might find this interesting:
>
> http://www.reddit.com/r/IAmA/comments/qisot/im_stephen_wolfram_mathematica_nks_wolframalpha/c3xz8by
>
> Quoting the last paragraph:
>
> "It'll probably be related to my goal in the next year or two of making
> Mathematica definitively the world's easiest to learn language..."
>
> --

> Szabolcs Horv=E1t
> Visit Mathematica.SE: http://mathematica.stackexchange.com/

[Bill Rowe]

On 6/15/12 at 3:40 AM, szho...@gmail.com (Szabolcs Horv=C3=A1t) wrote:

>On 2012.06.08. 9:37, Bill Rowe wrote:
>>On 6/7/12 at 5:22 AM, ralph....@gmail.com (Ralph Dratman) wrote:
>>

>>>It seems possible to me that Mathematica, in a slightly different
>>>form and possibly under a different name, could become for today's
>>>computers what Basic was for the Apple II and the IBM PC.

>>This almost certainly won't happen with Mathematica.

>You might find this interesting:

>http://www.reddit.com/r/IAmA/comments/qisot/im_stephen_wolfram_mathematica_nks_wolframalpha/c3xz8by

>Quoting the last paragraph:

>"It'll probably be related to my goal in the next year or two of
>making Mathematica definitively the world's easiest to learn
>language..."

I can accept this is Wolfram's goal and applaud their efforts to
achieve it. However, I still think this is a goal unlikely to be achieved.

It seems clear Wolfram has another goal as well for Mathematica
which is for users to be able to do any computation they can
think of in any field with Mathematica. To that end, the number
of available functions has grown significantly with every
release. Added functions generally increase learning difficulty
if for no other reason than there is more to learn.

Also, if you look at Mathematica as a programming language and
compare it to other programing languages such as BASIC, C++
etc., it seems clear a key distinction is the ability to do
advanced mathematics in Mathematica that cannot be done in other
languages without writing a significant amount of code. If you
don't have a background in mathematics, how could you take
advantage of this distinction? And if you aren't using the
mathematical tools Mathematica offers, why use it over something
like BASIC? Given the relative cost of a Mathematica license and
a BASIC interpreter/compiler, it seems rather silly to invest in
a Mathematica license and not use Mathematica's mathematics toolbox.

Mathematica may well become the easiest to learn language *for
doing mathematics and advanced computation*. But, I don't see
Mathematica becoming the easiest to learn language for say
creating web sites, games and a great many other things that are
created using programing languages.

[Richard Fateman]

On 6/15/2012 12:32 PM, Bill Rowe wrote:

>> Quoting the last paragraph:
>
>> "It'll probably be related to my goal in the next year or two of
>> making Mathematica definitively the world's easiest to learn
>> language..."

>
> I can accept this is Wolfram's goal and applaud their efforts to
> achieve it. However, I still think this is a goal unlikely to be achieved.

Yes.

>
> It seems clear Wolfram has another goal as well for Mathematica
> which is for users to be able to do any computation they can
> think of in any field with Mathematica. To that end, the number
> of available functions has grown significantly with every
> release. Added functions generally increase learning difficulty
> if for no other reason than there is more to learn.

Not if the added functionality is all orthogonal to existing functionality.

>
> Also, if you look at Mathematica as a programming language and
> compare it to other programing languages such as BASIC, C++
> etc., it seems clear a key distinction is the ability to do
> advanced mathematics in Mathematica that cannot be done in other
> languages without writing a significant amount of code.

You are comparing it to the wrong languages if you want to compare
languages by their ability to do (symbolic) computation.

If you
> don't have a background in mathematics, how could you take
> advantage of this distinction?

You would probably not take advantage of these features. If they
are orthogonal to what you want to write a program about, who cares?

> And if you aren't using the
> mathematical tools Mathematica offers, why use it over something
> like BASIC?

Because Stephen Wolfram "invented" Mathematica and he did not invent

BASIC?

> Given the relative cost of a Mathematica license and
> a BASIC interpreter/compiler, it seems rather silly to invest in
> a Mathematica license and not use Mathematica's mathematics toolbox.

You miss the point. Buying a Mathematica license contributes money
to SW.

>
> Mathematica may well become the easiest to learn language *for
> doing mathematics and advanced computation*.

I doubt that it will become any easier than it is now.

But, I don't see
> Mathematica becoming the easiest to learn language for say
> creating web sites, games and a great many other things that are
> created using programing languages.

One could write, in Mathematica, a package that includes selections from
palettes and such, that could be used for constructing web sites, games,
etc.

Such a package could also be written in other languages too.

Does WRI have the people who could design such a package that qualified
as "easiest to use"? Is SW motivated to pay such people to produce
such a product? Maintain it?

The comparison would be with (say) Ruby, Python, PHP, and many older
languages that have been made web-aware.


RJF
>
>