ERROR: LoadError: OutOfMemoryError()

[jda]

I've run into the following problem while running a simulation code (of a differential equation).  Because of the nature of the problem, the matrices are dense.  In particular one line of code that runs the backslash operator gives an OutOfMemory error on one computer but not on another computer.

The error is:

ERROR: LoadError: OutOfMemoryError()

 in .* at arraymath.jl:118

 in * at abstractarraymath.jl:54

 in functionName at C:\Users\user\Google Drive\codefile.jl:158

 in include at boot.jl:261

 in include_from_node1 at loading.jl:304

while loading C:\Users\user\Google Drive\codefile.jl, in expression starting

 on line 155

(error points to  backslash operator )

Info for the computer that works fine:

• mac laptop
• 1.6 GHz i5 processor
• 4 GB memory
• 250 GB
• running Julia 0.4.1
  

Info for the computer that gives the error:

• windows desktop
• Intel(R) Core(TM) i5-4590 CPU @3.30GHz
• 8 GB memory
• 420 GB drive
• running Julia 0.4.3

Both computers are not running any other programs, and are both fairly new with plenty of space available (the computers are specifically designated to run such code).  I am sure that my next step should be to install Julia 0.4.1 on the windows computer to see if that helps, but it may take me some time to sort through the instructions for installing old versions of Julia.  I wondered whether anyone would have some insight as to why/how this error could occur on a newer version of Julia and on a better machine?

[Milan Bouchet-Valat]

An important information is whether these are 64-bit OSes, and whether
you're using the 32-bit or 64-bit Julia. The latter will use more
memory to store pointers and integers.

> Both computers are not running any other programs, and are both
> fairly new with plenty of space available (the computers
> are specifically designated to run such code).  I am sure that my
> next step should be to install Julia 0.4.1 on the windows computer to
> see if that helps, but it may take me some time to sort through the
> instructions for installing old versions of Julia.

Should be very easy to do:
https://s3.amazonaws.com/julialang/bin/winnt/x86/0.4/julia-0.4.1-win32.exe
https://s3.amazonaws.com/julialang/bin/winnt/x64/0.4/julia-0.4.1-win64.exe

You can also install 0.4.3 on the Mac machine and see whether it still
works.

> I wondered whether anyone would have some insight as to why/how this
> error could occur on a newer version of Julia and on a better
> machine?

AFAIK there shouldn't be any large difference, but who knows?


Regards

[jda]

The OSes and the Julia versions are all 64-bit.  Thanks for the .exe links, I was looking for them.  Good suggestion to install the 0.4.3 version of Julia on the mac laptop.  

What I found has led me to a different question.  I have found that actually the same size matrix that before ran just fine on the mac laptop no longer run -- giving the OutOfMemory error now.  So this problem seems to *not* be related to the version of Julia but rather the state of the system.  I must say in advance that I am trying to push the limits -- trying to run simulations for the biggest grid possible for the best results.  In doing so it seems that I need to learn much more about optimizing memory usage and possible (?) other aspects of how the system is setup?  Any info that would help me to navigate this would be greatly appreciated.

[Milan Bouchet-Valat]

Le mardi 26 janvier 2016 à 12:59 -0800, jda a écrit :
> The OSes and the Julia versions are all 64-bit.  Thanks for the .exe
> links, I was looking for them.  Good suggestion to install the 0.4.3
> version of Julia on the mac laptop.  
>
> What I found has led me to a different question.  I have found that
> actually the same size matrix that before ran just fine on the mac
> laptop no longer run -- giving the OutOfMemory error now.  So this
> problem seems to *not* be related to the version of Julia but rather
> the state of the system.  I must say in advance that I am trying to
> push the limits -- trying to run simulations for the biggest grid
> possible for the best results.  In doing so it seems that I need to
> learn much more about optimizing memory usage and possible (?) other
> aspects of how the system is setup?  Any info that would help me to
> navigate this would be greatly appreciated.

It's hard to tell without seeing the code, but in general you should
avoid allocating temporary arrays by unrolling your loops and using in-
place versions of functions where possible. Of course, also make sure
that there is no type instability in your code which would trigger
unwanted memory allocations.

You could also use Float32 instead of Float64 if reduced precision
isn't an issue for your application.


Regards

[jda]

Thanks, I have considered much of what you mentioned while making my code but of course it can always be rechecked over again for these things.  Good point about float32; in my calculation precision ends up being important.  I do wonder whether my code which is mostly vectorized (i.e. matlab styled) is the most efficient way for matrix calculations, or if in Julia the best way of calculating would be better in fact if I inserted loops to replace vectorized statements.  What do you think about this?  In any case the backslash operator is really the key statement that's creating the OutOfMemory error and the backslash cannot be un-vectorized itself that I know.  I am looking through memory cleaners or other system optimizers but I am skeptical that this would be the main problem because the 8GB windows computer really should have done much much better than the 4GB laptop.  That is the part that I find to be most quizzical !  Could there be some useful info about how memory is utilized by Julia that I am missing?

[jda]

I eventually resolved this problem.  I found a place in my code where I was using the less efficient diag(sparse(vector)) instead of the more efficient diagm(vector).  This ended up being crucial for performance.  Now my code is running on both machines with simply the laptop running slower as expected in comparison to the desktop.