Need help understanding memory cleanup when using Numba

[Brian Merchant]

Hi all,

What am I using numba for?

I am using numba on a single computer for some math simulation which is very similar to N-body particle simulation. Mainly, I use numba to speed up for loops wherever I can find them (quite a few of them), and additionally, I use it to greatly speed up the function which integrates system ODEs (this is the function I supply to scipy.integrate.odeint). I always use numba in nopython mode.

How is simulation data stored?

I store data in numpy arrays, so I always pre-calculate how much data I need to store -- for example, if there are N particles, and each one of them have M associated variables, and I want to simulate things for T timesteps, then I need a TxNxM float array. I allocate this first right away (with zeroes or ones everywhere) and then fill the array in with data as I execute each simulation timestep. I keep these arrays in RAM, unless I need them to persist, in which case I store pickle and compress them.

When did the problem start to appear?

I have been using numba for about a year, and I think in general things work out quite well. In the last two months though, I have been having some memory issues (I run out of RAM, and then my OS starts to panic trying to shut Python down). This tends to happen when I have a "large" number of particles (N=16 is large for me), and it happens in cases that used to work well 4 months ago.

Have you made any changes to the simulation that might cause it to hog more RAM?

I have made some simulation changes (e.g. in the "rules" the particles follow to interact with each other), but I don't see how this should affect memory usage negatively, because I haven't increased M or T (if anything, M has dropped slightly). One way in which changing the interaction rules for particles might affect memory usage is if the integrator needs to do many more loops for solving each time step -- so, this might have increased a bit, but again, not drastically, because my simulations don't take much longer than before.

How much RAM does your computer have? Are you using 32-bit or 64-bit Python?

I have around 4GB of RAM, but I still use 64-bit Python (I recognize that's no good given how much RAM I have, so that in itself might be an issue?).

What memory usage pattern are you seeing when running a simulation?

Spyder lets me monitor my RAM usage in real time, and I notice that what tends to happen with my simulations is the following: usage starts out low ~30%, then increases steadily to around 85-90% then drops again to about 40-50% (so, did a lot of garbage collection just happen?), before increasing again -- this increase, drop off, increase cycle keep repeating. All's working well!

Occasionally though, (it seems especially when I am not using the computer for anything else in particular), I see messages from Windows prompting me that I have low RAM, and that it needs to shut down some programs. This doesn't tend to happen in a case where I would really expect it to happen: when I am multi-tasking and using other programs (e.g. playing games, I don't have a separate GPU, I use the one that came with my Intel processor) that are *not* Firefox (if I open a bunch of tabs in Firefox, this is almost guaranteed to happen).

Have you been able to notice any meaningful patterns regarding when "low memory" errors occur?

Sadly, no. For instance, just today, I re-ran a simulation that seemed to have crashed with a low memory error, and it ran just fine.

What makes you think this is a numba thing?

I am not certain if it is a numba thing, it could be another issue entirely.

Do you have any ideas regarding what might be going on?

Kind regards,

Brian