"RuntimeError: std::bad_alloc" for optimization problem with 100000 optimization varbiables.

[Anne Mai Ersdal]

Hi,

I am trying to solve a rather large optimization problem with Ipopt (around 100000 optimization variables). I use

solver.setOption("expand",True)
solver.setOption("linear_solver", "ma27")

which works fine with smaller versions of the optimization problem. I am running the binary version of Casadi 2.0.0 in Python.

With these large optimization problems I get the casadi error msg "RuntimeError: std::bad_alloc".

As I understand, the option solver.setOption("expand",True) will increase the memory usage, however I get the same errormsg if I remove it. Is there any way to deal with this, or is the problem just too large? Would it help to update Casadi? (I am a bit reluctant to do this as I am almost finished with this work, and as I understand there are some rather large changes to the newer versions?)

Best,
Anne Mai

[Anne Mai Ersdal]

Small update: When I remove solver.setOption("expand",True) I don't get the errormsg, but the optimization never seems to finish.

[Joel Andersson]

Hi Anne Mai,

So the "solution" should indeed be to remove that option. Whether the option is set to true or false should not matter for the optimization algorithm, it does not influence what is being calculated, just the way it is being done. But there might have been a bug present in CasADi 2.0 and in that case, it might help to update or find another way to work around it (find a computer with more memory, ha!). If you can scale down the problem enough so that you don't get an out-of-memory (e.g. decreasing the time horizon etc), and you still see a difference in IPOPT iterations, it's a clear indication of some bug.

I hope that helps,

Joel

[Joel Andersson]

Another tip: Chances are that much of the memory is needed for the calculation of the Hessian. So disabling exact Hessian (set "hessian_approximation" to "limited-memory") might help.

Joel

[Anne Mai Ersdal]

Thank you for your reply, Joel! I think I have figured out what the problem might be. The optimization when removing solver.setOption('expand', True) did finish eventually, also returning insufficient_memory, so I don't think there is a bug i Casadi. In task manager it doesn't seem like I'm running out of RAM, so it might be the fact that I am running a 32-bit version of casadi and python which limits the RAM-access? Anyway, I am now in the process of updating to versions of casadi precompiled for 64 bit, and I have encounter a problem when going from 2.2.0 to 2.3.0. I am using MA27, and on version 2.0.0-2.2.0 Ipopt has been pleased with libhsl.dll. When installing 2.3.0, however, Ipopt returns an error that it cannot locate libhsl.so. When googling it seems like .so is for UNIX and .dll is for Windows? Why the sudden change in preferances? I am running on windows 7.

When I go from one Casadi-version to another I simply replace the Casadi-catalogue in Python27/Lib/site-packages with the one in the .zip-file for the new version. In the .zip-file for 2.3.0 there where more catalogues than with previous versions. Is it a problem that I only included the casadi-catalogue? I have a binary installation.

Anne Mai

[Joris Gillis]

Dear Anne,

Somewhere in 2.*, a bug slipped in (fixed in 2.4) that made ipopt look for libhsl.so instead of libhsl.dll in windows. Simply copying libhsl.dll to libhsl.so should resolve this issue.

Best,
  Joris