SymPy solve() results in unusual memory usage on system of two big equations

[scurrier]

While running solve() on a system of two big equations over the course of three days, I came back to find what I'd consider bizarre memory usage.  Solve was not complete (not unexpected) and the python process had a commit charge of 100GB with only 6GB in the working set (unexpected).  The peak working set was a number I can't remember, but it was at most 50GB and most likely closer to 32GB.  The last time I saw memory usage like this (high commit charge, low working set), the python module I was using had a reference leak.

Is this memory usage suspicious?

Does sympy purport to be able to solve a system of two big nonlinear equations?

Anticipating a possible response to that second question:

I had played with using nonlinsolve() instead of solve() initially, but had better success with solve() on a simplified version of the equations where many symbols had been numerically substituted.  So I tried solve() for the non-substituted version and got the strange memory behavior above.

I am using the latest version of SymPy as pulled from the github repo a week or two ago.  Python 2.7.  Windows 7.  Intel Xeon E5-something v3, 6 cores w/ hyperthreading, 64GB physical memory.

I love SymPy, especially the mechanics module. Thanks for this great software.

[Aaron Meurer]

Did the solve actually finish?

If you could keyboard interrupt (ctrl-C) it when it is high usage and
post the resulting traceback, that would help to pinpoint the issue.
It would also help to post the system itself if you can. At the very
least can you tell us how many equations and variables there were, and
if the system is polynomial system (do all the variables being solved
for create a polynomial or do they appear in functions, like cos(x) or
sqrt(x)?).

But my guess is this is coming from the Groebner bases algorithm used
to solve polynomial systems. SymPy can solve nonlinear systems, but it
depends a lot on the specific system.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sympy/597e8b75-f709-4abb-a33b-ef6835a201f6%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[scurrier]

Hello Mr. Meurer,

Thanks for your help and for your efforts towards the project in general.

The solve did not finish. I eventually killed it.

I will run it again and ctrl+C it to get the traceback at an appropriate time after starting it.

There were only two equations and two unknowns that I was solving for, but there were 50+ other symbols that I was hoping to solve in terms of. It was not purely a polynomial system, it did have functions like cos(x). Not sure what my expectations should be with such a system.

If there is any hope and posting the system can help to further diagnose this, I will consider posting it. It's large, but not insane.

Thanks again.

[Aaron Meurer]

On Mon, Jun 5, 2017 at 4:24 PM, scurrier <shaun....@gmail.com> wrote:
> Hello Mr. Meurer,
>
> Thanks for your help and for your efforts towards the project in general.
>
> The solve did not finish. I eventually killed it.
>
> I will run it again and ctrl+C it to get the traceback at an appropriate time after starting it.
>
> There were only two equations and two unknowns that I was solving for, but there were 50+ other symbols that I was hoping to solve in terms of. It was not purely a polynomial system, it did have functions like cos(x). Not sure what my expectations should be with such a system.

Chances are you won't get a solution because chances are there isn't a
closed form solution, unless you have reason to believe that there is
one. Although I can't say that with absolute certainty, especially
without seeing the system itself.

I should point out that it you are only interested in a numeric
solution, you can use nsolve(), which will work much better, because
it isn't restricted to only finding closed-form symbolic solutions.

Aaron Meurer

>
> If there is any hope and posting the system can help to further diagnose this, I will consider posting it. It's large, but not insane.
>
> Thanks again.
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/5eb3a1f4-dc5a-44c6-9b33-bce902e241ab%40googlegroups.com.

[scurrier]

The system of equations is modeling a physical system. I'd think it's tractable, but I guess I'm not sure. I have seen closed form solutions for problems that are similar, but not identical, which were solved with Mathematica. Would it help to incorporate an assumption that all symbols are real? I did have real=True in the definition of most (not all) symbols already, but it's not clear if this has any effect. I'm not sure how or if it is possible to solve nonlinear systems like this with any of the SymPy assumption systems.

[Aaron Meurer]

real=True, or even positive=True can help SymPy to simplify things
sometimes (like sqrt(x**2) -> abs(x) for x real or x for x positive).

SymPy also has limited algorithms implemented for solving nonlinear
systems of equations, so it's quite possible that if there is a
solution, SymPy won't be able to find it.

Aaron Meurer

On Mon, Jun 5, 2017 at 4:54 PM, scurrier <shaun....@gmail.com> wrote:
> The system of equations is modeling a physical system. I'd think it's tractable, but I guess I'm not sure. I have seen closed form solutions for problems that are similar, but not identical, which were solved with Mathematica. Would it help to incorporate an assumption that all symbols are real? I did have real=True in the definition of most (not all) symbols already, but it's not clear if this has any effect. I'm not sure how or if it is possible to solve nonlinear systems like this with any of the SymPy assumption systems.
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sympy.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/a9de0f6f-fc6a-4529-b426-f936369e53d6%40googlegroups.com.

[scurrier]

Here's the traceback.

Traceback (most recent call last):
File "solve_closed_form_static_level.py", line 17, in <module>
print('Writing to disk complete.')
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\solvers\solvers.py", li
ne 1071, in solve
solution = _solve_system(f, symbols, **flags)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\solvers\solvers.py", li
ne 1681, in _solve_system
poly = g.as_poly(*symbols, extension=True)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\basic.py", line 70
5, in as_poly
poly = Poly(self, *gens, **args)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\polys\polytools.py", li
ne 129, in __new__
return cls._from_expr(rep, opt)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\polys\polytools.py", li
ne 239, in _from_expr
rep, opt = _dict_from_expr(rep, opt)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\polys\polyutils.py", li
ne 352, in _dict_from_expr
expr = expr.expand()
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\cache.py", line 93
, in wrapper
retval = cfunc(*args, **kwargs)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\compatibility.py",
line 809, in wrapper
result = user_function(*args, **kwds)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\expr.py", line 303
3, in expand
expr, hit = Expr._expand_hint(expr, hint, deep=deep, **hints)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\expr.py", line 296
6, in _expand_hint
arg, arghit = Expr._expand_hint(arg, hint, **hints)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\expr.py", line 296
6, in _expand_hint
arg, arghit = Expr._expand_hint(arg, hint, **hints)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\expr.py", line 296
6, in _expand_hint
arg, arghit = Expr._expand_hint(arg, hint, **hints)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\expr.py", line 297
4, in _expand_hint
newexpr = getattr(expr, hint)(**hints)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\mul.py", line 861,
in _eval_expand_mul
terms = self.func._expandsums(sums)
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\mul.py", line 825,
in _expandsums
right = Mul._expandsums(sums[L//2:])
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\mul.py", line 825,
in _expandsums
right = Mul._expandsums(sums[L//2:])
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\mul.py", line 825,
in _expandsums
right = Mul._expandsums(sums[L//2:])
File "C:\Program Files\conda\64bit\lib\site-packages\sympy\core\mul.py", line 827,
in _expandsums
terms = [Mul(a, b) for a in left for b in right]
MemoryError