Being more conscious of performance

[Aaron Meurer]

I just merged https://github.com/sympy/sympy/pull/8506, which fixes a couple of pretty major performance issues. Whether or not you see better performance in your own code after this will depend on what that code is doing, but I'd like to use this as an opportunity to pontificate a bit on performance. 

In SymPy, we always need to be aware of the tradeoff of symbolic capability and performance. Whenever you use a general algorithm, that algorithm has a potential to be slow. There is no way around this, since algorithms that work on general symbolic expressions can require computing arbitrary things to get answers. It's true that algorithms can be made faster, but this is also a limit of the mathematics itself.

Thus, there needs to be a very clear distinction between fast/dumb and slow/smart functions. Anything that is slow/dumb should only do the bare minimum to get an answer. Returning "I don't know" (in whatever form that means for the specific function) is preferable to spending a long time to try to return an answer. In cases where a wrong answer would be mathematically incorrect, we have to stipulate that the function works structurally, not mathematically. 

Here are some examples of things that should be fast/dumb/structural:

- == (structural equality)

- match

- xreplace

- any form of automatic evaluation/simplification (that is, it should never be slow to just create an object)

Of course, we can always have slow/smart/mathematical versions of these, like equals and subs. The assumptions should also be considered to be slow. So is printing (as a side-note, don't ever include the string form of an expression in an exception, as a large expression could take forever to print before the exception even gets raised).

The fast algorithms should never call the slow ones, unless the type of object being called is known to a degree that it will be known to be fast. 

In the pull request I merged, automatic evaluation of functions performed as_real_imag() in some cases, just to check if the expression should automatically evalf (example of a slow expression: sin((1.0 + 1.0*I)**10000 + 1)). Another issue was that match was calling signsimp(), which is slow enough to be considered a mathematical function.

In general, we should avoid doing things like this in the core. Assumptions is a big part of this, since the smarter the assumptions get, the slower they will be in general (even if the common cases get faster). We should not be using assumptions in the core (that is, in automatic evaluation, or other fast/dumb contexts). 

To achieve this, we need to do a few things:

- Be rigorous about this in pull requests.

- Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. 

- If a something is slow, investigate why it is slow with a profiler. The best Python profilers are snakevis (which is a front-end to the built-in profile module), line_profiler, and pyinstrument. These each profile Python in a different way, so each can be useful. 

- If a test is slow, don't just mark it is slow. Investigate why it is slow. Bisect to see if it didn't used to be slow. 

- Benchmark functions that are supposed to be fast when we create them, by stress testing them so that we can be assured that they won't slow something down (I think _aresame is an example of this). 

- Stress test SymPy in general. 

- When you fix a performance issue, add a test for it. Something that should run very fast, but would never finish if the wrong algorithm were used. Put a comment on the test that it is a performance test, so that it isn't changed when there is a regression. 

It's easy to just look at SymPy and brush off the speed as a side-effect of being in pure Python, but we should take the view that SymPy can be faster than it is.

Aaron Meurer

[Alan Bromborsky]

What about using parallel python (http://www.parallelpython.com/) were applicable in the code.

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6JmNoN%2Bdjis9a5dsa1nyDRMmp106QAhZeYxcOrxJh%2Bfqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Björn Dahlgren]

Hi all,

On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote:

- Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. 

I found a few hours and threw this together (motivated by having noticed some things I have been doing with sympy has been getting slower):

https://github.com/bjodah/sympy_benchmarks_bjodah

then, on a machine with asv 0.1.1, I ran:

   $ asv run ALL -s 100

which essensitally takes 100 equally spaced commits from the whle history and run the benchmarks one those.
You can view the results here:

http://hera.physchem.kth.se/~sympy_asv

some commits failed, maybe I should be targetting the merge commits but I haven't
figured out how to do that yet.

Furthermore: the comparison isn't quite fair since I belive the later versions return more correct results (Piecewise solutions),
I'm not sure how to deal with this fairly, maybe some kind of assertion should be made (like a tests) at the end of the individual benchmark,
a passing assertion means it should be included, a failing one that it should be excluded. But then this
assertion shouldn't be included in the timing. There could be something in asv for this but I haven't figured it out yet.

I am stil a bit new to using asv so I haven't quite figured out best practives here.
It would be nice to have a dedicated machine (or maybe several), maybe a Raspberry Pi 2 which
would make it easy for people to get the exact same hardware for comparisons?

Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an
interest.

[Ondřej Čertík]

Björn, thanks a lot for doing this! I need to setup something similar
for SymEngine, I am currently benchmarking every commit that I think
might slow things down by hand.


I agree with everything that Aaron said. SymEngine is only fixing the
"the speed as a side-effect of being in pure Python", i.e. with C++ we
are much closer to the metal, so for low level representations of the
symbolic expressions we have more freedom to nail the speed, but
algorithmically it's the same issue as with SymPy and everything that
Aaron said applies to SymEngine as well.

We can buy a dedicated machine via NumFOCUS, SymPy should have enough
money there, but I don't have time personally to make it happen. If
anybody is interested, let me or Aaron now.

Ondrej

[Jason Moore]

I just tried this out with jacobian() and subs() over the commits since 0.7.3 to master. It's showing me that the new caching is the killer slowdown:

https://github.com/sympy/sympy/commit/a63005e4

I've submitted a PR to Björn's repo: https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files

I'm assuming it uses fastcache because I have it installed, but maybe not. I'm not sure how to control for dependencies yet.

Jason
moorepants.info
+01 530-601-9791

--
You received this message because you are subscribed to the Google Groups "sympy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBBSj_BRVzPGubknQYmAvEaGLYhH%2BK4B7dXR90szteXNg%40mail.gmail.com.

[Ondřej Čertík]

Can you post your timings results? I am doing your benchmark with
SymEngine and Sage for comparison.

> https://groups.google.com/d/msgid/sympy/CAP7f1Aggf6HP_ka_3b73QwVGv67ShwRqtqgoRnCULVDmtbxTMg%40mail.gmail.com.

[Jason Moore]

I'm rerunning with the dependencies installed (I think). I'll post results once it is done.

Jason
moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBECZQHgY9KfpGCmymq%3DbGFWCr4LNxq6HY8UbakL6qWZg%40mail.gmail.com.

[Aaron Meurer]

Cool. For this benchmark, you're likely seeing the evolution in the integration algorithms as well. The risch algorithm doesn't handle this integral (too many symbolic parameters), but meijerg does, and so does heurisch. 

That's a great idea. 

Aaron Meurer

 

Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an
interest.

--

You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/5a7e4b53-6daa-4fc2-a8ac-a3288a0270c7%40googlegroups.com.

[Jason Moore]

Here is the last run I made:

http://www.moorepants.info/misc/sympy-asv/

from:

asv run sympy-0.7.3..master -s 200

Jason
moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2Bi13eMuPvm3bK_ME9kxpBjW9MpwWvJ5XyNJiWPYqAYMA%40mail.gmail.com.

[Ondřej Čertík]

On Mon, Jul 20, 2015 at 12:33 AM, Jason Moore <moore...@gmail.com> wrote:
> Here is the last run I made:
>
> http://www.moorepants.info/misc/sympy-asv/
>
> from:
>
> asv run sympy-0.7.3..master -s 200

Is caching causing the massive (10x) slowdown? If so, I know you can
turn it off. We should investigate this.

Ondrej

[Jason Moore]

Yes, it seems that the new cache commit is the slow down in these tests.

Jason
moorepants.info
+01 530-601-9791

--
You received this message because you are subscribed to the Google Groups "sympy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVA2QwER52pHrCTWvhQ%2B2fTSGDE9R0hF74aqmV%3DTM%2B3pxQ%40mail.gmail.com.

[Björn Dahlgren]

On Monday, 20 July 2015 09:02:23 UTC+2, Jason Moore wrote:

Yes, it seems that the new cache commit is the slow down in these tests.

Running with fastcache installed seems to make a minor difference (~10-30%)
http://hera.physchem.kth.se/~sympy_asv/

I haven't yet tried running tests with SYMPY_USE_CACHE=no
(it is my understanding that the environment variable needs to be set before importing sympy, which means I need to hardcode it to be
set in the *.py files in benchmarks/)

[Ondřej Čertík]

On Mon, Jul 20, 2015 at 1:02 AM, Jason Moore <moore...@gmail.com> wrote:
> Yes, it seems that the new cache commit is the slow down in these tests.

If this is the case, then I know that Peter Brady who wrote it will be
interested in this. We should get to the bottom of the issue.

Ondrej

[Peter Brady]

We visited the jacobian issue a while ago and I think the takeaway was that a larger cache size (about 2-3000) sped things up considerably.  Not sure if this is the same issue though.

[Peter Brady]

Elaborating a bit, the 2 main additional costs of a bounded cache compared to an unbounded one are:

1.  The extra cost of managing the LRU machinery (fastcache alleviates this by doing all the necessary management at the C level)

2.  The cost of repeated cache misses because the cache size is not appropriate for a particular problem.

Problem 2 is much harder to quantify and solve efficiently and could easily be the cause of the significant slowdown on some tests.  We could just make the cache bigger...  

[Jason Moore]

This now has every commit from 0.7.3 on:

http://www.moorepants.info/misc/sympy-asv/#diff.TimeJacobian.time_subs

first major slow down: new caching added

slight speedup: fastcache optional dep added

another speedup: cache increased from 500 to 1000

slight slow down: c removed from core

This goes for subs and jacobian.

Jason
moorepants.info
+01 530-601-9791

--

You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/a2484c2c-a031-46f9-8dc9-827efbc48927%40googlegroups.com.

[Aaron Meurer]

Awesome. This is exactly the sort of thing I've wanted to see for a long time. 

So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests).

Aaron Meurer

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AhWxQBUQY9fkJVmxOcAPuz%2BJefBOdmTDMNO5f1cSoHfOA%40mail.gmail.com.

[Björn Dahlgren]

On Monday, 20 July 2015 19:09:54 UTC+2, Aaron Meurer wrote:

So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests).

asv has support for tracking memory usage:
http://asv.readthedocs.org/en/latest/writing_benchmarks.html#memory

It's just a matter of adding memory benchmarks..

[Aaron Meurer]

Regarding the Raspberry Pi 2, as far as I can tell it is a good option, but I opened https://github.com/spacetelescope/asv/issues/292 to see if anyone else has any suggestions. 

Aaron Meurer

--

You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com.

[Jason Moore]

FYI, if I increased the cache size I can push the timings, post new cache, down to the normal speeds.

e.g.

SYMPY_CACHE_SIZE=5000 asv run 051850f2..880f5fa6

Jason
moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com.

[Ondřej Čertík]

On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore <moore...@gmail.com> wrote:
> I just tried this out with jacobian() and subs() over the commits since
> 0.7.3 to master. It's showing me that the new caching is the killer
> slowdown:
>
> https://github.com/sympy/sympy/commit/a63005e4
>
> I've submitted a PR to Björn's repo:
> https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files
>
> I'm assuming it uses fastcache because I have it installed, but maybe not.
> I'm not sure how to control for dependencies yet.

I modified your benchmark to first do the substitution for symbols,
only then do the Jacobian:

https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c

Here are the results:

certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py
Setup
Converting to SymEngine...
SymPy Jacobian:
Total time: 0.155253887177 s
SymEngine Jacobian:
Total time: 0.00277400016785 s

Speedup: 55.97x


I played with the cache size, it makes no difference. Your original
benchmark ran about 1s on my machine, with SYMPY_CACHE_SIZE=5000 it
took about 0.9s. So you can speedup SymPy itself just by using symbols
instead of functions by a factor of 5.8x. Btw, I tested that the
result that I get from symengine is exactly the same as in sympy
(https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c#diff-e423f8edd8641c702513b2ea9c10eaecR32).


How did you generate the benchmark? Can you generate a bit larger
matrix? Once the symengine benchmark runs in about 1s, then I can try
to speed it up (because I would see the result in the timing more
easily).


Ondrej

[Ondřej Čertík]

I actually found a much larger benchmark, where just a simple
differentiation of a single entry (after substituting functionsf or
symbols) takes 60s for sympy.

https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8

Here is where I am saving the matrix:

https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8#diff-39829995ea91876984e13e3b828d740aR252

Does this make sense? Is that a good benchmark? I love the size, I
just want to make sure I am saving the right thing. Is that the full
bicycle?

Ondrej

[Jason Moore]

Ondrej,

I'm not sure why you don't see performance increase with increased cache. The following shows that the benchmarks do run faster with a large cache. Interestingly the memory doesn't seem to change (but I'm not sure I understand how they measure mem usage). Notice that the jacobian wrt to symbols is not affected by the cache size but the jacobians wrt to the functions is, as is substitution. It seems that .diff() calls subs a lot (see https://github.com/sympy/sympy/issues/9701), so maybe the slow down is all due to the caching used for subs.

moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ asv --config asv.conf.conda.json run
· Fetching recent changes.
· Creating environments
· Discovering benchmarks
·· Uninstalling from py2.7-fastcache-mpmath.
·· Building for py2.7-fastcache-mpmath
·· Installing into py2.7-fastcache-mpmath..
· Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks)
[  0.00%] · For sympy commit hash 488f3c20:
[  0.00%] ·· Building for py2.7-fastcache-mpmath...
[  0.00%] ·· Benchmarking py2.7-fastcache-mpmath
[ 11.11%] ··· Running integrate.TimeIntegration01.time_doit                                                                                  365.73ms
[ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg                                                                          107.41ms
[ 33.33%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions                                                    37M
[ 44.44%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols                                                      37M
[ 55.56%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_subs                                                                      37M
[ 66.67%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions                                                     1.75s
[ 77.78%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols                                                     35.49ms
[ 88.89%] ··· Running large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions                                              1.79s
[100.00%] ··· Running large_exprs.TimeLargeExpressionOperations.time_subs                                                                       1.66s
moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ SYMPY_CACHE_SIZE=10000 asv --config asv.conf.conda.json run
· Fetching recent changes.
· Creating environments
· Discovering benchmarks
·· Uninstalling from py2.7-fastcache-mpmath.
·· Building for py2.7-fastcache-mpmath
·· Installing into py2.7-fastcache-mpmath..
· Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks)
[  0.00%] · For sympy commit hash 488f3c20:
[  0.00%] ·· Building for py2.7-fastcache-mpmath...
[  0.00%] ·· Benchmarking py2.7-fastcache-mpmath
[ 11.11%] ··· Running integrate.TimeIntegration01.time_doit                                                                                  361.81ms
[ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg                                                                          104.40ms
[ 33.33%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions                                                    37M
[ 44.44%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols                                                      37M
[ 55.56%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_subs                                                                      37M
[ 66.67%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions                                                   56.62ms
[ 77.78%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols                                                     33.04ms
[ 88.89%] ··· Running large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions                                            55.25ms
[100.00%] ··· Running large_exprs.TimeLargeExpressionOperations.time_subs                                                                     13.89ms
 

Jason
moorepants.info
+01 530-601-9791

Ondrej

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDZ_JSgC-C_j5J0zmSOXS9TAQsGFv_XcmbBvx3ijPhPrg%40mail.gmail.com.

[Ondřej Čertík]

It's because I didn't have fastcache installed.... After installing
it, by default I got:

certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py
Setup
Converting to SymEngine...
SymPy Jacobian:

Total time: 0.123499155045 s
SymEngine Jacobian:
Total time: 0.00305485725403 s

Speedup: 40.43x


(I am running some longer calculation at the same time, so the timing
is a bit different, but it got faster.)
But bigger cache doesn't seem to affect it, as you noted:

certik@redhawk:~/repos/symengine/benchmarks(py)$
SYMPY_CACHE_SIZE=50000 python kane2.py

Setup
Converting to SymEngine...
SymPy Jacobian:

Total time: 0.121557950974 s
SymEngine Jacobian:
Total time: 0.00302696228027 s

Speedup: 40.16x



More interestingly, I can now differentiate the actual bicycle
expression. So far I just took one element on the right hand side
vector, and differentiate with respect to q5. I substitute for symbols
first (in both SymPy and SymEngine). This is what I got:

certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane3.py
Converting to SymPy...
Converting to SymEngine...
Done.
Subs
Eval initial expression
-14960.1119361
-14960.111936127382563475771853345057230700491739739146860248076 +
0.00000000000000000000000000000000000000000000000000000000000000*I
SymPy diff:
Total time: 3.90587186813 s
SymEngine diff:
Total time: 0.0883350372314 s

Speedup: 44.22x

Subs
Eval derivative
-31789.936485
-31789.936484889832248148585748003998439133612431284784993412159 +
0.00000000000000000000000000000000000000000000000000000000000000*I
Converting SymPy derivative to SymEngine...
Subs
Eval derivative
-31789.936485
-31789.936484889832248148585748003998439133612431284784993279022 +
0.00000000000000000000000000000000000000000000000000000000000000*I


The floating point numbers are calculated by substituting integers for
the symbols and evaluating. The shorter number is using machine double
precision, the longer number is using MPC (evaluates in complex plane,
to make sure we don't get any negative values under square roots).
They both agree as you can see. The initial expression number I
checked against sympy (substituting the numbers using sympy and
evaluating using sympy). For the derivatives, currently I just the
sympy derivative to symengine to evaluate as a floating point, since
sympy is just too slow (it's been running for 30 minutes -- it's stuck
at the .n() method in sympy). The SymEngine evaluation is immediate
for double precision and a fraction of a second for the MPC. But the
conversion between symengine and sympy works well and I tested the
initial expression in both, so I think this part works. And the fact
that sympy derivative calculated using sympy and converted to
symengine and symengine derivative agrees I think proves with pretty
high confidence that things work.

Now the next step is to calculate the whole jacobian and again test
that we get the same expression. It's very important to spend time and
check the numerical evaluation of the expression, since there could be
bugs in symengine. I just fixed one here:
https://github.com/sympy/symengine/pull/556 (it was giving me
different numbers for the initial expression, but now they agree).
This was a bug in subs, but there could be a bug in the derivative as
well.

After we get the same expressions, we can start benchmarking. I can
start profiling the derivative code in C++ and see if it is possible
to speed it up. I can also implement cache in symengine, though I
don't know if it would speed up anything here. Still, even if we only
get 45x speedup, then instead of 4 minutes, the whole bicycle should
take 5s. So I think that's a big deal. I will also try to benchmark
this against other software like Sage to get an idea how we are doing.

Ondrej

> https://groups.google.com/d/msgid/sympy/CAP7f1AhavPNhXzdhUDUVH1ihgDgbc4FbdmRPBHrMW9HQ4ijZDA%40mail.gmail.com.

[Jason Moore]

Nice work!

Jason
moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVCNOioRCvXJ5SrdTn41m7R9RMpp_c9ASTY-oO5nVkR8LQ%40mail.gmail.com.

[Ondřej Čertík]

It eventually finished, and the floating point number calculated using
pure SymPy was exactly the same as with SymEngine. So I trust the
derivatives and conversion back and forth by now for this particular
expression.

I don't know how long it took, probably a few hours (compared to less
than 1s in symengine, so clearly the .n() algorithm is not the most
efficient, it probably has to do with the fact that it automatically
checks the accuracy --- showing that a better way to do that is just
keep evaluating at increasing precision and see if it converges, or
use something like Arb).

Overall I am very happy about this progress, we have a robust
foundation to build upon. I'll try to get this very benchmark working
with Sage to get an idea how long it takes there. The branch with
symengine+sympy code is here:
https://github.com/sympy/symengine/compare/master...certik:py, it's
not very polished, that I will do later.

Ondrej

[Jason Moore]

FYI, this database is building up: http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy cache set to 10000).

For the integration benchmarks it has every single functioning commit and maybe half the commits for the other benchmarks.

The repository to submit benchmarks to is now at:

https://github.com/sympy/sympy_benchmarks

Please submit benchmarks for core parts of SymPy that you work on.

Jason
moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVCB9%2B8EmZjEO%3DTH9ihU3eD-aGdKP7fymKnsXh3Y5bjsPA%40mail.gmail.com.

[Ondřej Čertík]

On Wed, Jul 22, 2015 at 4:16 PM, Jason Moore <moore...@gmail.com> wrote:
> FYI, this database is building up:
> http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy
> cache set to 10000).
>
> For the integration benchmarks it has every single functioning commit and
> maybe half the commits for the other benchmarks.
>
> The repository to submit benchmarks to is now at:
>
> https://github.com/sympy/sympy_benchmarks
>
> Please submit benchmarks for core parts of SymPy that you work on.

That is awesome. Essentially we can rerun it any time in the future as
well if we want to figure out why is some code slow and if it was
always that slow.

Can you post instructions how to run it on my own machine? (I'll be
happy to write it up into the README.)

What is the ideal time for a benchmark? Can in benchmark a quick
running code by running it several times? Does it work well if the
benchmark takes 10s? Or is that too long.

Ondrej

[Jason Moore]

We will work on the documentation. But essentially I'm a running:

SYMPY_CACHE_SIZE=10000 asv run ALL --parallel -k -e

You can use -s <num> to do a sparser mesh and then replace ALL with a git commit range to narrow down to commits in specific areas. Note that `asv find` is useful for bisecting to an offending commit.

To view the results do:

asv publish

asv preview

Bjorn and I discussed the ideal time. I'd say keep them under a second or two in general. There may be cases that you need to use longer benchmarks but I think you should profile the code and then write a benchmark that tests the slow portions (which will likely be short benchmarks). But it can run any length benchmark, but remember that they will ultimately be run on every single commit (~25000) so it can take a really long time.

I'll start running "asv run NEW" once we have things going and benchmark designs stabilize.

Jason
moorepants.info
+01 530-601-9791

Ondrej

--
You received this message because you are subscribed to the Google Groups "sympy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVCv6zYH5crbtEG76O_D27e-e7uvOj-FQv%2BYF8z8xtbx1A%40mail.gmail.com.

[Ondřej Čertík]

On Wed, Jul 22, 2015 at 6:26 PM, Jason Moore <moore...@gmail.com> wrote:
> We will work on the documentation. But essentially I'm a running:
>
> SYMPY_CACHE_SIZE=10000 asv run ALL --parallel -k -e
>
> You can use -s <num> to do a sparser mesh and then replace ALL with a git
> commit range to narrow down to commits in specific areas. Note that `asv
> find` is useful for bisecting to an offending commit.
>
> To view the results do:
>
> asv publish
> asv preview
>
> Bjorn and I discussed the ideal time. I'd say keep them under a second or
> two in general. There may be cases that you need to use longer benchmarks
> but I think you should profile the code and then write a benchmark that
> tests the slow portions (which will likely be short benchmarks). But it can
> run any length benchmark, but remember that they will ultimately be run on
> every single commit (~25000) so it can take a really long time.

Thanks for the info. Just to clarify, if I run 1 benchmark that takes
1s for the whole history, it would take less than 7 hours to finish:

In [2]: 25000 * 1. / 3600
Out[2]: 6.944444444444445

Is that right?

> https://groups.google.com/d/msgid/sympy/CAP7f1AiqsyHOQo%3Duy5pMNiqgriOhRrT%3DS18JO-uFTr%2BJUXC8-A%40mail.gmail.com.

[Jason Moore]

Yes, but it is likely to fail on many of the commits (probably more than half depending on the functionality being tested and how backwards compatible the benchmark is written).

There is also setup overhead time for each commit, so it will take longer than your calc.

Jason
moorepants.info
+01 530-601-9791

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDjgpvUJAREPZVWw1BTgPxhe9sYrfSt0zVsew%3DrhSJr1A%40mail.gmail.com.