How sympy runs the benchmarks so fast?

[Asish Kumar]

Hi there, 

I'm a GSoC student with TARDIS-SN this summer and I had a issue with the long time it takes to run benchmarks. Currently my benchmarks are getting run for 1.5 hrs (for only a single commit). I was wondering how sympy does it?

I checked the benchmark actions of sympy and was pretty surprised to see that it only takes 20 mins to run the whole benchmark.

Thank you
Asish Kumar

[Aaron Meurer]

Looking at your linked GitHub action, you run the benchmark in the
last 5 commits. SymPy's CI for PRs only runs the benchmarks for the PR
and master.

Aside from this, it's dependent on how long your benchmarks take to
run. Someone should probably write a guide on how to make a good
benchmark. Ideally a benchmark shouldn't take more than a second to
run, and that's at the very high end (10 or 100 ms is better), because
it needs to be run multiple times to get an average runtime.
Benchmarks that run too quickly are also bad because you are basically
benchmarking noise.

Also, if TARDIS has to be recompiled every time in order to install it
into the virtual environment, that could be a factor (SymPy is pure
Python, so installation time is not an issue). If this is an issue you
might consider caching nightly wheels for recent commits somewhere.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/1ff21b7f-382e-4d73-90fd-85316b9ca5bdn%40googlegroups.com.

[Asish Kumar]

I was recently checking https://github.com/sympy/sympy_benchmarks and had some questions: 

1. Are slow_benchmarks are also run by asv?
2. If yes, why it's in a seperate folder?
3. If No, why it's there?

[Aaron Meurer]

It looks like they aren't run on the run that runs on every pull
request https://github.com/sympy/sympy/blob/958cc95aebc35ced70d870586e68d0468684c1a0/asv.conf.actions.json#L67

There are some separate asv.conf.slow.json configurations in the
benchmarks repo for people who want to run the slow benchmarks. The
benchmarks are still run in the normal asv way of running along a
history of commits from time to time, but this is only done manually
when someone is interested in doing it.

A lot of these benchmarks are only there historically. Many of the
SymPy benchmarks predate asv and were moved from the SymPy repo to the
benchmarks repo. They need to be cleaned up so that they aren't as
slow.

In general, I wouldn't necessarily take the SymPy benchmarks to be the
gold standard in benchmark suites. There is quite a lot of work that
needs to be done there to clean things up and write more/better
benmarks, but no one has really had time to do it except on a case by
case basis when a benchmark is needed for something or other. It's
hard because writing good benchmarks is hard and also because there
are literally tens of thousands of functions and behaviors in SymPy
that could be benchmarked, so choosing the right things to benchmark
is important.

Aaron Meurer

On Thu, Jul 25, 2024 at 11:40 AM Asish Kumar

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/34cb7637-4acd-4ade-8b14-b684edaf31d2n%40googlegroups.com.