QuTiP v5 benchmarking live!
21 views
Skip to first unread message
Simon Cross
Nov 24, 2022, 11:25:36 AM11/24/22
to qu...@googlegroups.com
Greetings,
In addition to the tutorials, this year's GSoC saw a rewrite of
QuTiP's long unused benchmarking suite.
You can find plots of the benchmarks at https://qutip.org/qutip-benchmark/.
A big thank you to Xavier Spronken, the GSoC student who did all the
work, and to Eric Giguère for supervising Xavier's project.
Measuring software performance is a nuanced and complex art. The
results depend on many unexpected factors, from the precise details of
the underlying hardware and software, to the nature of the tasks
benchmarked and the specific parameters chosen for those tasks.
With that in mind, here is a description of what has been built.
Currently, benchmarks are run only for QuTiP 5, the upcoming QuTiP
release that is currently available in the dev.major branch. It would
have been interesting to include QuTiP 4, but practically we're
unlikely to make big changes to the QuTiP 4.7.x series of releases now
and it would have required significant extra work.
Benchmarks exist for three primary groups of tasks:
- Arithmetic operations on Qobjs
- Arithmetic operations on QobjEvos
- Solvers
And we produce two kinds of plots from the results:
- Historical plots that show changes in performance over time, and
- Scaling plots that show how performance scales with increased problem size
We now have a nice modular framework, so adding additional benchmarks
should be relatively straightforward.
Benchmarks are run nightly on GitHub Action virtual machines. Running
benchmarks on VMs is known to give very noisy results. However, it
turns out that much of the variability comes from which real CPU the
VM is running on, and by accounting for that one can get quite useful,
although still a little noisy, results.
On our plots, for example in the Qobj addition benchmarks
(https://qutip.org/qutip-benchmark/ops/add.html), you will see
differents styles of lines for different CPUs (dashed, dotted, solid)
and different colours for the different implementations (QuTiP dense,
QuTiP sparse, numpy, SciPy).
The benchmarks are written using pytest-benchmark and results are
stored in Amazon S3.
Since results are stored in S3, we can add results from machines
anywhere, and indeed we would very much like to set up daily runs on
raw hardware. If you're in a position to offer dedicated hardware to
run benchmarks on, please let us know!
Contributing:
The benchmarks are maintained in the
https://github.com/qutip/qutip-benchmark repository. Suggestions for
new benchmarks or bug reports are welcome (please open an issue or
pull request).
If you or your company would like to support the benchmarking effort
QuTiP would welcome contributions towards running benchmarks, either
by offering to pay for running benchmarks on additional cloud
instances (e.g. machines with better CPUs, more RAM, GPUs) or by
offering to run benchmarks on dedicated hardware you have available.
If this interests you, please email the qutip-admin mailing list.
At your service,
Simon Cross
In addition to the tutorials, this year's GSoC saw a rewrite of
QuTiP's long unused benchmarking suite.
You can find plots of the benchmarks at https://qutip.org/qutip-benchmark/.
A big thank you to Xavier Spronken, the GSoC student who did all the
work, and to Eric Giguère for supervising Xavier's project.
Measuring software performance is a nuanced and complex art. The
results depend on many unexpected factors, from the precise details of
the underlying hardware and software, to the nature of the tasks
benchmarked and the specific parameters chosen for those tasks.
With that in mind, here is a description of what has been built.
Currently, benchmarks are run only for QuTiP 5, the upcoming QuTiP
release that is currently available in the dev.major branch. It would
have been interesting to include QuTiP 4, but practically we're
unlikely to make big changes to the QuTiP 4.7.x series of releases now
and it would have required significant extra work.
Benchmarks exist for three primary groups of tasks:
- Arithmetic operations on Qobjs
- Arithmetic operations on QobjEvos
- Solvers
And we produce two kinds of plots from the results:
- Historical plots that show changes in performance over time, and
- Scaling plots that show how performance scales with increased problem size
We now have a nice modular framework, so adding additional benchmarks
should be relatively straightforward.
Benchmarks are run nightly on GitHub Action virtual machines. Running
benchmarks on VMs is known to give very noisy results. However, it
turns out that much of the variability comes from which real CPU the
VM is running on, and by accounting for that one can get quite useful,
although still a little noisy, results.
On our plots, for example in the Qobj addition benchmarks
(https://qutip.org/qutip-benchmark/ops/add.html), you will see
differents styles of lines for different CPUs (dashed, dotted, solid)
and different colours for the different implementations (QuTiP dense,
QuTiP sparse, numpy, SciPy).
The benchmarks are written using pytest-benchmark and results are
stored in Amazon S3.
Since results are stored in S3, we can add results from machines
anywhere, and indeed we would very much like to set up daily runs on
raw hardware. If you're in a position to offer dedicated hardware to
run benchmarks on, please let us know!
Contributing:
The benchmarks are maintained in the
https://github.com/qutip/qutip-benchmark repository. Suggestions for
new benchmarks or bug reports are welcome (please open an issue or
pull request).
If you or your company would like to support the benchmarking effort
QuTiP would welcome contributions towards running benchmarks, either
by offering to pay for running benchmarks on additional cloud
instances (e.g. machines with better CPUs, more RAM, GPUs) or by
offering to run benchmarks on dedicated hardware you have available.
If this interests you, please email the qutip-admin mailing list.
At your service,
Simon Cross
Shahnawaz Ahmed
Nov 24, 2022, 11:27:27 AM11/24/22
to Simon Cross, qu...@googlegroups.com
Awesome Simon, great work.
--
You received this message because you are subscribed to the Google Groups "QuTiP Board Admin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qutip-board-ad...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qutip-board-admin/CAD5NRCGWudW7qwiwP%3DujDtyA9WcDpG05XmVJ2pJwMggQUWTCWg%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages