Adventures in optimising the CCC code.
Paul Ollis
paragraphs).
To scratch an itch, I have looked at some optimisation of CCC, mainly
focussing on making step 3 faster.
The following based on revision 409. Timings are from a single run.
On my computer, running steps 0, 1, 2, 3, 5 revision 409 gives:
tot user sys
40m07 39m50 0m17
A big gain is obtained by changing the ``container`` function in
series.py to return a tuple instead of a ``constant_list`` object. I
think the resulting code is actually easier to understand.
tot user sys
31m48 31m41 0m07
Adding some temporary variables in ``combind`` to reduce the number of
list indexing operations. This could be argued to reduce readability
by a small amount and the gain is relatively modest.
tot user sys
31m24 31m17 0m07
Elimination of the calls to ``invalid`` and ``valid`` in series.py.
I did this by using direct comparison ``v == MISSING``,
``v != MISSING``. (I also added some asserts to giss_data.py
to catch the sitation where these comparisons do not work
for a given python implementation.) To my eyes the code seems
pretty much as clear.
tot user sys
26m39 26m32 0m07
Elimination of ``invalid`` and ``valid`` from the other steps and
modules. Plus, changing giss_io.py so to allow
``code.giss_data.MISSING`` to be replaced with ``MISSING``. This
involved converting some uses of ``filter`` to list comprehensions.
(I like these, but arguably some of the code ended up less readable.)
tot user sys
25m46 25m39 0m07
Which is all well and good, but my next experiment was very
illuminating.
Using a very recent PyPy on the optimised code I got:
tot user sys
7m48 7m24 0m24
Which is quite impressive. So I tried using PyPy on the original
revision 409 code and it was grindingly (as in looked like it would
never finish) slow. This turned out to be caused by the nested
``constant_list`` in series.py, which seems to to expose a PyPy bug.
A trivial change to series.py (I plan to make this change and check it
in)
and the resulting code gives:
tot user sys
7m50 7m30 0m20
Which means that, all my other optimisations are pretty much pointless
when the code is run using PyPy. So I conclude that if you want the
CCC code to run a lot faster, get PyPy and do not worry too much about
any
other optimisations. I suspect that the only way to get any more
useful gain is to go down the road of recoding some code in C/Cython.
Which, of course, is not what this project is about.
--
Paul
David Jones
On 2 Apr 2010, at 19:19, Paul Ollis wrote:
> For general information (the most interesting bit is in the last few
> paragraphs).
Quite interesting. Do you fancy knocking it up into a short blog post?
Cheers,
drj
PS My work-a-day machine is too small to compile PyPy and there is no
other way to obtain it.
David Jones
On 2 Apr 2010, at 19:19, Paul Ollis wrote:
> For general information (the most interesting bit is in the last few
> paragraphs).
This is really excellent work...
>
> Using a very recent PyPy on the optimised code I got:
>
> tot user sys
> 7m48 7m24 0m24
>
> Which is quite impressive. So I tried using PyPy on the original
> revision 409 code and it was grindingly (as in looked like it would
> never finish) slow. This turned out to be caused by the nested
> ``constant_list`` in series.py, which seems to to expose a PyPy bug.
> A trivial change to series.py (I plan to make this change and check it
> in)
> and the resulting code gives:
>
> tot user sys
> 7m50 7m30 0m20
>
> Which means that, all my other optimisations are pretty much pointless
> when the code is run using PyPy. So I conclude that if you want the
> CCC code to run a lot faster, get PyPy and do not worry too much about
> any
> other optimisations.
... and this take home message of "PyPy makes it 5 times faster" is
just perfect. I've added a note to readme.txt
Did you make the "trivial change"?
drj
Paul Ollis
only just noticed this.
On Jul 13, 11:47 am, David Jones <d...@ravenbrook.com> wrote:
> Reviewing this e-mail again...
>
> ... and this take home message of "PyPy makes it 5 times faster" is
> just perfect. I've added a note to readme.txt
>
> Did you make the "trivial change"?
>
> drj