osnews article
srepmub
Raphael Clifford
would be fair to look at pypy 1.7 at least.
Raphael
On 16 January 2012 15:53, srepmub <mark....@gmail.com> wrote:
> http://www.osnews.com/story/25512/Shed_Skin_6_Years_Later
>
> (send in the pypy and cython drones! ;-))
>
> --
> You received this message because you are subscribed to the Google Groups "shedskin-discuss" group.
> To post to this group, send email to shedskin...@googlegroups.com.
> To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.
>
Mark Dufour
On Mon, Jan 16, 2012 at 5:20 PM, Raphael Clifford <drr...@gmail.com> wrote:
> Well.. that is a very old version of pypy from November 2010 :) It
> would be fair to look at pypy 1.7 at least.
well, it should only take a few minutes do re-run these tests with
more recent versions of pypy (and shedskin).. please share the results
if you decide to do so.. :-)
of course shedskin has also had a new release since then, and could
have had two, were it not that I have been somewhat preoccupied since
0.9.
note also that shedskin performs much better on the first two tests
when using shedskin -b.. which turns off bounds checking.
thanks,
mark.
--
http://www.youtube.com/watch?v=E6LsfnBmdnk
Raphael Clifford
> On Mon, Jan 16, 2012 at 5:20 PM, Raphael Clifford <drr...@gmail.com> wrote:
>> Well.. that is a very old version of pypy from November 2010 :) It
>> would be fair to look at pypy 1.7 at least.
>
> well, it should only take a few minutes do re-run these tests with
> more recent versions of pypy (and shedskin).. please share the results
> if you decide to do so.. :-)
It's not quite as simple as it sounds to run the tests as they don't
provide exactly the code they ran it seems. Also, some of the code
comes in different versions for reasons I am not clear on. Here are
some example timings on my rather slow amd computer.
For pypy I used a recent nightly build. For shedskin I used 0.91. The
timings are rounded to the nearest half second to eliminate random
noise. Timings and memory were given by /usr/bin/time
For the matrix multiplication tests you have to set a size and I chose n = 800
matmul_v1.py
pypy : timing 6s , memory 188688maxresident
shedskin : timing 3s , memory 162640maxresident
matmul_v2.py
pypy: timing 4s, memory 154208maxresident
shedskin : 26s, memory 177328maxresident
matmul_v3.py
pypy : timing 5s, memory 154368maxresident
shedskin: timing 46s, memory 111776maxresident
The dictionary test seems very bad for both shedskin and pypy.
cat test.dict |/usr/bin/time ./dict_v1
pypy : 9s, memory 404592maxresident
shedskin : 10s, memory 787344maxresident
shedskin also outputs warning from the garbage collector GC.
BUT
cpython 2.6 : 6s, memory 308560maxresident
The Sudoku test doesn't come with the code they actually used that I
could see (it says the test is run 50 times). So instead I used the
sudoku17 set from http://people.csse.uwa.edu.au/gordon/sudoku17 and
piped the output to /dev/null as it's rather large.
pypy : 48s, 190384maxresident
shedskin : 31s, 8400maxresident
and just for fun
C : 7s, 1664maxresident
The pattern matching test seems to finish almost instantly with no
output so I omitted it. If anyone knows it should be run I would be
happy to run it.
Hope this helps.
Raphael
Raphael Clifford
> output so I omitted it. If anyone knows it should be run I would be
> happy to run it.
>
>
I got the pattern matching benchmarks to run.
First using
([a-zA-Z][a-zA-Z0-9]*)://([^ /]+)(/?[^ ]*)|([^ @]+)@([^ @]+)
pypy : timing 8s, memory 91072maxresident
shedskin : 39s, memory 7680maxresident
Now using
([a-zA-Z][a-zA-Z0-9]*)://([^ /]+)(/?[^ ]*)
pypy : timing 4.5s, ,memory 76640maxresident
shedskin : timing 3s, memory 7664maxresident
Now some notes and comments.
First, you will notice that in fact I rounded to the nearest second in
my first email, not the nearest half second as I claimed. Second, the
memory usage figures in linux are hard to interpret due to the way the
operating system works. The first answer
http://stackoverflow.com/questions/131303/linux-how-to-measure-actual-memory-usage-of-an-application-or-process
briefly sets out the main reasons why the simplest number you get
isn't necessarily what you want and the rest of the post tells you
ways to get better measurements. Lastly, you will see that shedskin
is very very variable in its timings. I will leave the devs to
discuss why this is :)
Raphael
Mark Dufour
thanks for testing. it looks like it may be the case something is
slowing shedskin down on your pc. perhaps the old cpu, an old version
of gcc or libgc or windows. could you perhaps provide some more
details about your system..? ;-)
I ran the matmul and sudoku tests here (linux 3.0, gcc 4.6.1, libgc
7.1, shedskin 0.9.1, Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz,
default FLAGS), and obtained the following results (each time taking
the best time out of three runs).
matmul v1:
cpython 253
pypy1.7 16.4
shedskin 2.9
shedskin -b 1.7
(so a factor of 5 faster than pypy, compared to the 2 times you see..
without -b that is ;-) -b is of course very safe since usually you are
only interested in index-out-of-bounds exceptions during testing.).
matmul v2
cpython 232
shedskin 14.8
shedskin -b 8.8
pypy1.7 7.0
(okay, something seems wrong here, and I'm guessing array element
access is sub-optimal. I will try to improve this and report back. but
usually one will use method v1 and not the array module of course.
this example may only exist to make cpython run faster.)
matmul v3
cpython 296
shedskin 22
shedskin -b 12.4
pypy1.7 4.6
(fastest pypy time here, still about 3 times slower than shedskin -b.
will investigate why shedskin takes 22 seconds here, probably
something silly.)
so I would say we can conclude that given the task of multiplying a
1000 by 1000 matrix, based on these tests, shedskin can do it 3 times
faster than pypy1.7 on a modern system.
as for the sudoku test:
sudoku
cpython 115
pypy1.7 14.8
shedskin 5.4
shedskin -b 3.5
it's great pypy is improving, but (again) I don't think it will be
easy to surpass a static compiler on a restricted subset of python.
jit compilers aren't magic. compiling away such things as arbitrary
size arithmetic, bounds- and wrap-around checking sound nice, but in
practice often end up fairly limited in their scope.
the dict benchmark mostly tests memory allocation within the dict
implementation, so there won't be much difference there. cpython is
obviously highly optimized here. the pattern matching benchmark mostly
tests the used re engine, libpcre in case of shedskin. not much I can
do about that.
thanks again!
mark.
> --
> You received this message because you are subscribed to the Google Groups "shedskin-discuss" group.
> To post to this group, send email to shedskin...@googlegroups.com.
> To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.
>
Mark Dufour
on 64-bit systems, though I haven't measured. anyone? :-))
Mark Dufour
often faster (though one has be sure that doesn't change the program).
then there's -fprofile-generate, -O3 is perhaps faster etc..)
On Fri, Jan 20, 2012 at 10:43 AM, Mark Dufour <mark....@gmail.com> wrote:
Mark Dufour
it turns out shedskin wasn't inlining array.__getitem__ properly yet
(array is quite a recent addition..). after a small change in GIT for
this, I now get these timings:
matmul v2
cpython 232
shedskin 11.2
pypy1.7 7.0
shedskin -b 5.9
matmul v3
cpython 296
shedskin 12.9
shedskin -b 5.8
pypy1.7 4.6
Raphael Clifford
Just a quick note that I was using a recent nightly build of pypy and
apparently they sped a lot of stuff up recently. It might be worth
rerunning your tests on that.
Best wishes,
Raphael
Raphael Clifford
> it turns out shedskin wasn't inlining array.__getitem__ properly yet
> (array is quite a recent addition..). after a small change in GIT for
> this, I now get these timings:
I reran the tests on a more modern computer using a recent pypy
nightly and git shedskin. Times rounded to the nearest half second
and pretty consistent between runs.
matmul v1
shedskin -b : 1.5s
shedskin : 2.5s
pypy : 4.5s
matmul v2
shedskin -b : 5s
shedskin : 10s
pypy : 4.5 s
matmul v3
shedskin -b : 13s
shedskin : 13s
pypy : 3.5s
I reran the sudoku test using http://magictour.free.fr/top1465, again
piping the output to /dev/null
shedskin -b : 1.5s
shedskin : 2s
pypy : 3s
I reran the pattern matching test
pattern ([a-zA-Z][a-zA-Z0-9]*)://([^ /]+)(/?[^ ]*)
shedskin : 1s
pypy : 2s
pattern ([a-zA-Z][a-zA-Z0-9]*)://([^ /]+)(/?[^ ]*)|([^ @]+)@([^ @]+)
shedskin : 14s
pypy : 4s
It's not at all clear to me why shedskin does so badly.
http://lh3lh3.users.sourceforge.net/reb.shtml implies that pcre is not
that much worse than python on this front. The slow patttern is
number 4 in that benchmark it seems.
All in, I think a fair summary of the current state of affairs is as follows.
a) shedskin without bounds checking can be up to 3 times faster than
pypy with bounds checking for some tasks (depending on how the program
is written).
b) shedskin even without bounds checking can be up to 4 times slower
than pypy for some tasks (depending on how the program is written).
c) shedskin's performance appears much more sensitive to the way code
is written than pypy's
d) Both shedskin and pypy can be slower than cpython for some tasks.
System: Intel(R) Core(TM)2 Duo CPU E8500 @ 3.16GHz, kernel
2.6.32-37-generic, 32 bit
Raphael
Mark Dufour
I made another improvement to array.__getitem__, and now get the
following timings for matmul_v3. note I also used GCC's profiling mode
(somewhat fairer, since pypy naturally has profiling information).
this probably helps a lot since there's a conditional in
array.__getitem__.
matmul_v3 (best of 5 runs)
shedskin 11.04
pypy 4.02
shedskin -b 2.66
shedskin -bw 2.29
shedskin -bw, -fprofile-generate/use 1.57
matmul_v2 should be quite a bit faster now as well. thanks for
triggering these array optimizations ^^
not much more time now though, will reply to your other comments later..
thanks,
mark.
Mark Dufour
> pattern ([a-zA-Z][a-zA-Z0-9]*)://([^ /]+)(/?[^ ]*)|([^ @]+)@([^ @]+)
>
> shedskin : 14s
> pypy : 4s
>
> It's not at all clear to me why shedskin does so badly.
> http://lh3lh3.users.sourceforge.net/reb.shtml implies that pcre is not
> that much worse than python on this front. The slow patttern is
> number 4 in that benchmark it seems.
well, using cpython here takes about the same time as shedskin. I
guess the pypy folks have adopted another engine? from this page,
there certainly seem to be faster engines than pcre.
> All in, I think a fair summary of the current state of affairs is as follows.
>
> a) shedskin without bounds checking can be up to 3 times faster than
> pypy with bounds checking for some tasks (depending on how the program
> is written).
> b) shedskin even without bounds checking can be up to 4 times slower
> than pypy for some tasks (depending on how the program is written).
it all certainly depends on the program. shedskin will be faster for
some things, and pypy for others. looking at just these benchmarks,
and after some very simple array optimizations, shedskin seems to win
in most cases though..
I do think that given a certain (algorithmic-type) task, it will often
be possible to create a faster version using shedskin than using pypy.
> c) shedskin's performance appears much more sensitive to the way code
> is written than pypy's
well, it's true with shedskin one may have to alter a program to work
at all.. ;-) shedskin is just not meant to work for arbitrary
programs, but to be really fast with a bit of programmer assistance.
not to say it's not a very worthwhile goal to have a transparent JIT
compiler, that works for, say, django applications.. :-)
> d) Both shedskin and pypy can be slower than cpython for some tasks.
sure, especially in pathological cases, where cpython has had more
than a decade more fine tuning.. ;-)