parrot performance vs.(trivial test) the good, the bad, and the ugly
Tupshin Harper
On a whim I took the primes.pasm example from the parrot examples page
and converted it to both c and perl5, with _interesting_ results.
Timing all three with a max of 100,000 produced the following results:
c -primes.c(lickety split):
real 0m7.710s
user 0m6.790s
sys 0m0.030s
parrot primes.pasm(default options not too impressive):
real 0m33.289s
user 0m29.130s
sys 0m0.080s
parrot primes.pasm(with -P...a bit better)
real 0m21.063s
user 0m20.000s
sys 0m0.050s
parrot primes.pasm (with jit -j option--omfg...i'm impressed)
real 0m12.625s
user 0m11.610s
sys 0m0.040s
perl5 primes.pl(ouch!!!!!!!!!, yes...this is the same algorithm)
real 6m53.454s
user 6m33.490s
sys 0m0.990s
FYI...all three used the identical algorithm taken from the primes.pasm
example complete with labels and gotos(makes for very disconcerting perl
code). Startup times and printf times were not significant in any of the
cases(<5%).
Further curious because of the python challenge, I took a different
(slower) primes algorithm(trying to be fair to python since it doesn't
have gotos), That someone had already written C and python versions of,
and ported it to parrot....drum roll please(only 10,000 primes due to
slower algorithm):
c - primes2.c
real 0m1.147s
user 0m1.050s
sys 0m0.000s
parrot primes2.pasm(default)
real 0m4.868s
user 0m4.700s
sys 0m0.020s
parrot primes2.pasm(-P)
real 0m3.058s
user 0m2.920s
sys 0m0.000s
parrot primes2.pasm(-J)
real 0m1.825s
user 0m1.700s
sys 0m0.000s
python primes2.py
real 0m25.922s
user 0m24.900s
sys 0m0.050s
So at least on simple looping benchmarks like this, parrot is much
closer to the performance of C than it is to perl or python...I liiiike
it (doing my best Tony the Tiger impersonation here). Congrats
guys...it's pretty impressive.
Testing platform is linux debian (sid) on an Athlon 2100, gcc 3.2.3,
perl 5.8.0, python 2.2.2, and parrot CVS head as of today.
-Tupshin
Code available if anybody cares.
Leopold Toetsch
> In case anyone is interested.....
Always :)
Did you have an optimized parrot compile?
( make progclean ; perl Configure.pl ... --optimize ; make -s)
> -Tupshin
>
> Code available if anybody cares.
Yes please.
TIA,
leo
Tupshin Harper
> Did you have an optimized parrot compile?
>
> ( make progclean ; perl Configure.pl ... --optimize ; make -s)
No I hadn't, but I just did, using those exact commands(no additional
options to Configure.pl), and had no perceivable performance change
using any of the parrot variances(default, -P, -j). Is there a way to
verify that an optimized build was truly made? Did you expect to see
much of any difference? GCC issues(running 3.2.3)?
>> -Tupshin
>>
>> Code available if anybody cares.
>
>
>
> Yes please.
Alright...all versions are attached (sorry for spamming the list, but
others might be interested). Some of the code is ugly (done around 3:00
in the morning), and some are in languages I am less then fluent in
(last touched any flavor of assembly in 1985, and barely touched it
then), so be kind. I don't believe I'm being too unfair to any of the
languages, though feel free to tell me otherwise.
The python and c versions of prime2 come from a tutorial on optimizing
python by writing extensions in C
http://kortis.to/radix/python_ext/
Modified slighly to behave more analogously to the first test, and some
compilation bugs fixed.
>
> TIA,
> leo
>
WATF
(welcome after the fact)
-Tupshin
Tupshin Harper
>On Tue, Feb 18, 2003 at 04:03:40AM -0800, Tupshin Harper wrote:
>
>
>>FYI...all three used the identical algorithm taken from the primes.pasm
>>example complete with labels and gotos(makes for very disconcerting perl
>>code). Startup times and printf times were not significant in any of the
>>cases(<5%).
>>
>>
>
>This is, unfortunately, a rigged demo. Its code optimized for Parrot
>and then mechanically translated into Perl and C. The algorithm would
>have been the same if it, for example, used a loop in Perl instead of
>gotos.
>
>I don't know that its all that interesting to show that goto is slow in
>Perl. OTOH, goto should be fast in C, no?
>
>I'm going to tinker with those benchmarks again.
>
>
Agreed...the second example is probably less rigged, since the
mechanical translation was to pasm(and don't know enough to optimize the
pasm ;-)
-Tupshin
Leopold Toetsch
> Leopold Toetsch wrote:
>
>> Did you have an optimized parrot compile?
>>
>> ( make progclean ; perl Configure.pl ... --optimize ; make -s)
>
>
> No I hadn't, but I just did, using those exact commands(no additional
> options to Configure.pl), and had no perceivable performance change
> using any of the parrot variances(default, -P, -j).
There sould be some advantage for non JIT runloops.
> ... Is there a way to
> verify that an optimized build was truly made? Did you expect to see
> much of any difference? GCC issues(running 3.2.3)?
You sould see some -Ox in Makefile's CFLAGS. The 'x' is taken from your
perl5 build options.
I have now checked in slightly modified versions:
- s/mod/cmod/ this is what C has
- made a real subroutine in primes2.pasm
- omitted printing, only summary
- reduced first count to 50000.
parrot -j is only slightly slower then -O3 primes*.c on my system.
$ time imcc -j examples/benchmarks/primes2.pasm
N primes calculated to 10000 is 1230
last is: 9973
real 0m2.219s
$ cc -O3 examples/benchmarks/primes2.c && time ./a.out
real 0m1.936s
$ time imcc -j examples/benchmarks/primes.pasm
real 0m3.902s
$ cc -O3 examples/benchmarks/primes.c && time ./a.out
real 0m3.497s
I don't have the time to give .pl and .py timings here:)
(Athlon 800; i386/linux; all -O3; gcc 2.95.2)
> -Tupshin
Thanks for your contributions.
leo
Leopold Toetsch
> On Tue, Feb 18, 2003 at 01:53:11PM +0100, Leopold Toetsch wrote:
>
>>Did you have an optimized parrot compile?
>>
>>( make progclean ; perl Configure.pl ... --optimize ; make -s)
>>
>
> --optimize may be broken. I tried it with a clean parrot source and
The ellipses should have been placeholders for other options. I
configure mine with:
perl Configure.pl --floatval=double --debugging --optimize
from Makefile:
CFLAGS = -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -g -O3
...
This should get perl5's default optimization for parrot.
leo
Leopold Toetsch
> I think --optimize alone is busted.
Probably my fault, when introducing this option. I did test only with
--debugging.
leo
Nicholas Clark
No no no. You're supposed to test with -march=... -fomit-frame-pointer
-ffancy-math -fuse-lots-of-resources-go-very-fast -fsacrifice-more-goats
-fsummon-cthulu-if-that-helps as root at nice -20, preferably in single
user mode and jumps should be aligned on pentagrams, not 8 byte boundaries.
Definitely not use debugging :-)
Nicholas Clark
Jim Meyer
Benchmarks are idiosyncratic and devious and I thank you for starting a
comparison whose results interest me greatly. =]
On Tue, 2003-02-18 at 10:03, Tupshin Harper wrote:
> [...]and some are in languages I am less then fluent in
> (last touched any flavor of assembly in 1985, and barely touched it
> then), so be kind. I don't believe I'm being too unfair to any of the
> languages, though feel free to tell me otherwise.
I looked at the .pl and .py versions and was struck by the very
dissimilar approaches taken in the two. I translated the GOTO-style
primes.pl to a loop syntax in both Perl and Python which I believe
accurately represents the logic of primes.pasm without resorting to
actual GOTO statements[1].
I'd be very curious as to the runtime of these on the system you used
for the earlier benchmarks. On my box, the retooled Python script takes
only approximately 50% of the time used by the earlier Perl version; the
retooled Perl version runs in roughly 30% the time of its Perl
predecessor.
Thanks again for taking this initiative!
--j
[1] "Go To Statement Considered Harmful", Edsger W. Dijkstra,
Communications of the ACM, Vol. 11, No. 3, March 1968, pp. 147-148; see
http://www.acm.org/classics/oct95/
--
Jim Meyer, Geek at Large jme...@pdi.com
Tupshin Harper
You are correct that the 2 versions I send out earlier are *very*
different. I started from two places, the primes.pasm which I converted
to C and perl versions and a pre-existing primes.py and primes.c that I
converted to primes.pasm. My interest was in comparing parrot to other
things, so I didn't pay any attention in trying to get comparable perl
and parrot ones. Thanks for taking an interest.
-Tupshin
sch...@pobox.com
> example complete with labels and gotos(makes for very disconcerting perl
> code). Startup times and printf times were not significant in any of the
> cases(<5%).
This is, unfortunately, a rigged demo. Its code optimized for Parrot
sch...@pobox.com
>
> ( make progclean ; perl Configure.pl ... --optimize ; make -s)
--optimize may be broken. I tried it with a clean parrot source and
I still get cc -g like so:
perl pmc2c.pl coroutine.pmc
Scanning /usr/local/src/parrot/classes/coroutine.pmc...
cc -g -pipe -pipe -fno-common -no-cpp-precomp -DHAS_TELLDIR_PROTOTYPE -pipe -fno-common -Wall -Wstrict-prototypes -Wmissing-prototypes -Winline -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Waggregate-return -Winline -W -Wno-unused -Wsign-compare -Wformat-nonliteral -Wformat-security -Wpacked -Wpadded -Wdisabled-optimization -I../include -o coroutine.o -c coroutine.c
investigating.
sch...@pobox.com
> > --optimize may be broken. I tried it with a clean parrot source and
>
>
> The ellipses should have been placeholders for other options. I
> configure mine with:
>
> perl Configure.pl --floatval=double --debugging --optimize
>
> from Makefile:
>
> CFLAGS = -I/usr/local/include -D_LARGEFILE_SOURCE
> -D_FILE_OFFSET_BITS=64 -g -O3
> ...
>
> This should get perl5's default optimization for parrot.
That's bizarre. perl Configure.pl --optimize gives:
CFLAGS = -g -pipe -pipe -fno-common -no-cpp-precomp -DHAS_TELLDIR_PROTOTYPE -pi
pe -fno-common -Wall -Wstrict-prototypes -Wmissing-prototypes -Winline -Wshado
w -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Waggregate-return -W
inline -W -Wno-unused -Wsign-compare -Wformat-nonliteral -Wformat-security -Wpac
ked -Wpadded -Wdisabled-optimization -I./include -DHAS_JIT -DPPC -DHAVE_COMPUT
ED_GOTO
no -O. perl Configure.pl --optimize --debugging gives
CFLAGS = -g -pipe -pipe -fno-common -no-cpp-precomp -DHAS_TELLDIR_PROTOTYPE -pi
pe -fno-common -g -Os -Wall -Wstrict-prototypes -Wmissing-prototypes -Winline
-Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Waggregate-re
turn -Winline -W -Wno-unused -Wsign-compare -Wformat-nonliteral -Wformat-securit
y -Wpacked -Wpadded -Wdisabled-optimization -I./include -DHAS_JIT -DPPC -DHAVE
_COMPUTED_GOTO
-Os (which is from Perl 5) and -g. I think --optimize alone is busted.
Simon Glover
You'd be right: the code adding the optimization flags to the list of
compiler flags is only executed if 'debugging' is defined, which is
only the case when the --debugging flag has been used.
The patches below seem to fix this problem and get the --optimize
option working as intended.
One minor niggle that remains is that when I use the --debugging option,
I get two copies of '-g' appended to the ldflags, ie:
LINKFLAGS = -L/usr/local/lib
LDFLAGS = -L/usr/local/lib -g -g
I'm not sure what's happening here, but fortunately it seems to be
harmless.
Simon
--- config/init/debug.pl.old Thu Feb 20 21:56:31 2003
+++ config/init/debug.pl Thu Feb 20 22:01:42 2003
@@ -10,11 +10,11 @@ $description="Enabling debugging...";
sub runstep {
if (Configure::Data->get('debugging')) {
- my($ccflags, $linkflags, $ldflags, $optimize) =
- Configure::Data->get(qw(ccflags linkflags ldflags optimize));
+ my($ccflags, $linkflags, $ldflags) =
+ Configure::Data->get(qw(ccflags linkflags ldflags));
my($cc_debug, $link_debug, $ld_debug) =
Configure::Data->get(qw(cc_debug link_debug ld_debug));
- $ccflags .= " $cc_debug $optimize";
+ $ccflags .= " $cc_debug";
$linkflags .= " $link_debug";
$ldflags .= " $ld_debug";
--- /dev/null Thu Aug 30 16:30:55 2001
+++ config/init/optimize.pl Thu Feb 20 22:01:57 2003
@@ -0,0 +1,26 @@
+package Configure::Step;
+
+use strict;
+use vars qw($description @args);
+use Parrot::Configure::Step;
+
+$description="Enabling optimization...";
+
+@args=();
+
+sub runstep {
+ if (Configure::Data->get('optimize')) {
+ my($ccflags, $optimize) =
+ Configure::Data->get(qw(ccflags optimize));
+ $ccflags .= " $optimize";
+
+ Configure::Data->set(
+ ccflags => $ccflags,
+ );
+ }
+ else {
+ print "(none requested) ";
+ }
+}
+
+1;
--- lib/Parrot/Configure/RunSteps.pm.old Thu Feb 20 21:59:48 2003
+++ lib/Parrot/Configure/RunSteps.pm Thu Feb 20 22:00:03 2003
@@ -10,6 +10,7 @@ use vars qw(@steps);
init/miniparrot.pl
init/hints.pl
init/debug.pl
+ init/optimize.pl
inter/progs.pl
inter/types.pl
inter/ops.pl
--- MANIFEST.old Thu Feb 20 22:13:05 2003
+++ MANIFEST Thu Feb 20 22:13:17 2003
@@ -112,6 +112,7 @@ config/init/hints/os2.pl
config/init/hints/vms.pl
config/init/manifest.pl
config/init/miniparrot.pl
+config/init/optimize.pl
config/inter/exp.pl
config/inter/ops.pl
config/inter/pmc.pl