Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

parrot performance vs.(trivial test) the good, the bad, and the ugly

10 views
Skip to first unread message

Tupshin Harper

unread,
Feb 18, 2003, 7:03:40 AM2/18/03
to perl6-i...@perl.org
In case anyone is interested.....

On a whim I took the primes.pasm example from the parrot examples page
and converted it to both c and perl5, with _interesting_ results.

Timing all three with a max of 100,000 produced the following results:

c -primes.c(lickety split):
real 0m7.710s
user 0m6.790s
sys 0m0.030s

parrot primes.pasm(default options not too impressive):
real 0m33.289s
user 0m29.130s
sys 0m0.080s

parrot primes.pasm(with -P...a bit better)
real 0m21.063s
user 0m20.000s
sys 0m0.050s

parrot primes.pasm (with jit -j option--omfg...i'm impressed)
real 0m12.625s
user 0m11.610s
sys 0m0.040s

perl5 primes.pl(ouch!!!!!!!!!, yes...this is the same algorithm)
real 6m53.454s
user 6m33.490s
sys 0m0.990s

FYI...all three used the identical algorithm taken from the primes.pasm
example complete with labels and gotos(makes for very disconcerting perl
code). Startup times and printf times were not significant in any of the
cases(<5%).

Further curious because of the python challenge, I took a different
(slower) primes algorithm(trying to be fair to python since it doesn't
have gotos), That someone had already written C and python versions of,
and ported it to parrot....drum roll please(only 10,000 primes due to
slower algorithm):

c - primes2.c
real 0m1.147s
user 0m1.050s
sys 0m0.000s

parrot primes2.pasm(default)
real 0m4.868s
user 0m4.700s
sys 0m0.020s

parrot primes2.pasm(-P)
real 0m3.058s
user 0m2.920s
sys 0m0.000s

parrot primes2.pasm(-J)
real 0m1.825s
user 0m1.700s
sys 0m0.000s

python primes2.py
real 0m25.922s
user 0m24.900s
sys 0m0.050s


So at least on simple looping benchmarks like this, parrot is much
closer to the performance of C than it is to perl or python...I liiiike
it (doing my best Tony the Tiger impersonation here). Congrats
guys...it's pretty impressive.

Testing platform is linux debian (sid) on an Athlon 2100, gcc 3.2.3,
perl 5.8.0, python 2.2.2, and parrot CVS head as of today.

-Tupshin

Code available if anybody cares.

Leopold Toetsch

unread,
Feb 18, 2003, 7:53:11 AM2/18/03
to Tupshin Harper, perl6-i...@perl.org
Tupshin Harper wrote:

> In case anyone is interested.....


Always :)


Did you have an optimized parrot compile?

( make progclean ; perl Configure.pl ... --optimize ; make -s)


> -Tupshin
>
> Code available if anybody cares.


Yes please.

TIA,
leo


Tupshin Harper

unread,
Feb 18, 2003, 1:03:52 PM2/18/03
to Leopold Toetsch, perl6-i...@perl.org
Leopold Toetsch wrote:

> Did you have an optimized parrot compile?
>
> ( make progclean ; perl Configure.pl ... --optimize ; make -s)

No I hadn't, but I just did, using those exact commands(no additional
options to Configure.pl), and had no perceivable performance change
using any of the parrot variances(default, -P, -j). Is there a way to
verify that an optimized build was truly made? Did you expect to see
much of any difference? GCC issues(running 3.2.3)?

>> -Tupshin
>>
>> Code available if anybody cares.
>
>
>
> Yes please.

Alright...all versions are attached (sorry for spamming the list, but
others might be interested). Some of the code is ugly (done around 3:00
in the morning), and some are in languages I am less then fluent in
(last touched any flavor of assembly in 1985, and barely touched it
then), so be kind. I don't believe I'm being too unfair to any of the
languages, though feel free to tell me otherwise.

The python and c versions of prime2 come from a tutorial on optimizing
python by writing extensions in C
http://kortis.to/radix/python_ext/
Modified slighly to behave more analogously to the first test, and some
compilation bugs fixed.

>
> TIA,
> leo
>
WATF
(welcome after the fact)

-Tupshin

primes.c
primes.pl
primes.pasm
primes2.c
primes2.py
primes2.pasm

Tupshin Harper

unread,
Feb 18, 2003, 2:24:53 PM2/18/03
to sch...@pobox.com, perl6-i...@perl.org
sch...@pobox.com wrote:

>On Tue, Feb 18, 2003 at 04:03:40AM -0800, Tupshin Harper wrote:
>
>
>>FYI...all three used the identical algorithm taken from the primes.pasm
>>example complete with labels and gotos(makes for very disconcerting perl
>>code). Startup times and printf times were not significant in any of the
>>cases(<5%).
>>
>>
>

>This is, unfortunately, a rigged demo. Its code optimized for Parrot
>and then mechanically translated into Perl and C. The algorithm would
>have been the same if it, for example, used a loop in Perl instead of
>gotos.
>
>I don't know that its all that interesting to show that goto is slow in
>Perl. OTOH, goto should be fast in C, no?
>
>I'm going to tinker with those benchmarks again.
>
>
Agreed...the second example is probably less rigged, since the
mechanical translation was to pasm(and don't know enough to optimize the
pasm ;-)

-Tupshin

Leopold Toetsch

unread,
Feb 18, 2003, 3:16:11 PM2/18/03
to Tupshin Harper, perl6-i...@perl.org
Tupshin Harper wrote:

> Leopold Toetsch wrote:
>
>> Did you have an optimized parrot compile?
>>
>> ( make progclean ; perl Configure.pl ... --optimize ; make -s)
>
>
> No I hadn't, but I just did, using those exact commands(no additional
> options to Configure.pl), and had no perceivable performance change
> using any of the parrot variances(default, -P, -j).


There sould be some advantage for non JIT runloops.

> ... Is there a way to

> verify that an optimized build was truly made? Did you expect to see
> much of any difference? GCC issues(running 3.2.3)?


You sould see some -Ox in Makefile's CFLAGS. The 'x' is taken from your
perl5 build options.

I have now checked in slightly modified versions:
- s/mod/cmod/ this is what C has
- made a real subroutine in primes2.pasm
- omitted printing, only summary
- reduced first count to 50000.

parrot -j is only slightly slower then -O3 primes*.c on my system.

$ time imcc -j examples/benchmarks/primes2.pasm
N primes calculated to 10000 is 1230
last is: 9973

real 0m2.219s

$ cc -O3 examples/benchmarks/primes2.c && time ./a.out

real 0m1.936s


$ time imcc -j examples/benchmarks/primes.pasm

real 0m3.902s


$ cc -O3 examples/benchmarks/primes.c && time ./a.out


real 0m3.497s


I don't have the time to give .pl and .py timings here:)

(Athlon 800; i386/linux; all -O3; gcc 2.95.2)

> -Tupshin


Thanks for your contributions.


leo


Leopold Toetsch

unread,
Feb 18, 2003, 3:27:17 PM2/18/03
to sch...@pobox.com, Tupshin Harper, perl6-i...@perl.org
sch...@pobox.com wrote:

> On Tue, Feb 18, 2003 at 01:53:11PM +0100, Leopold Toetsch wrote:
>
>>Did you have an optimized parrot compile?
>>
>>( make progclean ; perl Configure.pl ... --optimize ; make -s)
>>
>

> --optimize may be broken. I tried it with a clean parrot source and


The ellipses should have been placeholders for other options. I
configure mine with:

perl Configure.pl --floatval=double --debugging --optimize

from Makefile:

CFLAGS = -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -g -O3
...

This should get perl5's default optimization for parrot.


leo

Leopold Toetsch

unread,
Feb 18, 2003, 4:58:59 PM2/18/03
to sch...@pobox.com, Tupshin Harper, perl6-i...@perl.org
sch...@pobox.com wrote:

> I think --optimize alone is busted.


Probably my fault, when introducing this option. I did test only with
--debugging.

leo


Nicholas Clark

unread,
Feb 18, 2003, 5:17:35 PM2/18/03
to Leopold Toetsch, sch...@pobox.com, Tupshin Harper, perl6-i...@perl.org

No no no. You're supposed to test with -march=... -fomit-frame-pointer
-ffancy-math -fuse-lots-of-resources-go-very-fast -fsacrifice-more-goats
-fsummon-cthulu-if-that-helps as root at nice -20, preferably in single
user mode and jumps should be aligned on pentagrams, not 8 byte boundaries.

Definitely not use debugging :-)

Nicholas Clark

Jim Meyer

unread,
Feb 18, 2003, 9:13:05 PM2/18/03
to Tupshin Harper, Leopold Toetsch, perl6-i...@perl.org
Hello!

Benchmarks are idiosyncratic and devious and I thank you for starting a
comparison whose results interest me greatly. =]

On Tue, 2003-02-18 at 10:03, Tupshin Harper wrote:
> [...]and some are in languages I am less then fluent in

> (last touched any flavor of assembly in 1985, and barely touched it
> then), so be kind. I don't believe I'm being too unfair to any of the
> languages, though feel free to tell me otherwise.

I looked at the .pl and .py versions and was struck by the very
dissimilar approaches taken in the two. I translated the GOTO-style
primes.pl to a loop syntax in both Perl and Python which I believe
accurately represents the logic of primes.pasm without resorting to
actual GOTO statements[1].

I'd be very curious as to the runtime of these on the system you used
for the earlier benchmarks. On my box, the retooled Python script takes
only approximately 50% of the time used by the earlier Perl version; the
retooled Perl version runs in roughly 30% the time of its Perl
predecessor.

Thanks again for taking this initiative!

--j

[1] "Go To Statement Considered Harmful", Edsger W. Dijkstra,
Communications of the ACM, Vol. 11, No. 3, March 1968, pp. 147-148; see
http://www.acm.org/classics/oct95/
--
Jim Meyer, Geek at Large jme...@pdi.com

primes_loop.pl
primes_loop.py

Tupshin Harper

unread,
Feb 18, 2003, 9:26:09 PM2/18/03
to Jim Meyer, perl6-i...@perl.org
On my system, the perl takes 2.24 second and the python takes 3.76 seconds.
You are correct that the 2 versions I send out earlier are *very*
different. I started from two places, the primes.pasm which I converted
to C and perl versions and a pre-existing primes.py and primes.c that I
converted to primes.pasm. My interest was in comparing parrot to other
things, so I didn't pay any attention in trying to get comparable perl
and parrot ones. Thanks for taking an interest.

-Tupshin

sch...@pobox.com

unread,
Feb 18, 2003, 2:13:29 PM2/18/03
to Tupshin Harper, perl6-i...@perl.org
On Tue, Feb 18, 2003 at 04:03:40AM -0800, Tupshin Harper wrote:
> FYI...all three used the identical algorithm taken from the primes.pasm
> example complete with labels and gotos(makes for very disconcerting perl
> code). Startup times and printf times were not significant in any of the
> cases(<5%).

This is, unfortunately, a rigged demo. Its code optimized for Parrot

sch...@pobox.com

unread,
Feb 18, 2003, 2:19:39 PM2/18/03
to Leopold Toetsch, Tupshin Harper, perl6-i...@perl.org
On Tue, Feb 18, 2003 at 01:53:11PM +0100, Leopold Toetsch wrote:
> Did you have an optimized parrot compile?
>
> ( make progclean ; perl Configure.pl ... --optimize ; make -s)

--optimize may be broken. I tried it with a clean parrot source and
I still get cc -g like so:

perl pmc2c.pl coroutine.pmc
Scanning /usr/local/src/parrot/classes/coroutine.pmc...
cc -g -pipe -pipe -fno-common -no-cpp-precomp -DHAS_TELLDIR_PROTOTYPE -pipe -fno-common -Wall -Wstrict-prototypes -Wmissing-prototypes -Winline -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Waggregate-return -Winline -W -Wno-unused -Wsign-compare -Wformat-nonliteral -Wformat-security -Wpacked -Wpadded -Wdisabled-optimization -I../include -o coroutine.o -c coroutine.c

investigating.

sch...@pobox.com

unread,
Feb 18, 2003, 4:39:45 PM2/18/03
to Leopold Toetsch, Tupshin Harper, perl6-i...@perl.org
On Tue, Feb 18, 2003 at 09:27:17PM +0100, Leopold Toetsch wrote:
> > --optimize may be broken. I tried it with a clean parrot source and
>
>
> The ellipses should have been placeholders for other options. I
> configure mine with:
>
> perl Configure.pl --floatval=double --debugging --optimize
>
> from Makefile:
>
> CFLAGS = -I/usr/local/include -D_LARGEFILE_SOURCE
> -D_FILE_OFFSET_BITS=64 -g -O3
> ...
>
> This should get perl5's default optimization for parrot.

That's bizarre. perl Configure.pl --optimize gives:

CFLAGS = -g -pipe -pipe -fno-common -no-cpp-precomp -DHAS_TELLDIR_PROTOTYPE -pi


pe -fno-common -Wall -Wstrict-prototypes -Wmissing-prototypes -Winline -Wshado
w -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Waggregate-return -W
inline -W -Wno-unused -Wsign-compare -Wformat-nonliteral -Wformat-security -Wpac

ked -Wpadded -Wdisabled-optimization -I./include -DHAS_JIT -DPPC -DHAVE_COMPUT
ED_GOTO

no -O. perl Configure.pl --optimize --debugging gives

CFLAGS = -g -pipe -pipe -fno-common -no-cpp-precomp -DHAS_TELLDIR_PROTOTYPE -pi
pe -fno-common -g -Os -Wall -Wstrict-prototypes -Wmissing-prototypes -Winline

-Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Waggregate-re
turn -Winline -W -Wno-unused -Wsign-compare -Wformat-nonliteral -Wformat-securit

y -Wpacked -Wpadded -Wdisabled-optimization -I./include -DHAS_JIT -DPPC -DHAVE
_COMPUTED_GOTO

-Os (which is from Perl 5) and -g. I think --optimize alone is busted.

Simon Glover

unread,
Feb 20, 2003, 10:28:00 PM2/20/03
to sch...@pobox.com, perl6-i...@perl.org

On Tue, 18 Feb 2003 sch...@pobox.com wrote:
>
> I think --optimize alone is busted.
>

You'd be right: the code adding the optimization flags to the list of
compiler flags is only executed if 'debugging' is defined, which is
only the case when the --debugging flag has been used.

The patches below seem to fix this problem and get the --optimize
option working as intended.

One minor niggle that remains is that when I use the --debugging option,
I get two copies of '-g' appended to the ldflags, ie:

LINKFLAGS = -L/usr/local/lib
LDFLAGS = -L/usr/local/lib -g -g

I'm not sure what's happening here, but fortunately it seems to be
harmless.

Simon

--- config/init/debug.pl.old Thu Feb 20 21:56:31 2003
+++ config/init/debug.pl Thu Feb 20 22:01:42 2003
@@ -10,11 +10,11 @@ $description="Enabling debugging...";

sub runstep {
if (Configure::Data->get('debugging')) {
- my($ccflags, $linkflags, $ldflags, $optimize) =
- Configure::Data->get(qw(ccflags linkflags ldflags optimize));
+ my($ccflags, $linkflags, $ldflags) =
+ Configure::Data->get(qw(ccflags linkflags ldflags));
my($cc_debug, $link_debug, $ld_debug) =
Configure::Data->get(qw(cc_debug link_debug ld_debug));
- $ccflags .= " $cc_debug $optimize";
+ $ccflags .= " $cc_debug";
$linkflags .= " $link_debug";
$ldflags .= " $ld_debug";

--- /dev/null Thu Aug 30 16:30:55 2001
+++ config/init/optimize.pl Thu Feb 20 22:01:57 2003
@@ -0,0 +1,26 @@
+package Configure::Step;
+
+use strict;
+use vars qw($description @args);
+use Parrot::Configure::Step;
+
+$description="Enabling optimization...";
+
+@args=();
+
+sub runstep {
+ if (Configure::Data->get('optimize')) {
+ my($ccflags, $optimize) =
+ Configure::Data->get(qw(ccflags optimize));
+ $ccflags .= " $optimize";
+
+ Configure::Data->set(
+ ccflags => $ccflags,
+ );
+ }
+ else {
+ print "(none requested) ";
+ }
+}
+
+1;

--- lib/Parrot/Configure/RunSteps.pm.old Thu Feb 20 21:59:48 2003
+++ lib/Parrot/Configure/RunSteps.pm Thu Feb 20 22:00:03 2003
@@ -10,6 +10,7 @@ use vars qw(@steps);
init/miniparrot.pl
init/hints.pl
init/debug.pl
+ init/optimize.pl
inter/progs.pl
inter/types.pl
inter/ops.pl

--- MANIFEST.old Thu Feb 20 22:13:05 2003
+++ MANIFEST Thu Feb 20 22:13:17 2003
@@ -112,6 +112,7 @@ config/init/hints/os2.pl
config/init/hints/vms.pl
config/init/manifest.pl
config/init/miniparrot.pl
+config/init/optimize.pl
config/inter/exp.pl
config/inter/ops.pl
config/inter/pmc.pl

0 new messages