A Perl Task - Benchmarking
Leopold Toetsch
following idea:
examples/benchmarks/* has a bunch of programs e.g.
oo1.pasm
oo1.pl
oo1.py
stress.pasm
stress.pl
...
Now like perlbench is able to compare run times of different perl
versions, the goal of this task is to provide a script that compares
different interpreters and finally spits out:
Parrot-j Parrot-C Perl Python Ruby
oo1 100% 103% 75% 50% -
mops 100% 200% 40000%
stress ... - -
or some such.
To simplify the task, we could of course move used tests into a separate
directory. Unavailable interpreters (or missing scripts for that
language) are just skipped.
Any takers?
leo
Sebastian Riedel
Attached is a quick'n dirty parrotbench, instead of a complicated
test harness it uses bash to make time measurements, so that
new languages are very simple to add.
Currently it's just a proof of concept but if you like it i will make
a better version with pretty printing, extended reports and stuff.
Here's an example run:
parrot perl
python ruby
addit 8.469 7.379
- -
arriter - 1.657
- -
bench_newp 1.827 - -
-
fib -
0.594 - -
freeze 0.783 1.65
- -
gc_alloc_new 0.191 - -
-
gc_alloc_reuse 4.068 - -
-
gc_generations 6.363 - -
-
gc_header_new 1.168 - -
-
gc_header_reuse 5.772 - -
-
gc_waves_headers 1.302 - - -
gc_waves_sizeable_data 1.074 - - -
gc_waves_sizeable_headers 3.702 - - -
oo1 3.571 1.189
0.689 -
primes 27.991 383.851 -
-
primes2 17.325 -
44.379 -
primes2_p 29.753 -
- -
prop 0.14 -
- -
shared_ref 0.552 11.563 -
-
stress 1.988 0.905
- -
stress1 27.539 17.312 -
-
stress2 3.908 3.440
- -
stress3 19.050 -
- -
utf8 0.13 -
- -
vpm - 40.057
- -
Cheers,
Sebastian
Leopold Toetsch
^^^^
objective-ook? - SCNR
> Attached is a quick'n dirty parrotbench, instead of a complicated
> test harness it uses bash to make time measurements, so that
> new languages are very simple to add.
bash isn't really available on all systems, so it should better use one
of the time functions.
> Currently it's just a proof of concept but if you like it i will make
> a better version with pretty printing, extended reports and stuff.
Good. Please have a look at lib/Parrot/Test.pm:_run_command and
Config.pm for executable names. Maybe a config file could simplify the
task (user can put in executable names once).
> Here's an example run:
$ perl tools/dev/parrotbench.pl -regex '(oo|str|mops).*' \
-parrot='./parrot -j' -perl=`which perl` -python=`which python` \
-ruby=`which ruby`
parrot perl python ruby
mops 0.260 96.140 9.830 9.860
oo1 1.700 0.820 0.510 -
oo2 8.410 4.60 2.400 -
stress 0.980 0.640 - -
stress1 13.970 12.400 - -
stress2 1.670 2.450 - -
stress3 10.540 - - -
(Python and Ruby mops are running 1/10th of loops - files linked into
examples/benchmarks)
Nice. Yes please.
Any one out there, who speaks ruby and can translate tests, for which
we have a '.pl' file?.
Thanks,
leo
Sebastian Riedel
>Sebastian Riedel <s...@oook.de> wrote:
> ^^^^
> objective-ook? - SCNR
>
:)
>
>
>>Attached is a quick'n dirty parrotbench, instead of a complicated
>>test harness it uses bash to make time measurements, so that
>>new languages are very simple to add.
>>
>>
>
>bash isn't really available on all systems, so it should better use one
>of the time functions.
>
>
Attached is a new version using times()
>
>
>>Currently it's just a proof of concept but if you like it i will make
>>a better version with pretty printing, extended reports and stuff.
>>
>>
>
>Good. Please have a look at lib/Parrot/Test.pm:_run_command and
>Config.pm for executable names. Maybe a config file could simplify the
>task (user can put in executable names once).
>
>
Working on it
>
>
>>Here's an example run:
>>
>>
>
>$ perl tools/dev/parrotbench.pl -regex '(oo|str|mops).*' \
> -parrot='./parrot -j' -perl=`which perl` -python=`which python` \
> -ruby=`which ruby`
> parrot perl python ruby
>mops 0.260 96.140 9.830 9.860
>oo1 1.700 0.820 0.510 -
>oo2 8.410 4.60 2.400 -
>stress 0.980 0.640 - -
>stress1 13.970 12.400 - -
>stress2 1.670 2.450 - -
>stress3 10.540 - - -
>
>(Python and Ruby mops are running 1/10th of loops - files linked into
>examples/benchmarks)
>
>Nice. Yes please.
>
>Any one out there, who speaks ruby and can translate tests, for which
>we have a '.pl' file?.
>
>Thanks,
>leo
>
>
>
Cheers,
Sebastian
Sebastian Riedel
> Leopold Toetsch wrote:
>
>> Sebastian Riedel <s...@oook.de> wrote:
>> ^^^^
>> objective-ook? - SCNR
>>
> :)
>
>>
>>
>>> Attached is a quick'n dirty parrotbench, instead of a complicated
>>> test harness it uses bash to make time measurements, so that
>>> new languages are very simple to add.
>>>
>>
>>
>> bash isn't really available on all systems, so it should better use one
>> of the time functions.
>>
>>
> Attached is a new version using times()
>
>>
>>
>>> Currently it's just a proof of concept but if you like it i will make
>>> a better version with pretty printing, extended reports and stuff.
>>>
>>
>>
>> Good. Please have a look at lib/Parrot/Test.pm:_run_command and
>> Config.pm for executable names. Maybe a config file could simplify the
>> task (user can put in executable names once).
>>
>>
> Working on it
The attached version should do most of the things you wanted.
sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -regex oo -conf
../parrotbench.conf
parrot perl python ruby
oo1 100% 39% 23% -
oo2 100% 40% 22% -
I studied the config system, and now i wonder if it would make sense
to write a configure step to probe for enemies, or is that overkill?
>
>>
>>
>>> Here's an example run:
>>>
>>
>>
>> $ perl tools/dev/parrotbench.pl -regex '(oo|str|mops).*' \
>> -parrot='./parrot -j' -perl=`which perl` -python=`which python` \
>> -ruby=`which ruby`
>> parrot perl python ruby
>> mops 0.260 96.140 9.830 9.860
>> oo1 1.700 0.820 0.510 -
>> oo2 8.410 4.60 2.400 -
>> stress 0.980 0.640 - -
>> stress1 13.970 12.400 - -
>> stress2 1.670 2.450 - -
>> stress3 10.540 - - -
>>
>> (Python and Ruby mops are running 1/10th of loops - files linked into
>> examples/benchmarks)
>>
>> Nice. Yes please.
>>
>> Any one out there, who speaks ruby and can translate tests, for which
>> we have a '.pl' file?.
>>
>> Thanks,
>> leo
>>
>>
>>
> Cheers,
> Sebastian
Cheers,
Sebastian
Brent "Dax" Royal-Gordon
> I studied the config system, and now i wonder if it would make sense
> to write a configure step to probe for enemies, or is that overkill?
Overkill doesn't even begin to describe it.
It ought not to be too difficult to look for "enemies" at the beginning
of your little script. I'd imagine that you could reasonably stipulate
that the interpreters have to be in your PATH or specified on the
command line somehow. (And Perl's location should be in Parrot::Config
anyway.)
--
Brent "Dax" Royal-Gordon <br...@brentdax.com>
Perl and Parrot hacker
Oceania has always been at war with Eastasia.
Leopold Toetsch
> Sebastian Riedel wrote:
> The attached version should do most of the things you wanted.
> sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -regex oo -conf
> ../parrotbench.conf
> parrot perl python ruby
> oo1 100% 39% 23% -
> oo2 100% 40% 22% -
Good. Just some more notes:
I'd like the program to take a config like this:
parrot-C: ./parrot -C
parrot-j: ./parrot -j
perl: /usr/bin/perl
perl-58_th: /opt/perl-th/bin/perl
...
scheme: /usr/bin/rep: .scm
that is to be able to compare more different programs. Timings should be
relative to the first given program.
> I studied the config system, and now i wonder if it would make sense
> to write a configure step to probe for enemies, or is that overkill?
^^^^^^^
Please ...
No config step necessary. The -conf option is good enough.
leo
Ibotty
> Any one out there, who speaks ruby and can translate tests, for which
> we have a '.pl' file?.
just catched up mail and thought a bit practice might not be bad...
i intentionally did not wrote them optimal (performance-wise), but rather
elegantly (sort of ;).
thus this is only a performance of comparision of the language
implemenentation not the quality of the VM.
i thought this was kind of wanted (because we may have a ruby frontend one
day).
so have fun...
but dont be overly enthusiastic, ruby is kind of slow... (especially with
blocks)
~ibotty
Leopold Toetsch
> The attached version should do most of the things you wanted.
I've now added parrotbench.pl (slightly modifed) to the CVS tree.
thanks,
leo
Leopold Toetsch
> just catched up mail and thought a bit practice might not be bad...
> i intentionally did not wrote them optimal (performance-wise), but rather
> elegantly (sort of ;).
Yeah, that's always the problem. Where applicable, we could add a set of
optimized versions.
> so have fun...
Thanks, appplied
> # does the perl variant count as oo?
They say so :)
leo
Sebastian Riedel
>A very basic newbeish question..
>
>
>
>>>>sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -regex oo -conf
>>>>../parrotbench.conf
>>>> parrot perl python ruby
>>>>oo1 100% 39% 23% -
>>>>oo2 100% 40% 22% -
>>>>
>>>>
>>>>
>
>Are bigger numbers more desirable (as they would be if they mean number of
>times code run per second) or lower numbers are more desirable (as they
>would be if they mean time taken per run)? So what do these numbers mean?
>
>
All numbers are relative to the first one, lower is better.
The -t switch explains it:
sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -b oo -t
parrot(pasm) parrot(imc) python(py) perl(pl)
oo1 3.030s - 0.690s 1.200s
oo2 15.020s - 3.310s 6.020s
Yes, parrot looks not so good at the moment, maybe Dan will get his
cream pie at OSCON. :)
>--Abhijit
>
>
>
>
Cheers,
Sebastian Riedel
Abhijit A. Mahabal
> >
> >>sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -regex oo -conf
> >>../parrotbench.conf
> >> parrot perl python ruby
> >>oo1 100% 39% 23% -
> >>oo2 100% 40% 22% -
> >>
Are bigger numbers more desirable (as they would be if they mean number of
times code run per second) or lower numbers are more desirable (as they
would be if they mean time taken per run)? So what do these numbers mean?
--Abhijit
Dan Sugalski
>Abhijit A. Mahabal wrote:
>
>>A very basic newbeish question..
>>
>>
>>>>>sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -regex oo -conf
>>>>>../parrotbench.conf
>>>>> parrot perl python ruby
>>>>>oo1 100% 39% 23% -
>>>>>oo2 100% 40% 22% -
>>>>>
>>>>>
>>>>>
>>
>>Are bigger numbers more desirable (as they would be if they mean number of
>>times code run per second) or lower numbers are more desirable (as they
>>would be if they mean time taken per run)? So what do these numbers mean?
>>
>All numbers are relative to the first one, lower is better.
Can we add a caption to the output? Otherwise I'll end up forgetting.
>The -t switch explains it:
>
>sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -b oo -t
> parrot(pasm) parrot(imc) python(py) perl(pl)
>oo1 3.030s - 0.690s 1.200s
>oo2 15.020s - 3.310s 6.020s
>
>Yes, parrot looks not so good at the moment, maybe Dan will get his
>cream pie at OSCON. :)
Well... we'll see about that. :) There are a few tricks yet to be
played here, and this is definitely a first-cut of the object
implementation. There's an awful lot of unnecessary indirection at
the moment, which needs fixing up.
We also need to implement a method cache, which'll probably increase
parrot's speed just a touch, if every single other late-binding OO
language is anything to judge by. :)
--
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Sebastian Riedel
> At 10:10 PM +0100 3/8/04, Sebastian Riedel wrote:
>
>> Abhijit A. Mahabal wrote:
>>
>>> A very basic newbeish question..
>>>
>>>
>>>>>> sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -regex oo -conf
>>>>>> ../parrotbench.conf
>>>>>> parrot perl python ruby
>>>>>> oo1 100% 39% 23% -
>>>>>> oo2 100% 40% 22% -
>>>>>>
>>>>>>
>>>>>
>>>
>>> Are bigger numbers more desirable (as they would be if they mean
>>> number of
>>> times code run per second) or lower numbers are more desirable (as they
>>> would be if they mean time taken per run)? So what do these numbers
>>> mean?
>>>
>> All numbers are relative to the first one, lower is better.
>
>
> Can we add a caption to the output? Otherwise I'll end up forgetting.
>
Will be added in the next version.
>> The -t switch explains it:
>>
>> sri@odyssey:~/parrot$ tools/dev/parrotbench.pl -b oo -t
>> parrot(pasm) parrot(imc) python(py) perl(pl)
>> oo1 3.030s - 0.690s 1.200s
>> oo2 15.020s - 3.310s 6.020s
>>
>> Yes, parrot looks not so good at the moment, maybe Dan will get his
>> cream pie at OSCON. :)
>
>
> Well... we'll see about that. :) There are a few tricks yet to be
> played here, and this is definitely a first-cut of the object
> implementation. There's an awful lot of unnecessary indirection at the
> moment, which needs fixing up.
>
> We also need to implement a method cache, which'll probably increase
> parrot's speed just a touch, if every single other late-binding OO
> language is anything to judge by. :)
Seems Larry knows well how to motivate you. :)
Cheers,
Sebastian
Dan Sugalski
>Dan Sugalski wrote:
>
>>At 10:10 PM +0100 3/8/04, Sebastian Riedel wrote:
>>
>>>Abhijit A. Mahabal wrote:
>>
>>Can we add a caption to the output? Otherwise I'll end up forgetting.
>>
>Will be added in the next version.
Cool, thanks.
>>
>>We also need to implement a method cache, which'll probably
>>increase parrot's speed just a touch, if every single other
>>late-binding OO language is anything to judge by. :)
>
>Seems Larry knows well how to motivate you. :)
Heh. Turns out not to be the case--this one's pure ego and righteous
indignation. (Never underestimate the motivating power of righteous
indignation! :)
Leopold Toetsch
> The attached patch adds this, conf files now look so:
> parrot: /home/sri/parrot/parrot: .pasm .imc
Good.
> ruby: /usr/bin/ruby: .rb
> python: /usr/bin/python: .py
> python-C: /usr/bin/python -C: .py
^^^^^
That's probably parrot-C, anyway:
> Output looks so (could be prettier):
Yes ;)
> parrot(pasm) parrot(imc) python(py) perl(pl)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
These should just be one column: Some benchmarks are written in PASM some
in PIR, but they are totally equivalent and run through one parrot.
OTOH:
parrot-j: ./parrot -j: .imc .pasm
parrot-C: ./parrot -C: .imc .pasm
Should give two columns, one for 'parrot -j' and one for 'parrot -C' -
these are two different run loops with different timings.
So a line in the config specifies one program (with possibly multiple
file extensions) and is one column of timing report.
Also when comparing (percentage output) a benchmark shouldn't be run, if
there is only one program to run it.
[ please provide patches againt parrot root's directory ]
Applied and thanks,
leo
Sebastian Riedel
>Sebastian Riedel <s...@oook.de> wrote:
>
>
>
>>The attached patch adds this, conf files now look so:
>>
>>
>
>
>
>>parrot: /home/sri/parrot/parrot: .pasm .imc
>>
>>
>
>Good.
>
>
>
>>ruby: /usr/bin/ruby: .rb
>>python: /usr/bin/python: .py
>>python-C: /usr/bin/python -C: .py
>>
>>
> ^^^^^
>That's probably parrot-C, anyway:
>
>
>
Just an example. ;)
>>Output looks so (could be prettier):
>>
>>
>
>Yes ;)
>
>
>
>> parrot(pasm) parrot(imc) python(py) perl(pl)
>>
>>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
>These should just be one column: Some benchmarks are written in PASM some
>in PIR, but they are totally equivalent and run through one parrot.
>
>OTOH:
>
>parrot-j: ./parrot -j: .imc .pasm
>parrot-C: ./parrot -C: .imc .pasm
>
>Should give two columns, one for 'parrot -j' and one for 'parrot -C' -
>these are two different run loops with different timings.
>So a line in the config specifies one program (with possibly multiple
>file extensions) and is one column of timing report.
>
>
>
>Also when comparing (percentage output) a benchmark shouldn't be run, if
>there is only one program to run it.
>
>
Attached patch should fix that all.
>[ please provide patches againt parrot root's directory ]
>
>
Roger that.
>Applied and thanks,
>leo
>
>
>
Sebastian
Leopold Toetsch
Thanks, applied - just updated the sample conf.
leo