New shootout

[Antonio Cangiano]

Hi guys,

after a lot of work, I'm happy to announce the new version of The Great Ruby Shootout:

http://antoniocangiano.com/2008/12/09/the-great-ruby-shootout-december-2008/

Please feel free to send the link to your own Ruby VM related mailing lists.

Thanks,

Antonio

--
http://antoniocangiano.com - Zen and the Art of Programming
http://math-blog.com - Mathematics is wonderful!
http://stacktrace.it - Aperiodico di resistenza informatica
Currently writing "Ruby on Rails for Microsoft Developers" for Wrox.

[SASADA Koichi]

Antonio Cangiano wrote::

> Hi guys,
>
> after a lot of work, I'm happy to announce the new version of The Great Ruby
> Shootout:
> http://antoniocangiano.com/2008/12/09/the-great-ruby-shootout-december-2008/
>
> Please feel free to send the link to your own Ruby VM related mailing lists.

Great work!

Could you try it on ruby 1.9.1 branch? There are some difference
between trunk and 1.9.1 branch.

Regards,
Koichi

[Antonio Cangiano]

On Tue, Dec 9, 2008 at 9:33 AM, SASADA Koichi <sas...@gmail.com> wrote:
> Great work!

Ehm, you are the one who did great work. :)

> Could you try it on ruby 1.9.1 branch? There are some difference
> between trunk and 1.9.1 branch.

I don't plan to update the post with Ruby 1.9.1's branch results, but
I'd be happy to do a test run and send you the results.

Cheers,

[rogerdpack]

Nice job! Finally a somewhat comprehensive metric :)

Couple of thoughts on it...
it might be interesting to let spectators submit fine tuned versions
of their implementations:
ex: jruby guys with some "funky" command line options/warm-up stuffs

REE apparently runs faster with a tuned GC [1]

And if I remember correctly, tweaking 1.9's GC also benefits from
tweaking [2].
So it might change numbers. It's a little hard-core, but does help,
and is available.
Thanks!
-=R

[1] http://blog.erichsen.net/2008/12/05/how-ree-and-gc-tuning-reduced-spec-suite-runtime-to-one-third/
[2] I believe in my own experimenting changing [gc.c]
HEAP_MIN_SIZE to 100000 instead of 10000
sped things up. There's also some other experimental GC's
http://www.nabble.com/GC-patch-for-1.9-to-run-in-a-separate-thread-td20082948.html
I'll post back with 1.9 results if I get around to them sometime :)

On Dec 9, 7:18 am, "Antonio Cangiano" <acangi...@gmail.com> wrote:
> Hi guys,
>
> after a lot of work, I'm happy to announce the new version of The Great Ruby

> Shootout:http://antoniocangiano.com/2008/12/09/the-great-ruby-shootout-decembe...

>
> Please feel free to send the link to your own Ruby VM related mailing lists.
>
> Thanks,
> Antonio

> --http://antoniocangiano.com- Zen and the Art of Programminghttp://math-blog.com- Mathematics is wonderful!http://stacktrace.it- Aperiodico di resistenza informatica

[rogerdpack]

An option that might be nice sometime for the suite would be to do
each test until the "k best" are within a certain percentage of each
other, then accept the lowest.
like until the top 3 are within 3% of each other or something.
That would give me some added confidence to myself when running them
[was this just OS load?]
Thanks for the suite.
-=R

[rogerdpack]

> And if I remember correctly, tweaking 1.9's GC also benefits from
> tweaking [2].

example of what tweaking the gc can do [Cangiano benchmark]
Ruby 1.9 normal:

real-world/bm_hilbert_matrix.rb,
4.0054624080658,4.09904861450195,4.052255511283875,0.046793103218079,50
real-world/bm_hilbert_matrix.rb,
8.86907505989075,12.8570592403412,10.863067150115967,1.993992090225220,60

ruby 1.9 trunk with GC patch [1]

real-world/bm_hilbert_matrix.rb,
2.87246918678284,2.8920271396637,2.882248163223267,0.009778976440430,50
real-world/bm_hilbert_matrix.rb,
6.63441443443298,6.49161219596863,6.563013315200806,0.071401119232178,60

With GC.disable

real-world/bm_hilbert_matrix.rb,
2.61896514892578,2.60238766670227,2.610676407814026,0.008288741111755,50
real-world/bm_hilbert_matrix.rb,
5.82226586341858,5.81672692298889,5.819496393203735,0.002769470214844,60

So there is still room for improvement [i.e. with something like [0]],
but some benefit is gained with tuning.

Thanks!
-=R

[0] http://www.nabble.com/GC-patch-for-1.9-to-run-in-a-separate-thread-td20082948.html
[1]
Index: gc.c
===================================================================
--- gc.c (revision 20731)
+++ gc.c (working copy)
@@ -75,7 +75,7 @@
#endif /* __GNUC__ */

#ifndef GC_MALLOC_LIMIT
-#define GC_MALLOC_LIMIT 8000000
+#define GC_MALLOC_LIMIT 80000000
#endif

#define nomem_error GET_VM()->special_exceptions[ruby_error_nomemory]
@@ -281,7 +281,7 @@
int limit;
};

-#define HEAP_MIN_SLOTS 10000
+#define HEAP_MIN_SLOTS 100000
#define FREE_MIN 4096

struct gc_list {

[Charles Oliver Nutter]

I certainly have no problem with any impl providing tuning flags, but
there should at least be an attempt to make the shootout match what
"normal people" will be running and not include experimental patches or
non-standard changes.

The flags provided for JRuby are nothing unusual:

- switching the JVM to the optimized "server" JIT that ships with all
Hotspot-based JVMs
- increasing the per-thread stack size max to 4MB for the deeper benchmarks

There's not really anything "funky" about them, and I think it's
actually a benefit of using JRuby that there are many ways to tune it
without requiring a rebuild of anything.

[rogerdpack]

On Dec 9 2008, 7:18 am, "Antonio Cangiano" <acangi...@gmail.com>
wrote:

> Hi guys,
>
> after a lot of work, I'm happy to announce the new version of The Great Ruby

> Shootout:http://antoniocangiano.com/2008/12/09/the-great-ruby-shootout-decembe...

Nice as well might be to benchmark REE with and without

GC.copy_on_write_friendly=true

set.
Just thinking out loud :)
-=r