Remove prime benchmarks

[Evan Phoenix]

Hi everyone, especially Antonio,

I was looking over Antonio's new shootout results and I'd like to ask that we remove bm_prime entirely. It's a red herring benchmark entirely now, because nearly all implementations timeout on it, except for 1.9.2 because in 1.9.2, Prime is entirely in C.

So all it proves is that 1.9.2's Prime C code is faster than ruby code. It doesn't serve as a anything close to a reflection of overall implementation performance.

Even the benchmark itself has comments saying that it's now tuned to show great performance on 1.9.2, but timeout everywhere else.

For these reasons it's especially misleading and should be removed. If it's not removed, then 1.9.2 should use the Prime ruby code all other implementations use, NOT it's C code.

Thanks!

- Evan Phoenix

[Monty Williams]

A quick comparison on primes 3000 shows 1.9.1 & 1.9.2 are ~ 400x faster than 1.8.7.  Not useful for the casual observer. I agree with Evan that we should deprecate this benchmark.

[Antonio Cangiano]

On Tue, Jul 20, 2010 at 12:17 PM, Monty Williams <monty.w...@gemstone.com> wrote:

A quick comparison on primes 3000 shows 1.9.1 & 1.9.2 are ~ 400x faster than 1.8.7.

For some reason I don't think I received Evan's email. I agree that we can exclude/remove the benchmark.

Cheers,

Antonio

--
Antonio Cangiano
http://antoniocangiano.com

[Evan Phoenix]

On Jul 20, 2010, at 9:29 AM, Antonio Cangiano wrote:

On Tue, Jul 20, 2010 at 12:17 PM, Monty Williams <monty.w...@gemstone.com> wrote:

A quick comparison on primes 3000 shows 1.9.1 & 1.9.2 are ~ 400x faster than 1.8.7.

For some reason I don't think I received Evan's email. I agree that we can exclude/remove the benchmark.

Would you consider removing the results from the current shootout?

Cheers,

Antonio

--
Antonio Cangiano
http://antoniocangiano.com

--
The GitHub project is located at http://github.com/acangiano/ruby-benchmark-suite
 
You received this message because you are subscribed to the Google
Groups "Ruby Benchmark Suite" group.
To post to this group, send email to
ruby-bench...@googlegroups.com
To unsubscribe from this group, send email to
ruby-benchmark-s...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ruby-benchmark-suite?hl=en

[Antonio Cangiano]

On Tue, Jul 20, 2010 at 12:33 PM, Evan Phoenix <ev...@fallingsnow.net> wrote:

Would you consider removing the results from the current shootout?

I would consider adding a warning note to the time result section.

[Evan Phoenix]

On Jul 20, 2010, at 6:35 PM, Antonio Cangiano wrote:

On Tue, Jul 20, 2010 at 12:33 PM, Evan Phoenix <ev...@fallingsnow.net> wrote:

Would you consider removing the results from the current shootout?

I would consider adding a warning note to the time result section.

I'd like if you would, so that people know how to interpret the data better.

 - Evan

--
Antonio Cangiano
http://antoniocangiano.com

[Antonio Cangiano]

On Tue, Jul 20, 2010 at 9:43 PM, Evan Phoenix <ev...@fallingsnow.net> wrote:

I'd like if you would, so that people know how to interpret the data better.

Any preference on the wording of the note?

[Evan Phoenix]

  - Evan // via iPhone

On Jul 20, 2010, at 7:33 PM, Antonio Cangiano <acan...@gmail.com> wrote:

On Tue, Jul 20, 2010 at 9:43 PM, Evan Phoenix <ev...@fallingsnow.net> wrote:

I'd like if you would, so that people know how to interpret the data better.

Any preference on the wording of the note?

Perhaps: "The bm_prime benchmark was originally written to aid the development of the Prime class. As such in 1.9.2 it was rewritten in C, which makes it a poor representation of impl. performance. This benchmark will removed in the future."

--
Antonio Cangiano
http://antoniocangiano.com

--