Next steps

[Antonio Cangiano]

Hi guys,

Over the past two weeks there's been a fair bit of activity surrounding the RBS and I have read the emails you've sent to the group and to me privately. You've had a wealth of suggestions, so I'd like to summarize what I think should be done for the foreseeable future and to reply to some of your comments.

I'd like to get past the latest controversies and focus (each of) our limited amount of spare time on making the RBS the project that all we want it to be. Those who have a horse in this race (so to speak) are going to be biased. There is nothing wrong with it, but please understand that I will have to be impartial, so as to prevent two things from happening: 1) Discrediting the value of the suite; 2) Gaming (voluntarly or involuntarily) the suite by influencing the types of benchmarks that get added or removed. Please bear in mind that my only interest is to work together to create something really worthwhile - something that other communities can look up to even.

I'll number my points to make the discussion easier to follow (and in turn reply to).

0) We'll work on the existing repository. The old RBS is stored in deprecated_rbs/, we'll get rid of it once the new one is able to be viewed as a sufficient replacement. I'll update the current README and add a first "mission statement" as well.

1) As mentioned before, we are going to work with Brian's monitor approach which simplifies the process of creating or altering existing benchmarks. At this stage, this works through a bash script which implies that the current RBS is broken on plain-vanilla Windows. This is not as bad as it sounds, as long as it's able to work under Cygwin. Does anyone want to volunteer to verify that this is actually the case and to write down a note in the README about Windows requirements?

2) We can divide the benchmarks in two folders, micro-benchmarks and macro-benchmarks. The latter of which will aim to cover "real-worldish" benchmarks.

3) Among the micro-benchmarks, we can have tests which are useful to language implementers (e.g. core/standard classes or syntactic/language constructs). But I'd like to hang onto a only few benchmarks, those that meet two conditions: a) They must not be of specific interest for a single VM, but rather of generic appeal for all the VMs involved; b) They need to have some practical impact on real world programs. I realize this is hard to establish, but going this route should help us rule out obvious cases like raising an exception a million times, assigning a constant a million times and so on. This will definitely get rid of several YARV benchmarks (that were never intended to be used for VMs comparisons) and open the door to new ones (e.g., selected benchmarks from JRuby or Rubinius' repositories).

4) The benchmarks should run for a reasonable amount of time. By this I mean two things: we shouldn't have the majority of the tests each taking  0.2 seconds (or so) to run, and likewise we should reduce tests that take too long to run, in order to keep the overall execution time under an hour when run on average hardware with an average VM (e.g, MRI).

5) In an effort to put things into perspective, stress the importance of the RubySpec project and help showcase the progress that's been made by each VM over time; the results of RBS will always be presented along with a RubySpec score for each VM tested (at least in my shootout).

6) In order to give credibility to the suite - and obtain results that are more consistent with real performance - we should focus our attention on the "macro-benchmarks" and add in small programs that perform some form of real workload. Tim Bray's RX XML benchmarks is one that we need to add as soon as possible (and it wouldn't hurt to somehow plug in a Rails applications as well).

7) Where are we in regards to memory measurement on Linux/Mac OS X systems? I believe Roger had done some work in that sense. But I didn't follow this aspect too closely. Whatever the case, we need to have this implemented. Many people care more about memory consumption than speed. Who would like to take care of this?

8) I'm not sure how Ed's work and ideas can be integrated into the suite, but it's clear to me that there are some nice opportunities in this area.

9) There was a suggestion to switch to Tap (http://tap.rubyforge.org/) as it's arguably better than Rake, plus it's less memory hungry. What are your thoughts? I'm neutral on this.

10) Let's learn from other peoples' work. Brian suggested that we take a look at, and review, other benchmarks that have been/are used in the industry: SunSpider, Richards, SpecJVM2008, etc, to get a sense of their approaches and then be able to incorporate them into RBS. I'll add to that idea that we should also take a look at Google's approach with Unladen Swallow. They have been *very* selective in regards to their benchmark approach as you can read here: http://code.google.com/p/unladen-swallow/wiki/Benchmarks.

11) Brian has created rbs @ irc.freenode.net. Pop in there whenever you have some spare time to dedicate to the project. I think it will make our life easier, as we collaborate and collectively discuss some points together.

I think we can do a good job. Please get back to me with your thoughts when you can.

Thank you in advance,

Antonio

--
http://antoniocangiano.com - Zen and the Art of Programming
http://math-blog.com - Mathematics is wonderful!
http://stacktrace.it - Aperiodico di resistenza informatica
Follow me on Twitter: http://twitter.com/acangiano
Author of "Ruby on Rails for Microsoft Developers": http://bit.ly/rorforms

[M. Edward (Ed) Borasky]

On Fri, Apr 10, 2009 at 10:16 PM, Antonio Cangiano <acan...@gmail.com> wrote:
> 0) We'll work on the existing repository. The old RBS is stored in
> deprecated_rbs/, we'll get rid of it once the new one is able to be viewed
> as a sufficient replacement. I'll update the current README and add a first
> "mission statement" as well.
> 1) As mentioned before, we are going to work with Brian's monitor approach
> which simplifies the process of creating or altering existing benchmarks. At
> this stage, this works through a bash script which implies that the current
> RBS is broken on plain-vanilla Windows. This is not as bad as it sounds, as
> long as it's able to work under Cygwin. Does anyone want to volunteer to
> verify that this is actually the case and to write down a note in the README
> about Windows requirements?

I have Windows (64-bit Vista) and a full (32-bit) Cygwin. Still, since
we are presupposing Ruby, why does this need a "bash" script? Also,
what about the new "MinGW" based Windows Ruby? That uses an MSYS
command line rather than Cygwin, and I am guessing it will run faster
than the Cygwin Ruby does. It also includes a Windows non-Cygwin
"git".

> 4) The benchmarks should run for a reasonable amount of time. By this I mean
> two things: we shouldn't have the majority of the tests each taking  0.2
> seconds (or so) to run, and likewise we should reduce tests that take too
> long to run, in order to keep the overall execution time under an hour when
> run on average hardware with an average VM (e.g, MRI).

Actually, though, if you can tap into high-resolution timers, which go
down to at least microseconds and maybe even better, it's not
unreasonable to have benchmarks that only run five seconds or so.
You'd have to start the timers in the right place to avoid setup
times, but that shouldn't be hard either.

> 6) In order to give credibility to the suite - and obtain results that are
> more consistent with real performance - we should focus our attention on the
> "macro-benchmarks" and add in small programs that perform some form of real
> workload. Tim Bray's RX XML benchmarks is one that we need to add as soon as
> possible (and it wouldn't hurt to somehow plug in a Rails applications as
> well).

Is there a Rails benchmark other than PetStore? I have code to run
PetStore if that's the best we can do. It's in that huge amorphous
RubyForge repository of mine. :) IIRC it uses MySQL as the database,
so a port to SQLite might be a good idea.

> 7) Where are we in regards to memory measurement on Linux/Mac OS X systems?
> I believe Roger had done some work in that sense. But I didn't follow this
> aspect too closely. Whatever the case, we need to have this implemented.
> Many people care more about memory consumption than speed. Who would like to
> take care of this?

If you use Linux kernels 2.6.25 or later, you can get pretty good
process / thread memory usage data from /proc. That's Fedora 9 or
later, Ubuntu Intrepid Ibex or later and openSUSE 11.0 or later.
Speaking of Ubuntu and Fedora, they're both in beta now and I'm
playing around with them. I was running Ubuntu on my laptop most of
this week, but dropped back to openSUSE. It's not so much that the
Ubuntu beta was unstable -- there were a few minor glitches but it was
usable. But I missed the convenience of YaST for system
administration.

In any event, I'll volunteer to do the memory data collection for
openSUSE 11.1 / 2.6.27. It should backport fine to 2.6.25. To keep
things efficient, I will probably snapshot the raw data to a file in C
code and post-process it in Ruby, rather than trying to have the
benchmark driver open files and parse a bunch of stuff in Ruby during
a run.

> 8) I'm not sure how Ed's work and ideas can be integrated into the suite,
> but it's clear to me that there are some nice opportunities in this area.

As long as we agree on which version of Ruby is the baseline, we
should be OK. On an x86 / x86_64 system, that should probably be the
lowest common denominator, which would most likely be Ruby 1.8.7
compiled for an i386. :)

> 9) There was a suggestion to switch to Tap (http://tap.rubyforge.org/) as
> it's arguably better than Rake, plus it's less memory hungry. What are your
> thoughts? I'm neutral on this.

Tap? As in Test Anything Protocol? That's a big deal in the Perl and
PostgreSQL communities, so if we can leverage off of that, let's go
for it!

> 10) Let's learn from other peoples' work. Brian suggested that we take a
> look at, and review, other benchmarks that have been/are used in the
> industry: SunSpider, Richards, SpecJVM2008, etc, to get a sense of their
> approaches and then be able to incorporate them into RBS. I'll add to that
> idea that we should also take a look at Google's approach with Unladen
> Swallow. They have been *very* selective in regards to their benchmark
> approach as you can read here:
> http://code.google.com/p/unladen-swallow/wiki/Benchmarks.

Yeah ... what we have now is more like the Lawrence Livermore Loops
than anything else. :)

> 11) Brian has created rbs @ irc.freenode.net. Pop in there whenever you have
> some spare time to dedicate to the project. I think it will make our life
> easier, as we collaborate and collectively discuss some points together.
> I think we can do a good job. Please get back to me with your thoughts when
> you can.
>
> Thank you in advance,
> Antonio

Any particular time? I'm swamped with Open Source Bridge stuff early
this coming week, but by the end of the week I should have some spare
time to start looking at the Linux memory data collection stuff.

Oh yeah ... is anyone from the Windows Ruby Installer (like Luis
Lavena) on this mailing list? That team should be getting close to the
point where they'll want to start playing with GCC options and
profiling. No sense letting the *NIX folks have all the fun. ;)

--
M. Edward (Ed) Borasky
http://www.linkedin.com/in/edborasky

I've never met a happy clam. In fact, most of them were pretty steamed.

[Jim Deville]

Another, possibly minor, reason to go with something other than Cygwin is IronRuby. I don't really want to benchmark IronRuby on Mono.

JD

[Monty Williams]

> I have Windows (64-bit Vista) and a full (32-bit) Cygwin. Still, since
> we are presupposing Ruby, why does this need a "bash" script?

The only bash script is utils/timeout, adapted from http://www.shelldorado.com/scripts/cmds/timeout.
It's quite simple. It starts a 'watchdog' process then execs the ruby benchmark. The watchdog process sleeps for timeout seconds, then wakes up and kills -1 it's parent pid. As long as exec works correctly in Ruby on Windows, we should be able to replace it with a very short Ruby script.

In the meantime, you can verify the current timeout works on Vista with Cygwin or MSYS with a one line command that takes only seconds.
rake bench:file FILE= benchmark/rbs/core-features/bm_app_tak.rb
Can Ed or someone else with Windows verify this works under Cygwin?

> If you use Linux kernels 2.6.25 or later, you can get pretty good process
> / thread memory usage data from /proc

/proc doesn't exist on Mac OS X. It would be nice to have a universal way to measure memory usage, but it may have to be platform specific :-(

> As long as we agree on which version of Ruby is the baseline, we
> should be OK. On an x86 / x86_64 system, that should probably be the
> lowest common denominator, which would most likely be Ruby 1.8.7
> compiled for an i386. :)

It should have no effect on the test, so it needn't be the same version on all platforms. It seems the LCD should be the Ruby that comes installed on your system. Mac OS X comes with ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0] which Apple modifies, so some Mac users will be reluctant to compile from scratch. The

[Antonio Cangiano]

On Sat, Apr 11, 2009 at 12:14 PM, Jim Deville <jdev...@microsoft.com> wrote:
> Another, possibly minor, reason to go with something other than Cygwin is IronRuby.

> I don't really want to benchmark IronRuby on Mono.

A Ruby script is definitely the way to go. And by the way, I'd like to compare IronRuby on Mono running on Linux, to IronRuby running on Vista

[Antonio Cangiano]

On Sat, Apr 11, 2009 at 12:16 PM, Monty Williams <monty.w...@gemstone.com> wrote:
> /proc doesn't exist on Mac OS X. It would be nice to have a universal way to measure memory usage,

> but it may have to be platform specific :-(

To get started, I'd be happy if we only had (use) memory measurement on Linux. Later on though, we'll definitely need to resort to platform specific code to support Mac OS X and Windows. On Windows, can the Microsoft guys recommend what the best practice would be to check and report on memory usage?

 
> It should have no effect on the test, so it needn't be the same version on all platforms. It seems the LCD 

> should be the Ruby that comes installed on your system.   Mac OS X comes with ruby 1.8.6

> (2008-03-03 patchlevel 114) [universal-darwin9.0] which Apple modifies, so some Mac users will be reluctant 

> to compile from scratch. The

Agreed. Even though this means that the baseline on Ubuntu would be extremely slow (twice as slow as Ruby 1.8.7 built from source).

[Jim Deville]

I said that wrong. I didn't mean that I don't want Monlo benchmarked. I meant I don't want it to be the only way. I too would like to see Linux vs Windows.

JD

-----Original Message-----
From: Antonio Cangiano <acan...@gmail.com>
Sent: April 11, 2009 10:00 AM
To: ruby-bench...@googlegroups.com <ruby-bench...@googlegroups.com>
Subject: [RBS] Re: Next steps

On Sat, Apr 11, 2009 at 12:14 PM, Jim Deville <jdev...@microsoft.com<mailto:jdev...@microsoft.com>> wrote:
> Another, possibly minor, reason to go with something other than Cygwin is IronRuby.
> I don't really want to benchmark IronRuby on Mono.

A Ruby script is definitely the way to go. And by the way, I'd like to compare IronRuby on Mono running on Linux, to IronRuby running on Vista

[Antonio Cangiano]

On Sat, Apr 11, 2009 at 1:16 PM, Jim Deville <jdev...@microsoft.com> wrote:

I said that wrong. I didn't mean that I don't want Monlo benchmarked. I meant I don't want it to be the only way. I too would like to see Linux vs Windows.

That's what I thought you meant. :)

Could anyone in your team replace the bash file with a Ruby script that works on Windows? Alternatively, I'd be OK with modifying the code so that it could use a bash file on Linux/Mac and a compiled program on Windows. Either way we do need something, so as to make Windows happy.

Cheers,
Antonio

[Monty Williams]

> Agreed. Even though this means that the baseline on Ubuntu would be extremely slow (twice as slow as Ruby 1.8.7 built from source).

Sorry 'bout that :-( 

I was only talking about the Ruby used to run the rake script. rake bench VM=ruby187 could be used as the performance baseline, as could a compiled ruby 1.8.6 instead of the one that comes with Ubuntu. I only want to make sure that anyone who downloads the RBS can run it without having to install something that didn't come with their system.

[Monty Williams]

A universal Ruby script would be ideal. Then we'd avoid having to figure out which OS we're running on in the script. 

----- Original Message -----
From: "Antonio Cangiano" <acan...@gmail.com>
To: ruby-bench...@googlegroups.com
Sent: Saturday, April 11, 2009 10:41:31 AM GMT -08:00 US/Canada Pacific
Subject: [RBS] Re: Next steps

[Jim Deville]

fI'd be happy to work on a replacement. I'll have time forin the next week.

JD

-----Original Message-----
From: Antonio Cangiano <acan...@gmail.com>
Sent: April 11, 2009 10:42 AM
To: ruby-bench...@googlegroups.com <ruby-bench...@googlegroups.com>
Subject: [RBS] Re: Next steps

[Antonio Cangiano]

On Sat, Apr 11, 2009 at 1:47 PM, Monty Williams <monty.w...@gemstone.com> wrote:

I was only talking about the Ruby used to run the rake script. rake bench VM=ruby187 could be used as the performance baseline, as could a compiled ruby 1.8.6 instead of the one that comes with Ubuntu. I only want to make sure that anyone who downloads the RBS can run it without having to install something that didn't come with their system.

Oh I see what you mean. Yes, that's definitely fine. :)

A universal Ruby script would be ideal. Then we'd avoid having to figure out which OS we're running on in the script. 

+1

Cheers,

Antonio

-- 

[Antonio Cangiano]

On Sat, Apr 11, 2009 at 1:51 PM, Jim Deville <jdev...@microsoft.com> wrote:

I'd be happy to work on a replacement. I'll have time forin the next week.

Awesome. Thanks.

-- 

[Monty Williams]

Thanks Jim, that would be outstanding. Glad to have some Windows experts involved. I'll be glad to test your replacement on Ubuntu and Mac OS X.

Here's the relevant bash code (stripped of messages about incorrrect invocations).

As long as Kernel.exec, pid code, and kill work OK on Windows it shouldn't prove too difficult.

if [ -z "$parent_pid" ]

then

    # This is the first invokation of this script.

    # Start "watchdog" process, and then run the command.

    "$0" -p $$ -t $time_out & # Start watchdog

    exec "$@" # Run command

    exit 2 # NOT REACHED

else

    # We run in "watchdog" mode, $parent_pid contains the PID

    # of the process we should terminate after $time_out seconds.

    exec >/dev/null 0<&1 2>&1 # Suppress error messages

    sleep $time_out

    kill $parent_pid && # Give process time to terminate

     (sleep 2; kill -1 $parent_pid) && (sleep 2; kill -9 $parent_pid)

    exit 0

fi

----- Original Message -----
From: "Jim Deville" <jdev...@microsoft.com>
To: ruby-bench...@googlegroups.com

Sent: Saturday, April 11, 2009 10:51:06 AM GMT -08:00 US/Canada Pacific
Subject: [RBS] Re: Next steps

fI'd be happy to work on a replacement. I'll have time forin the next week.

JD

-----Original Message-----
From: Antonio Cangiano <acan...@gmail.com>
Sent: April 11, 2009 10:42 AM
To: ruby-bench...@googlegroups.com <ruby-bench...@googlegroups.com>
Subject: [RBS] Re: Next steps

On Sat, Apr 11, 2009 at 1:16 PM, Jim Deville <jdev...@microsoft.com> wrote:

I said that wrong. I didn't mean that I don't want Monlo benchmarked. I meant I don't want it to be the only way. I too would like to see Linux vs Windows.

That's what I thought you meant. :)

Could anyone in your team replace the bash file with a Ruby script that works on Windows? Alternatively, I'd be OK with modifying the code so that it could use a bash file on Linux/Mac and a compiled program on Windows. Either way we do need something, so as to make Windows happy.

Cheers,
Antonio

[M. Edward (Ed) Borasky]

On Sat, Apr 11, 2009 at 9:16 AM, Monty Williams
<monty.w...@gemstone.com> wrote:
>
>> I have Windows (64-bit Vista) and a full (32-bit) Cygwin. Still, since
>> we are presupposing Ruby, why does this need a "bash" script?
>
> The only bash script is utils/timeout, adapted from http://www.shelldorado.com/scripts/cmds/timeout.
> It's quite simple. It starts a 'watchdog' process then execs the ruby benchmark. The watchdog process sleeps for timeout seconds, then wakes up and kills -1 it's parent pid.  As long as exec works correctly in Ruby on Windows, we should be able to replace it with a very short Ruby script.
>
> In the meantime, you can verify the current timeout works on Vista with Cygwin or MSYS with a one line command that takes only seconds.
>  rake bench:file FILE= benchmark/rbs/core-features/bm_app_tak.rb
> Can Ed or someone else with Windows verify this works under Cygwin?

Yeah ...

1. Can I just do a "git clone" and type some simple thing at the command line?
2. I also have Luis Lavena's in-progress One-Click Ruby Installer on
that system, although I have no idea whether it functions or not or
passes any of the specs. I might as well take a shot at it while I'm
booted in Windows, since at the present time, I only boot into Windows
once a week to watch the StrangeLoveLive podcast, and that's only
going to happen until ALSA catches up to the sound card in my laptop.
:)

>
>> If you use Linux kernels 2.6.25 or later, you can get pretty good process
>> / thread memory usage data from /proc
>
> /proc doesn't exist on Mac OS X. It would be nice to have a universal way to measure memory usage, but it may have to be platform specific :-(

My recollection is that everything performance-related on MacOS X is
encapsulated in DTrace, although there is probably a usable
"getrusage" call in the C library that a Ruby interpreter could call.
I'll check the Linux libraries to see if getrusage has memory stuff
now ... it used to report only CPU statistics, because the kernel
wasn't keeping track of everything getrusage wants. Isn't Tim Bray our
resident DTrace expert? :)

>> As long as we agree on which version of Ruby is the baseline, we
>> should be OK. On an x86 / x86_64 system, that should probably be the
>> lowest common denominator, which would most likely be Ruby 1.8.7
>> compiled for an i386. :)
>
> It should have no effect on the test, so it needn't be the same version on all platforms. It seems the LCD should be the Ruby that comes installed on your system.   Mac OS X comes with ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0] which Apple modifies, so some Mac users will be reluctant to compile from scratch. The

My statistics work relative to a common baseline on any given
platform. So each platform with a different installed Ruby will have
different relative speeds for the implementations under test. Most of
the Linux implementations I've looked at recently, which was two,
openSUSE 11.1 and Ubunut 9.04 beta, are shipping 1.8.7. On a 64-bit
machine, they're compiled 64 bit, but on a 32-bit machine, openSUSE
compiles for i586 and most of the others compile for i386. That's why
I picked i386 and 1.8.7 as the LCD. But 1.8.6 / i386 should also work,
and there's be a side benefit of seeing if there were any performance
regressions from 1.8.6 to 1.8.7. :)

Brian, IIRC Engine Yard is now the keeper of the official Ruby 1.8.6
source tree. Do the Ruby maintainers have some kind of "continuous
integration" / "build farm" that could spit out "/bin" and "/lib"
trees people could use for Linux, Mac and Windows as baseline
implementations for the benchmark suite? That way, nobody would have
to recompile anything, just unpack a binary tarball or ZIP archive in
a local directory and change some search path settings. :)

[M. Edward (Ed) Borasky]

On Sat, Apr 11, 2009 at 9:58 AM, Antonio Cangiano <acan...@gmail.com> wrote:
> On Sat, Apr 11, 2009 at 12:14 PM, Jim Deville <jdev...@microsoft.com>
> wrote:
>> Another, possibly minor, reason to go with something other than Cygwin is
>> IronRuby.
>> I don't really want to benchmark IronRuby on Mono.
>
> A Ruby script is definitely the way to go. And by the way, I'd like to
> compare IronRuby on Mono running on Linux, to IronRuby running on Vista

What kind of shape is IronRuby in these days? I haven't heard much about it.

[Evan Phoenix]

This is where the --enable-pthread problem comes into play. It's pretty widely known that --enable-pthread hurts performance on MRI, and most (at least debian) linux distros compile with --enable-pthread for their packaged ruby.

So I think it's important that the output include some data from rbconfig.rb, specificly:

 

Config::CONFIG["configure_args"]

That will easily tell us how that particular MRI was configured. In addition, I'd imagine someone will benchmark different MRI configurations (likely --enable-pthread vs --disable-pthread).

My $0.02.

 - Evan

[Jim Deville]

We're chugging along for a 1.0 release this summer. http://ironruby.info has some stats on spec passage rates and some small startup and hello world benchmarks.

We're doing a push on .NET interop right now, and I am writing interop tests. In the next few weeks I plan to file any bugs that block the benchmark suite from completing.

Feel free to ping me offline if you want more info.

JD

-----Original Message-----
From: M. Edward (Ed) Borasky <zzn...@gmail.com>

Sent: April 11, 2009 11:46 AM
To: ruby-bench...@googlegroups.com <ruby-bench...@googlegroups.com>
Subject: [RBS] Re: Next steps

[Roger Pack]

> 0) We'll work on the existing repository. The old RBS is stored in
> deprecated_rbs/, we'll get rid of it once the new one is able to be viewed
> as a sufficient replacement. I'll update the current README and add a first
> "mission statement" as well.

nice.

> 1) As mentioned before, we are going to work with Brian's monitor approach
> which simplifies the process of creating or altering existing benchmarks. At
> this stage, this works through a bash script which implies that the current
> RBS is broken on plain-vanilla Windows. This is not as bad as it sounds, as
> long as it's able to work under Cygwin. Does anyone want to volunteer to
> verify that this is actually the case and to write down a note in the README
> about Windows requirements?

It seems that it currently "at least" requires cygwin [though I don't
have cygwin installed, so can't verify].

c:\dev\downloads\ruby-benchmark-suite>rake bench
(in c:/dev/downloads/ruby-benchmark-suite)
Running Ruby Benchmark Suite
Writing report to results/rbs/RBS-ruby-090411.130408.yaml
Running bm_app_factorial.rb
'c:/dev/downloads/ruby-benchmark-suite/utils/timeout' is not
recognized as an internal or external command,
operable program or batch file.

> 2) We can divide the benchmarks in two folders, micro-benchmarks and
> macro-benchmarks. The latter of which will aim to cover "real-worldish"
> benchmarks.

Sounds good. I notice that currently we've got them divided into
"core-features" "core-library" etc. still
My own way to handle this would be to leave all the tests [except
maybe real world] in the rbs_deprecated path, then copy them over as
they are deemed "useful" :)

> 5) In an effort to put things into perspective, stress the importance of the
> RubySpec project and help showcase the progress that's been made by each VM
> over time; the results of RBS will always be presented along with a RubySpec
> score for each VM tested (at least in my shootout).

Cool. Are you going to include that as a submodule or just use it
side by side the RBS for your own internal testing?

> 6) In order to give credibility to the suite - and obtain results that are
> more consistent with real performance - we should focus our attention on the
> "macro-benchmarks" and add in small programs that perform some form of real
> workload. Tim Bray's RX XML benchmarks is one that we need to add as soon as
> possible (and it wouldn't hurt to somehow plug in a Rails applications as
> well).

Yeah the kicker of going down this path is that you have to collect
those "real world" benchmarks. Maybe a plea to ruby talk? I'd be
happy to plugin in my rails substruct benchmarks (which are things
like load a page 1000x, save 1000 models, that type of thing). They're
in the rails branch still.
Do we need to certify that new tests run on jruby/MRI/KRI before
committing them or just add away?

> 7) Where are we in regards to memory measurement on Linux/Mac OS X systems?
> I believe Roger had done some work in that sense. But I didn't follow this
> aspect too closely. Whatever the case, we need to have this implemented.
> Many people care more about memory consumption than speed. Who would like to
> take care of this?

I can for windows and Linux. With Linux it's just a regex on
/proc/pid/status [I think]. With windows I use an external gem
dependency for it. There's code for it in the deprecated folder. It
is indeed interesting--I for one was surprised when I ran my "50 MB
GC" tests and found them taking 100MB--then realized I was running on
64-bit Linux--that's why.

> Thank you in advance,
> Antonio

re: luis' new (mingw) one click installer--once built it "should" be
as functional as the old mswin one click installer.

I've used it almost exclusively for quite awhile and it makes ruby
much less painful time wise on windows. As well as coming with its
own compiler :)

I have used the RBS on it a tidge--it seemed from preliminary results
that compiling on GCC4.3 versus 3.4.5 wasn't as large of a speedup as
I'd hoped. All this gleaned from the RBS.

re: shorter tests. If you're testing a fully functional ruby, you can
use the hitimes gem to get more accurate measurement. I can add it in
if desired.

Thanks!
-=r

[Monty Williams]

> Can I just do a "git clone" and type some simple thing at the command line?

Sure, you can probably do this in a cmd or git-bash shell:

git clone git://github.com/acangiano/ruby-benchmark-suite.git
cd ruby-benchmark-suite

rake bench:file FILE= benchmark/rbs/core-features/bm_app_tak.rb

rake bench:to_csv

Both the rake scripts will tell you their output files.

[Brian Ford]

Hi Ed,

On Apr 11, 11:44 am, "M. Edward (Ed) Borasky" <zzn...@gmail.com>
wrote:

>
> Brian, IIRC Engine Yard is now the keeper of the official Ruby 1.8.6
> source tree. Do the Ruby maintainers have some kind of "continuous
> integration" / "build farm" that could spit out "/bin" and "/lib"
> trees people could use for Linux, Mac and Windows as baseline
> implementations for the benchmark suite? That way, nobody would have
> to recompile anything, just unpack a binary tarball or ZIP archive in
> a local directory and change some search path settings. :)

I don't know what has been transitioned or set up yet for 1.8.6.
Official binaries would probably not be part of maintaining the
branch, that's generally up to package maintainers.

I'll ask around though.

Brian

> --
> M. Edward (Ed) Boraskyhttp://www.linkedin.com/in/edborasky

[M. Edward (Ed) Borasky]

Well ... the good news is that I'm booted up in Windows at the moment.
The bad news is that I need to reload Cygwin from scratch. That takes
a few hours on the slow laptop hard drive. So it might be tomorrow
before I have anything interesting.

If anyone else wants to play with the Windows / MSYS / MinGW / Git
stuff, the installer is at

http://code.google.com/p/msysgit/

I have that installed and running, but I haven't gone after the
Windows Ruby that uses it yet.

--
M. Edward (Ed) Borasky

[Monty Williams]

> It seems that it currently "at least" requires cygwin [though I don't
> have cygwin installed, so can't verify].

Roger,
Cygwin is a real pain. We're planning to replace "timeout" with a ruby version. If you're already using MinGW, wouldn't MSYS be a better alternative? Seems it should work, too.

> My own way to handle this would be to leave all the tests [except maybe
> real world] in the rbs_deprecated path, then copy them over as they are
> deemed "useful" :)

Please don't. I run these tests frequently and would prefer the unwanted ones were pruned rather than starting with a blank slate.

> Do we need to certify that new tests run on jruby/MRI/KRI
> before committing them or just add away?

It would be nice to have a process that ensures tests meet some criteria before they become an "official" part of RBS. In the current structure, we could have a "proposed" directory parallel to the "rbs" directory which would allow a user to run them, but they would not be run by the default command "rake bench" which runs everything beneath "rbs".

> re: shorter tests. If you're testing a fully functional ruby, you can
> use the hitimes gem to get more accurate measurement. I can add it in
> if desired.

Extremely short tests aren't that meaningful to me. If bm_app_factorial.rb with an input size of 1000 runs in .0008 this week instead of .0006 last seek did my implementation slow down by a third? Or did something occur that perturbed the result? What will viewers of RBS comparisons think? A difference between 80 seconds and 60 seconds _feels_ like it carries more weight.

I'd vote to keep the bench runner scripts as simple as possible. RBS needs to run on many platforms and _all_ ruby impls, some of which probably won't run hitimes. Even though your script had a "BAREBONES" option, I found following the internals of its measurements a bit difficult. Brian's utils/README seems pretty well thought out.

-- Monty

[Antonio Cangiano]

On Sat, Apr 11, 2009 at 2:50 PM, Evan Phoenix <ev...@fallingsnow.net> wrote:

This is where the --enable-pthread problem comes into play. It's pretty widely known that --enable-pthread hurts performance on MRI, and most (at least debian) linux distros compile with --enable-pthread for their packaged ruby.

So I think it's important that the output include some data from rbconfig.rb, specificly:

 

Config::CONFIG["configure_args"]

+1

 

That will easily tell us how that particular MRI was configured. In addition, I'd imagine someone will benchmark different MRI configurations (likely --enable-pthread vs --disable-pthread).

I would normally measure both Ubuntu's packaged Ruby (--enable-pthread) and Ruby built from source (--disable-pthread).

Cheers,

Antonio

-- 

[M. Edward (Ed) Borasky]

Well ... on Cygwin:

znmeb@DreamGate ~/Projects
$ git clone git://github.com/acangiano/ruby-benchmark-suite.git
Initialized empty Git repository in /cygdrive/c/Users/znmeb/Projects/ruby-benchm
ark-suite/.git/
remote: Counting objects: 4847, done.
remote: Compressing objects: 100% (3619/3619), done.
fatal: read error on input: Bad address
fatal: index-pack failed

znmeb@DreamGate ~/Projects
$

But on the Git "bash" prompt:

$ git clone git://github.com/acangiano/ruby-benchmark-suite.git
Initialized empty Git repository in
C:/Users/znmeb/Projects/ruby-benchmark-suite/.git/
remote: Counting objects: 4847, done.←[K
remote: Compressing objects: 100% (3619/3619), done.←[K
remote: Total 4847 (delta 1348), reused 4155 (delta 890)←[K
Receiving objects: 100% (4847/4847), 4.70 MiB | 422 KiB/s, done.
Resolving deltas: 100% (1348/1348), done.

znmeb@DREAMGATE ~/Projects
$

So ... I have the benchmark suite downloaded. Incidentally, I am
running Cygwin 1.7 Beta test ... not the "standard" Cygwin 1.5. Ruby
version is

$ ruby --version
ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-cygwin]

Stay tuned ... I have to install rubygems and rake in Cygwin.

[Monty Williams]

You could do this as a quick check:

cd C:/Users/znmeb/Projects/ruby-benchmark-suite/utils
./timeout
./timeout 5
./timeout 30

The first should print "timeout: please specify a command to execute"
The second should just sleep for 5 seconds
the their should be kill after 10 seconds and print "Terminated"

[M. Edward (Ed) Borasky]

Well ... I still don't know why the Cygwin "git clone" didn't work,
but the MSYS / git "git clone" worked fine. So:

1. Check out the repository "somehow". :)
2. Install "rubygems". I didn't check Cygwin to see if it was already
there. "rake" wasn't there, so I assumed I'd need to install
"rubygems" as well.
3. Type "gem install rake".
4. "cd" into the "ruby-benchmark-suite" and type "rake bench". Wait.
Collect the output.

I am currently at "wait". This is a 2.1 GHz Turion64 X2 (dual core
64-bit) running Windows Vista Home Premium 64 bit. It has 4 GB of RAM.
As I noted earlier, it looks like Ruby was compiled i386.

I agree with Monty -- Cygwin is a pain. While it does have some nice
Linux-like goodies, including X Windows, it does sneaky things to the
NTFS permission bits and isn't really a substitute for dual booting
with a real Linux distro.

I also have the "old" One-Click Ruby Installer on this machine.
Building the "new" One-Click Ruby Installer requires "a Ruby
interpreter", and the old One-Click will meet that. So I can benchmark
that as well before attempting a build of the new one.

Is Luis or someone from the One-Click Installer project on this list?
If not, we should probably let them know what we (I) are (am) doing,
so they won't be too surprised by any "results".

[Monty Williams]

Hi Ed,

Looks like you've made quite a bit of progress. If we can get by without Cygwin, I'll be happier.

"rake bench" will take quite a while, since it runs _all_ the benchmarks. Running a single file will let you know if things work much faster.

rake bench:file FILE= benchmark/rbs/core-features/bm_app_tak.rb

In either case, "rake bench:to_csv" will convert the .yaml file into a .csv file. You'll probably be more interested in perusing the .yaml file though.

-- Monty

[Monty Williams]

Ed,

I sent the wrong quick RBS test command. Shoulda sent this instead,

rake bench:file FILE=rbs/core-features/bm_app_tak.rb

I'll take a quick pass at a Ruby version of timeout and then leave it up to the Windows experts to fix or replace it if it doesn't work. I'll test it on Ubuntu and Mac OS X before I send it off.

-- Monty

[Roger Pack]

Sure I'll forward it on.

Ed would like to make us aware that he is "going to be benchmarking
the mingw version of ruby soon, in case we have any feedback for him
before he does."

His note is at the bottom of this message from the RBS mailing list.
Cheers!
-=r

[Antonio Cangiano]

On Sat, Apr 11, 2009 at 3:13 PM, Roger Pack <roger...@gmail.com> wrote:
> Cool.  Are you going to include that as a submodule or just use it
> side by side the RBS for your own internal testing?

It will probably be included in the RubySpec project, so all I'll have
to do is run it either by calling it from the RBS or manually starting
it so as to to obtain the completeness score (which will be published
along with the other results of the shootout).

> Yeah the kicker of going down this path is that you have to collect
> those "real world" benchmarks.  Maybe a plea to ruby talk?

Not a terrible idea.

>  I'd be
> happy to plugin in my rails substruct benchmarks (which are things
> like load a page 1000x, save 1000 models, that type of thing). They're
> in the rails branch still.

Cool.

> Do we need to certify that new tests run on jruby/MRI/KRI before
> committing them or just add away?

The programs must be standard Ruby code. Meaning Ruby code that can be
run by Ruby 1.8 as well as 1.9, without the need for external gems (if
we test a Rails app, that's an exception though). If other
implementaions fail to run it, then that's a bug they need to resolve.

> I can for windows and Linux.  With Linux it's just a regex on
> /proc/pid/status [I think]. With windows I use an external gem
> dependency for it.  There's code for it in the deprecated folder.  It
> is indeed interesting--I for one was surprised when I ran my "50 MB
> GC" tests and found them taking 100MB--then realized I was running on
> 64-bit Linux--that's why.

OK, I believe that Ed and Brian are looking into the Linux side of
things. It would be great if you could coordinate with them to add in
a Windows version.

Thanks in advance,

[Monty Williams]

>  I'd be
> happy to plugin in my rails substruct benchmarks (which are things
> like load a page 1000x, save 1000 models, that type of thing). They're
> in the rails branch still.

Will the rails tests be in their own directory? If so will it be underneath or parallel to the rbs directory? I expect it will be quite a while before we attempt to get MagLev to run rails.

I agree rails benchmarks are quite useful, but other nascent impls won't run rails initially either.

-- Monty

[M. Edward (Ed) Borasky]

I thought Rubinius had Rails running.

[Charles Oliver Nutter]

I wouldn't call Rubinius nascent...they're close to a 1.0 release.

[Antonio Cangiano]

On Thu, Apr 16, 2009 at 2:15 AM, Monty Williams
<monty.w...@gemstone.com> wrote:
> I agree rails benchmarks are quite useful, but other nascent impls won't run rails initially either.

I think we have enough VMs that are able to run Rails to warrant
introducing some form of Rails benchmark. Some VMs will fail such a
test. In fact, I would expect less mature VMs to also fail some other
macro-benchmarks that we'll come up with. But that's okay.

Cheers,

[Monty Williams]

I'm fine with including Rails benchmarks. They're what the larger audience will be interested in. Maglev isn't even to Beta yet, so I'd expect to fail some benchmarks due to unimplemented features.

Should we implement a tagging mechanism similar to MSpec? That way any implementation could skip some files or directories. The results would clearly show they weren't attempted. I include rbs as a submodule, and I run rbs with every build. It would be nice to be able skip ones known to fail or take a long time.

It would also be of value to have a verification stage for those benchmarks which could produce results, e.g. tak, tarai, fib, factorial. I'm sure Charles would agree you could appear to be much faster if you didn't actually run the code. I keep a really old RBS version around so I can diff outputs now and again.

-- Monty

[Monty Williams]

I certainly wasn't talking about Rubinius. But I thought the long term goal was to make this a useful tool for anyone creating a ruby implementation, and there are certain to be new ones in the future. Everyone who has been through this process knows it takes a long time between starting up and running Rails.

----- Original Message -----
From: "M. Edward (Ed) Borasky" <zzn...@gmail.com>
To: ruby-bench...@googlegroups.com
Sent: Wednesday, April 15, 2009 11:28:35 PM GMT -08:00 US/Canada Pacific
Subject: [RBS] Re: Next steps

[Antonio Cangiano]

On Thu, Apr 16, 2009 at 10:04 AM, Monty Williams
<monty.w...@gemstone.com> wrote:
> I'm fine with including Rails benchmarks. They're what the larger audience will be interested in. Maglev isn't even to Beta yet, so I'd expect to fail some benchmarks due to unimplemented features.

Agreed.

> Should we implement a tagging mechanism similar to MSpec? That way any implementation could skip some files or directories. The results would clearly show they weren't attempted. I include rbs as a submodule, and I run rbs with every build. It would be nice to be able skip ones known to fail or take a long time.

This is a good idea. We could implement something along the lines of .gitignore.

> It would also be of value to have a verification stage for those benchmarks which could produce results, e.g. tak, tarai, fib, factorial. I'm sure Charles would agree you could appear to be much faster if you didn't actually run the code. I keep a really old RBS version around so I can diff outputs now and again.

Yes, we need a way to verify that the results are correct.

[Roger Pack]

>>  I'd be
>> happy to plugin in my rails substruct benchmarks (which are things
>> like load a page 1000x, save 1000 models, that type of thing). They're
>> in the rails branch still.
>
> Cool.

The unfortunate aspect with them is that with each test you have to
startup rails--which takes like 20-30s on my boxes. So it won't be
quick, that's for sure. But still accurate probably.

-=r

[Antonio Cangiano]

On Thu, Apr 16, 2009 at 11:33 AM, Roger Pack <roger...@gmail.com> wrote:
> The unfortunate aspect with them is that with each test you have to
> startup rails--which takes like 20-30s on my boxes.  So it won't be
> quick, that's for sure.  But still accurate probably.

Don't worry about that. As far as I'm concerned, it's not an issue.

Cheers,

[rogerdpack]

> So I think it's important that the output include some data from  
> rbconfig.rb, specificly:
>
> Config::CONFIG["configure_args"]
>
> That will easily tell us how that particular MRI was configured. In  
> addition, I'd imagine someone will benchmark different MRI  
> configurations (likely --enable-pthread vs --disable-pthread).

Added something to attempt that. I don't see the equivalent of
COMPILER_VERSION in mri so I'm not sure what to do there-file a
feature request?
-=r