A status update
Antonio Cangiano
Hash: SHA1
Hi guys,
it has been a long time since the last message in this mailing list, and
I don't want to give you the impression that this project is dead. In
fact, if any of you read my blog, you'll know that my interest in the
project is as alive as it's ever been. I regret getting caught up with
other things and apologize for not having provided an update in this ML
earlier on, but I'd like to do so now and make a few points.
* I'm sure that in the past few months your respective teams managed to
advance the status of your Ruby implementations even further. I'm
excited to see (and report) on how they compare to each other
performance-wise.
* I'm still busy writing the final chapters of my 'Ruby on Rails for
Microsoft Developers' book, but I need to work on preparing the next
shootout, and modifying the GitHub repository accordingly, too - or it's
never going to get done.
* I finally have a decent machine that I intend to use for the shootout,
which I'm about to prepare. It's a quad core machine (Intel Core 2 Quad
2.4 GHz) with 8 GB of RAM. If your implementation supports native
threading, like JRuby and IronRuby do, please consider sending me
programs/benchmarks that showcase their ability to take advantage of the
available cores. Even one or two programs just to get started would be
sufficient.
* The shootout test will be run on that aforementioned machine for Linux
and for Windows. All the available implementations will be tested on
Linux, while only IronRuby on .NET and Ruby 1.8 will be tested on
Windows for comparison. Likewise on Mac OS X, only MacRuby and Ruby 1.8
will be tested. The hardware for Mac OS X will be necessarily different
(my MacBook Pro) as I don't have an Hackintosh setup.
* I'm about to perform a major refactoring of the Ruby Benchmark Suite.
This will need to account for two things: to exclude startup time
(excluding also compiling and parsing of module/class/method
definitions) and handle variable input sizes. The first of these
requirements is important for VM implementers as well as for trying to
keep the benchmark fair. The second criteria is important in order to
really understand the behavior of these VMs as the input size changes.
For example, VM1 could be much faster than VM2 at calculating fib(10),
but might become much slower when calculating fib(45), because the
numbers involved start to become fairly large.
John and Jim's idea of testing each benchmark for a variable amount of
iterations to extrapolate the startup time is a smart suggestion.
However we also need to keep in mind that we cannot make the total
execution time impractically large. For example, I had people contact me
and tell me that they are using the suite to determine how much faster
or slower a customer deployment machine would be, compared to their
development environment. The aim of keeping the runtime reasonable would
be particularly challenging with variable iterations, because it
requires a Cartesian product between the set of iterations and the set
of input sizes.
Since refactoring is requested for each test anyway, we may as well do a
good job and exclude "startup time" (also excluding loading libraries
and definitions) from the get-go, as well as measure the programs
against variable input sizes. This should make the results fair and
interesting (running each test five times and reporting the std dev as
well).
* A test that measures the startup time of each VM will be provided.
This will be measured by simply using the 'time' Unix command.
* Since there is quite a bit of work to do and I don't want to take ages
before publishing the shootout, for the first shootout that runs the
Ruby Benchmark Suite no memory profiling will be reported. After the
shootout, we'll see how to include memory allocation in the second
edition. I suspect Ed's work will become handy at that stage.
* After the first shootout, we'll also need to include some kind of "pet
store" Rails and Merb applications, to be tested with something like
RailsBench and ab.
How does this all sound to you guys? And once again, thank you for your
patience.
Cheers,
Antonio
- --
http://antoniocangiano.com - Zen and the Art of Programming
http://math-blog.com - Mathematics is wonderful!
http://stacktrace.it - Aperiodico di resistenza informatica
Currently writing "Ruby on Rails for Microsoft Developers" for Wrox.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkkpax0ACgkQqCqsu0qUj9RTKwCgwM/7fR8Kijlh0p1cl9BvWsY8
SGIAn07+f1Ya0cjs8oa79QSKi2OQtASr
=l4lX
-----END PGP SIGNATURE-----