Startup time and evening the odds
Charles Oliver Nutter
MRI and 1.9, startup time is almost negligible. JRuby and Rubinius have
roughly similar startup times, but JRuby needs a bit of JVM warmup time
for best results. And IronRuby and MagLev may have similar or worse
startup times. MagLev, however, you'd probably only ever load scripts
into an existing VM.
So leveling the playing field on these benchmarks will be important.
This means most of the benchmarks in there are not going to be
particularly good for any implementation that compiles the code, since
every load of those scripts will include parsing and compiling time. On
JRuby, it will just be parsing time unless the system is set up for
forced compilation (-X+C) in which case it will also include compilation
plus classloading time. Since those benchmarks are mostly not methods,
they also won't JIT under JRuby, which only JITs method bodies (since
they're what typically gets executed the most).
Any thoughts on this?
- Charlie
Antonio Cangiano
> BTW, I'm curious how the benchmark runs will eliminate startup time. For
> MRI and 1.9, startup time is almost negligible. JRuby and Rubinius have
> roughly similar startup times, but JRuby needs a bit of JVM warmup time
> for best results. And IronRuby and MagLev may have similar or worse
> startup times. MagLev, however, you'd probably only ever load scripts
> into an existing VM.
Hi Charles,
I was going to address John Lam's email but you raise a similar point so
I'm going to answer both of you.
Each test will be run multiple times, the best time will be published,
and the std. deviation will be provided to the readers.
The startup time will not be taken into consideration. Let me clarify
this point.
The run_benchmarks.rb file will be run directly by the VM being tested,
not by MRI. Each test would then be loaded within a block passed to a
timing method. What this means is that not only would the VM already be
"warmed up" by then, but also that the start up time would not be
considered at all. Meta-example:
time_it(5) { load "bm_test1.rb" }
I will use Rubinius' benchmark.rb and timeout.rb, aptly renamed, so that
each VM reuses the same code to execute the hypothetical "time_it" method.
That should be as fair as realistically possible for every VM, but
please let me know if such an approach causes you concern.
Cheers,
Antonio
--
http://antoniocangiano.com - Zen and the Art of Programming
http://math-blog.com - Mathematics is wonderful!
http://stacktrace.it - Aperiodico di resistenza informatica
Currently writing "Ruby on Rails for Microsoft Developers" for Wrox.
Charles Oliver Nutter
> The run_benchmarks.rb file will be run directly by the VM being tested,
> not by MRI. Each test would then be loaded within a block passed to a
> timing method. What this means is that not only would the VM already be
> "warmed up" by then, but also that the start up time would not be
> considered at all. Meta-example:
>
> time_it(5) { load "bm_test1.rb" }
>
> I will use Rubinius' benchmark.rb and timeout.rb, aptly renamed, so that
> each VM reuses the same code to execute the hypothetical "time_it" method.
>
> That should be as fair as realistically possible for every VM, but
> please let me know if such an approach causes you concern.
The problem with this approach is that it will include parse and compile
time into each iteration. In JRuby it also means that code at the root
of the script will never be compiled, since we usually just interpret
the script body, leaving compilation to be done just-in-time for method
bodies. We've done a lot of work recently to make our interpreter
faster, but the best optimizations come from compilation. So this would
be a severe penalty for at least JRuby.
Even worse, even if we did turn on compilation of scripts on load, we'd
pay a higher cost for no benefit: the compilation cost would actually
slow things down, and each load would result in a new compile, new code,
and new profiling data. Since Rubinius, MagLev, and Ruby 1.9 always
compile, they'll be hindered by this for every iteration. JRuby, MRI,
and IronRuby all have an interpreted mode, so we won't automatically pat
the price. If we turned on compile-before-load in JRuby or IronRuby,
however, the penalty would be similar, and probably severe.
So this method will not measure actual warmed up performance.
One possible way to avoid this would be to load the contents of each
file as a string, construct a method for each, and loop over calls to
that method. With JRuby's jit threshold set to 0
(-J-Djruby.jit.threshold=0) this would cause the code to compile with
all appropriate optimizations. And for all implementations, it would
avoid the repeat parsing and compilation cost.
files.each do |filename|
contents = File.read(filename)
eval "def do_bench\n#{contents}\nend"
5.times { do_bench }
end
- Charlie
Antonio Cangiano
> One possible way to avoid this would be to load the contents of each
> file as a string, construct a method for each, and loop over calls to
> that method.
I'm OK with this. Does anyone have a good reason not to proceed in this way?
Avi Bryant
>
> Charles Oliver Nutter wrote:
>> One possible way to avoid this would be to load the contents of each
>> file as a string, construct a method for each, and loop over calls to
>> that method.
>
> I'm OK with this. Does anyone have a good reason not to proceed in this way?
Works for me. I do think we want to separate out parsing/compilation
time from execution time, and if this helps to do that for JRuby or
other implementations, then great.
Avi
M. Edward (Ed) Borasky
I think as the suite evolves towards more real-world benchmarks and
fewer "Lawrence Livermore Loops" type benchmarks, this will be less of a
problem. That's one of the reasons I've invested so much time in the
C-level profiling of the implementations that are written in C.
Of course, there's really only one real-world application anybody seems
to care about, and so far, I've only been able to find one benchmark for
it -- "Petstore". And from what I've been able to determine, MRI spends
more time in the garbage collector than anywhere else when running
Rails. Somebody ought to rewrite ActiveRecord so that doesn't happen. :)
--
M. Edward (Ed) Borasky
http://ruby-perspectives.blogspot.com/
"A mathematician is a machine for turning coffee into theorems." --
Alfréd Rényi via Paul Erdős
Charles Oliver Nutter
> Of course, there's really only one real-world application anybody seems
> to care about, and so far, I've only been able to find one benchmark for
> it -- "Petstore". And from what I've been able to determine, MRI spends
> more time in the garbage collector than anywhere else when running
> Rails. Somebody ought to rewrite ActiveRecord so that doesn't happen. :)
Interesting...we've been doing various things to reduce the amount of
time spent in GC, from preallocating call frames to avoiding passing
parameters and constructing values whenever they won't be used. I'd be
curious to see any numbers you have about GC and ActiveRecord. For most
benchmarks, we're still no faster than MRI, even though the JVM has a
legendary garbage collector. But it might explain where some performance
is going, though I haven't seen that be the case myself.
- Charlie
M. Edward (Ed) Borasky
I'm in the middle of rebuilding my profiling toolset. I've got Igal
Koshevoy's script that runs all the Ruby specs working, and the profile
I ran last night before I started refactoring everything showed
something like 30 percent of the time going to "gc.c" at optimization
level 0. That suite includes a fair amount of Rails stuff. Then there's
this:
http://blog.pluron.com/2008/01/ruby-on-rails-i.html
Charles Oliver Nutter
Does this also help MagLev? I would assume loading a script from disk
would also require parsing and compilation time for you as well, yes?
- Charlie
John Lam
Charles Oliver Nutter wrote:
> BTW, I'm curious how the benchmark runs will eliminate startup time. For
> MRI and 1.9, startup time is almost negligible. JRuby and Rubinius have
> roughly similar startup times, but JRuby needs a bit of JVM warmup time
> for best results. And IronRuby and MagLev may have similar or worse
> startup times. MagLev, however, you'd probably only ever load scripts
> into an existing VM.
Each test will be run multiple times, the best time will be published,
and the std. deviation will be provided to the readers.
The startup time will not be taken into consideration. Let me clarify
this point.
The run_benchmarks.rb file will be run directly by the VM being tested,
not by MRI. Each test would then be loaded within a block passed to a
timing method. What this means is that not only would the VM already be
"warmed up" by then, but also that the start up time would not be
considered at all. Meta-example:
While this is *potentially* better, I would certainly recommend using my technique as a control experiment to see whether this technique adequately accounts for startup time.
Thanks,
-John
Avi Bryant
<charles...@sun.com> wrote:
>> Works for me. I do think we want to separate out parsing/compilation
>> time from execution time, and if this helps to do that for JRuby or
>> other implementations, then great.
>
> Does this also help MagLev? I would assume loading a script from disk
> would also require parsing and compilation time for you as well, yes?
Well, parsing/compilation is *so* slow with the current version of
MagLev that I've done some work around memoizing the results, so it
wouldn't have to be a problem. But yes, it makes life simpler not to
have to worry about it.
Avi