warmup

[Roger Pack]

There's been some suggestion that each test have a small warm-up
phase, where it basically runs a small n, throws away the timing.
Any thoughts on this one?
Thanks.
-rp

[Monty Williams]

My primary use of the RBS is to monitor changes in MagLev performance week to week. I'd prefer not including a warmup so I have access to that information. I have heard from some of my users that they are concerned about possible warmup times in MagLev, so it might be useful for others as well.

I'd be fine with a switch that turns warmup on/off.

I took a quick look at the Rubinius "tiered" benchmarks which do include a warmup -- i.e. they run a warmup and 5 iterations, and report the median time of the 5 iterations. Even if the warmup was significantly longer, the median was about the same between warmup and non-warmup runs.

-- Monty

[Shri Borde]

While it would be nice to have a single number to measure performance, the fact is that are a number of different aspects like cold startup time (how many disk accesses are needed at startup), warm startup time (how much computation is done during startup), memory working set (more important to multi-tasking client machines than to dedicated servers), raw throughput, etc. I think it makes to have dedicated micro-benchmarks for measuring the different aspects, and also use macro-benchmarks to measure end-to-end (non-warmup) runs.

Today, most of the RBS is computation micro-benchmark, and so I think excluding warmump is the right thing so that different implementation can make different tradeoffs on startup versus throughput, while still getting stable results out of RBS. Rails is a server app and so warmup is again not important. There could be new benchmarks added for startup for small scenarios like helloworld, and also for large client-side scenarios like, say, a Ruby Twitter client.

-- Monty

--
The GitHub project is located at http://github.com/acangiano/ruby-benchmark-suite

You received this message because you are subscribed to the Google
Groups "Ruby Benchmark Suite" group.
To post to this group, send email to
ruby-bench...@googlegroups.com
To unsubscribe from this group, send email to
ruby-benchmark-s...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/ruby-benchmark-suite?hl=en

[Shri Borde]

Btw, startup of Rails cold be used for the large app without warmup. Such a benchmark could just startup Rails, do a single request, and then exit. This benchmark number would be useful for development scenarios where the developer restarts Rails many times. No need for anything like a Ruby Twitter client.

The main point though is that such a startup benchmark should be separate from a throughput benchmarks which should exclude warmup time since warmup is not an issue in deployment scenarios

-----Original Message-----
From: Shri Borde
Sent: Thursday, February 18, 2010 11:12 AM
To: ruby-bench...@googlegroups.com
Subject: RE: [RBS] warmup

While it would be nice to have a single number to measure performance, the fact is that are a number of different aspects like cold startup time (how many disk accesses are needed at startup), warm startup time (how much computation is done during startup), memory working set (more important to multi-tasking client machines than to dedicated servers), raw throughput, etc. I think it makes to have dedicated micro-benchmarks for measuring the different aspects, and also use macro-benchmarks to measure end-to-end (non-warmup) runs.

Today, most of the RBS is computation micro-benchmark, and so I think excluding warmump is the right thing so that different implementation can make different tradeoffs on startup versus throughput, while still getting stable results out of RBS. Rails is a server app and so warmup is again not important. There could be new benchmarks added for startup for small scenarios like helloworld, and also for large client-side scenarios like, say, a Ruby Twitter client.

-----Original Message-----
From: ruby-bench...@googlegroups.com [mailto:ruby-bench...@googlegroups.com] On Behalf Of Monty Williams
Sent: Thursday, February 18, 2010 8:26 AM
To: ruby-bench...@googlegroups.com
Subject: Re: [RBS] warmup

-- Monty

--

[Roger Pack]

> Btw, startup of Rails cold be used for the large app without warmup. Such a benchmark could just startup Rails, do a single request, and then exit. This benchmark number would be useful for development scenarios where the developer restarts Rails many times. No need for anything like a Ruby Twitter client.

There's a benchmark out there that "just" starts up rails...I think
that's all it does, anyway, it might run all the migrations, too.

Currently it measures that time, before the tests even start, then
does a sleep x (where x is the total time divided by 10).

So I guess having a warmup phase wouldn't hurt this specific test.
Maybe a warmup phase is ok then.
-rp