Harmonic Averaging of Monitored Rate Data

[DrQ]

Follow-up slides from Monitorama 2013.

[Baron Schwartz]

I think there are three most common cases where metrics are averaged together, and it might be confusing to think about which mean should be applied:

1. I have a time-series metric that's sampled once per minute, and I want to downsample it to once per five minutes.
2. I have metrics about N things at the same sample rate, and I want to combine them into a single metric without changing the sample rate. (For example, the throughput of 10 servers.)
3. Both at the same time. Give me a single metric of throughput for all of my servers, sampled once per five minutes.

[steve jenkin]

DrQ wrote on 10/04/13 1:59 AM:

Follow-up slides from Monitorama 2013.

Slide 17 "Different Time Bins".

This would be a way to meaningfully report CPU utilisation when using a variable clock
or for asymmetrical CPU's: the new Big/Little ARM arrangement or the AMD x86/ARM hybrid in the pipeline.

How do you simply calculate a "moving average" with HM?
The sort of thing you want with CPU utilisation.

E.g. 1 minute and 5 minute CPU utilisations, given 1 second reporting.
i.e. moving average of last 60 and 300 samples.

-- 
Steve Jenkin, Info Tech, Systems and Design Specialist.
0412 786 915 (+61 412 786 915)
PO Box 48, Kippax ACT 2615, AUSTRALIA

stev...@gmail.com http://members.tip.net.au/~sjenkin

[DrQ]

This is a good observation and one I'd been mulling already. The biggest impediment to providing such an analysis is data---as usual. Does anyone have variable CPU-rate data that they can share?

Keep in mind, however, that:

1. Sampled variable rates (i.e., equal time intervals) can be averaged conventionally using AM.

2. Variable rates executing the same work (e.g., DB transaction), and therefore reported at aperiodic time intervals, should be averaged using HM. 

[steve jenkin]

DrQ wrote on 12/04/13 9:43 AM:

> 1. Sampled variable rates (i.e., equal time intervals) can be averaged
> conventionally using AM.

On Linux laptops with variable power settings, my understanding is that
CPU Utilisation is still reported as a simple percentage. But 80%
utilisation @ 0.9Ghz is very different to 60% at 3.2Ghz.

I can't see how AM gives a correct picture.
Caveat: I haven't researched this or looked at the kernel source to
confirm my assumption.

[DrQ]

You may well be right but please read what I write. To begin deciding whether we should be applying the AM or HM, the metric on the y-axis must be a rate metric, e.g., MIPS, IPC, GHz, etc. CPU utilization is not a rate metric. (see slide 6)

Let's see some data.

[steve jenkin]

DrQ wrote on 12/04/13 10:59 AM:

> You may well be right but please read what I write.

It's not a question of reading, but understanding the topic like you do.

--

[DrQ]

Right again. So there's no point plotting yet another strip-chart of CPU% just b/c that's the metric that Linux happens to cough up. We need to attack variable rates from a totally different angle, with different data, with our new insights, and then review what's truly appropriate (per your original question).

[steve jenkin]

DrQ wrote on 12/04/13 11:57 AM:

> Right again. So there's no point plotting yet another strip-chart of
> CPU% just b/c that's the metric that Linux happens to cough up. We
> need to attack variable rates from a totally different angle, with
> different data, with our new insights, and then review what's truly
> appropriate (per your original question).

Isn't the effective processing rate, Million-Instructions/sec, this:

MIPS = CPU_Utilisation * CPU_MIPS

These days, esp. for laptops/tablets:

CPU_MIPS = CPU_Rate_Function( CPU_Hz )

So a rate, MIPS, = CPU% * CPU_Rate_Func(CPU_Hz)

i.e. we can do better than a strip chart if we also catch Hz and know
the mapping function from Hz to Instruction_rate.

Data... yeah, none :-( I'd love to help if I still ran any systems to
probe.

--

[Jason Edgecombe]

Would sar data from some machines be sufficient? Does the CPU frequency
need to be varied? What data needs to be captured? Under what conditions?

Jason

[DrQ]

Almost right again, but it's a little incestuous to have MIPS on both sides of the equation. For better calibration, see "The Iron Law of Performance" Section 9.3.5 of the Perl::PDQ book.

[DrQ]

To know the actual rate of execution we would need something like CPI, which can be measured (in principle) but comes from hardware registers (if available), not Unix memory counters like SAR uses. Otherwise, you would need to slam a logic analyzer onto the memory bus.

Good questions about the type of workloads to measure and the conditions under which the clock freq varies. Since there are e lot of unknowns, these would probably best be settled once we know what platform we were talking about. Whatever those answers, we would need some kind of controlled measurements. 

[Jason Edgecombe]

I'm willing to devote a few hours to trying to produce some data.

I have a personal laptop with an "Intel Core2 Duo T6600 @ 2.20GHz"
processor running Ubuntu 12.10 (Quantal) 64bit. It's usually plugged
into mains power and running BOINC. I can vary the CPU frequency by
using the /proc power saving interfaces.

I'm a Linux admin, and I've used oprofile, but I haven't directly
queried the CPU registers before. I'm willing to spend a little more
time if it involves learning a tool like statsd/collectd that would be
very useful in a production Linux environment. I am proficient with R,
bash, and Perl. My 'C' skills are rusty.

In order to generate this data, I need guidance on how to go about doing
it. high-level instructions are fine.

Thanks,
Jason

[DrQ]

Let's forget about measuring MIPS; that's kind of a red herring.

All that needs to be done—in the first instance, anyway—is to run a fixed amount of work (e.g., some convenient benchmark workload) and measure the throughput (in whatever rate units) with several different clock settings, per your /proc knobs. Having data points of this type should be enough to compare AM and HM aggregations.

In the longer term, it would be interesting to see something similar but with the CPU clock varying automatically in the form of a time series (strip chart). I've heard a lot of muttering about this variable CPU speed behavior but I've never seen any data. So, I don't have a clear idea about this phase. Maybe others do.

[Karl Arao]

You might be interested on turbostat.c here http://goo.gl/jDUKg and here http://stuff.mit.edu/afs/sipb/contrib/linux/tools/power/x86/turbostat/turbostat.c

I've also done some CPU investigation about cores vs threads http://goo.gl/1MLFf

and others (measuring, monitoring, and sizing) ppt: http://goo.gl/YnYnn paper: http://goo.gl/mTrF4 but this is Oracle/database server type of workload 

-Karl

--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-planning+unsub...@googlegroups.com.
To post to this group, send email to guerrilla-capacity-planning@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

--
Karl Arao
Blog: karlarao.wordpress.com
Wiki: karlarao.tiddlyspot.com

Twitter: @karlarao

[DrQ]

I should probably add that the %CPU needs to be recorded during any controlled measurements, since that's the CPU metric that users are typically presented with (b/c that's the Linux counter that's there) and that was the basis of Steve's complaint about Linux on variable speed CPUs.

As a general matter, Unix/Linux don't know about appln throughput, but that would be measured by your benchmark or other test harness.

[Karl Arao]

Also the time bins is similar to the part 4 section of the paper (monitoring the cpu of massively consolidated environments) which I usually use Tableau to aggregate CPU data across databases and all are being lined up in a time dimension which is automatically created..  I find this pretty accurate when you have a sustained kind of database workload which most of the OLTP type databases have.. it gets tricky for mixed workload environments but you can easily filter those workload windows using Tableau and split the data sets.

[James Newsom]

I have been bitten in the buttocks of the variable CPU speed with the HP DL series servers pre-peak 2011. The machines ship from the factory in power savings mode so that the CPU will speed up and down as necessary for power savings. I don't have the charts as I no longer work at the DotCom back in 2011 but I think the CPU (AMD in this case) wouldn't spin up to full clock speed until 80% utilization.

We were planning capacity requirements using production stress studies where live customers were driving traffic to a few servers to get the data for calculation service times under load and we were averaging about 60% CPU utilization and with some new functionality we were planning on having to purchase a lot more servers. One of our DBAs was investigating something when he determined that the CPU clock speeds were varying under load. We had worked with HP and they didn't even know that their servers were shipping from the factory in power savings mode.

We flipped the bit and disabled power savings mode and the service times dropped by something like 1/3 (this is all from memory). It was a significant difference and the total power increase was something like 13W per server. Since our data center was built for maximum utilization of the server this wasn't an issue for us.

One of the major annoyances was that the CPU speed is not properly reported via Perfmon so no logging in possible directly from Perfmon/WMI. Our DBA found the true clock speed using CPU-Z (http://www.cpuid.com/softwares/cpu-z.html).

The company then switched from Windows/IIS to a J2EE system with AIX/Websphere. They thought capacity was going to be an issue for 2012 peak but fortunately some dude from Apple took over and got customers to shop elsewhere so capacity is no longer an issue.

James

On 4/12/2013 1:16 PM, DrQ wrote:

Let's forget about measuring MIPS; that's kind of a red herring.

All that needs to be done�in the first instance, anyway�is to run a fixed amount of work (e.g., some convenient benchmark workload) and measure the throughput (in whatever rate units) with several different clock settings, per your /proc knobs. Having data points of this type should be enough to compare AM and HM aggregations.

In the longer term, it would be interesting to see something similar but with the CPU clock varying automatically in the form of a time series (strip chart). I've heard a lot of muttering about this variable CPU speed behavior but I've never seen any data. So, I don't have a clear idea about this phase. Maybe others do.

On Friday, April 12, 2013 10:48:58 AM UTC-7, Jason Edgecombe wrote:

I'm willing to devote a few hours to trying to produce some data.

I have a personal laptop with an "Intel Core2 Duo �T6600 �@ 2.20GHz"

>>> � �CPU_MIPS = CPU_Rate_Function( CPU_Hz )

>>>
>>> So a rate, MIPS, = CPU% * CPU_Rate_Func(CPU_Hz)
>>>
>>> i.e. we can do better than a strip chart if we also catch Hz and know
>>> the mapping function from Hz to Instruction_rate.
>>>

>>> Data... yeah, none :-( �I'd love to help if I still ran any systems to

>>> probe.
>>>
>> Would sar data from some machines be sufficient? Does the CPU frequency
>> need to be varied? What data needs to be captured? Under what conditions?
>>
>> Jason
>>

--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.

Visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

�
�

[DrQ]

Got data?

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-planning+unsub...@googlegroups.com.

[James Newsom]

I wish I did, but I don't as I no longer work there. And even when the power-savings issue was found we were never able to get the real CPU clock speed out of Perfmon so there could not be any correlation with the clock speed. I was hoping to show the change in response times with clock speed with a nice 3D graph but Windows didn't want to play nice with reporting the actual clock speed via Perfmon.

James

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.

[Jason Edgecombe]

Would doing multiple trials of the "openssl speed" command with varying
CPU speeds be acceptable?

Here is some sample output for a single algorithm:
-----------begin-----------
% openssl speed md5
Doing md5 for 3s on 16 size blocks: 6749464 md5's in 2.97s
Doing md5 for 3s on 64 size blocks: 5017890 md5's in 2.99s
Doing md5 for 3s on 256 size blocks: 2793996 md5's in 2.97s
Doing md5 for 3s on 1024 size blocks: 909635 md5's in 2.69s
Doing md5 for 3s on 8192 size blocks: 124579 md5's in 2.54s
OpenSSL 1.0.1c 10 May 2012
built on: Tue Mar 19 19:10:34 UTC 2013
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial)
blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT
-DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2
-fstack-protector --param=ssp-buffer-size=4 -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions
-Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_NO_TLS1_2_CLIENT
-DOPENSSL_MAX_TLS1_2_CIPHER_LENGTH=50 -DMD32_REG_T=int
-DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5
-DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM
-DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
md5 36360.75k 107406.34k 240829.28k 346269.98k 401791.80k
%
-----------end-----------

The more I dig into this, the less sure I am about being able to do it.

Jason

On 04/12/2013 02:52 PM, DrQ wrote:
> I should probably add that the %CPU needs to be recorded during any
> controlled measurements, since that's the CPU metric that users are
> typically presented with (b/c that's the Linux counter that's there) and
> that was the basis of Steve's complaint about Linux on variable speed CPUs.
>
> As a general matter, Unix/Linux don't know about appln throughput, but that
> would be measured by your benchmark or other test harness.
>
> On Friday, April 12, 2013 11:16:42 AM UTC-7, DrQ wrote:
>> Let's forget about measuring MIPS; that's kind of a red herring.
>>

>> All that needs to be done�in the first instance, anyway�is to run a fixed

[DrQ]

On Friday, April 12, 2013 4:26:17 PM UTC-7, Jason Edgecombe wrote:

Would doing multiple trials of the "openssl speed" command with varying
CPU speeds be acceptable?

Probably not very useful, unfortunately. 

>>The more I dig into this,  the less sure I am about being able to do it. 

Welcome to performance analysis. :) What's needed is a more serious workload to drive the system under test such that the reported performance metrics are meaningful. In other words, a simulation. 

This is the type of thing that SPEC.org supervises on an industrial level and they also  provide the complete workload codes, scripts and harnesses. But ... they charge for that. Not an exorbitant amount but not worth it in this context.

TPC.org organizes competitive database benchmarks. They don't charge for the codes and harnesses, b/c they don't offer any. They only provide benchmark specifications. Worth perusing to get an idea of what serious benchmarks look like and the requirements for how the performance metrics are measured.

Something more intermediate b/w these big boys is Swingbench. Swingbench is a free load generator with benchmarks designed to stress test an Oracle database (10g,11g). Here's an overview of that setup.

And here's a list of other FOSS performance testing tools. 

This is just to give you an idea of what's likely involved. I'm sure others can list their favorite performance testing tools.

[Jason Edgecombe]

On 04/12/2013 08:59 PM, DrQ wrote:
> On Friday, April 12, 2013 4:26:17 PM UTC-7, Jason Edgecombe wrote:
>> Would doing multiple trials of the "openssl speed" command with varying
>> CPU speeds be acceptable?
>>
> Probably not very useful, unfortunately.
>
>>> The more I dig into this, the less sure I am about being able to do it.
> Welcome to performance analysis. :) What's needed is a more serious
> workload to drive the system under test such that the reported performance
> metrics are meaningful. In other words, a simulation.
>

> This is the type of thing that SPEC.org<http://www.spec.org/benchmarks.html>supervises on an industrial level and they also provide the complete

> workload codes, scripts and harnesses. But ... they charge for that. Not an
> exorbitant amount but not worth it in this context.
>

> TPC.org <http://www.tpc.org/information/benchmarks.asp> organizes

> competitive database benchmarks. They don't charge for the codes and
> harnesses, b/c they don't offer any. They only provide benchmark
> specifications. Worth perusing to get an idea of what serious benchmarks
> look like and the requirements for how the performance metrics are measured.
>

> Something more intermediate b/w these big boys is Swingbench<http://www.dominicgiles.com/swingbench.html>
> . Swingbench is a free load generator with benchmarks designed to stress
> test an Oracle database (10g,11g). Here's an overview<http://www.dominicgiles.com/Swingbench.pdf>of that setup.
>
> And here's a list of other FOSS performance testing<http://www.opensourcetesting.org/performance.php>tools.

>
> This is just to give you an idea of what's likely involved. I'm sure others
> can list their favorite performance testing tools.
>

Thanks. As a seasoned sysadmin, I should have known that things are
rarely that easy. ;)

I'm still not certain that the requirements have been clarified. The
benchmarks seem to just give a single fixed rate of a single run of the
benchmark.

Let's step back and try to define this more carefully. Please correct
this as needed:
----------------------------
We want to devise an experiment with following characteristics:
1. A quiescent computer system with an OS that is running a sustained
load (say 5 minutes)
2. measurements of system metrics will be recorded at regular intervals
during the test run. The measurements will likely be of the form:
units/interval (the likely interval is seconds). Each test run should be
long enough and the intervals small enough to have at least 30
measurements during a test run.
3. The test will be repeated at different CPU clock frequencies

Questions:
1. Are multiple test runs for a single CPU frequency required? If so,
how many? 10? 20? 30?
2. What is the minimum time duration for the test run to be valid? 1
minute? 5 minutes? 10 minutes?
3. Is the load a black box? What measurements, if any are needed of the
load itself? Do we need to measure transactions per second for the load?
do we need the measurements at each interval or just a single total
measurement for the entire run.

The end goal of the experiment is to determine how to correctly
downsample rate metrics (i.e. units/sec) to a lower granularity (i.e.
units/hour). The question is which is the correct method, arithmetic
mean, harmonic mean, or some other formula. How will the test data
answer the AM vs. HM question?

Thanks,
Jason

[DrQ]

On Saturday, April 13, 2013 8:06:56 AM UTC-7, Jason Edgecombe wrote:

On 04/12/2013 08:59 PM, DrQ wrote:
> On Friday, April 12, 2013 4:26:17 PM UTC-7, Jason Edgecombe wrote:
>> Would doing multiple trials of the "openssl speed" command with varying
>> CPU speeds be acceptable?
>>
> Probably not very useful, unfortunately.

A workload that is too small (for some value of small) will have it's effectiveness for the purposes of performance measurement reduced by clever compilers, large hardware caches, and so forth.

 

>>> The more I dig into this,  the less sure I am about being able to do it.

 

> Welcome to performance analysis. :) What's needed is a more serious
> workload to drive the system under test such that the reported performance
> metrics are meaningful. In other words, a simulation.
>
> This is the type of thing that SPEC.org<http://www.spec.org/benchmarks.html>supervises on an industrial level and they also  provide the complete
> workload codes, scripts and harnesses. But ... they charge for that. Not an
> exorbitant amount but not worth it in this context.
>
> TPC.org <http://www.tpc.org/information/benchmarks.asp> organizes
> competitive database benchmarks. They don't charge for the codes and
> harnesses, b/c they don't offer any. They only provide benchmark
> specifications. Worth perusing to get an idea of what serious benchmarks
> look like and the requirements for how the performance metrics are measured.
>
> Something more intermediate b/w these big boys is Swingbench<http://www.dominicgiles.com/swingbench.html>
> . Swingbench is a free load generator with benchmarks designed to stress
> test an Oracle database (10g,11g). Here's an overview<http://www.dominicgiles.com/Swingbench.pdf>of that setup.
>
> And here's a list of other FOSS performance testing<http://www.opensourcetesting.org/performance.php>tools.
>
> This is just to give you an idea of what's likely involved. I'm sure others
> can list their favorite performance testing tools.
>
Thanks. As a seasoned sysadmin, I should have known that things are
rarely that easy. ;)

And if you keep it up with these questions, you'll soon be in real danger of becoming a performance engineer. :)

I'm still not certain that the requirements have been clarified. The
benchmarks seem to just give a single fixed rate of a single run of the
benchmark.

Almost. And it depends somewhat on the benchmark workload and what it's measuring. 

Let's step back and try to define this more carefully. Please correct
this as needed:
----------------------------
We want to devise an experiment with following characteristics:
1. A quiescent computer system with an OS that is running a sustained
load (say 5 minutes)

It won't be "quiescent" under test (or something is broken). I think you mean isolated in that, while running the measurement workload, no other workloads will impact the measurement process. That is correct.

 

2. measurements of system metrics will be recorded at regular intervals
during the test run. The measurements will likely be of the form:
units/interval (the likely interval is seconds). Each test run should be
long enough and the intervals small enough to have at least 30
measurements during a test run.

There's actually several questions in here. 

All metrics, whether system metrics (i.e., coming from Linux, hardware, etc.) or application metrics (the workload being executed) are sampled at some predetermined rate.

The sampling rate could depends many factors, in particular, the nature of the benchmark simulation. The measured rate for TPC-C database benchmark, for example, is transactions per minute (tpmC).

I'm not sure where you're getting the "30 measurements" from. This is something that is best determined by trial and error. But you have the right idea. The main purpose of running the test for "long enough" is to generate performance data that is meaningful for statistical analysis. The key concept (which I find a surprising number of people do not understand), is steady state. See Section 5.5 and 5.6 of the TPC-C benchmark spec for an easy-to-read description of what that means. Roughly put, it means that every unit of work I throw at the SUT, it completes; rather than letting is backup in a queue or buffer somewhere or dropping it on the floor.

 

3. The test will be repeated at different CPU clock frequencies

Right and they could be set under discrete control vs. arbitrary variability.

 

Questions:
1. Are multiple test runs for a single CPU frequency required? If so,
how many? 10? 20? 30?

This question will be better answered after you have a know test rig and benchmark selected. 

One per would be good for starters. Three runs would be good for better statistics.

Maybe it's your sysadm talking, but performance testing is not a moonshot. As you said earlier, it's like an experiment which you are free to adjust over and over until satisfied. If we knew everything beforehand, it wouldn't be an experiment.

2. What is the minimum time duration for the test run to be valid? 1
minute? 5 minutes? 10 minutes?

The $64K question, which has no answer (a priori). The goal is to make all measurements in steady state. :)

 

3. Is the load a black box? What measurements, if any are needed of the
load itself? Do we need to measure transactions per second for the load?
do we need the measurements at each interval or just a single total
measurement for the entire run.

The use of the term "black box" is a bit confusing in this context. 

There are 3 essential pieces: the SUT (the platform being tested), the DRV (the system that is providing the stimulus to the SUT; usually in the form executing scripts) and the WRK (the workload itself that is defined in the test scripts to run on the SUT). The DRV and the SUT can be the same thing. That's called in-situ testing. More often they are separated by a network (LAN) to better ensure that the SUT is truly isolated (per your previous question).

As I said somewhere previously, any benchmark of this type is really a simulation. I think your question here translates to "A simulation of what?" That's an excellent question and the choice depends on what you want to accomplish. In the case of TPC benchmarks, for example, the TPC-A benchmark (now defunct as a competitive benchmark) is a simulation of bank ATM machines. The TPC-C benchmarks is an OLTP workload that simulates inventory control of distributed warehouses. Swingbench provides a similar type of workload.

TPC-H is a decision support or data mining class of benchmark. Why all these different simulations? Because end-users are generally looking to get some idea of how their precious workload will run on each vendor's platform, so they can choose he best performer. That's why they are competitive benchmarks.

In the case we're discussing here, we don't care about that aspect. So, we can choose anything that we think exerts a meaningful stimulus on the SUT. In my experiments that uncovered how the load average metric is determined, I just run a number of hot-loop instances over the course of an hour or so; killing them off about midway through the measurement process. The load average metric was collected every 5 mins via a separate shell script that issued the uptime command. Or something like that. You can read about it in Chap. 6 of my Perl:PDQ book.

 

The end goal of the experiment is to determine how to correctly
downsample rate metrics (i.e. units/sec) to a lower granularity (i.e.
units/hour). The question is which is the correct method, arithmetic
mean, harmonic mean, or some other formula. How will the test data
answer the AM vs. HM question?

I would separate this into:

1. Data collection phase

2. Data analysis phase

The reason for the distinction is this. Huxley's monkeys can generate data with lots of spiffy graphs. So that's not the point for performance analysis. 

Rather, we need to first look at the kind of data that is produced by the variable speed CPU. Is the selected benchmark simulation meaningful for that purpose? Are the reported benchmark metrics properly defined? What does it tell us? Is that what we expected? Does it jive with other things we know to be true? Did something go wrong in the measurement process? What needs to be corrected? Etc., etc.

Only once we have credible data is it worth moving on to phase 2.

All the above is more or less by way of background about what serious benchmarks look like. But now that I think about it, what we want to do for the purpose of examining variable CPU performance is more akin to applying a long-running batch workload of some type. 

We want to saw a tree trunk in half with a variable speed blade. To measure the performance of the blade speed accurately, we want to eliminate other variable effects by applying a giant redwood tree to the saw blade rather than a small pine tree.

With that in mind, wouldn't one of your BOINC-SETI workloads do the trick? We just need it to be long-running and repeatable.

 

[Jason Edgecombe]

> mean *isolated* in that, while running the measurement workload, no other

> workloads will impact the measurement process. That is correct.
>
>
>> 2. measurements of system metrics will be recorded at regular intervals
>> during the test run. The measurements will likely be of the form:
>> units/interval (the likely interval is seconds). Each test run should be
>> long enough and the intervals small enough to have at least 30
>> measurements during a test run.
>>
> There's actually several questions in here.
>
> All metrics, whether system metrics (i.e., coming from Linux, hardware,

> etc.) or application metrics (the workload being executed) are *sampled* at

> some predetermined rate.
>
> The sampling rate could depends many factors, in particular, the nature of
> the benchmark simulation. The measured rate for TPC-C database benchmark,
> for example, is transactions per minute (tpmC).
>
> I'm not sure where you're getting the "30 measurements" from. This is
> something that is best determined by trial and error. But you have the
> right idea. The main purpose of running the test for "long enough" is to
> generate performance data that is meaningful for statistical analysis. The
> key concept (which I find a surprising number of people do not understand),

> is *steady state*. See Section 5.5 and 5.6 of the TPC-C benchmark spec for

> That's called *in-situ* testing. More often they are separated by a network

> (LAN) to better ensure that the SUT is truly isolated (per your previous
> question).
>
> As I said somewhere previously, any benchmark of this type is really a
> simulation. I think your question here translates to "A simulation of
> what?" That's an excellent question and the choice depends on what you want
> to accomplish. In the case of TPC benchmarks, for example, the
> TPC-A benchmark (now defunct as a competitive benchmark) is a simulation of
> bank ATM machines. The TPC-C benchmarks is an OLTP workload that simulates
> inventory control of distributed warehouses. Swingbench provides a similar
> type of workload.
>
> TPC-H is a decision support or data mining class of benchmark. Why all
> these different simulations? Because end-users are generally looking to get
> some idea of how their precious workload will run on each vendor's
> platform, so they can choose he best performer. That's why they are
> competitive benchmarks.
>
> In the case we're discussing here, we don't care about that aspect. So, we
> can choose anything that we think exerts a meaningful stimulus on the SUT.
> In my experiments that uncovered how the load average metric is determined,
> I just run a number of hot-loop instances over the course of an hour or so;
> killing them off about midway through the measurement process. The load
> average metric was collected every 5 mins via a separate shell script that
> issued the uptime command. Or something like that. You can read about it in

> Chap. 6 of my Perl:PDQ book <http://www.perfdynamics.com/iBook/ppa_new.html>
> .

I don't think that I'll have time to work on this.

Jason

[James Newsom]

In my particular case with a major e-commerce site, we did not know this
and furthermore the technical account managers didn't know that the
machines were shipping to us with a default setting of power savings.
There was a mad pre-peak rush to bounce the boxes and setup the proper
CMOS settings to disable power savings mode. I'm sure that somebody at
HP knew that these machines shipped in power savings mode but
unfortunately our account managers didn't (and this was after dumping
the CMOS settings of the machine and sending to HP for double checking).
The moral of the story is to always double check.

We also ran into some interesting issues with .NET where 8 and 16 core
machines worked just fine by 12 core machines had terrible garbage
collection but that's a story for another time.

James

> It seems that if power savings mode is on the clock speed varies. Our
> sizing teams don't seem to know that. Hopefully their sizing
> calculations are done after switching off power savings. These are
> banks. So the admins. there must know this.
> Are the virtualization considerations too ? What about sizing the
> hardware for two or more VM's running on it ?
> Thanks.

[DrQ]

Since Jason bailed on benchmarking, I'm still looking for data.

[Jason Edgecombe]

Sorry.