Determining uncertainty of the Universal Scalability Law

[Baron Schwartz]

The input below is real data, but I did not measure it and I don't
know how good it is. Nor do I have any details on the systems
themselves, either software or hardware. So I think I must accept the
data as it is. After doing the least-squares regression and finding
sigma and kappa as shown in GCaP, I am unsure how to estimate the
uncertainty in the model itself. Any guidance?

Thanks,
Baron

threads txn/sec
1 955.16
2 1878.91
3 2688.01
4 3548.68
5 4315.54
6 5130.43
7 5931.37
8 6531.08
9 7219.8
10 7867.61
11 8278.71
12 8646.7
13 9047.84
14 9426.55
15 9645.37
16 9897.24
17 10097.6
18 10240.5
19 10532.39
20 10798.52
21 11151.43
22 11518.63
23 11806
24 12089.37
25 12075.41
26 12177.29
27 12211.41
28 12158.93
29 12155.27
30 12118.04
31 12140.4
32 12074.39

[DrQ]

Hi Baron,

I'm in the middle of something else, so I don't have time to look at
it right away. In the meantime, I'm not quite sure what you mean by
"uncertainty in the model" in this case, but maybe this blog post can
help http://perfdynamics.blogspot.com/2010/09/confidence-bands-for-universal.html

[M. Edward (Ed) Borasky]

I did a quick scatterplot in oocalc and I saw an issue that you might
want to check out if you can ask questions of the data source. From 1
to 18 threads, things look OK, but there's a "kink" in the curve
between 18 and 19 threads. From 19 up, it's fine. So it looks like you
have *two* GCap curves superimposed - there's some kind of "regime
change" around 18 - 19 threads.
--
M. Edward (Ed) Borasky
http://borasky-research.net http://twitter.com/znmeb

"A mathematician is a device for turning coffee into theorems." - Paul Erdos

Quoting Baron Schwartz <baron.s...@gmail.com>:

> --
> You received this message because you are subscribed to the Google
> Groups "Guerrilla Capacity Planning" group.
> To post to this group, send email to
> guerrilla-cap...@googlegroups.com.
> To unsubscribe from this group, send email to
> guerrilla-capacity-...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
>
>

[M. Edward (Ed) Borasky]

There are a couple of ways to do this, depending on what tools you
have available beyond, say, Excel or a pocket calculator. I'll assume
R, since many of us on this list know it well enough to piece together
some working code.

1. Use a direct non-linear regression model from R and use the
goodness-of-fit data / confidence intervals from that. That is, rather
than doing the algebra to convert GCap into a quadratic, fit the data
directly to the equation. I had some R code that did this a while back
- I don't remember whether I posted it anywhere or not. It may or may
not be in one of my Github repositories and it may or may not be
somewhere on my hard drive. ;-)

2. Use any regression - transformed to quadratic or direct - and do a
bootstrap or jackknife resampling. The jackknife is simpler - you just
re-run the analysis N times, once for each data point, leaving a data
point out each time. Then you compute a few statistics and you've got
confidence intervals. The most "user friendly" description of the
process is in "Data Analysis and Regression: A Second Course in
Statistics" by Mosteller and Tukey. You can do a jackknife in Excel, I
think - last time I did one I did it in FORTRAN but you should be able
to construct the iterations in VBA.

If you want to bootstrap, there are R routines to do it. A bootstrap
works by drawing repeated samples from your data, performing the
regressions, and then computing the statistics just like you would
with a jackknife.

I'd recommend the first option - it's really not that difficult unless
your data set is flaky. In that case, a simple scatterplot will tell
you it's flaky and you can simply remove bad points before doing the
fitting.

--
M. Edward (Ed) Borasky
http://borasky-research.net http://twitter.com/znmeb

"A mathematician is a device for turning coffee into theorems." - Paul Erdos

Quoting Baron Schwartz <baron.s...@gmail.com>:

[Baron Schwartz]

Hi Neil,

I'm not a stats expert -- so thanks for correcting my terminology.
That's what I'm looking for, I think. I'll study that.

> --
> You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
> To post to this group, send email to guerrilla-cap...@googlegroups.com.
> To unsubscribe from this group, send email to guerrilla-capacity-...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/guerrilla-capacity-planning?hl=en.
>
>

--
Baron Schwartz
Percona Inc <http://www.percona.com/>
Consulting, Training, Support & Services for MySQL

[Baron Schwartz]

Ed,

Thanks. I noticed the same thing. I think the data is suspect. I
looked at another set from the same benchmarks and the C(1)
measurement is way too low -- it makes the C(2 .. 5) measurements look
like they are 110% over linear speedup. I think the benchmarks might
not be properly warmed up, or something else like that might be going
on.

- Baron

--

[Baron Schwartz]

Hi Ed,

Thanks. I have been using gnuplot, which can do regressions against
any closed-form equation such as the USL. Does anyone have experience
with gnuplot, or advice about it? I have no experience using R, but I
can see that I need to learn someday. I'd like to compare gnuplot's
notes with R. Using gnuplot against the dataset I gave, I get

sigma 0.0207163 +/- 0.001323 (6.385%)
kappa 0.000861226 +/- 5.414e-05 (6.287%)
R^2 0.999624

I computed the R^2 myself with awk. Apparently gnuplot does not
really believe in R^2, but I am not educated enough to understand the
explanation given in the gnuplot manual. It just looks to me like
someone knows a lot more than I do, and can't explain it to me!

My final model (still using gnuplot) gives me a peak predicted
capacity of 12253 at N=33.

If the above is different from the results others on this list obtain,
then I think I need to take the time to learn R, and learn the
techniques you and Dr. Gunther described (jackknife, etc). But if
gnuplot's +/- results for sigma and kappa are usable, then I'll save
the time I don't have to learn R, and go on with my work.

Thank you!

- Baron

[DrQ]

I finally found some time to take a look at your data in R.

> summary(usl)

Formula: Norm ~ N/(1 + alpha * (N - 1) + beta * N * (N - 1))

Parameters:
Estimate Std. Error t value Pr(>|t|)
alpha 2.072e-02 1.322e-03 15.67 5.48e-16 ***
beta 8.612e-04 5.412e-05 15.91 3.61e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.2005 on 30 degrees of freedom

Number of iterations to convergence: 7
Achieved convergence tolerance: 8.628e-08


R plot of USL analysis ...
<http://guerrilla-capacity-planning.googlegroups.com/web/RplotBS.png?
hl=en&gsc=yTkW9xYAAABEd-l0PdqDmr79WJEbYuleLOn3cX5K69EHMinC2sQ4jA>

Seems to agree with your gnuplot results.

R plot with confidence bands ...
<http://guerrilla-capacity-planning.googlegroups.com/web/RplotBS-
CI.png?hl=en&gsc=yTkW9xYAAABEd-
l0PdqDmr79WJEbYuleLOn3cX5K69EHMinC2sQ4jA>

Hopefully, these images can be accessed. Google groups is truly awful
in this respect.

> >http://borasky-research.nethttp://twitter.com/znmeb

>
> > "A mathematician is a device for turning coffee into theorems." - Paul Erdos
>