Improving USL results

[Mohit Chawla]

Hello folks, 

I am modeling an application's behavior with USL.

I am happy to report that the data perfectly aligns with Little's Law - the computed number from Little's Law is equal to the actual number of service threads in the system. 

So, load('N') and throughput data is good. But, when applying USL to the data, the best R-squared value I have been able to get so far is 0.87. I would appreciate some help to improve the results of the model. 

Here is the summary of applying USL to the data:

Call:

usl(formula = throughput ~ N, data = values)

Scale Factor for normalization: 3.3

Efficiency:

   Min     1Q Median     3Q    Max

0.5401 0.6916 0.7475 0.8282 1.0000

Residuals:

     Min       1Q   Median       3Q      Max

-151.626  -19.735    8.142   32.249   96.047

Coefficients:

       Estimate   Std. Error

sigma  6.649e-04  1.409e-04

kappa  2.452e-06  4.857e-07

Residual standard error: 42.32 on 233 degrees of freedom

Multiple R-squared: 0.8786, Adjusted R-squared: 0.878.

I have uploaded a screenshot of the data and the model curve at https://i.imgur.com/4q6qUHV.png

Thanks,

Mohit

[DrQ]

Hi Mohit,

I'm not sure why you think there's an issue that needs to be improved.

Be aware that R^2 does not tell the whole story about the fit. It's only the leading indicator.

And, you shouldn't expect it to be close to 100% (if that's what you're thinking), especially 

when you have some degree of scatter (which you do). 

It looks about right to me, given the residuals about the USL curve.

What does R report as the significance levels?

Quite honestly, I wouldn't be sweating bullets over R^2. I'd be looking at the USL projections 

beyond N = 350 threads or whatevers and assessing the magnitudes of the USL coefficients.

Do the projections (the thing that can't be seen in the data) make sense?

The other thing that looks unusual to me, is the shear number of data points. You have enough 

data to really calculate the error bars for those measurements.  Since those are also related to 

R^2, are they acceptable?

[Mohit Chawla]

Hello Dr. Gunther, 

Thanks for your reply. 

The projections for the same day do make sense, to an extent, as can be seen at http://i.imgur.com/SmiKkUG.png  - the red dots are the continued projection points. 

I was trying to use the model applied to the data from one day to predict the measurements for another day with similar load pattern, and I found out that the predictions were not close to the measured values. So I thought of improving the model's correctness. But maybe I need to repeat the calculations for a week to be able to make such predictions. Any ideas here would be helpful as well. 

Since the data is multivalued, I tried to use average of points for same values of 'N' instead, but that included rounding and converting to integers and that somehow brought down the R-squared value quite a bit. 

I do not know how I can get R to report the significance levels for the data - is there a library or function for that ? Error bars sound useful, I'll try to plot them. As for their acceptability, I am not sure how I can ascertain that, and what to do to clean them up (the data fits well with Little's Law). Any suggestions ? 

Thanks,

Mohit

[Mohit Chawla]

Hello Dr. Gunther, 

Thanks for your reply. 

The projections for the same day do make sense, to an extent, as can be seen at http://i.imgur.com/SmiKkUG.png  - the red dots are the continued projection points. 

I was trying to use the model applied to the data from one day to predict the measurements for another day with similar load pattern, and I found out that the predictions were not close to the measured values. So I thought of improving the model correctness. But maybe I need to repeat the calculations for a week to be able to make such predictions. Any ideas here would be helpful as well. 

Since the data is multivalued, I tried to use average of points for same values of 'N' instead, but that included rounding and converting to integers and that somehow brought down the R-squared value quite a bit. 

I do not know how I can get R to report the significance levels for the data. Error bars sound useful, I'll try to plot them. As for their acceptability, I am not sure how I can ascertain that, and what to do to clean them up. Any suggestions ? 

Thanks, 

Mohit

--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at https://groups.google.com/group/guerrilla-capacity-planning.
For more options, visit https://groups.google.com/d/optout.

[DrQ]

Here is my USL fit (blue curve) to the initial data (black points) ...

Extended data (red points) are then superimposed. Those data are inferior to the USL prediction based on the initial data set.

Next, I fit to all those data (black + red) to get the new USL coefficients and see how much they changed.

This clearly tells you the newer data "failed" to scale as well as the USL predicted using the original measurements. Someone now needs to explain what went "wrong". Of course, this may be just how it is. Maybe the system (whatever it is) can't scale as well as anticipated by the USL based on the lower scaling data. Nonetheless, it should be explained why not. It might be a potential performance opportunity. :)

I also interpolated the the N=1 throughput value using R and found it to be  X(1) = 3.344884.

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-planning+unsub...@googlegroups.com.
To post to this group, send email to guerrilla-capacity-planning@googlegroups.com.

[Mohit Chawla]

Hello Dr. Gunther,

Thanks a lot for taking the time to analyse and interpret the data ! I'll take up the performance opportunity and check it further.

Thanks,
Mohit

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.

Visit this group at https://groups.google.com/group/guerrilla-capacity-planning.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.

[Baron Schwartz]

I'd suggest looking at the data as a time-series as well. Can you plot the throughput and concurrency as time series and provide the links in a reply?

I was going to reply and suggest exactly what Dr. Gunther did: the "tail" of the data all fall under the predicted scaling curve, which is highly suggestive that the system is underperforming at the higher concurrencies and that this trend will continue and accelerate. I'm basing this on my own experience; when I've seen this, the system is already getting into serious trouble at the upper end and can't be counted on to produce increasing throughput under increasing load. Basically, you're already looking at its peak in the data you have at present.

[DrQ]

Here are the significance levels for the respective USL models.

For some reason they don't show up if you use Stefan Moeding's pkg.

> summary(usl.fit)

Formula: Norm ~ N/(1 + alpha * (N - 1) + beta * N * (N - 1))

Parameters:

       Estimate Std. Error t value Pr(>|t|)    

alpha 6.406e-04  1.399e-04   4.579 7.61e-06 ***

beta  2.686e-06  4.832e-07   5.559 7.40e-08 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 12.61 on 233 degrees of freedom

Number of iterations to convergence: 13 

Achieved convergence tolerance: 4.1e-06

> summary(usl.fit2)

Formula: Norm ~ N/(1 + alpha * (N - 1) + beta * N * (N - 1))

Parameters:

       Estimate Std. Error t value Pr(>|t|)    

alpha 2.026e-04  1.026e-04   1.975   0.0492 *  

beta  4.340e-06  3.250e-07  13.354   <2e-16 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 12.8 on 287 degrees of freedom

Number of iterations to convergence: 13 

Achieved convergence tolerance: 9.683e-06

On Sunday, July 17, 2016 at 7:14:29 PM UTC-7, Mohit Chawla wrote:

Hello Dr. Gunther,

Thanks a lot for taking the time to analyse and interpret the data ! I'll take up the performance opportunity and check it further.

Thanks,
Mohit

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-planning+unsub...@googlegroups.com.
To post to this group, send email to guerrilla-capacity-planning@googlegroups.com.

Visit this group at https://groups.google.com/group/guerrilla-capacity-planning.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-planning+unsub...@googlegroups.com.
To post to this group, send email to guerrilla-capacity-planning@googlegroups.com.

[Harry van der horst]

Mohit,

It looks a bit as if your database starts to fuck up at the higher utilitions.

Maybe there is a single data element that is causing contention.

kind regards 

harry

--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.

Visit this group at https://groups.google.com/group/guerrilla-capacity-planning.
For more options, visit https://groups.google.com/d/optout.

--

met hartelijke groeten/kind regards
harry van der Horst
M 0031643016999

[Stefan Moeding]

Hi!

On Monday, July 18, 2016 at 7:08:41 AM UTC+2, DrQ wrote:

Here are the significance levels for the respective USL models.

For some reason they don't show up if you use Stefan Moeding's pkg.

> summary(usl.fit)

Formula: Norm ~ N/(1 + alpha * (N - 1) + beta * N * (N - 1))

Parameters:

       Estimate Std. Error t value Pr(>|t|)    

alpha 6.406e-04  1.399e-04   4.579 7.61e-06 ***

beta  2.686e-06  4.832e-07   5.559 7.40e-08 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The significance levels were never part of the summary output for an usl model. It's just a different piece of code and while the calculation was already implemented, the output was not.

Anyway, this has been fixed with the latest release of the usl package for R. Now the summary command includes the t-value and the two sided p-value for both coefficients. So the output now looks similar to the one shown here.

Version 1.7.0 of the package should be available on your favorite CRAN mirror shortly.

Regards

Stefan