Java thread pools and Little's law

[KriRad]

Hi,

       http://www.infoq.com/articles/Java-Thread-Pool-Performance-Tuning 

Thanks,

Mohan

[DrQ]

Little's law for Java. Wow! Will wonders never cease?

BTW: "Little’s law says that the number of requests in a system equals the rate at which they arrive, multiplied by the average amount of time it takes to service an individual request." is wrong (as the formula indicates) but let me know if it ruins the thread of the article. :)

[KriRad]

I am learning these things. Wish I could mediate between two experts - You and Kirk Pepperdine :-)

Thanks,

Mohan

[DrQ]

Let me correct his statement for you.

There are really 2 variants of Little's law: LL1 and LL2. 

LL1

His statement refers to the usual variant LL1, which states that the mean number of requests (N) in a system equals the rate of  arrivals (λ) or departures (X), times the mean time a request/thread spends in the system; aka the residence time ( R). In other words (or symbols): N = X*R. In my books, you'll more often see this written as Q = X*R b/c we are usually talking about queues and their lengths (buffer occupancy).

Although LL was known to Erlang (100+ years ago), John Little's proof c.1961 was written in terms of the waiting-time (W) and the waiting-line length (L). Hence, L = λ*W, which appears in the title of John Little's paper and is the formula quoted in the Java article. The difference b/w the two ways of writing LL1 arises from R = W + S, which says your time to get through a grocery store checkout ( R), is the time you spent waiting in line to get to the cashier (W) plus the time it takes to ring up your groceries (your service time S) once you get there.

LL2

The 2nd form of LL corresponds to ignoring the waiting time in LL1. Starting with N = X*R = X*(W + S) = X*W + X*S, and subtracting the W term from both sides, we get N - X*W = X*S. The left-hand side is the total number in residence (all customers in the checkout lane) minus those waiting (X*W). That can only leave the mean number of customers in service. Do we have a notation for that? Yes, we do. N - X*W = ρ, which is standard queueing theory notation for the utilization of the service facility. In short, ρ = X*S. This leads to a largely unknown alternative interpretation of CPU utilization: it measures the average number of threads in service (as opposed to the mean number in residence).

[KriRad]

Did you mean that all three should be averages ( the *average* number of requests in a system )?

Thanks.

On Thursday, 7 November 2013 10:03:59 UTC+5:30, DrQ wrote:

[DrQ]

Yes, mean means average. :) 

Most queueing theory equations, and especially LL, are expressed in terms of averaged metrics b/c the system is assumed to be in steady state.

[Raja Chatterjee]

A better methodology is to use Little's law for calculating concurrency and using that as an input to Erlang-B as stated in http://www.michaelnygard.com/blog/2008/11/thread_pools_and_erlang_models.html . This removes the uncertainty surrounding various headroom estimation (aka peak multiplier) that accompanies every Little's Law estimation.

The above holds good for peak traffic (not bursts - more effort is required to size for bursty traffic)

Thanks,

Raja

---

Date: Wed, 6 Nov 2013 23:23:53 -0800
From: red...@yahoo.com
To: guerrilla-cap...@googlegroups.com
Subject: Re: Java thread pools and Little's law

--
You received this message because you are subscribed to the Google Groups "Guerrilla Capacity Planning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.
To post to this group, send email to guerrilla-cap...@googlegroups.com.
Visit this group at http://groups.google.com/group/guerrilla-capacity-planning.
For more options, visit https://groups.google.com/groups/opt_out.

[DrQ]

You don't need Erlang-B or C to determine the least number of servers/threads. Revisiting the Wide Awake Developer's 2 cases:

Case 1 (using my previous notation)
λ = 0.27777 requests / ms
S = 250 ms / request

Take the product λS (LL2 from my previous reply):

> 0.27777 * 250

[1] 69.4425

Thus (rounding up), you need at least 70 servers. But that corresponds to each server/thread being close to 100% busy, in which case, the queue will grow unbounded. You would need more than 70 to avoid that situation.

Case 2 (using my previous notation)
λ = 0.27777 requests / ms
S = 250 + 100 ms / request

Taking the product λS (LL2):

> 0.27777 * (250 + 100)

[1] 97.2195

By the same argument, you need at least 98 servers.

 

To determine how many more servers you need, depends on what other performance criteria are important. Often, this will be either the residence time ( R) in the pool or the queue occupancy of waiting threads (Q). In that case, you don't need to calculate the Erlang probability function, either.

To determine R use my Guerrilla formula: R = S / [1 - (λS)^m], where m is the number of servers/threads.

To determine Q use: Q = λR, which is just LL1.

Raja

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-planning+unsub...@googlegroups.com.

[Raja Chatterjee]

Agreed. However we are looking at sizing the thread pool (max pool size given the traffic/concurrency). Erlang is one of the methods that let us size for peak using the population average (figured it out / validated with live data as against using Little's Law with a peak multiplier - a snapshot in time and dependent on the accuracy of input data w.r.t to expected peak).

Utilization is also not always a good metric to use to measure thread pool capacity. For a high volume thread pool it will lead to over provisioning as gains related to statistical multiplexing will not be realized (since we will have alarms all over if utilization hits say 70%). Erlang's blocking probability is a better SLA (size the pool such that we are OK to bounce 1 call out of 1000 ....),

Thanks,

Raja

---

Date: Thu, 7 Nov 2013 08:38:05 -0800

To unsubscribe from this group and stop receiving emails from it, send an email to guerrilla-capacity-...@googlegroups.com.

[DrQ]

How about showing us a worked example?