Load average vs runq

[Louis Coilliot]

Hello, on a Linux host I have performance stats that I don't
understand.

For example:
runq vs load:
$ vmstat 1 60 | awk '/[0-9]/ { sum+=$1 } END {print sum}'; uptime
65
11:59:26 up 336 days, 19:32, 2 users, load average: 31.59, 18.81,
11.87

runq+bprocs vs load:
$ vmstat 1 60 | awk '/[0-9]/ { rsum+=$1;bsum+=$2 } END {print rsum"
"bsum}'; uptime
55 28
14:06:57 up 336 days, 21:39, 2 users, load average: 17.16, 15.35,
11.06


So the 1 min load (ex: 17) is far greater the the mean runq (ex:
(55+28)/60=1.4)

That's not what I expected, in regard to the definition of the Load
related to the runq, from wikipedia, cf
http://en.wikipedia.org/wiki/Load_%28computing%29
"An idle computer has a load number of 0 and each process using or
waiting for CPU (the ready queue or run queue) increments the load
number by 1."

It is also said in the article that "However, Linux also includes
processes in uninterruptible sleep states (usually waiting for disk
activity), which can lead to markedly different results if many
processes remain blocked in I/O due to a busy or stalled I/O system.
(...) Such circumstances can result in an elevated load average"

But even if I add the 'b' processes to the 'r' processes, I'm still
far greater with the Load avg.

I would be thankful if someone have a logical explanation for these
stats.

Louis Coilliot

[Ken Robson]

Generally have found the following articles to be of use:

http://www.teamquest.com/resources/gunther/display/5/index.htm
http://www.teamquest.com/resources/gunther/display/7/index.htm
http://www.teamquest.com/resources/gunther/display/8/index.htm

Hope they address some of your questions.

On Jul 27, 1:54 pm, Louis Coilliot <louis.coill...@gmail.com> wrote:
> Hello, on a Linux host I have performance stats that I don't
> understand.
>
> For example:
> runq vs load:
> $ vmstat 1 60 | awk '/[0-9]/ { sum+=$1 } END {print sum}'; uptime
> 65
>  11:59:26 up 336 days, 19:32,  2 users,  load average: 31.59, 18.81,
> 11.87
>
> runq+bprocs vs load:
> $ vmstat 1 60 | awk '/[0-9]/ { rsum+=$1;bsum+=$2 } END {print rsum"
> "bsum}'; uptime
> 55 28
>  14:06:57 up 336 days, 21:39,  2 users,  load average: 17.16, 15.35,
> 11.06
>
> So the 1 min load (ex: 17) is far greater the the mean runq (ex:
> (55+28)/60=1.4)
>
> That's not what I expected, in regard to  the definition of the Load

> related to the runq, from wikipedia, cfhttp://en.wikipedia.org/wiki/Load_%28computing%29

[Louis Coilliot]

Well, I've already seen these links in the archives of this group
(subject "Understanding the load average").

It explains how the datas are (properly or not) statistically smoothed
and decayed. But not the gap I have:

active_tasks = count_active_tasks();
/* then statistical function */

This should match the runq, I think. Or I missed something.

Going further in the code or the maths is out of my league, so I was
looking for the advice of an expert.

Louis

[Federico Lucifredi]

Hello Luis,
if I read correctly, you are summing up the first field (or the first
and second) of vmstat output for a minute and comparing it with the
uptime load average for a minute, and the former is bigger then the
latter.

At a quick glance, the wikipedia page gives you that processes
"running or waiting" are the ones being counted. in vmstat, the man
page defines r as giving you processes waiting for run time, and b as
giving you those sleeping waiting for IO. You seem to be missing those
actually running in your count, or the man page may be inaccurate.

Of course, if processes are missing that would only *increase* your
count, and therefore increases your discrepancy.

But you are skipping over the math: while count_active_tasks is
actually counting processes in RUNNING or UNINTERRUPTIBLE as you are
trying to:

608 for_each_task(p) {
609 if ((p->state == TASK_RUNNING ||
610 (p->state & TASK_UNINTERRUPTIBLE)))
611 nr += FIXED_1;

avenrun feeds the result of that count to the exponentiating function,
and only the output of that is the load average:

633 active_tasks = count_active_tasks();
634 CALC_LOAD(avenrun[0], EXP_1, active_tasks);

So, even if your count was correct, you would still need to feed it to
CALC_LOAD to get the load average out of it you seek to match with
uptime's output.

Hope this helps.

Best -Federico

PS: Load average is always a fun subject. Besides the Wikipedia page,
there is another interesting treatment (although not necessarily fully
up
to date) on Linux Journal:

http://www.linuxjournal.com/article/9001

[Federico Lucifredi]

Hello Computing Performance,
  I just noticed that the kernel has now distributed logic to computing these numbers - and there is an update pending for the next kernel, too.  Looks like the old code was too taxing sampling-wise for many-core CPUs.

  The patch for the upcoming 3.5 kernel is here (RC7), for those interested.

  https://patchwork.kernel.org/patch/1162981/

  Quite a bit of inline documentation there... nice!

Best -Federico