[REPORT] cfs-v5 vs sd-0.46
Michael Gerdau
with cfs-v5 finally booting on my machine I have run my daily
numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on
top of a stock openSUSE 10.2 (X86_64). Config for both kernel
is the same except for the X boost option in cfs-v5 which on
my system didn't work (X still was @ -19; I understand this will
be fixed in -v6). HZ is 250 in both.
System is a Dell XPS M1710, Intel Core2 2.33GHz, 4GB,
NVIDIA GeForce Go 7950 GTX with proprietary driver 1.0-9755
I'm running three single threaded perl scripts that do double
precision floating point math with little i/o after initially
loading the data.
Both cfs and sd showed very similar behavior when monitored in top.
I'll show more or less representative excerpt from a 10 minutes
log, delay 3sec.
sd-0.46
top - 00:14:24 up 1:17, 9 users, load average: 4.79, 4.95, 4.80
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.8%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.0%si, 0.0%st
Mem: 3348628k total, 1648560k used, 1700068k free, 64392k buffers
Swap: 2097144k total, 0k used, 2097144k free, 828204k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6671 mgd 33 0 95508 22m 3652 R 100 0.7 44:28.11 perl
6669 mgd 31 0 95176 22m 3652 R 50 0.7 43:50.02 perl
6674
mgd 31 0 95368 22m 3652 R 50 0.7 47:55.29 perl
cfs-v5
top - 08:07:50 up 21 min, 9 users, load average: 4.13, 4.16, 3.23
Tasks: 3 total, 3 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.5%us, 0.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3348624k total, 1193500k used, 2155124k free, 32516k buffers
Swap: 2097144k total, 0k used, 2097144k free, 545568k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6357 mgd 20 0 92024 19m 3652 R 100 0.6 8:54.21 perl
6356 mgd 20 0 91652 18m 3652 R 50 0.6 10:35.52 perl
6359 mgd 20 0 91700 18m 3652 R 50 0.6 8:47.32 perl
What did surprise me is that cpu utilization had been spread 100/50/50
(round robin) most of the time. I did expect 66/66/66 or so.
What I also don't understand is the difference in load average, sd
constantly had higher values, the above figures are representative
for the whole log. I don't know which is better though.
Here are excerpts from a concurrently run vmstat 3 200:
sd-0.46
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
5 0 0 1702928 63664 827876 0 0 0 67 458 1350 100 0 0 0
3 0 0 1702928 63684 827876 0 0 0 89 468 1362 100 0 0 0
5 0 0 1702680 63696 827876 0 0 0 132 461 1598 99 1 0 0
8 0 0 1702680 63712 827892 0 0 0 80 465 1180 99 1 0 0
3 0 0 1702712 63732 827884 0 0 0 67 453 1005 100 0 0 0
4 0 0 1702792 63744 827920 0 0 0 41 461 1138 100 0 0 0
3 0 0 1702792 63760 827916 0 0 0 57 456 1073 100 0 0 0
3 0 0 1702808 63776 827928 0 0 0 111 473 1095 100 0 0 0
3 0 0 1702808 63788 827928 0 0 0 81 461 1092 99 1 0 0
3 0 0 1702188 63808 827928 0 0 0 160 463 1437 99 1 0 0
3 0 0 1702064 63884 827900 0 0 0 229 479 1125 99 0 0 0
4 0 0 1702064 63912 827972 0 0 1 77 460 1108 100 0 0 0
7 0 0 1702032 63920 828000 0 0 0 40 463 1068 100 0 0 0
4 0 0 1702048 63928 828008 0 0 0 68 454 1114 100 0 0 0
11 0 0 1702048 63928 828008 0 0 0 0 458 1001 100 0 0 0
3 0 0 1701500 63960 828020 0 0 0 189 470 1538 99 1 0 0
3 0 0 1701476 63968 828020 0 0 0 57 461 1111 100 0 0 0
4 0 0 1701508 63996 828044 0 0 0 105 458 1093 99 1 0 0
4 0 0 1701428 64012 828044 0 0 0 127 471 1341 100 0 0 0
5 0 0 1701356 64028 828040 0 0 0 55 458 1344 100 0 0 0
3 0 0 1701356 64028 828056 0 0 0 15 462 1291 100 0 0 0
cfs-v5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
6 0 0 2157728 31816 545236 0 0 0 103 543 748 100 0 0 0
4 0 0 2157780 31828 545256 0 0 0 63 435 752 100 0 0 0
4 0 0 2157928 31852 545256 0 0 0 105 424 770 100 0 0 0
4 0 0 2157928 31868 545268 0 0 0 261 457 763 100 0 0 0
4 0 0 2157928 31884 545280 0 0 0 113 435 765 100 0 0 0
4 0 0 2157928 31900 545288 0 0 0 52 422 745 100 0 0 0
4 0 0 2157556 31932 545284 0 0 0 169 436 1010 100 0 0 0
4 0 0 2157556 31952 545296 0 0 0 72 424 736 100 0 0 0
4 0 0 2157556 31960 545304 0 0 0 35 428 743 100 0 0 0
4 0 0 2157556 31984 545308 0 0 0 91 425 710 99 1 0 0
4 0 0 2157556 31992 545320 0 0 0 35 428 738 100 0 0 0
5 0 0 2157556 32016 545320 0 0 0 105 425 729 100 0 0 0
4 0 0 2157432 32052 545336 0 0 0 197 434 989 99 1 0 0
5 0 0 2157448 32060 545352 0 0 0 36 421 767 100 0 0 0
4 0 0 2157448 32076 545356 0 0 0 127 441 752 100 0 0 0
6 0 0 2157448 32092 545368 0 0 0 69 422 784 99 1 0 0
4 0 0 2157324 32116 545388 0 0 0 191 445 734 100 0 0 0
4 0 0 2157200 32148 545400 0 0 0 123 427 773 100 0 0 0
4 0 0 2157200 32156 545412 0 0 0 39 428 713 100 0 0 0
7 0 0 2156844 32184 545412 0 0 1 161 429 1360 99 1 0 0
6 0 0 2156348 32192 545416 0 0 0 32 427 723 100 0 0 0
Last not least I'd like to add that at least on my system having X
niced to -19 does result in kind of "erratic" (for lack of a better
word) desktop behavior. I'll will reevaluate this with -v6 but for
now IMO nicing X to -19 is a regression at least on my machine despite
the claim that cfs doesn't suffer from it.
Best,
Michael
PS: Only learning how to test these things I'm happy to get pointed
out the shortcomings of what I tested above. Of course suggestions for
improvements are welcome.
--
Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
Vote against SPAM - see http://www.politik-digital.de/spam/
Michael Gerdau email: m...@technosis.de
GPG-keys available on request or at public keyserver
Ingo Molnar
* Michael Gerdau <m...@technosis.de> wrote:
> I'm running three single threaded perl scripts that do double
> precision floating point math with little i/o after initially loading
> the data.
thanks for the testing!
> What I also don't understand is the difference in load average, sd
> constantly had higher values, the above figures are representative for
> the whole log. I don't know which is better though.
hm, it's hard from here to tell that. What load average does the vanilla
kernel report? I'd take that as a reference.
> Here are excerpts from a concurrently run vmstat 3 200:
>
> sd-0.46
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 5 0 0 1702928 63664 827876 0 0 0 67 458 1350 100 0 0 0
> 3 0 0 1702928 63684 827876 0 0 0 89 468 1362 100 0 0 0
> 5 0 0 1702680 63696 827876 0 0 0 132 461 1598 99 1 0 0
> 8 0 0 1702680 63712 827892 0 0 0 80 465 1180 99 1 0 0
> cfs-v5
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 6 0 0 2157728 31816 545236 0 0 0 103 543 748 100 0 0 0
> 4 0 0 2157780 31828 545256 0 0 0 63 435 752 100 0 0 0
> 4 0 0 2157928 31852 545256 0 0 0 105 424 770 100 0 0 0
> 4 0 0 2157928 31868 545268 0 0 0 261 457 763 100 0 0 0
interesting - CFS has half the context-switch rate of SD. That is
probably because on your workload CFS defaults to longer 'timeslices'
than SD. You can influence the 'timeslice length' under SD via
/proc/sys/kernel/rr_interval (milliseconds units) and under CFS via
/proc/sys/kernel/sched_granularity_ns. On CFS the value is not
necessarily the timeslice length you will observe - for example in your
workload above the granularity is set to 5 msec, but your rescheduling
rate is 13 msecs. SD default to a rr_interval value of 8 msecs, which in
your workload produces a timeslice length of 6-7 msecs.
so to be totally 'fair' and get the same rescheduling 'granularity' you
should probably lower CFS's sched_granularity_ns to 2 msecs.
> Last not least I'd like to add that at least on my system having X
> niced to -19 does result in kind of "erratic" (for lack of a better
> word) desktop behavior. I'll will reevaluate this with -v6 but for now
> IMO nicing X to -19 is a regression at least on my machine despite the
> claim that cfs doesn't suffer from it.
indeed with -19 the rescheduling limit is so high under CFS that it does
not throttle X's scheduling rate enough and so it will make CFS behave
as badly as other schedulers.
I retested this with -10 and it should work better with that. In -v6 i
changed the default to -10 too.
> PS: Only learning how to test these things I'm happy to get pointed
> out the shortcomings of what I tested above. Of course suggestions for
> improvements are welcome.
your report was perfectly fine and useful. "no visible regressions" is
valuable feedback too. [ In fact, such type of feedback is the one i
find the easiest to resolve ;-) ]
Since you are running number-crunchers you might be able to give
performacne feedback too: do you have any reliable 'performance metric'
available for your number cruncher jobs (ops per minute, runtime, etc.)
so that it would be possible to compare number-crunching performance of
mainline to SD and to CFS as well? If that value is easy to get and
reliable/stable enough to be meaningful. (And it would be nice to also
establish some ballpark figure about how much noise there is in any
performance metric, so that we can see whether any differences between
schedulers are systematic or not.)
Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Michael Gerdau
> > constantly had higher values, the above figures are representative for
> > the whole log. I don't know which is better though.
>
> hm, it's hard from here to tell that. What load average does the vanilla
> kernel report? I'd take that as a reference.
I will redo this test with sd-0.46, cfs-v5 and mainline later today.
> interesting - CFS has half the context-switch rate of SD. That is
> probably because on your workload CFS defaults to longer 'timeslices'
> than SD. You can influence the 'timeslice length' under SD via
> /proc/sys/kernel/rr_interval (milliseconds units) and under CFS via
> /proc/sys/kernel/sched_granularity_ns. On CFS the value is not
> necessarily the timeslice length you will observe - for example in your
> workload above the granularity is set to 5 msec, but your rescheduling
> rate is 13 msecs. SD default to a rr_interval value of 8 msecs, which in
> your workload produces a timeslice length of 6-7 msecs.
>
> so to be totally 'fair' and get the same rescheduling 'granularity' you
> should probably lower CFS's sched_granularity_ns to 2 msecs.
I'll change default nice in cfs to -10.
I'm also happy to adjust /proc/sys/kernel/sched_granularity_ns to 2msec.
However checking /proc/sys/kernel/rr_interval reveals it is 16 (msec)
on my system.
Anyway, I'll have to do some urgent other work and won't be able to
do lots of testing until tonight (but then I will).
Best,
Michael
Ingo Molnar
* Michael Gerdau <m...@technosis.de> wrote:
> > so to be totally 'fair' and get the same rescheduling 'granularity'
> > you should probably lower CFS's sched_granularity_ns to 2 msecs.
>
> I'll change default nice in cfs to -10.
>
> I'm also happy to adjust /proc/sys/kernel/sched_granularity_ns to
> 2msec. However checking /proc/sys/kernel/rr_interval reveals it is 16
> (msec) on my system.
ah, yeah - there due to the SMP rule in SD:
rr_interval *= 1 + ilog2(num_online_cpus());
and you have a 2-CPU system, so you get 8msec*2 == 16 msecs default
interval. I find this a neat solution and i have talked to Con about
this already and i'll adopt Con's idea in CFS too. Nevertheless, despite
the settings, SD seems to be rescheduling every 6-7 msecs, while CFS
reschedules only every 13 msecs.
Here i'm assuming that the vmstats are directly comparable: that your
number-crunchers behave the same during the full runtime - is that
correct? (If not then the vmstat result should be run at roughly the
same type of "stage" of the workload, on all the schedulers.)
Michael Gerdau
> number-crunchers behave the same during the full runtime - is that
> correct?
Yes, basically it does (disregarding small fluctuations)
I'll see whether I can produce some type of absolute performance
measure as well. Thinking about it I guess this should be fairly
simple to implement.
Ingo Molnar
* Michael Gerdau <m...@technosis.de> wrote:
> > Here i'm assuming that the vmstats are directly comparable: that
> > your number-crunchers behave the same during the full runtime - is
> > that correct?
>
> Yes, basically it does (disregarding small fluctuations)
ok, good.
> I'll see whether I can produce some type of absolute performance
> measure as well. Thinking about it I guess this should be fairly
> simple to implement.
oh, you are writing the number-cruncher? In general the 'best'
performance metrics for scheduler validation are the ones where you have
immediate feedback: i.e. some ops/sec (or ops per minute) value in some
readily accessible place, or some "milliseconds-per-100,000 ops" type of
metric - whichever lends itself better to the workload at hand. If you
measure time then the best is to use long long and nanoseconds and the
monotonic clocksource:
unsigned long long rdclock(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
(link to librt via -lrt to pick up clock_gettime())
The cost of a clock_gettime() (or of a gettimeofday()) can be a couple
of microseconds on some systems, so it shouldnt be done too frequently.
Plus an absolute metric of "the whole workload took X.Y seconds" is
useful too.
Michael Gerdau
Yep.
> In general the 'best'
> performance metrics for scheduler validation are the ones where you have
> immediate feedback: i.e. some ops/sec (or ops per minute) value in some
> readily accessible place, or some "milliseconds-per-100,000 ops" type of
> metric - whichever lends itself better to the workload at hand.
I'll have to see whether that works out. I don't have an easily
available ops/sec but I guess I could create something similar.
> If you
> measure time then the best is to use long long and nanoseconds and the
> monotonic clocksource:
[snip]
Thanks, I will implement that, for Linux anyway.
> Plus an absolute metric of "the whole workload took X.Y seconds" is
> useful too.
That's the easiest to come by and is already available.
Con Kolivas
> Hi list,
>
> with cfs-v5 finally booting on my machine I have run my daily
> numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on
> top of a stock openSUSE 10.2 (X86_64).
Thanks for testing.
You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the
task is currently on that it is using; it is not the percentage of
the "overall cpu available on the machine". Since you have 3 tasks and 2
cpus, the extra task will always be on one or the other cpu taking half of
the cpu but never on both cpus.
> What I also don't understand is the difference in load average, sd
> constantly had higher values, the above figures are representative
> for the whole log. I don't know which is better though.
There isn't much useful to say about the load average in isolation. It may be
meaningful or not depending on whether it just shows the timing of when the
cpu load is determined, or whether there is more time waiting in runqueues.
Only throughput measurements can really tell them apart.
What is important is that if all three tasks are fully cpu bound and started
at the same time at the same nice level, that they all receive close to the
same total cpu time overall showing some fairness is working as well. This
should be the case no matter how many cpus you have.
--
-ck
Michael Gerdau
> > numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on
> > top of a stock openSUSE 10.2 (X86_64).
>
> Thanks for testing.
I actually enjoyed it -- the more extensive test I had promised
two days ago is almost finished. There is just one test I have
yet to (re)run and I will have a slot for it later today so I'll
mail out the results comparing
2.6.21-rc7 (mainline)
2.6.21-rc7-sd046
2.6.21-rc7-cfs-v6-rc2 (X @ nice 0)
2.6.21-rc7-cfs-v6-rc2 (X @ nice -10)
during the early afternoon (my time).
> You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the
> task is currently on that it is using; it is not the percentage of
> the "overall cpu available on the machine". Since you have 3 tasks and 2
> cpus, the extra task will always be on one or the other cpu taking half of
> the cpu but never on both cpus.
I had assumed that given the interval of 3 sec the three tasks would
be evenly distributed among the 2 CPUs thus resulting in a CPU% of
66 each because that's what they get in the long run anyway.
Apparently 3 sec is too short an interval to see this.
> What is important is that if all three tasks are fully cpu bound and started
> at the same time at the same nice level, that they all receive close to the
> same total cpu time overall showing some fairness is working as well. This
> should be the case no matter how many cpus you have.
They are started via 'make -j3' which implies they start at the
same time (i.e. within a few msec). They initially load some data
and then perform extensive computations on that data.
Best,
Michael