Poor KVM guest performance on an HP rack server
Ozan Çağlayan
We have an HP Proliant DL580G5 rack server. It has 4 Intel Xeon X7460(6
core, 2.67GHz, 16MB L3) processor with 32GB of memory. /proc/cpuinfo has
24 x the following entry:
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 29
model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
stepping : 1
cpu MHz : 2666.891
cache size : 16384 KB
physical id : 3
siblings : 6
core id : 5
cpu cores : 6
apicid : 29
initial apicid : 29
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm
constant_tsc arch_perfmon pebs bts xtopology pni dtes64 monitor ds_cpl
vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi
flexpriority
bogomips : 5333.59
clflush size : 64
---
I'm running 2.6.30.9-pae on top of it. We were actually planning to use
it for a virtualization server for giving people dedicated *guest*
access for their personal compile-farm needs.
For testing purposes, we created a guest VM (2.6.30.9 too) on top of it
with 2GB of virtual memory stored in a raw partition:
qemu-kvm -cpu host -smp 2 -m 2047 -drive
file=/dev/cciss/c1d0p1,if=virtio,cache=none,boot=on -net
nic,model=virtio,macaddr=DE:AD:BE:EF:10:28 -net
tap,ifname=tap0,script=/usr/bin/qemu-ifup -k tr -nographic -daemonize
The problem is that I'm seeing very poor performance within the guest.
I've googled a bit and seen similar bug reports/discussions ending with
some tweaks (setting rotational to 1 for virtio_blk, using cache=none,
etc.) and an explanation from Avi Kivity about the bad scalability of
KVM on pre-Nehalem boxes under high build load.
But I fear that I'm far behind that *bad scalability*. I've made some
comparisons with my QuadCore Q8300 (2MB cache) box. I won't give the
whole numbers but for example,
Running the autotools configure script of CUPS on that KVM guest (I can
easily follow the output of configure line per line, it really really
waits on some checks):
real 0m52.876s
user 0m4.892s
sys 0m55.705s
On the host (while running the guest vm):
real 0m8.193s
user 0m3.099s
sys 0m4.055s
On the quadcore box:
real 0m8.424s
user 0m2.651s
sys 0m2.879s
Both with cold cache (echo 3 > /proc/sys/vm/drop_caches)
So it's not even a high build load. I've tried with -smp 8 (which showed
worse numbers than -smp 2 and 4), with IDE instead of virtio, without
-cpu host parameter but can't get near 30 (I've got 35 seconds with
tuning read_ahead_kb, on top of IDE instead of virtio, etc.) seconds at all.
I've also tried hugetlbfs for backing the memory within the guest.
I'm using the latest kvm-mod-2.6.32 built on top of 2.6.30.9.
So is this huge performance difference should be accepted as normal or
am I missing some big things?
Thanks a lot
Ozan Caglayan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Jeremy Fitzhardinge
> We have an HP Proliant DL580G5 rack server. It has 4 Intel Xeon X7460(6
> core, 2.67GHz, 16MB L3) processor with 32GB of memory. /proc/cpuinfo has
> 24 x the following entry:
>
>
>
> it for a virtualization server for giving people dedicated *guest*
> access for their personal compile-farm needs.
>
Why are you running a 32-bit kernel on this machine? Sounds like a bad
mismatch.
J
Avi Kivity
> Hi,
>
> We have an HP Proliant DL580G5 rack server. It has 4 Intel Xeon X7460(6
> core, 2.67GHz, 16MB L3) processor with 32GB of memory. /proc/cpuinfo has
> 24 x the following entry:
>
> it for a virtualization server for giving people dedicated *guest*
> access for their personal compile-farm needs.
>
First, as Jeremy notes, large memory machines want an x86_64 kernel.
But that doesn't explain the slowness.
> For testing purposes, we created a guest VM (2.6.30.9 too) on top of it
> with 2GB of virtual memory stored in a raw partition:
>
> qemu-kvm -cpu host -smp 2 -m 2047 -drive
> file=/dev/cciss/c1d0p1,if=virtio,cache=none,boot=on -net
> nic,model=virtio,macaddr=DE:AD:BE:EF:10:28 -net
> tap,ifname=tap0,script=/usr/bin/qemu-ifup -k tr -nographic -daemonize
>
>
> The problem is that I'm seeing very poor performance within the guest.
> I've googled a bit and seen similar bug reports/discussions ending with
> some tweaks (setting rotational to 1 for virtio_blk, using cache=none,
> etc.) and an explanation from Avi Kivity about the bad scalability of
> KVM on pre-Nehalem boxes under high build load.
>
-smp 2 should work perfectly well.
First, are you sure that kvm is enabled? 'info kvm' in the monitor.
Second, is the workload cpu bound or I/O bound? Both from the guest's
and host's point of view.
--
error compiling committee.c: too many arguments to function
Ozan Çağlayan
>
> Why are you running a 32-bit kernel on this machine? Sounds like a
> bad mismatch.
>
> J
Hmm, that's because we're using/developing a Linux distribution which
doesn't have 64-bit support yet. But since the guest will be a
compile-farm for our needs, it's worth trying to run an 64-bit distro on
the host to see the difference, thanks for the tip but how much should
this affect the performance?
Thanks.
Ozan Çağlayan
>
> First, are you sure that kvm is enabled? 'info kvm' in the monitor.
kvm support: enabled
(qemu) info cpus
* CPU #0: pc=0x00000000c011ceb3 (halted) thread_id=26023
CPU #1: pc=0x00000000c011ceb3 (halted) thread_id=26024
(qemu) info block
virtio0: type=hd removable=0 file=/dev/cciss/c1d0p1 ro=0 drv=host_device
encrypted=0
ide1-cd0: type=cdrom removable=1 locked=0 [not inserted]
floppy0: type=floppy removable=1 locked=0 [not inserted]
sd0: type=floppy removable=1 locked=0 [not inserted]
-------
KVM related messages on the host server:
~ # dmesg |grep kvm
[ 3649.058945] loaded kvm module (kvm-kmod-2.6.32)
[ 3683.736737] kvm: emulating exchange as write (It's interesting, what
does it mean?)
On the guest side:
~ # dmesg | grep kvm
[ 0.000000] kvm-clock: cpu 0, msr 0:64a3e1, boot clock
[ 0.000000] kvm-clock: cpu 0, msr 0:20103e1, primary cpu clock
[ 0.065075] kvm-clock: cpu 1, msr 0:20203e1, secondary cpu clock
>
> Second, is the workload cpu bound or I/O bound? Both from the guest's
> and host's point of view.
>
First of all if there are some specific outputs that will reveal the
IO/CPU boundness of the workload, I'd like to give them too. I'm not
much familiar with diagnostic/analysis tools..
From host's POV:
----------------
Guest is idle:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26117 root 20 0 2153m 175m 1356 S 5 0.5 0:33.57 qemu-kvm
Guest is configuring CUPS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26117 root 20 0 2129m 180m 1356 S 143 0.6 0:44.34 qemu-kvm (
<-- peak at 140,150~ %CPU)
load average reached 0.91 at the end of the configuration.
SMP view of top command shows that only Cpu2 and Cpu9 are involved,
other ones are zeroed (maybe coherent with -smp 2, don't have much idea):
Cpu2 : 28.4%us, 31.4%sy, 0.0%ni, 40.2%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu9 : 27.7%us, 26.7%sy, 0.0%ni, 45.5%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
vmstat -n 1 output on host:
procs -----------memory---------- ---swap-- -----io---- -system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy
id wa st
.
.
.
2 0 0 30161636 117300 116416 0 0 0 36 20366 3868 2
2 96 0 0 (./configure)
1 0 0 30161468 117300 116416 0 0 0 4 21673 4153 2
3 95 0 0
2 0 0 30160848 117300 116416 0 0 0 0 16549 4215 2
2 95 0 0
2 0 0 30160396 117300 116416 0 0 0 0 21089 4196 2
3 95 0 0
1 0 0 30160308 117300 116416 0 0 0 0 17031 4189 2
3 95 0 0
1 0 0 30160088 117300 116420 0 0 0 128 21772 4184 2
3 95 0 0
4 0 0 30159848 117304 116416 0 0 0 28 22378 4117 2
3 95 0 0
2 0 0 30159704 117312 116420 0 0 0 20 23530 4082 2
3 95 0 0
1 0 0 30159624 117312 116420 0 0 0 0 20997 4098 2
3 95 0 0
1 0 0 30159608 117324 116416 0 0 0 40 18519 4290 2
3 95 0 0
1 0 0 30159600 117324 116416 0 0 0 76 20959 4187 2
3 95 0 0
1 0 0 30159764 117324 116416 0 0 0 0 19718 4205 2
3 95 0 0
1 0 0 30159848 117324 116416 0 0 0 0 19934 4095 2
3 95 0 0
1 0 0 30159724 117324 116416 0 0 0 0 21821 4094 2
3 95 0 0
1 0 0 30159684 117328 116412 0 0 0 16 19621 4093 2
2 95 0 0
1 0 0 30159848 117328 116416 0 0 0 76 19630 4162 2
3 95 0 0
1 0 0 30159724 117328 116416 0 0 0 0 22810 4033 2
3 95 0 0
1 0 0 30159848 117328 116416 0 0 0 0 20525 4105 2
3 95 0 0
1 0 0 30159560 117328 116416 0 0 0 0 15715 4188 2
2 95 0 0
1 0 0 30159516 117328 116416 0 0 0 0 17686 4223 2
2 95 0 0
1 0 0 30159476 117328 116416 0 0 0 76 17308 4174 2
2 95 0 0
1 0 0 30159888 117328 116416 0 0 0 0 17973 4186 2
3 95 0 0
1 0 0 30159848 117328 116416 0 0 0 0 18795 4147 2
3 95 0 0
1 0 0 30159808 117328 116416 0 0 0 0 18346 4181 2
2 95 0 0
1 0 0 30160096 117328 116416 0 0 0 0 17054 4202 2
2 95 0 0
1 0 0 30159892 117328 116416 0 0 0 76 16767 4256 2
2 95 0 0
2 0 0 30159600 117328 116416 0 0 0 8 19817 4191 2
3 95 0 0
2 0 0 30159600 117328 116416 0 0 0 0 19709 4125 2
3 95 0 0
1 0 0 30159476 117328 116416 0 0 0 0 17322 4153 2
2 95 0 0
1 0 0 30159600 117328 116416 0 0 0 0 20106 4189 2
3 95 0 0
1 0 0 30159808 117328 116416 0 0 0 80 17810 4202 2
2 95 0 0
1 0 0 30159848 117328 116416 0 0 0 0 16723 4217 2
2 95 0 0
1 0 0 30160052 117328 116416 0 0 0 0 17799 4140 2
2 95 0 0
1 0 0 30159560 117328 116416 0 0 0 0 15878 4244 2
2 95 0 0
2 0 0 30159764 117328 116416 0 0 0 0 19632 4124 2
3 95 0 0
1 0 0 30160144 117328 116416 0 0 0 188 18119 4255 2
2 95 0 0
1 0 0 30159860 117328 116416 0 0 0 0 17664 4139 2
2 95 0 0
2 0 0 30159856 117328 116416 0 0 0 0 23323 4099 2
3 95 0 0
2 0 0 30159776 117328 116416 0 0 0 12 30120 3790 2
3 94 0 0
1 0 0 30159484 117328 116416 0 0 0 0 25599 4172 2
3 95 0 0
2 0 0 30159732 117328 116416 0 0 0 128 41030 3579 2
4 94 0 0
2 0 0 30159528 117328 116416 0 0 0 0 39064 3644 2
4 94 0 0
1 0 0 30159816 117328 116416 0 0 0 0 39337 3556 2
4 94 0 0
1 0 0 30159940 117328 116416 0 0 0 0 41252 3617 2
4 94 0 0
2 0 0 30160112 117328 116416 0 0 0 0 36845 3668 2
3 94 0 0
2 0 0 30160484 117328 116416 0 0 0 116 39011 3737 2
4 94 0 0
1 0 0 30160196 117328 116416 0 0 0 0 38782 3646 2
4 94 0 0
1 0 0 30160072 117328 116416 0 0 0 0 38546 3684 2
4 94 0 0
1 0 0 30160444 117328 116416 0 0 0 0 37738 3641 2
4 94 0 0
0 0 0 30160336 117328 116416 0 0 0 20 26019 3596 2
3 95 0 0
0 0 0 30160460 117328 116416 0 0 0 184 2373 2286 0
0 100 0 0 (finished)
0 0 0 30160460 117328 116416 0 0 0 0 2398 2279 0
0 100 0 0
0 0 0 30160460 117328 116416 0 0 0 0 2356 2251 0
0 100 0 0
0 0 0 30160460 117328 116416 0 0 0 0 2395 2275 0
0 100 0 0
0 0 0 30160460 117328 116416 0 0 0 0 2354 2249 0
0 100 0 0
0 0 0 30160460 117328 116416 0 0 0 0 2395 2283 0
0 100 0 0
0 0 0 30160460 117328 116416 0 0 0 0 2367 2230 0
0 100 0 0
0 0 0 30160460 117328 116416 0 0 0 0 2396 2277 0
0 100 0 0
0 0 0 30160460 117328 116416 0 0 0 0 2344 2223 0
0 100 0 0
I also checked iotop on host and qemu-kvm appears occasionally with disk
writes at max 200 K/s.
From guest's POV:
-----------------
Configuring CUPS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19258 root 20 0 3396 1656 1192 R 18 0.1 0:05.20 configure (
Saw maximum 22 %CPU)
The following is the highest load average instant during configuration:
top - 11:35:32 up 33 min, 3 users, load average: 0.56, 0.25, 0.15
Tasks: 99 total, 4 running, 95 sleeping, 0 stopped, 0 zombie
Cpu0 : 4.7%us, 68.3%sy, 0.0%ni, 27.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu1 : 4.3%us, 66.3%sy, 0.0%ni, 29.3%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 2061660k total, 137564k used, 1924096k free, 58672k buffers
Swap: 307192k total, 0k used, 307192k free, 49240k cached
vmstat -n 1:
------------
(context switches and interrupts are much more higher during
configuration than the idle state)
procs -----------memory---------- ---swap-- -----io---- -system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy
id wa st
0 0 0 1926012 58552 49172 0 0 27 4 165 46 1 8
91 0 0
0 0 0 1925956 58552 49176 0 0 0 0 33 18 0 0
100 0 0
0 0 0 1925832 58560 49172 0 0 0 32 71 78 0 0
100 0 0
0 0 0 1925832 58560 49176 0 0 0 0 42 21 0 0
100 0 0
0 0 0 1925832 58560 49176 0 0 0 0 74 73 0 1
99 0 0
0 0 0 1925832 58560 49176 0 0 0 0 26 15 0 0
100 0 0
0 0 0 1925832 58560 49176 0 0 0 0 95 216 0 0
100 0 0
0 0 0 1925832 58560 49176 0 0 0 0 33 35 0 0
100 0 0
0 0 0 1925832 58568 49176 0 0 0 12 143 166 1 0
99 0 0
0 0 0 1925840 58568 49176 0 0 0 0 181 224 0 1
99 0 0
0 0 0 1925880 58568 49176 0 0 0 0 67 71 0 0
100 0 0
0 0 0 1925880 58568 49176 0 0 0 0 22 12 0 0
100 0 0
2 0 0 1925880 58568 49176 0 0 0 0 108 119 0 0
100 0 0
0 0 0 1925624 58576 49176 0 0 0 12 222 212 1 4
95 0 0
0 0 0 1925540 58576 49176 0 0 0 0 220 185 0 2
98 0 0
0 0 0 1925500 58576 49176 0 0 0 0 34 26 0 0
100 0 0
0 0 0 1925508 58576 49176 0 0 0 12 164 121 0 1
98 0 0
0 0 0 1925508 58576 49176 0 0 0 0 54 59 0 0
100 0 0
0 0 0 1925508 58584 49176 0 0 0 36 81 87 0 0
100 0 0
0 0 0 1925508 58584 49176 0 0 0 0 47 30 0 1
100 0 0
0 0 0 1925508 58584 49176 0 0 0 0 77 80 0 0
100 0 0
0 0 0 1925508 58584 49176 0 0 0 0 71 78 0 0
100 0 0
0 0 0 1925436 58584 49176 0 0 0 0 175 216 0 0
100 0 0
0 0 0 1925428 58584 49176 0 0 0 0 80 66 0 1
99 0 0
0 0 0 1925428 58592 49176 0 0 0 48 84 91 0 0
100 0 0
1 0 0 1924788 58592 49176 0 0 0 0 1413 401 2 33
65 0 0 (./configure)
1 0 0 1924092 58592 49072 0 0 0 4 2210 599 5 57
38 0 0
1 0 0 1924544 58592 49080 0 0 0 0 2131 349 5 51
45 0 0
1 0 0 1924580 58592 49088 0 0 0 0 2188 564 5 54
42 0 0
2 0 0 1924796 58592 49092 0 0 0 0 2153 386 5 53
42 0 0
2 0 0 1924892 58604 49092 0 0 0 112 2148 409 5 54
41 0 0
1 0 0 1924732 58604 49096 0 0 0 0 2153 415 5 56
39 0 0
1 0 0 1924916 58604 49100 0 0 0 0 2197 508 5 56
39 0 0
1 0 0 1924788 58604 49100 0 0 0 0 2134 376 5 56
40 0 0
1 0 0 1924612 58604 49100 0 0 0 4 2199 522 6 52
42 0 0
1 0 0 1924772 58612 49100 0 0 0 84 2184 405 6 53
41 0 0
3 0 0 1924960 58612 49116 0 0 0 0 2187 453 5 53
42 0 0
1 0 0 1924600 58612 49112 0 0 0 0 2156 474 5 54
42 0 0
1 0 0 1924164 58612 49128 0 0 0 0 2151 420 5 55
40 0 0
2 0 0 1924944 58612 49116 0 0 0 0 2130 357 5 53
42 0 0
2 0 0 1924536 58620 49120 0 0 0 80 2218 691 6 54
41 0 0
1 0 0 1924252 58620 49144 0 0 0 0 2121 358 5 54
41 0 0
1 0 0 1924888 58620 49128 0 0 0 0 2158 422 6 54
41 0 0
1 0 0 1922456 58620 49132 0 0 0 0 2125 386 6 50
44 0 0
1 0 0 1924696 58620 49144 0 0 0 0 2195 451 5 53
43 0 0
2 0 0 1924780 58628 49140 0 0 0 80 2143 379 6 51
44 0 0
2 0 0 1924660 58628 49144 0 0 0 0 2151 389 6 51
44 0 0
1 0 0 1924380 58628 49144 0 0 0 0 2130 315 7 50
43 0 0
1 0 0 1924504 58628 49152 0 0 0 0 2169 400 6 52
42 0 0
2 0 0 1924548 58628 49156 0 0 0 0 2126 320 6 51
44 0 0
1 0 0 1924660 58636 49156 0 0 0 80 2161 367 5 52
43 0 0
1 0 0 1924412 58636 49160 0 0 0 8 2133 323 5 52
43 0 0
1 0 0 1924392 58636 49176 0 0 0 0 2182 575 5 55
40 0 0
1 0 0 1924516 58636 49180 0 0 0 0 2128 318 6 52
43 0 0
1 0 0 1923572 58636 49180 0 0 0 0 2164 432 7 51
43 0 0
1 0 0 1924192 58644 49184 0 0 0 80 2164 451 5 52
43 0 0
1 0 0 1923980 58644 49192 0 0 0 0 2160 389 7 50
44 0 0
2 0 0 1924528 58644 49192 0 0 0 0 2122 318 6 51
43 0 0
2 0 0 1924448 58644 49212 0 0 0 0 2166 418 7 51
43 0 0
1 0 0 1923872 58644 49200 0 0 0 72 2137 322 6 52
43 0 0
1 0 0 1923276 58652 49208 0 0 0 84 2171 401 5 53
42 0 0
1 0 0 1923052 58652 49220 0 0 0 0 2175 554 6 54
41 0 0
1 0 0 1924736 58652 49220 0 0 0 0 2172 395 6 54
40 0 0
2 0 0 1924560 58652 49100 0 0 0 4 2145 1591 6 59
36 0 0
2 0 0 1924460 58652 49212 0 0 0 28 2212 573 5 60
36 0 0
3 0 0 1924060 58660 49224 0 0 0 96 2201 583 5 63
33 0 0
1 0 0 1924128 58660 49240 0 0 0 0 2189 611 5 67
28 0 0
1 0 0 1923668 58660 49240 0 0 0 0 2169 652 6 67
28 0 0
4 0 0 1924212 58660 49240 0 0 0 0 2225 705 4 68
28 0 0
1 0 0 1924136 58660 49240 0 0 0 0 2158 571 5 66
29 0 0
1 0 0 1924176 58672 49228 0 0 0 152 2211 693 4 69
27 0 0
2 0 0 1924072 58672 49240 0 0 0 0 2169 569 5 67
29 0 0
4 0 0 1924180 58672 49240 0 0 0 0 2183 620 5 68
28 0 0
1 0 0 1924140 58672 49240 0 0 0 0 2208 811 4 68
28 0 0
1 0 0 1924240 58672 49244 0 0 0 20 2190 606 4 64
32 0 0
0 0 0 1924800 58672 49236 0 0 0 0 646 1352 3 18
78 0 0
0 0 0 1924808 58684 49236 0 0 0 200 111 162 0 1
99 0 0
0 0 0 1924808 58684 49236 0 0 0 0 74 84 0 0
100 0 0 (finished)
0 0 0 1924808 58684 49236 0 0 0 0 139 170 0 0
100 0 0
0 0 0 1924808 58684 49236 0 0 0 0 53 67 0 0
99 0 0
Thanks a lot.
Ozan Çağlayan
> Avi Kivity wrote:
>> First, are you sure that kvm is enabled? 'info kvm' in the monitor.
> (qemu) info kvm
> kvm support: enabled
Resending it as my mail client wrapped all the lines, sorry..
(qemu) info kvm
kvm support: enabled
(qemu) info cpus
* CPU #0: pc=0x00000000c011ceb3 (halted) thread_id=26023
CPU #1: pc=0x00000000c011ceb3 (halted) thread_id=26024
(qemu) info block
virtio0: type=hd removable=0 file=/dev/cciss/c1d0p1 ro=0 drv=host_device encrypted=0
ide1-cd0: type=cdrom removable=1 locked=0 [not inserted]
floppy0: type=floppy removable=1 locked=0 [not inserted]
sd0: type=floppy removable=1 locked=0 [not inserted]
-------
KVM related messages on the host server:
~ # dmesg |grep kvm
[ 3649.058945] loaded kvm module (kvm-kmod-2.6.32)
[ 3683.736737] kvm: emulating exchange as write (It's interesting, what does it mean?)
On the guest side:
~ # dmesg | grep kvm
[ 0.000000] kvm-clock: cpu 0, msr 0:64a3e1, boot clock
[ 0.000000] kvm-clock: cpu 0, msr 0:20103e1, primary cpu clock
[ 0.065075] kvm-clock: cpu 1, msr 0:20203e1, secondary cpu clock
First of all if there are some specific outputs that will reveal the
IO/CPU boundness of the workload, I'd like to give them too. I'm not
much familiar with diagnostic/analysis tools..
From host's POV:
----------------
Guest is idle:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26117 root 20 0 2153m 175m 1356 S 5 0.5 0:33.57 qemu-kvm
Guest is configuring CUPS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26117 root 20 0 2129m 180m 1356 S 143 0.6 0:44.34 qemu-kvm (peak at 140,150~ %CPU)
Configuring CUPS:
19258 root 20 0 3396 1656 1192 R 18 0.1 0:05.20 configure (Saw maximum 22 %CPU)
vmstat -n 1:
------------
Jeremy Fitzhardinge
> Hmm, that's because we're using/developing a Linux distribution which
> doesn't have 64-bit support yet. But since the guest will be a
> compile-farm for our needs, it's worth trying to run an 64-bit distro on
> the host to see the difference, thanks for the tip but how much should
> this affect the performance?
32-bit PAE is marginal above 4G of memory and the limit is at about 8G.
Above that, so much lowmem (<~1GB) is occupied by page and other kernel
structures that there's barely any memory to do any real work. If this
is what's happening, you'd expect to see poor host performance, and slow
guest performance would be secondary to that.
With 64-bit, all the memory can be directly addressed so none of this
applies. I would expect to see significant performance gains.
There's nothing to stop you from running 32-bit PAE guests, so you can
do your distro development that way (at least in part).
J
Avi Kivity
>
> KVM related messages on the host server:
>
> ~ # dmesg |grep kvm
> [ 3649.058945] loaded kvm module (kvm-kmod-2.6.32)
> [ 3683.736737] kvm: emulating exchange as write (It's interesting, what does it mean?)
>
An exchange instruction (cmpxchg, likely) was emulated using a write.
Since it happens under lock, this is safe.
> First of all if there are some specific outputs that will reveal the
> IO/CPU boundness of the workload, I'd like to give them too. I'm not
> much familiar with diagnostic/analysis tools..
>
> From host's POV:
> ----------------
>
> Guest is idle:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 26117 root 20 0 2153m 175m 1356 S 5 0.5 0:33.57 qemu-kvm
>
> Guest is configuring CUPS:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 26117 root 20 0 2129m 180m 1356 S 143 0.6 0:44.34 qemu-kvm (peak at 140,150~ %CPU)
>
>
Definitely looks cpu bound.
> load average reached 0.91 at the end of the configuration.
>
> SMP view of top command shows that only Cpu2 and Cpu9 are involved,
> other ones are zeroed (maybe coherent with -smp 2, don't have much idea):
>
> Cpu2 : 28.4%us, 31.4%sy, 0.0%ni, 40.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu9 : 27.7%us, 26.7%sy, 0.0%ni, 45.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>
Can you provide kvm_stat output while this happens?
> Configuring CUPS:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19258 root 20 0 3396 1656 1192 R 18 0.1 0:05.20 configure (Saw maximum 22 %CPU)
>
Do you have any idea what this configure does? If it is a very
fork/exec intensive workload, you can expect some slowdown on older
processors.
> The following is the highest load average instant during configuration:
>
> top - 11:35:32 up 33 min, 3 users, load average: 0.56, 0.25, 0.15
> Tasks: 99 total, 4 running, 95 sleeping, 0 stopped, 0 zombie
> Cpu0 : 4.7%us, 68.3%sy, 0.0%ni, 27.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu1 : 4.3%us, 66.3%sy, 0.0%ni, 29.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem: 2061660k total, 137564k used, 1924096k free, 58672k buffers
> Swap: 307192k total, 0k used, 307192k free, 49240k cached
>
> vmstat -n 1:
> ------------
>
> (context switches and interrupts are much more higher during
> configuration than the idle state)
>
> 1 0 0 1924788 58592 49176 0 0 0 0 1413 401 2 33 65 0 0 (./configure)
> 1 0 0 1924092 58592 49072 0 0 0 4 2210 599 5 57 38 0 0
> 1 0 0 1924544 58592 49080 0 0 0 0 2131 349 5 51 45 0 0
>
That's consistent with a fork/exec load, though the numbers are not that
high.
--
error compiling committee.c: too many arguments to function
--
Ozan Çağlayan
> On 12/13/09 02:12, Ozan Çağlayan wrote:
>> Hmm, that's because we're using/developing a Linux distribution which
>> doesn't have 64-bit support yet. But since the guest will be a
>> compile-farm for our needs, it's worth trying to run an 64-bit distro on
>> the host to see the difference, thanks for the tip but how much should
>> this affect the performance?
>
> 32-bit PAE is marginal above 4G of memory and the limit is at about 8G.
> Above that, so much lowmem (<~1GB) is occupied by page and other kernel
> structures that there's barely any memory to do any real work. If this
> is what's happening, you'd expect to see poor host performance, and slow
> guest performance would be secondary to that.
Updates on the issue after 3 months:
------------------------------------
The host is now running an x86_64 2.6.32.9 kernel with kvm-2.6.32.7 built as an external module.
The guest is 32-bit 2.6.32.9.
I've booted the guest with -smp 4 -m 4096
and I'm seeing the same slowness that I've mentioned in this thread during CUPS configuration.
I'll try to send kvm_stat output in this week and try the same configure on another guest OS to see if it differs or not.
I'm also thinking to build an optimized 2.6.33 for that rack server to see if things get better or not.
Thanks,
Ozan Caglayan
Avi Kivity
> Jeremy Fitzhardinge wrote:
>
>> On 12/13/09 02:12, Ozan Çağlayan wrote:
>>
>>> Hmm, that's because we're using/developing a Linux distribution which
>>> doesn't have 64-bit support yet. But since the guest will be a
>>> compile-farm for our needs, it's worth trying to run an 64-bit distro on
>>> the host to see the difference, thanks for the tip but how much should
>>> this affect the performance?
>>>
>> 32-bit PAE is marginal above 4G of memory and the limit is at about 8G.
>> Above that, so much lowmem (<~1GB) is occupied by page and other kernel
>> structures that there's barely any memory to do any real work. If this
>> is what's happening, you'd expect to see poor host performance, and slow
>> guest performance would be secondary to that.
>>
> Updates on the issue after 3 months:
> ------------------------------------
>
> The host is now running an x86_64 2.6.32.9 kernel with kvm-2.6.32.7 built as an external module.
> The guest is 32-bit 2.6.32.9.
>
You're running an old kvm on a new kernel. It's simpler and better to
run the kvm that came with 2.6.32.9.
> I've booted the guest with -smp 4 -m 4096
>
> and I'm seeing the same slowness that I've mentioned in this thread during CUPS configuration.
> I'll try to send kvm_stat output in this week and try the same configure on another guest OS to see if it differs or not.
>
> I'm also thinking to build an optimized 2.6.33 for that rack server to see if things get better or not.
>
Better to find out what's wrong first and build kernels later.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.