DRBD performance
Jorge Cabello
I am having an odd performance issue with DRBD. When DRBD is used the
performance of disks writes is really bad. Looking at documentation (ganeti and
drbd) it seems this shouldn't happen. Anyone has had similar problems?
It seems some kind of problem beetween DRBD and LVM as:
1) KVM on top of LVM without DRBD works fine (performance test A)
2) DRBD syncronization works "fine" (performance test C)
3) Network works fine (performance test B)
4) KVM on top of LVM with DRBD doesn't work as expected (performance test D, E)
Any kind of help will be welcome as I'm out of ideas.
Thanks.
Test environment:
* The nodes use a separate gigabit network for DRBD.
* Linux: Debian Squeeze x86_64
* Linux Kernel: 2.6.32-5-amd64
* DRBD: version: 8.3.7 (api:88/proto:86-91) on both nodes
* KVM: version 0.12.5 on both nodes
* LVM: 2.02.66(2), Version: 4.15.0 on both nodes
* Instances disk type is paravirtual
* We have played with DRBD max-epoch-size, max-buffers, unplug-watermark, al-
extents, no-barriers and rate parameters without reaching more than 10MB/s.
==================================================
Performance:
A) Plain disk instance (no DRBD): (~44MB)
$ dd if=/dev/zero of=file bs=8k count= count=20000
20000+0 records in
20000+0 records out
163840000 bytes (164 MB) copied, 3,74366 s, 43,8 MB/s
B) Copying (scp) from node2 to node1 using DRBD network: (~48MB)
100% 391MB 48.8MB/s 00:08
C) Resync of instances: (~28MB)
$ cat /proc/drbd
0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
ns:0 nr:401928 dw:8485156 dr:0 al:0 bm:354 lo:1579 pe:0 ua:1579 ap:0 ep:1
wo:b oos:6316
[==================>.] sync'ed: 99.0% (6316/401888)K
finish: 0:00:00 speed: 32,376 (28,252) K/sec
D) From inside instance (DRBD Protocol C): (~6MB)
dd if=/dev/zero of=file bs=8k count=20000
20000+0 records in
20000+0 records out
163840000 bytes (164 MB) copied, 26,008 s, 6,3 MB/s
E) From inside instance (DRBD Protocol B): (~6MB)
$ dd if=/dev/zero of=file bs=8k count=20000
20000+0 records in
20000+0 records out
163840000 bytes (164 MB) copied, 29,4272 s, 5,6 MB/s
Iustin Pop
> Hi,
>
> I am having an odd performance issue with DRBD. When DRBD is used the
> performance of disks writes is really bad. Looking at documentation (ganeti and
> drbd) it seems this shouldn't happen. Anyone has had similar problems?
>
> It seems some kind of problem beetween DRBD and LVM as:
> 1) KVM on top of LVM without DRBD works fine (performance test A)
> 2) DRBD syncronization works "fine" (performance test C)
> 3) Network works fine (performance test B)
> 4) KVM on top of LVM with DRBD doesn't work as expected (performance test D, E)
>
> Any kind of help will be welcome as I'm out of ideas.
OK, this might be related to your question, but one of the reasons we
still use Xen and not KVM is that I/O performance is much more
predictable. For Xen, our internal benchmarks show that there is indeed
a performance degradation when using DRBD of 50% but not more (for large
block sizes).
I don't know if in your case KVM is the problem, or some other stuff,
just saying that we have found Xen more consistent.
However, your numbers are really low, so I suspect there's something
else.
> Test environment:
> * The nodes use a separate gigabit network for DRBD.
> * Linux: Debian Squeeze x86_64
> * Linux Kernel: 2.6.32-5-amd64
> * DRBD: version: 8.3.7 (api:88/proto:86-91) on both nodes
> * KVM: version 0.12.5 on both nodes
> * LVM: 2.02.66(2), Version: 4.15.0 on both nodes
> * Instances disk type is paravirtual
> * We have played with DRBD max-epoch-size, max-buffers, unplug-watermark, al-
> extents, no-barriers and rate parameters without reaching more than 10MB/s.
Just 10MB/s? Strange.
> ==================================================
> Performance:
>
> A) Plain disk instance (no DRBD): (~44MB)
> $ dd if=/dev/zero of=file bs=8k count= count=20000
> 20000+0 records in
> 20000+0 records out
> 163840000 bytes (164 MB) copied, 3,74366 s, 43,8 MB/s
You're not using oflag=direct, and bs=8k is *really small*, so this is
not really meaningful.
Try to use oflag=direct consistently across your tests to ensure
reliable data.
> B) Copying (scp) from node2 to node1 using DRBD network: (~48MB)
> 100% 391MB 48.8MB/s 00:08
scp will use big buffers (not 8k), and it's also doing cached writes.
But ack that the network is good.
> C) Resync of instances: (~28MB)
> $ cat /proc/drbd
> 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
> ns:0 nr:401928 dw:8485156 dr:0 al:0 bm:354 lo:1579 pe:0 ua:1579 ap:0 ep:1
> wo:b oos:6316
> [==================>.] sync'ed: 99.0% (6316/401888)K
> finish: 0:00:00 speed: 32,376 (28,252) K/sec
This tells us that the node itself can do good writes.
> D) From inside instance (DRBD Protocol C): (~6MB)
> dd if=/dev/zero of=file bs=8k count=20000
> 20000+0 records in
> 20000+0 records out
> 163840000 bytes (164 MB) copied, 26,008 s, 6,3 MB/s
Again, missing odirect and small block size. What do you get for this?
dd if=/dev/zero of=file bs=64k count=5000 oflag=direct
> E) From inside instance (DRBD Protocol B): (~6MB)
> $ dd if=/dev/zero of=file bs=8k count=20000
> 20000+0 records in
> 20000+0 records out
> 163840000 bytes (164 MB) copied, 29,4272 s, 5,6 MB/s
thank,
iustin
Jorge Cabello
show a performance degradation of around 50%.
Without DRBD)
$ dd if=/dev/zero of=file bs=64k count=5000 oflag=direct
5000+0 records in
5000+0 records out
327680000 bytes (328 MB) copied, 62,1705 s, 5,3 MB/s
With DRBD)
$ dd if=/dev/zero of=file bs=64k count=5000 oflag=direct
5000+0 records in
5000+0 records out
327680000 bytes (328 MB) copied, 132,032 s, 2,5 MB/s
As you have pointed it seems to be a KVM problem. I'm going to search
information about IO performance problems in KVM and I will keep you updated.
Thanks!
Iustin Pop
> Using bs=64k and oflag=direct numbers are more consistent (but really worst) and
> show a performance degradation of around 50%.
Which is what we've seen for Xen too, so expected.
> Without DRBD)
> $ dd if=/dev/zero of=file bs=64k count=5000 oflag=direct
> 5000+0 records in
> 5000+0 records out
> 327680000 bytes (328 MB) copied, 62,1705 s, 5,3 MB/s
Even without DRBD, just 5MB/s? Wow.
> With DRBD)
> $ dd if=/dev/zero of=file bs=64k count=5000 oflag=direct
> 5000+0 records in
> 5000+0 records out
> 327680000 bytes (328 MB) copied, 132,032 s, 2,5 MB/s
Really low, indeed.
> As you have pointed it seems to be a KVM problem. I'm going to search
> information about IO performance problems in KVM and I will keep you updated.
For the record, on a normal machine (SATA disks, nothing fancy), with a
standard Ganeti install (Xen, DRBD 8.3.7), I get this:
$ dd if=/dev/zero of=bigf bs=64k count=5000 oflag=direct
5000+0 records in
5000+0 records out
327680000 bytes (328 MB) copied, 9.38699 seconds, 34.9 MB/s
Increasing the block size results in:
$ dd if=/dev/zero of=bigf bs=1024k count=256 oflag=direct
256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 4.72125 seconds, 56.9 MB/s
Which is about the max we usually get.
iustin
Lance Albertson
> On Thu, Mar 03, 2011 at 10:37:12AM +0100, Jorge Cabello wrote:
>> As you have pointed it seems to be a KVM problem. I'm going to search
>> information about IO performance problems in KVM and I will keep you updated.
>
> For the record, on a normal machine (SATA disks, nothing fancy), with a
> standard Ganeti install (Xen, DRBD 8.3.7), I get this:
>
> $ dd if=/dev/zero of=bigf bs=64k count=5000 oflag=direct
> 5000+0 records in
> 5000+0 records out
> 327680000 bytes (328 MB) copied, 9.38699 seconds, 34.9 MB/s
>
> Increasing the block size results in:
>
>
> $ dd if=/dev/zero of=bigf bs=1024k count=256 oflag=direct
> 256+0 records in
> 256+0 records out
> 268435456 bytes (268 MB) copied, 4.72125 seconds, 56.9 MB/s
>
> Which is about the max we usually get.
I was curious what performance we got on our clusters and it was
surprisingly similar to what Jorge encountered in some cases.
prod cluster: 9MB/s (6-SAS disks, 18 instances per node)
blade cluster: 3MB/s (2-SAS disks, 4 instances per node)
sc cluster: 33MB/s (6-SAS disks, w/ DRBD)
49MB/s w/o DRBD
phpbb cluster: 72MB/s (6-SCSI disks, 4-5 instances per node)
All the clusters except phpbb are using DRBD. We use KVM on all our
clusters so I can't test Xen.
--
Lance Albertson
Systems Administrator / Architect Open Source Lab
Network Services Oregon State University
Iustin Pop
> On 03/03/2011 01:41 AM, Iustin Pop wrote:
> > On Thu, Mar 03, 2011 at 10:37:12AM +0100, Jorge Cabello wrote:
> >> As you have pointed it seems to be a KVM problem. I'm going to search
> >> information about IO performance problems in KVM and I will keep you updated.
> >
> > For the record, on a normal machine (SATA disks, nothing fancy), with a
> > standard Ganeti install (Xen, DRBD 8.3.7), I get this:
> >
> > $ dd if=/dev/zero of=bigf bs=64k count=5000 oflag=direct
> > 5000+0 records in
> > 5000+0 records out
> > 327680000 bytes (328 MB) copied, 9.38699 seconds, 34.9 MB/s
> >
> > Increasing the block size results in:
> >
> >
> > $ dd if=/dev/zero of=bigf bs=1024k count=256 oflag=direct
> > 256+0 records in
> > 256+0 records out
> > 268435456 bytes (268 MB) copied, 4.72125 seconds, 56.9 MB/s
> >
> > Which is about the max we usually get.
>
> I was curious what performance we got on our clusters and it was
> surprisingly similar to what Jorge encountered in some cases.
>
> prod cluster: 9MB/s (6-SAS disks, 18 instances per node)
> blade cluster: 3MB/s (2-SAS disks, 4 instances per node)
> sc cluster: 33MB/s (6-SAS disks, w/ DRBD)
> 49MB/s w/o DRBD
This is interesting. It shows that on the sc cluster, you have *much*
higher bandwidth than on the others. Is there a difference in software
versions (kernel/KVM especially)?
thanks,
iustin
Martin Beauchamp
Martín
From: "Iustin Pop" <iu...@k1024.org>
To: gan...@googlegroups.com
Sent: Thursday, March 3, 2011 1:27:53 PM
Subject: Re: DRBD performance
Lance Albertson
The sc cluster doesn't have very many VMs on it so its basically sitting
idle. Its also the newest hardware out of all the clusters I listed. The
prod cluster is the busiest and using a slightly older kernel than the
rest while sc probably is using the newest. The hardware in prod is also
a few years old although the HDD's are fairly new.
As far as versions go this is what they are:
sc cluster: 2.6.35-gentoo-r13 kernel, app-emulation/qemu-kvm-0.12.5-r1,
sys-cluster/drbd-8.3.8.1, using drbd module in mainline
prod: 2.6.33-gentoo, app-emulation/qemu-kvm-0.12.5-r1, sys-cluster/drbd-
8.3.7, using drbd module in mainline
I have a few more clusters I could do some testing on but I picked the
most busy/newest/fastest ones I had. Also all guests are Gentoo.
Serge van Ginderachter
About a small year ago, I have been conducting some various tests on
comparing different virtualisation technologies on different
platforms.
* tested VMWare ESXi, Xen on Debian and CentOS and KVM on Ubuntu and CentOS
* these tests have nothing to do with Ganeti, nor was there a test with DRBD,
* on Xen and KVM I tested using HVM/Virtio - not so on VMWare
* made different tests, stressing CPU, disk, memory and network on 1
till 10 concurrent vm's.
* client vm machines were all Debian Lenny
Some general conclusions were:
* VMWare ESXi was the clear winner,
* Xen on Debian Lenny was a close second, and the winner amongst Open
Source solutions.
* Xen performance is way better than KVM, especially when looking at disk access
* Xen performs in a more stable and predictable way, whilst KVM seemed
to perform more at random (which confirms Iustin's observations, )
* CentOS (5.4) performed remarkably well for being older sofwtare
versions (KVM, Xen, Linux kernel)
* performance on Ubuntu was really bad. The then recent Ubuntu Lucid
was far worse than CentOS 5.5 (both KVM)
* Disk speed on bare metal was 80MB/s
* on vm, those dropped to 40-20MB/s depending on the platform end
thenumber of concurrent access (= number of tested vm's)
Also, testing latency (lots of small writes): time dd if=/dev/zero
of=/dev/vda5 bs=512 count=100000 oflag=direct
* 4.6 MB/s on bare metal
* 728 KB/s on vm Ubuntu + KVM
* on vm Debian Lenny + Xen:don't have the number anymore but noted a
performance drop of only 20%
HTH,
--
Met vriendelijke groet,
Serge van Ginderachter
Miguel Di Ciurcio Filho
> Hi,
>
> I am having an odd performance issue with DRBD. When DRBD is used the
> performance of disks writes is really bad. Looking at documentation (ganeti and
> drbd) it seems this shouldn't happen. Anyone has had similar problems?
>
> It seems some kind of problem beetween DRBD and LVM as:
> 1) KVM on top of LVM without DRBD works fine (performance test A)
> 2) DRBD syncronization works "fine" (performance test C)
> 3) Network works fine (performance test B)
> 4) KVM on top of LVM with DRBD doesn't work as expected (performance test D, E)
>
What is KVM's command line? Are you using virtio? Did you change the elevators?
Here is what I do:
For maximum DRBD throughput:
- I use a secondary Gigabit switch for DRBD, so on the network
interface of the "replication" network, change the MTU to a higher
value, this will depend on your network card. Mine is 6000. Increase
the txqueuelen too, mine is 5000 packets. Also, increase the host's
tcpbacklog to 5000 too: sysctl -w net.core.netdev_max_backlog=5000.
This will make sure that DRBD have plenty of room on the network.
For the disks:
- Use virtio. I'm very aggressive and use writeback as caching method too.
- Change the elevator on the host and on the VM to deadline (Google
more about it). This may vary, you will have to experiment for your
workload.
echo deadline > /sys/block/vda/queue/scheduler
You didn't specify if there is any type of RAID setup involved. I use
a software RAID5 with crappy 1TB SATA2 disks on both hosts, and here
are my results:
[root@recife ~]# dd if=/dev/zero of=file bs=8k count=20000
20000+0 records in
20000+0 records out
163840000 bytes (164 MB) copied, 0.272316 seconds, 602 MB/s
Well this is totally a lie because caused by my use of writeback on
virtio, but it is good, isn't it? :-D
Something more realistic, using O_DIRECT:
dd if=/dev/zero of=file bs=8k count=20000 oflag=direct
20000+0 records in
20000+0 records out
163840000 bytes (164 MB) copied, 2.69948 seconds, 60.7 MB/s
Hosts are 2 x 4 quad core and 16GB of RAM and onboard Intel ICH10 SATA
controller. No fakeraid.
DRBD is 100% managed by Ganeti.
Regards,
Miguel
Eric Rostetter
> I had been curious about DRBD disk performance in the past and took
> steps to optimize my setup. I'd like to add the following results:
So did I, I guess you could say... ;)
> CentOS 5.5 [...] 2.6.18-194.17.1.el5 [...] kvm-83-164.el5_5.21
Same but kernel 2.6.18-194.26.1.el5
> (2) Dell R610 w/RAID-6 (6 SAS 10,000k drives)
(2) Dell R310 w/RAID-10 (4 SATA DISKS)
> (3) 1 Gbps links using Linux bonding in round robin mode
(2) 1 Gbps links using linux bonding mode=4
> On a drbd instance:
> [root@stg-a ~]# dd if=/dev/zero of=bigf bs=1024k count=256 oflag=direct
[...]
> 268435456 bytes (268 MB) copied, 3.4105 seconds, 78.7 MB/s
268435456 bytes (268 MB) copied, 2.64135 seconds, 102 MB/s
268435456 bytes (268 MB) copied, 2.49242 seconds, 108 MB/s
(those are numbers for a VM on each of the two machines)
> On a the master node (no drbd):
> [root@node-a ~]# dd if=/dev/zero of=bigf bs=1024k count=256 oflag=direct
> 268435456 bytes (268 MB) copied, 2.97024 seconds, 90.4 MB/s
268435456 bytes (268 MB) copied, 0.803979 seconds, 334 MB/s
> This node is not under heavy utilization at the moment. I'd be
> curious to see more benchmarks.
My nodes are under very low utilization at the moment. :)
22:04:25 up 96 days, 58 min, 1 user, load average: 0.08, 0.06, 0.03
> Best,
> Martín
--
Eric Rostetter
The Department of Physics
The University of Texas at Austin
Go Longhorns!
Manfred Herrmann
- I/O-scheduler type > deadline (default scheduler bad on multi-threaded I/O > concurrent VMs)
- blk (not file/image / blk is ganeti default?)
- virtio blk (qemu -drive file=/dev/mapper/ImagesVolumeGroup-Guest1,cache=none,if=virtio)
- cache=none (qemu -drive file=/dev/mapper/ImagesVolumeGroup-Guest1,cache=none,if=virtio)
- prefer ext3 (not ext4)
- ??? please add more
Bas Tichelaar
On Mar 26, 6:16 pm, Manfred Herrmann <herrmann.manf...@googlemail.com>
wrote:
>
> Benchmarking disk I/O needs a well defined testcase and reproducable SUT
> (system under test). Different static and dynamic parameters change
> test-results a lot.
>
> KVM-blk-I/O tuning areas:
>
> - I/O-scheduler type > deadline (default scheduler bad on multi-threaded
> I/O > concurrent VMs)
> - blk (not file/image / blk is ganeti default?)
> - virtio blk (qemu -drive
> file=/dev/mapper/ImagesVolumeGroup-Guest1,cache=none,*if=virtio*)
> - cache=none (qemu -drive file=/dev/mapper/ImagesVolumeGroup-Guest1,*
> cache=none*,if=virtio)
> - prefer ext3 (not ext4)
> - ??? please add more