DRBD performance measurment / settings

[Lucas, Sascha]

Hi,

I'm doing benchmarking on DRBD to figure out, how it performs. My benchmark is a random read/write¹. I've done following observations:

* the native IOPS performance of a single node without drbd is more then 2 times faster then with drbd (second node is identical to the first one)

* Tuning parameters (disk-barriers: bf, meta-barriers: True, net-custom: --max-buffers 8000 --max-epoch-size 8000 --unplug-watermark=16-8000) have no significant influence

* the primary node does lots of I/Os on the meta-LV

* the primary node's load rices to 64 (= queue depth?)

Can someone confirm my observations? Or is this a normal DRBD behavior? My Hardware is HW RAID10 with 6x SAS10k and flash-backed write-cache.

What can I expect placing the meta-LV(VG) on a SSD? can I reach close to native IOPS?

BTW I think, that following ganeti parameters have invers logic compared to drbd itself:

* disk-barriers: n = none means barriers/flush/drain enabled, while with drbdsetup(8) none means no write ordering
* meta-barriers: True means disabled meta-flushes, but True is sometimes understood as Enabled

[1] "fio iometer-file-access-server": it works on a 4GB test file with DIRECTIO (no pagecache involved), it does mixed read/write (80/20) on different block sizes (wighted from 512 to 64k) with a queue depth of 64.

Thanks, Sascha.

Vorsitzender des Aufsichtsrates: Ralf Hiltenkamp
Geschäftsführung: Michael Krüger (Sprecher), Stephan Drescher
Sitz der Gesellschaft: Halle/Saale
Registergericht: Amtsgericht Stendal | Handelsregister-Nr. HRB 208414
UST-ID-Nr. DE 158253683
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Empfänger sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder des Inhalts dieser Mail sind nicht gestattet. Diese Kommunikation per E-Mail ist nicht gegen den Zugriff durch Dritte geschützt. Die GISA GmbH haftet ausdrücklich nicht für den Inhalt und die Vollständigkeit von E-Mails und den gegebenenfalls daraus entstehenden Schaden. Sollte trotz der bestehenden Viren-Schutzprogramme durch diese E-Mail ein Virus in Ihr System gelangen, so haftet die GISA GmbH - soweit gesetzlich zulässig - nicht für die hieraus entstehenden Schäden.

[Thomas Thrainer]

Hi Sascha,

Probably your question concerning performance would be better placed at the DRBD user list (drbd...@lists.linbit.com).

You are right that the disk-barriers and meta-barriers parameters are not named perfectly. They are, however, properly documented. Changing their names would probably mean more hassles for development and during cluster upgrades than the improved readability would be worth it.

Thanks,

Thomas

--

Thomas Thrainer | Software Engineer | thom...@google.com | 

Google Germany GmbH

Dienerstr. 12

80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

[John N.]

Hi Sascha,

Just a small thought here: have you already tried using 2 CPUs and more memory (2GB for example) for your dom0? Depending if you have a high i/o load this should in theory from what I read help.

Best,
John

[Jake Anderson]

You are tuning the wrong end.

There is some stuff in the wiki about it but the simple things are aio=native and cache=none that should improve your performance to around 80 percent raw and fix the system load. Assuming your drbd sync rate is high enough.

Sent from Samsung Galaxy Note

[Lucas, Sascha]

Hi John,

From: John N.
Date: Wed, 29. Mai 2013 23:52

> Just a small thought here: have you already tried using 2 CPUs and more memory (2GB for example) for your dom0? Depending if you have a high i/o load this should in theory from what I read help.

I already noticed that most $FILESYSTEMS perform better with many CPUs. What I missed to tell in the starting thread is that I use KVM, so there is no dom0.

[Lucas, Sascha]

Hi Jake,

From: Jake Anderson
Date: Thu, 30. May 2013 01:17

> You are tuning the wrong end.

I just followed the DRBD tuning guide: http://www.drbd.org/users-guide-8.3/s-throughput-tuning.html

> There is some stuff in the wiki about it but the simple things are aio=native and cache=none that should improve your performance to around 80 percent raw and fix the system load.

That is a very interesting point. It belongs to KVM tuning wrt I/O. cache=none can be set with Ganeti. And I already have this set, else my benchmark results are one order of magnitude better then on the host (because all reads are cached on the host side, physical disks just need to do the writes).

I can't set aio=native via Ganeti, right? This setting is often recommended by IBM, redhat etc. and I think libvirt defaults to this. What's your way to set it?

[Jake Anderson]

i believe aio=native is setable with ganeti now
https://groups.google.com/forum/?fromgroups#!topic/ganeti/nAHZPzVGccc
I haven't looked into it I'm still running an old version and using a wrapper to do cpu=host etc

http://code.google.com/p/ganeti/wiki/PerformanceTuning has some more info

[Lucas, Sascha]

Answering myself:

From: Lucas, Sascha
Date: Wed, 29. Mai 2013 16:13

> What can I expect placing the meta-LV(VG) on a SSD? can I reach close
> to native IOPS?

Yes. I've put the meta-LV of an instance on a SSD and my benchmark shows the same result over DRBD just like on a single node without DRBD/ssd.

[Lucas, Sascha]

From: Lucas, Sascha
Date: Wed, 29. May 2013 16:13

> * the native IOPS performance of a single node without drbd is more
> then 2 times faster then with drbd (second node is identical to the
> first one)

FYI: I've changed the I/O scheduler from CFQ to deadline or noop (node and instance). Now I have 100% the same performance in a DRBD based KVM instance like on a plain filesystem (no DRBD) on the host/node.

Of course there are still the same amount of extra I/Os on the meta-LV, but now the RAID controller can handle these totally in cache. Why to this enterprise distros have CFQ as default? :-).

[Ansgar Jazdzewski]

Hi Sascha

Tanks for the hint, so i have to Do some updates on my Puppet modules.