drbd sync very slow

[Seth Galitzer]

I'm running ganeti 2.15.2 on debian 9.3 (stretch) with kvm. When creating instances with disk type drbd, the sync rate is very slow. Settings for sync rates (c-max-rate, resync-rate) are set to 60M, but dstat is showing I'm only getting about 30-35M over 1Gbps link. Network is healthy, NICs are on same network segment (same switch even), linking at 1G/full-duplex, and iperf is showing near-1G throughput. I've tried tuning drbd per https://docs.linbit.com/doc/users-guide-84/s-throughput-tuning/, but it has made no effect. Disks are high performance enterprise SSDs in RAID0 arrays. What can I do to improve performance?

Thanks.

Seth

[candlerb]

I have seen RAID0 SSD arrays drop to 10MB/sec per disk sequential write performance after only a minute or two (not using DRBD - just using direct copy from NFS to SSD).  I believe it's to do with the drives heating up and thermal throttling becoming active.

To check if this is happening for you, try using smartctl to check the drive temperatures.  If they rapidly rise from (say) 25 degrees to 40 degrees when your DRBD syncing takes place, you may be seeing the same issue.  You can also try creating a large logical volume directly on the RAID0 array, and copying data to it directly, to test performance independently of DRBD.

[Iustin Pop]

You need to understand what limits the performance of the sync, if
possible.

Is any of the disks (source, destination) maxing at 100% utilization? Is
a single CPU core on either source or destination maxing out? Etc.

regards,
iustin

[Bosko Koprivica]

This is increasing drbd sync. It would be good to put this as default inside cluster drbd config, didn't figured out if that's possible.

Primary:

drbdsetup disk-options --c-min-rate=300M <resource number>

Secondary:

drbdsetup disk-options --c-min-rate=300M <resource number>
drbdsetup disk-options --c-fill-target=20M <resource number>
drbdsetup show <resource number> - then take a look at address:port for _this_host and _remote_host and then do this:

drbdsetup net-options <_this_host address:port> <_remote_host address:port> --max-buffers=36k --sndbuf-size=1024k --rcvbuf-size=2048k

Regards,
Bosko Koprivica

[candlerb]

> This is increasing drbd sync. It would be good to put this as default inside cluster drbd config, didn't figured out if that's possible.

gnt-cluster modify --disk-parameters drbd:foo=XXX,bar=YYY,...

Full details in "man gnt-cluster": http://docs.ganeti.org/ganeti/2.15/html/man-gnt-cluster.html#init

Some selected parameters which may be relevant to what you want are:

       resync-rate

              Static re-synchronization rate.  [KiB/s]

       dynamic-resync

              Boolean  indicating  whether  to  use  the  dynamic resync speed controller or not.  If enabled, c-plan-ahead must be

              non-zero and all the c-* parameters will be used by DRBD.  Otherwise, the value of resync-rate  will  be  used  as  a

              static resync speed.

       c-plan-ahead

              Agility  factor  of the dynamic resync speed controller.  (the higher, the slower the algorithm will adapt the resync

              speed).  A value of 0 (that is the default) disables the controller.  [ds]

       c-fill-target

              Maximum amount of in-flight resync data for the dynamic resync speed controller.  [sectors]

       c-min-rate

              Minimum resync speed for the dynamic resync speed controller.  [KiB/s]

       net-custom

              String containing additional parameters to be appended to the arguments list of drbdsetup net.