Disk Backlog 10 minutes.

[Adam J. McPartlan]

Hi,

Netdata is reporting a disk backlog of around 10 minutes on a node in my cluster.

The VM is heavy on the IOs. I am assuming (stabbing in the dark) the disk is not fast enough to cope or the DRDB replication is sluggish.

Looking at Ganeti it shows a degraded primary disk. I am unsure what to do to correct this and thought I'd ask for some opinions/advice.

It's not affecting the cluster and a gnt-cluster verify reports nothing unusual...

      on primary: /dev/drbd0 (147:0) in sync, status *DEGRADED*
      name: None
      UUID: 710e9f84-a181-474b-8788-aad26ada0b34
      child devices:
        - child 0: plain, size 200.0G
          logical_id: vg0/603396dd-29ae-478c-86d1-23722e1da843.disk0_data
          on primary: /dev/vg0/603396dd-29ae-478c-86d1-23722e1da843.disk0_data (253:0)
          on secondary: /dev/vg0/603396dd-29ae-478c-86d1-23722e1da843.disk0_data (253:0)
          name: None
          UUID: 21f740fa-5720-40df-aa0c-3b7efaa25ac1
        - child 1: plain, size 128M
          logical_id: vg0/603396dd-29ae-478c-86d1-23722e1da843.disk0_meta
          on primary: /dev/vg0/603396dd-29ae-478c-86d1-23722e1da843.disk0_meta (253:1)
          on secondary: /dev/vg0/603396dd-29ae-478c-86d1-23722e1da843.disk0_meta (253:1)
          name: None
          UUID: fa1b0e39-0079-4f49-a247-5d727737ebf2

cat /proc/drbd
version: 8.4.11 (api:1/proto:86-101)
srcversion: FC3433D849E3B88C1E7B55C
 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
    ns:0 nr:0 dw:1453831088 dr:1501319775 al:56559455 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:183001276
 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:260983204 nr:0 dw:255965036 dr:8834543 al:13877 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 3: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:11987748 nr:0 dw:11988156 dr:1748798 al:2163 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
 4: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:64642468 nr:0 dw:65225092 dr:3152849 al:9298 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 7: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
    ns:0 nr:73400320 dw:73400320 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Thanks in advance for your time and patience.

Adam

[Sascha Lucas]

Hi Adam,

On Thu, 16 Jun 2022, Adam J. McPartlan wrote:

> The VM is heavy on the IOs. I am assuming (stabbing in the dark) the disk
> is not fast enough to cope or the DRDB replication is sluggish.

If there is no networking issue, DRBD stays in sync, regardless of the
performance of your disks or the IO-load of the VMs.

> It's not affecting the cluster and a gnt-cluster verify reports nothing
> unusual...
>

> on primary: /dev/drbd0 (147:0) in sync, status *DEGRADED*B

Please provide the output of "gnt-cluster verify-disks". This should
detect the degraded DRBD disks.

Normally a cron-job runs every 5 minutes on the master node, which should
repair degraded DRBD disks. You may run "ganeti-watcher" by hand, to see
if that helps.

If this does not help (which I assume), you are in a case, that ganeti can
not detect/repair reliable.

> cat /proc/drbd
> version: 8.4.11 (api:1/proto:86-101)
> srcversion: FC3433D849E3B88C1E7B55C
> 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r-----
> ns:0 nr:0 dw:1453831088 dr:1501319775 al:56559455 bm:0 lo:0 pe:0 ua:0
> ap:0 ep:1 wo:f oos:183001276

...

> 7: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
> ns:0 nr:73400320 dw:73400320 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
> wo:f oos:0

This two DRBD resources (0, 7) are out-of-sync (oos). For completeness,
can you also provide the content of /proc/drbd from the secondary node of
the instance with /dev/drbd0?

Resource 7 must belongs to an other instance... check to find/repair it,
too.

Last but not least, your problem should be fixed! "gnt-instance
activate-disks <inst>". Should to do it.

Thanks, Sascha.

[Adam McPartlan]

Thanks for your reply

> Please provide the output of "gnt-cluster verify-disks". This should
> detect the degraded DRBD disks

gnt-cluster verify-disks
Submitted jobs 144555
Waiting for job 144555 ...

No disks need to be activated.

  > This two DRBD resources (0, 7) are out-of-sync (oos). For completeness,
  > can you also provide the content of /proc/drbd from the secondary node of
  > the instance with /dev/drbd0?

Secondary Node:

 3: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:11992588 dw:11992996 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 6: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

  > Last but not least, your problem should be fixed! "gnt-instance
  > activate-disks <inst>". Should to do it.

gnt-instance activate-disks cpe-nms
xho1.domain.co.uk:disk/0:/dev/drbd0

Doesn't seem to have made a difference... as both now appear degraded. Apologies if I have misunderstood.

  Disks:
    - disk/0: drbd, size 200.0G
      access mode: rw
      nodeA: xho1.nynet.co.uk, minor=0
      nodeB: xho3.nynet.co.uk, minor=0
      port: 11001

      on primary: /dev/drbd0 (147:0) in sync, status *DEGRADED*

      on secondary: /dev/drbd0 (147:0) in sync, status *DEGRADED*
      name: None

  > Resource 7 must belongs to an other instance... check to find/repair it,
  > too.

Is there an easy way to find this out?

Thanks

Adam

[Sascha Lucas]

Hi Adam,

On Thu, 16 Jun 2022, Adam McPartlan wrote:

> Secondary Node:
>
> 3: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
> ns:0 nr:11992588 dw:11992996 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
> wo:f oos:0
>
> 6: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

...

> Disks:
> - disk/0: drbd, size 200.0G
> access mode: rw
> nodeA: xho1.nynet.co.uk, minor=0
> nodeB: xho3.nynet.co.uk, minor=0
> port: 11001
> on primary: /dev/drbd0 (147:0) in sync, status *DEGRADED*
> on secondary: /dev/drbd0 (147:0) in sync, status *DEGRADED*
> name: None

The provided content of /proc/drbd seems not match for the secondary node?
I expect a line stating with "0:" on xho3.

Can you please double check if there is really no "0:" on xho3?

Thanks, Sascha.

[Adam J. McPartlan]

> Can you please double check if there is really no "0:" on xho3?

D'oh... Sorry... Yes, it's there. A copy/paste fail.

 0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown   r-----

    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5066752

 3: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

    ns:0 nr:13234628 dw:13235036 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

 6: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

--
You received this message because you are subscribed to the Google Groups "ganeti" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ganeti+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ganeti/alpine.DEB.2.22.394.2206162353120.289572%40eyu.loc.

[Adam McPartlan]

Further to this... I have tried the following:

drbdsetup /dev/drbd0 invalidate-remote
/dev/drbd0: State change failed: (-15) Need a connection to start verify or resync

From the Primary:

 drbdsetup status
resource0 role:Primary
  disk:UpToDate
  peer connection:Connecting

resource1 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

resource3 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

resource4 role:Primary
  disk:UpToDate
  peer role:Secondary
    replication:Established peer-disk:UpToDate

resource7 role:Secondary
  disk:UpToDate
  peer connection:Connecting

From the Secondary:

 drbdsetup status
resource0 role:Secondary
  disk:UpToDate

resource3 role:Secondary
  disk:UpToDate
  peer role:Primary
    replication:Established peer-disk:UpToDate

resource6 role:Secondary
  disk:UpToDate
  peer role:Primary
    replication:Established peer-disk:UpToDate

All servers can see each other though I don't know why drbd does not sync for that instance.

Thanks again,

Adam

[Sascha Lucas]

Hi Adam,

On Fri, 17 Jun 2022, Adam J. McPartlan wrote:

>> Can you please double check if there is really no "0:" on xho3?
>
> D'oh... Sorry... Yes, it's there. A copy/paste fail.
>
> 0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r-----
> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:5066752

Ok, this looks somehow as expected. System logs (i.e. during
"activate-disks") should explain why DRBD connecting "0" fails. I guess
you have split brain. Ganeti can not solve this (but should warn).

Solving split brain can be done by invalidating the secondary copy: on
xho3 please try "drbdsetup invalidate 0". Afterwards connect DRBD by
running "gnt-instance activate-disks <your instance>".

Thanks, Sascha.

[Adam McPartlan]

Hi Sascha,

Thank you very much. This worked like a charm.

Where should I see the warning - is there a particular log file?

Thanks

Adam

[Sascha Lucas]

Hi Admam,

On Mon, 20 Jun 2022, Adam McPartlan wrote:

> Thank you very much. This worked like a charm.

good to hear.

> Where should I see the warning - is there a particular log file?

What I meant was, that Ganeti should warn, when running "gnt-cluster
verify-disks", but it does not ATM. This seems a bug?

This is why it is recommended to run additional DRBD monitoring on every
node.

The reason why "activate-disks" can't assemble the DRBD connection can be
found in the kernels ring buffer using "dmesg" or in the usual log file
location like /var/log/syslog etc.

On Thu, 16 Jun 2022, Adam J. McPartlan wrote:

> 7: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----

> ns:0 nr:73400320 dw:73400320 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
> wo:f oos:0

You still need to fix other DRBDs? Use the following to find the effected
instances:

$ drbdsetup show 7
...
_this_host {
...
disk "/dev/vg/3be453ea-a28b-454b-b4de-823cf90cd552.disk0_data";

$ lvs -o name,tags /dev/vg/3be453ea-a28b-454b-b4de-823cf90cd552.disk0_data
LV LV Tags
3be453ea-a28b-454b-b4de-823cf90cd552.disk0_data originstname+test.vm

Here it's test.vm.

Thanks, Sascha.