What to do when DRBD is stalled

[Roberto Espinoza]

Hello!

Today we had to power cycle our nodes for some external maintenance.

Two instances didn't come up, KVM actually was started but there was a DRBD problem.

The problem was that the DRBD device used for those two instances was stalled (one at 67%, another at 31%).

The information from ganeit was just "UNCERTAIN STATE" but with drbd-overview I could see the inconsistency.

Anyway, what is the procedure here?

How can I tell ganeti to start the sync again or how can I safely shutdown the DRBD devices?

I tried to shutdown the instances but as you may guess, Ganeti couldn't shutdown the DRBD devices.

At the end, I had to power cycle the nodes involved with the problem and everything worked just fine but I would like to know if it is possible to fix this problem from the command line.

Also when I try running command with drbdadm, for example "drbdadm verify" it said no resources where available which makes sense as this is handled by ganeti, so there wasn't much for me to do.

I am also not very familiar with all the commands for DRBD devices and the consequences of doing this, specifically, how this can affect the Ganeit operation.

Any tips appreciated.

Thank you,

Roberto

[Guido Trotter]

Hi Roberto,

drbdadm only works for "configured" drbd devices (with the config file
and such), what you need in order to interact with kernel level drbd
devices is drbdsetup. There you'll find commands to shutdown,
invalidate, and do other nasty tortures to your drbds. Check it out
and see if it helps, if not we can also try to help you with some of
our internal usage guides (which I don't have handy at the moment).

Best regards,

Guido

>
>
> Any tips appreciated.
>
>
> Thank you,
> Roberto

--
Guido Trotter
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370

[Jean-François Maeyhieux]

In case of DRBD split brain ( drbd not in sync anymore: drbd devices from /proc/drbd are seen as Primary/Unknown or Secondary/Unknown):

You want to discard the data of one drbd node and trust the other one to recontruct the drbd device as a RAID 1 disk.

1) To start, you need to know the drbd resource involved (DRBD minor, device and port):

[MASTER NODE] # gnt-instance info $INSTANCE | grep -A9 -- "- disk/"

    - disk/0: drbd, size 20.0G

      access mode: rw

      nodeA: node01.mycloud.net, minor=4                                 *** DRBD MINOR ***

      nodeB: node02.mycloud.net, minor=4                                 *** DRBD MINOR ***

      port: 11020                                                                       *** DRBD PORT ***

      auth key: 6af168277b7abc986b0d2c3abf12348ab5fa388f

      on primary: /dev/drbd4 (147:4) in sync, status ok                 *** DRBD DEVICE ***

      on secondary: /dev/drbd4 (147:4) in sync, status ok             *** DRBD DEVICE ***

      name: None

      UUID: ab12e4c3-dcb4-bc45-a340-36fff8924c2

Usually on ganeti, minor, device and resource keep the same number, but it's better to check it anyway.

Here, minor=4   device=drbd4   port=11020

2) Then you need to know the corresponding drbd resource:

# drbdsetup show $MINOR   (here: drbdsetup show 4)

resource resource4 {

    options {

    }

    net {

        cram-hmac-alg   "md5";

        shared-secret   "6af168277b7abc986b0d2c3abf12348ab5fa388f";

        after-sb-0pri   discard-zero-changes;

        after-sb-1pri   consensus;

    }

    _remote_host {

        address ipv4 192.168.0.2:11020;

    }

    _this_host {

        address ipv4 192.168.0.1:11020;

        volume 0 {

            device minor 4;

            disk "/dev/vg-cluster/d3764526-e655-4d59-bd8b-57333ee83497.disk0_data";

            meta-disk "/dev/vg-cluster/d3764526-e655-4d59-bd8b-57333ee83497.disk0_meta";

            disk {

                size             41943040s; # bytes

                resync-rate     1048576k; # bytes/second

            }

        }

    }

}

2) You want to discard data from node01 and rebuild your drbd device with data from node02:

- use drbdsetup to force primary/secondary if you have both primary or secondary:

drbdsetup primary $MINOR [--force]

drbdsetup secondary $MINOR [--force]

- disconnect the drbd device on each node:

[NODE01] drbdsetup disconnect 192.168.0.1:$PORT 192.168.0.2:$PORT --force=yes

[NODE02] drbdsetup disconnect 192.168.0.2:$PORT 192.168.0.1:$PORT --force=yes

- reconnect drbd device on each node discarding data on node01:

[NODE01] drbdsetup connect $RESOURCE 192.168.0.1:$PORT 192.168.0.2:$PORT --discard-my-data=yes

[NODE02] drbdsetup connect $RESOURCE 192.168.0.2:$PORT 192.168.0.1:$PORT

Note: If you can't reconnect here, you need to set the correct net part or empty it on both node to have the same net section on your drbd resource

=> drbd devices will resync fully, you could watch it via /proc/drbd

Here you're ok but your drbd resource should have an empty net section.

Stop and start the instance to fix it.

Jean-François

[Roberto Espinoza]

Hello Guido,

If you have the guides, I would appreciate it.

Thank you

[Roberto Espinoza]

Hello Jean-François,

Thank you for your detailed description.

In my case, there was no split-brain problem, the sync was just stalled.

What should be the procedure here?

Certainly I could discard the data but I don't think is needed in this case.

Like I said, I had to power cycle the two involved nodes so the sync could happen again, everything worked just fine, no problems.

Is there a way to force a reconnect or resync with the drbdsetup without the need to discard data?

Maybe Ganeti is down this internally, not sure, like I said, I have only minimal experience using the DRBD tools because pretty much all the operations are done by Ganeti.

Thank you.