Instance has disks in *DEGRADED* *UNCERTAIN STATE* after evacuate

[John N.]

Hello,

In order to reboot a node I did a gnt-node migrate and evacuate to get rid of all instances. The migration went well but unfortunately the evacuate did not.

The output of gnt-node evacuate is the following:

       1:2013-07-23 23:26:34.449584:message  - WARNING: Communication failure to node node1b.domain.tld: Error 52: Empty reply from server
        2:2013-07-23 23:26:34.473885:message Replacing disk(s) 0 for instance 'dnsa2.domain.tld'
        3:2013-07-23 23:26:34.486700:message Current primary node: node1a.domain.tld
        4:2013-07-23 23:26:34.497959:message Current seconary node: node2a.domain.tld
        5:2013-07-23 23:26:34.508358:message STEP 1/6 Check device existence
        6:2013-07-23 23:26:34.520174:message  - INFO: Checking disk/0 on node1a.domain.tld
        7:2013-07-23 23:26:34.741285:message  - INFO: Checking volume groups
        8:2013-07-23 23:26:34.894084:message STEP 2/6 Check peer consistency
        9:2013-07-23 23:26:34.909832:message  - INFO: Checking disk/0 consistency on node node1a.domain.tld
        10:2013-07-23 23:26:35.332237:message STEP 3/6 Allocate new storage
        11:2013-07-23 23:26:35.350610:message  - INFO: Adding new local storage on node1b.domain.tld for disk/0
        12:2013-07-23 23:26:36.483138:message STEP 4/6 Changing drbd configuration
        13:2013-07-23 23:26:36.594757:message  - INFO: activating a new drbd on node1b.domain.tld for disk/0
        14:2013-07-23 23:26:37.951522:message  - INFO: Shutting down drbd for disk/0 on old node
        15:2013-07-23 23:26:38.525043:message  - INFO: Detaching primary drbds from the network (=> standalone)
        16:2013-07-23 23:26:39.034398:message  - INFO: Updating instance configuration
        17:2013-07-23 23:26:49.175566:message Copy of file /var/lib/ganeti/config.data to node node1b.domain.tld failed: Error 52: Empty reply from server
        18:2013-07-23 23:26:49.190128:message Copy of file /var/lib/ganeti/config.data to node node2b.domain.tld failed: Error 52: Empty reply from server
        19:2013-07-23 23:26:49.204236:message Copy of file /var/lib/ganeti/config.data to node node2a.domain.tld failed: Error 52: Empty reply from server
        20:2013-07-23 23:26:49.218991:message  - INFO: Attaching primary drbds to new secondary (standalone => connected)
        21:2013-07-23 23:28:50.968477:message  - WARNING: Can't attach drbd disks on node node1b.domain.tld: Error 52: Empty reply from server
        22:2013-07-23 23:28:50.990314:message       Hint: please do a gnt-instance info to see the status of disks
        23:2013-07-23 23:28:51.003257:message  - WARNING: Can't attach drbd disks on node node1a.domain.tld: Timeout in disk reconnecting
        24:2013-07-23 23:28:51.016014:message       Hint: please do a gnt-instance info to see the status of disks
        25:2013-07-23 23:28:51.026057:message STEP 5/6 Sync devices
        26:2013-07-23 23:28:51.039155:message  - INFO: Waiting for instance dnsa2.domain.tld to sync disks
        27:2013-07-23 23:29:03.830639:message  - INFO: Instance dnsa2.domain.tld's disks are in sync


I suppose what generated this problem is that earlier on that same day I have replaced the switch connecting all the nodes, that's all really what changed.

My instance is mysteriously still working but a gnt-instance info shows the following disk statuses:

      on primary:   /dev/drbd2 (147:2) in sync, status *DEGRADED*
      on secondary: /dev/drbd2 (147:2) in sync, status *DEGRADED* *UNCERTAIN STATE*

Afaik this is pretty bad... I already tried a gnt-cluster verify-disks but that did not help. What should I do? I am a bit loss here and would really appreciate some help to get the disks back in sync.

Regards,
John

[Thomas Thrainer]

You could try a `gnt-instance activate-disks`, which will hopefully reconnect the DRBD nodes which should start the sync automatically.

Cheers,

Thomas

--

Thomas Thrainer | Software Engineer | thom...@google.com | 

Google Germany GmbH

Dienerstr. 12

80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores

[John N.]

On Wednesday, July 24, 2013 9:08:22 AM UTC+2, Thomas Thrainer wrote:

Hi Thomas,

You could try a `gnt-instance activate-disks`, which will hopefully reconnect the DRBD nodes which should start the sync automatically.

I'm impressed, this worked for both of my instances which had the degraded state... I always thought that activate-disks was just to use if you wanted to mount an instance's disk locally on the node. I'll have to read more about that command but if I understand correctly my issue here was that the DRBD device and config somehow have not been created correctly. Or what exactly did activate-disks do to fix this state?

Best,
J.

[Thomas Thrainer]

Hi,

`gnt-instance activate-disks` just makes sure that disks are active (either for instances to use them, or for the admin to mount them). It is OK to activate disks multiple times, they won't get deactivated before.

In the case of DRBD activate disks checks the state of the DRBD device on both nodes. The DRBD state is actually two things: the state of the local disk and the state of the remote disk.

In your case, DRBD probably lost the connection between the two nodes (maybe because of changing the switch). So the remote disk state on both nodes is broken, and activate disks will issue the proper commands to re-connect the nodes (while leaving the local disk in place). DRBD then figures out that a sync is required and performs it.

DRBD on its own does not, as far as I know, re-establish a connection once it's lost.

We considered it a bug/missing feature that Ganeti does not automatically fix this problem for you. So in 2.9 `gnt-cluster verify` will report instances in such a state, `gnt-cluster verify-disks` will issue the `gnt-instance activate-disks` automatically for you, and as the watcher is calling `gnt-cluster verify-disks` regularly, the cluster will heal itself essentially ;-).

Cheers,

Thomas

[John N.]

On Wednesday, July 24, 2013 10:08:33 AM UTC+2, Thomas Thrainer wrote:

`gnt-instance activate-disks` just makes sure that disks are active (either for instances to use them, or for the admin to mount them). It is OK to activate disks multiple times, they won't get deactivated before.

In the case of DRBD activate disks checks the state of the DRBD device on both nodes. The DRBD state is actually two things: the state of the local disk and the state of the remote disk.

In your case, DRBD probably lost the connection between the two nodes (maybe because of changing the switch). So the remote disk state on both nodes is broken, and activate disks will issue the proper commands to re-connect the nodes (while leaving the local disk in place). DRBD then figures out that a sync is required and performs it.

DRBD on its own does not, as far as I know, re-establish a connection once it's lost.

Thanks for your comprehensive explanation, it all makes sense to me now and it must definitely be because of swapping the switch, I was already worried that something is not setup correctly on my ganeti cluster... I didn't know that DRBD does not automatically re-initiate the connection between two nodes if it's lost.

We considered it a bug/missing feature that Ganeti does not automatically fix this problem for you. So in 2.9 `gnt-cluster verify` will report instances in such a state, `gnt-cluster verify-disks` will issue the `gnt-instance activate-disks` automatically for you, and as the watcher is calling `gnt-cluster verify-disks` regularly, the cluster will heal itself essentially ;-).

I totally agree, this is quite surprising in fact from DRBD as this means human intervention is in theory required if the state of the network changes... Happy to hear that you will implement a workaround to this issue in 2.9... what a pitty it's not yet implemented already in 2.7 ;-)

Best,
J.

[Valentin Lazarev]

John, did you solve this problem? After 'gnt-instance activate-disks instance_name' i have
      on primary:   /dev/drbd2 (147:0) in sync, status *DEGRADED*
      on secondary: /dev/drbd2 (147:0) in sync, status *DEGRADED* *UNCERTAIN STATE*
again =(

[Guido Trotter]

I believe in this case you need a bit more than "activate-disks". You
may want to check the drbd manual, check that the data on the device
is correct, and then manually mark the data as good.

drbdsetup /dev/drbd2 primary --force

will force the local data to be up to date again, and resync the secondary.

Be careful to make sure your underlying disks exist (lvm is allright
on the two nodes, no missing partitions, gnt-cluster verify happy,
etc)

Thanks,

Guido

[Sokhushu]

Hi Ultrotter,

Can you please help my vm disks show
on primary:   /dev/drbd0 (147:0) in sync, status *DEGRADED*
      on secondary: /dev/drbd0 (147:0) in sync, status *DEGRADED* *UNCERTAIN STATE*, after running "gnt-instance activate-disks instancename"

I have also run "drbdsetup /dev/drbd2 primary --force "

But there was no change.

gnt-cluster veirfy is far from being happy:
Verifying node status
Thu Nov 21 10:59:03 2013   - ERROR: on NodeA: drbd minor 38 of instance EDUTEST is not active
Thu Nov 21 10:59:03 2013   - ERROR: on NodeB: while contacting node: Error while executing backend function: 'spice_ip_version'
Thu Nov 21 10:59:03 2013 * Verifying instance status
Thu Nov 21 10:59:03 2013   - ERROR: instance ACC: couldn't retrieve status for disk/0 on NodeB: Can't find device <DRBD8(hosts=NodeB/39-nodeA/39, port=11139, configured as xx.xx.xx.2:11139 xx.xx.xx.1:11139, backend=<LogicalVolume(/dev/all/cf482dba-ac1a-4fc5-9116-eb04489f2fb0.disk0_data, not visible, size=20480m)>, metadev=<LogicalVolume(/dev/all/cf482dba-ac1a-4fc5-9116-eb04489f2fb0.disk0_meta, not visible, size=128m)>, visible as /dev/disk/0, size=20480m)>
Thu Nov 21 10:59:03 2013   - ERROR: node NodeB: instance ACC, connection to secondary node failed
Thu Nov 21 10:59:03 2013   - ERROR: instance ADOBE: couldn't retrieve status for disk/0 on NodeA: Can't find device <DRBD8(hosts=NodeA/15-NOdeB/15, port=11089, configured as xx.xx.xx.2:11089 xx.xx.xx.1:11089, backend=<LogicalVolume(/dev/all/97915b2e-2ab3-46e6-95c2-9fb195fa84cc.disk0_data, not visible, size=102400m)>, metadev=<LogicalVolume(/dev/all/97915b2e-2ab3-46e6-95c2-9fb195fa84cc.disk0_meta, not visible, size=128m)>, visible as /dev/disk/0, size=102400m)>
Thu Nov 21 10:59:03 2013   - ERROR: node NodeB: instance ADOBE, connection to secondary node failed
Thu Nov 21 10:59:03 2013   - ERROR: node NodeB : instance AFTLR, connection to primary node failed


I think I have a split brain problem and unable to resolve it. Please help i have been trying to fix this with no win.

Regards,
Sakhamuzi Hadebe