how to remove ghost instance

[Raphael Thoma]

Hey there

Due to the bug mentioned in https://bugs.launchpad.net/ubuntu/+source/drbd8/+bug/1185756 we now have a ghost instance on one of our nodes:

Tue Apr 29 18:13:40 2014   - ERROR: cluster: ghost instance 'eccbe8be-d85d-4da8-b7ee-39f11d98b6f5' in temporary DRBD map

Unfortunately I didn't find a way to remove the instance so far. How can we cleanup this?

Cheers

Raphi

[Thomas Thrainer]

There are two reasons for this error to appear:

 * The temporary DRBD minor allocation was not cleaned up properly. In this case, it's enough to restart Ganeti on the master node.

 * The configuration is corrupt and there is really a broken instance somewhere. I doubt that's the case, as there would probably be more errors/warning then.

So could you please just restart the Ganeti master daemon and see if this fixes the situation?

Cheers,

Thomas

--

Thomas Thrainer | Software Engineer | thom...@google.com | 

Google Germany GmbH

Dienerstr. 12

80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Christine Elizabeth Flores

[Dieter Verhelst]

I have the exact same issue and the problem remains after restarting the ganeti from init.d

Ubuntu Trusty 14.04.2
Ganeti 2.14.1

I only have this 'issue' on one node.

---
Fri Jul 31 13:31:25 2015   - ERROR: cluster: ghost disk '66c9ce9d-8058-440b-8479-a614885c3617' in temporary DRBD map
Fri Jul 31 13:31:25 2015   - ERROR: cluster: ghost disk '39d3c715-5493-4943-b477-3690cffe547d' in temporary DRBD map
Fri Jul 31 13:31:25 2015   - ERROR: cluster: ghost disk 'c646e7c7-b952-4f96-9c90-0ac47f58f1da' in temporary DRBD map
...
---

Any other fixes I could try?

[Bo Kersey]

I had the same problem.  Restart of Ganeti did not fix it.
I found the offending disk ids in /var/lib/ganeti/tempres.data
I stopped ganeti, removed /var/lib/ganeti/tempres.data, started ganeti
gnt-cluster verify is now clean.

[Jean-François Maeyhieux]

Take care with only removing it from /var/lib/ganeti/tempres.data because the underlying LVM logical volume could still exist.

I got the same problem because I've try to add a disk to an instance B right after removing a disk on instance A (without hotplug option).

In fact removing a disk without hotplug and without have restart the corresponding instance have left a drbd on running instance node.

I think ganeti have try to allocate the removed drbd minor when trying to add the disk but since this drbd minor was still existing an a node, it failed.

This failure have left a ghosting disk.

To solve this situation, you have to find the ghost disk LV:

$ for n in $(gnt-node list --no-headers -o name); do ssh $n -- 'lvs --noheadings --separator " " -o lv_host,lv_path,lv_size,lv_tags | sed "s|originstname+||g"';done | sort -k 2 | uniq -c -f1 | grep '^ *1 '

then use lv-remove to delete it

Jean-François