How add a second drbd disk based on another VG on a specific instance

[Jean-François Maeyhieux]

Hello,

    we have a ganeti cluster with different instances installed on drbd ( metavg: vg-cluster ).

Is it possible to add a second drbd disk on a specific instance with a storage located on another volume group (vg-data for exemple) ?

Jean-François

[Iustin Pop]

Have you tried:

gnt-instance modify --disk add:size=4g,vg=vg-data instance

I don't know if it works, let us know if it creates the volumes
correctly.

thanks,
iustin

[Jean-François Maeyhieux]

1) That works perfectly:

- that create the drbd disk, start the drbd sync between nodes and return on the shell directly.

- I've wait the sync to be effective by watching /proc/drbd

- I stop/start the instance and I could see the /dev/vdb

- I mkfs it and I add it manually to my fstab

- I mount it and I could start to use it

2) No sync information/notice:

I think when the command finish and return to the shell, we should read an advice that tell us to check the progression of the drbd sync before start to use the disk.

Something like:

     The sync of disk/1:/dev/drbd3 is in progress.

     You could check the progression of the sync with:  'watch cat /proc/drbd'

     Wait the sync is finish before start to use the device.

3) The command tell that the modification could be effective after the next start of the instance

I don't know if it's possible to do it and use the disk whithout shutdown the instance.

In my case, I've shutdown the instance before use it.

4) metadata on main VG

The second point is that the metadata drbd volume land on the main volume group and not on the command line specified one:

 - disk/1: drbd8, size 1.1T

      access mode:  rw

      auth key:     ce5b456a30412b651f72e92487b74a781ce68345

      on primary:   /dev/drbd3 (147:3) in sync, status ok

      on secondary: /dev/drbd3 (147:3) in sync, status ok

      child devices:

        - child 0: lvm, size 1.1T

          logical_id:   vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data

          on primary:   /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:17)               <= VG specified on command line to add the disk

          on secondary: /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:13)             <= VG specified on command line to add the disk

        - child 1: lvm, size 128M

          logical_id:   vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta

          on primary:   /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:18)                   <= main VG for instance

          on secondary: /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:14)                 <= main VG for instance

Is it normal ? could we have a way to specify it ?

Jean-Francois

Le lundi 12 novembre 2012 14:40:06 UTC+1, Iustin Pop a écrit :

[Guido Trotter]

Hi,

No, currently we don't have a feature to attach disks to a running instance.

As for the meta vg you can specify it with the metavg=... option to the same command. It might be a bug though that it didn't choose the default one, if it was specified.

Thanks for the resync message suggestion: would you file an issue for that?

Thanks

Guido

[Jean-François Maeyhieux]

Issue open:  http://code.google.com/p/ganeti/issues/detail?id=306

[Dimitris Aragiorgis]

Hi,

* Guido Trotter <ultr...@google.com> [2012-11-13 06:24:35 +0100]:

> Hi,
>
> No, currently we don't have a feature to attach disks to a running
> instance.
>

I have implemented a disk/nic hotplug feature for ganeti, that makes use of
specific qemu monitor commands and therefor kvm >=1.0 is needed. This is
premature but seems to work. All you have to do is enable hotplug when
creating an instance, by adding --hotplug option:

# gnt-instance add --hotplug ... vmname

and then pass this option when modifying the instance:

# gnt-instance modify --hotplug --disk add:... vmname
# gnt-instance modify --hotplug --disk 1:remove vmname
# gnt-instance modify --hotplug --net add:... vmname
# gnt-instance modify --hotplug --net 0:remove vmname
# gnt-instance modify --hotplug --net 0:... vmname

There are still some limitations about the current implementation. First the PCI
slots are partitioned in order to separate hot-plugable devices from the rest.
Secondly the first disk is not hot-plugable. Furthermore if a device is
hot-added it should be removed using hotplug. Still you can add/remove/modify
devices with the traditional way (i.e. without --hotplug option) and the
modification will take place after reboot).

In case you are interested for digging further into this please refer to:

https://code.grnet.gr/git/ganeti-local stable-2.6-ippool-hotplug-esi branch

The implementation (along with ippool and external storage interface) is build
on top of stable 2.6.1

I mention it here because it would be great to have someone else test it and
provide feedback or even make use of it, if it satisfies his/her needs.

Regards,
dimara

[Jean-François Maeyhieux]

Dimitris Aragiorgis: you make my day ! This hotplug patch is awesome.

So it's possible with it to do disk provisionning whithout instance down time.

That should be really usefull most especially with huge drbd based volume that could take severals hours of syncing.

I will try to take some time to try it and report eventual failure/sucess use case.

Do I only need to use this GIT branch to test it ?

What is implication of this branch on a prod environment ?

Do you think is it enough safe to test hotplug functionality on test instance next to production instance ?

Jean-Francois

[Dimitris Aragiorgis]

* Jean-François Maeyhieux <zen2...@gmail.com> [2012-11-14 02:03:18 -0800]:

>
> Dimitris Aragiorgis: you make my day ! This hotplug patch is awesome.
>

I am glad!

> So it's possible with it to do disk provisionning whithout instance down
> time.
> That should be really usefull most especially with huge drbd based volume
> that could take severals hours of syncing.
>
> I will try to take some time to try it and report eventual failure/sucess
> use case.
>

Feedback on this is more than welcome.

> Do I only need to use this GIT branch to test it ?

Yes. You can also use debian-2.6 branch and git-buildpackage to build the deb
package. Please note that this branch has also ip pool support (gnt-network ...)
and external storage interface support (-t ext --disk 0:size=10G,provider=...).
IP pool makes use of python-bitarray that is currently existing only in wheezy.
I have backported it to squeeze (you can get it from our local testing repo
deb http://apt.dev.grnet.gr sid main ). Because of ippool management you have
to run cfgupgrade (nodegroups get an extra slot: networks).

> What is implication of this branch on a prod environment ?

IP pool and extstorage are so far tested more that hotplug in our clusters.
All features are backward compatible extentions and do not break the rest
funtionality. Note that currently old instances can not use hotplug feature,
It should be enabled during instance creation (maybe a proper cfgupgrade could
bypass this limitation).

> Do you think is it enough safe to test hotplug functionality on test
> instance next to production instance ?
>

I don't see anything that could affect eachother.

Of course feel free to ask any implementation details or mention any suggestions/
improvements you come up with.


Thanks again for the interest,
dimara

[tschend]

Hi,

just wanted to drop a link about the hotplug patch.

I am using it currently and it works without problems.
It is a real time saver and makes ganeti even more useful!

Keep up the great work.

Kind Regards
Thomas

On 14 Nov., 12:27, Dimitris Aragiorgis <dim...@grnet.gr> wrote:
> * Jean-François Maeyhieux <zen2dr...@gmail.com> [2012-11-14 02:03:18 -0800]:

>
>
>
> > Dimitris Aragiorgis: you make my day ! This hotplug patch is awesome.
>
> I am glad!
>
> > So it's possible with it to do disk provisionning whithout instance down
> > time.
> > That should be really usefull most especially with huge drbd based volume
> > that could take severals hours of syncing.
>
> > I will try to take some time to try it and report eventual failure/sucess
> > use case.
>
> Feedback on this is more than welcome.
>
> > Do I only need to use this GIT branch to test it ?
>
> Yes. You can also use debian-2.6 branch and git-buildpackage to build the deb
> package. Please note that this branch has also ip pool support (gnt-network ...)
> and external storage interface support (-t ext --disk 0:size=10G,provider=...).
> IP pool makes use of python-bitarray that is currently existing only in wheezy.
> I have backported it to squeeze (you can get it from our local testing repo

> debhttp://apt.dev.grnet.grsid main ). Because of ippool management you have

> to run cfgupgrade (nodegroups get an extra slot: networks).
>
> > What is implication of this branch on a prod environment ?
>
> IP pool and extstorage are so far tested more that hotplug in our clusters.
> All features are backward compatible extentions and do not break the rest
> funtionality. Note that currently old instances can not use hotplug feature,
> It should be enabled during instance creation (maybe a proper cfgupgrade could
> bypass this limitation).
>
> > Do you think is it enough safe to test hotplug functionality on test
> > instance next to production instance ?
>
> I don't see anything that could affect eachother.
>
> Of course feel free to ask any implementation details or mention any suggestions/
> improvements you come up with.
>
> Thanks again for the interest,
> dimara
>

>  signature.asc
> < 1 KBAnzeigenHerunterladen

[Jean-François Maeyhieux]

So this patch seems to be pratically stable.

 Dimitris Aragiorgis: do you know if this branch will be merged on next ganeti version ?

Jean-François.

[Dimitris Aragiorgis]

Hi,

* Jean-François Maeyhieux <zen2...@gmail.com> [2012-11-17 04:09:17 -0800]:

> So this patch seems to be pratically stable.
>

Nice to hear that. It is a first implementation of live instance modifications,
so there is still much to be added/changed, especially on the feedback and error
path. The whole functionallity should not change a lot although there are
plenty of design issues to discuss.

Currently the feature is hypervisor specific (kvm) but the actual provisions
are done by cmdlib. (*)

Another thing that should be noted is that the runtime file should change
anytime a hotplug succeeds. (**)

Current implementation has tried to keep 100% compatibility with existing
funtionality. So instances are devided into hotplug-able or not. Although this
is stable it may be not ideal. (***)

> Dimitris Aragiorgis: do you know if this branch will be merged on next
> ganeti version ?
>
>

I don't know if this will be the case but I am not the proper person to ask.
Our priority for the next Ganeti version is to have the IP pool management
functionality merged and not the hotplug. However, this maybe a good time
to submit a design doc regarding hotplug, for a first round of conversation.

In the following weeks I am going to refactor the code (not the API) so that
the whole funtionallity will be delivered to hypervisor in order the slots
provisioning to be shifted away from masterd. (****)

Thats all for now. Thanks again for the interest and sorry for the long mail..

Regards,
dimara




(*) The whole idea is that qemu can handle devices based on their names and
their pci slot. To this end all is needed is to ensure unique device names
(via a strictly ascending index) and keep tracking the free and occupied pci
slots of the instance. All this info is currently saved in a per instance slot
in config data (hotplug info).

(**) This is needed so that migration does not fail due to inconsistent VM
state (the --incoming instance should have identical pci configuration as the
original one). To this end block devices should be removed from the cmdline and
treated separately just like nics. This patch does this already.

(***) Non hotplug-able keep the original way of passing devices to kvm cmdline and
the way info is kept in runtime file. Hotplug-able ones obtain hotplug info
and devices are named properly and put to specific pci slots.


(****) This to take place, code only needs to parse correctly pci info output
(and other qemu monitor commands) and find the next free slot to put the
device. The success/failure of this action should be returned to the masterd
(and eventually to the user).