- that create the drbd disk, start the drbd sync between nodes and return on the shell directly.
- I've wait the sync to be effective by watching /proc/drbd
- I stop/start the instance and I could see the /dev/vdb
- I mkfs it and I add it manually to my fstab
- I mount it and I could start to use it
2) No sync information/notice:
I think when the command finish and return to the shell, we should read an advice that tell us to check the progression of the drbd sync before start to use the disk.
Something like:
The sync of disk/1:/dev/drbd3 is in progress.
You could check the progression of the sync with: 'watch cat /proc/drbd'
Wait the sync is finish before start to use the device.
3) The command tell that the modification could be effective after the next start of the instance
I don't know if it's possible to do it and use the disk whithout shutdown the instance.
In my case, I've shutdown the instance before use it.
4) metadata on main VG
The second point is that the metadata drbd volume land on the main volume group and not on the command line specified one:
- disk/1: drbd8, size 1.1T
access mode: rw
auth key: ce5b456a30412b651f72e92487b74a781ce68345
on primary: /dev/drbd3 (147:3) in sync, status ok
on secondary: /dev/drbd3 (147:3) in sync, status ok
child devices:
- child 0: lvm, size 1.1T
logical_id: vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data
on primary: /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:17) <= VG specified on command line to add the disk
on secondary: /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:13) <= VG specified on command line to add the disk
- child 1: lvm, size 128M
logical_id: vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta
on primary: /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:18) <= main VG for instance
on secondary: /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:14) <= main VG for instance
Is it normal ? could we have a way to specify it ?
Jean-Francois
Le lundi 12 novembre 2012 14:40:06 UTC+1, Iustin Pop a écrit :
No, currently we don't have a feature to attach disks to a running
instance.
As for the meta vg you can specify it with the metavg=... option to the
same command. It might be a bug though that it didn't choose the default
one, if it was specified.
Thanks for the resync message suggestion: would you file an issue for that?
Thanks
Guido
On 12 Nov 2012 19:13, "Jean-François Maeyhieux" <zen2dr...@gmail.com> wrote:
> - that create the drbd disk, start the drbd sync between nodes and return
> on the shell directly.
> - I've wait the sync to be effective by watching /proc/drbd
> - I stop/start the instance and I could see the /dev/vdb
> - I mkfs it and I add it manually to my fstab
> - I mount it and I could start to use it
> 2) No sync information/notice:
> I think when the command finish and return to the shell, we should read an
> advice that tell us to check the progression of the drbd sync before start
> to use the disk.
> Something like:
> The sync of disk/1:/dev/drbd3 is in progress.
> You could check the progression of the sync with: 'watch cat
> /proc/drbd'
> Wait the sync is finish before start to use the device.
> 3) The command tell that the modification could be effective after the
> next start of the instance
> I don't know if it's possible to do it and use the disk whithout shutdown
> the instance.
> In my case, I've shutdown the instance before use it.
> 4) metadata on main VG
> The second point is that the metadata drbd volume land on the main volume
> group and not on the command line specified one:
> - disk/1: drbd8, size 1.1T
> access mode: rw
> auth key: ce5b456a30412b651f72e92487b74a781ce68345
> on primary: /dev/drbd3 (147:3) in sync, status ok
> on secondary: /dev/drbd3 (147:3) in sync, status ok
> child devices:
> - child 0: lvm, size 1.1T
> logical_id:
> vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data
> on primary:
> /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:17)
> <= VG specified on command line to add the disk
> on secondary:
> /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:13)
> <= VG specified on command line to add the disk
> - child 1: lvm, size 128M
> logical_id:
> vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta
> on primary:
> /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:18)
> <= main VG for instance
> on secondary:
> /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:14)
> <= main VG for instance
> Is it normal ? could we have a way to specify it ?
> Jean-Francois
> Le lundi 12 novembre 2012 14:40:06 UTC+1, Iustin Pop a écrit :
>> On Mon, Nov 12, 2012 at 05:36:51AM -0800, Jean-Fran�ois Maeyhieux
>> wrote:
>> > Hello,
>> > we have a ganeti cluster with different instances installed on drbd
>> > ( metavg: vg-cluster ).
>> > Is it possible to add a second drbd disk on a specific instance with a
>> > storage located on another volume group (vg-data for exemple) ?
> No, currently we don't have a feature to attach disks to a running > instance.
> As for the meta vg you can specify it with the metavg=... option to the > same command. It might be a bug though that it didn't choose the default > one, if it was specified.
> Thanks for the resync message suggestion: would you file an issue for > that?
> Thanks
> Guido > On 12 Nov 2012 19:13, "Jean-François Maeyhieux" <zen2...@gmail.com<javascript:>> > wrote:
>> 1) That works perfectly:
>> - that create the drbd disk, start the drbd sync between nodes and return >> on the shell directly.
>> - I've wait the sync to be effective by watching /proc/drbd
>> - I stop/start the instance and I could see the /dev/vdb
>> - I mkfs it and I add it manually to my fstab
>> - I mount it and I could start to use it
>> 2) No sync information/notice:
>> I think when the command finish and return to the shell, we should read >> an advice that tell us to check the progression of the drbd sync before >> start to use the disk.
>> Something like:
>> The sync of disk/1:/dev/drbd3 is in progress.
>> You could check the progression of the sync with: 'watch cat >> /proc/drbd'
>> Wait the sync is finish before start to use the device.
>> 3) The command tell that the modification could be effective after the >> next start of the instance
>> I don't know if it's possible to do it and use the disk whithout shutdown >> the instance.
>> In my case, I've shutdown the instance before use it.
>> 4) metadata on main VG
>> The second point is that the metadata drbd volume land on the main volume >> group and not on the command line specified one:
>> - disk/1: drbd8, size 1.1T
>> access mode: rw
>> auth key: ce5b456a30412b651f72e92487b74a781ce68345
>> on primary: /dev/drbd3 (147:3) in sync, status ok
>> on secondary: /dev/drbd3 (147:3) in sync, status ok
>> child devices:
>> - child 0: lvm, size 1.1T
>> logical_id: >> vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data
>> on primary: >> /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:17) >> <= VG specified on command line to add the disk
>> on secondary: >> /dev/vg-storage/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_data (253:13) >> <= VG specified on command line to add the disk
>> - child 1: lvm, size 128M
>> logical_id: >> vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta
>> on primary: >> /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:18) >> <= main VG for instance
>> on secondary: >> /dev/vg-cloud/12ea7dbe-7b7f-4de1-af5f-a28ade1ada43.disk1_meta (253:14) >> <= main VG for instance
>> Is it normal ? could we have a way to specify it ?
>> Jean-Francois
>> Le lundi 12 novembre 2012 14:40:06 UTC+1, Iustin Pop a écrit :
>>> On Mon, Nov 12, 2012 at 05:36:51AM -0800, Jean-Fran�ois Maeyhieux >>> wrote:
>>> > Hello,
>>> > we have a ganeti cluster with different instances installed on >>> drbd >>> > ( metavg: vg-cluster ).
>>> > Is it possible to add a second drbd disk on a specific instance with a >>> > storage located on another volume group (vg-data for exemple) ?
> No, currently we don't have a feature to attach disks to a running
> instance.
I have implemented a disk/nic hotplug feature for ganeti, that makes use of
specific qemu monitor commands and therefor kvm >=1.0 is needed. This is
premature but seems to work. All you have to do is enable hotplug when
creating an instance, by adding --hotplug option:
# gnt-instance add --hotplug ... vmname
and then pass this option when modifying the instance:
There are still some limitations about the current implementation. First the PCI
slots are partitioned in order to separate hot-plugable devices from the rest.
Secondly the first disk is not hot-plugable. Furthermore if a device is
hot-added it should be removed using hotplug. Still you can add/remove/modify
devices with the traditional way (i.e. without --hotplug option) and the
modification will take place after reboot).
In case you are interested for digging further into this please refer to:
Dimitris Aragiorgis: you make my day ! This hotplug patch is awesome.
So it's possible with it to do disk provisionning whithout instance down time.
That should be really usefull most especially with huge drbd based volume that could take severals hours of syncing.
I will try to take some time to try it and report eventual failure/sucess use case.
Do I only need to use this GIT branch to test it ?
What is implication of this branch on a prod environment ?
Do you think is it enough safe to test hotplug functionality on test instance next to production instance ?
Jean-Francois
Le lundi 12 novembre 2012 14:36:51 UTC+1, Jean-François Maeyhieux a écrit :
> Dimitris Aragiorgis: you make my day ! This hotplug patch is awesome.
I am glad!
> So it's possible with it to do disk provisionning whithout instance down > time.
> That should be really usefull most especially with huge drbd based volume > that could take severals hours of syncing.
> I will try to take some time to try it and report eventual failure/sucess > use case.
Feedback on this is more than welcome.
> Do I only need to use this GIT branch to test it ?
Yes. You can also use debian-2.6 branch and git-buildpackage to build the deb
package. Please note that this branch has also ip pool support (gnt-network ...)
and external storage interface support (-t ext --disk 0:size=10G,provider=...).
IP pool makes use of python-bitarray that is currently existing only in wheezy.
I have backported it to squeeze (you can get it from our local testing repo
deb http://apt.dev.grnet.gr sid main ). Because of ippool management you have
to run cfgupgrade (nodegroups get an extra slot: networks).
> What is implication of this branch on a prod environment ?
IP pool and extstorage are so far tested more that hotplug in our clusters.
All features are backward compatible extentions and do not break the rest
funtionality. Note that currently old instances can not use hotplug feature,
It should be enabled during instance creation (maybe a proper cfgupgrade could
bypass this limitation).
> Do you think is it enough safe to test hotplug functionality on test > instance next to production instance ?
I don't see anything that could affect eachother.
Of course feel free to ask any implementation details or mention any suggestions/
improvements you come up with.
> > Dimitris Aragiorgis: you make my day ! This hotplug patch is awesome.
> I am glad!
> > So it's possible with it to do disk provisionning whithout instance down
> > time.
> > That should be really usefull most especially with huge drbd based volume
> > that could take severals hours of syncing.
> > I will try to take some time to try it and report eventual failure/sucess
> > use case.
> Feedback on this is more than welcome.
> > Do I only need to use this GIT branch to test it ?
> Yes. You can also use debian-2.6 branch and git-buildpackage to build the deb
> package. Please note that this branch has also ip pool support (gnt-network ...)
> and external storage interface support (-t ext --disk 0:size=10G,provider=...).
> IP pool makes use of python-bitarray that is currently existing only in wheezy.
> I have backported it to squeeze (you can get it from our local testing repo
> debhttp://apt.dev.grnet.grsid main ). Because of ippool management you have
> to run cfgupgrade (nodegroups get an extra slot: networks).
> > What is implication of this branch on a prod environment ?
> IP pool and extstorage are so far tested more that hotplug in our clusters.
> All features are backward compatible extentions and do not break the rest
> funtionality. Note that currently old instances can not use hotplug feature,
> It should be enabled during instance creation (maybe a proper cfgupgrade could
> bypass this limitation).
> > Do you think is it enough safe to test hotplug functionality on test
> > instance next to production instance ?
> I don't see anything that could affect eachother.
> Of course feel free to ask any implementation details or mention any suggestions/
> improvements you come up with.
> > > Dimitris Aragiorgis: you make my day ! This hotplug patch is awesome.
> > I am glad!
> > > So it's possible with it to do disk provisionning whithout instance > down > > > time. > > > That should be really usefull most especially with huge drbd based > volume > > > that could take severals hours of syncing.
> > > I will try to take some time to try it and report eventual > failure/sucess > > > use case.
> > Feedback on this is more than welcome.
> > > Do I only need to use this GIT branch to test it ?
> > Yes. You can also use debian-2.6 branch and git-buildpackage to build > the deb > > package. Please note that this branch has also ip pool support > (gnt-network ...) > > and external storage interface support (-t ext --disk > 0:size=10G,provider=...). > > IP pool makes use of python-bitarray that is currently existing only in > wheezy. > > I have backported it to squeeze (you can get it from our local testing > repo > > debhttp://apt.dev.grnet.grsid main ). Because of ippool management you > have > > to run cfgupgrade (nodegroups get an extra slot: networks).
> > > What is implication of this branch on a prod environment ?
> > IP pool and extstorage are so far tested more that hotplug in our > clusters. > > All features are backward compatible extentions and do not break the > rest > > funtionality. Note that currently old instances can not use hotplug > feature, > > It should be enabled during instance creation (maybe a proper cfgupgrade > could > > bypass this limitation).
> > > Do you think is it enough safe to test hotplug functionality on test > > > instance next to production instance ?
> > I don't see anything that could affect eachother.
> > Of course feel free to ask any implementation details or mention any > suggestions/ > > improvements you come up with.
Nice to hear that. It is a first implementation of live instance modifications,
so there is still much to be added/changed, especially on the feedback and error
path. The whole functionallity should not change a lot although there are
plenty of design issues to discuss.
Currently the feature is hypervisor specific (kvm) but the actual provisions
are done by cmdlib. (*)
Another thing that should be noted is that the runtime file should change
anytime a hotplug succeeds. (**)
Current implementation has tried to keep 100% compatibility with existing
funtionality. So instances are devided into hotplug-able or not. Although this
is stable it may be not ideal. (***)
> Dimitris Aragiorgis: do you know if this branch will be merged on next
> ganeti version ?
I don't know if this will be the case but I am not the proper person to ask.
Our priority for the next Ganeti version is to have the IP pool management
functionality merged and not the hotplug. However, this maybe a good time
to submit a design doc regarding hotplug, for a first round of conversation.
In the following weeks I am going to refactor the code (not the API) so that
the whole funtionallity will be delivered to hypervisor in order the slots
provisioning to be shifted away from masterd. (****)
Thats all for now. Thanks again for the interest and sorry for the long mail..
Regards,
dimara
(*) The whole idea is that qemu can handle devices based on their names and
their pci slot. To this end all is needed is to ensure unique device names
(via a strictly ascending index) and keep tracking the free and occupied pci
slots of the instance. All this info is currently saved in a per instance slot
in config data (hotplug info).
(**) This is needed so that migration does not fail due to inconsistent VM
state (the --incoming instance should have identical pci configuration as the
original one). To this end block devices should be removed from the cmdline and
treated separately just like nics. This patch does this already.
(***) Non hotplug-able keep the original way of passing devices to kvm cmdline and
the way info is kept in runtime file. Hotplug-able ones obtain hotplug info
and devices are named properly and put to specific pci slots.
(****) This to take place, code only needs to parse correctly pci info output
(and other qemu monitor commands) and find the next free slot to put the
device. The success/failure of this action should be returned to the masterd
(and eventually to the user).
