Re: gnt-network help
171 views
Skip to first unread message
Guido Trotter
Apr 24, 2013, 2:41:15 AM4/24/13
to gan...@googlegroups.com
On Tue, Apr 23, 2013 at 11:05 PM, <mitc...@ucar.edu> wrote:
Hi,
> I'm trying to understand the usage of gnt-network and gnt-group and some
> stuff is not making sense. I'm not sure if it's something I'm doing or if
> I'm hitting some kind of bug. What I would like to accomplish (and I don't
> think the new network type really incorporates all of this) is for the
> 'network' to be something which is only active on one node at a time. My
> goal is to be able to move a group of instances and their associated subnet
> from one location to another where there is only layer 3 connectivity
> between them. To do so, I would have some concept of starting a network on a
> node before the instances associated with it were booted and if I needed to
> move the instances to a different node the network would have to be shutdown
> first. It would be a lot like what has to be done with the DRBD disks
> associated with an instance, only I imagine that usually the network would
> be used by multiple instances which would have to move as a group. I plan
> initially to try and kludge something together using the RAPI and wanted to
> get a good grip on how networks and groups work first.
Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
Idea to get this implemented.
The only other alternative would be hacking it up in the "ifup" script
for kvm (supposing you use kvm. if you use xen we have a xen hackathon
project to port the kvm ifup script to xen). Finally an option for
your case would be to connect and disconnect the network to all nodes
of a nodegroup at once. So your workload would be:
- connect the network to the second nodegroup
- migrate all instances from nodegroup1 to nodegroup2
- disconnect the network from nodegroup1
A hook at "connect" time could take care of bringing it up or down on
all the nodes of the target group.
If you have interested students (since you come from a .edu domain)
perhaps we can work together on a design for the GSoC project, and you
could mentor them together with us.
> One problem is I can't seem to get an instance to be associated with a
> network. I created an instance whose IP is defined inside of the range of
> the network, but that wasn't enough to make it associated. And when I try to
> add the network manually I get an error I don't really understand.
>
> # gnt-network info cnet1
> Network name: cnet1
> UUID: bcd9523d-bab6-4af0-acf2-5120ccb83559
> Serial number: 1
> Subnet: 192.168.5.0/24
> Gateway: 192.168.5.251
> IPv6 Subnet: None
> IPv6 Gateway: None
> Mac Prefix: None
> Size: 256
> Free: 253 (98.83%)
> Usage map:
> 0 X............................................................... 63
> 64 ................................................................
> 127
> 128 ................................................................
> 191
> 192 ...........................................................X...X
> 255
> (X) used (.) free
> externally reserved IPs:
> 192.168.5.0, 192.168.5.251, 192.168.5.255
> connected to node groups:
> ['default', 'bridged', 'xen-br1']
> ['mlgroup', 'bridged', 'xen-br1']
> ['nwscgroup', 'bridged', 'xen-br1']
> not used by any instances
>
> # gnt-instance modify --net 0:ip=192.168.5.5 testvm6.local
> Failure: prerequisites not met for this operation:
> error type: wrong_state, error details:
> Conflicting IP address found: '192.168.5.5' != 'cnet1'
Indeed this error could probably use some improvement.
What I think it means is that it won't allow you to use an ip from the
"cnet1" network unless the nic itself is actually connected to that
network. But this definitely deserves a bug so that it can be reworded
properly.
Can we see a gnt-instance info, to be sure?
Also you can try:
gnt-instance modify --net 0:network=cnet1
or
gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
(which one depends on which ip the nic already has configured)
> #
>
> I've also been trying to create some node groups but the output of gnt-group
> always says they are empty. This is despite the face that gnt-instance
> actually reports the instance as being in the group.
>
This is a bug introduced while moving the queries to confd. Will file an issue.
Thanks,
Guido
Hi,
> I'm trying to understand the usage of gnt-network and gnt-group and some
> stuff is not making sense. I'm not sure if it's something I'm doing or if
> I'm hitting some kind of bug. What I would like to accomplish (and I don't
> think the new network type really incorporates all of this) is for the
> 'network' to be something which is only active on one node at a time. My
> goal is to be able to move a group of instances and their associated subnet
> from one location to another where there is only layer 3 connectivity
> between them. To do so, I would have some concept of starting a network on a
> node before the instances associated with it were booted and if I needed to
> move the instances to a different node the network would have to be shutdown
> first. It would be a lot like what has to be done with the DRBD disks
> associated with an instance, only I imagine that usually the network would
> be used by multiple instances which would have to move as a group. I plan
> initially to try and kludge something together using the RAPI and wanted to
> get a good grip on how networks and groups work first.
Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
Idea to get this implemented.
The only other alternative would be hacking it up in the "ifup" script
for kvm (supposing you use kvm. if you use xen we have a xen hackathon
project to port the kvm ifup script to xen). Finally an option for
your case would be to connect and disconnect the network to all nodes
of a nodegroup at once. So your workload would be:
- connect the network to the second nodegroup
- migrate all instances from nodegroup1 to nodegroup2
- disconnect the network from nodegroup1
A hook at "connect" time could take care of bringing it up or down on
all the nodes of the target group.
If you have interested students (since you come from a .edu domain)
perhaps we can work together on a design for the GSoC project, and you
could mentor them together with us.
> One problem is I can't seem to get an instance to be associated with a
> network. I created an instance whose IP is defined inside of the range of
> the network, but that wasn't enough to make it associated. And when I try to
> add the network manually I get an error I don't really understand.
>
> # gnt-network info cnet1
> Network name: cnet1
> UUID: bcd9523d-bab6-4af0-acf2-5120ccb83559
> Serial number: 1
> Subnet: 192.168.5.0/24
> Gateway: 192.168.5.251
> IPv6 Subnet: None
> IPv6 Gateway: None
> Mac Prefix: None
> Size: 256
> Free: 253 (98.83%)
> Usage map:
> 0 X............................................................... 63
> 64 ................................................................
> 127
> 128 ................................................................
> 191
> 192 ...........................................................X...X
> 255
> (X) used (.) free
> externally reserved IPs:
> 192.168.5.0, 192.168.5.251, 192.168.5.255
> connected to node groups:
> ['default', 'bridged', 'xen-br1']
> ['mlgroup', 'bridged', 'xen-br1']
> ['nwscgroup', 'bridged', 'xen-br1']
> not used by any instances
>
> # gnt-instance modify --net 0:ip=192.168.5.5 testvm6.local
> Failure: prerequisites not met for this operation:
> error type: wrong_state, error details:
> Conflicting IP address found: '192.168.5.5' != 'cnet1'
Indeed this error could probably use some improvement.
What I think it means is that it won't allow you to use an ip from the
"cnet1" network unless the nic itself is actually connected to that
network. But this definitely deserves a bug so that it can be reworded
properly.
Can we see a gnt-instance info, to be sure?
Also you can try:
gnt-instance modify --net 0:network=cnet1
or
gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
(which one depends on which ip the nic already has configured)
> #
>
> I've also been trying to create some node groups but the output of gnt-group
> always says they are empty. This is despite the face that gnt-instance
> actually reports the instance as being in the group.
>
This is a bug introduced while moving the queries to confd. Will file an issue.
Thanks,
Guido
Constantinos Venetsanopoulos
Apr 24, 2013, 4:12:11 AM4/24/13
to gan...@googlegroups.com, Guido Trotter
Hi,
Yes indeed. That is what it means.
> But this definitely deserves a bug so that it can be reworded
> properly.
>
> Can we see a gnt-instance info, to be sure?
>
> Also you can try:
>
> gnt-instance modify --net 0:network=cnet1
> or
> gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
Just a clarification here. The network feature was introduced to
handle automatic allocation of IPs from the range. So, yes:
If you want Ganeti to allocate an IP from your range you would
do:
gnt-instance modify --net 0:network=cnet1
If you want to put the NIC in that net and also pick the IP
yourself you would do:
gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
Ganeti will then complain if this IP is already allocated for
another NIC.
Finally, if you do:
gnt-instance modify --net 0:ip=192.168.5.5
Ganeti will not put the NIC into any network and will just try
to assign the IP (for backwards compatibility). However,
if this IP is included in one of your networks, Ganeti will
complain again (that's your case). Indeed the message
should be more descriptive.
Also keep in mind that you first need to connect the
network to the nodegroup before you can use it, as
Guido pointed previously.
Kind Regards,
Constantinos
> But this definitely deserves a bug so that it can be reworded
> properly.
>
> Can we see a gnt-instance info, to be sure?
>
> Also you can try:
>
> gnt-instance modify --net 0:network=cnet1
> or
> gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
handle automatic allocation of IPs from the range. So, yes:
If you want Ganeti to allocate an IP from your range you would
do:
gnt-instance modify --net 0:network=cnet1
yourself you would do:
gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
another NIC.
Finally, if you do:
gnt-instance modify --net 0:ip=192.168.5.5
to assign the IP (for backwards compatibility). However,
if this IP is included in one of your networks, Ganeti will
complain again (that's your case). Indeed the message
should be more descriptive.
Also keep in mind that you first need to connect the
network to the nodegroup before you can use it, as
Guido pointed previously.
Kind Regards,
Constantinos
Dimitris Aragiorgis
Apr 24, 2013, 12:24:04 PM4/24/13
to gan...@googlegroups.com, Guido Trotter
Hi!
* Constantinos Venetsanopoulos <cv...@grnet.gr> [2013-04-24 11:12:11 +0300]:
> Hi,
>
> On 04/24/2013 09:41 AM, Guido Trotter wrote:
> >On Tue, Apr 23, 2013 at 11:05 PM, <mitc...@ucar.edu> wrote:
> >
> >Hi,
> >
>
In case you want Ganeti to allocate the first available IP inside the
network you should do:
gnt-instance add --net 0:ip=pool,network=cnet1 ...
gnt-instance modify --net 0:ip=pool,network=cnet1 ...
In case you want instances with no IP but in the same collision domain:
gnt-instance add --net 0:network=cnet1 ...
gnt-instance modify --net 0:ip=none,network=cnet1 ...
> If you want to put the NIC in that net and also pick the IP
> yourself you would do:
>
> gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
>
> Ganeti will then complain if this IP is already allocated for
> another NIC.
>
..or the IP is not included in the network.
> Finally, if you do:
>
> gnt-instance modify --net 0:ip=192.168.5.5
>
> Ganeti will not put the NIC into any network and will just try
> to assign the IP (for backwards compatibility). However,
> if this IP is included in one of your networks, Ganeti will
> complain again (that's your case). Indeed the message
> should be more descriptive.
>
The conflict check can be overridden with --no-conflicts-check option.
Note that the NIC will not inherit network's link and mode and the
IP will not be reserved in the network.
> Also keep in mind that you first need to connect the
> network to the nodegroup before you can use it, as
> Guido pointed previously.
>
> Kind Regards,
> Constantinos
Thanks for the feedback,
dimara
* Constantinos Venetsanopoulos <cv...@grnet.gr> [2013-04-24 11:12:11 +0300]:
> Hi,
>
> On 04/24/2013 09:41 AM, Guido Trotter wrote:
> >On Tue, Apr 23, 2013 at 11:05 PM, <mitc...@ucar.edu> wrote:
> >
> >Hi,
> >
>
> Just a clarification here. The network feature was introduced to
> handle automatic allocation of IPs from the range. So, yes:
>
> If you want Ganeti to allocate an IP from your range you would
> do:
>
> gnt-instance modify --net 0:network=cnet1
>
Just to clear things up:
> handle automatic allocation of IPs from the range. So, yes:
>
> If you want Ganeti to allocate an IP from your range you would
> do:
>
> gnt-instance modify --net 0:network=cnet1
>
In case you want Ganeti to allocate the first available IP inside the
network you should do:
gnt-instance add --net 0:ip=pool,network=cnet1 ...
gnt-instance modify --net 0:ip=pool,network=cnet1 ...
In case you want instances with no IP but in the same collision domain:
gnt-instance add --net 0:network=cnet1 ...
gnt-instance modify --net 0:ip=none,network=cnet1 ...
> If you want to put the NIC in that net and also pick the IP
> yourself you would do:
>
> gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
>
> Ganeti will then complain if this IP is already allocated for
> another NIC.
>
> Finally, if you do:
>
> gnt-instance modify --net 0:ip=192.168.5.5
>
> Ganeti will not put the NIC into any network and will just try
> to assign the IP (for backwards compatibility). However,
> if this IP is included in one of your networks, Ganeti will
> complain again (that's your case). Indeed the message
> should be more descriptive.
>
Note that the NIC will not inherit network's link and mode and the
IP will not be reserved in the network.
> Also keep in mind that you first need to connect the
> network to the nodegroup before you can use it, as
> Guido pointed previously.
>
> Kind Regards,
> Constantinos
dimara
David Mitchell
Apr 24, 2013, 2:02:06 PM4/24/13
to gan...@googlegroups.com
On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>
> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
> Idea to get this implemented.
> The only other alternative would be hacking it up in the "ifup" script
> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
> project to port the kvm ifup script to xen). Finally an option for
> your case would be to connect and disconnect the network to all nodes
> of a nodegroup at once. So your workload would be:
> - connect the network to the second nodegroup
> - migrate all instances from nodegroup1 to nodegroup2
> - disconnect the network from nodegroup1
# gnt-network disconnect cnet1 mlgroup
Wed Apr 24 11:48:51 2013 - WARNING: IP addresses from network 'cnet1', which is about to disconnect from node group 'mlgroup', are in use: testvm6.local: nic0/192.168.5.5
Failure: prerequisites not met for this operation:
error type: wrong_state, error details:
Conflicting IP addresses found; remove/modify the corresponding network interfaces
error type: wrong_state, error details:
# gnt-instance info testvm6
Instance name: testvm6.local
UUID: d02c154b-7e28-4a96-834a-dd64da51786c
Serial number: 11
Creation time: 2013-04-23 14:23:39
Modification time: 2013-04-24 11:42:51
State: configured to be up, actual state is up
Nodes:
- primary: vhip2.ucar.edu
group: nwscgroup (UUID 2c7eb36c-56df-42d9-ab0a-d2bf15e54c76)
- secondaries: vhip1.ucar.edu (group mlgroup, group UUID 9f759620-69ab-418d-b179-45c0b48b5f14)
>
> A hook at "connect" time could take care of bringing it up or down on
> all the nodes of the target group.
> If you have interested students (since you come from a .edu domain)
> perhaps we can work together on a design for the GSoC project, and you
> could mentor them together with us.
>
>
> Also you can try:
>
> gnt-instance modify --net 0:network=cnet1
> or
> gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
This second one seems to have exactly the desired effect. I now see the instance and reserved IP as I would expect to:
>
> gnt-instance modify --net 0:network=cnet1
> or
> gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5
# gnt-instance modify --net 0:network=cnet1,ip=192.168.5.5 testvm6
Wed Apr 24 11:30:20 2013 - INFO: Reserving IP 192.168.5.5 in network cnet1
Modified instance testvm6
- nic.ip/0 -> 192.168.5.5
- nic.link/0 -> xen-br1
- nic.mode/0 -> bridged
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.
# gnt-network info cnet1
Network name: cnet1
UUID: bcd9523d-bab6-4af0-acf2-5120ccb83559
Serial number: 1
Subnet: 192.168.5.0/24
Gateway: 192.168.5.251
IPv6 Subnet: None
IPv6 Gateway: None
Mac Prefix: None
Size: 256
Free: 252 (98.44%)
Network name: cnet1
UUID: bcd9523d-bab6-4af0-acf2-5120ccb83559
Serial number: 1
Subnet: 192.168.5.0/24
Gateway: 192.168.5.251
IPv6 Subnet: None
IPv6 Gateway: None
Mac Prefix: None
Size: 256
Usage map:
0 X....X.......................................................... 63
64 ................................................................ 127
128 ................................................................ 191
192 ...........................................................X...X 255
(X) used (.) free
externally reserved IPs:
192.168.5.0, 192.168.5.251, 192.168.5.255
connected to node groups:
['default', 'bridged', 'xen-br1']
['mlgroup', 'bridged', 'xen-br1']
['nwscgroup', 'bridged', 'xen-br1']
used by 1 instances:
128 ................................................................ 191
192 ...........................................................X...X 255
(X) used (.) free
externally reserved IPs:
192.168.5.0, 192.168.5.251, 192.168.5.255
connected to node groups:
['default', 'bridged', 'xen-br1']
['mlgroup', 'bridged', 'xen-br1']
['nwscgroup', 'bridged', 'xen-br1']
testvm6.local : 0:192.168.5.5
>
> (which one depends on which ip the nic already has configured)
>
>> #
>>
>> I've also been trying to create some node groups but the output of gnt-group
>> always says they are empty. This is despite the face that gnt-instance
>> actually reports the instance as being in the group.
>>
>
> This is a bug introduced while moving the queries to confd. Will file an issue.
>
> Thanks,
>
> Guido
| David Mitchell (mitc...@ucar.edu) Network Engineer IV |
| Tel: (303) 497-1845 National Center for |
| FAX: (303) 497-1818 Atmospheric Research |
-----------------------------------------------------------------
Guido Trotter
Apr 29, 2013, 8:10:29 AM4/29/13
to gan...@googlegroups.com
On Wed, Apr 24, 2013 at 8:02 PM, David Mitchell <mitc...@ucar.edu> wrote:
>
> On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>
>>
>> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
>> Idea to get this implemented.
>> The only other alternative would be hacking it up in the "ifup" script
>> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
>> project to port the kvm ifup script to xen). Finally an option for
>> your case would be to connect and disconnect the network to all nodes
>> of a nodegroup at once. So your workload would be:
>> - connect the network to the second nodegroup
>> - migrate all instances from nodegroup1 to nodegroup2
>> - disconnect the network from nodegroup1
>
> There is actually some internal logic right now which seems to be preventing this. Since the secondary node for the VM is in nodegroup1, I am not allowed to disconnect nodegroup1 from the network. If the prerequisites only checked to see if the primary node was connected then I think I could get the workflow outlined above going using hook scripts. Nothing in Ganeti would prevent me from connecting the network to two nodegroups at once and breaking my setup, but that's a secondary concern I can address once I got the basics working.
>
Yes, this only works if you move both primary and secondary out of a
>
> On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>
>>
>> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
>> Idea to get this implemented.
>> The only other alternative would be hacking it up in the "ifup" script
>> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
>> project to port the kvm ifup script to xen). Finally an option for
>> your case would be to connect and disconnect the network to all nodes
>> of a nodegroup at once. So your workload would be:
>> - connect the network to the second nodegroup
>> - migrate all instances from nodegroup1 to nodegroup2
>> - disconnect the network from nodegroup1
>
> There is actually some internal logic right now which seems to be preventing this. Since the secondary node for the VM is in nodegroup1, I am not allowed to disconnect nodegroup1 from the network. If the prerequisites only checked to see if the primary node was connected then I think I could get the workflow outlined above going using hook scripts. Nothing in Ganeti would prevent me from connecting the network to two nodegroups at once and breaking my setup, but that's a secondary concern I can address once I got the basics working.
>
nodegroup. If you want to leave the redundancy across nodegroups
(which is not exactly supported nowadays in Ganeti) then the network
has to be connected. It can still be deactivated with a vif-level
code, though. Then we should of course design/implement something
better for the future.
Guido
--
Guido Trotter
Ganeti Engineering
Google Germany GmbH
Dienerstr. 12, 80331, München
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens
Steuernummer: 48/725/00206
Umsatzsteueridentifikationsnummer: DE813741370
David Mitchell
May 1, 2013, 3:29:22 PM5/1/13
to gan...@googlegroups.com
On Apr 29, 2013, at 6:10 AM, Guido Trotter <ultr...@google.com> wrote:
> On Wed, Apr 24, 2013 at 8:02 PM, David Mitchell <mitc...@ucar.edu> wrote:
>>
>> On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>>
>>>
>>> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
>>> Idea to get this implemented.
>>> The only other alternative would be hacking it up in the "ifup" script
>>> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
>>> project to port the kvm ifup script to xen). Finally an option for
>>> your case would be to connect and disconnect the network to all nodes
>>> of a nodegroup at once. So your workload would be:
>>> - connect the network to the second nodegroup
>>> - migrate all instances from nodegroup1 to nodegroup2
>>> - disconnect the network from nodegroup1
>>
>> There is actually some internal logic right now which seems to be preventing this. Since the secondary node for the VM is in nodegroup1, I am not allowed to disconnect nodegroup1 from the network. If the prerequisites only checked to see if the primary node was connected then I think I could get the workflow outlined above going using hook scripts. Nothing in Ganeti would prevent me from connecting the network to two nodegroups at once and breaking my setup, but that's a secondary concern I can address once I got the basics working.
>>
>
> Yes, this only works if you move both primary and secondary out of a
> nodegroup. If you want to leave the redundancy across nodegroups
> (which is not exactly supported nowadays in Ganeti) then the network
> has to be connected. It can still be deactivated with a vif-level
> code, though. Then we should of course design/implement something
> better for the future.
-David Mitchell
Guido Trotter
May 2, 2013, 3:01:28 AM5/2/13
to gan...@googlegroups.com
On Wed, May 1, 2013 at 9:29 PM, David Mitchell <mitc...@ucar.edu> wrote:
>
> On Apr 29, 2013, at 6:10 AM, Guido Trotter <ultr...@google.com> wrote:
>
>> On Wed, Apr 24, 2013 at 8:02 PM, David Mitchell <mitc...@ucar.edu> wrote:
>>>
>>> On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>>>
>>>>
>>>> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
>>>> Idea to get this implemented.
>>>> The only other alternative would be hacking it up in the "ifup" script
>>>> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
>>>> project to port the kvm ifup script to xen). Finally an option for
>>>> your case would be to connect and disconnect the network to all nodes
>>>> of a nodegroup at once. So your workload would be:
>>>> - connect the network to the second nodegroup
>>>> - migrate all instances from nodegroup1 to nodegroup2
>>>> - disconnect the network from nodegroup1
>>>
>>> There is actually some internal logic right now which seems to be preventing this. Since the secondary node for the VM is in nodegroup1, I am not allowed to disconnect nodegroup1 from the network. If the prerequisites only checked to see if the primary node was connected then I think I could get the workflow outlined above going using hook scripts. Nothing in Ganeti would prevent me from connecting the network to two nodegroups at once and breaking my setup, but that's a secondary concern I can address once I got the basics working.
>>>
>>
>> Yes, this only works if you move both primary and secondary out of a
>> nodegroup. If you want to leave the redundancy across nodegroups
>> (which is not exactly supported nowadays in Ganeti) then the network
>> has to be connected. It can still be deactivated with a vif-level
>> code, though. Then we should of course design/implement something
>> better for the future.
>
> I would be willing to help with that however I can. For the time being I'm just going to alter my design a little. Ideally I would have a group of instances which are on the same subnet and I would move/migrate that subnet and the associated instances between nodes as a group. As an alternative I plan to simply advertise /32 host routes into the network, one per instance. This should work with the existing Ganeti code by attaching to the instance boot/shutdown hook scripts and advertising/withdrawing the host routes from there. Host routes won't really scale very well, but for the relatively small number of instances I am planning on it should work fine.
>
I guess the best idea would be writing a design doc, or asking a
>
> On Apr 29, 2013, at 6:10 AM, Guido Trotter <ultr...@google.com> wrote:
>
>> On Wed, Apr 24, 2013 at 8:02 PM, David Mitchell <mitc...@ucar.edu> wrote:
>>>
>>> On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>>>
>>>>
>>>> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
>>>> Idea to get this implemented.
>>>> The only other alternative would be hacking it up in the "ifup" script
>>>> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
>>>> project to port the kvm ifup script to xen). Finally an option for
>>>> your case would be to connect and disconnect the network to all nodes
>>>> of a nodegroup at once. So your workload would be:
>>>> - connect the network to the second nodegroup
>>>> - migrate all instances from nodegroup1 to nodegroup2
>>>> - disconnect the network from nodegroup1
>>>
>>> There is actually some internal logic right now which seems to be preventing this. Since the secondary node for the VM is in nodegroup1, I am not allowed to disconnect nodegroup1 from the network. If the prerequisites only checked to see if the primary node was connected then I think I could get the workflow outlined above going using hook scripts. Nothing in Ganeti would prevent me from connecting the network to two nodegroups at once and breaking my setup, but that's a secondary concern I can address once I got the basics working.
>>>
>>
>> Yes, this only works if you move both primary and secondary out of a
>> nodegroup. If you want to leave the redundancy across nodegroups
>> (which is not exactly supported nowadays in Ganeti) then the network
>> has to be connected. It can still be deactivated with a vif-level
>> code, though. Then we should of course design/implement something
>> better for the future.
>
> I would be willing to help with that however I can. For the time being I'm just going to alter my design a little. Ideally I would have a group of instances which are on the same subnet and I would move/migrate that subnet and the associated instances between nodes as a group. As an alternative I plan to simply advertise /32 host routes into the network, one per instance. This should work with the existing Ganeti code by attaching to the instance boot/shutdown hook scripts and advertising/withdrawing the host routes from there. Host routes won't really scale very well, but for the relatively small number of instances I am planning on it should work fine.
>
student to apply to our "dynamic network" proposal for Google Summer
of Code (unfortunately proposals are due by tomorrow) :).
I would be for doing it node by node, as I don't see an advantage to
doing it at the nodegroup level (at least initially), and also
attaching/detaching to a nodegroup signifies more of a policy (nodes
on this nodegroup *can* connect to this network) while per-node there
would be the mechanism (this node *is* attached to this network).
What do you think?
Thanks,
Guido
Guido Trotter
May 2, 2013, 4:52:55 AM5/2/13
to gan...@googlegroups.com
On Thu, May 2, 2013 at 9:01 AM, Guido Trotter <ultr...@gmail.com> wrote:
> On Wed, May 1, 2013 at 9:29 PM, David Mitchell <mitc...@ucar.edu> wrote:
>>
>> On Apr 29, 2013, at 6:10 AM, Guido Trotter <ultr...@google.com> wrote:
>>
>>> On Wed, Apr 24, 2013 at 8:02 PM, David Mitchell <mitc...@ucar.edu> wrote:
>>>>
>>>> On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>>>>
>>>>>
>>>>> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
>>>>> Idea to get this implemented.
>>>>> The only other alternative would be hacking it up in the "ifup" script
>>>>> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
>>>>> project to port the kvm ifup script to xen). Finally an option for
>>>>> your case would be to connect and disconnect the network to all nodes
>>>>> of a nodegroup at once. So your workload would be:
>>>>> - connect the network to the second nodegroup
>>>>> - migrate all instances from nodegroup1 to nodegroup2
>>>>> - disconnect the network from nodegroup1
>>>>
>>>> There is actually some internal logic right now which seems to be preventing this. Since the secondary node for the VM is in nodegroup1, I am not allowed to disconnect nodegroup1 from the network. If the prerequisites only checked to see if the primary node was connected then I think I could get the workflow outlined above going using hook scripts. Nothing in Ganeti would prevent me from connecting the network to two nodegroups at once and breaking my setup, but that's a secondary concern I can address once I got the basics working.
>>>>
>>>
>>> Yes, this only works if you move both primary and secondary out of a
>>> nodegroup. If you want to leave the redundancy across nodegroups
>>> (which is not exactly supported nowadays in Ganeti) then the network
>>> has to be connected. It can still be deactivated with a vif-level
>>> code, though. Then we should of course design/implement something
>>> better for the future.
>>
>> I would be willing to help with that however I can. For the time being I'm just going to alter my design a little. Ideally I would have a group of instances which are on the same subnet and I would move/migrate that subnet and the associated instances between nodes as a group. As an alternative I plan to simply advertise /32 host routes into the network, one per instance. This should work with the existing Ganeti code by attaching to the instance boot/shutdown hook scripts and advertising/withdrawing the host routes from there. Host routes won't really scale very well, but for the relatively small number of instances I am planning on it should work fine.
Note about this: we have a "routed" mode for instances, as opposed to
> On Wed, May 1, 2013 at 9:29 PM, David Mitchell <mitc...@ucar.edu> wrote:
>>
>> On Apr 29, 2013, at 6:10 AM, Guido Trotter <ultr...@google.com> wrote:
>>
>>> On Wed, Apr 24, 2013 at 8:02 PM, David Mitchell <mitc...@ucar.edu> wrote:
>>>>
>>>> On Apr 24, 2013, at 12:41 AM, Guido Trotter <ultr...@google.com> wrote:
>>>>
>>>>>
>>>>> Indeed, Ganeti doesn't do this yet. There is a Google Summer of Code
>>>>> Idea to get this implemented.
>>>>> The only other alternative would be hacking it up in the "ifup" script
>>>>> for kvm (supposing you use kvm. if you use xen we have a xen hackathon
>>>>> project to port the kvm ifup script to xen). Finally an option for
>>>>> your case would be to connect and disconnect the network to all nodes
>>>>> of a nodegroup at once. So your workload would be:
>>>>> - connect the network to the second nodegroup
>>>>> - migrate all instances from nodegroup1 to nodegroup2
>>>>> - disconnect the network from nodegroup1
>>>>
>>>> There is actually some internal logic right now which seems to be preventing this. Since the secondary node for the VM is in nodegroup1, I am not allowed to disconnect nodegroup1 from the network. If the prerequisites only checked to see if the primary node was connected then I think I could get the workflow outlined above going using hook scripts. Nothing in Ganeti would prevent me from connecting the network to two nodegroups at once and breaking my setup, but that's a secondary concern I can address once I got the basics working.
>>>>
>>>
>>> Yes, this only works if you move both primary and secondary out of a
>>> nodegroup. If you want to leave the redundancy across nodegroups
>>> (which is not exactly supported nowadays in Ganeti) then the network
>>> has to be connected. It can still be deactivated with a vif-level
>>> code, though. Then we should of course design/implement something
>>> better for the future.
>>
>> I would be willing to help with that however I can. For the time being I'm just going to alter my design a little. Ideally I would have a group of instances which are on the same subnet and I would move/migrate that subnet and the associated instances between nodes as a group. As an alternative I plan to simply advertise /32 host routes into the network, one per instance. This should work with the existing Ganeti code by attaching to the instance boot/shutdown hook scripts and advertising/withdrawing the host routes from there. Host routes won't really scale very well, but for the relatively small number of instances I am planning on it should work fine.
"bridged". If you plan to advertize routes would you mind experiement
with that and report any problems/improvements? :)
Thanks,
Guido
David Mitchell
May 6, 2013, 4:44:17 PM5/6/13
to gan...@googlegroups.com
On May 2, 2013, at 2:52 AM, Guido Trotter <ultr...@google.com> wrote:
>>>
>
> Note about this: we have a "routed" mode for instances, as opposed to
> "bridged". If you plan to advertize routes would you mind experiement
> with that and report any problems/improvements? :)
Here's what I ended up with for working network parameters:
- nic/0: MAC: aa:00:00:52:d9:85, IP: 128.117.42.129, mode: routed, link: main, network: None
The setup on the VM side is somewhat counter-intuitive as well, but I'm not sure that's a Ganeti issue at all. Conceptually I think of the VM as having just a single interface with a /32 host address. The default route specifies a device rather than IP, sending everything out eth0. This works when I set it up by hand but the Debian config scripts don't like it (that I can work out.) The workaround is to rely on the proxy_arp which is enabled by kvm_ifup. With proxy_arp enabled on the node for that interface, almost anything will work for a network configuration inside the VM. One downside to this is that the arp entry on the VM side for the default route ends up outdated after a migrate because the MAC of tap0 has changed on the node side. Is there a way to have routed mode set the tap0 MAC address to something fixed?
Once I worked through those issues, the resulting setup is very nice. I actually don't need any hook scripts at all to achieve what I want, which is a very pleasant surprise! I have Quagga installed on the nodes and have it configured to redistribute static and connected routes into BGP (with appropriate filtering.) Using routed mode everything just works as Quagga adds and removes the advertisement based on what Ganeti is doing automatically. This even extends to advertising the cluster master IP address as well! I can migrate a running VM between layer 3 domains with roughly 30 seconds of unreachability. And about half of that is the ARP issue above which I'm sure can be worked around.
-David Mitchell
Guido Trotter
May 22, 2013, 4:50:47 AM5/22/13
to gan...@googlegroups.com
On Mon, May 6, 2013 at 10:44 PM, David Mitchell <mitc...@ucar.edu> wrote:
This generally works, and works quite well. Just a couple of caveats. Getting the right parameters set up was tricky. Specifically, I had to specify 'main' as the link parameter in order to get the kvm-ifup script to set up the route properly. From what I can tell, there is no way to convince gnt-instance to set the 'link' parameter to be blank which would have the effect of having kvm-ifup not specify a routing table when it installs the route. And having 'xen-br0' or any such similar value in there causes the route addition to fail. By the way, I don't think the output from the 'ip' commands issued by kvm-ifup are captured anywhere. At least I couldn't find them.
On May 2, 2013, at 2:52 AM, Guido Trotter <ultr...@google.com> wrote:
>>>
>
> Note about this: we have a "routed" mode for instances, as opposed to
> "bridged". If you plan to advertize routes would you mind experiement
> with that and report any problems/improvements? :)
Here's what I ended up with for working network parameters:
- nic/0: MAC: aa:00:00:52:d9:85, IP: 128.117.42.129, mode: routed, link: main, network: None
The setup on the VM side is somewhat counter-intuitive as well, but I'm not sure that's a Ganeti issue at all. Conceptually I think of the VM as having just a single interface with a /32 host address. The default route specifies a device rather than IP, sending everything out eth0. This works when I set it up by hand but the Debian config scripts don't like it (that I can work out.) The workaround is to rely on the proxy_arp which is enabled by kvm_ifup. With proxy_arp enabled on the node for that interface, almost anything will work for a network configuration inside the VM. One downside to this is that the arp entry on the VM side for the default route ends up outdated after a migrate because the MAC of tap0 has changed on the node side. Is there a way to have routed mode set the tap0 MAC address to something fixed?
Not currently, I think, but we can discuss about this. Should we have a nic parameter for this, perhaps?
Once I worked through those issues, the resulting setup is very nice. I actually don't need any hook scripts at all to achieve what I want, which is a very pleasant surprise! I have Quagga installed on the nodes and have it configured to redistribute static and connected routes into BGP (with appropriate filtering.) Using routed mode everything just works as Quagga adds and removes the advertisement based on what Ganeti is doing automatically. This even extends to advertising the cluster master IP address as well! I can migrate a running VM between layer 3 domains with roughly 30 seconds of unreachability. And about half of that is the ARP issue above which I'm sure can be worked around.
Cool, glad to hear. Do you have any suggestion for improving the documentation so that future users find it easier?
Thanks!
Guido
Guido Trotter
May 22, 2013, 4:51:28 AM5/22/13
to gan...@googlegroups.com
On Mon, May 6, 2013 at 10:44 PM, David Mitchell <mitc...@ucar.edu> wrote:
This generally works, and works quite well. Just a couple of caveats. Getting the right parameters set up was tricky. Specifically, I had to specify 'main' as the link parameter in order to get the kvm-ifup script to set up the route properly. From what I can tell, there is no way to convince gnt-instance to set the 'link' parameter to be blank which would have the effect of having kvm-ifup not specify a routing table when it installs the route. And having 'xen-br0' or any such similar value in there causes the route addition to fail. By the way, I don't think the output from the 'ip' commands issued by kvm-ifup are captured anywhere. At least I couldn't find them.
On May 2, 2013, at 2:52 AM, Guido Trotter <ultr...@google.com> wrote:
>>>
>
> Note about this: we have a "routed" mode for instances, as opposed to
> "bridged". If you plan to advertize routes would you mind experiement
> with that and report any problems/improvements? :)
Here's what I ended up with for working network parameters:
- nic/0: MAC: aa:00:00:52:d9:85, IP: 128.117.42.129, mode: routed, link: main, network: None
The setup on the VM side is somewhat counter-intuitive as well, but I'm not sure that's a Ganeti issue at all. Conceptually I think of the VM as having just a single interface with a /32 host address. The default route specifies a device rather than IP, sending everything out eth0. This works when I set it up by hand but the Debian config scripts don't like it (that I can work out.) The workaround is to rely on the proxy_arp which is enabled by kvm_ifup. With proxy_arp enabled on the node for that interface, almost anything will work for a network configuration inside the VM. One downside to this is that the arp entry on the VM side for the default route ends up outdated after a migrate because the MAC of tap0 has changed on the node side. Is there a way to have routed mode set the tap0 MAC address to something fixed?
Not currently, I think, but we can discuss about this. Should we have a nic parameter for this, perhaps?
Once I worked through those issues, the resulting setup is very nice. I actually don't need any hook scripts at all to achieve what I want, which is a very pleasant surprise! I have Quagga installed on the nodes and have it configured to redistribute static and connected routes into BGP (with appropriate filtering.) Using routed mode everything just works as Quagga adds and removes the advertisement based on what Ganeti is doing automatically. This even extends to advertising the cluster master IP address as well! I can migrate a running VM between layer 3 domains with roughly 30 seconds of unreachability. And about half of that is the ARP issue above which I'm sure can be worked around.
Reply all
Reply to author
Forward
0 new messages