floating ip reachability with the stitched network for the openstack-git profile
14 views
Skip to first unread message
Shunmuga Priya
Oct 9, 2019, 10:54:07 AM10/9/19
to cloudlab-users
Hi,
I have stitched two cluster sites and everything went well until the floating ip comes into the picture.
When I create a floating ip for an instance, it was not reachable outside.
I tried using GRE Tunnel , VXlan and as well flat network configuration for the data network.
And the neutron router namespace has the proper SNAT and DNAT mapping for the floating ip.
But with the stitched network, even the neutron router namespace cant reach the floating IP of the VM instance.
I assigned flat network interface to the VM with the hope that the compute and network nodes should be able to access the VM using the flat network interface.
But the VM is not accesible either through the flat network interface or the floating ip.
When I have the cluster in the same site, I didnt encounter any of the above mentioned issues with the same openstack-git profile.
Your help would be much appreciated. Thank you.
Regards,
Priya.
David M. Johnson
Oct 9, 2019, 5:16:33 PM10/9/19
to cloudla...@googlegroups.com, Shunmuga Priya
On 10/9/19 8:54 AM, Shunmuga Priya wrote:
>
> Hi,
>
> I have stitched two cluster sites and everything went well until the
> floating ip comes into the picture.
>
> When I create a floating ip for an instance, it was not reachable outside.
>
> I tried using GRE Tunnel , VXlan and as well flat network configuration
> for the data network.
>
> And the neutron router namespace has the proper SNAT and DNAT mapping
> for the floating ip.
> But with the stitched network, even the neutron router namespace cant
> reach the floating IP of the VM instance.
>
> I assigned flat network interface to the VM with the hope that the
> compute and network nodes should be able to access the VM using the flat
> network interface.
> But the VM is not accesible either through the flat network interface or
> the floating ip.
Hi. I ran a multi-site test using 1 compute node at Emulab (site 1),
>
> Hi,
>
> I have stitched two cluster sites and everything went well until the
> floating ip comes into the picture.
>
> When I create a floating ip for an instance, it was not reachable outside.
>
> I tried using GRE Tunnel , VXlan and as well flat network configuration
> for the data network.
>
> And the neutron router namespace has the proper SNAT and DNAT mapping
> for the floating ip.
> But with the stitched network, even the neutron router namespace cant
> reach the floating IP of the VM instance.
>
> I assigned flat network interface to the VM with the hope that the
> compute and network nodes should be able to access the VM using the flat
> network interface.
> But the VM is not accesible either through the flat network interface or
> the floating ip.
and 1 compute node at Cloudlab Utah (site 2). I created a VM on each
compute host, assigned each a floating IP, and was able to remotely
login to both VMs. So floating IPs should work fine across clusters.
I don't see that you have a running experiment. Next time this happens
to you, please include the URL of your experiment's status page so we
can look at it.
> Priya.
David
David M. Johnson
Oct 9, 2019, 10:43:26 PM10/9/19
to Priya Ramanathan, cloudlab-users
On 10/9/19 5:39 PM, Priya Ramanathan wrote:
> Hi David,
>
> Thank you for your mail.
>
> Today I configured 2 sites with flat network (with GRE tunneling)
> multiplexed with the management network, which is also of flat type.
> And, beautifully I can access both the VMs internal IP and floating IP
> through the router namespace. But the horizon dashboard which used to
> work till yesterday is not getting connected. The working experiment
> model name: FlatNw.
> (https://www.cloudlab.us/status.php?uuid=ef70a38d-ead5-11e9-b1eb-e4434b2381fc)
Thanks for the pointers to your experiments, that's helpful! (Please
keep cloudla...@googlegroups.com cc'd too.)
I clicked on the dashboard link in the Profile Instructions tab in the
page linked above, and was able to login successfully with the password
in the instructions. Maybe it was a transient and/or network problem?
> Yesterday, I configured 2 sites with flat network (with the VXLAN
> tunneling) multiplexed with the management network, which is of VPN
> connectivity. Here even the internal IP is not accessible by the router
> namespace. But the dashboard is working fine. The non-working experiment
> model name: VXLAN
> (https://www.cloudlab.us/status.php?uuid=d92040c6-ea4a-11e9-b1eb-e4434b2381fc)
Right. If you look at the VM in the openstack dashboard, and click the
Log tab, and click the View Full Log button, you will see that the VM
tried to dhcp for its IP ("Starting Raise Network Interfaces"), and
never received anything, so that is why it is not pinging. I noticed
there were quite a few exceptions on the nm node in
/var/log/neutron/neutron-openvswitch-agent.log, saying that the
[DEFAULT]/lock_path option wasn't set. I think this was the problem. I
fixed up your nm node, and now your VM is reachable; and I've also
modified the OpenStack profile to set this variable. (This is an odd
problem, and I'm unsure why this doesn't show up when there's not a
dedicated networkmanager node; and why the ovs agent is insisting the
variable be set in the DEFAULT section -- it should be in the
oslo_concurrency section, which is where we ensure it's set.)
> For the past 2 days, I played around changing layer 2 connectivity,
> nothing helped me. Today I changed the management network from VPN to
> flat. And my requirement is satisfied.
>
> I don't know whether some changes in the cloudlab configuration solved
> the problem. Or is it to do with the management network to be of flat
> type for inter-site clustering.
No, the management network type doesn't matter for this problem. The
test I ran earlier today used the default setting for the management
network.
> It would be helpful if I could get the GUI of the openstack horizon
> dashboard up again without causing any VM internet connectivity issue.
> Thank you for your support.
>
> Regards,
> Priya.
David
> --
> winners do not do different things.They do things differently
> Hi David,
>
> Thank you for your mail.
>
> Today I configured 2 sites with flat network (with GRE tunneling)
> multiplexed with the management network, which is also of flat type.
> And, beautifully I can access both the VMs internal IP and floating IP
> through the router namespace. But the horizon dashboard which used to
> work till yesterday is not getting connected. The working experiment
> model name: FlatNw.
> (https://www.cloudlab.us/status.php?uuid=ef70a38d-ead5-11e9-b1eb-e4434b2381fc)
Thanks for the pointers to your experiments, that's helpful! (Please
keep cloudla...@googlegroups.com cc'd too.)
I clicked on the dashboard link in the Profile Instructions tab in the
page linked above, and was able to login successfully with the password
in the instructions. Maybe it was a transient and/or network problem?
> Yesterday, I configured 2 sites with flat network (with the VXLAN
> tunneling) multiplexed with the management network, which is of VPN
> connectivity. Here even the internal IP is not accessible by the router
> namespace. But the dashboard is working fine. The non-working experiment
> model name: VXLAN
> (https://www.cloudlab.us/status.php?uuid=d92040c6-ea4a-11e9-b1eb-e4434b2381fc)
Right. If you look at the VM in the openstack dashboard, and click the
Log tab, and click the View Full Log button, you will see that the VM
tried to dhcp for its IP ("Starting Raise Network Interfaces"), and
never received anything, so that is why it is not pinging. I noticed
there were quite a few exceptions on the nm node in
/var/log/neutron/neutron-openvswitch-agent.log, saying that the
[DEFAULT]/lock_path option wasn't set. I think this was the problem. I
fixed up your nm node, and now your VM is reachable; and I've also
modified the OpenStack profile to set this variable. (This is an odd
problem, and I'm unsure why this doesn't show up when there's not a
dedicated networkmanager node; and why the ovs agent is insisting the
variable be set in the DEFAULT section -- it should be in the
oslo_concurrency section, which is where we ensure it's set.)
> For the past 2 days, I played around changing layer 2 connectivity,
> nothing helped me. Today I changed the management network from VPN to
> flat. And my requirement is satisfied.
>
> I don't know whether some changes in the cloudlab configuration solved
> the problem. Or is it to do with the management network to be of flat
> type for inter-site clustering.
No, the management network type doesn't matter for this problem. The
test I ran earlier today used the default setting for the management
network.
> It would be helpful if I could get the GUI of the openstack horizon
> dashboard up again without causing any VM internet connectivity issue.
> Thank you for your support.
>
> Regards,
> Priya.
David
> winners do not do different things.They do things differently
Priya Ramanathan
Oct 10, 2019, 11:26:47 AM10/10/19
to David M. Johnson, cloudlab-users
Thank you, David, for your attention and quick response.
Now, I am able to access both the dashboard as well as the VM in both the experiments.
Thank you for the explanation part as well, which is helpful for me to learn and debug.
Regards,
Priya.
Priya Ramanathan
Oct 30, 2019, 7:47:49 PM10/30/19
to David M. Johnson, cloudlab-users
Hi David,
I am having the inter-site cluster created for one of my openstack experiments. The profile is openstack-git and the experiment name is EPCVM https://www.cloudlab.us/status.php?uuid=194fd2a1-fb68-11e9-b1eb-e4434b2381fc
I am getting the same old error of not able to login the dashboard. Timeout error is happening even after long time wait. Could you please help in resolving this issue?
Thank you.
Regards,
Priya
David M. Johnson
Oct 30, 2019, 11:48:09 PM10/30/19
to Priya Ramanathan, cloudlab-users
On 10/30/19 5:47 PM, Priya Ramanathan wrote:
> Hi David,
>
> Hi David,
>
> I am having the inter-site cluster created for one of my openstack
> experiments. The profile is openstack-git and the experiment name is
> EPCVM https://www.cloudlab.us/status.php?uuid=194fd2a1-fb68-11e9-b1eb-e4434b2381fc
>
> I am getting the same old error of not able to login the dashboard.
> Timeout error is happening even after long time wait. Could you please
> help in resolving this issue?
Sorry, I can't reproduce this in your experiment. I logged in to the
> experiments. The profile is openstack-git and the experiment name is
> EPCVM https://www.cloudlab.us/status.php?uuid=194fd2a1-fb68-11e9-b1eb-e4434b2381fc
>
> I am getting the same old error of not able to login the dashboard.
> Timeout error is happening even after long time wait. Could you please
> help in resolving this issue?
dashboard and got the nearly immediate expected response.
> Priya
David
Priya Ramanathan
Nov 4, 2019, 7:41:15 AM11/4/19
to David M. Johnson, cloudlab-users
Hi David,
I am getting the same floating ip connectivity issue with the inter cluster stitched model between Utah and Wisconsin for one of my new experiments.
We are planning to give a live demo using the cloudlab set up for the forthcoming IEEE NFV-SDN conference. Your timely help would be much appreciated.
I am using the GTP tunneling with the multiplex flat network.
Profile: openstack
Experiment name: VMMigration
Thank you.
Regards,
Priya.
Priya Ramanathan
Nov 4, 2019, 12:12:03 PM11/4/19
to David M. Johnson, cloudlab-users
Hi David,
FYI, if I use flat network interface type , the floating ip is reachable created through that router.
Experiment Name: VMMigrationFlat
Thanks and Regards,
Priya
Priya Ramanathan
Nov 4, 2019, 2:02:39 PM11/4/19
to David M. Johnson, cloudlab-users
Hi David,
Further observations:
With myVMMigrationFlat model, the nodes in Wisconsin is not linked with the Utah control node. When I view the Hypervisors list, the Wisconsin node is not being listed. http://ctl.vmmigrationflat.cloudlab-pg0.utah.cloudlab.us/horizon/admin/hypervisors/
With my VMMigration model, the nodes are linked and are listed. But the VM floating ip is not reachable. http://ctl.vmmigration.cloudlab-pg0.utah.cloudlab.us/horizon/admin/hypervisors/
Thank you.
Regards,
Priya
David M. Johnson
Nov 4, 2019, 2:11:29 PM11/4/19
to Priya Ramanathan, cloudlab-users
On 11/4/19 12:02 PM, Priya Ramanathan wrote:
> Hi David,
>
> Hi David,
>
> Further observations:
>
> With myVMMigrationFlat model, the nodes in Wisconsin is not linked with
> the Utah control node. When I view the Hypervisors list, the Wisconsin
> node is not being
> listed. http://ctl.vmmigrationflat.cloudlab-pg0.utah.cloudlab.us/horizon/admin/hypervisors/
>
> With my VMMigration model, the nodes are linked and are listed. But the
> VM floating ip is not
> reachable. http://ctl.vmmigration.cloudlab-pg0.utah.cloudlab.us/horizon/admin/hypervisors/
Hi. I've been looking at this for quite some time, and haven't been
>
> With myVMMigrationFlat model, the nodes in Wisconsin is not linked with
> the Utah control node. When I view the Hypervisors list, the Wisconsin
> node is not being
> listed. http://ctl.vmmigrationflat.cloudlab-pg0.utah.cloudlab.us/horizon/admin/hypervisors/
>
> With my VMMigration model, the nodes are linked and are listed. But the
> VM floating ip is not
> reachable. http://ctl.vmmigration.cloudlab-pg0.utah.cloudlab.us/horizon/admin/hypervisors/
able to figure it out yet. There are two problems. First, floating IPs
aren't reachable if the VMs are on the GRE tunnel network. Second, even
in this experiment, the node at Wisconsin (cp-s2-1) isn't reachable from
the ctl node (e.g. `ctl$ ping cp-s2-1` fails). I was able to reproduce
this second problem with a stitched link between Utah and Wisconsin in a
separate experiment, but we're in a bit of a resource crunch and so I
haven't been able to do more testing yet, to see if it's a stitching
problem, or a profile problem. I did check most of the virtual network
configuration, and what I saw looked fine. I could see incoming traffic
to the floating IPs in the virtual tun0-net router.
As for the floating IPs on the tun0-net problem, I haven't been able to
reproduce that in unstitched, single-cluster experiments, so I'm hoping
that's just related to the inability to get traffic across the stitched
link; it could well be.
I'll keep looking and let you know what I find.
> Thank you.
>
> Regards,
> Priya
David
> <mailto:shunmugapriy...@gmail.com>> wrote:
>
> Hi David,
>
> FYI, if I use flat network interface type , the floating ip is
> reachable created through that router.
> Experiment Name: VMMigrationFlat
> http://ctl.vmmigrationflat.cloudlab-pg0.utah.cloudlab.us/horizon/project/instances/?action=row_update&table=instances&obj_id=7e1d0974-8994-4630-b250-5c5126bfffe7
>
> Thanks and Regards,
> Priya
>
> On Mon, Nov 4, 2019 at 6:41 AM Priya Ramanathan
> <shunmugapriy...@gmail.com
>
> Hi David,
>
> FYI, if I use flat network interface type , the floating ip is
> reachable created through that router.
> Experiment Name: VMMigrationFlat
> http://ctl.vmmigrationflat.cloudlab-pg0.utah.cloudlab.us/horizon/project/instances/?action=row_update&table=instances&obj_id=7e1d0974-8994-4630-b250-5c5126bfffe7
>
> Thanks and Regards,
> Priya
>
> On Mon, Nov 4, 2019 at 6:41 AM Priya Ramanathan
> <shunmugapriy...@gmail.com
> <mailto:shunmugapriy...@gmail.com>>> wrote:
> >
> > Thank you, David, for your attention and quick response.
> > Now, I am able to access both the dashboard as well as
> the VM in
> > both the experiments.
> > Thank you for the explanation part as well, which is
> helpful for me
> > to learn and debug.
> >
> > Regards,
> > Priya.
> >
> > On Wed, Oct 9, 2019 at 9:43 PM David M. Johnson
> > <john...@flux.utah.edu
> <mailto:john...@flux.utah.edu>
> >
> > Thank you, David, for your attention and quick response.
> > Now, I am able to access both the dashboard as well as
> the VM in
> > both the experiments.
> > Thank you for the explanation part as well, which is
> helpful for me
> > to learn and debug.
> >
> > Regards,
> > Priya.
> >
> > On Wed, Oct 9, 2019 at 9:43 PM David M. Johnson
> > <john...@flux.utah.edu
> <mailto:john...@flux.utah.edu>
Priya Ramanathan
Nov 4, 2019, 2:25:24 PM11/4/19
to David M. Johnson, cloudlab-users
Hi David,
One observation regd. the reachability of Wisconsin node to Utah controller is related to the dns problem. I am using the profile: Openstack.
Regards,
Priya
David M. Johnson
Nov 4, 2019, 9:37:42 PM11/4/19
to Priya Ramanathan, cloudlab-users
We'll let you know when we have more information. Thanks for your patience.
>> Thank you.
>>
>> Regards,
>> Priya
>
> David
Priya Ramanathan
Nov 4, 2019, 9:53:48 PM11/4/19
to cloudlab-users, David M. Johnson, andreaf
Hi David,
In my earlier experimentations, I was able to use the flat network for the inter-networking. And you managed to enable the floating ip for my vxlan based connectivity.
Now, I am repeating the same set up. But it is having floating ip reachability issue always. If the management network is selected as flat network, then the floating ip is reachable but the hostname of the second cluster is having the dns issue.
I am having the IEEE conference by next week. Hope things shall start working up soon. Thank you.
Regards,
Priya.
Reply all
Reply to author
Forward
0 new messages