[geni-users] Layer2 topology-stitching successful, resources not ready

[Aditi Ghag]

Hi,

I created a layer2 topology using the stitching service last night. The reservation was successful as indicated-

23:15:00 INFO     stitch.Aggregate: DCN circuit 67921 is ready

23:15:00 INFO     stitch.Aggregate: DCN circuit 67941 is ready

23:15:00 INFO     stitch.Aggregate: DCN circuit 67911 is ready

23:15:00 INFO     stitch.Aggregate: DCN circuit 67901 is ready

23:15:00 INFO     stitch.Aggregate: DCN circuit 67891 is ready

23:15:00 INFO     stitch.Aggregate: DCN circuit 67931 is ready

23:15:07 INFO     stitch.Aggregate: Allocation at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+am> complete.

23:15:07 INFO     stitch.launcher: All aggregates are complete.

23:15:07 INFO     stitcher: Writing to 'OFtopov4-manifest-rspec-stitching-combined.xml'

23:15:07 INFO     stitcher: Saved combined reservation RSpec at 8 AMs to file OFtopov4-manifest-rspec-stitching-combined.xml

Stitching success: Reserved resources in slice OFtopov4 at 8 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 6 link(s)

I then ran the readyToLogin.py script to get the login information for all the nodes.My topology had end hosts connected via intermediate nodes acting as OVS. I used this image URL- https://www.utahddc.geniracks.net/image_metadata.php?uuid=468c846d-5395-11e3-880c-000000000000, which was given by Niky. I tried using the latest OVS image, but apparently the image was unavailable at one of the AMs(iirc, it was a protogeni rack). So to be consistent I used the older image. 

 While I am able to connect to the end hosts, when I run the readyToLogin script at the OVS nodes, its saying that the resource is not ready yet. 

Switch2's geni_status is: notready (am_status:notready) 

User aghag3 logs in to Switch2 using:

ssh -p 34618  -i /Users/aditi/.ssh/geni_key_portal agh...@pc3.instageni.illinois.edu

ssh -p 34618  -i /Users/aditi/.ssh/geni_cert_portal_key agh...@pc3.instageni.illinois.edu

I remember Niky had said that it takes time to boot up the image, but its been around 12 hours, and the resource is still not ready.

The slice name is OFtopov4

Could anyone suggest what the issue is, why the OVS resources ain't ready yet?

Thanks,

Aditi

[Vicraj Thomas]

Hello Aditi,

I'm sorry your node running OVS isn't booting up. Perhaps somebody from
the InstaGENI dev team can look to see what might be wrong.

The newest VM image with OVS has the URN
urn:publicid:IDN+utahddc.geniracks.net+image+emulab-ops:Ubuntu12-64-OVS.
It has been pushed to all the InstaGENI racks. Niky says this image
is much more reliable.

< Vic

On 2/14/14 10:40 AM, Aditi Ghag wrote:
> Hi,
>
> I created a layer2 topology using the stitching service last night. The
> reservation was successful as indicated-
>
> 23:15:00 INFO stitch.Aggregate: DCN circuit 67921 is ready
>
> 23:15:00 INFO stitch.Aggregate: DCN circuit 67941 is ready
>
> 23:15:00 INFO stitch.Aggregate: DCN circuit 67911 is ready
>
> 23:15:00 INFO stitch.Aggregate: DCN circuit 67901 is ready
>
> 23:15:00 INFO stitch.Aggregate: DCN circuit 67891 is ready
>
> 23:15:00 INFO stitch.Aggregate: DCN circuit 67931 is ready
>
> 23:15:07 INFO stitch.Aggregate: Allocation at <Aggregate
> urn:publicid:IDN+ion.internet2.edu

> <http://ion.internet2.edu>+authority+am> complete.

>
> 23:15:07 INFO stitch.launcher: All aggregates are complete.
>
> 23:15:07 INFO stitcher: Writing to
> 'OFtopov4-manifest-rspec-stitching-combined.xml'
>
> 23:15:07 INFO stitcher: Saved combined reservation RSpec at 8 AMs to
> file OFtopov4-manifest-rspec-stitching-combined.xml
>
> Stitching success: Reserved resources in slice OFtopov4 at 8 Aggregates
> (including 2 intermediate aggregate(s) not in the original request),
> creating 6 link(s)
>
>
> I then ran the readyToLogin.py script to get the login information for
> all the nodes.My topology had end hosts connected via intermediate nodes
> acting as OVS. I used this image URL-
> https://www.utahddc.geniracks.net/image_metadata.php?uuid=468c846d-5395-11e3-880c-000000000000,
> which was given by Niky. I tried using the latest OVS image, but
> apparently the image was unavailable at one of the AMs(iirc, it was a
> protogeni rack). So to be consistent I used the older image.
>
> While I am able to connect to the end hosts, when I run the
> readyToLogin script at the OVS nodes, its saying that the resource is
> not ready yet.
>
> Switch2's geni_status is: notready (am_status:notready)
>
> User aghag3 logs in to Switch2 using:
>
> ssh -p 34618 -i /Users/aditi/.ssh/geni_key_portal

> agh...@pc3.instageni.illinois.edu <mailto:agh...@pc3.instageni.illinois.edu>

>
> ssh -p 34618 -i /Users/aditi/.ssh/geni_cert_portal_key

> agh...@pc3.instageni.illinois.edu <mailto:agh...@pc3.instageni.illinois.edu>

>
> I remember Niky had said that it takes time to boot up the image, but
> its been around 12 hours, and the resource is still not ready.
>
> The slice name is OFtopov4
>
> Could anyone suggest what the issue is, why the OVS resources ain't
> ready yet?
>
>
> Thanks,
>
> Aditi
>

> --
> GENI Users is a community supported mailing list, so please help by
> responding to questions you know the answer to.
>
> If this is your first time posting a question to this list, please
> review
> http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
> ---
> You received this message because you are subscribed to the Google
> Groups "GENI Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to geni-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Leigh Stoller]

> The newest VM image with OVS has the URN urn:publicid:IDN+utahddc.geniracks.net+image+emulab-ops:Ubuntu12-64-OVS. It has been pushed to all the InstaGENI racks. Niky says this image is much more reliable.

Right, this is the image to use. I looked at Illinois and the image
is there, so you should recreate using that image.

If there is an AM without the image, please tell us so we can
fix it.

Leigh

[Aditi Ghag]

Thanks Vic.

When I was creating the topology using Flack, I did come across an AM, where the latest image was missing in the drop-down menu. If I come across it again, I will let you know. But now that I know the urn, next time I'll use that. 

Do I have to reserve the resources all over again? It took me several attempts to finally get all the resources reserved using the stitcher.py script. 

Niky mentioned in her email that the older image still works. Is there a way I can still work with my existing resources? I can upgrade the image later(if thats possible). 

I don't want to lose the reserved resources. 

Please advise. 

Aditi

[Leigh Stoller]

> Do I have to reserve the resources all over again? It took me several attempts to finally get all the resources reserved using the stitcher.py script.
> Niky mentioned in her email that the older image still works. Is there a way I can still work with my existing resources? I can upgrade the image later(if thats possible).
> I don't want to lose the reserved resources.

I tried restarting the VMs at Illinois ... check back in on them in
about 10 minutes.

Lbs

[Aditi Ghag]

Thanks a bunch. 

Please note that I have two more OVS VMs at missouri and nysernet. 

Aditi

[Leigh Stoller]

> Please note that I have two more OVS VMs at missouri and nysernet.

I restarted Missouri. The ones at Nysernet look fine.

Lbs

[Aditi Ghag]

Thanks!

I am sorry, the third one is at Wisconsin. Could you please restart that as well?

Lbs

[Leigh Stoller]

> I am sorry, the third one is at Wisconsin. Could you please restart that as well?

Done

[Aditi Ghag]

Thanks very much.

Illinois and Wisconsin are ready. 

There seems to be some issue with Missouri-

Switch1's geni_status is: failed (am_status:failed) 

On Fri, Feb 14, 2014 at 4:42 PM, Leigh Stoller <lbst...@gmail.com> wrote:

> I am sorry, the third one is at Wisconsin. Could you please restart that as well?

Done

[Niky Riga]

On 2/14/14 4:01 PM, Aditi Ghag wrote:

Niky mentioned in her email that the older image still works.

Just to clarify this point for other users in the list as well, the image that works
as well is the default Ubuntu Image that loads on xen VMs if no image is specified
(it comes with user-space OVS).

Any older OVS images that I have individually  shared the urn/url with users
should not be used anymore since they are known to be unstable in booting,
please delete any reference you have to these images.

Thanks,
Niky

[Aditi Ghag]

Hi,

I just wanted to let you know that the resource at Missouri is not ready yet.

Thanks!

Aditi

[Aditi Ghag]

Hey,

Could someone please find out why the Missouri resource isn't ready yet?

Thanks,

Aditi

[Leigh Stoller]

> Could someone please find out why the Missouri resource isn't ready yet?

This is a known problem with the XEN image that is running at Missouri.

Josh, maybe you could kill off your plastic slices at Missouri?

Lbs

[Josh Smift]

LS> This is a known problem with the XEN image that is running at Missouri.
LS>
LS> Josh, maybe you could kill off your plastic slices at Missouri?

Hmm, sure; remind me, what's the issue? And am I staying off in general,
or just clearing out while you reboot or something?

-Josh (j...@bbn.com)

[Leigh Stoller]

> Hmm, sure; remind me, what's the issue? And am I staying off in general,
> or just clearing out while you reboot or something?

The XEN41 image has that problem where it will no longer allow
any more containers cause of some kernel bug. I was not able to
push out the XEN43 image on friday, still hoping to do it this
friday.

Lbs

[Josh Smift]

LS> The XEN41 image has that problem where it will no longer allow any
LS> more containers cause of some kernel bug. I was not able to push out
LS> the XEN43 image on friday, still hoping to do it this friday.

Mm, ok; so the theory is that deleting my slivers will let other people
create VMs?

If so, I'll work on shutting things down in an orderly fashion, stay tuned.

-Josh (j...@bbn.com)

[Leigh Stoller]

> Mm, ok; so the theory is that deleting my slivers will let other people
> create VMs?

Yep. Thanks.

[Josh Smift]

JBS> Mm, ok; so the theory is that deleting my slivers will let other
JBS> people create VMs?
JBS>
JBS> If so, I'll work on shutting things down in an orderly fashion, stay
JBS> tuned.

Ok, I'm all shut down.

-Josh (j...@bbn.com)

[Leigh Stoller]

> Ok, I'm all shut down.

Thanks. I started up the container and it is running.

Lbs

[Aditi Ghag]

Cool. That did the trick. I'm now able to ssh to the missouri VM. 

Thanks!

Aditi