geni-lib Context credentials

[asep...@gmail.com]

Hi all,

I've been reading this tutorial to get to know about geni-lib
http://geni-lib.readthedocs.org/en/latest/tutorials/gettingstarted.html

However, I could not find information on how/where to get portal user key & certificate.
FYI I only have cloudlab portal user.

framework = FrameworkRegistry.get("portal")()
  framework.cert = "/home/user/.ssh/portal-user.pem"
  framework.key = "/home/user/.ssh/portal-user.key"

Is geni-lib only for generating RSpec file or can I also use it to start an experiment without having to interact with cloudlab web-portal?

Thank you.

Asep

[Leigh Stoller]

> I've been reading this tutorial to get to know about geni-lib
> http://geni-lib.readthedocs.org/en/latest/tutorials/gettingstarted.html
>
> However, I could not find information on how/where to get portal user key & certificate.
> FYI I only have cloudlab portal user.
>
> framework = FrameworkRegistry.get("portal")()
> framework.cert = "/home/user/.ssh/portal-user.pem"

> framework.key = "/home/user/.ssh/portal-user.key”

Hi. The CloudLab portal is a different kind of portal then the Gene portal.
The above code fragment refers to your cert/key from the Geni Portal at
https://portal.geni.net, if you have a Geni portal account.

If you do not have a Geni portal account, but instead have only a CloudLab
portal account, then we need to get you your cert/key. No one has asked for
this yet, so first lets figure out what you need before we implement
something new.

Can you describe what you would like to do? If it is just to use geni-lib
to create your topology, then you can already do that via the CloudLab
portal. See section 5.3 in the manual (http://docs.cloudlab.us/).

Leigh

[Asep Noor Mukhdari Sutrisna]

Hi Leigh,

I’m trying to utilise CloudLab infrastructure (bare metal nodes ) to run as ToMaTo hosts on demand. (http://tomato-lab.org/)
So, when we need extra resources to do an experiment or running a tutorial, we can just instantiate several nodes on CloudLab and automatically add them to our tomato-backend.

What I need to do is a scripted/automated way to instantiate and terminate a node and get information about that node (ip address/hostname) which I believe can be achieved by using geni-lib.

I know I can generate topology/RSpec using geni-lib via cloudlab portal, but I also need to instantiate that topology from outside CloudLab portal.

If I have a GENI portal account, can I also access CloudLab hardware from that account?

Thank you for your help.

Regards,

Asep

[Leigh Stoller]

> If I have a GENI portal account, can I also access CloudLab hardware from that account?

Yes, you can.

Leigh

[Robert Ricci]

To expand a bit on the answer...

Yes, for this case, you will want to download and use your keys locally
in the script. We do generate certificates and credentials for you, but
at the moment, we don't have the interface in our web UI to download
them; As Leigh said, in the meantime, if you have a GENI portal account,
those credentials will work with CloudLab, so you can use them.

The geni-lib support that Leigh pointed you to is intended for a case
where you want a nice flexible way to build the topology by writing it
as a python script (as opposed to drawing it with a GUI or giving us a
hand-created RSpec); basically, it's all about generating RSpecs. It's
not (so far) intended for what you want to do, where you are using
geni-lib to make API calls.

> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To post to this group, send email to cloudla...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/6A94C46E-4A93-49C0-B5E9-63091158C7F2%40gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Asep Noor Mukhdari Sutrisna]

> On Feb 9, 2015, at 9:25 PM, Robert Ricci <ri...@cs.utah.edu> wrote:
>
> To expand a bit on the answer...
>
> Yes, for this case, you will want to download and use your keys locally
> in the script. We do generate certificates and credentials for you, but
> at the moment, we don't have the interface in our web UI to download
> them; As Leigh said, in the meantime, if you have a GENI portal account,
> those credentials will work with CloudLab, so you can use them.
>
> The geni-lib support that Leigh pointed you to is intended for a case
> where you want a nice flexible way to build the topology by writing it
> as a python script (as opposed to drawing it with a GUI or giving us a
> hand-created RSpec); basically, it's all about generating RSpecs. It's
> not (so far) intended for what you want to do, where you are using
> geni-lib to make API calls.

Hi, thanks for the explanation.

I have created GENI user, tried to use it and I can query cloudlab aggregate using the account.
However, If I use GENI account, how can I join my existing project in CloudLab?
I tried to call createsliver method by providing my cloudlab project name (cloudlab-tomato), but it gave me exception error (No slice credential returned).
Do I have to create another project in Geni portal too?

If it’s a planned feature and it’s actually possible to generate cloudlab portal credential for me, that would be great.
Thank you.



Regards,

Asep

[Leigh Stoller]

> I tried to call createsliver method by providing my cloudlab project name (cloudlab-tomato), but it gave me exception error (No slice credential returned).

Hi, are you using omni and the —project (-r) argument.

Leigh

[Asep Noor Mukhdari Sutrisna]

> On Feb 10, 2015, at 1:47 AM, Leigh Stoller <lbst...@gmail.com> wrote:
>
>> I tried to call createsliver method by providing my cloudlab project name (cloudlab-tomato), but it gave me exception error (No slice credential returned).
>
> Hi, are you using omni and the —project (-r) argument.
>

No, I’m using geni-lib examples to build context first and then use it as parameter to createsliver.

Here’s mycontext.py
rom geni.aggregate import FrameworkRegistry
from geni.aggregate.context import Context
from geni.aggregate.user import User

def buildContext ():

framework = FrameworkRegistry.get("portal")()

framework.cert = “/my/.keys/geni-sutrisna.pem"
framework.key = “/my/.keys/geni-sutrisna.pem"

user = User()
user.name = "sutrisna"
user.urn = "urn:publicid:IDN+ch.geni.net+user+sutrisna"
user.addKey(“/my/.keys/id_rsa.pub")

context = Context()
context.addUser(user, default = True)
context.cf = framework
context.project = "cloudlab-tomato”

return context


And here the actual script that creates topology and runs it:

import mycontext
import geni.aggregate.cloudlab
import geni.rspec.pg as PG

context = mycontext.buildContext()

CL = geni.aggregate.cloudlab

# Debian 7.6 Proxmox VE tomato-hostmanager disk image
DISK_IMAGE="https://www.apt.emulab.net/image_metadata.php?uuid=ed0e54ed-a71d-11e4-9439-db9edc46fe2c"

r = PG.Request()

node = PG.RawPC(“thm-node")
node.disk_image=DISK_IMAGE

r.addResource(node)
r.writeXML("thm-gl.xml")

m = CL.UtahDDC.createsliver(context, "thm-node", r)
m.writeXML("thm-manifest.xml”)

So, in theory can my geni-portal account uses my cloudlab-portal project?

Regards,

Asep

[Leigh Stoller]

> No, I’m using geni-lib examples to build context first and then use it as
> parameter to createsliver.

Well, internally geni-lib is using omni, and I do not think it can
do InstaGeni projects. So your certificate and key are not enough,
you need a slice credential in the right project too.

We are going to have to discuss this internally and get back to you.

Leigh

[Asep Noor Mukhdari Sutrisna]

Oh, I see.. I don’t really know about omni and maybe still confused about difference of InstaGeni, ProtoGeni, Cloudlab..

Thank you, I’m looking forward to hearing from you.

Regards,

Asep

[Leigh Stoller]

> Oh, I see.. I don’t really know about omni and maybe still confused about
> difference of InstaGeni, ProtoGeni, Cloudlab..

InstaGeni and ProtoGeni are the same thing, I use them interchangeably but
I really shouldn't.

CloudLab uses the *Geni APIs to create the physical instantiations of your
profiles on the Cloudlab/APT clusters. Its a portal, but a different kind
of portal then the GPO portal.

Leigh

[Robert Ricci]

Sorry for all of the questions :) but one more (I want to make sure we
are spending our time solving the right problems...

Do you explicitly care about the project stuff in the GENI portal, or if
we gave you some example code that called the APIs without getting GENI
projects involved, would that fit what you are trying to do?

[Asep Noor Mukhdari Sutrisna]

> On Feb 10, 2015, at 6:13 PM, Robert Ricci <ri...@cs.utah.edu> wrote:
>
> Sorry for all of the questions :) but one more (I want to make sure we

> are spending our time solving the right problems…

No problem. I’m glad with your support.

> Do you explicitly care about the project stuff in the GENI portal, or if
> we gave you some example code that called the APIs without getting GENI
> projects involved, would that fit what you are trying to do?

Actually no, as long as I can instantiate my topology in cloudlab x86 cluster using a script (not web interface), that should be sufficient.
Yes, some example would be great.

Thanks,

Asep

[Asep Noor Mukhdari Sutrisna]

>
>> Do I have to create another project in Geni portal too?
>

> No, your project is sufficient, but you do need to create a _slice_
> belonging to that project (with the "thm-node" name, matching the one
> used in your script's CreateSliver call).
>
> You can do that on the portal "Projects" page:
>
> https://portal.geni.net/secure/projects.php
>
> using the "Create Slice" button in the right hand column of the project
> list. You can reuse the same slice for many sliver instantiations, and
> slices can last for quite a long time (the default portal slice lifetime
> is one week, and they can be extended), so it shouldn't be too much of
> a burden to use the web interface for slice creation.
>

Hmm.. I couldn’t find the "Create Slice” button on the project page..
Perhaps because I didn’t join any project in Geni Portal? (and my account isn’t Project Lead)
Currently I only have a project in CloudLab portal, as I was planning to use only that account.

So, maybe I should ask someone to create or let me join a project in geni portal.


> (It is also possible to create slices via geni-lib. However, you
> probably don't want to do it in the same script as your sliver
> creation one, since the expectation is that slices will live for
> longer than slivers, and recreating a slice with the same name
> before it expires will fail.)

I haven’t really dig in geni-lib, but I hope there’s also a way to check slice or slivers expiration.
Thank you for the explanation.

Regards,

Asep

[Robert Ricci]

Okay, sounds like the best thing for us to do is just going to be to get
you your CloudLab certificate and credentials. We'll work on the UI for
this and get back to you soon.

[Leigh Stoller]

> Okay, sounds like the best thing for us to do is just going to be to get
> you your CloudLab certificate and credentials. We'll work on the UI for
> this and get back to you soon.

Hi. If you login to the CloudLab portal, you will see a new item
in the Actions menu; Download Credentials. Note that the passphrase
on the key is the same as your web login password. We don’t have
a UI to change that yet, but that can be done offline with standard
openssl tools if you need to.

Leigh

[Robert Ricci]

You will also need to modify your context.py file to use a different
framework, and will need to make sure to make a call to create the slice
(from your main script) before adding nodes to it. We're testing this
out to make sure that geni-lib supports this properly for CloudLab.

[Asep Noor Mukhdari Sutrisna]

Wow, great! Thanks a lot guys.

btw, my user.urn, is it:
user.urn = “urn:publicid:IDN+cloudlab.us+user+sutrisna” ?


Regards,

Asep

[Leigh Stoller]

> Wow, great! Thanks a lot guys.
>
> btw, my user.urn, is it:
> user.urn = “urn:publicid:IDN+cloudlab.us+user+sutrisna” ?

Hi. Your URN is urn:publicid:IDN+emulab.net+user+sutrisna

Leigh

[Asep Noor Mukhdari Sutrisna]

> On Feb 13, 2015, at 4:05 AM, Robert Ricci <ri...@cs.utah.edu> wrote:
>

> You will also need to modify your context.py file to use a different
> framework, and will need to make sure to make a call to create the slice
> (from your main script) before adding nodes to it. We're testing this
> out to make sure that geni-lib supports this properly for CloudLab.
>

Hi, sorry for asking too much.
What framework should I use? Could you give me a little example plus creating the slice part?

I’ve modified my context.py to use the new credentials (keys & urn), but when I tried this:

r = context.cf.createslice(context,'THMNODES')
17:52:06 ERROR : Error from server creating slice. Code 2: [AUTHORIZATION] AUTHORIZATION_ERROR (Client urn:publicid:IDN+emulab.net+user+sutrisna is not authorized to make API calls.)
17:52:06 ERROR : Create Slice Failed for slice name THMNODES.

I’ve also tried to change ‘portal’ to ‘pg':
framework = FrameworkRegistry.get("portal")() to framework = FrameworkRegistry.get("pg")()
This resulted to an exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/geni_lib-1.0-py2.7.egg/geni/aggregate/frameworks.py", line 62, in createslice
args = ["--warn", "--AggNickCacheName", context.nickCache, "-c", context.cfg_path, "-f", self.name, "--usercredfile", context.usercred_path, "createslice"]
File "/usr/local/lib/python2.7/dist-packages/geni_lib-1.0-py2.7.egg/geni/aggregate/context.py", line 158, in cfg_path
l.extend(self.cf.getConfig())
AttributeError: 'ProtoGENI' object has no attribute ‘getConfig'

I’ve also updated my geni-lib installation.


Thank you,

Asep

[Asep Noor Mukhdari Sutrisna]

>
> After that, I believe the script should successfully create both a
> slice and a sliver for you. Please let me know if it does not!

Thank you so much, it’s working now :)
The disk_image property doesn’t accept disk image URL though, it works with URN.

Btw, should I create a slice every time I create sliver, or as you suggest earlier? (create slice on different script as sliver)

Thanks again,

Asep

[Asep Noor Mukhdari Sutrisna]

Hi, all
It’s been a while ;)

I’ve encountered this error when running a geni-lib script to create slice and instantiate CloudLab profile again.
The very same script worked fine last time I run (mid Feb 15).

12:28:27 ERROR : Failed to create new GENI Clearinghouse slice urn:publicid:IDN+emulab.net:cloudlab-tomato+slice+THMNODESLICE04: Malformed arguments: *** verifygenicred:
invalid credential (expired at 2015-02-14T19:46:51Z) (code 1)

I have re-downloaded my credentials from CloudLab portal and checked the cert file:
openssl x509 -noout -enddate -in .keys/cloudlab-cred.pem
notAfter=Dec 17 13:49:13 2015 GMT

However I’m still getting the same error.
Did I miss something?

regards,

Asep

[Robert Ricci]

From the message below, it looks like your credential expired February
14, you'll need to get a fresh one, which I believe you can do on the
GENI portal.

For what it's worth, we had a discussion with the primary author of
geni-lib and have a plan sketched out so that you won't need Gary's
patches to use it against CloudLab in the future. This is planned for
the 1.0 release of geni-lib.

[Asep Noor Mukhdari Sutrisna]

> On Apr 15, 2015, at 11:14 PM, Robert Ricci <ri...@cs.utah.edu> wrote:
>
> From the message below, it looks like your credential expired February
> 14, you'll need to get a fresh one, which I believe you can do on the
> GENI portal.

I thought GENI Portal credential and the one I downloaded from CloudLab are different?
Anyway, I refreshed and downloaded new credential from GENI Portal but still getting the same expired credential error (same date in the error msg: 14 Feb 2015).

> For what it's worth, we had a discussion with the primary author of
> geni-lib and have a plan sketched out so that you won't need Gary's
> patches to use it against CloudLab in the future. This is planned for
> the 1.0 release of geni-lib.

That’s great! Many thanks :)

Regards,

Asep