cloudlab geni-lib advertisements/context help?

[Brian Kroth]

Hi, I'm trying to work through a few geni-lib examples for making
somewhat more dynamic/scripted profiles.

As a first start, I'm just trying to run either the getads.py sample
script from geni-lib [1] or jacks-context.py [2] scripts to see what
data I can from the advertisements. I've tried specifying individual
sites like ig-utah or ig-wisconsin as well as letting it churn through
all of them.

Trouble is that I keep getting the following exception back from
everything:

[AUTHORIZATION] AUTHORIZATION_ERROR (Client urn:publicid:IDN+emulab.net+user+bpkroth is not authorized to make API calls.)

My config.py looks like the following currently, where the
cloudlab_geni-lib certification material was obtained from
https://www.cloudlab.us/getcreds.php and split out and decrypted via the
usual openssl commands:


from geni.aggregate import FrameworkRegistry
from geni.aggregate.context import Context
from geni.aggregate.user import User

import os

def buildContext ():
portal = FrameworkRegistry.get("portal")()
#portal = FrameworkRegistry.get("pg")()
portal.cert = os.environ['HOME']+"/private/cloudlab/cloudlab_geni-lib_cert.pem"
portal.key = os.environ['HOME']+"/private/cloudlab/cloudlab_geni-lib_key.pem"

bpkroth = User()
bpkroth.name = "bpkroth"
bpkroth.urn = "urn:publicid:IDN+emulab.net+user+bpkroth"
bpkroth.addKey(os.environ['HOME']+"/.ssh/id_rsa.pub")

context = Context()
context.addUser(bpkroth, default = True)
context.cf = portal
context.project = "bpk-test"

return context


I grabbed the urn and tried the alternative "pg" FrameworkRegistry based
on stuff I read in the following thread:
https://groups.google.com/forum/#!msg/cloudlab-users/qIbI9PvyRE8/iS27Y5q3i8sJ

The urn also matches what I see in the subjectaltnames field in the cert
itself. The cert/key match, and isn't expired:

# openssl x509 -in $HOME/private/cloudlab/cloudlab_geni-lib_cert.pem -noout -text | grep -B1 -i -e before -e bpkroth
Validity
Not Before: Oct 9 14:59:55 2015 GMT
Not After : Oct 8 14:59:55 2016 GMT
Subject: C=US, ST=Utah, O=Utah Network Testbed, OU=utahemulab.bpkroth, CN=76cab590-5486-11e4-afea-001143e453fe/emailAddress=bpk...@emulab.net
--
X509v3 Subject Alternative Name:
URI:urn:publicid:IDN+emulab.net+user+bpkroth, email:bpk...@emulab.net, URI:urn:uuid:76cab590-5486-11e4-afea-001143e453fe



Any other advice or things to look into?

Thanks,
Brian

[1] <https://bitbucket.org/barnstorm/geni-lib/src>
[2] <https://github.com/Jacks-GENI/jacks-context>

[Leigh Stoller]

> My config.py looks like the following currently, where the
> cloudlab_geni-lib certification material was obtained from
> https://www.cloudlab.us/getcreds.php and split out and decrypted via the
> usual openssl commands:
>

> portal = FrameworkRegistry.get("portal")()
> #portal = FrameworkRegistry.get("pg")()

Ah, this might be the problem. In the Geni world, the "portal" refers to
the GPO portal, not the Cloudlab portal. I think you really want the "pg"
framework? But I'm just guessing wildly.

Leigh

[Nicholas Bastin]

On Tue, Mar 29, 2016 at 4:26 PM, Brian Kroth <bpk...@cs.wisc.edu> wrote:

Trouble is that I keep getting the following exception back from everything:

[AUTHORIZATION] AUTHORIZATION_ERROR (Client urn:publicid:IDN+emulab.net+user+bpkroth is not authorized to make API calls.)

My config.py looks like the following currently, where the cloudlab_geni-lib certification material was obtained from https://www.cloudlab.us/getcreds.php and split out and decrypted via the usual openssl commands:

You should definitely not post-process the certificates you get from the cloudlab portal - just copy them into a single cloudlab.pem file. 

 

from geni.aggregate import FrameworkRegistry
from geni.aggregate.context import Context
from geni.aggregate.user import User

import os

def buildContext ():
portal = FrameworkRegistry.get("portal")()
#portal = FrameworkRegistry.get("pg")()

You want to use:

cf = FrameworkRegistry.get("emulab-ch2")()

cf.cert = "/path/to/cloudlab.pem"

cf.key = "/path/to/cloudlab.pem"

There are known problems with this endpoint that we're trying to work out, but I believe it works for some basic stuff.

 

bpkroth = User()
bpkroth.name = "bpkroth"
bpkroth.urn = "urn:publicid:IDN+emulab.net+user+bpkroth"
bpkroth.addKey(os.environ['HOME']+"/.ssh/id_rsa.pub")

context = Context()
context.addUser(bpkroth, default = True)
context.cf = portal
context.project = "bpk-test"

And just change this to context.cf = cf of course, but otherwise it looks fine.

--

Nick

[Brian Kroth]

Nicholas Bastin <nick....@gmail.com> 2016-03-29 18:44:

I tried your suggested tweaks with a freshly updated and built geni-lib from the dev branch (which I believe was the suggested code to be using last I'd checked), but it's returning errors:

# ./getads.py
Traceback (most recent call last):
File "./getads.py", line 36, in <module>
do_parallel()
File "./getads.py", line 27, in do_parallel
context = geni.util.loadContext()
File "/usr/local/lib/python2.7/dist-packages/geni_lib-0.9.1-py2.7.egg/geni/util.py", line 264, in loadContext
context.project = obj["project"]
File "/usr/local/lib/python2.7/dist-packages/geni_lib-0.9.1-py2.7.egg/geni/aggregate/context.py", line 165, in project
self.cf.project = val
File "/usr/local/lib/python2.7/dist-packages/geni_lib-0.9.1-py2.7.egg/geni/aggregate/frameworks.py", line 339, in project
self._sa = EmulabCH2.SA % (val)
TypeError: not all arguments converted during string formatting

If I switch the SA/MA attribute definitions to the commented out versions that actually contain format string placeholders, it at least tries to run, though still reports as offline with some other errors:

[ig-utah] OFFLINE

'EntryPoint' object has no attribute 'resolve'

(I added the printing of the exception string bit to the code, but that was it)


BTW, I also had to tweak the example json formatted context config file a bit to look more like the python one and stash it in a ~/.bssw/geni/ directory first, which I didn't see referenced in the tutorial docs. The sample code/docs might just need updating.

Thanks,
Brian

[Nicholas Bastin]

On Wed, Mar 30, 2016 at 5:53 PM, Brian Kroth <bpk...@cs.wisc.edu> wrote:

I tried your suggested tweaks with a freshly updated and built geni-lib from the dev branch (which I believe was the suggested code to be using last I'd checked), but it's returning errors:

Unfortunately you're a bit of a guinea pig here as we haven't debugged all of this.  That being said I've used my Cloudlab credentials to do basic things, and your problems seem odd... (oh right more below, not as odd as I'd thought)

 

If I switch the SA/MA attribute definitions to the commented out versions that actually contain format string placeholders, it at least tries to run, though still reports as offline with some other errors:

Oh right.  So there's some debate among the stakeholders right now as to whether there's a per-project sub-SA, which is what the string replacement was for (if that means nothing to you, I understand).  Unfortunately Utah needs to address this before we can really make progress - the tool can't help you if the infrastructure isn't sure how to give you a credential.  You should be able to call `listresources` with just a user credential, but obviously you need slice credentials to make real reservations.  Hopefully someone from Cloudlab/Utah can weigh in on when we'll have some idea of how to address this.

BTW, I also had to tweak the example json formatted context config file a bit to look more like the python one and stash it in a ~/.bssw/geni/ directory first, which I didn't see referenced in the tutorial docs.  The sample code/docs might just need updating.

The samples are horribly out of date.  The docs should be mostly ok for constructing a context in code, but they don't document constructing one in json.  (You can sortof guess, and your guess is probably right, but there's no docs and some of the fields don't make sense when coming from cloudlab)

--

Nick 

[Leigh Stoller]

> Oh right. So there's some debate among the stakeholders right now as to
> whether there's a per-project sub-SA, which is what the string
> replacement was for (if that means nothing to you, I
> understand). Unfortunately Utah needs to address this before we can
> really make progress - the tool can't help you if the infrastructure
> isn't sure how to give you a credential. You should be able to call
> `listresources` with just a user credential, but obviously you need slice
> credentials to make real reservations. Hopefully someone from
> Cloudlab/Utah can weigh in on when we'll have some idea of how to address
> this.

There is no ETA on when the emulab-CH2 API will do the right thing.
The PG interface does work and will do per-project sub-SA slices and
credentials (has for a very long time). I thought the "pg" framework
supported this?

Leigh

[Nicholas Bastin]

On Wed, Mar 30, 2016 at 7:16 PM, Leigh Stoller <lbst...@gmail.com> wrote:

There is no ETA on when the emulab-CH2 API will do the right thing.
The PG interface does work and will do per-project sub-SA slices and
credentials (has for a very long time). I thought the "pg" framework
supported this?

The "pg" framework is basically unimplemented.  I would need to invest some serious time into making the methods work.

--

Nick 

[Leigh Stoller]

> The "pg" framework is basically unimplemented.

Ah, I thought geni-lib could talk to the PG SA. My mistake!

Leigh

[Nicholas Bastin]

Sadly it can't, this is why we were hoping to leverage the work done on the CH2 API.

--

Nick

[Brian Kroth]

Nicholas Bastin <nick....@gmail.com> 2016-03-30 18:28:
<snip/>

>BTW, I also had to tweak the example json formatted context config file a
>> bit to look more like the python one and stash it in a ~/.bssw/geni/
>> directory first, which I didn't see referenced in the tutorial docs. The
>> sample code/docs might just need updating.
>>
>
>The samples are horribly out of date. The docs should be mostly ok for
>constructing a context in code, but they don't document constructing one in
>json. (You can sortof guess, and your guess is probably right, but there's
>no docs and some of the fields don't make sense when coming from cloudlab)

One of the sample scripts wanted the json version through some part that
was buried in another library that I didn't quite bother to track down.
I mostly just ran it by hand and kept adjusting the json until it
stopped complaining and moved on to a different sort of error message.
Aside from a few of the field names being slightly different, it more or
less ended up looking like the python:

In case anyone else needs it, the minimal set seemed to be something
like this:

# cat ~/.bssw/geni/context.json
{
"framework" : "emulab-ch2",
"cert-path" : "/path/to/cloudlab.pem",
"key-path" : "/path/to/cloudlab.pem",
"user-name" : "yourusername",
"user-urn" : "urn:publicid:IDN+emulab.net+user+yourusername",
"user-pubkeypath" : "/home/yourusername/.ssh/id_rsa.pub",
"project" : "yourprojectname"
}

In other words, I ended up collapsing the users and frameworks arrays
that were in the sample into just tagged key-value pairs.

Cheers,
Brian

[Nicholas Bastin]

On Wed, Mar 30, 2016 at 8:54 PM, Brian Kroth <bpk...@cs.wisc.edu> wrote:

In case anyone else needs it, the minimal set seemed to be something like this:

# cat ~/.bssw/geni/context.json
{               "framework" : "emulab-ch2",
        "cert-path" : "/path/to/cloudlab.pem",
        "key-path" : "/path/to/cloudlab.pem",
        "user-name" : "yourusername",
        "user-urn" : "urn:publicid:IDN+emulab.net+user+yourusername",
        "user-pubkeypath" : "/home/yourusername/.ssh/id_rsa.pub",
        "project" : "yourprojectname"
}

In other words, I ended up collapsing the users and frameworks arrays that were in the sample into just tagged key-value pairs.

Right, I should have mentioned this - this is the right format (and user-urn does not have to be correct, but must be supplied).  You can then load this using:

geni.util.loadContext("/path/to/cloudlab/context.json", key_passphrase=True)

This actually works for me for getting advertisements:

>>> context = util.loadContext("~/.bssw/geni/clcontext.json", key_passphrase=True)            
Private key passphrase:
>>> IGAM.UtahDDC.listresources(context)
<geni.rspec.pgad.Advertisement object at 0x7fd91d8ca050>

What calls are you making that fail?

--

Nick

[Leigh Stoller]

> Sadly it can't, this is why we were hoping to leverage the work done on the CH2 API.

So, what if Brian already had a slice credential in the correct project?
Is the geni-sa/geni-ma API needed after that?

Leigh

[Nicholas Bastin]

I did a bunch of work last night and now the only thing that doesn't work is listSlices (which is annoying, but not a showstopper).  I filed a few tickets that would help resolve some usability issues , but you can get slice credentials in geni-lib now from the emulab CH2 endpoint now.

--

Nick

[Leigh Stoller]

> I did a bunch of work last night and now the only thing that doesn't work
> is listSlices (which is annoying, but not a showstopper). I filed a few
> tickets that would help resolve some usability issues , but you can get
> slice credentials in geni-lib now from the emulab CH2 endpoint now.

Thanks Nick, much appreciated.

Leigh

[Brian Kroth]

Nicholas Bastin <nick....@gmail.com> 2016-03-31 07:35:

Same ones. Though, I got it to spit out a better backtrace now and I
think it's actually a problem in python cryptography module on my
system. I'll see what I can do about that first.

# ./getads-wisc.py
Private key passphrase:

Traceback (most recent call last):

File "./getads-wisc.py", line 24, in <module>
context = geni.util.loadContext("~/.bssw/geni/context.json", key_passphrase=True)
File "/usr/local/lib/python2.7/dist-packages/geni_lib-0.9.1-py2.7.egg/geni/util.py", line 254, in loadContext
cf.setKey(obj["key-path"], key_passphrase)
File "/usr/local/lib/python2.7/dist-packages/geni_lib-0.9.1-py2.7.egg/geni/aggregate/frameworks.py", line 125, in setKey
key = serialization.load_pem_private_key(open(path, "rb").read(), passwd, default_backend())
File "/usr/local/lib/python2.7/dist-packages/cryptography-1.3.1-py2.7-linux-x86_64.egg/cryptography/hazmat/backends/__init__.py", line 35, in default_backend
_default_backend = MultiBackend(_available_backends())
File "/usr/local/lib/python2.7/dist-packages/cryptography-1.3.1-py2.7-linux-x86_64.egg/cryptography/hazmat/backends/__init__.py", line 22, in _available_backends
"cryptography.backends"
AttributeError: 'EntryPoint' object has no attribute 'resolve'

Thanks,
Brian

[Nicholas Bastin]

Maybe is a problem with your setuptools version?

https://github.com/pyca/cryptography/issues/2838

--

Nick

[Brian Kroth]

Sorry for the delay - busy last week or so.

Nicholas Bastin <nick....@gmail.com> 2016-03-31 13:23:

>Maybe is a problem with your setuptools version?
>
>https://github.com/pyca/cryptography/issues/2838

Probably. I ended up just pinning the cryptography module requirement
to version 1.2.1 for now in setup.py. With that I was able to run one
of the sample scripts for fetching ads.

I'll probably poke some more at actually using them later this week.
Will prod the list again if I run into more troubles.

Thanks for your help.

Cheers,
Brian