non-interactive webservice workflow

[Jim Rae]

Has anyone successfully created an non-interactive workflow. Basically, what I am trying to accomplish is this. From my CRM, I am trying to do a Valance API callout to my LMS that will create an LMS user when the user is created in the CRM.

This needs to occur without any "human interaction".

I am using the Valance Java SDK (https://github.com/Desire2Learn-Valence/valence-sdk-java) as a framework for my code, at at its core, am executing this Flow:

• D2LAppContext DAC = new D2LAppContext(aid,key);
  
• URL targetURL = new URL(protocol+endpt + actiontype);
  
• URL authEP = DAC.createWebUrlForAuthentication(endpt , hport, targetURL);
  
• httprequest (Get) authEP (which returns a 200 status code - OK)
  
  
• ID2LUserContext DUC = DAC.createUserContext(authEP, endpt, hport, true);  ( I have also tried this with passing the userid and userkey generated from the API test application, based on logging in as an interactive user.
  
• URL userURL = DUC.createAuthenticatedUri(actiontype, 'GET');
  
• httprequest (Get) userURL (which returns a 403 status code - Forbidden)

Does anyone have a suggestion on how to accomplish this? One question I have is which userid and userkey is the API expecting? The one I use to login to the LMS with via the UI? Or something returned from the Authorization call?

Thanks!

Jim

[Desire2Learn Staff: Sarah-Beth]

Hi Jim

This is a very common scenario for folks using the Valence Learning Framework APIs. We answered a similar question on StackOverflow a while back, but I've repeated the answer below so that we have it here in the ValenceUsers forum as well.

Your app will need to run with a Service Account, or other Account with sufficient privileges to perform the actions you need to take with the APIs.

Once you create a Service Account, you need to manually harvest the user tokens using a utility such as the API Test Tool (https://apitesttool.desire2learnvalence.com/) to authenticate with your LMS. You then need to store those keys securely, and configure your LMS to ensure the user tokens are long-lived. Many systems have a token timeout of 30 days, but when a headless integration is in place like the one you're proposing, it's often a good idea to make the timeout infinite. You can contact Desire2Learn Support to verify the timeout value for the user tokens.

[Desire2Learn Staff: Viktor]

There are several aspects of your stated workflow that seem odd to me. With the most recent version of the Java SDK available on the D2L GitHub site, I think the right workflow is something like this. NOTE: this uses methods that are updated, and not the "deprecated" older methods; the workflow is thus subtly different.

Before you start

You need to have this information:

- Your Application's ID/Key pair (let's say 'appID' and 'appKey', but they won't look like that)

- The domain for your back-end service (which LMS are you calling): 'https://d2l.someUniversity.edu' or something like that

- The callback point for your app/service where the LMS can send User ID/Key pairs when you request them: 'https://yourServer.org/authCallback' or something like that

Build an app context

- Construct an App context with D2LAppContext( "appID", "appKey", "https://d2l.someUniversity.edu"); this fashions an app context that you can then use to make the request to fetch a User ID/Key pair from the LMS

- authURL = DAC.createWebUrlForAuthentication( new URI("https://yourServer.org/authCallback")); this makes the "authURL", you need to direct a web-container (the user's browser) to this URL: that requests the LMS send back User ID/Key pair for the user session that will get established inside the requesting web container.

The LMS will send the User ID/Key pair by redirecting back to https://yourServer.org/authCallback with the User ID/Key pair attached as query parameters on the redirect URL. Let's call this full redirect URL 'fullRedirectUrl'.

Build a user context

- You can either build a user context by using fullRedirectUrl: DAC.createUserContext( fullRedirectUrl )

- OR, you can extract the User ID/Key pair from fullRedirectUrl (let's say 'userID' and 'userKey', but they'll look different), and use: DAC.createUserContext( "userID", "userKey" )

Make calls

- You can now use your user context, DUC, to make calls: DUC.createAuthenticatedUri( "/d2l/api/lp/1.3/users/whoami", "GET")

- Note that here you provide the API route only, plus the HTTP method (uppercase); make sure you use a route in an API contract that your back-end LMS supports -- here we make the whoami call from the v1.3 API contract, but your LMS may not support that contract (or it may support a more recent one).

Hope this helps.

[Sean M]

One note for the uninitiated like myself: "user tokens" above does not mean the timeout set via the d2l.SessionTimeout server setting. Rather, this is set via the d2l.Security.Api.TokenTimeout setting. While the SessionTimeout setting seems at first glance to be relevant to this question, it isn't.

Putting this another way, "user tokens" above does not mean the returned x_a/x_b/x_c and cookies set during a standard user login. This timeout is specific to the API tokens sent back via the API login process.

[Desire2Learn Staff: Sarah-Beth]

Thanks for contributing to the conversation, Sean! You bring up a very good point.

To clarify further: 

• The d2l.SessionTimeout config variable refers to the timeout period of the web session (i.e. how long someone can be idle before the LMS web UI times out and prompts the user to log back in). This timeout is often set at an interval of several minutes to an hour.
• The d2l.Security.Api.TokenTimeout config variable refers to the timeout period of the user id\key pair associated with the Valence authentication process. Put another way, these tokens are the x_b value and the basis for the x_d value that are sent as query parameters on every API call to identify the user. This timeout is often set at an interval of several days, or could be set to never timeout. In the latter case, the user id\key pair can only be invalidated by changing the user's password or explicitly revoking app access. Once the user id\key pair becomes invalid, the app will have to re-initiate the authentication process to receive a new user id\key pair to sign API calls.

[Desire2Learn Staff: Viktor]

It's probably useful to point out that the default setting for d2l.Security.Api.TokenTimeout should be "indefinite" on most LMS installations. This means that the User ID/Key pair will only become expired when the user's password changes, or when an admin revokes the user's access to their party applications.

However, clients can request for a defined expiry time for this value, if they feel they have need, and in those cases, D2L typically recommends a month. This means that any application or service using the Valence Learning Framework APIs (including mobile applications like D2L's Assignment Grader) will likely require the user to re-authenticate themselves with the LMS along with that expiry cycle (once a month or more often).

As general guidance, because API access is essentially tied to a user's password to gain access to an LMS, admins should think of the expiry cycle for API access in a similar way -- typically unless admins have particular concerns, we recommend that a fixed expiry time for this value is only needed if you want API access to operate on a shorter cycle than user passwords (because changing a user's password will force User ID/Key expiry anyway).

[Jeffrey Kahn]

This response was very helpful.  Here is what I have done, with specifics changed:

            String appID = "myAppID";
            String appKey = "myAppKey";
            String host = "me.desire2learndemo.com";        
            D2LAppContext DAC = new D2LAppContext(appID, appKey, host);

            URI authURL = DAC.createWebUrlForAuthentication( new URI("http:/myserver.com/d2l/authCallback.jsp"));
            ID2LUserContext DUC = DAC.createUserContext(authURL);
           
            URI coursesURL = DUC.createAuthenticatedUri( "/d2l/api/lp/1.3/courses/schema", "GET");
            System.out.println("coursesURL " + coursesURL);            

This all seems to be working well.  The authURL contains x_a, x_b, and x_target values.  The coursesURL contains x_a, x_b, x_c, x_d, and x_t.

When I call this URI, I get a 403 error.

When I paste this URI to my browser, I am redirected to login, which I am doing with admin credentials.  The login passes, but I see the message:

"This application is not authorized on this LMS instance. Ask your administrator to authorize this application."

What should I do?

[Sean M]

I've seen that error in two general cases:

1) The user you're connecting with doesn't have rights to that specific information (general course structure info, in this case)

2) The app ID/app key you're using is authorized for a different server than the one you're connecting to. Check the URI listed in the Valence keytool to make sure it's actually authorized for me.desire2learndemo.com (or whatever server you're actually connecting to)

Wish I could give you very specific info, but narrowing that one is a process. Maybe someone more experienced will weigh in!

[Desire2Learn Staff: Viktor]

On Wednesday, 25 June 2014 11:31:34 UTC-4, Jeffrey Kahn wrote:

This response was very helpful.  Here is what I have done, with specifics changed:

            String appID = "myAppID";
            String appKey = "myAppKey";
            String host = "me.desire2learndemo.com";        
            D2LAppContext DAC = new D2LAppContext(appID, appKey, host);

            URI authURL = DAC.createWebUrlForAuthentication( new URI("http:/myserver.com/d2l/authCallback.jsp"));

 // user interaction required here

 

            ID2LUserContext DUC = DAC.createUserContext(authURL);

There's a problem directly here.

The URL that you 'create for authentication' is the URL your service sends to the LMS to ask for a set of user tokens for the "user currently driving the browser". The callback you provide (as you noted) is for the callback-endpoint on your service where the LMS should send the user ID/Key pair once it can determine who that user is.

When the LMS redirects the flow, with UserID/Key known, back to your callback, it will attach the User ID/Key pair onto that callback as query parameters that you can harvest.

The DAC.createUserContext(resultURL) call therefore will not be the 'authURL' from your sample, but will actually be your callback endpoint with query parameters added containing the user token data.

There is a point at which user interaction will be required (see my inserted comment in your code). Your app/service needs to send the user's browser to that authURL and then trust that the LMS will auth the user appropriately, and eventually, redirect the flow back to your stated endpoint. At that point, you'll have a UserID/Key and you can proceed to make calls.

There's no easy way to do this in a headless manner -- you'll need to manually step through this process least once to get the LMS to authenticate a user, and then you'll get tokens for that user, which you securely cache and use to proceed (this is if you need to have a two legged scenario where your service is directly working, headlessly, with the LMS). The auth process for the Valence LF API is specifically designed to work in a three-legged scenario where a user with a user agent is interacting with your service, and with the LMS.

 

           
            URI coursesURL = DUC.createAuthenticatedUri( "/d2l/api/lp/1.3/courses/schema", "GET");
            System.out.println("coursesURL " + coursesURL);            

This all seems to be working well.  The authURL contains x_a, x_b, and x_target values.  The coursesURL contains x_a, x_b, x_c, x_d, and x_t.

When I call this URI, I get a 403 error.

When I paste this URI to my browser, I am redirected to login, which I am doing with admin credentials.  The login passes, but I see the message:

"This application is not authorized on this LMS instance. Ask your administrator to authorize this application."

Here you also have a problem -- your LMS is not set up to have your application (as identified by your App ID/Key pair) enabled, so the auth sequence will not work.

Overall the auth workflow looks like this:

1) User visits your site, in order to do something; your site needs to make Valence API calls to help them do that.

2) Your site uses the "get me user tokens" API call for the LMS (http://docs.valence.desire2learn.com/res/apiprop.html#get--d2l-auth-api-token) by getting the user's browser to visit that API URL. NOTE that, in this call, your x_a parm contains your AppID, and your x_b parm contains your App Signature, and your x_target parm contains your site's callback endpoint.

3) The LMS redirects the user's browser through the LMS's authentication process to establish an LMS user web session for that user. If the User is already logged into the LMS in the browser, then this auth process can be shortcut; if not, the LMS gets the user to "log in" using it's standard way of doing that (which varies wildly LMS to LMS). Once the LMS knows who the user is, it can build the correct UserID/Key pair.

4) LMS sends the UserID/Key pair for that user AND your app back to your site by redirecting the flow back to your x_target endpoint (specified in step 2), with the User ID/Key attached as query parameters. Once you get this callback to your endpoint, you can grab THAT URL (with its query parameters) and use it to create your "User Context".

5) User your UserContext object to do the decoration on all the API calls you want to make -- these calls will now be identified as coming from your site, and that user, when they get to the LMS.

 

What should I do?

[Jeffrey Kahn]

OK, I think I am onto something better.  I went to the apitesttool.desire2learnvalence.com site.  I entered my host and appID and appKey.  I clicked on authenticate, logged in system an admin account, and saw the userID and userKey.  I copied these into my program.  So now I have the host, the appID, the appKey, the userID and the userKey as static values.

In my program, I call this to start:

ID2LUserContext DUC = new D2LUserContext(_host, _appId, _appKey, userId, userKey);

then

URI whoamiURL = DUC.createAuthenticatedUri( "/d2l/api/lp/1.3/users/whoami", "GET");

and now I get a JSON response!

[Desire2Learn Staff: Jacob Parker]

Sounds good!

Please be mindful that if you are hard-coding tokens for a user with the authorization to do X, Y and Z then if your service is compromised then an attacker could gain that same authorization. Thus, when creating the user you should attempt to give it as few permissions/enrollments as necessary for your service.

You should also be careful with where you store "service account" user tokens (in source code, configuration files etc.) and make sure to never transmit them indirectly to an end-user.

[Rodregus]

Is there any way to get userId and userKey, without an interactive way? ie. something like , i could pass username and password and the api returning me the userId and UserKey? If so i could avoid using a browser just to get username and password from user, which is a weired experience.

[Desire2Learn Staff: Jacob Parker]

We don't support anything other than the current browser-based auth workflow. For client-side (e.g. you give the user a .exe) the open-a-new-browser method is the best possible at this time. Alternatively, depending on your environment, you could open the browser within your app (e.g. WebViews on iOS/Android, etc.) which may be less jarring.

(If you tried real hard you could avoid using the browser directly, obviously, but it is via an avenue which we do not support and is subject to breaking changes--so its not a good idea.)

[UCB VALENCE]

I have been doing this same technique , but continue to get the 403 error.

        AuthenticationSecurityFactory factory1 = new AuthenticationSecurityFactory();
        D2LUserContextParameters ducp = new D2LUserContextParameters(APPID, APPKEY,USERID,USERKEY,HOST,PORT,true);
        ID2LAppContext appContext2 = factory1.createSecurityContext(APPID, APPKEY); ;
        ID2LUserContext userContext2;
        userContext2 = appContext2.createUserContext(ducp);
       
        URI newUri1 = userContext2.createAuthenticatedUri("/d2l/api/versions", "GET");
        URLConnection connection1 = null;
       
        try{
            connection1 = newUri1.toURL().openConnection();
            System.out.println("Opened connection");
            HttpURLConnection httpConnection = (HttpURLConnection) connection1;
            httpConnection.setRequestMethod("GET");
            connection1.setRequestProperty("Accept", "application/json");
           
            BufferedReader br = new BufferedReader(new InputStreamReader((connection1.getInputStream())));
        
                String output;
                System.out.println("Output from Server .... \n");
                while ((output = br.readLine()) != null) {
                    System.out.println(output);
                } 

1) The userid, userkey are static and are harvested from the apitesttool after the first authentication using the registered appid and appkey.  the host is ucbdev.desire2learn.com.  port is 443.  When I look at the "TYPE" in the keytool, it says it is limited.  Would that be part of the problem?      

[Desire2Learn Staff: Sarah-Beth]

An important clarification - Keytool issues App ID\Key pairs for apps that you register. In the case of the App type, Limited means it's restricted to a specific LMSID which is usually tied to only one environment. The alternative is a Universal app which works against all instances, but these are only issued for D2L products and certain D2L Partner products. So the fact that you have a Limited app is expected. Provided Keytool also show this app as Approved, then you should be in good shape on that front.

App ID\Key pairs associated with the app you register in Keytool are different than User ID\Key pairs The type of app (Limited vs Universal) has no impact on if/when the User ID\Key pairs expire.

Here's something to check - in the API Test Tool, input the target Host, the App ID\Key pair values, and click Manually Set Credentials. Then paste in the User ID\Key pair values that you're attempting to use in your app. Then make a call like the WhoAmI call in the API Test Tool. Does this call work, or are you getting a 403? If it works in the API Test Tool, this suggests that the App ID\Key pair and the User ID\Key pair are all functioning correctly. 

[UCB VALENCE]

Hi Sarah-Beth,

When I perform the test you suggested in the last paragraph, and submit the /d2l/api/versions/ GET request or /d2l/api/lp/1.0/users/whoami GET request, those requests work correctly.  But when I use the same App ID\Key pairs and User ID\Key pairs in the code above, a 403 is returned for the /d2l/api/versions/ GET request

Kevin

[Desire2Learn Staff: Sarah-Beth]

That, to me, suggests that there's something wrong with how your code is handling those Auth parameters. 

Have you tried running a Fiddler trace to get more details on exactly what's being passed and where in process the error is occurring?

Anyone else have suggestions, based on the code shared in the thread above? 

[Brian Looker]

I'm spoiled coding in Python and using the interpreter to check that everything is working as I go along. In any case, my Java is a little rusty. 

The route for versions in your code should have a trailing slash. 

URI newUri1 = userContext2.createAuthenticatedUri("/d2l/api/versions/", "GET");

I don't think that would result in a 403, but it can cause problems. 

[kkm...@colorado.edu]

thanks Brian.  I am posting some additional steps to take if others are getting 403 with Java.

1) use the trailing slash Brian posted above. 

URI newUri1 = userContext2.createAuthenticatedUri("/d2l/api/versions/", "GET");

2) make sure you are listed as a watcher in https://github.com/Desire2Learn-Valence/valence-sdk-java so as to use the latest api.
3) The last x_t is a timestamp parameter.  A skew is calculated in D2LUserContext.calculateServerSkewFromResponse().  Make sure the client has its clock set within a reasonable time of a network time protocol server.

[Desire2Learn Staff: Sarah-Beth]

Glad to know that Brian's suggestion helped. (Great catch, Brian!)

And thanks for posting those tips!