Re: [learningreg-dev] Slice & Resumption_token

[Grata, Walt]

Sorry about that Patrick, it looks like there is a precedence problem between the data in the resumption_token and the any_tags params.  It will give correct results if you remove the any_tags from subsequent requests.  I'll create a ticket and get a fix in.

On Tue, Apr 9, 2013 at 5:48 PM, Patrick Walsh <pat...@departmentofdigitalwizardry.com> wrote:

Greetings,

I'm hoping someone can provide more information regarding an issue I'm having with the use of resumption_token and the slice operation.

I am sending requests to node01.public.learningregistry.net taking the form initially /slice?any_tags=pbs, and subsequently /slice?any_tags=pbs&resumption_token=<token_from_prev_response>.

As much as I can tell from the documentation and LR resources, this is the correct syntax. However, the subsequent requests with resumption_token return the same page of results.

Can anyone confirm this behavior or acknowledge it as a known issue?

Respectfully submitted,

Patrick Walsh

--
You received this message because you are subscribed to the Google Groups "Learning Registry Developers List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to learningreg-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Patrick Walsh]

Hi Walt,

I gave that a try, but the result wasn't any better:

Can you confirm that with any_tags removed and the request sent with only the resumption_token, the resultCount shoots up and documents no longer contain the tags queried for?

Is it true that resumption_token contains the HMAC signed params for the next query? Someone who has the secret might be able to see what's going wrong there, if that's the case.

[Jim Klo]

On Apr 10, 2013, at 1:58 PM, Patrick Walsh <pat...@departmentofdigitalwizardry.com>

 wrote:

Hi Walt,

I gave that a try, but the result wasn't any better:

Can you confirm that with any_tags removed and the request sent with only the resumption_token, the resultCount shoots up and documents no longer contain the tags queried for?

Walt and I had discussed this earlier and the API is indeed broken… Sorry :(  Apparently one of our unit tests wasn't catching the problem. A fix is in the works. 

The end result is that we are going to make slice work more like the other services that use the resumption_token; where first request is with parameters, and subsequent requests should only require the resumption token. Yeah for consistency!

WALT: can you make sure that backwards compatibility works (i.e. don't throw an error if both resumption_token and other params are provided, just ignore other params)?

Is it true that resumption_token contains the HMAC signed params for the next query? Someone who has the secret might be able to see what's going wrong there, if that's the case.

It is true that we implemented the resumption_token a JSON Web Signatures (JWS) <http://datatracker.ietf.org/doc/draft-ietf-jose-json-web-signature/>. So those of you wanting to peek at what's in the resumption should be able to do that if you like.

- JK

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

t. @nsomnac

[Patrick Walsh]

Thanks guys! Good to know. If it's not too much trouble, can you pass along the ticket IDs so that I may follow along?

[Patrick Walsh]

Hi All,

Does anyone know if progress has been made on this issue?

Regards,

Patrick

[Grata, Walt]

I believe I have a fix for this, finishing up testing this AM.  Once the fix is tested we'll do a point release to our public network.

[Patrick Walsh]

Hi Walt,

Thank you for the update! Will you post to the board to let us know when the production nodes have been updated?

Respectfully submitted,

Patrick Walsh

[finke...@gmail.com]

Hi Walt,
        NSDL is back on track to try to start posting our collections into the LR. We were using http://sandbox.learningregistry.org to test our code but it seems like slice with resumption token is also not working on there. Its behaving the same way as Patrick noted, i always get the same results.

Is sandbox still the place that we should be testing our code out, we need to be able to delete and update as well as slice

thanks
Dave Finke - NSDL

[Grata, Walt]

The fix is in, just waiting on the pull request to be reviewed by another team member before it's merged and deployed.   You should be able to delete and update on sandbox as it's running the latest master code containing that capability.  As soon as I do the deploy I'll send out an email letting everyone know.

[Jim Klo]

Apologies… been slammed… I'll review the pull request this morning…

- JK

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

e. jim...@sri.com

t. @nsomnac

p. 805.542.9330 x121

m.  805.286.1350

[finke...@gmail.com]

Walt,
     I recoded our stuff and tried delete and replace but I get errors when i try to publish to sandbox(http://sandbox.learningregistry.org), I followed the spec at http://docs.learningregistry.org/en/latest/start/20min.html . Added replaces and removed some of the attributes that it says may be omitted but i get an error for all my records that I try to delete

2013-05-06T10:54:02MDT : ERROR : lr_node.publish_documents(295) : REPOID:c0c579855b7646a7a9e12ddc2ce2670e DOCID:Unknown ID ERROR:
"u'resource_data' is a required property,\nu'none' is not one of [u'inline'],\nAdditional properties are not allowed (u'replaces' was          unexpected),\nu'resource_locator' is a required property,\nu'payload_locator' is a required property,\nu'none' is not one of [u'linked         '],\nAdditional properties are not allowed (u'replaces' was unexpected),\nu'resource_locator' is a required property,\nu'resource_data         ' is a required property,\nu'none' is not one of [u'inline'],\nu'resource_locator' is a required property,\nu'0.23.0' is not one of [u         '0.49.0'],\nu'payload_locator' is a required property,\nu'none' is not one of [u'linked'],\nu'resource_locator' is a required property         ,\nu'0.23.0' is not one of [u'0.49.0'],\nAdditional properties are not allowed (u'payload_schema_locator' was unexpected),\nu'0.23.0'          is not one of [u'0.49.0']"


Just wanted to make sure that this is the sandbox that contains replaces and deletes code on it

thanks
-Dave

[Jim Klo]

Dave,

There might be a minor error in the doc, I'll check. 

My quick look says resource_data_type is still required which is missing from the example doc in the 20 min guide. 

The definitive schema for delete is:

https://github.com/LearningRegistry/LearningRegistry/blob/master/LR/lr/schema/v_0_49/deleted_resource_data.json

Which inherits:

https://github.com/LearningRegistry/LearningRegistry/blob/master/LR/lr/schema/abstract_resource_data.json

The error you see is because the validator can't find any match against any variation. I have an update to this which can produce better info but not quite ready to be pulled in yet. 

Sent from my iPhone

[finke...@gmail.com]

Jim,
       I saw another post with this error, sorry i should have posted my reply there. From here on out i will do that so everyone can see.

i'm posting this for the delete which seems to be matching the format at url. https://github.com/LearningRegistry/LearningRegistry/blob/master/LR/lr/schema/v_0_49/deleted_resource_data.json And still getting error saying replaces is unexpected and all the other attributes that are missing are required properties

{u'doc_type': u'resource_data',
 u'digital_signature': { removed to keep clean},
 u'TOS': {u'submission_TOS': u'http://www.learningregistry.org/information-assurances/open-information-assurances-1-0'}, 'replaces': u'7c94f100459749689bb492e95e435a49',
 u'resource_data_type': u'metadata',
 u'payload_placement': 'none',
 u'doc_version': '0.49.0', <-- tried '0.23.0' too, since it keep saying that was an error too, it just goes back and forth though
u'active': True,
 u'doc_ID': u'7c94f100459749689bb492e95e435a49',
u'identity':
       {u'signer': u'NSDL', u'submitter': u'NSDL', u'submitter_type': u'agent', u'curator': u'NSDL'}}

error message
node.publish_documents(305) : REPOID:7c94f100459749689bb492e95e435a49 DOCID:Unknown ID ERROR:
"u'payload_schema' is a required property,\nu'resource_data' is a required property,\nu'none' is not one of [u'inline'],\nAdditional properties are not allowed (u'replaces' was unexpected),\nu'resource_locator' is a required property,\nu'0.49.0' is not one of [u'0.23.0'],\nu'payload_locator' is a required property,\nu'payload_schema' is a required property,\nu'none' is not one of [u'linked'],\nAdditional properties are not allowed (u'replaces' was unexpected),\nu'resource_locator' is a required property,\nu'0.49.0' is not one of [u'0.23.0'],\nu'payload_schema' is a required property,\nu'resource_data' is a required property,\nu'none' is not one of [u'inline'],\nu'resource_locator' is a required property,\nu'7c94f100459749689bb492e95e435a49' is not of type u'a', u'r', u'r', u'a', u'y',\nu'payload_locator' is a required property,\nu'payload_schema' is a required property,\nu'none' is not one of [u'linked'],\nu'resource_locator' is a required property,\nu'7c94f100459749689bb492e95e435a49' is not of type u'a', u'r', u'r', u'a', u'y',\nu'7c94f100459749689bb492e95e435a49' is not of type u'a', u'r', u'r', u'a', u'y'"
  
finally i thought I should try it on http://alpha.learningregistry.org. But i wasn't able to reguest auth because that url /auth is a 404

thanks for all you help
Dave - NSDL

[Jim Klo]

No worries…

Hmm so this test… https://gist.github.com/jimklo/5484651 seems to work in my testing (note the script changes the "doc_ID" and the "replaces" values with different ones).

The structure of the delete message that it uses is:

{
        "doc_ID": "urn:learningregistry.org:testing:uuid:C2879C2E-660C-47A5-8585-6B863D445324",
        "doc_type": "resource_data",
        "doc_version": "0.49.0",
        "replaces": ["urn:learningregistry.org:testing:uuid:C134DD8D-6580-4A68-B456-506B07D5A288"],
        "resource_data_type": "paradata",
        "active": true,
        "identity": {
            "submitter_type": "agent",
            "owner": "Jim Klo",
            "curator": "Jim Klo <j...@learningregistry.org>"
        },
        "TOS": {
            "submission_TOS": "http://creativecommons.org/publicdomain/zero/1.0/legalcode"
        },
        "payload_placement": "none"
    }

Can you make it work on sandbox? 

FWIW: We have some issues with redirection for the /auth on sandbox and alpha…

https://<node address>/apps/oauth-key-management/ is the ultimate redirection for /auth which should work on each node.

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

t. @nsomnac

On May 6, 2013, at 3:39 PM, <finke...@gmail.com>

 wrote:

[finke...@gmail.com]

Thanks Jim, Your script works for me. I'm going to use it to try to delete some resources that I have created and try to figure out the reason my format in the datapump isn't working.

-Dave

[Patrick Walsh]

Hi Jim,

The fix for /slice doesn't seem to have made it into production yet; can you tell me whether this is the case and we just need to wait for a release, or is the bug still present? According to #253, you merged, but this was a month ago, so I submitted #255 to be on the safe side; I apologize if it ends up being a duplicate.

Respectfully submitted,

Patrick

[Jim Klo]

You are correct. We ran into some issues in testing the fix, in the process discovering a few critical bugs, which has delayed the release. 

We are targeting Wednesday, May 29 to release. Apologies for the delay - we didn't want to release another broken build. 

Have a good Memorial Day. And to anyone who is serving or has served - thank you for your service!

- Jim

Sent from my iPhone

[Patrick Walsh]

Hi Jim,

Thank you for the update--and a good Memorial Day to you!

Regards,

Patrick

[finke...@gmail.com]

It looks like the resumption fix made its way on to the public nodes 01 and 02 but it is still not working on sandbox. Wasn't sure if sandbox is being used for something else but is there anyway the fix can be put onto sandbox for testing too. We are really close to start pushing our data to the public nodes but still need to do a little more testing on sandbox via slice and update/delete.

thanks
Dave - NSDL

[Grata, Walt]

Try sandbox now, I think there was a hick up restarting uwsgi so it didn't pick up the changes.

[finke...@gmail.com]

Works perfectly on sandbox now, thanks for doing that Walt.