Delete Operation Error Message

[Letha Kay Goger]

Hi Everyone, 

We are looking for help to address a problem we are having using the delete operation. Our developer sent the feedback, below.

Can you confirm the delete operation is currently supported on the registry? 

We want to entirely delete a set of paradata records. We will then be adding new records of aggregated and enriched data. 

Thanks,

Letha Goger

I've prepared a script to delete the existing paradata records and replace them with correct data. But it seems that LR nodes are not updated yet to support delete operations. When I try to delete a document according to documentation (http://docs.learningregistry.org/en/latest/start/20min.html#deleting-a-learning-registry-document) I receive the following error:

u'resource_data_type' is a required property,\
u'payload_schema' is a required property,\
u'resource_data' is a required property,\
u'none' is not one of [u'inline'],\
Additional properties are not allowed (u'replaces' was unexpected),\
u'resource_locator' is a required property,\
u'0.49.0' is not one of [u'0.23.0'],\
u'resource_data_type' is a required property,\
u'payload_locator' is a required property,\
u'payload_schema' is a required property,\
u'none' is not one of [u'linked'],\
Additional properties are not allowed (u'replaces' was unexpected),\
u'resource_locator' is a required property,\
u'0.49.0' is not one of [u'0.23.0'],\
u'resource_data_type' is a required property,\
u'payload_schema' is a required property,\
u'resource_data' is a required property,\
u'none' is not one of [u'inline'],\
u'resource_locator' is a required property,\
u'resource_data_type' is a required property,\
u'payload_locator' is a required property,\
u'payload_schema' is a required property,\
u'none' is not one of [u'linked'],\
u'resource_locator' is a required property,\
u'resource_data_type' is a required property\""

I've tried it on both sandbox and node01 nodes.

[Jim Klo]

Can you provide more details as to what exactly you are trying to publish as the replacement document??

Publishing a resource data description document to signal delete needs to conform to the JSON Schema (and it's inheritance chain) here: https://github.com/LearningRegistry/LearningRegistry/blob/master/LR/lr/schema/v_0_49/deleted_resource_data.json

If you can post a sample document that you are using to perform the delete that would be helpful to diagnose on what you might be doing incorrectly.

- Jim

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

e. jim...@sri.com

t. @nsomnac

p. 805.542.9330 x121

m.  805.286.1350

On Apr 26, 2013, at 8:51 AM, Letha Kay Goger <letha...@gmail.com>

 wrote:

--
You received this message because you are subscribed to the Google Groups "Learning Registry Developers List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to learningreg-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jim Klo]

I'll also mention the error output you are seeing here is the result of resource data description documents (rd3) being checked against the following schema:

https://github.com/LearningRegistry/LearningRegistry/blob/master/LR/lr/schema/any/resource_data.json

For more information about how to interpret JSON Schema see http://json-schema.org.  You might need to look at the previous version of their documentation; JSON Schema recently revved to Draft 4… The rd3 schemas are authored using Draft 3.  The key to interpreting the schema is knowing that each schema is evaluated independently of any extended schema, meaning the rd3 instance is evaluated against each schema listed in the extension hierarchy.

Also, I'm looking right now at our test cases for publish and the schema validator.  While our publish test cases don't currently cover explicitly the publishing of a replacement rd3 with "payload_placement" ==  "none", the tests for the schema validator certainly cover this - and provide some useful example data models: https://github.com/LearningRegistry/LearningRegistry/blob/master/LR/lr/tests/data/validation/any_resource_data.json#L6

- Jim

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

e. jim...@sri.com

t. @nsomnac

p. 805.542.9330 x121

m.  805.286.1350

On Apr 26, 2013, at 8:51 AM, Letha Kay Goger <letha...@gmail.com>

 wrote:

[Letha Kay Goger]

> Can you provide more details as to what exactly you are trying to publish as the replacement document??

Our developer is posting a document (below) that is almost exact copy of example from documentation (http://docs.learningregistry.org/en/latest/start/20min.html#deleting-a-learning-registry-document), but the server is not validating it and we receive the error message in my first post. 

{
  "doc_type": "resource_data",
  "replaces": [
    "c646902998674c0da74f205043ae8357"
  ],
  "TOS": {
    "submission_TOS": "http://www.learningregistry.org/tos/cc0/v0-5/"
  },
  "payload_placement": "none",
  "doc_version": "0.49.0",
  "active": true,
  "identity": {
    "submitter": "OER Commons",
    "submitter_type": "agent"
  }
}

[Jim Klo]

Can you provide the signed version of this?  Also the document that this supposedly replaces doesn't exist (however it shouldn't matter).

The digital signatures of both the original and the replacement must match (they must use the same key to sign).  The document sample you've provided has no digital signature (which is fine if you are providing it as input to LR signature tool), however the output of that tool is what I need to debug.

- JK

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

e. jim...@sri.com

t. @nsomnac

p. 805.542.9330 x121

m.  805.286.1350

On Apr 29, 2013, at 8:47 AM, Letha Kay Goger <letha...@gmail.com>

[Jim Klo]

Here's a sample script that performs the action you are trying on sandbox…

https://gist.github.com/jimklo/5484651

heres a link to the tombstone: http://sandbox.learningregistry.org/obtain?request_ID=urn:www.example.com:uuid1:abfb2085-b10d-11e2-a67d-109add5279f3-0&by_doc_ID=true

heres a link to the replacement document  (which is empty - effectively a delete): http://sandbox.learningregistry.org/obtain?request_ID=urn:www.example.com:uuid1:abfb2085-b10d-11e2-a67d-109add5279f3-1&by_doc_ID=true

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

e. jim...@sri.com

t. @nsomnac

p. 805.542.9330 x121

m.  805.286.1350

[Letha Kay Goger]

> Can you provide the signed version of this?

Here's the signed version:

{"documents": [{"doc_type": "resource_data", "replaces": ["61bfdaa9296a4e2792e944fdb7c1891e"], "digital_signature": {"key_location": ["http://www.oercommons.org/public-key.txt"], "key_owner": "OER Commons <in...@oercommons.org>", "signing_method": "LR-PGP.1.0", "signature": "-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

72987b0b21edbe9832835796c155ece9e4c3522e2ad6e7fb65ec3d7d329b0094
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)

iQEcBAEBAgAGBQJRf82/AAoJEKPVkXeiZE8+9l4H/iLkgwWyupn4FpFbcFHpi0qs
ZsrJZI2ctDjJm4g7xpC4z6O3RreWRJQ27rtgZcL2bKiVkCwcxcf6YK4MSMu8L2y7
oXY840zv4LA4mdFu6cMPzDFHhuhDs77BcyUsRAptEavnw7FKXZ5fFEK4v9vg3EEq
opSaJZ84eTeEYGpBz/xHimobz4uh2Mr8MbmV7PqbhB2oNFagr9PMvB6Fd/xS9qj5
IigXMnGh0VDdzkvPGeUzHWh5R37OBOVUTvX4c+HNUP9OcZbY4WXdd88cwaGz5WMr
fURpcJ27WZizNu/695WL0RN7E9B6r236aZUDc3equ5xJsYCeRd94LYQf7hkM+xo=
=q5jp
-----END PGP SIGNATURE-----
"}, "TOS": {"submission_TOS": "http://www.learningregistry.org/tos/cc0/v0-5/"}, "payload_placement": "none", "doc_version": "0.49.0", "active": true, "identity": {"submitter": "OER Commons", "submitter_type": "agent"}}]}

The document that I'm trying to delete is: http://sandbox.learningregistry.org/obtain?request_ID=61bfdaa9296a4e2792e944fdb7c1891e&by_doc_ID=true

[Letha Kay Goger]

The Delete operation continues to not work for us.

I'm bringing forward my April 30th post for reference.

Below is the message I received last night from our developer.  

I've attached the two files he mentions in his feedback to me.

Thank you,

Letha

Letha,

I've checked the discussion and I didn't find any useful information. I also double checked that the Delete operation still does not work for documents that we send.

I'm attaching two JSON files that I've used for testing.

1. doc.json — this is a test document that I submit to LR
2. delete-doc.json — this is replacement document used for delete operation

I submit these documents with the following command "cat doc.json | python -m LRSignature.cmd sign [other parameters omitted]". I get "OK" response for both documents.

Here's the published document (doc.json): http://sandbox.learningregistry.org/obtain?request_ID=804026fc367a4a44a76bd5b4c24809d8&by_doc_ID=true

It should be replaced with a "tombstone" record but it is not.

This is the replacement document used for Delete operation (delete-doc.json):http://sandbox.learningregistry.org/obtain?request_ID=5569a004388e4621ba8b414969455f8b&by_doc_ID=true It was created correctly.

[finke...@gmail.com]

Letha,
      I'm sure others will chime in here because something else might be wrong with the submission but NSDL had the same problem. Update and delete would not work for us, I tried to host the key on one of our servers like you guys do. I also tried different key server.

Finally I used the same key server that Jim uses. http://pool.sks-keyservers.net and that worked . If you run out of ideas I would try publishing your public key there and using that url to then re-publish the document and then try to delete it. I have a feeling that all the documents prior to using this key server were not seen as having a valid signature.

Dave - NSDL

[Steve Midgley]

Thanks Dave. Letha, the method you are using *should* work but it sounds like there may be a problem in processing public keys from some URLs. Jim is at ISTE and can't address this until he gets back next week, but if you want to public *the same* key to the sks ring, you might find that it does work for now.. http://sks-keyservers.net/i/#submit

To be clear, you should be able to use the same public key but just host it there, so all your signing should remain the same except for the URL to the key. I like where you put your public key on your own server personally, so we don't want to discourage that, but in the short term it might be worth trying Dave's work around?

Let us know what happens..

Best,

Steve

[andrey....@zojax.com]

Steve,

We've tried to use the delete operation using the public key hosted at sks-keyservers.net (http://pool.sks-keyservers.net:11371/pks/lookup?op=get&search=0xA3D59177A2644F3E) and it worked perfectly. Here's the link to the tombstone record: http://sandbox.learningregistry.org/obtain?request_ID=804026fc367a4a44a76bd5b4c24809d8&by_doc_ID=true

But then I noticed that public key URL (http://pool.sks-keyservers.net:11371/pks/lookup?op=get&search=0xA3D59177A2644F3E) returns HTML code containing the key instead of the plain text version of the key. So I've decided to try other public key locations and see how it works.

It turned out that I can use any URL as public key location in delete operation. I submitted a test document using our hosted public key (http://www.oercommons.org/public-key.txt).

Then I submitted a request to delete this document with http://example.iana.org/ as public key location. And it worked! Here's the tombstone record: http://sandbox.learningregistry.org/obtain?request_ID=9fd67a9a17f9404294ecacb280ceb773&by_doc_ID=true

My guess is that delete operation works for invalid public key locations, but does not work for valid public keys.

On Wednesday, June 26, 2013 4:27:43 AM UTC+6, Steve Midgley wrote:

Thanks Dave. Letha, the method you are using *should* work but it sounds like there may be a problem in processing public keys from some URLs. Jim is at ISTE and can't address this until he gets back next week, but if you want to public *the same* key to the sks ring, you might find that it does work for now.. http://sks-keyservers.net/i/#submit

To be clear, you should be able to use the same public key but just host it there, so all your signing should remain the same except for the URL to the key. I like where you put your public key on your own server personally, so we don't want to discourage that, but in the short term it might be worth trying Dave's work around?

Let us know what happens..

[Jim Klo]

Hmm…  I'll have to perform some additional tests.. The LRSignature test suite, which does several checks on the the fetching of the keys from a variety of sources, should be extracting from both Text and HTML content.  In fact it shouldn't care as it only looks for the PGP start and end headers, so for that matter, it should be able to find the string within any file content that can be parsed as a string.

It's possible that the PGP client is retrieving keys via another protocol from SKS servers. The algorithm I believe uses the local key server configuration first before looking at the key location in the envelope.  The tests currently use the following as test data locations:

https://sites.google.com/site/learningregistrytestdata/home/single-public-pgp-key

http://keyserver.pgp.com/vkd/DownloadKey.event?keyid=0xA8A790EA220403B7

http://pool.sks-keyservers.net:11371/pks/lookup?op=get&search=0xA8A790EA220403B7

http://sites.google.com/site/learningregistrytestdata/home/single-public-pgp-key/key0xA8A790EA220403B7.asc?attredirects=0&d=1

http://sites.google.com/site/learningregistrytestdata/home/multiple-public-pgp-keys

https://docs.google.com/uc?id=0ByJYdR0YE41yNDg2ZTkzYTItNmJjNC00YTgwLTkyYzctZTNiMzAwYzhjYTdh&export=download&hl=en

- JK

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

e. jim...@sri.com

t. @nsomnac

p. 805.542.9330 x121

m.  805.286.1350

On Jun 27, 2013, at 2:34 AM, <andrey....@zojax.com>

 wrote:

[Steve Midgley]

This is a little worrying - just to be clear it sounds like the delete operation is taking envelopes with non-matching public keys and executing the delete operation against them? That would be a big bug and one we should resolve quickly, if I'm understanding the problem.

Andrey - thanks for bringing this issue up. It sounds like the *opposite* of some other problems which were that public keys at some locations were not being recognized? I wonder if Dave will have any thoughts on this vs his issue?

Steve

[Jim Klo]

On Jun 27, 2013, at 10:30 AM, Steve Midgley <steve....@mixrun.com>

 wrote:

This is a little worrying - just to be clear it sounds like the delete operation is taking envelopes with non-matching public keys and executing the delete operation against them? That would be a big bug and one we should resolve quickly, if I'm understanding the problem.

Andrey is only mentioning that the URL to the public key is 'bad'.

What the validation process does is:

1. Extract the actually public key fingerprint used to sign the envelope from the signature (not looking at the public key yet)

2. We then search for the public key in a variety ways.  The easiest and most efficient method is to first use the key server services built into gnupg, which by default I believe is the sks pool (might be others, but most of the OpenPGP key servers are interconnected). Unless you did not submit your key to one of the major public key servers, the system generally finds the public key in this manner about 90% of the time. If the key is not found in the first attempt, we then use the public_key_location field in the envelope to fetch as a last resort. In most cases it's possible we don't get to this step ever. (FYI: Public keys can exist safely in a volatile environment, as long as you have the fingerprint, no 2 public keys are identical and cannot me modified without modifying the fingerprint).

3. Once we get the key for the fingerprint of the replacement document. We validate to ensure that signature is valid against the signed content.  If it's valid then, a look to see if it's a replacement then proceed.

4. If it's a replacement, we look for all the matching doc_ID's specified… validate that the fingerprints from both signatures match.  As long as the two match, then a tombstone is created by updating the existing doc_ID specified in the replaces field.  If the specified doc_ID doesn't exist… we just create a tombstone for the missing doc_ID.

Following this procedure, it is possible to specify a "junk" key location and the envelope still validate, provided the public key is found some other way.  The server does indeed cache public keys, so if validation ever fetched the public key previously - it won't ever go out and get it again from the network.

[finke...@gmail.com]

I just tried exactly what Andrey did and I got the same result, I was able to delete documents with a bogus URL in key_locations.. So I feel like the issue we are our experiencing at NSDL is exactly the same as what Andrey is experiencing. Which is great because I thought we were the only ones that couldn't get update/delete to work.

Jim's post explains it perfectly. We first published the key to http://pool.sks-keyservers.net and submitted documents using this key, which is a valid public key I guess you would call it and now all publishes that we do with that key will be using that one. Regardless of what is in key locations, which makes sense.

So I feel like the main question is why can't Andrey and I host our public key in places besides http://pool.sks-keyservers.net and get it to work, even though Jim's tests say that it should. NSDL is fine with hosting our key at http://pool.sks-keyservers.net but I feel like others might run into this issue down the road which might create more headaches.
 

Dave - NSDL

[Steve Midgley]

Oh great. Phew. So LR validation is just smart, and looks for your key in well known locations so that if it were to go down off your specified location but can still be found in other well advertised places, the envelope will still work. Yay. Glad that's a feature not a bug!!

Definitely seems like we need to make it work with other key servers, per previous bug, and I'm sure Jim will take that on at some point soon, but not nearly as heart-stopping as what I thought I understood earlier..

Steve

 

Dave - NSDL

[Jim Klo]

I believe I found the issue. Indeed it was a problem with fetching self hosted keys (admittedly, PEBCAK on my part, inadvertently using a Java convention instead of the Python convention )

Those interested in the problem can see the delta here: https://github.com/jimklo/LearningRegistry/compare/http-public-key-fix

Pull request in the morning provided it passes tests tonight...

- JK

Jim Klo

Senior Software Engineer

Center for Software Engineering

SRI International

t. @nsomnac