LRSignature bug...

36 views
Skip to first unread message

Jim Klo

unread,
May 16, 2013, 2:17:03 PM5/16/13
to learnin...@googlegroups.com
Greetings,

Dave Finke reported an issue with my LRSignature code…  Here's the text Dave noted the issue:

2) Also In the LR Paradata Spec 1.0, it shows an activity having a measure property which then has a float sub property(ie count, value) stays as a float in the JSON. We are using LRSignature written by Jim Klo, which during signing, normalizes the document and converts all float numbers and booleans into strings. So our float numbers during publishing are converted to strings.
So should the spec specify the floats as strings instead or are we missing something? On a side note I would think that if floats are converted to strings, perhaps for consistency sake, integers in the activity should also be strings, ie scaleMinscaleMax, count and sampleSize. Then a consumer of the documents can just convert all string values into floats.

So I've reviewed the 3 other Signature implementations that I know of… 

https://github.com/adlnet/LR.Net/  .NET implemention doesn't covert numbers, hence they will dropped in the BEncode step.
https://github.com/navnorth/LRJavaLib/blob/master/src/com/navnorth/learningregistry/LREnvelope.java#L100 Java implementation doesn't deal with numbers - hence they would get dropped in the BEncode step.

https://github.com/jeffreyhill/LRPHP/blob/master/lib/signature.php  PHP implementation - seems to mirror my implementation, converts numbers to strings.


It seems there's a 50/50 split on how this is currently implemented.  Keeping in mind that LRSignature is currently used in node signing, the question is how should we address? 

1. Should we just fix the implementation which would invalidate any envelopes that contain floating point values (which would be encoded in JSON)?
2. Change the Spec?

My gut says that I think the number of envelopes that have resource data that is a JSON object AND contains a non-stringified floating point number is low, and that option 1 is the correct approach, meaning Jeff Hill probably needs to get his solution updated as well…

The problem with changing the spec around numbers is the precision problem with Floating point…  LR Spec is a spec… which isn't tied necessarily to an implementation.  So someone could go on and build a LR Node in Java, Ruby, NodeJS, Scala, etc.. and the problem is that floating point representation may be different across languages - so theres a potential there for floating point to get serialized into a canonical format differently because of the underlying language representation.  This link: http://docs.couchdb.org/en/latest/json-structure.html#number-handling  does a pretty good job at explaining the problem.

If anything, I'd like to hear a proposal for other signature algorithms, rather than modify the algorithm…

Thoughts anyone?


Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI International
t. @nsomnac

Steve Midgley

unread,
May 20, 2013, 1:53:40 PM5/20/13
to learnin...@googlegroups.com
Let me be sure I understand what you're getting at.. If you put a native floating point number in any Json field in an LR envelope, the PGP signing might break b/c of different implementations of float representation by different languages? But if you convert the float to a string, then you solve that problem, but it's not obvious that your number is a number, to the consumers of the envelope?

Solutions include:

1) Submitters convert any float to a string before signing it
1a) Prohibit native floats in envelopes
2) Change the spec

I'm not sure I understand what the change in the spec would mean? Allow floating point? But then we have the problem you alluded to which is floats are implementation dependent and hard for a developer to be sure they're getting the same result on different platforms and languages?

I would vote that we should just use stringified floats everywhere in the envelope as a simple and fairly workable concept, since duck typing in json is not so painful with most language libs.. 

***

Regarding alternative signing, does the Mozilla Persona system (not that it's running) offer any way to tie identity and signatures to that instead of PGP keys?

Useful input?
Steve

Jim Klo

unread,
May 20, 2013, 2:21:59 PM5/20/13
to <learningreg-dev@googlegroups.com>, learnin...@googlegroups.com


Sent from my iPad

On May 20, 2013, at 10:53 AM, "Steve Midgley" <steve....@mixrun.com> wrote:

Let me be sure I understand what you're getting at.. If you put a native floating point number in any Json field in an LR envelope, the PGP signing might break b/c of different implementations of float representation by different languages?

Yes.. Since the normalization for signing and validation could be on different platforms that represent the same floating point value differently.  


But if you convert the float to a string, then you solve that problem, but it's not obvious that your number is a number, to the consumers of the envelope?

Yes, as long as the original content has the float encoded as a string, the problem is avoided - that's where schemas for content structure come in. If you use a Schema that specifies numbers to be encoded as strings we avoid the issue. 


Solutions include:

1) Submitters convert any float to a string before signing it
1a) Prohibit native floats in envelopes

I'm not sure this works. The original value must always be encoded as a string.

ie: say I sign on Python, verify in Ruby. Since parsing the float to string would be different:

Say float is 1.012345678901234567890

Python parses that as 1.01234567890123456

Ruby parses 1.01234567890123

Conversion to string would yield different results - causing an inconsistent hash across both platforms. 

2) Change the spec


Maybe we don't change the spec, but create an additional signing method that is acceptable - and works with floating point values better, but not sure there is a solution. 


I'm not sure I understand what the change in the spec would mean? Allow floating point? But then we have the problem you alluded to which is floats are implementation dependent and hard for a developer to be sure they're getting the same result on different platforms and languages?


The "change the spec" would be to specify an alternate solution that addresses the general signing problem, as well as this floating point issue. 

I would vote that we should just use stringified floats everywhere in the envelope as a simple and fairly workable concept, since duck typing in json is not so painful with most language libs.. 


This would be my initial solution, updating the LR Signature toolkit to drop floats as it should.  


--
You received this message because you are subscribed to the Google Groups "Learning Registry Developers List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to learningreg-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Steve Midgley

unread,
May 20, 2013, 4:04:53 PM5/20/13
to learnin...@googlegroups.com
Solutions include:

1) Submitters convert any float to a string before signing it
1a) Prohibit native floats in envelopes

I'm not sure this works. The original value must always be encoded as a string.

ie: say I sign on Python, verify in Ruby. Since parsing the float to string would be different:

Say float is 1.012345678901234567890

Python parses that as 1.01234567890123456

Ruby parses 1.01234567890123

Conversion to string would yield different results - causing an inconsistent hash across both platforms. 

SM: I don't get this problem. Shouldn't the signature validation be done on the string version of the float, not the "to_float()" version of the float? So ruby and python just signature validate against a strong that have some numbers and a period in it? Once the signature validation is done, who cares what the various implementations do to the float - it'll all be close enough?

I must be missing something.

Steve

Jim Klo

unread,
May 20, 2013, 5:40:00 PM5/20/13
to <learningreg-dev@googlegroups.com>, learnin...@googlegroups.com


Sent from my iPad

On May 20, 2013, at 1:04 PM, "Steve Midgley" <steve....@mixrun.com> wrote:

Solutions include:

1) Submitters convert any float to a string before signing it
1a) Prohibit native floats in envelopes

I'm not sure this works. The original value must always be encoded as a string.

ie: say I sign on Python, verify in Ruby. Since parsing the float to string would be different:

Say float is 1.012345678901234567890

Python parses that as 1.01234567890123456

Ruby parses 1.01234567890123

Conversion to string would yield different results - causing an inconsistent hash across both platforms. 

SM: I don't get this problem. Shouldn't the signature validation be done on the string version of the float, not the "to_float()" version of the float? So ruby and python just signature validate against a strong that have some numbers and a period in it? Once the signature validation is done, who cares what the various implementations do to the float - it'll all be close enough?


The problem is how the initial data structure is parsed into a canonical format for hashing...

JSON.parse( '{"rating":1.01234567890123456789, "title": "some rating", "author": "Martin Yee"}' )

If the validating platform converts the float into a string differently than the original signed version, there's no chance at generating a hash that's close. If there were parsers out there that didn't convert to float first, it wouldn't be an issue. 

Steve Midgley

unread,
May 20, 2013, 6:57:30 PM5/20/13
to learnin...@googlegroups.com
What about just representing floats at IEEE 754 in string hex or binary or something? Maybe that way you don't convert before verifying signature?

Steve



Jim Klo

unread,
May 30, 2013, 2:28:12 PM5/30/13
to <learningreg-dev@googlegroups.com>
I am trying to fix the Hashing Algorithm in LRSignature.

For those not aware, there is an issue in that the canonicalization step is converting floating point values to strings before the bencode step.

As a refresher the spec says:

  • Create a canonical document to sign:
    • Make a working copy of the JSON form of the resource data description document.
    • Eliminate all objects that are generated by a node in the Learning Registry network, leaving only those objects that are supplied by the user . Objects to be eliminated include: doc_id, publishing_node, update_timestamp, ▲node_timestamp, create_timestamp
    • Eliminate all implementation-specific objects, i.e., in Couch these are the _* objects, e.g., _id, _rev (NB: These will exist only when verifying a signature.)
    • Eliminate all objects where the object is a number. (NB: There are currently no numeric objects.)
    • Eliminate the digital_signature object.
    • For a boolean object with value true, change the value to a string with the value “true”.
    • For a boolean object with value false, change the value to a string with the value “false”.
    • For an object with value null, change the value to a string with the value “null”.
    • Encode the resulting JSON object using Bencode. The Bencoded output SHALL conform to the Bittorent Protocol Specification.
    • Hash the Bencoded output using SHA-256.
My question is how should I interpret "Eliminate"?  Consider:

{
"name": "foo",
"int_val": 1
}

Is the correct interpretation for canonicalization:

{
"name": "foo"
}

or is it
{
"name": "foo", 
"int_val": undefined  // not valid JSON so could treat as null
}

which then possibly could convert to:
{
"name": "foo",
"int_val": "null"
}


Here's my rationale:

{
"name": "foo",
        "my_val": 1.0
}

and 

{
"name": "foo",
"your_val":1.1
}


depending upon the interpretation would be canonicalized to the same value using the the first interpretation (dropping the key and value) whereas the second, we literally "drop the value" by making it undefined, however that's technically not permitted in JSON, so instead we could say it's actually null, of which our rules for null state to convert to "null"

Thoughts, opinions, flames?

Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI International

Steve Midgley

unread,
May 30, 2013, 2:38:42 PM5/30/13
to learnin...@googlegroups.com

Intuitively I'd think dropping the value only increases entropy as little as possible..

I'm shooting from the hip. I can't see specific implications between them (other than dropping k/v pairs or just dropping values)..

Steve

Marie Bienkowski

unread,
May 30, 2013, 3:49:10 PM5/30/13
to <learningreg-dev@googlegroups.com>
"Eliminating the object" does appear to mean dropping the key and value pair. Otherwise why not say 'change the value to a string with the value "null" '?

That said, I can see where more information makes a better hash, so your keeping the key and setting the value to null works for me. But if you do that, I'd recommend changing the spec.
• For an object that is a number, change the value to a string with the value “null”.

Marie



On May 30, 2013, at 11:38 AM, Steve Midgley <steve....@mixrun.com>

Steve Midgley

unread,
May 30, 2013, 4:00:43 PM5/30/13
to learnin...@googlegroups.com

+1

Jim Klo

unread,
May 30, 2013, 4:20:02 PM5/30/13
to <learningreg-dev@googlegroups.com>
So I think there is a "small" consensus that it means to drop key and values - I think I'll treat it that way; as to take it literally it would mean to change to object to "undefined" in JavaScript land, and since undefined is not permissible in JSON; in the process creating a cascading coercion nightmare. Dropping the key is probably the next best thing even though we loose entropy in the process. In reality  I think we are better served with just having proper schemas for JSON objects that specify that numbers should be encoded as Strings before insertion into LR (that might be a note we add to the spec.) and use a pattern to for restricting the value of said string.

Rather than change the spec for signing - which has unevaluated impacts.  I'd rather change the spec in a bigger way that allows us to not be lossy than to change.  Changing the algorithm has a bigger impact on things like sign by proxy, signature validation, replacement documents, etc; which I'm not really ready to tackle at the moment unless there is a strong desire. 

FWIW I may alter LRSignature to optionally raise an exception when signing something that produces lossy results.


Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI International
t. @nsomnac

On May 30, 2013, at 12:49 PM, Marie Bienkowski <marie.bi...@sri.com>
 wrote:

Message has been deleted

finke...@gmail.com

unread,
May 30, 2013, 4:25:36 PM5/30/13
to learnin...@googlegroups.com
Hope you guys don't mind me adding my two cents about this, I might be way of base though.

1) When i brought up this issue I was actually referring to the issue in LRSignature that during normalizing its actually making changes to the envelope too. Because if my JSON has a float, the float is not only converted to a string
for figuring out the hash but ts also being converted to a string in the envelope that is being published. I think it has do to
sigObj = envelope.copy(). Unless this was purposefully done I would think this should be a deep copy so the envelope that has a float stays a float and not converted to a string in the envelope

2) From what I take away from this thread is that the envelope that we are publishing may contain booleans, ints and floats.
Its just that when we are signing/verifying they either need to be removed/changed so the hashes stay consistent across platforms. So I agree with everyone else having the closest hash would be the best, so leaving the key and making the value "null" sounds good. The only thing I wonder though is if the hash is allowed to still have integers, why can't we at least convert the float into an integer. 3.9->3. And hash that, then at least in the hash the most that anything could be off by since it was signed would be .9999.......

Dave - NSDL

Steve Midgley

unread,
May 30, 2013, 5:54:43 PM5/30/13
to learnin...@googlegroups.com
Converting float to int for purposes of signing seems scary in principle but perhaps not bad for our purposes. I'm open.

Jim, I thought the consensus prior to Dave (Marie and me) was to convert floats to the string "null" -- but Dave's proposal of converts floats to ints has merit too I think.. (How can it be worse than converting to null?)

I guess it could be worse if someone nefarious intercepts an envelope and changes 3.4 to 3.1 and that has some dramatic impact on the consumer.

Steve


Jim Klo

unread,
May 30, 2013, 6:26:32 PM5/30/13
to <learningreg-dev@googlegroups.com>


On May 30, 2013, at 2:54 PM, Steve Midgley <steve....@mixrun.com>
 wrote:

Converting float to int for purposes of signing seems scary in principle but perhaps not bad for our purposes. I'm open.

Jim, I thought the consensus prior to Dave (Marie and me) was to convert floats to the string "null" -- but Dave's proposal of converts floats to ints has merit too I think.. (How can it be worse than converting to null?)


The question I was looking to answer was:

1. How should I interpret this bit of the spec when the object is the value of a key?:
  • Eliminate all objects where the object is a number. (NB: There are currently no numeric objects.)

The small consensus was that the interpretation should drop the key/value (even though retaining an object's key that originally holds a number value would be helpful in adding entropy to the hash). Which was my original gut instinct.


The second, but implied question was should we change this?

There's a consensus seems that if we change it right now, acceptable solutions might be:
1. convert the numeric value to "null"
2. convert the numeric value to a signed integer (Q: could there be rounding issues here? - truncation vs rounding is probably preferred).

My concern on making this minor change is that:
1. it doesn't 'solve' the problem; it just makes it 'less likely'.
2. the problem is 'avoidable' with proper knowledge.
3. there's a lot effected downstream - I'd rather not make minor changes unless they really solve the problem
- Anywhere that validates or generate the signature would need inspection and updating.
• sign by proxy
• replacement documents and V1 signature vs V2 signature.

Jim Klo

unread,
May 30, 2013, 8:01:11 PM5/30/13
to <learningreg-dev@googlegroups.com>

On May 30, 2013, at 1:25 PM, <finke...@gmail.com>
 wrote:

Hope you guys don't mind me adding my two cents about this, I might be way of base though.

1) When i brought up this issue I was actually referring to the issue in LRSignature that during normalizing its actually making changes to the envelope too. Because if my JSON has a float, the float is not only converted to a string
for figuring out the hash but ts also being converted to a string in the envelope that is being published. I think it has do to
sigObj = envelope.copy(). Unless this was purposefully done I would think this should be a deep copy so the envelope that has a float stays a float and not converted to a string in the envelope


it shouldn't be changing the actual envelope at all… if it is that's a really big bug (and I think I totally misunderstood your initial claim previously)…  but not so bad in that at least the final output contains the modified version with the signature - it's just a pain because it didn't publish exactly what you expected.

 I see that it is doing a dict.copy() first, which I now see in the Python docs is a shallow copy - I will fix globally as I think this is pervasive to other areas.



2) From what I take away from this thread is that the envelope that we are publishing may contain booleans, ints and floats.
Its just that when we are signing/verifying they either need to be removed/changed so the hashes stay consistent across platforms. So I agree with everyone else having the closest hash would be the best, so leaving the key and making the value "null" sounds good. The only thing I wonder though is if the hash is allowed to still have integers, why can't we at least convert the float into an integer. 3.9->3. And hash that, then at least in the hash the most that anything could be off by since it was signed would be .9999…….

FWIW: The spec says to drop ALL numbers.  JSON (and ECMAScript for that matter) has no notion of Float vs Integer in reality - both are of type Number http://ecma262-5.com/ELS5_HTML.htm#Section_8.5.  Off by one can change the hash dramatically so it's a pretty binary problem - it either needs to match or not, close doesn't count in this case. Hence there are 3 bugs in LRSignature, 1 retention of integers (which may have been dropped by the bencode process, not sure - need to verify), 2 conversion of floats to strings, 3 not performing a deep-copy for hash computation.

I agree conversion to 'null' is probably the best way to preserve entropy for the hash, however I'm not very keen on changing the spec just to solve a portion of it's lossy-ness - which can be avoided through encoding numbers as strings to retain the correct precision.  I highly recommend those using JSON as resource_data, to consider using JSON Schema v4 for defining a schemas - it's relatively straight forward to use.

To admit this is the after-effect of not understanding the Resource Date Document resource_data property fully.  It was my understanding at the time that it was only a String and not a union of possible types String or Object.  My original understanding was that the string could contain XML, JSON, any thing you wanted to encode.  It was not until after the original release that I understood that resource_data could actually be something other than a String. It's been on several plans to redesign, but never scheduled as there was always a workaround.

Jim Klo

unread,
May 30, 2013, 8:37:22 PM5/30/13
to <learningreg-dev@googlegroups.com>
I've just updated LRSignature to version 0.1.14.

Changes include:
Removed the inclusion of numbers (float, integer) when computing canonical form.
Fixed "deep copy" issue. When "resource_data" as JSON object contained a key/value containing a null or boolean, the origin JSON retained the a manipulation indented for hashing.

Updated to be included in continuous integration testing at learnreg2.sri.com


Available via pip or pypi: https://pypi.python.org/pypi/LRSignature

Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI International
t. @nsomnac

On May 30, 2013, at 5:01 PM, Jim Klo <jim...@sri.com>
 wrote:


On May 30, 2013, at 1:25 PM, <finke...@gmail.com>
 wrote:

Hope you guys don't mind me adding my two cents about this, I might be way of base though.

1) When i brought up this issue I was actually referring to the issue in LRSignature that during normalizing its actually making changes to the envelope too. Because if my JSON has a float, the float is not only converted to a string
for figuring out the hash but ts also being converted to a string in the envelope that is being published. I think it has do to
sigObj = envelope.copy(). Unless this was purposefully done I would think this should be a deep copy so the envelope that has a float stays a float and not converted to a string in the envelope


it shouldn't be changing the actual envelope at all… if it is that's a really big bug (and I think I totally misunderstood your initial claim previously)…  but not so bad in that at least the final output contains the modified version with the signature - it's just a pain because it didn't publish exactly what you expected.

 I see that it is doing a dict.copy() first, which I now see in the Python docs is a shallow copy - I will fix globally as I think this is pervasive to other areas.

<Notification Center-1.png>

Marie Bienkowski

unread,
May 31, 2013, 10:31:52 AM5/31/13
to <learningreg-dev@googlegroups.com>
Please clarify
2. the problem is 'avoidable' with proper knowledge.

I agree with not making a minor change that does not solve the problem.

M



Steve Midgley

unread,
May 31, 2013, 11:01:30 AM5/31/13
to learnin...@googlegroups.com
I may be more confused than is helpful. That said, I think it makes more sense to keep the integer equivalent (agree that it's truncate not round) of a number rather than drop the entire number value (and replacing with string "null")? 

I can see all kinds of problems with having multiple json renderings permitted, so better to keep it simple. That said, beyond reducing all numbers to only significant digits, could we suggest in/near the spec that prior to signing, submitters can convert floats to IEEE 754 string versions of their floats and provide a json schema somewhere (else?) that indicates those fields are string encoded numbers if maintaining full security over every bit of their envelope is vital? Tools like this would make it pretty easy to do the conversion if it really matters to people? http://www.h-schmidt.net/FloatConverter/

This would only be necessary if the submitter can't handle the security exposure of having the least significant digits of their floats being unsigned (which is all I think we're chasing in this bug anyway?)..

Am I missing the main story?

Steve





Jim Klo

unread,
May 31, 2013, 12:52:25 PM5/31/13
to <learningreg-dev@googlegroups.com>

On May 31, 2013, at 8:01 AM, Steve Midgley <steve....@mixrun.com>
 wrote:

I may be more confused than is helpful. That said, I think it makes more sense to keep the integer equivalent (agree that it's truncate not round) of a number rather than drop the entire number value (and replacing with string "null")? 


True, truncation to integer may provide better guarantees over dropping, however - It doesn't close the gap completely. So, in my opinion, since it's just a minor improvement on a partial guarantee (as it's still a partial guarantee even with the improvement), it doesn't seem compelling enough to rev the algorithm, when parsing strings into numbers is pretty commonplace and closes the gap (and can be done with no change to the spec).

I can see all kinds of problems with having multiple json renderings permitted, so better to keep it simple.

Agreed, which is why I'm adverse to adding one more step into the already complex signing algorithm; I'd rather scrap it wholesale for something simpler that can provide a full guarantee.  To be clear this change to the spec would be to:

Remove the wording:

  • Eliminate all objects where the object is a number. (NB: There are currently no numeric objects.)

Replacing it with:

  • Truncate all objects where the object is a number into an integer. 
    • (NB: There are currently no numeric objects in the resource data description document)
    • (NB: It is recommended to encode numbers IEEE 754 values as strings)
However not sure I'm ready to make that change quite yet.

That said, beyond reducing all numbers to only significant digits, could we suggest in/near the spec that prior to signing, submitters can convert floats to IEEE 754 string versions of their floats and provide a json schema somewhere (else?) that indicates those fields are string encoded numbers if maintaining full security over every bit of their envelope is vital? Tools like this would make it pretty easy to do the conversion if it really matters to people? http://www.h-schmidt.net/FloatConverter/


I think it would be worthy to add additional notes on the explanation of the existing algorithm:

• numerical objects are dropped from the JSON, when the object is the value of a key/value pair, both the key and value should be dropped.
• recommendation is to encode the complete JSON as a string, or encode all numerical values as strings to retain published precision.


This would only be necessary if the submitter can't handle the security exposure of having the least significant digits of their floats being unsigned (which is all I think we're chasing in this bug anyway?)..

Am I missing the main story?


Maybe just a tad… My main argument is that while I acknowledge that preservation of an object's key/value pair to a truncated integer permits the digital signing algorithm to provide a better guarantee than dropping the entire key/value pair, that better is still not a full guarantee. Plus when a full guarantee is available when encoding numbers as strings, I'm really hard pressed to modify the algorithm as well as handle all the backwards compatibility issues that would arise as it doesn't actually solve the problem of making numbers in JSON metadata fully guaranteed, and you still need to leverage the same number encoded into a string technique to achieve full guarantees.

If there is a good solution for dealing with FP values across the board that gets us to full guarantees - that would be a worthwhile reason for changing the algorithm.

finke...@gmail.com

unread,
May 31, 2013, 1:44:49 PM5/31/13
to learnin...@googlegroups.com
Thanks for the quick updates to LRSignature Jim,

I updated my LRSignature but it breaks on the bencode when trying to publish LRData. I had to remove the raise TypeError("Numbers not permitted") and uncomment out return str(obj) for the numbers. I think its because in the if DictType it catches TypeErrors and gives it a pass. Thus leaving the floats as is, and bencode breaks because of that.


That being said, is the idea of raising a TypeError supposed to make it all the way out of LRSignature and stop our code so we don't have numbers in our envolopes? or did you mean to catch it and remove it from the dict?

Also I tried to publish some stuff to sandbox using the new LRSignature but sandbox its not having it. I guess the new LRSignature hasn't made its way to sandbox yet.

thanks
Dave - NSDL







Jim Klo

unread,
May 31, 2013, 1:48:03 PM5/31/13
to <learningreg-dev@googlegroups.com>
Hmm… It should be catching that TypeError as the "fix" to adhere to the spec required dropping all Numbers… :(

Can you send me some example data where it's failing so I can amend the unit tests and fix?

- JK

Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI International

On May 31, 2013, at 10:44 AM, <finke...@gmail.com>
 wrote:

Steve Midgley

unread,
Jun 1, 2013, 11:27:53 AM6/1/13
to learnin...@googlegroups.com
Well, truncation is bad and it gives a false sense of security that you have signed something that you haven't signed. So it sounds like, keep things as they are, and just put a strongly worded advisory that tells people to put their integers *and* floats into strings when submitting to LR, and ideally provide a schema referenced from the envelope/payload that indicates which strings are numbers?

Steve

finke...@gmail.com

unread,
Jun 3, 2013, 11:51:10 AM6/3/13
to learnin...@googlegroups.com
Jim,
      Sorry i took a half day on Friday and didn't have time to send you an example of whats not working on LRSignature so you could add it to your test cases.

The example below has a bunch of numbers in it but from the way the code looks, if numbers are ever found in inner dicts. I would think the _bnormal would bomb. Because the obj that is sent in still has those keys, and the assigning is just overwriting it. I would think in the elif isinstance(obj, types.DictType): That it would be putting together a new dict instead of updating the one sent in.

After reading Steve's reply. I'm taking it that the best practice, is for us to always convert our numbers strings. Therefore I think eventually the LRParadata 1.0 spec should put quotes around their numbers. So no one else goes down the path of using integers and floats like we did.


Here is my JSON that is messing up Bencode.

{u'doc_type': u'resource_data', u'resource_data': {u'collection': {u'totalitems': 6, u'items': [{u'activity': {u'content': u"This record contains paradata for the resource 'Slope and Y-intercept'", u'verb': {u'action': u'downloaded', u'date': u'2010-10-20T11:00:32/2012-02-09T16:20:41', u'measure': {u'measureType': u'count', u'value': 1.0}}, u'object': u'http://www.shodor.org/interactivate/discussions/SlopeAndYintercept/'}}, {u'activity': {u'content': u"This record contains paradata for the resource 'Slope and Y-intercept'", u'verb': {u'action': u'favorited', u'date': u'2010-10-20T11:00:32/2012-02-09T16:20:41', u'measure': {u'measureType': u'count', u'value': 2.0}}, u'object': u'http://www.shodor.org/interactivate/discussions/SlopeAndYintercept/'}}, {u'activity': {u'content': u"This record contains paradata for the resource 'Slope and Y-intercept'", u'verb': {u'action': u'commented', u'date': u'2010-10-20T11:00:32/2012-02-09T16:20:41', u'measure': {u'measureType': u'count', u'value': 0.0}}, u'object': u'http://www.shodor.org/interactivate/discussions/SlopeAndYintercept/'}}, {u'activity': {u'content': u"This record contains paradata for the resource 'Slope and Y-intercept'", u'verb': {u'action': u'viewed', u'date': u'2010-10-20T11:00:32/2012-02-09T16:20:41', u'measure': {u'measureType': u'count', u'value': 20.0}}, u'object': u'http://www.shodor.org/interactivate/discussions/SlopeAndYintercept/'}}, {u'activity': {u'content': u"This record contains paradata for the resource 'Slope and Y-intercept'", u'verb': {u'action': u'subscribed', u'date': u'2010-10-20T11:00:32/2012-02-09T16:20:41', u'measure': {u'measureType': u'count', u'value': 1.0}}, u'object': u'http://www.shodor.org/interactivate/discussions/SlopeAndYintercept/'}}, {u'activity': {u'content': u"This record contains paradata for the resource 'Slope and Y-intercept'", u'verb': {u'action': u'recommended', u'date': u'2010-10-20T11:00:32/2012-02-09T16:20:41', u'measure': {u'measureType': u'count', u'value': 0.0}}, u'object': u'http://www.shodor.org/interactivate/discussions/SlopeAndYintercept/'}}]}}, u'keys': [u'paradata', u'NSDL4_COLLECTION_ncs-NSDL-COLLECTION-000-003-112-057', u'oai:nsdl.org:2200/20111021180114887T'], u'TOS': {u'submission_TOS': u'http://nsdl.org/help/?pager=termsofuse'}, u'resource_data_type': u'paradata', u'payload_placement': u'inline', u'payload_schema': [u'LR Paradata 1.0'], u'doc_version': u'0.49.0', u'active': True, u'resource_locator': u'http://www.shodor.org/interactivate/discussions/SlopeAndYintercept/', u'identity': {u'submitter': u'National Science Digital Library (NSDL)', u'submitter_type': u'agent', u'curator': u'CTE Online'}}


Dave - NSDL
Reply all
Reply to author
Forward
0 new messages