incorrect salmon signatures

[Mike Macgirvin]

During some of my compatibility testing w/mistpark, I've discovered
that several projects are using incorrect implementations of salmon
magic signatures. In fact, I have not yet found one that is
compliant.

This creates a huge mess because one must send two slaps to each of
these providers - the first using the correct implementation, and then
if that fails - repeat using the incorrect format. These have to be
verified in the same way, falling back to the incorrect format if the
correct one fails.

Unfortunately, this means that each of you who wishes to provide
federated services must also implement this duplication of effort
until such time as everybody has upgraded and the broken
implementations begin to fade from existence (this could take a few
years - which is why everybody needs to start fixing it today(!)).

The crux of the problem is in section 8 of draft-panzer-magicsig-00
(section 9.1 if you're working off the trunk).

The implementations I've encountered are only signing the
base64url_encoded data itself. The correct method is to append the
string ".YXBwbGljYXRpb24vYXRvbSt4bWw=.YmFzZTY0dXJs.UlNBLVNIQTI1Ng=="
to the armoured data and use that both for signing and verification.
(This is a simplification, you should calculate this string yourself
to allow for future revisions).

[Tuomas Koski]

Hi Mike,

well noticed.

On 27 October 2010 00:46, Mike Macgirvin <mi...@macgirvin.com> wrote:
> During some of my compatibility testing w/mistpark, I've discovered
> that several projects are using incorrect implementations of salmon
> magic signatures. In fact, I have not yet found one that is
> compliant.

If I'm not mistaken, this one should be working by the spec:

http://www.madebymonsieur.com/ostatus_discovery/magic_env/readme

I also send you, Mike, a salmon message. Your server replied
successfully at least. Did you receive it?

On 27 October 2010 00:46, Mike Macgirvin <mi...@macgirvin.com> wrote:
> This creates a huge mess because one must send two slaps to each of
> these providers - the first using the correct implementation, and then
> if that fails - repeat using the incorrect format. These have to be
> verified in the same way, falling back to the incorrect format if the
> correct one fails.

True. It's quite a pain in the ...

On 27 October 2010 00:46, Mike Macgirvin <mi...@macgirvin.com> wrote:
> Unfortunately, this means that each of you who wishes to provide
> federated services must also implement this duplication of effort
> until such time as everybody has upgraded and the broken
> implementations begin to fade from existence (this could take a few
> years -  which is why everybody needs to start fixing it today(!)).

I think it's just matter of when status.net's implementation is
"fixed". It seems to have the same signing fault as well. Deeply sorry
if I'm wrong.

So in other words, I remove my implementation's "fallback to wrong
signing" as soon as identi.ca is "fixed".

Cheers,
--
Tuomas

[Mike Macgirvin]

> I also send you, Mike, a salmon message. Your server replied
> successfully at least. Did you receive it?

If you received a 200 reply the signature verified. Mistpark does not
accept unsolicited messages from the wild (with the exception of
"follow" activities), but it will indicate that the message was received
intact - or throw a 4xx error if the verification failed.

> I think it's just matter of when status.net's implementation is
> "fixed". It seems to have the same signing fault as well.

You're certainly optimistic. There's usually an uncomfortable time lag
between "fixed" and "widely deployed".

[James Walker]

Hey Mike (et al),

On Tue, Oct 26, 2010 at 6:46 PM, Mike Macgirvin <mi...@macgirvin.com> wrote:
> During some of my compatibility testing w/mistpark, I've discovered
> that several projects are using incorrect implementations of salmon
> magic signatures. In fact, I have not yet found one that is
> compliant.

It's true, the signing message format has changed (I apparently missed
it getting promoted from 'experimental').

> This creates a huge mess because one must send two slaps to each of
> these providers - the first using the correct implementation, and then
> if that fails - repeat using the incorrect format. These have to be
> verified in the same way, falling back to the incorrect format if the
> correct one fails.
>
> Unfortunately, this means that each of you who wishes to provide
> federated services must also implement this duplication of effort
> until such time as everybody has upgraded and the broken
> implementations begin to fade from existence (this could take a few
> years -  which is why everybody needs to start fixing it today(!)).

Agreed it's a hassle, but it's the nature of the beast when
implementing draft specs that are still under heavy revision. I
believe Salmon is getting "real close now"(tm) - but I'll let John
provide official word there.

That said, I can speak for StatusNet and say - thanks for bringing
this to our attention, I'll be rolling out a fix ASAP.

> The crux of the problem is in section 8 of draft-panzer-magicsig-00
> (section 9.1 if you're working off the trunk).
>
> The implementations I've encountered are only signing the
> base64url_encoded data itself. The correct method is to append the
> string ".YXBwbGljYXRpb24vYXRvbSt4bWw=.YmFzZTY0dXJs.UlNBLVNIQTI1Ng=="
> to the armoured data and use that both for signing and verification.
> (This is a simplification, you should calculate this string yourself
> to allow for future revisions).
>

Good catch. Thanks again!
--
James Walker :: http://walkah.net/ :: http://james.status.net/

[John Panzer]

Hi all,

Sorry for the rough edges around the specification.  I want to provide some background here for those who are interested in it; James pretty much nailed the high level summary of "real close now".

The signature specification has been going through various changes, and we've deliberately not been committing them to specific version numbers or anything like that because we don't want anyone to be stuck long term with having to support stuff that was based on a draft spec (possibly, one with security bugs -- though none have been found so far, knock wood).

We have some experience with the process of getting people to rev their libraries in the OAuth world, in which pretty much everyone was able to rev their services and library code within a week or two when a session fixation attack was discovered and a solution found.  That was in a spec that had already been declared "done" by an informal working group and implemented widely, so my hope is that upgrading experimental Salmon signatures to 1.0 will be much easier than that.

That said, it is a pain to experiment with at the moment because, well, everybody is experimenting.  The signature specification bleeding edge has moved to the Magic Signatures Experimental Draft (http://salmon-protocol.googlecode.com/svn/trunk/draft-panzer-magicsig-experimental-00.html).  That draft has gone through a few rounds of feedback from implementors and we think it's ready for running code -- in fact many services have upgraded already.  I'm personally nearly ready to declare it a 1.0 candidate (and deal only with final reviews, wording fixes, etc. that shouldn't affect existing code). 

There's one complication.  One of the motivations for the changes in Experimental was to see if we could also use it as a basis for signed JSON OAuth Tokens, so that people using both (probably a lot of people) would have fewer arbitrarily different signing mechanisms to have to deal with.  I worked on this with Dirk Balfanz and we came up with a spec proposal for the OAuth Working Group.  It turned out that two other people also had proposals, and Mike Jones of Microsoft is working on "normalizing" them into one coherent proposal, which is being thrashed out right now and will be discussed face to face at the Internet Identity Workshop in Mountain View next week.  If the consensus that comes out of that is one that wants to leverage Magic Signatures, AND there are only one or two small changes to MagicSig needed to make that happen, I'd like to propose making those changes.  If it looks like they're diverging anyway, then I'd like to just label MagicSig 1.0 and be done.  Either way, we should have something that we believe will be stable for a while in a couple of weeks.  The Experimental draft is much improved from the feedback it's received by considering additional environments and use cases, including supporting more general data types and a compact serialization for space savings, so the effort has been worth while in any case.

The changes being asked for so far amount mainly to things like not including the == padding in the base64 encoded data.  If that were the only thing keeping us from converging I'd change it in a second.  

As an aside:  Does anyone else have an opinion about including the padding or not including it just based on the merits, not on compatibility with OAuth Tokens?  

Pros: Shorter, does not require URL escaping.

Cons: Nonstandard, requires library switch or manual truncation of content & re-padding if library doesn't support.  Also, you can detect and fix padding over-escaping fairly easily.

 

--
John Panzer / Google
jpa...@google.com / abstractioneer.org / @jpanzer

[Mike Macgirvin]

> Cons: Nonstandard

That pretty well sums it up for me. With all due respects John - salmon
magic envelopes already require the addition of a lot of bloat to
support its unique way of doing things - at least on the php platform.

I understand the reasoning and appreciate the mechanisms. But couldn't
some of this stuff be done with standard libraries and not require
special salmon flavours of everything (e.g. encryption libs, key
formats, base64 encoders, etc.)?

[John Panzer]

We're using bog-standard encryption libraries for  RSA-SHA256.  I'm curious as to what encryption problems PHP brings to the table?

(Note that the reason for not requiring certificates, and using a simple RSA keypair format, is that you end up needing to install compiled C libraries to parse ASN.1 stuff for a lot (all?) non-Java languages, which in some environments is a nonstarter.)

--
John Panzer / Google
jpa...@google.com / abstractioneer.org / @jpanzer

[Kingsley Idehen]

What is the current state of SWATO if all the current Salmon Slap based implementations are broken?

As you can imagine, our submission (ODS) was naturally ignored (zero feedback re. interop tests) since it used WebID and Semantic Pingbacks i.e., secure notifications based on semantically provable identity that resides in the structured data.

For sake of bridge building, we've also gone through the process of implementing Salmon Slap without a broken implementation of Salmon Slap magic envelopes. Thus, are we going to redo SWATO interop testing since it's currently broken re. those implementations that are Salmon Slap based?

-- 

Regards,

Kingsley Idehen	      
President & CEO 
OpenLink Software     
Web: http://www.openlinksw.com
Weblog: http://www.openlinksw.com/blog/~kidehen
Twitter/Identi.ca: kidehen 

[James Walker]

On Fri, Oct 29, 2010 at 6:00 PM, John Panzer <jpa...@google.com> wrote:
> We're using bog-standard encryption libraries for  RSA-SHA256.  I'm curious
> as to what encryption problems PHP brings to the table?

Plenty - but that's PHP's problem ... lack of quality libraries. (imnsho).

That said - it was an issue when we first started implementing salmon,
but the phpseclib library that StatusNet is using has proven to be
reliable since. Thought, the PHP gmp extension makes a *world* of
difference for performance (but that's true of *any* crypto -
diffe-hellman for openid, et al has the same issue).

I think the only "non standard" bit is the base64url encoding but it's
a single line to implement in PHP on top of the standard
base64_encode/decode.

> (Note that the reason for not requiring certificates, and using a simple RSA
> keypair format, is that you end up needing to install compiled C libraries
> to parse ASN.1 stuff for a lot (all?) non-Java languages, which in some
> environments is a nonstarter.)
>
> --
> John Panzer / Google
> jpa...@google.com / abstractioneer.org / @jpanzer
>
>
> On Fri, Oct 29, 2010 at 2:27 PM, Mike Macgirvin <mi...@macgirvin.com> wrote:
>>>
>>> Cons: Nonstandard
>>
>> That pretty well sums it up for me. With all due respects John - salmon
>> magic envelopes already require the addition of a lot of bloat to support
>> its unique way of doing things - at least on the php platform.
>>
>> I understand the reasoning and appreciate the mechanisms. But couldn't
>> some of this stuff be done with standard libraries and not require special
>> salmon flavours of everything (e.g. encryption libs, key formats, base64
>> encoders, etc.)?
>>
>
>

--

[John Panzer]

On Fri, Oct 29, 2010 at 3:10 PM, James Walker <wal...@walkah.net> wrote:

On Fri, Oct 29, 2010 at 6:00 PM, John Panzer <jpa...@google.com> wrote:
> We're using bog-standard encryption libraries for  RSA-SHA256.  I'm curious
> as to what encryption problems PHP brings to the table?

Plenty - but that's PHP's problem ... lack of quality libraries. (imnsho).

That said - it was an issue when we first started implementing salmon,
but the phpseclib library that StatusNet is using has proven to be
reliable since. Thought, the PHP gmp extension makes a *world* of
difference for performance (but that's true of *any* crypto -
diffe-hellman for openid, et al has the same issue).

I think the only "non standard" bit is the base64url encoding but it's
a single line to implement in PHP on top of the standard
base64_encode/decode.

OK, that's good to know.  (Note: I'd call base64url "not as well supported as we'd like" rather than non standard, since it's in the original base64 encoding standard; this is actually somewhat important since if a library DOES implement it, I think it's likely to be interoperable with other libraries.)

[Evan Prodromou]

On Thu, 2010-10-28 at 12:58 -0400, James Walker wrote:

That said, I can speak for StatusNet and say - thanks for bringing
this to our attention, I'll be rolling out a fix ASAP.

Needless to say, this is a pretty important fix to make.

James is going to make sure that new versions of StatusNet support "old" (=bad) and "new" (=compliant) signatures.

It's our issue for implementing the spec early; we don't want to be screwing up the whole network as the bad players. So, people shouldn't have to do double-posts for much much longer.

-Evan

Evan Prodromou, CEO

StatusNet Inc., 1124 rue Marie-Anne Est #32, Montreal, QC H2J 2T5

T: 438-380-4801 x101 C: 514-554-3826 W: http://evan.status.net/ E: ev...@status.net