Lift crypto security vulnerability

[David Pollak]

Folks,

A kind person who was exploring Lift's "most secure web framework" claims found:
https://www.assembla.com/spaces/liftweb/tickets/858-security-vulnerability-in-lift-s-3des---blowfish-encryption

I will be checking in a fix for Lift 2.3-SNAPSHOT.

Is there anyone out there that thinks we should backport the fix to Lift 2.0, 2.1 or 2.2?

Thanks,

David

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics

[Paul Dale]

I'd vote for backporting to 2.2.

2.2 was a big release and it would be a shame to have to say 2.2 is great except for ...

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

[Derek Chen-Becker]

+1. I would even be +1 to backporting to 2.0 and 2.1 if people are using them in production.

Derek

[David Pollak]

None of the routines in question are being use by Lift... they are utilities made available to other end-user applications.

My inclination is to simply deprecate the existing routines (they are vulnerabilities because the same message will result in the same encrypted bytes) and add routines that prepend 16 bytes of random to each encrypted message... that's pretty much what the initialization vectors do, but that way there's no need to share the IVs as well as the keys.

So, if anyone are using these methods in production, please send me a private note with the version of Lift you're using (if you send the note to dpp at athena dot com and your SMTP server supports sending messages over TLS, the message will be encrypted end to end) to I can figure out how widespread the issue is.

[Derek Chen-Becker]

Sounds reasonable to me.

[Ján Raska]

Hi,

after upgrade to 2.3-SNAPSHOT I discovered that blowfish methods are not there any more. I know by now, that it's due to those vulnerabilities, but, shouldn't such changes be done by deprecating methods first, and delete them afterwards, so that the code doesn't blow up after update? Not a pleasant thing to discover, if I'm hurrying with the release of the app. I needed to use 2.3-SNAPSHOT because of SHtml.onEvent method I needed, so there was no way for me to go back to 2.2.

Rusho

[David Pollak]

On Wed, Feb 2, 2011 at 4:51 PM, Ján Raska <ras...@gmail.com> wrote:

Hi,

after upgrade to 2.3-SNAPSHOT I discovered that blowfish methods are not there any more. I know by now, that it's due to those vulnerabilities, but, shouldn't such changes be done by deprecating methods first, and delete them afterwards, so that the code doesn't blow up after update? Not a pleasant thing to discover, if I'm hurrying with the release of the app. I needed to use 2.3-SNAPSHOT because of SHtml.onEvent method I needed, so there was no way for me to go back to 2.2.

I made a command decisions that rather than someone complaining that Lift was somehow insecure, I'd just remove the potentially insecure code.  Here's the patch:

https://github.com/lift/lift/commit/04170b134c8f7ffabfa57a72c88e4ff43f5f43af

I encourage you to copy/paste the removed code into your code as long as you know the caveats of using crypto code with known weaknesses.
 

[Jeppe Nejsum Madsen]

David Pollak <feeder.of...@gmail.com> writes:

> On Wed, Feb 2, 2011 at 4:51 PM, Ján Raska <ras...@gmail.com> wrote:
>
>> Hi,
>> after upgrade to 2.3-SNAPSHOT I discovered that blowfish methods are not
>> there any more. I know by now, that it's due to those vulnerabilities, but,
>> shouldn't such changes be done by deprecating methods first, and delete them
>> afterwards, so that the code doesn't blow up after update? Not a pleasant
>> thing to discover, if I'm hurrying with the release of the app. I needed to
>> use 2.3-SNAPSHOT because of SHtml.onEvent method I needed, so there was no
>> way for me to go back to 2.2.
>>
>
> I made a command decisions that rather than someone complaining that Lift
> was somehow insecure, I'd just remove the potentially insecure code. Here's
> the patch:
>
> https://github.com/lift/lift/commit/04170b134c8f7ffabfa57a72c88e4ff43f5f43af
>
> I encourage you to copy/paste the removed code into your code as long as you
> know the caveats of using crypto code with known weaknesses.

I was in the same boat and did the same thing :-)

I haven't followed the issue closely, but is there a safe way to bring
back the original functionality. We use it a lot for encoding url
parameters in public urls.

/Jeppe

[David Pollak]

The better practice is to put 8 bytes of salt into each encrypted thing.  That will increase the length by 8 but ensure that the same bytes will not result in the same encrypted value.
 

/Jeppe

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

[Jeppe Nejsum Madsen]

David Pollak <feeder.of...@gmail.com> writes:

Ahh yes, I see. Wouldn't it make sense then, to reintroduce the
encrypt/decrypt functions and just always append 8 random bytes?

/Jeppe

[Ján Raska]

I'm not an encryption expert, so I might be wrong, but I think those algorithms need to be compatible in general. Meaning, that if I use a blowfish in Lift with certain key to encrypt a message that will be send to some other system, I must be able to decrypt it on that other system, that might run on something else then Lift... However, appending bytes, thats what I did with original functions anyway. I always salt my encryption, even hashes, and in some cases I even use "random" amount of random bytes as a salt with certain byte serving as an information holder. That's why I was unpleasantly surprised by the removal of functions (though I understand it), as I'm sure, that my approach removed that vulnerability.

And my main concern was not the removal, but the way it has been removed. If methods have been marked as deprecated for 2.3-M1 and removed in 2.3-M2, then I'd have been prepared and I'd write my own functions to do the job without a stress. Now I have been put into situation, that I had a choice to either remove a functionality requested by client (ajax checked select box using SHtml.onEvents) or quickly fix encryption so that it can go to production

Also I'm just wondering, as I don't completely know in detail how blowfish and other symmetrical encryption functions work. Is it possible to encrypt something by blowfish so that if I encrypt again same data I'll get different encryption string, and that at the same time, I still can decrypt the message on other systems that are build with let's say C#? Without using salt or anything (if I use salt, author of the other system has to know about it, if not, it's enough if we share keys).

Finally, I think, that if blowfish or 3DES always produce same encrypted string for same data, then just appending/prepending will not do the job. Because if I was an intruder and came to a point that I can capture encrypted messages, then if I'd just send two or three messages, capture encrypted strings and I'd very quickly discover that part of it is always the same... I'd assume then that varying part is a salt and try to work it out without it. Might not be the case, but if it had some value to me, I'd be worthy the try. So a real salting IMHO needs to be done in some better way, not just appending/prepending.

Rusho

[David Pollak]

I'm sorry that we removed without deprecation, but in the future, I would do the same thing.

The particular methods are not used anywhere in Lift and they can lead to less that secure systems.  See http://stackoverflow.com/questions/3008139/why-is-using-a-non-random-iv-with-cbc-mode-a-vulnerability

So, rather than getting into the business of providing APIs that can be mis-used, I decided to get out of this business altogether.  The cost is about 20 minutes per site that uses these methods (copy/paste them into your own code).  The benefit is that one place where Lift libraries can be mis-used is removed and the probability of Lift getting tarred with the "It's not as secure as they claim" gets reduced.

For everyone who had to deal with the issue of copy/pasting the former Lift blowfish/3DES code into their own, I will buy you a nice lunch or a nice dinner next time we're in the same town and I will spend that meal answering any questions you have on Lift.

2011/2/4 Ján Raska <ras...@gmail.com>

[David Pollak]

On Fri, Feb 4, 2011 at 12:55 PM, David Pollak <feeder.of...@gmail.com> wrote:

I'm sorry that we removed without deprecation, but in the future, I would do the same thing.

The particular methods are not used anywhere in Lift and they can lead to less that secure systems.  See http://stackoverflow.com/questions/3008139/why-is-using-a-non-random-iv-with-cbc-mode-a-vulnerability

So, rather than getting into the business of providing APIs that can be mis-used, I decided to get out of this business altogether.  The cost is about 20 minutes per site that uses these methods (copy/paste them into your own code).  The benefit is that one place where Lift libraries can be mis-used is removed and the probability of Lift getting tarred with the "It's not as secure as they claim" gets reduced.

For everyone who had to deal with the issue of copy/pasting the former Lift blowfish/3DES code into their own, I will buy you a nice lunch or a nice dinner next time we're in the same town and I will spend that meal answering any questions you have on Lift.

Oh... and this is the first time we've torn out APIs without warning in Lift's history and it's likely to be the last.

[Ján Raska]

No problem and no need for a meal, I really do appreciate all the work and effort you put into Lift :). And I completely understand why you did it, and have nothing against it now, that my site is there and running :)

I'm just wondering, isn't it possible to make those functions available in Lift while using random IVs (thus removing the vulnerability)? If no, then I suggest to remove this wiki site (that's how I found out about them): https://www.assembla.com/wiki/show/liftweb/Obscuring_Sensitive_Data or at least change it, so that the example is about using SHA or MD5.

[Timothy Perrett]

On Saturday, February 5, 2011 10:21:18 AM UTC, Rusho wrote:

I'm just wondering, isn't it possible to make those functions available in Lift while using random IVs (thus removing the vulnerability)?

I would also be interested in this :-) 

[Ján Raska]

I've done some research and testing on this. Lift's blowfish method was completely unsafe. It actually run in ECB mode, thus not even using IV at all and just encrypting each block separately. Thus same plaintext always resulted in the same encrypted message. 3DES was a bit better on it, using CBC with PKCS5Padding, but an all-zero IV. 

Appending/prepending suggested by David is only a partial solution in my opinion, in ECB mode in fact no solution at all (I described it earlier). In CBC, it'd be more or less something that IV does, however, IV in CBC mode goes further, and XORs it with the first block, thus removing the possibility of attacker discovering/guessing that there is something appended/prepended.

As David said earlier, using non-zero IV requires IVs to be shared, but if IVs are hardcoded in both sides of communication, we are at the same level of security as having them all-zero. In case of random IVs, these need to be sent to the receiver, so that receiver can reconstruct the cipher and decrypt a message. And as I just learned recently, it's completely safe to send IVs as a plaintext, without any encryption or security at all. That is, because an attacker cannot determine what the encrypted IV looks like without knowing the key, so they actually have no way to know what has been XORed with the first block of plaintext.

Thus if there's a will to have blowfish/3DES methods back in Lift, I'd suggest following:

1. Use Blowfish/CBC/PKCS5Padding

2. For both blowfish and 3DES encryption methods, use random IV generated using java.security.SecureRandom

3. Encryption methods will return tuple (encryptedMessage, generatedIV), or return generatedIV some other way.

4. Decryption methods will have one more parameter - IV, that will be used to initialize Cipher. Thus these methods could be used to decrypt messages, encrypted by other parties, if IV is send with the message, or can be otherwise reconstructed

5. Include AES algorithm too, as it is considered politically safe (US NIST uses it). Some people consider it safer then Blowfish, as there's been less analysis done on Blowfish.

If there's a will, I can try to refactor removed methods so that they'll cover specification described above.

[Christopher Taylor]

another alternative may be the Keyczar library (http://www.keyczar.org/ byďż˝ the Google security team), which is explicitely designed to abstract from these algorithm subtleties and "do the right thing" by default.

Regards,
ďż˝ --Chris

On 02/05/2011 02:37 PM, J�n Raska wrote:

I've done some research and testing on this. Lift's blowfish method was completely unsafe. It actually run in ECB mode, thus not even using IV at all and just encrypting each block separately. Thus same plaintext always resulted in the same encrypted message. 3DES was a bit better on it, using CBC with�PKCS5Padding, but an all-zero IV.�

Appending/prepending suggested by David is only a partial solution in my opinion, in ECB mode in fact no solution at all (I described it earlier). In CBC, it'd be more or less something that IV does, however, IV in CBC mode goes further, and XORs it with the first block, thus removing the possibility of attacker discovering/guessing that there is something appended/prepended.

As David said earlier, using non-zero IV requires IVs to be shared, but if IVs are hardcoded in both sides of communication, we are at the same level of security as having them all-zero. In case of random IVs, these need to be sent to the receiver, so that receiver can reconstruct the cipher and decrypt a message. And as I just learned recently, it's completely safe to send IVs as a plaintext, without any encryption or security at all. That is, because�an attacker cannot determine what the�encrypted�IV looks like�without knowing the key, so they actually have no way to know what has been XORed with the�first block of plaintext.

Thus if there's a will to have blowfish/3DES methods back in Lift, I'd suggest following:

1. Use�Blowfish/CBC/PKCS5Padding

2. For both blowfish and 3DES encryption methods, use random IV generated using�java.security.SecureRandom

3. Encryption methods will return tuple (encryptedMessage, generatedIV), or return generatedIV some other way.

4. Decryption methods will have one more parameter - IV, that will be used to initialize Cipher. Thus these methods could be used to decrypt messages, encrypted by other parties, if IV is send with the message, or can be otherwise reconstructed

5. Include AES algorithm too, as it is considered politically safe (US NIST uses it). Some people consider it safer then Blowfish, as there's been less analysis done on Blowfish.

If there's a will, I can try to refactor removed methods so that they'll cover specification described above.

On Feb 5, 2011, at 13:20 , Timothy Perrett wrote:

On Saturday, February 5, 2011 10:21:18 AM UTC, Rusho wrote:

I'm just wondering, isn't it possible to make those functions available in Lift while using random IVs (thus removing the vulnerability)?

I would also be interested in this :-)ďż˝

ďż˝

If no, then I suggest to remove this wiki site (that's how I found out about them):�https://www.assembla.com/wiki/show/liftweb/Obscuring_Sensitive_Data or at least change it, so that the example is about using SHA or MD5.

On Feb 4, 2011, at 21:55 , David Pollak wrote:

On Fri, Feb 4, 2011 at 12:55 PM, David Pollak <feeder.of...@gmail.com> wrote:

I'm sorry that we removed without deprecation, but in the future, I would do the same thing.

The particular methods are not used anywhere in Lift and they can lead to less that secure systems. �See�http://stackoverflow.com/questions/3008139/why-is-using-a-non-random-iv-with-cbc-mode-a-vulnerability

So, rather than getting into the business of providing APIs that can be mis-used, I decided to get out of this business altogether. �The cost is about 20 minutes per site that uses these methods (copy/paste them into your own code). �The benefit is that one place where Lift libraries can be mis-used is removed and the probability of Lift getting tarred with the "It's not as secure as they claim" gets reduced.

For everyone who had to deal with the issue of copy/pasting the former Lift blowfish/3DES code into their own, I will buy you a nice lunch or a nice dinner next time we're in the same town and I will spend that meal answering any questions you have on Lift.

Oh... and this is the first time we've torn out APIs without warning in Lift's history and it's likely to be the last.

ďż˝

2011/2/4 J�n Raska <ras...@gmail.com>

[David Pollak]

Can I suggest that you create the code and publish it on Scala-tools.org.

2011/2/5 Ján Raska <ras...@gmail.com>

[Ján Raska]

Sure, I can do that, if I have your permission to use original Lift's code as a base. Can you also provide me some directions how to publish stuff there, I've never published source code through maven, I usually did so on github or sourceforge.

[David Pollak]

2011/2/6 Ján Raska <ras...@gmail.com>

Sure, I can do that, if I have your permission to use original Lift's code as a base.

Yes.  Lift code is under Apache 2.0 license ( http://www.apache.org/licenses/LICENSE-2.0.html ).

You can copy/paste the code to your heart's content as long as you include the original copyright notice.
 

Can you also provide me some directions how to publish stuff there, I've never published source code through maven, I usually did so on github or sourceforge.

Perhaps Indrajit or DavidB can jump in here.  I'm all thumbs when it comes to how to use Maven or SBT to publish code.

Also, please see http://scala-tools.org for instructions on how to apply for a Scala-tools account.
 

[Indrajit Raychaudhuri]

J�n,

David Pollak wrote:
>
>
> 2011/2/6 J�n Raska <ras...@gmail.com <mailto:ras...@gmail.com>>

>
> Sure, I can do that, if I have your permission to use original
> Lift's code as a base.
>
>
> Yes. Lift code is under Apache 2.0 license (
> http://www.apache.org/licenses/LICENSE-2.0.html ).
>
> You can copy/paste the code to your heart's content as long as you
> include the original copyright notice.
>
> Can you also provide me some directions how to publish stuff there,
> I've never published source code through maven, I usually did so on
> github or sourceforge.
>
>
> Perhaps Indrajit or DavidB can jump in here. I'm all thumbs when it
> comes to how to use Maven or SBT to publish code.

Please have your stuff on github and get going. You are free to use
either SBT or Maven (whatever makes you comfortable). Once done,
publishing can be taken care of. Just notify on the list (DavidB, Derek
or myself would take care).

>
> Also, please see http://scala-tools.org for instructions on how to apply
> for a Scala-tools account.

You still need to do this to have your Nexus credentials :)

- Indrajit

>
>
> On Feb 6, 2011, at 1:53 , David Pollak wrote:
>
>> Can I suggest that you create the code and publish it on

>> Scala-tools.org <http://Scala-tools.org>.
>>
>> 2011/2/5 J�n Raska <ras...@gmail.com <mailto:ras...@gmail.com>>

>>>> 2011/2/4 J�n Raska <ras...@gmail.com>

>>>>
>>>>
>>>> On Feb 4, 2011, at 9:11 , Jeppe Nejsum Madsen wrote:
>>>>
>>>> > David Pollak <feeder.of...@gmail.com> writes:
>>>> >
>>>> >> On Thu, Feb 3, 2011 at 12:57 AM, Jeppe Nejsum
>>>> Madsen <je...@ingolfs.dk>wrote:
>>>> >>
>>>> >>> David Pollak <feeder.of...@gmail.com> writes:
>>>> >>>

>>>> >>>> On Wed, Feb 2, 2011 at 4:51 PM, J�n Raska

>>>> http://liftweb.net <http://liftweb.net/>

>>>> Beginning Scala
>>>> http://www.apress.com/book/view/1430219890
>>>> Follow me: http://twitter.com/dpp

>>>> Blog: http://goodstuff.im <http://goodstuff.im/>

>>>> Surf the harmonics
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lift, the simply functional web framework

>>>> http://liftweb.net <http://liftweb.net/>

>>>> Beginning Scala http://www.apress.com/book/view/1430219890
>>>> Follow me: http://twitter.com/dpp

>>>> Blog: http://goodstuff.im <http://goodstuff.im/>

>>>> Surf the harmonics
>>>>
>>>> --
>>>> You received this message because you are subscribed to
>>>> the Google Groups "Lift" group.
>>>> To post to this group, send email to
>>>> lif...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> liftweb+u...@googlegroups.com.
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/liftweb?hl=en.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "Lift" group.
>>> To post to this group, send email to lif...@googlegroups.com

>>> <mailto:lif...@googlegroups.com>.

>>> To unsubscribe from this group, send email to
>>> liftweb+u...@googlegroups.com

>>> <mailto:liftweb+u...@googlegroups.com>.

>>> For more options, visit this group at
>>> http://groups.google.com/group/liftweb?hl=en.
>>
>>
>>
>>
>> --
>> Lift, the simply functional web framework http://liftweb.net

>> <http://liftweb.net/>

>> Beginning Scala http://www.apress.com/book/view/1430219890
>> Follow me: http://twitter.com/dpp

>> Blog: http://goodstuff.im <http://goodstuff.im/>

>> Surf the harmonics
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Lift" group.
>> To post to this group, send email to lif...@googlegroups.com

>> <mailto:lif...@googlegroups.com>.

>> To unsubscribe from this group, send email to
>> liftweb+u...@googlegroups.com

>> <mailto:liftweb+u...@googlegroups.com>.

[Ján Raska]

Thanks for the info.
Code is on github: https://github.com/rusho/softwave-util

I also applied for Scala-tools account a minute ago, if you need any info about that please let me know. I used softwave.util package for the code, and sk.softwave.softwave-util as a maven package name, but if you want to have it under net.liftweb or anything else, you're welcome to change it.

On Feb 7, 2011, at 9:16 , Indrajit Raychaudhuri wrote:

> Ján,
>
> David Pollak wrote:
>>
>>
>> 2011/2/6 Ján Raska <ras...@gmail.com <mailto:ras...@gmail.com>>

>>
>> Sure, I can do that, if I have your permission to use original
>> Lift's code as a base.
>>
>>
>> Yes. Lift code is under Apache 2.0 license (
>> http://www.apache.org/licenses/LICENSE-2.0.html ).
>>
>> You can copy/paste the code to your heart's content as long as you
>> include the original copyright notice.
>>
>> Can you also provide me some directions how to publish stuff there,
>> I've never published source code through maven, I usually did so on
>> github or sourceforge.
>>
>>
>> Perhaps Indrajit or DavidB can jump in here. I'm all thumbs when it
>> comes to how to use Maven or SBT to publish code.
>
> Please have your stuff on github and get going. You are free to use either SBT or Maven (whatever makes you comfortable). Once done, publishing can be taken care of. Just notify on the list (DavidB, Derek or myself would take care).
>
>>
>> Also, please see http://scala-tools.org for instructions on how to apply
>> for a Scala-tools account.
>
> You still need to do this to have your Nexus credentials :)
>
> - Indrajit
>
>>
>>
>> On Feb 6, 2011, at 1:53 , David Pollak wrote:
>>
>>> Can I suggest that you create the code and publish it on
>>> Scala-tools.org <http://Scala-tools.org>.
>>>

>>> 2011/2/5 Ján Raska <ras...@gmail.com <mailto:ras...@gmail.com>>

>>>>> 2011/2/4 Ján Raska <ras...@gmail.com>

>>>>>
>>>>>
>>>>> On Feb 4, 2011, at 9:11 , Jeppe Nejsum Madsen wrote:
>>>>>
>>>>> > David Pollak <feeder.of...@gmail.com> writes:
>>>>> >
>>>>> >> On Thu, Feb 3, 2011 at 12:57 AM, Jeppe Nejsum
>>>>> Madsen <je...@ingolfs.dk>wrote:
>>>>> >>
>>>>> >>> David Pollak <feeder.of...@gmail.com> writes:
>>>>> >>>

>>>>> >>>> On Wed, Feb 2, 2011 at 4:51 PM, Ján Raska

[Ján Raska]

Fixed security functions have just made their way to scala-tools.org in as sk.softwave.softwave-util package. It's only in snapshot repository so far, anybody interested please check it out.

Thanks to Derek and Indrajit for their help with publishing :)

Jan

[David Pollak]

Awesome.  Once it's published, I'd be happy to put an article on it on liftweb.net

2011/2/12 Ján Raska <ras...@gmail.com>