Chrome: From NSS to OpenSSL

[Kathleen Wilson]

Draft Design Doc posted by Ryan Sleevi regarding Chrome migrating from
NSS to OpenSSL:

https://docs.google.com/document/d/1ML11ZyyMpnAr6clIAwWrXD53pQgNR-DppMYwt9XvE6s/edit?pli=1
"Switching to OpenSSL, however, has the opportunity to bring significant
performance and stability advantages to iOS, Mac, Windows, and ChromeOS
immediately out of the gate. Switching Linux to use OpenSSL will take
longer, due to the desire to continue to support PKCS#11-based smart
card authentication, which will require more work. The biggest risk/cost
to such a switch is no longer being able to help Firefox benefit from
these efforts, nor benefiting from Firefox’s efforts in these areas."

Kathleen

[yann.s...@gmail.com]

Nice work,

I am not clear on the cons for OpenSSL:

Limited-to-no extension points for dealing with networking fetching (AIA, CRL, OCSP)

OCSP and CRL are available ? What do you mean with this sentence. Which may be a good point for security coverage.

Thanks

[Julien Pierre]

On 1/27/2014 10:28, Kathleen Wilson wrote:
> Draft Design Doc posted by Ryan Sleevi regarding Chrome migrating from
> NSS to OpenSSL:
>
> https://docs.google.com/document/d/1ML11ZyyMpnAr6clIAwWrXD53pQgNR-DppMYwt9XvE6s/edit?pli=1
>
> "Switching to OpenSSL, however, has the opportunity to bring
> significant performance and stability advantages to iOS, Mac, Windows,
> and ChromeOS immediately out of the gate. Switching Linux to use
> OpenSSL will take longer, due to the desire to continue to support
> PKCS#11-based smart card authentication, which will require more work.
> The biggest risk/cost to such a switch is no longer being able to help
> Firefox benefit from these efforts, nor benefiting from Firefox’s
> efforts in these areas."
>
> Kathleen

Strange that "PKCS#11 support" is listed as a "con" for NSS .

[Julien Pierre]

[Alan Braggins]

On 31/01/14 10:24, Julien Pierre wrote:
>
> On 1/27/2014 10:28, Kathleen Wilson wrote:
>> Draft Design Doc posted by Ryan Sleevi regarding Chrome migrating from
>> NSS to OpenSSL:
>>
>> https://docs.google.com/document/d/1ML11ZyyMpnAr6clIAwWrXD53pQgNR-DppMYwt9XvE6s/edit?pli=1

> Strange that "PKCS#11 support" is listed as a "con" for NSS .

It is at least listed under "pro" as well.... (Having ENGINE_pkcs11
listed under both for OpenSSL might make sense too.)

[Ryan Sleevi]

It was not accidental that it was listed under "Con", nor do I see
ENGINE_pkcs11 as a "Pro"

As part of its fundamental design, NSS performs all operations using
PKCS#11 tokens. Even the internal cryptographic implementation is exposed
as a PKCS#11 token - softoken.

I realize that from a design and layering perspective, there are those
(especially within NSS developers) that feel this is a Pro. However, in
practical terms, this works out as a significant negative for performance.
It's not "merely" the function call overhead, but the abstraction prevents
significant compiler optimizations for space and size.

Further, the PKCS#11 API itself - while suitable for interacting with IO
contrained hardware tokens - is absolutely horrible for maximizing CPU
thoroughput.

Consider AES-GCM or AES-CBC-PAD. The former requires establishing a new
PKCS#11 context every time, which forces levels of key expansion. I
realize some NSS developers have suggested possible ways to improve this -
but the engineering effort is recognized to be complex. We've also been
committed to the newly-formed OASIS TC to try to improve things, but the
reality is that the current spec is "broken for purpose".

Or consider AES-CBC-PAD, which is essentially *the same issue*. Because of
this, NSS's SSL layer only uses AES-CBC, and handles padding manually, so
that it can keep a single context around for the duration of the SSL
session.

On the server side, NSS is virtually non-existent on any platform that
deeply cares about performance because of this. Heck, there is a whole
"PKCS#11 bypass" mode precisely for this.

This reliance on PKCS#11 means that there are non-trivial overheads when
doing something as "simple" as hashing with SHA-1. For something that is
such a "simple" transformation, multiple locks must be acquired and the
entire NSS internals may *block* if using NSS on multiple threads, in
order to prevent any issues with PKCS#11's threading design.

I tried not to write too much on the negatives of NSS or OpenSSL, because
both are worthy of long rants, but I'm surprised to hear anyone who has
worked at length with PKCS#11 - like Oracle has (and Sun before) - would
be particularly praising it.

It's good for interop with smart cards. That's about it.

Cheers,
Ryan

[Julien Pierre]

Ryan,

On 1/31/2014 10:28, Ryan Sleevi wrote:
> I tried not to write too much on the negatives of NSS or OpenSSL,
> because both are worthy of long rants, but I'm surprised to hear
> anyone who has worked at length with PKCS#11 - like Oracle has (and
> Sun before) - would be particularly praising it. It's good for interop
> with smart cards. That's about it. Cheers, Ryan

I wasn't praising it. I agree with most of your points about the
shortcomings of PKCS#11.
However, as far as your table is concerned, the entry "PKCS#11 support"
under Cons, without any explanation, is confusing.
All the other entries have at least a sentence to explain each pro/con.

Julien

[Alan Braggins]

On 31/01/14 18:28, Ryan Sleevi wrote:
> On Fri, January 31, 2014 9:18 am, Alan Braggins wrote:
>> On 31/01/14 10:24, Julien Pierre wrote:
>>> On 1/27/2014 10:28, Kathleen Wilson wrote:
>>>> Draft Design Doc posted by Ryan Sleevi regarding Chrome migrating from
>>>> NSS to OpenSSL:
>>>>
>>>> https://docs.google.com/document/d/1ML11ZyyMpnAr6clIAwWrXD53pQgNR-DppMYwt9XvE6s/edit?pli=1
>>
>>> Strange that "PKCS#11 support" is listed as a "con" for NSS .
>>
>> It is at least listed under "pro" as well.... (Having ENGINE_pkcs11
>> listed under both for OpenSSL might make sense too.)
>
> It was not accidental that it was listed under "Con", nor do I see
> ENGINE_pkcs11 as a "Pro"
>
> As part of its fundamental design, NSS performs all operations using
> PKCS#11 tokens. Even the internal cryptographic implementation is exposed
> as a PKCS#11 token - softoken.

That conflates two different things though.

Having support for PKCS#11 tokens at all is a pro, even if one
irrelevant to the vast majority of users.

Having a design that shoe-horns all cryptographic operations
through a PKCS#11 token layer has the disadvantages you describe,
but isn't really the same thing as "PKCS#11 support".

> It's good for interop with smart cards.

Which is what it was designed for.....

[Ryan Sleevi]

On Mon, February 3, 2014 4:30 am, David Woodhouse wrote:

> On Mon, 2014-02-03 at 12:13 +0000, Alan Braggins wrote:
> >
> > Having support for PKCS#11 tokens at all is a pro, even if one
> > irrelevant to the vast majority of users.
>

> That gets less true as we start to use PKCS#11 a little more. It isn't
> *just* about hardware tokens — things like gnome-keyring offer
> themselves as a PKCS#11 token, giving you a consistent certificate store
> which is automatically unlocked when you log in, etc. Integration with
> key stores on other operating systems is also fairly easy if you have a
> PKCS#11 interface to them.

At the risk of diverging even further from a suitable NSS discussion, I
will just say that it is categorically *not* the case that it's fairly
easy, as the concepts used in PKCS#11 do not reliably map over well to
other cryptographic APIs (CDSA/CSSM, Security.framework, CryptoAPI, CNG).
This is already true in Chromium's integration, where we use private (in
that they're not upstream, but are open sourced) patches to NSS to avoid
having to emulate PKCS#11 tokens. This is especially true when considering
the different UI interaction requirements.

>
> And it isn't just about keys either — we are starting to use it to get a
> coherent set of trusted certificates system-wide. Which has historically
> been a real PITA on Linux with a different set of trusted certs per
> crypto library, or even per application. Hell, even *Firefox* isn't
> properly using the NSS "Shared System Database" setup, and stupidly
> persists in having its *own* separate store. Fedora 19 starts to make
> sense of this and exports the trust database to a flat file for
> compatibility with OpenSSL, but it's not ideal and still only addresses
> *certs*, not keys.

See p11-kit, which was also referenced in the design doc, which Red Hat
actively contributes to, and which I think is an increasingly viable
solution both for NSS and other applications. This is something we're
looking at supporting, even with our migration to OpenSSL, so that whether
you're using Firefox or Chromium, things should "just work" if you're
using p11-kit.

>
> There is a middle ground between having decent PKCS#11 support where you
> can identify a key/cert by a PKCS#11 URL and everything Just Worksâ„¢
> without jumping through hoops, and having PKCS#11 insinuate itself into
> *everything* as NSS has done. I think GnuTLS has the balance about
> right, and probably would have been a good choice — if it wasn't for the
> GPL allergy, although I note GnuTLS has at least switched back to
> LGPL2.1 in the latest releases.
>
> For OpenSSL to become viable, it's going to need a whole lot more than
> ENGINE_PKCS11.

Our experience has been that when integrated into the library itself (eg:
as NSS has done), it leads for a much poorer experience than when provided
by the application (eg: Chromium), even if it does require more work to
get there.

There's a balance to be struck, for sure, and NSS arguably errs too much
on one particular pattern, which is, again, part of our reasoning for
believing that the transition to OpenSSL will be more in the benefit of
our users and our growing list of needs from an SSL/cryptographic library.

[Wan-Teh Chang]

On Fri, Jan 31, 2014 at 10:28 AM, Ryan Sleevi
<ryan-mozde...@sleevi.com> wrote:
>
> Further, the PKCS#11 API itself - while suitable for interacting with IO
> contrained hardware tokens - is absolutely horrible for maximizing CPU
> thoroughput.
>
> Consider AES-GCM or AES-CBC-PAD. The former requires establishing a new
> PKCS#11 context every time, which forces levels of key expansion. I
> realize some NSS developers have suggested possible ways to improve this -
> but the engineering effort is recognized to be complex. We've also been
> committed to the newly-formed OASIS TC to try to improve things, but the
> reality is that the current spec is "broken for purpose".
>
> Or consider AES-CBC-PAD, which is essentially *the same issue*. Because of
> this, NSS's SSL layer only uses AES-CBC, and handles padding manually, so
> that it can keep a single context around for the duration of the SSL
> session.

Just wanted to correct a flaw in the eloquent argument. The last
paragraph (about AES-CBC-PAD) is not true.

The padding used in TLS has two important differences from the PKCS #7
padding used in CKM_AES_CBC_PAD. 1) The padding value used in TLS
differs by one from the padding value used in PKCS #7 padding. 2) TLS
allows a padding longer than a block size. So CKM_AES_CBC_PAD cannot
be used to implement TLS.

Also, I suspect there was little incentive to add an
AES-CBC-with-TLS-padding mechanism to PKCS #11 because the stream
ciphers (really just RC4) and CBC block ciphers (with implicit IV) in
SSL 3.0 and TLS 1.0 encourage keeping a single encryption context
around for the duration of the SSL/TLS session. Although TLS 1.1 adds
an explicit IV to CBC block ciphers, the spec helpfully provides a
trick to allow using a single encryption context for the duration of
the TLS session.

By the way, OpenSSL (I checked version 0.9.7*, which only supports SSL
3.0 and TLS 1.0) also keeps a single context around for the duration
of the SSL/TLS session. See the |ds| variable in the tls1_enc()
function in t1_enc.c.

Wan-Teh Chang

[Jean-Marc Desperrier]

Ryan Sleevi a écrit :

That was an interesting rant, thanks.

> reliance on PKCS#11 means that there are non-trivial overheads when
> doing something as "simple" as hashing with SHA-1. For something that is
> such a "simple" transformation, multiple locks must be acquired and the

> entire NSS internals may*block* if using NSS on multiple threads, in

> order to prevent any issues with PKCS#11's threading design.

I don't believe that PKCS#11's threading design mandates that.
Implementation easily can have that problem, and NSS sure does, but I
think it would be possible to design a PKCS#11 implementation than let's
you do hashing without requiring locks.
Or maybe, it's more because of PKCS#11 session management rules that you
hardly can avoid them.

> [...] I'm surprised to hear anyone who has

> worked at length with PKCS#11 - like Oracle has (and Sun before) - would
> be particularly praising it.
>
> It's good for interop with smart cards. That's about it.

More or less. Already with HSM there's some serious performance
concerns, and it's probably quite related to what you describe.
And for smart cards, there's the fact that's it's a particularly error
prone interface. All PKCS#11 code that I know that's been tested to
interfaces with many cards/HSM has a very large number of quirks to run
around various failures or buggy behaviors.

[Alan Braggins]

On 08/04/14 13:11, Jean-Marc Desperrier wrote:
> Ryan Sleevi a écrit :

>> reliance on PKCS#11 means that there are non-trivial overheads when
>> doing something as "simple" as hashing with SHA-1. For something that is
>> such a "simple" transformation, multiple locks must be acquired and the

>> entire NSS internals may*block* if using NSS on multiple threads, in

>> order to prevent any issues with PKCS#11's threading design.
>

> I don't believe that PKCS#11's threading design mandates that.
> Implementation easily can have that problem, and NSS sure does, but I
> think it would be possible to design a PKCS#11 implementation than let's
> you do hashing without requiring locks.
> Or maybe, it's more because of PKCS#11 session management rules that you
> hardly can avoid them.

If you do all your hashing in one session, then the rules require
you to use locks. The obvious answer is "don't do that then", but
PKCS#11 libraries for dumb devices are allowed to only support a
limited number of sessions and force you to juggle C_GetOperationState/
SetOperationState.

But if https://developer.mozilla.org/en/docs/PKCS11_FAQ is accurate,
it will only fall back on the single session solution if the device
requires it.

And for hashing without secret keys (plain digest, not HMAC, no
C_DigestKey calls), there's never any reason to use a limited device
for that operation.

[Martin Paljak]

On Tue, Apr 8, 2014 at 1:11 PM, Jean-Marc Desperrier <jmd...@gmail.com> wrote:
>> It's good for interop with smart cards. That's about it.

I'd say that PKCS#11 for smart cards is the last resort an actual user
would want to use. It is more useful for HSM-s than smart cards, which
require usability in addition to a technical low level interface,
which PKCS#11 provides.

Even if the spec would be nice, real life implementations from both
sides (apps and providers) lack a lot of juice.


--
Martin
+372 515 6495

[Martin Paljak]

On Tue, Apr 8, 2014 at 1:11 PM, Jean-Marc Desperrier <jmd...@gmail.com> wrote:

>> It's good for interop with smart cards. That's about it.

[Florian Weimer]

* Julien Pierre:

> Strange that "PKCS#11 support" is listed as a "con" for NSS .

I found the PKCS#11 approach rather difficult to deal with if you're
adding cryptography to some library whose client code has no idea that
there is cryptography involved (and that NSPR and NSS need
initialization). The global state is horrible in this scenario, and
with an increased push towards use of cryptography, it comes up more
and more often.

[Alan Braggins]

Moving to OpenSSL won't remove that issue though:
http://wiki.openssl.org/index.php/Library_Initialization

(OpenSSL, unlike PKCS#11, doesn't require initialization
on fork, though there are related problems:
http://wiki.openssl.org/index.php/Random_fork-safety)

And of course if you use OpenSSL's PKCS#11 engine, then
you have the problem that you reimpose PKCS#11 requirements
on a layer than didn't know it was using PKCS#11.