[net] OpenSSL everywhere?

[Chris Bentzel]

Chromium supports both NSS and OpenSSL for the SSL implementation as
well as for many crypto primitives.

This adds to additional engineering burden: we need different binding
code for both libraries (such as SSLClientSocketNSS versus
SSLClientSocketOpenSSL), occasionally need to implement new features
in both NSS and OpenSSL, and assorted other issues like maintaining
the linux_redux build.

There's also potentially some performance benefits for moving to
OpenSSL on all platforms, although some of the performance gains could
be had by tuning the assembly used in NSS.

What would it take to move to OpenSSL on all platforms?

Would it be worth it?

[William Chan (陈智昌)]

Dumb question: why pick OpenSSL over NSS? Don't we primarily use NSS
everywhere (mod Android)? Is it because OpenSSL has better overall
support outside of the Chromium community?

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev

[David Turner]

First, here's my recollection of why Android uses OpenSSL instead of NSS (I didn't make this decision, I joined the project later):

• code size: OpenSSL is much smaller than NSS
  
• speed: OpenSSL provides optimized assembly routines for ARM, and NSS does not.
  
• maintenance: We use the OpenSSL version maintained by the Android team ($ANDROID/external/openssl), though we have a few of our own patches on top of it, i.e. we share maintenance work with them.
  
• compatibility: To work around platform bugs that existed before Android 4.2, we sometime have to get an OpenSSL keypair handle (EVP_PKEY) from the system, and use it with our own copy of OpenSSL to perform signing / verification (this works because these object layouts are stable between OpenSSL releases, but there is no guarantee that this will always be the case). This is not an initial reason, but is important if we consider switching to something else.
  

IMHO, if considering moving all platforms to move to OpenSSL:

Pros

• Reduced code size
  
  
• Reduced maintenance work (alleged, see note below).

Cons

• OpenSSL doesn't natively use platform Keychain APIs to perform certificate lookup/validation/installation, this must be performed through client-provided callbacks. We wrote them for Android, but new versions would be needed for Win, OS X, etc... There may be some API mismatches making this non-trivial though. In all cases, that's still a lot of new code which will have to be carefully tested (and most of these system APIs _cannot_ be unit-tested, as far as I understand, ain't that fun?).
  
  
• On Linux, I believe we currently use the NSS-provided certificate store, because there are multiple incompatible keychain APIs (gnome-keyring and kwallet being the main ones), though there is an effort to provide a standardized API at http://www.freedesktop.org/wiki/Specifications/secret-storage-spec/. I believe OpenSSL allows you to implement your own store as well, but I'm unsure its API has all the features we'd need (especially with regards to client certificate storage / filtering). Hopefully some experts could chime in.
  
  
• OpenSSL APIs are really really horrible, from a comprehension and maintenance point of view. The library is no doubt written by talented cryptographers, but the API isn't (for historical reasons that would be hard to fix without breaking lots of existing client sources). I have worked extensively with it in the past year, and I'm convinced it's still impossible to use this library properly without multiple passes over its source code to really understand what's going on. While some of that comes from the subject's complexity (crypto _is_ hard), the majority of the issues are non-existing-documentation-at-all, existing-but-badly-written-documentation, and legacy implementation details that are exposed by the public API and make usage painful / ambiguous / perplexing. It's completely possible to write client code for it (we did), but there is a huge mental effort involved that doesn't seem to be required when looking at the NSS APIs (I had to work with them too).
  
  
• The Android OpenSSL port isn't tested for Windows / OS X / whatever. It's also compiled for Linux host for testing though. It looks like it's slightly tricky to build the current upstream sources with MSVC (http://developer.covenanteyes.com/building-openssl-for-visual-studio/), but it seems to be related to the Perl-based OpenSSL Configure script and build system (I assume we'd be using a gyp/gn files with pre-generated config headers instead, just like we currently do for Android and Linux).

So in the end, that's probably worthwhile long-term, but the initial engineering effort to migrate to this library will not be trivial.

There might be other considerations, like FIPS certification, that come into play. I really can't comment on them though :)

Hope this helps, don't hesitate to correct me if I'm wrong.

- Digit

[Paweł Hajdan, Jr.]

For completeness, what would be pros and cons of switching Chrome for Android from OpenSSL to NSS? Is it possible? (I can imagine reasons why it wouldn't, e.g. interacting/integrating with the underlying Android system).

Paweł

[Ryan Sleevi]

See the reasons David listed why we use OpenSSL, especially compatibility and size.

The absence of those (which NSS implies) are clear cons.

[Jonathan Dixon]

On 6 December 2013 03:23, David Turner <di...@chromium.org> wrote:

First, here's my recollection of why Android uses OpenSSL instead of NSS (I didn't make this decision, I joined the project later):

• code size: OpenSSL is much smaller than NSS
  
• speed: OpenSSL provides optimized assembly routines for ARM, and NSS does not.
  
• maintenance: We use the OpenSSL version maintained by the Android team ($ANDROID/external/openssl), though we have a few of our own patches on top of it, i.e. we share maintenance work with them.
  
• compatibility: To work around platform bugs that existed before Android 4.2, we sometime have to get an OpenSSL keypair handle (EVP_PKEY) from the system, and use it with our own copy of OpenSSL to perform signing / verification (this works because these object layouts are stable between OpenSSL releases, but there is no guarantee that this will always be the case).

Yikes! Really? wow.

There's no guarantee that any given OEM has not modified their version of the open ssl library (or even removed it altogether. It's not part of the NDK...)

• This is not an initial reason, but is important if we consider switching to something else.
  

There's an additional code-size saving for Android webview because it dynamically links to the system libopenssl. (Chrome does not do this as OpenSSL is not part of the platform public API. Although from the comment above it sounds like we might as well be....)

Some time ago I heard rumor of various other private chromium ports that were using openssl (related to Google TV?) I don't know of current state of that though.

 

IMHO, if considering moving all platforms to move to OpenSSL:

Pros

• Reduced code size
  
  

+ Speed on ARM. (Assumedly)

• Reduced maintenance work (alleged, see note below). 

Moving to a single SSL stack everywhere surely has some maintenance benefits (single config to test & debug, less gyp shenanigans, consistent compat & security story across all ports, etc) regardless of which is used. 

Cons

• OpenSSL doesn't natively use platform Keychain APIs to perform certificate lookup/validation/installation, this must be performed through client-provided callbacks. We wrote them for Android, but new versions would be needed for Win, OS X, etc... There may be some API mismatches making this non-trivial though. In all cases, that's still a lot of new code which will have to be carefully tested (and most of these system APIs _cannot_ be unit-tested, as far as I understand, ain't that fun?).
  
  
• On Linux, I believe we currently use the NSS-provided certificate store, because there are multiple incompatible keychain APIs (gnome-keyring and kwallet being the main ones), though there is an effort to provide a standardized API at http://www.freedesktop.org/wiki/Specifications/secret-storage-spec/. I believe OpenSSL allows you to implement your own store as well, but I'm unsure its API has all the features we'd need (especially with regards to client certificate storage / filtering). Hopefully some experts could chime in.
  
  
• OpenSSL APIs are really really horrible, from a comprehension and maintenance point of view. The library is no doubt written by talented cryptographers, but the API isn't (for historical reasons that would be hard to fix without breaking lots of existing client sources). I have worked extensively with it in the past year, and I'm convinced it's still impossible to use this library properly without multiple passes over its source code to really understand what's going on. While some of that comes from the subject's complexity (crypto _is_ hard), the majority of the issues are non-existing-documentation-at-all, existing-but-badly-written-documentation, and legacy implementation details that are exposed by the public API and make usage painful / ambiguous / perplexing. It's completely possible to write client code for it (we did), but there is a huge mental effort involved that doesn't seem to be required when looking at the NSS APIs (I had to work with them too).
  
  

This ^ is not really a con if the alternative is retaining both the NSS and OpenSSL ports. That is to say, with the status quo we already incur this specific downside in spades.

OTOH, if the alternative is to move purely to NSS, this is most certainly a drawback of picking OpenSSL.

[Peter Kasting]

On Fri, Dec 6, 2013 at 3:52 PM, Jonathan Dixon <jo...@chromium.org> wrote:

This ^ is not really a con if the alternative is retaining both the NSS and OpenSSL ports. That is to say, with the status quo we already incur this specific downside in spades.

OTOH, if the alternative is to move purely to NSS, this is most certainly a drawback of picking OpenSSL.

My only random-layman comment is that it seems like either uniform world is better than the current state.

(I feel like I've heard a lot of things over the years that portray OpenSSL in a poor light compared to NSS, but I have no idea how much of that is just my uninformed perception.)

PK

[Chris Bentzel]

Is android_webview dynamically linking to system OpenSSL a hard requirement?

Arguably it would be easier to shift to NSS everywhere - and for
performance we could always spot-optimize the assembly for common
ciphers although there may be some systemic slowness due to
abstraction and layers-of-indirection.

However, if android_webview requires system OpenSSL we would always
need to support OpenSSL.

[Torne (Richard Coles)]

Actually I don't think anyone got around to doing the work to use the system OpenSSL for the WebView yet..

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Chris Bentzel]

So is android WebView just packaging its own version of OpenSSL at
this time (similar to Chrome-for-Android)?

In that case we're not going to get the potential memory/binary size
optimizations from sharing a library.

[Torne (Richard Coles)]

Yes, unless I'm missing something it looks like we just statically link it the same way that Chrome for Android does.

[Paweł Hajdan, Jr.]

I think one argument was that OpenSSL is smaller - this would matter even if we compile it in statically.

I did some measurements on my Gentoo system:

Total package size (all installed files, obtained using equery size):

dev-libs/nss-3.15.2  : 17.35 MiB

dev-libs/openssl-1.0.1e-r1 : 15.38 MiB

Just the shared libraries (obtained using equery files <package> 2>/dev/null | grep '.so' | grep -v debug | xargs du -hsc):

dev-libs/nss-3.15.2 : 3.6 MB

dev-libs/openssl-1.0.1e-r1 : 2.6 MB

It's not obvious to me whether the savings are "big enough" (and whether we use all of the .so libraries - that may change the picture as well; maybe one way to measure would be to compare sizes of linux and linux redux build).

Isn't it also an advantage of NSS that Wan-Teh is a developer of NSS? :)

Paweł

[Ryan Sleevi]

On Wed, Dec 11, 2013 at 12:01 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

I think one argument was that OpenSSL is smaller - this would matter even if we compile it in statically.

I did some measurements on my Gentoo system:

Total package size (all installed files, obtained using equery size):

dev-libs/nss-3.15.2  : 17.35 MiB

dev-libs/openssl-1.0.1e-r1 : 15.38 MiB

Just the shared libraries (obtained using equery files <package> 2>/dev/null | grep '.so' | grep -v debug | xargs du -hsc):

dev-libs/nss-3.15.2 : 3.6 MB

dev-libs/openssl-1.0.1e-r1 : 2.6 MB

It's not obvious to me whether the savings are "big enough" (and whether we use all of the .so libraries - that may change the picture as well; maybe one way to measure would be to compare sizes of linux and linux redux build).

Isn't it also an advantage of NSS that Wan-Teh is a developer of NSS? :)

As am I. Although Adam Langley (as well as a number of other Googlers) are effective contributors to OpenSSL, and the Opera team (I believe) previously used OpenSSL as well.

I've been holding off on this thread to see other reactions.

David did an excellent summary about the pros/cons of OpenSSL for Android vs NSS.

NSS

- Pros

  * Supports PKCS#11, which is the standard for all smart card interactions.

  * Provides a stable ABI, with glacial changes (ie: 10 year support, roughly, due to Red Hat's business requirements imposed on upstream NSS)

  * Has a complete RFC 5280 certificate path verification engine AS WELL AS a path *building* engine.

  * Readily available hooks for integrating into existing networking stacks for additional functionality (such as revocation checking)

  * FIPS 140-2 validation designed to be encapsulated within a single DLL, and has been FIPS validated for many years on multiple platforms and at multiple levels

  * Has long supported Windows and Mac as "first class" citizens, due to Firefox's use of it

  * Improvements to NSS for Chromium eventually trickle into Firefox

- Cons

  * The ABI requirements create complex layers of DLL abstraction that can prevent effective compiler optimizations and add additional overhead

  * The 10 year support has lead to an incredible amount of dead code. This is in part due to Oracle's acquisition of Sun and subsequent sacking of the NSS team at Sun while they were mid-way on a whole-scale refactoring effort

  * The path building and verification library is C code made to look like Java code, with simulated exception handling and roughly 7 layers of macro expansion for every function prolog/epilog.

  * Has received for optimizations like assembly, due to it's limited-to-non-existent use on the (web) server side

  * Lots (and LOTS) of Big Global Locks, due to PKCS#11 requirements

OpenSSL

- Pros

  * The ubiquitous, de-facto standard for servers and embedded devices, thus has received heavy optimization for a variety of platforms

  * A very direct API that exposes many internal details, but which makes compiler optimizations very easy

  * Upstream uses git, not hg. Seriously, this is a pro. Trust me. :)

  * Smaller size / memory footprint, due to leaving it up to applications to implement things like memory caches and memoization of parsing results

  * Opera team has experience (is this still the case, Opera guys?)

- Cons

  * Lacks any form of reasonable path building algorithms, and the path verification algorithms are so minimal that they would be unsuitable for "The Web".

  * A truly evolutionary API from one of the first open source crypto libraries, there is a wholesale lack of consistency in terms of API expectations and behaviours. There's often 3 ways to do something, 2 of which are deprecated, and one which doesn't quite work the way you'd think, but looks cool.

  * Has not received a terrible amount of focus on the *client* side, particularly on Windows/Mac.

  * Taking advantage of the optimized assembly requires perl scripts to meta-parse the assembly into the appropriate target syntax.

  * No smartcard support. No, the ENGINE API does not count - as an API or as smart card support :)

  * Limited-to-no AIA/CRL/OCSP fetching support (at least, as easily integrated into a project with its own networking stack)

In the end, we could use NSS on Android (Firefox team has done this, both with Firefox OS and Firefox for Android), or we could use OpenSSL everywhere else (Opera has shown this). There's no question that either effort would be a non-trivial effort with real long-term commitment - are we aggressively working with upstream NSS to remove all the useless, bloated crap that has lingered on for years (eg: I just removed debug defines added 12 years ago that hadn't been used beyond the first week they were added), or are we working on building our own certificate verification engine for OpenSSL?

My arm-chair strategy quarterbacking: We know we need to rewrite NSS's certificate verification library (of which it has technically 2.5, all of which stink-but-are-passable-but-barely). We do that first, and reap the obvious and immediate benefits across platforms. It's easy to then port that bit to OpenSSL's API and perform a more realistic comparison of the pros/cons on the basis of memory and whether things like Smart Card support (on Linux) or FIPS 140-2 (on all platforms) are important for Chrome, and how important are they versus the other gains.

Right now, I don't think the dual-maintenance cost is a gross burden, and, if anything, forces API design decisions to be properly interoperable (I am, for example, thinking of WebCrypto in this case, which benefits from having two compatible underlying library implementations). Long term, we can realize some savings, but I don't think they'll be extreme.

Cheers

[Jiang Jiang]

On Wed, Dec 11, 2013 at 9:58 PM, Ryan Sleevi <rsl...@chromium.org> wrote:
> Right now, I don't think the dual-maintenance cost is a gross burden, and,
> if anything, forces API design decisions to be properly interoperable (I am,
> for example, thinking of WebCrypto in this case, which benefits from having
> two compatible underlying library implementations). Long term, we can

> realise some savings, but I don't think they'll be extreme.

Speaking of WebCrypto and Windows/Mac support, has anyone thought
about using the Crytography: Next Generation [1] and Common Crypto [2]
API as performance optimisations?

I know that some of these APIs are already used in chromium but not
widely and not used for WebCrypto implementation yet, but we have at
least found some quite significant performance improvements by using
CNG and Common Crypto over NSS for digest calculation at least. We
have yet to measure how they compare with OpenSSL, but if they have a
stable API and provides equal or higher performance, would they be a
better fit for Windows and Mac?

- Jiang

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa376210(v=vs.85).aspx
[2] https://developer.apple.com/library/mac/documentation/security/conceptual/cryptoservices/GeneralPurposeCrypto/GeneralPurposeCrypto.html

[Ryan Sleevi]

On Wed, Dec 11, 2013 at 1:58 PM, Jiang Jiang <jia...@opera.com> wrote:

On Wed, Dec 11, 2013 at 9:58 PM, Ryan Sleevi <rsl...@chromium.org> wrote:
> Right now, I don't think the dual-maintenance cost is a gross burden, and,
> if anything, forces API design decisions to be properly interoperable (I am,
> for example, thinking of WebCrypto in this case, which benefits from having
> two compatible underlying library implementations). Long term, we can

> realise some savings, but I don't think they'll be extreme.

Speaking of WebCrypto and Windows/Mac support, has anyone thought
about using the Crytography: Next Generation [1] and Common Crypto [2]
API as performance optimisations?

We have, and they have not turned out to be performance optimizations in practice.

 

I know that some of these APIs are already used in chromium but not
widely and not used for WebCrypto implementation yet, but we have at
least found some quite significant performance improvements by using
CNG and Common Crypto over NSS for digest calculation at least. We
have yet to measure how they compare with OpenSSL, but if they have a
stable API and provides equal or higher performance, would they be a
better fit for Windows and Mac? 

- Jiang

In the beginning of Chrome, we used the native SSL stacks from each platform. SChannel on Windows. Secure Transport on Mac. NSS on Linux (hey, it's in LSB, it counts).

Performance and security stunk for Windows and Mac. AES on XP? Good luck. TLS False Start or SPDY support? Not gonna happen. Apple's bug reporting is often a black hole that only shines light once a major release.

That's not a good state to be in for the web.

Further, these are not accessible from the sandboxed processes - and exposing these APIs to the sandbox would be Very Dangerous, because neither OS offers the ability to say "I want the crypto access, but don't give me access the the users' Keys" (at least, not in a sandbox-proof way; OS X requires some elements of Keychain Services, and Windows requires the root stores).

While there is certainly optimizations that the OS vendors have placed into their crypto stacks that work for a variety of platforms (and there is no question, MSFT has done a *lot* of heavy lifting here for Windows), when you add in the API cost overhead, as well as the fact that we target multiple platforms with disparate sets of cryptographic primitives (eg: no CNG on Windows XP), our options really are: NSS and OpenSSL.

[Marcus Bulach]

I think David, Joth and Ryan already covered all the points, but as one of the original folks that implemented the OpenSSL support in chrome, let me give some historical facts to better understand the decision back then. :)

We're talking about 2010 here:

- Dinosaurs roamed this planet.

- chrome on android was nowhere near being public.

- It was a pre-GingerBread universe: "Google Tablet" was nothing but rumors [1]

- NDK was on its infancy as well.

- there were no "unbundled apps", i.e., "browser", "gmail", etc.., were all part of the system image.

- as such, "chrome" architecture was to become part of the system.. and to shave off a few bytes of the Nexus One (reference device back then), and keep in-line with the existing android browser, it was natural to use "system libraries" as much as possible.... Skia was far simpler, but OpenSSL obviously required far more changes in the chrome side.. icu/i18n was another pain...

Ryan himself was an external contributor and immensely helped us :) even though he almost gave us a heart attack when he naturally uncover our "secret" plan... ;)

Then, around 2011/12:

- android launched HoneyComb (which was only for tablets), and then shortly afterwards we had IceCreamSandwich and "unbundled" apps.

- at this stage, chrome then became an unbundled app, downloadable via the playstore, and we switched to only use the NDK apis, i.e., statically link against all libraries...

- yay! we launched on early 2012.

Thanks,

Marcus

[1] The rumors were "google adds geolocation to chrome on windows, mac and linux. hence a tablet is coming"... non-sequitur, however, both myself and joth worked on geolocation, and then later moved on to chrome on android.. ;)