Trusted PEM distribution of Mozilla's CA bundle

[Gregory Szorc]

I Googled "mozilla ca bundle" [1] over the weekend, found something that
made me go "wat," and wanted to run my observations past this list in
case I missed something.

Before I present my findings, I should say that while I have basic
understanding of what CAs are, how the chain of trust works, etc, I'm
largely ignorant of how the CA bundles are distributed and consumed in
the wild. Some of the statements I'm about to make may be inaccurate or
influenced for novice understanding. However, the statements were
influenced by content I found on the internets, so they may be
indicative of what "average" developers and system admins believe.

Skipping the back story of how I got here, I found myself wanting to
obtain Mozilla's CA bundle in PEM format so I could explicitly point
some Python code at it. This led me to Google "mozilla ca bundle" so I
could find a copy.

Long story short, the internets is full of guides ([2][3][4][5][6] etc)
detailing how to obtain the bundle in PEM format. The common solution
seems to be:

a) download http://curl.haxx.se/ca/cacert.pem (over regular HTTP!)

or

b) Run
https://raw.githubusercontent.com/bagder/curl/master/lib/mk-ca-bundle.pl
to produce the PEM file that haxx.se distributes.

"b" is a somewhat gnarly-looking Perl script that downloads certdata.txt
[7] from http://hg.mozilla.org/ or http://mxr.mozilla.org/ (more
non-HTTPS URLS!) (hostname depends on which version / instruction you
are looking at), and somehow munges it into a PEM file.

Wat?

Is the best way to obtain a copy of Mozilla's trusted CA bundle in PEM
format (or any other popular format for that matter) really to download
it from a 3rd party or to execute a Perl script that downloads it over HTTP?

It feels silly to be exposing downstream consumers of this all-important
data to poor security hygiene (executing code written by others, 3rd
party trust, non-HTTPS downloads) and needless increases in attack
surface area (hg.mozilla.org and mxr.mozilla.org).

Is there a good reason Mozilla can't host copies of the trusted CA
bundle in popular formats so people can obtain a copy directly from
Mozilla? And while we're at it, can we add some PGP signatures for
additional verification?

[1] https://www.google.ca/search?q=mozilla+ca+bundle
[2] http://curl.haxx.se/docs/caextract.html
[3]
https://stackoverflow.com/questions/23032165/where-is-the-official-download-location-for-cacert-pem
[4] https://gist.github.com/jjb/996292
[5] http://notetoself.vrensk.com/2008/09/verified-https-in-ruby/
[6] http://www.petefreitag.com/item/830.cfm
[7]
https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt

[Anne van Kesteren]

On Mon, Oct 20, 2014 at 4:10 AM, Gregory Szorc <g...@mozilla.com> wrote:
> "b" is a somewhat gnarly-looking Perl script that downloads certdata.txt

> from http://hg.mozilla.org/ or http://mxr.mozilla.org/ (more non-HTTPS
> URLS!) (hostname depends on which version / instruction you are looking at),
> and somehow munges it into a PEM file.

These servers are also available over TLS. I filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1077394 a while ago to
restrict them to TLS.


--
https://annevankesteren.nl/

[Gervase Markham]

On 20/10/14 03:10, Gregory Szorc wrote:
> Is there a good reason Mozilla can't host copies of the trusted CA
> bundle in popular formats so people can obtain a copy directly from
> Mozilla? And while we're at it, can we add some PGP signatures for
> additional verification?

One issue is, perhaps, that Mozilla doesn't perhaps have an official
stance on reuse of this data.

Mozilla is the /de facto/ maintainer of the only "open process" root
program. That is to say, if you want a curated root store with good web
compatibility and you don't want to do your own due diligence, you can
choose Microsoft's, Apple's or Mozilla's. (Google uses whatever the
platform store is, with some tweaks.) Of those three, we are the only
ones who run an open and transparent process. Which makes ours popular.

[The fact that we maintain this is sort of tied to the fact that we also
make NSS, and our root store is the default one in NSS (and so in most
things NSS gets into), but that's not a required link. If NSS had never
existed, we could still be doing this.]

All this means that people take our root store and embed it in things.
And we certainly don't try and stop them. Good luck to 'em, I say. And
when we make decisions about our program, we do sometimes think about
other consumers of our list. For example, we maintain a "code-signing?"
bit for each trusted root even though Mozilla software has, in recent
history, done little or nothing with code signing. (Although that may
change.)

But there's perhaps a difference between "we don't try and stop you",
and "we encourage you". Saying "here it is in a useful format - download
it" would definitely be the latter.

Perhaps we just need to jump that gap and accept what is /de facto/ true.

Gerv

[Anne van Kesteren]

On Mon, Oct 20, 2014 at 3:41 PM, Gervase Markham <ge...@mozilla.org> wrote:
> Perhaps we just need to jump that gap and accept what is /de facto/ true.

Yeah, as with publicsuffix.org we should own this up.


--
https://annevankesteren.nl/

[Ryan Sleevi]

On Mon, October 20, 2014 7:17 am, Anne van Kesteren wrote:
> On Mon, Oct 20, 2014 at 3:41 PM, Gervase Markham <ge...@mozilla.org> wrote:
> > Perhaps we just need to jump that gap and accept what is /de facto/
> > true.
>
> Yeah, as with publicsuffix.org we should own this up.
>

I would, in fact, argue strongly against this, despite recognizing the
value that the open root program has.

The decisions made for the root program are directly tied to the
capabilities and behaviours of the Mozilla software package it's
distributed with - in particular, Firefox. The behaviours, limitations,
bugs, and features of Firefox/NSS (e.g. including both NSS and
mozilla::pkix) play very heavily into the discussion and maintenance of
the root program.

Consider the 1024-bit root removals. For NSS and mozilla::pkix using
applications, a known set of tradeoffs were made to minimize any backwards
compatibility issues. However, a large number of programs with sub-optimal
to non-existent chain building and discovery algorithms (read: OpenSSL)
experienced issue, because the software was too dumb to discover paths to
the 2048-bit roots.

Accepting the Root Program as a product for "general" PKI purposes
inherently means such flawed behaviours are in scope and "supported",
equivalent to public API surfaces. Having to weigh such considerations
when making decisions about how best to secure the public Internet, and
Firefox users at large, is not a desirable point to be.

I have seen plenty of bungled attempts at repackaging the Mozilla list,
and I have zero faith that having an 'official' supported way would in any
way reduce the bungling. In many cases, the bungling is done by
well-intentioned people with ideological axes to grind, rather than people
who understand the issues at play. Adding roots that have never been
audited, re-adding removed roots that haven't been audited for years,
botching the trust records, etc.

Consider this (long) email an encouragement to "caveat repackager", and
say that it's only supported when used with the Mozilla product it's
packaged with - NSS and Firefox. Maintaining a trust store for multiple
PKI products, with differences in behaviour, nuance, and bugs, is not a
scalable operation.

[Brian Smith]

On Mon, Oct 20, 2014 at 8:33 AM, Ryan Sleevi <
ryan-mozde...@sleevi.com> wrote:

> On Mon, October 20, 2014 7:17 am, Anne van Kesteren wrote:
> > On Mon, Oct 20, 2014 at 3:41 PM, Gervase Markham <ge...@mozilla.org>
> wrote:
> > > Perhaps we just need to jump that gap and accept what is /de facto/
> > > true.
> >
> > Yeah, as with publicsuffix.org we should own this up.
>
> I would, in fact, argue strongly against this, despite recognizing the
> value that the open root program has.
>

I strongly agree with Ryan. Besides his very good points, I will add:

Not all of the relevant information about the roots is even available in
certdata.txt. For example, the name constraints on DCSSI are not encoded in
certdata.txt. For a long time there were hard-coded restrictions on some
Comodo and the Diginotar certificates, which weren't encoded in
certdata.txt. None of Google's CRLSet information is in certdata.txt, and
none of Mozilla's OneCRL information is in certdata.txt. One of the key
pinning information is in certdata.txt.

More generally, when Mozilla or Google address issues related to the root
program, they may do so with a code change to Firefox or Chrome that never
touches certdata.txt. And, they might do so in a hidden bug that people
trying to reuse certdata.txt may not see for many months. It's not
reasonable to give everybody who wants to use certdata.txt access to those
bugs, and it's not reasonable to constrain fixes to require they be done
through certdata.txt. AFAICT, none of the libraries or programs that try to
reuse the Mozilla root set even have enough flexibility to add all of these
additional data sources to their validation process.

For example, let's say some CA in Mozilla's root program mis-issues a
google.com certificate. Because google.com is pinned to certain CAs in
Firefox, Mozilla might not take any immediate action and may not make the
public aware of the issue for a long time (or ever).

Note that a consequence of this, even applications that use the NSS cert
verification APIs might not be as safe as they expect to be when trusting
the NSS root CA set.

Cheers,
Brian

[Anne van Kesteren]

On Mon, Oct 20, 2014 at 6:47 PM, Gregory Szorc <gregor...@gmail.com> wrote:
> Quite frankly, I don't care what that messaging around use is as long as it
> is coming straight from Mozilla. Even if the conclusion is "you probably
> shouldn't use this CA bundle," I think adding a link to "but here is the
> bundle in PEM format because many people want it" is still a win.

This seemed like a good suggestion and based on what Ryan and Brian
said I added this question to the FAQ:

https://wiki.mozilla.org/CA:FAQ#Can_I_use_Mozilla.27s_set_of_CA_certificates.3F

Based on what was said about certdata.txt I filed this bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=1086194


--
https://annevankesteren.nl/

[Gervase Markham]

On 21/10/14 10:06, Anne van Kesteren wrote:
> This seemed like a good suggestion and based on what Ryan and Brian
> said I added this question to the FAQ:
>
> https://wiki.mozilla.org/CA:FAQ#Can_I_use_Mozilla.27s_set_of_CA_certificates.3F

Anne wrote:

"The decisions Mozilla makes with regards to the inclusion of CA
certificates is directly tied to the capabilities and behaviors of the
software Mozilla distributes. It would therefore be irresponsible to
bundle Mozilla's set of CA certificates with other software."

I don't agree with that. I think Ryan's "caveat embeddor" is a much
better way to put it. I would say something like:

"The decisions Mozilla makes with regards to the inclusion or exclusion
of CA certificates in its root store are directly tied to the
capabilities and behaviours of the software Mozilla distributes.
Sometimes, a security change is made wholly or partly in the software
instead of the root store. Further, Mozilla does not promise to take
into account the needs of other users of its root store when making such
decisions.

Therefore, anyone considering bundling Mozilla's root store with other
software needs to be aware of the issues surrounding providing a root
store, and committed to making sure that they maintain security for
their users by carefully observing Mozilla's actions and taking
appropriate steps of their own.

Gerv

[Gregory Szorc]

This is an interesting and new-to-me perspective.

Reading this, I couldn't help but think that
https://wiki.mozilla.org/CA:FAQ needs a "Should I use the Mozilla CA
bundle?" entry. I'm not qualified to write it, otherwise I would add it.

While I'm here, I should say that I didn't arrive at "use the Mozilla CA
bundle" by accident: I was guided to that solution by a trove of
information on the internet. Like it or not, Mozilla's CA bundle is used
beyond its original intent. (Also, searching for "should I use Mozilla's
CA bundle" doesn't return anything helpful - just how-to-use-it
results.) At the very least, I think Mozilla needs to step up and own
the messaging around using the bundle. Today, the messaging around use
isn't controlled by Mozilla: it's controlled by haxx.se, random blogs,
Stack Overflow answers, etc and pretty much nothing addresses the
"should I" aspect. IMO 3rd party hosting of the messaging is a gross
violation of the chain of trust and significantly undermines a lot of
the effort that goes into the CA program.

[Peter Bowen]

On Tue, Oct 21, 2014 at 3:04 AM, Gervase Markham <ge...@mozilla.org> wrote:
>>
>> https://wiki.mozilla.org/CA:FAQ#Can_I_use_Mozilla.27s_set_of_CA_certificates.3F

>
> "The decisions Mozilla makes with regards to the inclusion or exclusion
> of CA certificates in its root store are directly tied to the
> capabilities and behaviours of the software Mozilla distributes.
> Sometimes, a security change is made wholly or partly in the software
> instead of the root store. Further, Mozilla does not promise to take
> into account the needs of other users of its root store when making such
> decisions.
>
> Therefore, anyone considering bundling Mozilla's root store with other
> software needs to be aware of the issues surrounding providing a root
> store, and committed to making sure that they maintain security for
> their users by carefully observing Mozilla's actions and taking
> appropriate steps of their own.

Should there also be a warning that not all certificates in Mozilla's
CA list are trusted? I've seen people completely skip the flags when
converting certdata.txt, resulting in trusting things explicitly
distrusted.

[Kathleen Wilson]

Thanks to all of you for your suggestions about this.

I updated the wiki page based on the input so far:
https://wiki.mozilla.org/CA:FAQ#Can_I_use_Mozilla.27s_set_of_CA_certificates.3F

Kathleen