Intermediate CA Preloading is enabled for desktop Nightly users

[J.C. Jones]

# tl;dr #

At the end of February I enabled Intermediate CA Preloading for all
desktop Nightly users to begin gathering telemetry. This means all
intermediate CAs disclosed to Mozilla will be pre-loaded into
profiles, so the common secure website misconfiguration of forgetting
this certificate won’t stop Firefox users. It has other benefits, too,
but for those read on.




In Bug 1404934 we've added support to download the Intermediate
Certificate Authorities that have been disclosed to the Mozilla CA
Root Program from Kinto in the background during normal Firefox
operation. I turned this on for all desktop nightly users a bit more
than a week ago, prompting Nightly users to download 100 intermediate
CA certs into their profile each day. I do not have a timeline at
present for this to ride the trains.


# What it does #

Websites using encryption should provide two digital PKI certificates
[0] when connecting to clients: One for the website itself, and one
for the intermediate CA that produced the website's digital
certificate. Sometimes, websites are set up incorrectly.


When other browsers encounter this case, they use AIA-chasing, a
mechanism where the user's browser then, in the background, connects
to the CA and downloads the certificate during the TLS handshake.

## Performance benefit ##

Preloading the intermediate CA data avoids the just-in-time network
fetch, which delays the connection.

## Privacy ##

Avoiding the network fetch is also a privacy improvement, as it
doesn't disclose your browsing pattern to the certificate authority
that issued the misconfigured website's certificate.

It's also helpful because it's possible to "fingerprint" a user by a
website carefully analyzing what other websites load or do not load
due to which intermediate CAs have been "seen" by a user [1].
Preloading ensures that all users have "seen" all the same
intermediate CAs.

## Protection ##

The Mozilla CA program requires all members to disclose unconstrained
intermediate CAs before using them, as a safety measure. However,
there is not currently a way to technically enforce that. Intermediate
Preloading could eventually be used to perform that enforcement, by
only trusting intermediate CAs which were disclosed already. This
protects against a scenario where a compromised CA is coerced into
producing an undisclosed secret intermediate CA, which is then used to
attack people on the Internet.


## Future usefulness ##

The experimental CRLite concept [2] (aiming to replace OCSP) requires
a mapping of Intermediate CAs to an “enrollment status” flag. Since we
need to enumerate the Intermediate CAs for that future functionality,
actually preloading their data is a rational extension.


# How it works #

Intermediate Preloading fetches ~100 intermediate certificate
authorities' certificates once a day during the Kinto main update [3],
and loads them into your profile, as if you had visited a site that
used that intermediate. At 100 per day, summing to between 300-500 kB,
it will take approximately three weeks for a Firefox profile to
preload all intermediates [4]. We will likely increase the rate after
the initial experiments.

The certificate data is loaded into the NSS Certificate Database, as
is done for normal web browsing. In the future, we will use the faster
Rust "rkv" database, in Bug 1530545.

Currently preloading is only enabled for desktop users on Nightly. We
will want to (A) restrict the download to be only over WiFi and maybe
plugged in, and (B) switch to the faster database before enabling on
mobile. (Bug 1520297)


# What we expect #

Intermediate Preloading should reduce the number of
SEC_ERROR_UNKNOWN_ISSUER errors seen by Firefox users over time, which
is our most common secure connection error. It will also remove the
tracking vector (above, Privacy), and give us a path to enforce CA
disclosure.


# Things to keep in mind #

Right now, these certificates go into the NSS Certificate Database,
and are visible in / totally overcrowding the Certificate Manager (Go
To about:preferences#privacy , View Certificates.., Authorities tab).
That’ll change when this moves to rkv.

This is potentially a bunch of data, approximately 3 megabytes of
binary, but it’s data that changes only very slowly.


# Telemetry #

Telemetry for Intermediate Preloading is available in the histograms:

"INTERMEDIATE_PRELOADING_ERRORS"
"INTERMEDIATE_PRELOADING_UPDATE_TIME_MS"

and the scalars:

"security.intermediate_preloading_num_pending"
"security.intermediate_preloading_num_preloaded"

... which show how many of the ~2300 intermediates have been loaded,
per profile.

# More info #

Well, maybe. ;) There's a wiki page about this which can be used to
help explain the same things as are in this post. Otherwise, send your
questions to me, Mark Goodwin, or Dana Keeler.

https://wiki.mozilla.org/Security/CryptoEngineering/Intermediate_Preloading

:: Footnotes::

[0] The WebPKI generally has one root CA certificate, one intermediate
CA certificate, and then one end-entity (specific website)
certificate. Sometimes there can be more than one intermediate CA
certificate, potentially much more than one. Sometimes that’s because
of cross-certification CA certificates, sometimes it’s technical debt
someone hasn’t cleaned up yet. See https://signed.bad.horse for an
egregious (and untrusted) example.
(https://tls-observatory.services.mozilla.com/static/certsplainer.html?id=188088012)

[1] https://shiftordie.de/blog/2017/02/21/fingerprinting-firefox-users-with-cached-intermediate-ca-certificates-fiprinca/

[2] https://wiki.mozilla.org/CA/Revocation_Checking_in_Firefox#CRLite

[3] 100/day is configurable by a pref, it is likely to change. See
https://searchfox.org/mozilla-central/source/security/manager/ssl/security-prefs.js#166
.

[4] The data is loaded from Kinto here:
https://settings.prod.mozaws.net/v1/buckets/security-state/collections/intermediates/records
. This data is exported from the Common CA Database maintained by the
Mozilla root program.

[Tom Ritter]

How does kinto know which certificates you yet need to download?

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[J.C. Jones]

Tom,

Kinto provides the whole list of metadata to clients as soon as it syncs
[1]. The metadata uses the Kinto attachment
<https://github.com/Kinto/kinto-attachment> mechanism to store the
DER-encoded certificate for separate download.

Firefox maintains a "local field" boolean in the dataset to of whether a
given metadata entry's certificate attachment has been downloaded or not,
toggling as each one is pulled. Currently we don't deduplicate with the
local NSS Cert DB, the inserts that are already there will fail and emit
telemetry -- the amount of data saved didn't seem worth it for the
experimental phase.

Thanks!

[1]
https://settings.prod.mozaws.net/v1/buckets/security-state/collections/intermediates/records

[Nicholas Alexander]

On Wed, Mar 13, 2019 at 2:23 PM J.C. Jones <j...@mozilla.com> wrote:

> Tom,
>
> Kinto provides the whole list of metadata to clients as soon as it syncs
> [1]. The metadata uses the Kinto attachment
> <https://github.com/Kinto/kinto-attachment> mechanism to store the
> DER-encoded certificate for separate download.
>
> Firefox maintains a "local field" boolean in the dataset to of whether a
> given metadata entry's certificate attachment has been downloaded or not,
> toggling as each one is pulled. Currently we don't deduplicate with the
> local NSS Cert DB, the inserts that are already there will fail and emit
> telemetry -- the amount of data saved didn't seem worth it for the
> experimental phase.
>

J.C. -- I don't think this answers Tom's question, but perhaps it does. In
that case I'll ask what I think is the same question:

How is the set of certificates that _might_ be pushed to clients
determined? In some way we must determine a set of relevant intermediate
certificates: how do we determine that set? Is it that the set of
intermediates for every CA that we trust is known?

Nick

[J.C. Jones]

Nicholas,

Mozilla's root program mandates all members disclose all intermediates via
the Common CA Database <https://ccadb.org/>. That database has enough
metadata to determine which CA certificates chain to roots in our program.
The CCADB exports a list on-demand <https://ccadb.org/resources> of all
intermediates that are disclosed and in our program (Note, the
Mozilla-speciifc one isn't linked there, contact me off-list if you want
access). As part of our CRLite server-side code, we're packaging up that
list of intermediates regularly and update Kinto. Obviously they don't
change quickly. :)

By policy, Firefox users should not encounter any undisclosed intermediate
CAs that are trusted via a root in our root program. In principal, we could
eventually use this functionality to affirm that.

Cheers!
J.C.



On Thu, Mar 14, 2019 at 8:26 AM Nicholas Alexander <nalex...@mozilla.com>
wrote:

[Tom Ritter]

On Thu, Mar 14, 2019 at 3:26 PM Nicholas Alexander
<nalex...@mozilla.com> wrote:
> J.C. -- I don't think this answers Tom's question, but perhaps it does. In that case I'll ask what I think is the same question:

Actually, what I was worried about was Mozilla being able to track
users based on what the client sends.

"I've got 100-200, send me some more"
"I've got 100-300, send me some more"
and so on

It sounds like The client says something more like "Send me 200-300",
"Send me 300-400". Which is not _that_ much different, but it
slightly better I think...

-tom

[J.C. Jones]

That's a good argument for us never "optimizing" it to avoid re-downloading
already-known certs. Just download the whole set once, everywhere - the
bandwidth savings are limited.

[Tom Ritter]

On Fri, Mar 15, 2019 at 4:47 PM J.C. Jones <j...@mozilla.com> wrote:
> That's a good argument for us never "optimizing" it to avoid re-downloading already-known certs. Just download the whole set once, everywhere - the bandwidth savings are limited.

Yes and No. As ekr pointed out to me offline, there are so many myriad
of ways that Mozilla can correlate you that trying to solve any
individual one without an overall survey and consensus-building effort
for future development isn't really worth the headache and time...

-tom