On GitHub, Leaked Keys, and getting practical about revocation

[Matthew Hardeman]

Hi all,

I'm sure questions of certificates leaked to the public via GitHub and other file sharing / code sharing / deployment repository hosting and sharing sites have come up before, but last night I spent a couple of hours constructing various search criteria which I don't think were even especially clever, but still I was shocked and amazed at what I found:

At least 10 different Apple Development and Production environment certificates and key pairs. Of these minimum of 10 that I am counting, I validated that the certificate is within its validity period. Of these, I validated that the key I found matches the public key information in the certificate. Most of these certificates are TLS Client Authentication certificates which also have additional Apple proprietary extended key usages. These certificates are utilized for authenticating to the Apple Push Notification System. A couple of certificates were Apple Developer ID certificates appropriate for development and production environment deployment of executable code to Apple devices. (Those developer ID certificates I have reported to Apple for revocation.) There were more Apple Push authentication certificates than I cared to write up and send over.

I was shocked at the level of improper distribution and leaking of these keys and certificates.

Once in a while, the key was represented in encrypted form. In _every_ instance for which I found an encrypted key and dug further, either a piece of code, a configuration file, or sometimes a README-KEY-PASSWORD.txt (or similar) within the same repository successfully decrypted the encrypted key.

Additionally, I did find some TLS server certificates. There were many more that I did not bother to carefully analyze. Some were expired. One was a in-validity-window DV certificate issued by Let's Encrypt. Utilizing the certificate's private key, I was able to successfully use the Let's Encrypt ACME API to automatically request revocation of that certificate. Minutes later, I verified that OCSP responses for that certificate were, in fact, indicating that the certificate was revoked.

Of course, revocation even with a really nice OCSP responder system is not very effective today.

I have this suspicion that human nature dictates that eliminating these kinds of key material leaks is not even a goal worth having. Disappointment, I suspect, lives down that road.

Because live OCSP checks for certificates en-masse is not appealing to either the CAs or the browsers or the end users (consequences of network delay, reliability, etc.), revocation means very little pragmatically today.

This only reinforces the value and importance of either/both:

- Quite short lived certificates, automatically replaced and deployed, to reduce the risks associated with key compromise

and/or

- OCSP must-staple, which I believe is only pragmatically gated at the moment by a number of really poor server-side implementations of OCSP stapling. Servers must cache good responses. Servers must use those while awaiting a new good response further into the OCSP response validity period. Servers must validate the response and not server random garbage as if OCSP. Etc, etc. Ryan Sleevi's work documenting the core issues is clearly a step in the right direction.

Both NGINX's and Apache HTTPD's implementations of OCSP stapling are lacking in several material respects.

It would certainly be a significant undertaking, but I believe that organizations who are working to ensure a secure Web (and that reap the benefits of a secure and trustworthy web) could do much to achieve better deployment of OCSP stapling in relatively short time:

1. Direct contribution of funds / bounty to the core developers of each of those two web server projects for building a server-side OCSP stapling implementation which is trivial to configure and which meets the needs of an ideal implementation with respect to caching of good results, validating new responses to staple, scheduling the deployment of successful new responses or scheduling retries of fails, etc. Insist that the code be written with a view to maximal back-port capability for said implementations.

2. If such contributions are infeasible, funding competent external development of code which achieves the same as item 1 above.

3. High level engagement with major distributions. Tackle the technical and administrative hurdles to get these changes into the stable and development builds of all currently shipping versions of at least RedHat's, Canonical's, and Debian's distributions. Get these changes into the standard default version httpd and nginx updates.

4. Same as above but for common docker images, prevalent VM images, etc.

5. Ensure that the browsers are ready to support and enforce fail-hard on certificates which feature the OCSP must-staple extension.

6. Monitor progress in readiness and incentivize deployment of OCSP must-staple at the operations level. Maybe CAs are able to / encouraged to offer longer validity periods on OCSP must-staple certificates?

In terms of dollars and cents, I think the various giant organizations (Hi, Google, how are you today?) most interested and best positioned to benefit from a secure web would regard the costs of a coordinated effort such as this pretty insubstantial.

Through a little Google digging, I find numerous comments and references from well informed parties going back quite several years lamenting the poor state of support of OCSP stapling in both Apache HTTPD and NGINX. I'm well aware of the rising power that is Caddy, but it's not there yet. The whole ecosystem could be greatly helped by making the default shipping versions of those two daemons in the major distros be ideal OCSP-stapling ready.

Just a thought...

Matt Hardeman

[Hanno Böck]

On Wed, 21 Jun 2017 10:40:01 -0700 (PDT)
Matthew Hardeman via dev-security-policy

<dev-secur...@lists.mozilla.org> wrote:

> Through a little Google digging, I find numerous comments and
> references from well informed parties going back quite several years
> lamenting the poor state of support of OCSP stapling in both Apache
> HTTPD and NGINX. I'm well aware of the rising power that is Caddy,
> but it's not there yet. The whole ecosystem could be greatly helped
> by making the default shipping versions of those two daemons in the
> major distros be ideal OCSP-stapling ready.

There is some movement here for apache, see discussion over at the
apache dev list:
https://lists.apache.org/thread.html/1a61e9dfbd685c4102b097e8189bccb7d5da39bf9f32fcbe7407a760@%3Cdev.httpd.apache.org%3E

I'm slightly optimistic that we'll have a better stapling
implementation in apache soon.
Also CII is interested in funding efforts that improve the state of ocsp
stapling.

--
Hanno Böck
https://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42

[Jakob Bohm]

Please note that Apache and NGINX are by far not the only TLS servers
that will need working OCSP stapling code before must-staple can become
default or the only method checked by Browsers and other TLS clients.

What is needed is:

1. An alternative checking mechanism, such as the compact super-CRL
developed by some researchers earlier this year. This needs to be
deployed and used *before* turning off traditional CRL and OCSP
checking in relying party code, since leaving a relying party
application without any checking for revoked certificates is a pretty
obvious security hole.

2. Full OCSP stapling support in all TLS libraries, including the LTS
branches of OpenSSL, mbedTLS, NSS, Java, Android etc. Providing this
only in the "next great release that is incompatible with many
existing users" is a common malpractice among security library
vendors, just as there are still systems that refuse to support TLS
1.2 and ubiquitous SHA-256 signatures in todays world, simply because
the original system/library vendor refuses to backport the needed
changes.

Here is the status as I know it so far:
OpenSSL 1.0.x (LTS, backward compatible): Partial support for leaf
OCSP stapling, no code to actually help programs maintain fresh
OCSP responses to be stapled, no support for full chain OCSP
stapling.
OpenSSL 1.1.x (current, redesigned API): Maybe more implemented, not
sure.
OpenSSL clones (LibreSSL, BoringSSL etc.): Unknown status, probably no
better than OpenSSL itself.
mbedTLS: Apparently no OCSP stapling support
NSS as a client: Apparently at least some OCSP stapling support.
NSS as a server: Unknown status.
Java runtime (Oracle/OpenJDK): Unknown status
Android Java runtime/BouncyCastle: Unknown status
Microsoft SCHANNEL: Apparently at least full support for leaf OCSP
stapling, at least since the version shipped with Windows 6.0 .
The version shipped with Windows 6.0 doesn't support TLS 1.1/1.2.
Microsoft CryptoAPI (for non-TLS cert checks): Support for SHA-256
signatures is inconsistent between Windows 5.x and Windows 6.x
because different API identifiers are used for SHA-256 related
operations. Related code signing doesn't support SHA-256 signatures
until Windows 6.1 .

3. Once #2 is achieved, actual TLS servers and clients using those
libraries can begin to enable OCSP stapling.

4. Once #3 is achieved and deployed, then OCSP stapling might become
mandatory by default.

Note that the real world time to achieve #2..#4 is unfortunately
significant, hence my suggesting to use #1 as a faster solution.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Ryan Sleevi]

On Thu, Jun 22, 2017 at 1:59 PM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Please note that Apache and NGINX are by far not the only TLS servers
> that will need working OCSP stapling code before must-staple can become
> default or the only method checked by Browsers and other TLS clients.
>
> What is needed is:
>
> 1. An alternative checking mechanism, such as the compact super-CRL
> developed by some researchers earlier this year. This needs to be
> deployed and used *before* turning off traditional CRL and OCSP
> checking in relying party code, since leaving a relying party
> application without any checking for revoked certificates is a pretty
> obvious security hole.

There is reasonable and well-argued disagreement with you on this. Without
hardfail, it is not a security hole, and now mainstream client has ever
shipped hardfail, so it is not a reasonable requirement to introduce into
this discussion.

There's equally a host of pragmatic and practical concerns, which you can
find in the years of discussion on this topic and related solutions (as the
"super-CRL" you refer to is just the application of one particular
technology, which had previously been outlined years prior, and which
alternatives of similar constraints also existed even before that).

I suspect this may simply resolve to irreconcilable differences, as some
people may incorrectly choose to believe that soft-fail revocation provides
a defensible security boundary, or may believe that CAs revocation is a
trustworthy reason to deny access, both of which ample evidence and
arguments exist to show otherwise. I mention this largely to avoid
rehashing the same conversations that have been had, but if you are
unfamiliar with them, and are interested in learning of this other
perspective, I would be happy to provide them. I just figure as shorthand
that we disagree on this.

>
> 2. Full OCSP stapling support in all TLS libraries, including the LTS
> branches of OpenSSL, mbedTLS, NSS, Java, Android etc. Providing this
> only in the "next great release that is incompatible with many
> existing users" is a common malpractice among security library
> vendors, just as there are still systems that refuse to support TLS
> 1.2 and ubiquitous SHA-256 signatures in todays world, simply because
> the original system/library vendor refuses to backport the needed
> changes.

There is zero reason to introduce this as a dependency beforehand. Perhaps
your assumption is that it is unreasonable to require of the ecosystem what
the ecosystem does not support, but equally realize, the ecosystem will not
support it until it is required.

It is already true that a sufficient and meaningful majority support most
of the necessary work, and so the argument that ALL servers or libraries
must support is to ignore both the market realities and to place the
(unreasonable) perfect ahead of the achievable good.

Further, you conflate OCSP Multi-Staple as necessary, when in fact it is
entirely undesirable from a performance perspective and largely unnecessary
from a deployment scenario, given the existence of CRLSets, CDLs,
OneCRL-et-al.

>
> 3. Once #2 is achieved, actual TLS servers and clients using those
> libraries can begin to enable OCSP stapling.
>
> 4. Once #3 is achieved and deployed, then OCSP stapling might become
> mandatory by default.

This is to completely upend the only action that has been seen to
meaningfully improve the ecosystem, which is the mandate that thus spurs
implementation or innovation.

On a more pragmatic level, there is nothing wrong with saying "The only
revocation supported will be stapling and OneCRL". There is no need to go
above and beyond this, because collectively, this achieves the goal of
providing a compelling revocation story.

The disconnect that results in proposals like yours is that they presume
revocation is for the benefit of the relying party, as opposed to being for
the benefit of the site operator.

A site operator cares about revocation for cases of key compromise and
impersonation. A relying party may care about revocation for reasons like
misrepresentation (which is not, despite some views contrary, an accepted
concern of the Mozilla policies - c.f. malware and phishing), apathetic
server compromise (that is, they did not enable stapling. However, the root
cause/risk is the apathy, for which revocation does not fix), or should the
user want to deny themselves access to a site (which no user does).

If we focus on stapling, the position is that it is not necessary for the
browser to protect the user from servers' apathy (in not enabling
stapling), or from CAs' capricious opinions about certificates (which the
so-called supercrls try to enable, as a business model), but to allow
servers to protect themselves. There is similarly no concern given to CAs
that want to use OCSP or CRLs to "rent a cert" (as some tried to in the
past), because that's hostile to servers and the user security experience.

For these reasons, and more, it's merely sufficient to OFFER sites the
ability to use stapling and OneCRL. No other revocation methods are needed
or justified. No ecosystem dependencies are introduced - if your server
software doesn't support it, change servers, don't hold the ecosystem back.

The introduction of mandatory stapling only applies if you trust CAs enough
to issue revocation status accurately (empirically not true), you wish to
cede control of your user experience to a third party (no browser does), or
you plan to use the stapling in conjunction with application-specific risk
evaluations.

[Jakob Bohm]

On 22/06/2017 15:02, Ryan Sleevi wrote:
> On Thu, Jun 22, 2017 at 1:59 PM Jakob Bohm via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>

> (Snip long repeat of the same opinion)

You seem to argue:

- Because the recent research on efficient central CRL distribution was
based on a novel optimization of a previously inefficient algorithm,
then it is nothing new and should be ignored.

- Operating a central CRL distribution service is a suspect commercial
enterprise with a suspect business model, not a service to the
community.

- OCSP stapling of intermediary certificates is inefficient because we
should all just use central CRL distribution in a form where not all
revocations are included.

- Because most/all browsers contain security holes in their revocation
checking, making those holes bigger is not a problem.

- Revocation by the issuing CA is not trustworthy, thus nothing is,
therefore everybody should just trust compromised keys as if they were
not compromised.

- An attacker with a stolen/compromised key not doing OCSP stapling is
the fault of the legitimate key holder.

- Forcing people to use OCSP stapling will magically cause software that
allows this to spring into existence overnight. If this doesn't happen
it is the fault of the server operator, not because the demand was
premature.

[Ryan Sleevi]

On Thu, Jun 22, 2017 at 3:53 PM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 22/06/2017 15:02, Ryan Sleevi wrote:

> > On Thu, Jun 22, 2017 at 1:59 PM Jakob Bohm via dev-security-policy <
> > dev-secur...@lists.mozilla.org> wrote:
> >

> > (Snip long repeat of the same opinion)
>
> You seem to argue:

Your summary is neither accurate nor supported by what I said. However, as
your reply doesn't seek to learn or understand, and rather chooses to
belittle and misrepresent, nor does it provide any new or useful
information to refute the generally accepted conclusions you appear to
disagree with, it does seem that we should simply agree to disagree.

If you should have interest in trying to provide persuasive arguments,
factually supported, in a respectful way, perhaps there is an opportunity
to learn from eachother, but as this is otherwise an unnecessarily
combative, overly reducative, and textually unsupported strawman, it might
be best if you were to take a break from this discussion.