Let's Encrypt Blocklist Incident, November 21 2016

[jo...@letsencrypt.org]

Between 11:30am and 4pm Pacific on November 21, 2016, a problem with the Let’s Encrypt issuance blocklist was identified, confirmed, and fixed.

The issue was initially identified by a Let’s Encrypt operations engineer during routine maintenance. A script is used to assemble a final blocklist configuration from a set of input files. The engineer was adding a suffix to the blocklist and noticed that it wasn’t being propagated to the final blocklist configuration. Further investigation confirmed a bug in the script - it incorrectly and silently failed to process a small number of blocklist entries based on a formatting characteristic. The bug has been fixed and we are reviewing policy around the code in question. Testing for the code will be improved.

While a fix was being developed, Let’s Encrypt staff worked to identify all blocks that had failed to propagate as well as any certificates that were issued for those domains. The following certificates were found to have been mis-issued by policy, though there is no sign that they were used maliciously and domain control was properly demonstrated via DV validation.

gov.ir
https://crt.sh/?id=49145557 (Revoked)
https://crt.sh/?id=17321835 (Expired)
https://crt.sh/?id=17320010 (Expired)

gov.sy
https://crt.sh/?id=24753847 (Expired)

mil
https://crt.sh/?id=31920262 (Revoked)
https://crt.sh/?id=29886368 (Revoked)
https://crt.sh/?id=52210328 (Revoked)
https://crt.sh/?id=51226007 (Revoked)
https://crt.sh/?id=48632604 (Revoked)
https://crt.sh/?id=47382849 (Revoked)
https://crt.sh/?id=47464047 (Revoked)
https://crt.sh/?id=43269410 (Revoked)
https://crt.sh/?id=43268871 (Revoked)
https://crt.sh/?id=40478677 (Revoked)
https://crt.sh/?id=36321880 (Revoked)
https://crt.sh/?id=30291839 (Revoked)
https://crt.sh/?id=25207594 (Expired)

Issuance to gov.ir and gov.sy is not allowed as these entities are sanctioned by the U.S. government and we are a U.S.-based organization. Issuance to .mil is not allowed due to contractual obligations that are reflected in our Certification Practice Statement.

All unexpired certificates have been revoked. Account contacts were notified.

[Kathleen Wilson]

On Tuesday, November 22, 2016 at 12:16:43 PM UTC-8, jo...@letsencrypt.org wrote:
> Between 11:30am and 4pm Pacific on November 21, 2016, a problem with
> the Let’s Encrypt issuance blocklist was identified, confirmed, and fixed.
>

> <snip>

> The following certificates were found to have been mis-issued by policy,
> though there is no sign that they were used maliciously and domain control
> was properly demonstrated via DV validation.
>

I have filed https://bugzilla.mozilla.org/show_bug.cgi?id=1319609 to track Mozilla's response to this incident report.

Kathleen

[Gervase Markham]

On 22/11/16 20:16, jo...@letsencrypt.org wrote:
> organization. Issuance to .mil is not allowed due to contractual
> obligations that are reflected in our Certification Practice
> Statement.

I have just been investigating this issue, as documented in the bug
Kathleen links to. Mozilla policy requires that certificates issued in
contravention of a CA's CP/CPS should be revoked, which LE have done.
Other than that, Mozilla policy does not directly require (somewhat to
my surprise) that a CA operate in accordance with its CP and CPS. We
require this indirectly because the audits that we require, require it.

So: should Mozilla's policy directly require that CAs operate in
accordance with the appropriate CP/CPS for the root(s) in our store? I
can see both pros and cons to directly mandating this.

Gerv

[Myers, Kenneth (10421)]

This is one of the issues with a WebTrust audit in that WebTrust Auditors may not look at a CP/CPS depending on the management assertion. The trust in PKI is based on documented procedures so to not operate against a CP/CPS degrades the trust in PKI. The US Federal PKI have run into a similar issue where trust in Federal PKI is based on assurance strength of the certificate policies in a CP/CPS. The audit must verify a CA is following its operational practices to maintain that trust. This model only works if the validation is by a certificate policy and not simple path validation.

If this was added to the Mozilla CP, how would it be enforced and verified? The WebTrust letter would say it explicitly?

What are your pro/cons?

Date: Wed, 23 Nov 2016 10:47:06 +0000
From: Gervase Markham <ge...@mozilla.org>
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Let's Encrypt Blocklist Incident, November 21 2016
Message-ID: <XuadnTBSnNk37qjF...@mozilla.org>
Content-Type: text/plain; charset=utf-8

NOTICE: Protiviti is a global consulting and internal audit firm composed of experts specializing in risk and advisory services. Protiviti is not licensed or registered as a public accounting firm and does not issue opinions on financial statements or offer attestation services. This electronic mail message is intended exclusively for the individual or entity to which it is addressed. This message, together with any attachment, may contain confidential and privileged information. Any views, opinions or conclusions expressed in this message are those of the individual sender and do not necessarily reflect the views of Protiviti Inc. or its affiliates. Any unauthorized review, use, printing, copying, retention, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email message to the sender and delete all copies of this message. Thank you.

[cda]

On Tuesday, November 22, 2016 at 9:16:43 PM UTC+1, jo...@letsencrypt.org wrote:
> Issuance to gov.ir and gov.sy is not allowed as these entities are sanctioned by the U.S. government and we are a U.S.-based organization. Issuance to .mil is not allowed due to contractual obligations that are reflected in our Certification Practice Statement.

This is perhaps not the place to broaden the argument on Let's Encrypt's compliance polices, but to clarify this where it arises (and of course, I am neither a lawyer nor providing legal advice). Iranian General License D-1 paragraph (a)(6) authorizes "publicly available, no cost services and software to the Government of Iran." That applies where the software or service falls under a subset of the G.L. Annex, of which the eleventh item covers SSL certificates. There are limitations to this related to blocked or designated entities, but that's not only a problem related to governmental entities. So on face value, Let's Encrypt need not restrict use by .gov.ir domains if it would otherwise like to offer its services.

I take it as a positive sign that the government of Iran would use Let's Encrypt as a CA, and further promote its use in a country that is especially in need of the initiative due to sanctions. Neat, unusual incident.

[jo...@letsencrypt.org]

I'll provide a bit of context here, but if you'd like to provide input on something like this we should probably take the discussion somewhere else. Our community forums are probably a good place, in the Issuance Policy category.

The legal issues here are complex, and the landscape changes frequently. For example - we used to block issuance to Cuban government domains but we stopped doing that in early 2016 based on relevant changes in law/regulations.

Generally speaking our internal policy is to not issue to any person or entity on the U.S. Treasury Department's Specially Designated Nationals (SDN) list. There are some other relevant laws and regulations that we take into account, complicating that otherwise straightforward policy.

[Gervase Markham]

On 23/11/16 17:12, Myers, Kenneth (10421) wrote:
> What are your pro/cons?

The pro is that we would then be stating explicitly what everyone
presumes to be already the case, and that if it did come to our
attention that there was a violation, we would have grounds for complaint.

The con is that one could argue it's not wise to make a requirement
unless there is some reasonable detection and enforcement mechanism.
Whether that is true in this case is a debatable question. Some issues
are easier to detect than others.

Gerv

[Jakob Bohm]

One could argue that it would, in some ways, be one of the easier ones
to both enforce and comply with (given that the entity needing to
comply is actually writing the rules to be enforced).

It would also form a clear cut basis for the existing requirement to
supply the CP and CPS documents in a language understood by the Mozilla
organization.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Gervase Markham]

On 23/11/16 10:47, Gervase Markham wrote:
> So: should Mozilla's policy directly require that CAs operate in
> accordance with the appropriate CP/CPS for the root(s) in our store? I
> can see both pros and cons to directly mandating this.

https://github.com/mozilla/pkipolicy/issues/43 .

Gerv