GoDaddy Underscore Revocation Disclosure

[Joanna Fox]

GoDaddy received a certificate problem report on 1/29/2019 for 2 unrevoked unexpired certificates have underscores in the DNS name that did not meet the January 15th deadline for revocation. The certificates reported are as follows:
https://crt.sh/?opt=zlint&id=626981823
https://crt.sh/?opt=zlint&id=637047181

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Certificate Problem Reporting via email address supplied in CP/CPS on Tuesday, January 29, 2019 11:18:42 AM

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

1/29/2019 11:18:42 AM AZ time Problem Report received by GoDaddy
1/29/2019 3:55 PM AZ time Problem report was viewed by an agent and escalated appropriately.
1/29/2019 to 1/30/2019 Researched as to why these certificates were not caught in the initial data set for underscore revocation.
1/30/2019 Identified root cause as order of an operation problem where certificates could be CT logged but never be delivered to the certificate requester.
1/30/2019 23:36:32 UTC and 1/30/2019 23:35:55 UTC Certificates that were reported are revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We have prevented new certificates with underscores from being provisioned since November 8, 2018.

4. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?opt=zlint&id=626981823
https://crt.sh/?opt=zlint&id=637047181

5. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our initial effort to bring underscore certificates into compliance was dependent upon an ‘active’ flag in our database. In rare cases, due to an order of operation problem, non-active certificates were being logged to CT. These non-active certificates were fully vetted meeting the BR standards but failed to be delivered to the certificate requester due to a software defect.

6. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have prevented new certificates with underscores from being provisioned since November 8, 2018. These non-active certificates were revoked and we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks.

[Ryan Sleevi]

Thanks Joanna,

Just to make sure - this is
https://bugzilla.mozilla.org/show_bug.cgi?id=1524815 correct?

If so, this sounds remarkably similar to the root cause analysis that
Entrust shared, in https://bugzilla.mozilla.org/show_bug.cgi?id=1521520 .
Similar to that issue, please provide an update as to what steps are being
taken to ensure that future compliance objectives are met - that is, what
process changes have been made with respect to the data sources you use,
how tools will be reviewed that use those data sources, and what steps may
be taken to improve the quality and accuracy of those data sources (e.g.
disclosing active certificates to CT)

While Mozilla policy does not require or mandate the disclosure to CT,
those non-active certificates are still relevant to the BR expectations of
compliance, and should they be discovered (e.g. via auditors or if they
were to be shared outside GoDaddy), the BR non-compliance may appear
significantly worse to the community. Rather than not logging non-active
certificates, it may be more useful to approach this by ensuring your tools
consider both active and non-active certificates - that is, anything that
has been signed with the CA's private key.

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Wayne Thayer]

Perhaps more concerning, this sounds a lot like bug #1462844 in which
misissued certificates were reported that had not been found and revoked
despite GoDaddy having previously scanned their database for the issue.
GoDaddy never identified or described how they would remediate the cause of
that bug.

[Ryan Sleevi]

Good catch, thanks Wayne!

I also realized it sounds very similar to
https://bugzilla.mozilla.org/show_bug.cgi?id=1526154 from DigiCert, which
similarly included an overly-restrictive query.

It's also similar to Symantec's incident investigation in
https://wiki.mozilla.org/CA:Symantec_Issues#Issue_D:_Test_Certificate_Misissuance_.28April_2009_-_September_2015.29
,
which saw multiple revisions to the set of misissued certificates - as also
captured at
https://security.googleblog.com/2015/10/sustaining-digital-certificate-security.html
.

I highlight these to indicate that there is a systemic pattern of CAs
having trouble detecting the set of misissued certificates or certificates
that require action. When faced with systemic patterns, especially
well-publicized ones, I do think it's reasonable to expect a higher bar in
both the response and the remediation plans. In particular, each CA that
has such an incident should examine past patterns (at both their and other
CAs) and come up with comprehensive plans to address. Other CAs - those
that have not had such issues - should also consider implementing similar
best practices, as the goal of these reports is to highlight gaps and
develop and operationalize best practices. Failure to do so will be seen
very unfavorably - as the comment on
https://bugzilla.mozilla.org/show_bug.cgi?id=1462844#c9 explicitly called
out.

[Joanna Fox]

I agree on the surface this bug appears to be the same, but the root cause is a different. The issue for bug 1462844 was a specific status not counting as active when it was. To mitigate this issue, we updated the query to include the missing status. However, we are in the process of simplifying the data structures to simplify these types of queries.

For the underscore certificates, these were non-active, not even considered as provisioned since they were not delivered to a customer and not publicly used for any encryption. These certificates were effectively abandoned by our system.

[Santhan Raj]

On Friday, February 8, 2019 at 4:09:32 PM UTC-8, Joanna Fox wrote:
> I agree on the surface this bug appears to be the same, but the root cause is a different. The issue for bug 1462844 was a specific status not counting as active when it was. To mitigate this issue, we updated the query to include the missing status. However, we are in the process of simplifying the data structures to simplify these types of queries.
>
> For the underscore certificates, these were non-active, not even considered as provisioned since they were not delivered to a customer and not publicly used for any encryption. These certificates were effectively abandoned by our system.

Is the term "certificate" accurate in this case? Assuming you embed SCTs within the EE cert, what you have is technically a pre-cert that was abandoned (not meant to be submitted to CT). Right? I ask because both the cert you linked are pre-certs, and I understand signing a pre-cert is intent to issue and is treated the same way, but still wanted to clarify.

Or by non-active certificate, are you actually referring to a fully signed EE that was just not delivered to the customer?

Thanks,
Santhan

[Jakob Bohm]

And in either case, the only reasonable sequences of events within
expectations (and possibly within requirements) would have been:

Good Scenario A:
1. Pre-certificate is logged in CT.
2. Matching certificate is signed by CA key.
3. Signed certificate is logged in CT.
4. At this point the customer _might_ retrieve the certificate from CT
without knowledge of CA.
5. Thus certificate is in reality active, despite any records the CA
may have of not delivering it directly to customer.

Good Scenario B:
1. Pre-certificate is logged in CT.
2. CA decides (for any reason) not to actually sign the actual
certificate.
3. The serial number listed in the CT pre-certificate is formally
revoked in CRL and OCSP. This is done once the decision not to
sign is final.
4. If possible, label these revocations differently in CRL and OCSP
responses, to indicate to independent CT auditors that the EE was
never signed and therefore not logged to CT.

Scenario B should be very rare, as the process from pre-cert logging to
final cert logging should typically be automatic or occur within a
single root key ceremony. Basically, something unusual would have to
happen at the very last moment, such as an incoming report that a
relied-upon external system (Global DNS, Internet routing, reliable
identity database etc.) may have been compromised during the vetting.

In neither case would a CT-logged serial number be left in a
not-active but not-revoked state beyond a relevant procedural
delay.

As pointed out in other recent cases, CA software must allow
revoking a certificate without making it publicly valid first, in
case Scenario B happens.



Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Santhan Raj]

My assumption is,

1) Pre-cert is generated
2) something breaks
3) pre-cert is never submitted to CT (and is termed a non-active certificate)

The basis for my assumption is "In rare cases, due to an order of operation problem, non-active certificates were being logged to CT." and in another response "These non-active certificates were revoked and we are taking steps to prevent non-active certificates from being logged to CT within the next 2 weeks. "

There is no reason to "prevent non-active certificates from being logged to CT" if the pre-cert is already in CT. To me, it makes sense to say "prevent non-active pre-cert from being logged to CT".

Thanks,
Santhan