Incident report: Failure to verify authenticity for some partner requests

[Tim Hollebeek]

Hi everyone,

There was a bug in our OEM integration that led to a lapse in the
verification of authenticity of some OV certificate requests coming in
through the reseller/partner system.

As you know, BR 3.2.5 requires CAs to verify the authenticity of a request
for an OV certificate through a Reliable Method of Communication (RMOC).
Email can be a RMOC, but in these cases, the email address was a constructed
email address as in BR 3.2.2.4.4. Despite the fact that these addresses are
standardized in RFC 2142 or elsewhere, we do not believe this meets the
standard of "verified using a source other than the Applicant
Representative."

The issue was discovered by TBS Internet on Dec 30, 2018. Apologies for the
delay in reporting this. Because of the holidays, it took longer than we
wanted to collect the data we needed. We patched the system to prevent
continued use of constructed emails for authenticity verification early, but
getting the number of impacted orgs took a bit more time. We are using the
lessons learned to implement changes that will benefit overall user security
as we migrate the legacy Symantec practices and systems to DigiCert.

Here's the incident report:

1. How your CA first became aware of the problem (e.g. via a problem
report submitted to your Problem Reporting Mechanism, via a discussion in
mozilla.dev.security.policy, or via a Bugzilla bug), and the date.

Email from JP at TBS about the issue on Dec 30, 2017.

2. A timeline of the actions your CA took in response.

A. Dec 30, 2017 - Received report that indirect accounts did not require a
third-party source for authenticity checks. Constructed emails bled from the
domain verification approval list to the authenticity approval list.
B. Dec 30, 2017 - Investigation began. Shut off email verification of
authenticity.
C. Jan 3, 2017 - Call with JP to investigate what he was seeing and
confirmed that all indirect accounts were potentially impacted.
D. Jan 3, 2017 - Fixed issue where constructed emails were showing as a
permitted address for authenticity verification.
E. Jan 5, 2017 - Invalidated all indirect order's authenticity checks.
Started calling on verified numbers to confirm authenticity for impacted
accounts.
F. Jan 6, 2017 - Narrowed scope to only identify customers impacted (where
the validation staff used a constructed email rather than a verified
number).
G. Jan 10, 2017 - This disclosure.

Ongoing:
H. Reverification of all impacted accounts
I. Training of verification staff on permitted authenticity verification

3. Confirmation that your CA has stopped issuing TLS/SSL certificates
with the problem.

Confirmed. Email verification of authenticity remains disabled until we can
ensure additional safeguards.

4. A summary of the problematic certificates. For each problem: number of
certs, and the date the first and last certs with that problem were issued.

There are 3,437 orgs impacted, with a total of 5,067 certificates. The
certificates were issued between December 1st and December 30th.

5. The complete certificate data for the problematic certificates. The
recommended way to provide this is to ensure each certificate is logged to
CT and then list the fingerprints or crt.sh IDs, either in the report or as
an attached spreadsheet, with one list per distinct problem.

Will add to CT once we grab it all. I will provide a list of affected
certificates in a separate email (it's big, so it was getting this post
moderated).

6. Explanation about how and why the mistakes were made or bugs
introduced, and how they avoided detection until now.

In truth, it comes down to a short timeframe to implement the
Symantec-DigiCert system integration and properly train everyone we hired.
We are implementing lessons learned to correct this and improve security
overall as we migrate legacy Symantec practices and systems to DigiCert. In
this case, there are mitigating controls. For example, these are mostly
existing Symantec certs that are migrating to the DigiCert backend. The
verification by Symantec previously means that the number of potentially
problematic certs is pretty low. There's also a mitigating factor that we
did not use method 1 to confirm domain control. In each case, someone from
the approved constructed emails had to sign off on the certificate before
issuance. This is limited to OV certificates, meaning EV certificates were
not impacted. Despite the mitigating factors, we believe this is a
compliance issue, even though we believe the security risk is minimal.

7. List of steps your CA is taking to resolve the situation and ensure
such issuance will not be repeated in the future, accompanied with a
timeline of when your CA expects to accomplish these things.

A. We clarified in the system what is required for an authenticity check.
B. We removed email verification for authenticity checks until appropriate
new safeguards can be added.
C. We are re-validating authenticity for all potentially problematic
certificates and will revoke any that fail to validate (none have failed so
far)
D. We improved training for all validation specialists on the rules for
authenticity checking.
E. We are undergoing quarterly audits on the OEM system to ensure all checks
are in place (per the root transfer requirements from Google).

-Tim

[Wayne Thayer]

Thank you for the report Tim. I just created
https://bugzilla.mozilla.org/show_bug.cgi?id=1429639 to track this issue.
Please follow up in the bug and on this thread.

- Wayne

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
>

[Bruce]

On Wednesday, January 10, 2018 at 4:24:54 PM UTC-5, Tim Hollebeek wrote:

> As you know, BR 3.2.5 requires CAs to verify the authenticity of a request
> for an OV certificate through a Reliable Method of Communication (RMOC).
> Email can be a RMOC, but in these cases, the email address was a constructed
> email address as in BR 3.2.2.4.4. Despite the fact that these addresses are
> standardized in RFC 2142 or elsewhere, we do not believe this meets the
> standard of "verified using a source other than the Applicant
> Representative."

I agree. The intention for the constructed email from BR 3.2.2.4.4 was supposed to be to "confirm the Applicant's control over the FQDN" and not to perform the BR 3.2.5 requirement "to verify the authenticity of the Applicant Representative’s certificate request."

I also don't think a CA should use the information from 3.2.2.4.2 (Email, Fax, SMS, or Postal Mail) or 3.2.2.4.3 (phone number) to get the BR 3.2.5 authorization. The issue is the CA may end up using the same data to perform both 3.2.2.4 and 3.2.5 and will not mitigate the risk that the attacker controls the WHOIS data.

It would be more secure if the CA used two separate methods of communication for 3.2.2.4 and 3.2.5.

[Amus]

I updated the bugzilla thread (https://bugzilla.mozilla.org/show_bug.cgi?id=1429639). We ended up revoking 35 certs where we couldn't complete the authenticity check. I don't think these were actually issued to the wrong organization. Most of them are foreign, which means getting them on the phone when they already had their cert was pretty difficult.