Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.

Dismiss

Incident Report : GoDaddy certificates with ROCA Fingerprint

376 views

Skip to first unread message

Daymion Reynolds

unread,

Oct 24, 2017, 1:08:50 PM10/24/17

to mozilla-dev-s...@lists.mozilla.org

Godaddy LLC first became aware of possible ROCA vulnerability exposure on Monday October 16th 2017 at 9:30am. The following are the steps we took for detection, revocation, and the permanent fix of certificate provisioning:

• Monday October 16th 2017 AZ, first became aware of the ROCA vulnerability. We downloaded and modified the open source detection tool to audit 100% of the non-revoked and non-expired certs we had issued.
• Early am Wednesday October 18th AZ we had our complete list of 7 certs with the ROCA defect. We verified the results and proceeded to start the revocation process. While cert revocation was in progress we started researching the long-term detection and prevention of the weak CSR vulnerability.
• Early am Wednesday October 18th Rob Stradling released a list of certs with the vulnerability. 2/7 we revoked were on the list. https://misissued.com/batch/28/
• Thursday October 19th by 2:02am AZ, we completed the 7 cert revocations. Revocations included customer outreach to advise the customer of the vulnerability.
• Thursday October 19th AZ, two CSRs were submitted for commonNames “scada2.emsglobal.net” & “scada.emsglobal.net” and were issued. Each request had used the vulnerable keys for CSR generation. We revoked the certs again on Thursday October 19th AZ. During this period, we reached out to the customer to educate them regarding the vulnerability and informing them they needed to generate a new keypair from an unimpacted device. Customer was unreachable. Friday October 20thAZ, another cert was issued for commonName “scada.emsglobal.net” using a CSR generated with a weak key. We then took measures to prevent future certs from being issued to the same common name and revoked the cert on October 20th 2017 AZ.
commonName crt.sh-link
scada.emsglobal.net https://crt.sh/?id=3084867

scada.emsglobal.net https://crt.sh/?id=238721704

scada.emsglobal.net https://crt.sh/?id=238721807

scada2.emsglobal.net https://crt.sh/?id=238720969

scada2.emsglobal.net https://crt.sh/?id=238721559

• Saturday October 21st 2017 AZ & Sunday October 22nd 2017 AZ, we scanned our cert store and identified 0 vulnerable certs.
• Monday October 23, 2017 AZ, we have deployed a permanent fix to prevent future CSRs generated using weak keys from being submitted. Post scanning of the environment concluded 0 certificates at risk.

Below is a complete list of certs under GoDaddy management impacted by this vulnerability.

Alias crt.sh-link
alarms.realtimeautomation.net https://crt.sh/?id=33966207

scada.emsglobal.net https://crt.sh/?id=3084867
https://crt.sh/?id=238721704
https://crt.sh/?id=238721807

www.essicorp-scada.com https://crt.sh/?id=238720405

marlboro.bonavistaenergy.com https://crt.sh/?id=238720743

scada2.emsglobal.net https://crt.sh/?id=238720969
https://crt.sh/?id=238721559

www.jointboardclearscada.com https://crt.sh/?id=238721242

*.forgenergy.com https://crt.sh/?id=238721435

Regards,
Daymion Reynolds
GoDaddy PKI

Alex Gaynor

unread,

Oct 27, 2017, 11:07:58 AM10/27/17

to Daymion Reynolds, mozilla-dev-s...@lists.mozilla.org

Thank you for writing this up.

Do any of the other CAs with trusted server certificates intend to publish
similar reports? (Based on CT logs that'd be Comodo, Symantec, and
GlobalSign).

Alex

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

Matthew Hardeman

unread,

Oct 27, 2017, 11:42:41 AM10/27/17

to mozilla-dev-s...@lists.mozilla.org

I can not help but notice that the host names of the certificates involved rather strongly suggest that a series of device or embedded server is creating these CSRs / utilizing these certificates.

As you mentioned, some users subsequently requested certs for the same keys already previously utilized.

The concentration of these certificates across host names that suggest they are embedded industrial control and monitoring systems, like SCADA, etc, points to specific equipment or applications that have a problematic security problem.

Are you aware of the particulars of what devices are involved? Are the customers aware of the vulnerability? Has the manufacturer been made aware?

It's certainly true that publicly trusted CAs can block issuance of certs over weak ROCA keys. However, this may just drive the problematic devices to self-signed certs or corporate PKI. Unless these devices can perform proper TLS, users should probably take caution to ensure these are not reachable from the public internet.

Thanks,

Matt

Ryan Sleevi

unread,

Oct 27, 2017, 12:14:35 PM10/27/17

to Daymion Reynolds, mozilla-dev-security-policy

Daymion,

Thanks for providing this detailed report. I want to especially thank you
for providing an actual timeline - so many CAs have unfortunately
misunderstood what a timeline means, or how to effectively communicate it.
Your timeline provides a useful description of pre-existing state, when the
issue or incident was introduced, when it was detected, what steps were
taken initially, when the issue was resolved, and what steps will be taken
in the future.

In looking at what the expectations of CAs are, and how well GoDaddy upheld
them, the specific view is that:
- With the disclosure of the ROCA vulnerability, private keys subject to it
are noted to have suffered a Key Compromise event (BRs 1.5.1, Section
1.6.1, "Key Compromise")
- CAs are required to revoke a certificate within 24 hours if "The CA
obtains evidence that the Subscriber's Private Key corresponding to the
Public Key in the Certificate has suffered a Key compromise" (BRs 1.5.1,
4.9.1.1, "Reasons for Revoking a Subscriber Certificate", Item 3)
- CAs are required to reject CSRs if the private key does not meet the
requirements set forth in Sections 6.1.5, 6.1.6, or if it has a known-weak
Private Key (BRs 1.5.1, Section 6.1.1.3, "Subscriber Key Pair Generation")

Looking at the timing, it looks like:
- ~36 hours to detecting certificates
- ~60 hours to revoke
- ~60 hours to set up initial CSR rejection
- ~1 week to setup full scanning/rejection

That said, the level of detail provided - and the many challenges a number
of folks encountered with the initial ROCA code (including seemingly some
obfuscation by the authors) - arguably establish that this is a reasonable
and understandable timeline. This further is in line with some of the
discussions proceeding in the CA/Browser Forum with respect to revocation
timelines and transparency. While the BRs presently provide only 24 hours
for CAs to fully implement revocation (and 0 hours to implement CSR
rejection), truly exceptional cases such as this are arguably justifiable.

I think that, with respect to CA incident reporting, the Mozilla Module
Peers should consider this closed. I defer to Gerv and Kathleen whether or
not the deviations in timeline warrant filing a bug to ensure tracking, but
I do not think this report should be seen as looking unfavorably on
GoDaddy. In particular, it's worth noting that a number of CAs -
Symantec/GeoTrust, GlobalSign, and Symantec/VeriSign still have unrevoked
certificates out there despite awareness of this issue.

Jeremy Rowley

unread,

Nov 1, 2017, 9:19:56 PM11/1/17

to Alex Gaynor, Daymion Reynolds, mozilla-dev-s...@lists.mozilla.org

Hey Alex - we intend to publish a report for the former Symantec certs. For now, here's what I know:

1) The scope was 15 TLS certs. We became aware of the certs through your posting.
2) We are revoking all 15 certs. I'm still waiting for their serial numbers. We kicked off the 24 hour period today so they should all be revoked tomorrow.

That's about all I know right now.
Jeremy

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+jeremy.rowley=digice...@lists.mozilla.org] On Behalf Of Alex Gaynor via dev-security-policy
Sent: Friday, October 27, 2017 9:08 AM
To: Daymion Reynolds <drey...@godaddy.com>
Cc: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Incident Report : GoDaddy certificates with ROCA Fingerprint

Thank you for writing this up.

Do any of the other CAs with trusted server certificates intend to publish similar reports? (Based on CT logs that'd be Comodo, Symantec, and GlobalSign).

Alex

On Tue, Oct 24, 2017 at 12:28 PM, Daymion Reynolds via dev-security-policy < dev-secur...@lists.mozilla.org> wrote:

> Godaddy LLC first became aware of possible ROCA vulnerability exposure
> on Monday October 16th 2017 at 9:30am. The following are the steps we
> took for detection, revocation, and the permanent fix of certificate provisioning:
>
> • Monday October 16th 2017 AZ, first became aware of the ROCA
> vulnerability. We downloaded and modified the open source detection
> tool to audit 100% of the non-revoked and non-expired certs we had issued.
> • Early am Wednesday October 18th AZ we had our complete list of 7
> certs with the ROCA defect. We verified the results and proceeded to
> start the revocation process. While cert revocation was in progress we
> started researching the long-term detection and prevention of the weak
> CSR vulnerability.
> • Early am Wednesday October 18th Rob Stradling released a list of
> certs with the vulnerability. 2/7 we revoked were on the list.

> https://clicktime.symantec.com/a/1/1SkDS7vkKe6aFPef3aMSvQFIofogSXtMjIDOMPSrNdU=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJD53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2At2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXXB-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNHK-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Fmisissued.com%2Fbatch%2F28%2F

> • Thursday October 19th by 2:02am AZ, we completed the 7 cert
> revocations. Revocations included customer outreach to advise the
> customer of the vulnerability.
> • Thursday October 19th AZ, two CSRs were submitted for commonNames “
> scada2.emsglobal.net” & “scada.emsglobal.net” and were issued. Each
> request had used the vulnerable keys for CSR generation. We revoked
> the certs again on Thursday October 19th AZ. During this period, we
> reached out to the customer to educate them regarding the
> vulnerability and informing them they needed to generate a new keypair from an unimpacted device.
> Customer was unreachable. Friday October 20thAZ, another cert was
> issued for commonName “scada.emsglobal.net” using a CSR generated with
> a weak key. We then took measures to prevent future certs from being
> issued to the same common name and revoked the cert on October 20th 2017 AZ.
> commonName crt.sh-link
> scada.emsglobal.net

> https://clicktime.symantec.com/a/1/4ypxaC37c0Q-AJ7bui52CmnZ0rpGYzh75RU
> ZYnpk23A=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJ
> D53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ
> 4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2A
> t2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXX
> B-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNH
> K-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7
> keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85
> zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Fcrt.sh%2F%3Fid%
> 3D3084867
>
> scada.emsglobal.net
> https://clicktime.symantec.com/a/1/OB-olazZASanecDv3efdwDvRV66LosOv7dP
> Crn-2tbc=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJ
> D53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ
> 4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2A
> t2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXX
> B-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNH
> K-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7
> keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85
> zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Fcrt.sh%2F%3Fid%
> 3D238721704
>
> scada.emsglobal.net
> https://clicktime.symantec.com/a/1/Ye0C0KwxJT7gG0xeS9X1OFkk2WwbbLhRhFQ
> yq-MGe3o=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJ
> D53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ
> 4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2A
> t2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXX
> B-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNH
> K-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7
> keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85
> zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Fcrt.sh%2F%3Fid%
> 3D238721807
>
> scada2.emsglobal.net
> https://clicktime.symantec.com/a/1/6NRvdXoMgB5jSRDjmk9juUJX8IQNeARCwe1
> POZfqqPw=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJ
> D53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ
> 4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2A
> t2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXX
> B-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNH
> K-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7
> keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85
> zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Fcrt.sh%2F%3Fid%
> 3D238720969
>
> scada2.emsglobal.net
> https://clicktime.symantec.com/a/1/xhHbR0IKzrZ8gR083W-b8u464POP3G3W7hG
> 7MSCM4o8=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJ
> D53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ
> 4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2A
> t2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXX
> B-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNH
> K-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7
> keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85
> zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Fcrt.sh%2F%3Fid%
> 3D238721559

>
> • Saturday October 21st 2017 AZ & Sunday October 22nd 2017 AZ, we
> scanned our cert store and identified 0 vulnerable certs.
> • Monday October 23, 2017 AZ, we have deployed a permanent fix to
> prevent future CSRs generated using weak keys from being submitted.
> Post scanning of the environment concluded 0 certificates at risk.
>
> Below is a complete list of certs under GoDaddy management impacted by
> this vulnerability.
>
> Alias crt.sh-link
> alarms.realtimeautomation.net

>
>
> Regards,
> Daymion Reynolds
> GoDaddy PKI
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org

> https://clicktime.symantec.com/a/1/yD3t6IcFjMFt6ppvUEqLi3GkL8qMZLUXFmq
> YrGxwTVI=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJ
> D53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ
> 4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2A
> t2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXX
> B-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNH
> K-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7
> keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85
> zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Flists.mozilla.o
> rg%2Flistinfo%2Fdev-security-policy

>
_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org

https://clicktime.symantec.com/a/1/yD3t6IcFjMFt6ppvUEqLi3GkL8qMZLUXFmqYrGxwTVI=?d=y7maDzQwn0t2gfiNBTRLLoxLptvalFLxhxxGV50FFf2HN_GpCO0GEQ5_rJD53axlha3VgyPx5e47idtKKh9Q430x5oQoja_2JjwYYimO70LL-IABmm8rLDDwsSe6D-SQ4vUvLFK8QmovkdoVYa5s4bx_lJw2M8RHGF6MxFEinJ-dFtEuoLiaF_FuBO7KEhoOnoqj2At2y3L-V1_T2U3QXqMynZvbpNH7wBgbuTN89gmguJAKE4Wff-cB1Q590BZYVEFmTUCwDBXXB-aCCKdMZU-CPbiC27t9PqIsHpBTeMTdqeYJIPkES4fzuq6TW8no6Bh0q9461T37F4JqNHK-9ybFxA8-HEacA0WU6u25efXCjiK0bzUgwVRxB9vTYtCoXtet39vQB7YyQRj0Rgyh8TU7keVb3WjzVERe0Pp3r7l3JmD6_4RyNRVIUiI8Kjf9WRn-N0rHOgbvv9p_hin3dr7wSFSS85zlq4g65z0_L1idMgpHVkqfaf33dUSsaghwqNn5&u=https%3A%2F%2Flists.mozilla.org%2Flistinfo%2Fdev-security-policy

0 new messages