info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Google Trust Services and EJBCA serial number behavior
1,122 views
Skip to first unread message
Ryan Hurst
Mar 1, 2019, 7:43:04 PM3/1/19
to mozilla-dev-s...@lists.mozilla.org
Dear m.d.s.p,
We at Google Trust Services have been following the thread discussing Dark Matter’s root inclusion request. In particular the elements of the thread that discuss the EJBCA serial number generation logic stood out to us.
This is because we use EJBCA for some of our own CAs. This element of the thread spurred us to review how our EJBCA based CAs were generating serial numbers. As a result of this review we determined our legacy EJBCA CAs were exhibiting the same behavior.
Though we believe this not to represent a material security issue and we believe that this issue is systemic given it is a result of behavior of the most common CA software in use in the WebPKI, we are actively working on a post mortem and are evaluating if and how to replace the affected certificates.
It is noteworthy that the associated EJBCA CAs have been patched and any new certificates will not have this issue. Additionally these CAs were already actively being deprecated for a new generation of EJBCA and a bespoke CA code base that do not exhibit this behavior.
We will follow up with a post mortem when our investigation is complete.
Ryan Hurst
Product Manager
Google Trust Services
We at Google Trust Services have been following the thread discussing Dark Matter’s root inclusion request. In particular the elements of the thread that discuss the EJBCA serial number generation logic stood out to us.
This is because we use EJBCA for some of our own CAs. This element of the thread spurred us to review how our EJBCA based CAs were generating serial numbers. As a result of this review we determined our legacy EJBCA CAs were exhibiting the same behavior.
Though we believe this not to represent a material security issue and we believe that this issue is systemic given it is a result of behavior of the most common CA software in use in the WebPKI, we are actively working on a post mortem and are evaluating if and how to replace the affected certificates.
It is noteworthy that the associated EJBCA CAs have been patched and any new certificates will not have this issue. Additionally these CAs were already actively being deprecated for a new generation of EJBCA and a bespoke CA code base that do not exhibit this behavior.
We will follow up with a post mortem when our investigation is complete.
Ryan Hurst
Product Manager
Google Trust Services
Ryan Hurst
Mar 5, 2019, 1:47:11 PM3/5/19
to mozilla-dev-s...@lists.mozilla.org
Dear m.d.s.p,
We wanted to follow-up to this thread and give an update.
We have decided to replace and revoke the certificates with 63 bit serial numbers, so far we have finished about 95% of the affected certificates.
We are actively working with the remaining subscribers to replace their certificates as soon as possible without creating a service disruption. We have made the decision to work with subscribers to enable a smooth transition prior to revocation since the issue in question does not reflect a material security issue.
We will share more information as we have it and will publish a complete post mortem once the associated response is complete.
We wanted to follow-up to this thread and give an update.
We have decided to replace and revoke the certificates with 63 bit serial numbers, so far we have finished about 95% of the affected certificates.
We are actively working with the remaining subscribers to replace their certificates as soon as possible without creating a service disruption. We have made the decision to work with subscribers to enable a smooth transition prior to revocation since the issue in question does not reflect a material security issue.
We will share more information as we have it and will publish a complete post mortem once the associated response is complete.
Ryan Sleevi
Mar 5, 2019, 2:46:46 PM3/5/19
to Ryan Hurst, mozilla-dev-security-policy
Please review the recently updated [1] guidance regarding revocation and
incident response [2]. Unfortunately, I don't think messages like this meet
the bar of expectations, which were recently discussed at length regarding
DigiCert and underscore characters [3]. The consequence of this is to
highlight that CAs making statements that the matter "does not reflect a
material security issue" is not a sufficient or accepted reply.
If my understanding is correct, and that we are approaching or have passed
the 5 day mark at which Google Trust Services was made aware of this, then
the appropriate next step is to file an incident report, thus making the
total matter two incident reports - one incident report regarding the
serial entropy, for which your initial message begins evaluating, and
another for a failure to revoke according to the BR allotted timeframe.
If I've misunderstood the timing, then this may simply serve as a reminder
about the level of expectations regarding revocation and BR violations.
However, based on this and your previous reply, it does sound that this
represents an incident regarding the failure to revoke 5% of those
certificates, and an incident report is expected regarding that, including
the timeline and plans for remediation. This helps ensure a consistent
evaluation regarding the risks, as the CA represents them, and helps
demonstrates CA's clear and hard commitments regarding remediation timing.
Please also note that, as per [2], CAs are encouraged to file an incident
report much sooner. On the particular topic of revocation, CAs are
encouraged to file a report "immediately", preferably before the
BR-mandated deadline has been exceeded. You may then provide additional
details and investigation notes, as information becomes available and as
questions are raised.
[1]
https://groups.google.com/d/msg/mozilla.dev.security.policy/HdirGOy6TJI/oIHKXeSuCAAJ
[2] https://wiki.mozilla.org/CA/Responding_To_An_Incident
[3]
https://groups.google.com/d/msg/mozilla.dev.security.policy/0oy4uTEVnus/pnywuWbmBwAJ
Ryan Hurst
Mar 5, 2019, 8:08:15 PM3/5/19
to mozilla-dev-s...@lists.mozilla.org
Sleevi,
Thanks you for the links to both the reporting requirements and the underscore issue with DigiCert.
Regarding the statement about the severity of the issue, it was not intended to diminish the non-compliance. Instead it was an attempt to frame the issue with sufficient context to help others who follow this thread answer the question of impact on the community.
As for the incident response, we have been making every effort to gather complete and accurate information to enable us to provide a useful and actionable incident report. This has unfortunately taken longer than we had hoped. We now have that report ready and you can find it below.
You are right that some affected certificates could not be revoked within the 5 day requirement. In the last section of the report you'll find more information about the reasons for this along with our plan for revoking the remaining certificates.
Ryan Hurst
Google Trust Services
Product Manager
Summary
---
Some certificates issued by GTS utilize EJBCA and as a result had serial numbers with an effective entropy of 63 bits. These serial numbers were created from a 64 bit CSPRNG output and were believed to be in compliance with Section 7.1 of the Baseline Requirements. Upon closer investigation we learned that EJBCA’s logic for serial number generation only selected output numbers having a leading 0 bit, which reduced their effective entropy to 63 bits.
Though GTS agree that the issuance of the certificates based on the above behavior qualifies as missisuance, we also believe that this issue does not represent a material security risk to the community.
To ensure that all of our certificates comply with the community’s interpretation of the Baseline Requirements (BR) we have updated the associated EJBCA CAs to mitigate the problematic behaviour.
At this time approximately 95% of the affected certificates have been replaced and revoked. The remaining certificates expire over the next 3 months.
We are actively working with the subscribers of these remaining certificates to facilitate a replacement with the goal of minimizing disruption of services. Should this not be possible, we will revoke these certificates no later than 2019-03-31..
Certificates issued from non-EJBCA CAs have been checked and are not affected.
Incident Report
---
1. How your CA first became aware of the problem
We have been following the thread discussing Dark Matter’s root inclusion request. When concerns regarding the EJBCA serial number generation logic were raised, we analyzed the behaviour of our EJBCA installations and found that they were affected as well.
2. A timeline of the actions your CA took in response.
2019-02-22 - A thread on m.d.s.p. mentions serial entropy issue of Dark Matter certificates.
2019-02-26 - GTS begins reviewing the serial number generation behaviour of its CAs.
2019-02-27 - A third-party reports that serial numbers in all certificates issued from a specific GTS CA have a leading bit of 0 and suggests that we may have the same issue as Dark Matter.
2019-02-27 - GTS requests clarification from PrimeKey. It is confirmed that EJBCA serial generation logic causes the issue and that in order to create compliant serial numbers, the logic has to be replaced.
2019-02-27 - The associated CAs used an earlier version of EJBCA where the serial number logic was not configurable. As a result code from a newer version of EJBCA that supports configurable serial number length is backported and configured to use 16 byte serials.
2019-02-28 - The backported code is deployed to production.
2019-02-28 - Ongoing discussion on m.d.s.p. revolves around interpretation of Section 7.1 BR. A consensus emerges that the affected certificates must be considered to have been misissued.
2019-02-28 - We inventory the number of certificates issued since Section 7.1 BR went into effect in September 2016, the number that were currently valid as well as their validity period. The results are provided in the section on remediation actions below.
2019-03-01 - GTS decides to replace and revoke all affected certificates. Customers are contacted to work out revocation plans.
2019-03-01 - Issuance of replacement certificates begins.
2019-03-02 - A first notification is posted to m.d.s.p
2019-03-04 - Certificate revocation begins.
2019-03-05 - An update on progress is posted to m.d.s.p
2019-03-05 - This post mortem is posted to m.d.s.p
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.
GTS has stopped using the incorrect serial number generation logic. As of 2019-02-28, all GTS certificates issued from its EJBCA CAs have serial numbers with at least 64 bits of entropy.
4. A summary of the problematic certificates.
All certificates issued from GIAG3 (https://crt.sh/?id=109354897, https://crt.sh/?id=158511650) between 2016-09-30 and 2019-02-28 were affected.
5. The complete certificate data for the problematic certificates.
Given the large number of certificates (> 100k) they are not enumerated here. All certificates issued from the CAs mentioned above are affected. The CA certificates themselves are not affected. All affected certs were logged to multiple CT logs.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
GTS relied on the affected EJBCA CA to generate 64 bit long certificate serial numbers as per its configuration and relied on EJBCA to perform this serial number generation in a BR compliant manner.
Since the serial numbers had the expected length, the leading bit of 0 was not discovered as limiting the serial number space.
Regular internal audits examined the technical profile of samples of issued certificates but did not detect the issue, because the audit was performed on a certificate-by-certificate basis. A comparison of serial numbers across a larger certificate population would have been required to identify it. Such tests were not implemented for the affected EJBCA CA because it is a legacy system scheduled for decommissioning later this year.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future.
The issue only affects certificates from one of our legacy EJBCA CAs. Most of the GTS issuance volume has previously been migrated to a bespoke CA that is not affected.
Since the affected CA was patched with updated logic, it is issuing with 16 byte serials. At the end of August 2019, it will be decommissioned.
An inventory of the affected certificates provided the following:
Total number issued since September 2016 - 100,836
Total valid as of 2019-02-28 - 7,171
Expiring within 90 days - 7,137
Revocation Schedule
---
Revocation was performed with the objective of meeting the timeline prescribed for cases that fall under the second paragraph of Section 4.9.1 BR (5 days).
by 2019-03-04 : 6,199 (86.5%)
by 2019-03-05 : 575 (8%)
The remaining 397 certificates (5.5%) will expire or be revoked no later than 2019-03-31.
For 86.5% of the affected certificates the BR revocation timeline could be met.
954 certificates could not be replaced within the five day target without causing outages or other business issues. Revoking these certificates without providing adequate time for the associated subscribers to replace the certificates certificates without providing adequate time for the associated subscribers to replace the certificates would have caused significant business damage to the respective subscribers. Our goal being to revoke all certificates as quickly as possible while minimizing disruption on the subscribers and the users of their services.
We are actively working with remaining subscribers who have affected certs which have not been revoked to replace their certificates in the shortest time window possible. The 397 remaining certificates will expire or be revoked by 2019-03-31.
Thanks you for the links to both the reporting requirements and the underscore issue with DigiCert.
Regarding the statement about the severity of the issue, it was not intended to diminish the non-compliance. Instead it was an attempt to frame the issue with sufficient context to help others who follow this thread answer the question of impact on the community.
As for the incident response, we have been making every effort to gather complete and accurate information to enable us to provide a useful and actionable incident report. This has unfortunately taken longer than we had hoped. We now have that report ready and you can find it below.
You are right that some affected certificates could not be revoked within the 5 day requirement. In the last section of the report you'll find more information about the reasons for this along with our plan for revoking the remaining certificates.
Ryan Hurst
Google Trust Services
Product Manager
Summary
---
Some certificates issued by GTS utilize EJBCA and as a result had serial numbers with an effective entropy of 63 bits. These serial numbers were created from a 64 bit CSPRNG output and were believed to be in compliance with Section 7.1 of the Baseline Requirements. Upon closer investigation we learned that EJBCA’s logic for serial number generation only selected output numbers having a leading 0 bit, which reduced their effective entropy to 63 bits.
Though GTS agree that the issuance of the certificates based on the above behavior qualifies as missisuance, we also believe that this issue does not represent a material security risk to the community.
To ensure that all of our certificates comply with the community’s interpretation of the Baseline Requirements (BR) we have updated the associated EJBCA CAs to mitigate the problematic behaviour.
At this time approximately 95% of the affected certificates have been replaced and revoked. The remaining certificates expire over the next 3 months.
We are actively working with the subscribers of these remaining certificates to facilitate a replacement with the goal of minimizing disruption of services. Should this not be possible, we will revoke these certificates no later than 2019-03-31..
Certificates issued from non-EJBCA CAs have been checked and are not affected.
Incident Report
---
1. How your CA first became aware of the problem
We have been following the thread discussing Dark Matter’s root inclusion request. When concerns regarding the EJBCA serial number generation logic were raised, we analyzed the behaviour of our EJBCA installations and found that they were affected as well.
2. A timeline of the actions your CA took in response.
2019-02-22 - A thread on m.d.s.p. mentions serial entropy issue of Dark Matter certificates.
2019-02-26 - GTS begins reviewing the serial number generation behaviour of its CAs.
2019-02-27 - A third-party reports that serial numbers in all certificates issued from a specific GTS CA have a leading bit of 0 and suggests that we may have the same issue as Dark Matter.
2019-02-27 - GTS requests clarification from PrimeKey. It is confirmed that EJBCA serial generation logic causes the issue and that in order to create compliant serial numbers, the logic has to be replaced.
2019-02-27 - The associated CAs used an earlier version of EJBCA where the serial number logic was not configurable. As a result code from a newer version of EJBCA that supports configurable serial number length is backported and configured to use 16 byte serials.
2019-02-28 - The backported code is deployed to production.
2019-02-28 - Ongoing discussion on m.d.s.p. revolves around interpretation of Section 7.1 BR. A consensus emerges that the affected certificates must be considered to have been misissued.
2019-02-28 - We inventory the number of certificates issued since Section 7.1 BR went into effect in September 2016, the number that were currently valid as well as their validity period. The results are provided in the section on remediation actions below.
2019-03-01 - GTS decides to replace and revoke all affected certificates. Customers are contacted to work out revocation plans.
2019-03-01 - Issuance of replacement certificates begins.
2019-03-02 - A first notification is posted to m.d.s.p
2019-03-04 - Certificate revocation begins.
2019-03-05 - An update on progress is posted to m.d.s.p
2019-03-05 - This post mortem is posted to m.d.s.p
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.
GTS has stopped using the incorrect serial number generation logic. As of 2019-02-28, all GTS certificates issued from its EJBCA CAs have serial numbers with at least 64 bits of entropy.
4. A summary of the problematic certificates.
All certificates issued from GIAG3 (https://crt.sh/?id=109354897, https://crt.sh/?id=158511650) between 2016-09-30 and 2019-02-28 were affected.
5. The complete certificate data for the problematic certificates.
Given the large number of certificates (> 100k) they are not enumerated here. All certificates issued from the CAs mentioned above are affected. The CA certificates themselves are not affected. All affected certs were logged to multiple CT logs.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
GTS relied on the affected EJBCA CA to generate 64 bit long certificate serial numbers as per its configuration and relied on EJBCA to perform this serial number generation in a BR compliant manner.
Since the serial numbers had the expected length, the leading bit of 0 was not discovered as limiting the serial number space.
Regular internal audits examined the technical profile of samples of issued certificates but did not detect the issue, because the audit was performed on a certificate-by-certificate basis. A comparison of serial numbers across a larger certificate population would have been required to identify it. Such tests were not implemented for the affected EJBCA CA because it is a legacy system scheduled for decommissioning later this year.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future.
The issue only affects certificates from one of our legacy EJBCA CAs. Most of the GTS issuance volume has previously been migrated to a bespoke CA that is not affected.
Since the affected CA was patched with updated logic, it is issuing with 16 byte serials. At the end of August 2019, it will be decommissioned.
An inventory of the affected certificates provided the following:
Total number issued since September 2016 - 100,836
Total valid as of 2019-02-28 - 7,171
Expiring within 90 days - 7,137
Revocation Schedule
---
Revocation was performed with the objective of meeting the timeline prescribed for cases that fall under the second paragraph of Section 4.9.1 BR (5 days).
by 2019-03-04 : 6,199 (86.5%)
by 2019-03-05 : 575 (8%)
The remaining 397 certificates (5.5%) will expire or be revoked no later than 2019-03-31.
For 86.5% of the affected certificates the BR revocation timeline could be met.
954 certificates could not be replaced within the five day target without causing outages or other business issues. Revoking these certificates without providing adequate time for the associated subscribers to replace the certificates certificates without providing adequate time for the associated subscribers to replace the certificates would have caused significant business damage to the respective subscribers. Our goal being to revoke all certificates as quickly as possible while minimizing disruption on the subscribers and the users of their services.
We are actively working with remaining subscribers who have affected certs which have not been revoked to replace their certificates in the shortest time window possible. The 397 remaining certificates will expire or be revoked by 2019-03-31.
Ryan Hurst
Mar 5, 2019, 8:16:52 PM3/5/19
to mozilla-dev-s...@lists.mozilla.org
I have created a bug to track this issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1532842
Ryan Sleevi
Mar 5, 2019, 8:28:39 PM3/5/19
to Ryan Hurst, mozilla-dev-s...@lists.mozilla.org
Ryan,
Thanks for providing the update. One area that I do need to push back on is
the disclosure of the 100K certificates mentioned.
As demonstrated through past CA distrust discussions and whose need is
evidenced by past incident reports, one of the purposes of having CAs
disclose the affected certificates is to provide assurance to the community
that the CA has completed a rigorous investigation of the issue.
While I realize that, as a systemic issue, it’s easy to highlight that it
is 100% of issuance from a given CA, in the goal of ensuring consistent
incident reporting, please provide the following necessary and required
information:
- The full list of certificates affected (past CAs, when faced with such
large volumes due to systemic issues, have used CSVs with the necessary
information)
- The full list of certificates not revoked according to the BR timeline
(again, CSVs suffice)
Can you also confirm that, based on the information provided so far, that
it is accurate to say that 100% of these certificates were issued to a
single organization? As the discussions around revocation captured, because
the needs and nature of BR revocation violations can vary across
organizations, separate incident reports should be filed for each affected
customer. It would appear that there is only one customer (including
Affiliates) - Google - but I do not see that explicitly stated. The use of
subscribers (plural) leaves it ambiguous as to whether there have been
multiple Subscriber Agreements/Terms of Use, or whether that is simply
accounting for different teams within Google and it’s Affiliates under the
auspices of a single Agreement/Terms of Use.
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
Thanks for providing the update. One area that I do need to push back on is
the disclosure of the 100K certificates mentioned.
As demonstrated through past CA distrust discussions and whose need is
evidenced by past incident reports, one of the purposes of having CAs
disclose the affected certificates is to provide assurance to the community
that the CA has completed a rigorous investigation of the issue.
While I realize that, as a systemic issue, it’s easy to highlight that it
is 100% of issuance from a given CA, in the goal of ensuring consistent
incident reporting, please provide the following necessary and required
information:
- The full list of certificates affected (past CAs, when faced with such
large volumes due to systemic issues, have used CSVs with the necessary
information)
- The full list of certificates not revoked according to the BR timeline
(again, CSVs suffice)
Can you also confirm that, based on the information provided so far, that
it is accurate to say that 100% of these certificates were issued to a
single organization? As the discussions around revocation captured, because
the needs and nature of BR revocation violations can vary across
organizations, separate incident reports should be filed for each affected
customer. It would appear that there is only one customer (including
Affiliates) - Google - but I do not see that explicitly stated. The use of
subscribers (plural) leaves it ambiguous as to whether there have been
multiple Subscriber Agreements/Terms of Use, or whether that is simply
accounting for different teams within Google and it’s Affiliates under the
auspices of a single Agreement/Terms of Use.
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
Ryan Hurst
Mar 5, 2019, 8:42:50 PM3/5/19
to mozilla-dev-s...@lists.mozilla.org
Posting from a personal account but commenting in a professional capacity.
Our decision not to include the list was intended for brevity sake only. It is a reasonable request to provide a CSV and we will do that within 24 hours.
Regarding the number of subscribers, yes in this case it is appropriate to say there is a single subscriber, Alphabet and it’s affiliates, including Google.
Our decision not to include the list was intended for brevity sake only. It is a reasonable request to provide a CSV and we will do that within 24 hours.
Regarding the number of subscribers, yes in this case it is appropriate to say there is a single subscriber, Alphabet and it’s affiliates, including Google.
Ryan Hurst
Mar 6, 2019, 12:10:00 PM3/6/19
to mozilla-dev-s...@lists.mozilla.org
We have attached two files to the bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1532842), one that provides a list of all certificates issued after ballot 164 that contain 63 bit serial numbers and one that lists all certificates in that set that have not yet been revoked.
Ryan Hurst
Mar 11, 2019, 2:39:24 PM3/11/19
to mozilla-dev-s...@lists.mozilla.org
Dear m.d.s.p,
We wanted to follow-up to this thread and give a brief update.
We have revoked all but 26 of the affected certificates and are working with the associated subscribers to enable a smooth transition prior to revocation which will occur as each certificate is replaced or by 2019-03-31 whichever happens first.
We wanted to follow-up to this thread and give a brief update.
We have revoked all but 26 of the affected certificates and are working with the associated subscribers to enable a smooth transition prior to revocation which will occur as each certificate is replaced or by 2019-03-31 whichever happens first.
0 new messages