Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Incident Report: TrustCor Serial Number Entropy
427 views
Skip to first unread message
Neil Dunbar
Mar 2, 2019, 10:52:56 AM3/2/19
to mozilla-dev-security-policy
All,
Included is an incident report, formatted per the Mozilla recommendations, with timelines and resolutions.
Wayne - if you want to create a bug to track this, I’m happy to respond either there or here on m.d.s.p.
Regards,
Neil Dunbar
CA Administrator,
TrustCor Systems S. de R.L.
-----
1. How your CA first became aware of the problem (e.g. via a problem
report submitted to your Problem Reporting Mechanism, a discussion in
mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit),
and the time and date.
Following a recent discussion on mozilla.dev.security.policy [1], an incident was reported which led us to believe there could be an issue involving serial numbers in certificate issuance.
Specifically, the output of certificates would contain a 64-bit serial number, but depending on the software used, there is a high probability that the most significant bit (big-endian) of the serial numbers is 0. In this scenario, those certificate serial numbers would not meet Section 7.1 of the Baseline Requirements, having an effective entropy input of 63-bits.
This prompted us to perform an immediate investigation as to whether in our specific software configuration, sufficient entropy was being used in TrustCor's certificate serial number generation.
2. A timeline of the actions your CA took in response. A timeline is a
date-and-time-stamped sequence of all relevant events. This may include
events before the incident was reported, such as when a particular
requirement became applicable, or a document changed, or a bug was
introduced, or an audit was done.
(All times are UTC)
[2019-02-25 08:25:00] TrustCor first becomes aware the potential issue.
[2019-02-25 09:25:00] TrustCor suspended issuance of all SSL certificates pending investigation.
[2019-02-25 13:00:00] Investigation into how to change serial number length, ideally on a per-CA basis.
[2019-02-26 15:30:00] TrustCor Policy Authority accepts that compliance is only guaranteed for at least 64-bit serial numbers, and orders identification and revocation of all certificates whose serial numbers contain less than the minimum required value, and a solution should to be found to reissue affected certificates and generate all future certificates with profiles greater than the required 64-bit.
[2019-02-27 18:05:00] Operational testing solution for 96-bit serial numbers successfully applied.
[2019-02-27 21:30:00] Downstream applications confirmed to function properly with larger serial numbers.
[2019-02-28 16:20:00] Risk acceptance performed into pushing the tested configuration into production. Change Approval Ticket generated.
[2019-02-28 20:17:32] Change Approval granted.
[2019-03-01 09:05:29] TrustCor identifies five (5) mis-issued certificates (all of which were internal certificates). All certificates identified as mis-issued are revoked.
[2019-03-01 09:30:00] Change to production issuance systems configuration complete.
[2019-03-01 11:14:00] Certificate issuance resumed, using larger entropy.
[2019-03-01 14:40:31] All revoked certificates are successfully reissued.
[2019-03-01 16:00:00] Verification that all certificate related services are working with the new serial numbers (CRLs, OCSP, CT publication, etc).
3. Whether your CA has stopped, or has not yet stopped, issuing
certificates with the problem. A statement that you have will be
considered a pledge to the community; a statement that you have not
requires an explanation.
When it became clear that there might have been a problem, TrustCor made the decision to suspend issuance until it could be established whether or not mis-issuance had taken place.
4. A summary of the problematic certificates. For each problem: number
of certs, and the date the first and last certs with that problem were
issued.
Five (5) valid (ie, unrevoked and unexpired) certificates were identified as exhibiting this problem. Three of the five certificates were TrustCor's own test website certificates. The other two were also owned by TrustCor for internal API websites. Once TrustCor identified a mis-issuance, all five of the mis-issued certificates were immediately revoked.
The first certificate with the problem was issued on 2017-11-30 22:01:39.
The last such certificate exhibiting the problem was issued on 2018-12-20 12:53:05.
5. The complete certificate data for the problematic certificates. The
recommended way to provide this is to ensure each certificate is logged
to CT and then list the fingerprints or crt.sh IDs, either in the report
or as an attached spreadsheet, with one list per distinct problem.
The certificates (now revoked) are:
https://crt.sh/?id=1044400819
https://crt.sh/?id=1044636961
https://crt.sh/?id=285016280
https://crt.sh/?id=295418448
https://crt.sh/?id=295418456
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
It would appear that there was not general acceptance by the software vendor that rejecting 64-bit serial numbers with the top bit set was actually equivalent to using 63-bit entropy; therefore this bug has been allowed to persist for some time [1] However, the vendor has announced a future software release that will change this behaviour, defaulting to larger serial numbers [2].
Under the current version, a longer serial number setting is configurable within the software. However, it is not a per-CA setting, and by the vendor's own admission, is not a convenient setting to adjust - in fact, comments in the configuration file example explicitly make clear that the 64-bit setting should not normally be adjusted.
---
# It is really recommended to use at least 64 bits, so please leave
# as default unless you are really sure, and have a really good
# reason to change it.
---
TrustCor was of the belief that the vendor's setting used the entire 64-bits and had concluded that the software would produce BR-compliant certificates.
7. List of steps your CA is taking to resolve the situation and ensure
such issuance will not be repeated in the future, accompanied with a
timeline of when your CA expects to accomplish these things.
TrustCor established a method to strengthen the entropy of random serial numbers under its current platform to ensure compliance for all future Public Trust certificates. This mitigation was tested, staged, approved for production and put into production, issuance was restarted.
The global setting for the software has been changed to use 96-bits of entropy for serial numbers. While we could use 128-bits, we have been considering a prefix scheme to identify which equipment was used to generate serial numbers, and using all the keyspace would inhibit that. We could use 159-bit wide serial numbers, but that would require more extensive changes to some of our downstream software.
These changes were made live on 2019-03-01 11:14:00 UTC.
All five of the mis-issued certificates were reissued using higher entropy that meets the Baseline Requirements.
A scanning tool has been compiled which scans the database for certificates issued (post fix date), and measures both the parity of the serial number and distance between the top and bottom set bits within the serial. If a rolling average of those numbers falls outside of a reasonable tolerance of 50% parity, or the expected 96-bit distance, an alert is generated which will start a security incident, to which TrustCor personnel must respond promptly. This tool runs every 15 minutes. This tool will also be deployed to the test array, such that large numbers of certificates can be generated, scanned and a histogram of bit widths to store serial numbers viewed.
It is possible that a set of serial numbers could be generated whose top bits are genuinely zero, but we consider this such an unlikely event that it is worth generating an alert for investigation, and is far more likely to be a misbehaviour of the serial number generation method.
[1] https://groups.google.com/d/msg/mozilla.dev.security.policy/nnLVNfqgz7g/OVKywVZIBgAJ
[2] https://jira.primekey.se/browse/ECA-4991
Included is an incident report, formatted per the Mozilla recommendations, with timelines and resolutions.
Wayne - if you want to create a bug to track this, I’m happy to respond either there or here on m.d.s.p.
Regards,
Neil Dunbar
CA Administrator,
TrustCor Systems S. de R.L.
-----
1. How your CA first became aware of the problem (e.g. via a problem
report submitted to your Problem Reporting Mechanism, a discussion in
mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit),
and the time and date.
Following a recent discussion on mozilla.dev.security.policy [1], an incident was reported which led us to believe there could be an issue involving serial numbers in certificate issuance.
Specifically, the output of certificates would contain a 64-bit serial number, but depending on the software used, there is a high probability that the most significant bit (big-endian) of the serial numbers is 0. In this scenario, those certificate serial numbers would not meet Section 7.1 of the Baseline Requirements, having an effective entropy input of 63-bits.
This prompted us to perform an immediate investigation as to whether in our specific software configuration, sufficient entropy was being used in TrustCor's certificate serial number generation.
2. A timeline of the actions your CA took in response. A timeline is a
date-and-time-stamped sequence of all relevant events. This may include
events before the incident was reported, such as when a particular
requirement became applicable, or a document changed, or a bug was
introduced, or an audit was done.
(All times are UTC)
[2019-02-25 08:25:00] TrustCor first becomes aware the potential issue.
[2019-02-25 09:25:00] TrustCor suspended issuance of all SSL certificates pending investigation.
[2019-02-25 13:00:00] Investigation into how to change serial number length, ideally on a per-CA basis.
[2019-02-26 15:30:00] TrustCor Policy Authority accepts that compliance is only guaranteed for at least 64-bit serial numbers, and orders identification and revocation of all certificates whose serial numbers contain less than the minimum required value, and a solution should to be found to reissue affected certificates and generate all future certificates with profiles greater than the required 64-bit.
[2019-02-27 18:05:00] Operational testing solution for 96-bit serial numbers successfully applied.
[2019-02-27 21:30:00] Downstream applications confirmed to function properly with larger serial numbers.
[2019-02-28 16:20:00] Risk acceptance performed into pushing the tested configuration into production. Change Approval Ticket generated.
[2019-02-28 20:17:32] Change Approval granted.
[2019-03-01 09:05:29] TrustCor identifies five (5) mis-issued certificates (all of which were internal certificates). All certificates identified as mis-issued are revoked.
[2019-03-01 09:30:00] Change to production issuance systems configuration complete.
[2019-03-01 11:14:00] Certificate issuance resumed, using larger entropy.
[2019-03-01 14:40:31] All revoked certificates are successfully reissued.
[2019-03-01 16:00:00] Verification that all certificate related services are working with the new serial numbers (CRLs, OCSP, CT publication, etc).
3. Whether your CA has stopped, or has not yet stopped, issuing
certificates with the problem. A statement that you have will be
considered a pledge to the community; a statement that you have not
requires an explanation.
When it became clear that there might have been a problem, TrustCor made the decision to suspend issuance until it could be established whether or not mis-issuance had taken place.
4. A summary of the problematic certificates. For each problem: number
of certs, and the date the first and last certs with that problem were
issued.
Five (5) valid (ie, unrevoked and unexpired) certificates were identified as exhibiting this problem. Three of the five certificates were TrustCor's own test website certificates. The other two were also owned by TrustCor for internal API websites. Once TrustCor identified a mis-issuance, all five of the mis-issued certificates were immediately revoked.
The first certificate with the problem was issued on 2017-11-30 22:01:39.
The last such certificate exhibiting the problem was issued on 2018-12-20 12:53:05.
5. The complete certificate data for the problematic certificates. The
recommended way to provide this is to ensure each certificate is logged
to CT and then list the fingerprints or crt.sh IDs, either in the report
or as an attached spreadsheet, with one list per distinct problem.
The certificates (now revoked) are:
https://crt.sh/?id=1044400819
https://crt.sh/?id=1044636961
https://crt.sh/?id=285016280
https://crt.sh/?id=295418448
https://crt.sh/?id=295418456
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
It would appear that there was not general acceptance by the software vendor that rejecting 64-bit serial numbers with the top bit set was actually equivalent to using 63-bit entropy; therefore this bug has been allowed to persist for some time [1] However, the vendor has announced a future software release that will change this behaviour, defaulting to larger serial numbers [2].
Under the current version, a longer serial number setting is configurable within the software. However, it is not a per-CA setting, and by the vendor's own admission, is not a convenient setting to adjust - in fact, comments in the configuration file example explicitly make clear that the 64-bit setting should not normally be adjusted.
---
# It is really recommended to use at least 64 bits, so please leave
# as default unless you are really sure, and have a really good
# reason to change it.
---
TrustCor was of the belief that the vendor's setting used the entire 64-bits and had concluded that the software would produce BR-compliant certificates.
7. List of steps your CA is taking to resolve the situation and ensure
such issuance will not be repeated in the future, accompanied with a
timeline of when your CA expects to accomplish these things.
TrustCor established a method to strengthen the entropy of random serial numbers under its current platform to ensure compliance for all future Public Trust certificates. This mitigation was tested, staged, approved for production and put into production, issuance was restarted.
The global setting for the software has been changed to use 96-bits of entropy for serial numbers. While we could use 128-bits, we have been considering a prefix scheme to identify which equipment was used to generate serial numbers, and using all the keyspace would inhibit that. We could use 159-bit wide serial numbers, but that would require more extensive changes to some of our downstream software.
These changes were made live on 2019-03-01 11:14:00 UTC.
All five of the mis-issued certificates were reissued using higher entropy that meets the Baseline Requirements.
A scanning tool has been compiled which scans the database for certificates issued (post fix date), and measures both the parity of the serial number and distance between the top and bottom set bits within the serial. If a rolling average of those numbers falls outside of a reasonable tolerance of 50% parity, or the expected 96-bit distance, an alert is generated which will start a security incident, to which TrustCor personnel must respond promptly. This tool runs every 15 minutes. This tool will also be deployed to the test array, such that large numbers of certificates can be generated, scanned and a histogram of bit widths to store serial numbers viewed.
It is possible that a set of serial numbers could be generated whose top bits are genuinely zero, but we consider this such an unlikely event that it is worth generating an alert for investigation, and is far more likely to be a misbehaviour of the serial number generation method.
[1] https://groups.google.com/d/msg/mozilla.dev.security.policy/nnLVNfqgz7g/OVKywVZIBgAJ
[2] https://jira.primekey.se/browse/ECA-4991
Wayne Thayer
Mar 4, 2019, 3:21:32 PM3/4/19
to Neil Dunbar, mozilla-dev-security-policy
Neil,
On Sat, Mar 2, 2019 at 8:52 AM Neil Dunbar via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:
> All,
>
> Included is an incident report, formatted per the Mozilla recommendations,
> with timelines and resolutions.
>
> Thank you for completing an excellent incident report.
Wayne - if you want to create a bug to track this, I’m happy to respond
> either there or here on m.d.s.p.
>
> I have created https://bugzilla.mozilla.org/show_bug.cgi?id=1532399 to
track this issue
On Sat, Mar 2, 2019 at 8:52 AM Neil Dunbar via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:
> All,
>
> Included is an incident report, formatted per the Mozilla recommendations,
> with timelines and resolutions.
>
Wayne - if you want to create a bug to track this, I’m happy to respond
> either there or here on m.d.s.p.
>
track this issue
0 new messages