CA Incident Transparency and Public Audits
955 views
Skip to first unread message
Wayne
Apr 24, 2024, 3:12:47 AMApr 24
to dev-secur...@mozilla.org
Hello,
As part of an incident response CAs generally provide a list of impacted certs in the form of SHA256 sums and a link to crt.sh for each certificate. Additionally each certificate will need revoked and we have a Certificate Revocation List to check timestamps and reasons.
In order to audit we must be able to take a list of crt.sh SHA256 links, and turn them into something we can check against a CRL: serial numbers. This is the biggest barrier currently.
Note that there is a limitation in obtaining only the latest copy of a CRL as they generally do not include expired domains and are trimmed automatically. To reflect this limitation this dataset will be working on a CRL obtained 2024-04-20 and 2024-04-23 with merged de-duped records. The vast majority of certificates issued by Entrust lie in the 3-month period going by samples.
Proof of Concept 1: Entrust: CPS typographical (text placement) error
Original Issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1890896
Impacted Certs: 6008*
The naive approach for a proof of concept is to check crt.sh's postgres db, or website for each url and grab them one-by-one to extract the serial number. In practice this took over 20 hours - noting that there isn't a good mechanism for checking in bulk and this is not a shortcoming of crt.sh as a public service. Thank you for your work Sectigo.
Next I grabbed the most relevant CRL (L1K) and parsed it to provide a lookup table - but only from 2024-03-12. This date was chosen as it is a day before their major issues recently started snowballing. A decision was made to give a cutoff at 2024-04-23 08:00:00 UTC applying to all future notes.
Combining both of these aspects onto a single cursed google sheet gets us some brief analysis and should suffice as a proof of concept.
Full google sheet: https://docs.google.com/spreadsheets/d/1QuwPJh0sCHQCQaQEtUsr9qw_QRlf8fpwf8B7W6MPOqw/edit?usp=sharing
Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vT8ulLxKWSdCo80Xqf5UyAaqznIOcGDl_TEd2T2ayCZbaItnWfvLs4AC_F8BYmI16R-F1UgHJiWgu5J/pubhtml
NOTE: Avoid CPS_6K_List_56K_WARNING (5,995 rows), and CRL_L1K_56K_WARNING (12,013 rows)
This issue in particular is one where Entrust insist they will not revoke so it's a good testbed for a worst-case scenario. There is a breakdown on just the 6,008* cert batch, and also on the CRL attached to 99% of their certificates.
*Caveat: This was only checking Entrust's L1K CRL, and it turns out there are 35 certificates from L1F unaccounted for in this preliminary test. Additionally there are also 14 certificates not logged on crt.sh despite being on that list for inclusion.[1]
With that in mind we can see a pattern to their revocation at the CRL level: 4am is an automated system for processing, they generally only take requests/process them Monday-Friday. The first time we breach over 500 revocations per-day occurs 2024-04-19 (Friday).
Proof of Concept 2: Entrust's Publicly Listed CRLs
Given the sample above and handling one of their largest pools went fine I expanded the test to all reasonably available CRLs and every entry. Given Entrust's scale this should test the limits of Google Sheets and provide a wider picture.
NOTE: Avoid CRL_DATA_56K_WARNING (86k rows)
Full google sheet: https://docs.google.com/spreadsheets/d/1V2vLoOdD5CxDeCfVbTFHP_8HWYCViS9wo5Q23VWDJPE/edit?usp=sharing
Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vQ9RhMAJVmaBWZ_chwUIK_KYOMgERe-PZH2MbZhkS16EDJiOkRJJVO8QnCfdbbGcak6EEI_wvIT2ak_/pubhtml
The advantage of this approach is that we can take a step back from checking an individual certificate against each CRL. If a CA is responding to an incident where they must revoke 10k certicates within 5 days, then we should see approximately 10k revocations in a short time period, no?
While this has the full data I've also trimmed CRL_GRAPH to show from 2024-03-12 onward. There have been 24,154 revocations across every CRL, but there's an unusual spike this weekend (see linked image below or the google sheets link). 12,012 on L1K - the main OV intermediary, and 11,370 on L1M - the main EV intermediary.
CRL Graph: https://i.imgur.com/xN3U8O3.png
Between 04-20 and 04-22 on L1K (OV) accounts for 710 revocations, while L1M (EV) accounts for 2,598 revocations alone. The weekend is normally quiet however EV certificates started being handled differently, but that isn't the topic of this post.
Now we can check these revocations against a single larger issue and get an idea of what's happening. Notably we won't be checking each certificate given the SHA256<->serial bottleneck mentioned above for a smaller separate pool.
https://bugzilla.mozilla.org/show_bug.cgi?id=1883843#c32
Entrust: EV TLS Certificate cPSuri missing
26653 pre-certs provided, confirmed as mis-issued by Entrust 2024-03-05 but only action started 2024-03-18. Given these are EV certificates we know they will be primarily from intermediary L1M (and sampling confirms this).
As noted early L1M has had 11,370 revocations since 2024-03-12, or if we count from Entrust acknowledging revocation as necessary (2024-03-18) a total of 11,268. This is imperfect as Entrust have decided that expiry is the same as revocation, and includes certificates issued outside of the affected batch. Additionally CRLs are pruned for expired certificates, however given the issue was from 2024-03-22 and a CRL from 2024-04-20 is sampled these should be accounted for.
Looking at a statement made 2024-04-15 (https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c10) we are told 8,572 revoked or expired. The CRL for L1M at that time notes from 2024-03-12 to 2024-04-15 6,989 revocations (inclusive). From acknowledgement we get 2024-03-18 to 2024-04-15 with 6,887. Regardless both would be approximately 80% revoked:expired read generously.
The latest statement (https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c11) by Entrust on revocation and expiring (2024-04-19) notes 10,013 revoked/expired. Against the total pool of 26,653 requiring revocation within 5 days we get 37.5%. This is noting a ramp-up in revocations is starting, but given the information provided this could be a separate issue unrelated to this batch.
Closing Remarks
There is room for a more thorough analysis by someone far more talented, but I hope this brief picture gives enough of an idea of what can be done in the future.
Takeaways
- We need a better method for the public to transform SHA256 certs en-mass to serial numbers to check CRLs
- We need CAs to be more forthcoming with breakdowns of issuing intermediaries for each batch of certificates requiring revocation
- There isn't any big technical barrier or processing issue to ascertaining a CA's compliance in an ongoing issue from someone with better tooling
- There is further research required in checking for undisclosed issues via CRLs and encouraging transparency
- A CA shouldn't collapse in their public trust to evoke this level of public research
Baseline Requirements Notes:
4.10.1 Operational characteristics
Revocation entries on a CRL or OCSP Response MUST NOT be removed until after the Expiry Date of the revoked Certificate.
7.2 CRL profile
nextUpdate
Indicates the date by which the next CRL will be issued. For CRLs covering Subscriber Certificates, at most 10 days after the thisUpdate. For other CRLs, at most 12 months after the thisUpdate.
revokedCertificates
MUST be present if the CA has issued a Certificate that has been revoked and the corresponding entry has yet to appear on at least one regularly scheduled CRL beyond the revoked Certificate’s validity period. The CA SHOULD remove an entry for a corresponding Certificate after it has appeared on at least one regularly scheduled CRL beyond the revoked Certificate’s validity period. See the “revokedCertificates Component” table for additional requirements.
Possible CRL Issue:
While compiling this I noticed that 'Entrust Enterprise Intermediate CA-ICA1' isn't refreshing the CRL daily (same CRL 2024-04-20 and 2024-04-23):
2048.crl:
Last Update: Apr 5 15:55:58 2024 GMT
Next Update: Apr 5 15:55:57 2025 GMT
I presume that'd still cover: https://crt.sh/?caid=32 which has 13 unexpired certs, but this may be a misinterpretation on my behalf.
[1]: Certificates not on crt.sh but listed in 20240409-OV-TLS-Certificates-typo-finalcerts:
-----
https://crt.sh/?sha256=1086E495E6D4CDC926CDFE3C3F083272D70BE65F3BF75336393C6975A11E6D8A
https://crt.sh/?sha256=125E7D5030D1B4C57DFF13DAD4E9525024DEC3649F77745C906C229765E24F5F
https://crt.sh/?sha256=4323087C85F6FB5F7F2D8E66D9C5EC8BC4007D37CBF93ACC159D6173D0793AFA
https://crt.sh/?sha256=637D768E1D386ACDEBED435C84B0B08C2C6B09B840045A34503DB518645657B5
https://crt.sh/?sha256=692991CF2926851395CE4C15A37AC4B6AD13D46979E5A7912F39FDD9FFFAC4A9
https://crt.sh/?sha256=749109CEAEE9CF2D88E6999D4768697696E7C67E4C4DB2648A19137A5D1669ED
https://crt.sh/?sha256=76ADEF1237C4B07652D8A597337580947E6B6E1A83F3D25332BE3863BA180CAC
https://crt.sh/?sha256=7DD2FF3533BD7C64DEA51E89F469EDD29B007C5DC4708597A9F0594B76954EE1
https://crt.sh/?sha256=AF4274B727520335A80F82C0E615EFB87D37C5F03971AFDF9114ACBCFA5882EA
https://crt.sh/?sha256=C5E3974E1DBE2A36450B662A6867988B31361F9DFC67AC4ACA38B386B43432F5
https://crt.sh/?sha256=EF5F20F6B5913C3A5F9477E2BE2D6801D85DD80B9BB42A5363AA3B1E82566FD9
https://crt.sh/?sha256=F5455CBE0B50EC0002A5C1F83AC49CC38842B30D84B830D8A23257AEFA406117
https://crt.sh/?sha256=F7F70E1FD16BB7B09A21486AAC9295CFD8259BB1C6EF363D5A61C1230B1E9426
https://crt.sh/?sha256=FE5BEA6F8D4333CFEFA3277AC2F4D544987FFF7C3ED83514EAF92A34919F19E6
-----
- Wayne
I've been watching the Entrust saga of issues over the past month and keeping an eye on their incident response like many in the community. Given that, I was curious how the public could reasonably audit a non-compliant CA and ascertain how honest their claims were, as well as what bottlenecks exist for public analysis.
As part of an incident response CAs generally provide a list of impacted certs in the form of SHA256 sums and a link to crt.sh for each certificate. Additionally each certificate will need revoked and we have a Certificate Revocation List to check timestamps and reasons.
In order to audit we must be able to take a list of crt.sh SHA256 links, and turn them into something we can check against a CRL: serial numbers. This is the biggest barrier currently.
Note that there is a limitation in obtaining only the latest copy of a CRL as they generally do not include expired domains and are trimmed automatically. To reflect this limitation this dataset will be working on a CRL obtained 2024-04-20 and 2024-04-23 with merged de-duped records. The vast majority of certificates issued by Entrust lie in the 3-month period going by samples.
Proof of Concept 1: Entrust: CPS typographical (text placement) error
Original Issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1890896
Impacted Certs: 6008*
The naive approach for a proof of concept is to check crt.sh's postgres db, or website for each url and grab them one-by-one to extract the serial number. In practice this took over 20 hours - noting that there isn't a good mechanism for checking in bulk and this is not a shortcoming of crt.sh as a public service. Thank you for your work Sectigo.
Next I grabbed the most relevant CRL (L1K) and parsed it to provide a lookup table - but only from 2024-03-12. This date was chosen as it is a day before their major issues recently started snowballing. A decision was made to give a cutoff at 2024-04-23 08:00:00 UTC applying to all future notes.
Combining both of these aspects onto a single cursed google sheet gets us some brief analysis and should suffice as a proof of concept.
Full google sheet: https://docs.google.com/spreadsheets/d/1QuwPJh0sCHQCQaQEtUsr9qw_QRlf8fpwf8B7W6MPOqw/edit?usp=sharing
Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vT8ulLxKWSdCo80Xqf5UyAaqznIOcGDl_TEd2T2ayCZbaItnWfvLs4AC_F8BYmI16R-F1UgHJiWgu5J/pubhtml
NOTE: Avoid CPS_6K_List_56K_WARNING (5,995 rows), and CRL_L1K_56K_WARNING (12,013 rows)
This issue in particular is one where Entrust insist they will not revoke so it's a good testbed for a worst-case scenario. There is a breakdown on just the 6,008* cert batch, and also on the CRL attached to 99% of their certificates.
*Caveat: This was only checking Entrust's L1K CRL, and it turns out there are 35 certificates from L1F unaccounted for in this preliminary test. Additionally there are also 14 certificates not logged on crt.sh despite being on that list for inclusion.[1]
With that in mind we can see a pattern to their revocation at the CRL level: 4am is an automated system for processing, they generally only take requests/process them Monday-Friday. The first time we breach over 500 revocations per-day occurs 2024-04-19 (Friday).
Proof of Concept 2: Entrust's Publicly Listed CRLs
Given the sample above and handling one of their largest pools went fine I expanded the test to all reasonably available CRLs and every entry. Given Entrust's scale this should test the limits of Google Sheets and provide a wider picture.
NOTE: Avoid CRL_DATA_56K_WARNING (86k rows)
Full google sheet: https://docs.google.com/spreadsheets/d/1V2vLoOdD5CxDeCfVbTFHP_8HWYCViS9wo5Q23VWDJPE/edit?usp=sharing
Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vQ9RhMAJVmaBWZ_chwUIK_KYOMgERe-PZH2MbZhkS16EDJiOkRJJVO8QnCfdbbGcak6EEI_wvIT2ak_/pubhtml
The advantage of this approach is that we can take a step back from checking an individual certificate against each CRL. If a CA is responding to an incident where they must revoke 10k certicates within 5 days, then we should see approximately 10k revocations in a short time period, no?
While this has the full data I've also trimmed CRL_GRAPH to show from 2024-03-12 onward. There have been 24,154 revocations across every CRL, but there's an unusual spike this weekend (see linked image below or the google sheets link). 12,012 on L1K - the main OV intermediary, and 11,370 on L1M - the main EV intermediary.
CRL Graph: https://i.imgur.com/xN3U8O3.png
Between 04-20 and 04-22 on L1K (OV) accounts for 710 revocations, while L1M (EV) accounts for 2,598 revocations alone. The weekend is normally quiet however EV certificates started being handled differently, but that isn't the topic of this post.
Now we can check these revocations against a single larger issue and get an idea of what's happening. Notably we won't be checking each certificate given the SHA256<->serial bottleneck mentioned above for a smaller separate pool.
https://bugzilla.mozilla.org/show_bug.cgi?id=1883843#c32
Entrust: EV TLS Certificate cPSuri missing
26653 pre-certs provided, confirmed as mis-issued by Entrust 2024-03-05 but only action started 2024-03-18. Given these are EV certificates we know they will be primarily from intermediary L1M (and sampling confirms this).
As noted early L1M has had 11,370 revocations since 2024-03-12, or if we count from Entrust acknowledging revocation as necessary (2024-03-18) a total of 11,268. This is imperfect as Entrust have decided that expiry is the same as revocation, and includes certificates issued outside of the affected batch. Additionally CRLs are pruned for expired certificates, however given the issue was from 2024-03-22 and a CRL from 2024-04-20 is sampled these should be accounted for.
Looking at a statement made 2024-04-15 (https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c10) we are told 8,572 revoked or expired. The CRL for L1M at that time notes from 2024-03-12 to 2024-04-15 6,989 revocations (inclusive). From acknowledgement we get 2024-03-18 to 2024-04-15 with 6,887. Regardless both would be approximately 80% revoked:expired read generously.
The latest statement (https://bugzilla.mozilla.org/show_bug.cgi?id=1886532#c11) by Entrust on revocation and expiring (2024-04-19) notes 10,013 revoked/expired. Against the total pool of 26,653 requiring revocation within 5 days we get 37.5%. This is noting a ramp-up in revocations is starting, but given the information provided this could be a separate issue unrelated to this batch.
Closing Remarks
There is room for a more thorough analysis by someone far more talented, but I hope this brief picture gives enough of an idea of what can be done in the future.
Takeaways
- We need a better method for the public to transform SHA256 certs en-mass to serial numbers to check CRLs
- We need CAs to be more forthcoming with breakdowns of issuing intermediaries for each batch of certificates requiring revocation
- There isn't any big technical barrier or processing issue to ascertaining a CA's compliance in an ongoing issue from someone with better tooling
- There is further research required in checking for undisclosed issues via CRLs and encouraging transparency
- A CA shouldn't collapse in their public trust to evoke this level of public research
Baseline Requirements Notes:
4.10.1 Operational characteristics
Revocation entries on a CRL or OCSP Response MUST NOT be removed until after the Expiry Date of the revoked Certificate.
7.2 CRL profile
nextUpdate
Indicates the date by which the next CRL will be issued. For CRLs covering Subscriber Certificates, at most 10 days after the thisUpdate. For other CRLs, at most 12 months after the thisUpdate.
revokedCertificates
MUST be present if the CA has issued a Certificate that has been revoked and the corresponding entry has yet to appear on at least one regularly scheduled CRL beyond the revoked Certificate’s validity period. The CA SHOULD remove an entry for a corresponding Certificate after it has appeared on at least one regularly scheduled CRL beyond the revoked Certificate’s validity period. See the “revokedCertificates Component” table for additional requirements.
Possible CRL Issue:
While compiling this I noticed that 'Entrust Enterprise Intermediate CA-ICA1' isn't refreshing the CRL daily (same CRL 2024-04-20 and 2024-04-23):
2048.crl:
Last Update: Apr 5 15:55:58 2024 GMT
Next Update: Apr 5 15:55:57 2025 GMT
I presume that'd still cover: https://crt.sh/?caid=32 which has 13 unexpired certs, but this may be a misinterpretation on my behalf.
[1]: Certificates not on crt.sh but listed in 20240409-OV-TLS-Certificates-typo-finalcerts:
-----
https://crt.sh/?sha256=1086E495E6D4CDC926CDFE3C3F083272D70BE65F3BF75336393C6975A11E6D8A
https://crt.sh/?sha256=125E7D5030D1B4C57DFF13DAD4E9525024DEC3649F77745C906C229765E24F5F
https://crt.sh/?sha256=4323087C85F6FB5F7F2D8E66D9C5EC8BC4007D37CBF93ACC159D6173D0793AFA
https://crt.sh/?sha256=637D768E1D386ACDEBED435C84B0B08C2C6B09B840045A34503DB518645657B5
https://crt.sh/?sha256=692991CF2926851395CE4C15A37AC4B6AD13D46979E5A7912F39FDD9FFFAC4A9
https://crt.sh/?sha256=749109CEAEE9CF2D88E6999D4768697696E7C67E4C4DB2648A19137A5D1669ED
https://crt.sh/?sha256=76ADEF1237C4B07652D8A597337580947E6B6E1A83F3D25332BE3863BA180CAC
https://crt.sh/?sha256=7DD2FF3533BD7C64DEA51E89F469EDD29B007C5DC4708597A9F0594B76954EE1
https://crt.sh/?sha256=AF4274B727520335A80F82C0E615EFB87D37C5F03971AFDF9114ACBCFA5882EA
https://crt.sh/?sha256=C5E3974E1DBE2A36450B662A6867988B31361F9DFC67AC4ACA38B386B43432F5
https://crt.sh/?sha256=EF5F20F6B5913C3A5F9477E2BE2D6801D85DD80B9BB42A5363AA3B1E82566FD9
https://crt.sh/?sha256=F5455CBE0B50EC0002A5C1F83AC49CC38842B30D84B830D8A23257AEFA406117
https://crt.sh/?sha256=F7F70E1FD16BB7B09A21486AAC9295CFD8259BB1C6EF363D5A61C1230B1E9426
https://crt.sh/?sha256=FE5BEA6F8D4333CFEFA3277AC2F4D544987FFF7C3ED83514EAF92A34919F19E6
-----
- Wayne
Wayne
Apr 27, 2024, 10:50:00 AMApr 27
to dev-secur...@mozilla.org, Wayne
Continuing on with this research I've analysed the open issues on bugzilla for recent mis-issuances requiring revocation with more than 100 certificates.
The full dataset is held at: https://docs.google.com/spreadsheets/d/1DmqupuUfx5bQvtm99I6OZZnYj3BL9Eawo1gu0uPST0E/edit?usp=sharing
Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vRM7a7yCe3jYJAvrb-dR90iIh5xu3WAbj7EU42r14pgjLS1G7z7W4Vp2jOAesmStkMM6hw4emLKw9-K/pubhtml
To reiterate this is checking mis-issued certificate batches against Certificate Revocation Lists. For every mis-issued batch (bar some exceptionally large issues) I've obtained the serial number and checked the CRL to see revocation timeframes and progress at 5 days, 15 days, and 30 days. This is to also to see if there's any data in the murmured proposals of an extension.
General Caveats:
- There is imprecise wording and different interpretations on when the revocation start timer begins, and how CAs note this in their incident reports. To that end I've separated the time that a CA seems to have internally confirmed the issue, and when they actively started some process of revocation: contacting subscribers or actively revoking.
- A number of these issues are still pending revocation and will be updated as more information arrives, consider the spreadsheet a living document
This covers 20 issues across 17 CAs - 17 of which get a detailed breakdown of individual certs to revocations:
1: ACCV
Issue: ACCV: Certificates issued with cRLIssuer in CDP extension
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884532
Impacted: 837
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- An outlier for all surveyed CAs in this dataset: the CRL involved is 24MB and has 537k entries, useful for testing
- CA provided serial numbers speeding up this report
2: Actalis
Issue: Actalis: Certificates issued with invalid RDN order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884532
Impacted: 263
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- CA provided serial numbers speeding up this report
3: Buypass (1)
Issue: Buypass: TLS certificates with incorrect Subject attribute order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1864204
Impacted: 591
Missed at 5d: 544 (92.05%)
Missed at 15d: 15 (2.54%)
Missed at 30d: 2 (0.34%)
- There is an extreme outlier (so far) as full revocation time is over 65 days
- There isn't a clear start hour for when revocation procedures occurred
4: Buypass (2)
Issue: Buypass: Using an external DNS Resolver for DNS lookups
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1872371
Impacted: 177060
Missed at 5d: 177060 (100%)
Missed at 15d: 90576 (51.16%)
Missed at 30d: 0 (0.00%)
- This is included based off of comments from the CA
- No detailed analysis is provided, the scale of this is out of scope
5: Certigna
Issue: Certigna: TLS certificates with Basic constraint non-critical
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883416
Impacted: 3794
Missed at 5d: 1243 (32.76%)
Missed at 15d: 1 (0.03%)
Missed at 30d: 0 (0.00%)
- Note: This is an ongoing incident
- There isn't a clear start hour for when revocation procedures occurred
- Counting it generously from 2024-03-21 (revocation start) to 2024-03-26 23:59 gives us 1243, when CA reports 1241?
6: certSIGN
Issue: certSIGN: Certificates with incorrect Subject attribute order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886624
Impacted: 625
Missed at 5d: 306 (48.96%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- Casualty of the fuzziness of the precise revocation start time, the 306 certs are revoked within 68 minutes after the deadline
7: CFCA
Issue: CFCA: certificate basicConstraints extension not marked as critical
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886135
Impacted: 2095*
Missed at 5d: 2089 (99.71%)
Missed at 15d: 1268 (60.53%)
Missed at 30d: 526 (25.11%)
- Note: This is an ongoing incident
- *Incident provides 2098 SHA256 sums, however this is 2095 de-duped
8: Chunghwa Telecom
Issue: Chunghwa Telecom: Wrong Extended Key Usage setting by GTLSCA
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1887096
Impacted: 6453
Missed at 5d: 6450 (99.95%)
Missed at 15d: 6450 (99.95%)
Missed at 30d: 6448 (99.92%)
- Note: This is an ongoing incident
- The CRL is 6315 entries long, and this incident is 6298 of them currently
9: D-Trust
Issue: D-Trust: LDAP-URL in Subscriber Certificate Authority Information Access field
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884714
Impacted: 2601*
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- *2 certificates are not CT-logged and not included in this figure
10: Entrust (1)
Issue: Entrust: EV TLS Certificate cPSuri missing
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883843
Impacted: 26668
Missed at 5d: 22838 (85.64%)
Missed at 15d: 20554 (77.07%)
Missed at 30d: 16655 (62.45%)
- Note: This is an ongoing incident
- This is included based off of comments from the CA
- 15d figure is taken from a comment at 14 days
- 30d figure is taken from a comment at 31 days
- No detailed analysis is provided, the scale of this is out of scope
11: Entrust (2)
Issue: Entrust: clientAuth TLS Certificates without serverAuth EKU
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886467
Impacted: 1176
Missed at 5d: 1076 (91.50%)
Missed at 15d: 1062 (90.31%)
Missed at 30d: 998 (84.87%)
- No caveats
12: Entrust (3)
Issue: Entrust: CPS typographical (text placement) error
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1890896
Impacted: 6008
Missed at 5d: 6008* (100%)
Missed at 15d: 6008* (100%)
Missed at 30d: 6008* (100%)
- CA has refused to revoke as per the Baseline Requirements and their own CPS
- No detailed analysis is provided due to this
13: Firmaprofesional
Issue: Firmaprofesional: Policy Qualifiers other than id-qt-cps present for certificate
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1889420
Impacted: 499
Missed at 5d: 1 (0.20%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- No caveats
14: FNMT
Issue: FNMT: Certificates issued included Policy qualifiers other than id-qt-cps
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1875942
Impacted: 712
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- CA provided serial numbers speeding up this report
15: Hongkong Post
Issue: Hongkong Post: TLS certificates with Certificate Policies extension that does not assert http scheme
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886406
Impacted: 1176
Missed at 5d: 1160 (98.64%)
Missed at 15d: 1160 (98.64%)
Missed at 30d: 1160 (98.64%)
- Note: This is an ongoing incident
16: NETLOCK
Issue: NETLOCK: Policy Qualifiers other than id-qt-cps is included in TLS certificates
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1889570
Impacted: 522
Missed at 5d: 521 (99.81%)
Missed at 15d: 255 (48.85%)
Missed at 30d: N/A
- Note: This is an ongoing incident
- No 30d figure as incident ongoing, at time of writing (22d)
17: Sectigo
Issue: Sectigo: EV Certificate issuance with incorrect subject:serialNumber attribute value
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1891245
Impacted: 166
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- CA provided serial numbers speeding up this report
18: Telekom Security
Issue: Telekom Security: TLS certificates with basicConstraints not marked as critical
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1875820
Impacted: 816
Missed at 5d: 336 (41.18%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- No caveats
19: TWCA
Issue: TWCA: TLS certificates with non-critical basicConstraints
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1885132
Impacted: 16481
Missed at 5d: 4330* (22.23%)
Missed at 15d: 1933* (11.73%)
Missed at 30d: 1846* (11.20%)
- CA provided serial numbers speeding up this report
- However incident data included expired certificates inflating the missed certificate figures
20: VikingCloud
Issue: VikingCloud: OV Precertificates with incorrect Subject RDN encoding order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883779
Impacted: 3208
Missed at 5d: 3123 (97.35%)
Missed at 15d: 2433 (75.84%)
Missed at 30d: 1596 (49.75%)
- Strictly speaking this is two incidents in one: additional small batch of EV certs were discovered 21 days into start of revocation
- Wayne
The full dataset is held at: https://docs.google.com/spreadsheets/d/1DmqupuUfx5bQvtm99I6OZZnYj3BL9Eawo1gu0uPST0E/edit?usp=sharing
Published HTML version: https://docs.google.com/spreadsheets/d/e/2PACX-1vRM7a7yCe3jYJAvrb-dR90iIh5xu3WAbj7EU42r14pgjLS1G7z7W4Vp2jOAesmStkMM6hw4emLKw9-K/pubhtml
To reiterate this is checking mis-issued certificate batches against Certificate Revocation Lists. For every mis-issued batch (bar some exceptionally large issues) I've obtained the serial number and checked the CRL to see revocation timeframes and progress at 5 days, 15 days, and 30 days. This is to also to see if there's any data in the murmured proposals of an extension.
This information is provided to give an overall picture of compliance across all prominent incidents recently. If there are any questions or corrections feel free to email me. There are a number of caveats per-CA that I'll detail below alongside some figures. Consider this a rough summation given the various fuzzy factors involved.
Information was all updated as of 2024-04-27 14:00 UTC
General Caveats:
- There is imprecise wording and different interpretations on when the revocation start timer begins, and how CAs note this in their incident reports. To that end I've separated the time that a CA seems to have internally confirmed the issue, and when they actively started some process of revocation: contacting subscribers or actively revoking.
- A number of these issues are still pending revocation and will be updated as more information arrives, consider the spreadsheet a living document
This covers 20 issues across 17 CAs - 17 of which get a detailed breakdown of individual certs to revocations:
1: ACCV
Issue: ACCV: Certificates issued with cRLIssuer in CDP extension
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884532
Impacted: 837
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- An outlier for all surveyed CAs in this dataset: the CRL involved is 24MB and has 537k entries, useful for testing
- CA provided serial numbers speeding up this report
2: Actalis
Issue: Actalis: Certificates issued with invalid RDN order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884532
Impacted: 263
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- CA provided serial numbers speeding up this report
3: Buypass (1)
Issue: Buypass: TLS certificates with incorrect Subject attribute order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1864204
Impacted: 591
Missed at 5d: 544 (92.05%)
Missed at 15d: 15 (2.54%)
Missed at 30d: 2 (0.34%)
- There is an extreme outlier (so far) as full revocation time is over 65 days
- There isn't a clear start hour for when revocation procedures occurred
4: Buypass (2)
Issue: Buypass: Using an external DNS Resolver for DNS lookups
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1872371
Impacted: 177060
Missed at 5d: 177060 (100%)
Missed at 15d: 90576 (51.16%)
Missed at 30d: 0 (0.00%)
- This is included based off of comments from the CA
- No detailed analysis is provided, the scale of this is out of scope
5: Certigna
Issue: Certigna: TLS certificates with Basic constraint non-critical
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883416
Impacted: 3794
Missed at 5d: 1243 (32.76%)
Missed at 15d: 1 (0.03%)
Missed at 30d: 0 (0.00%)
- Note: This is an ongoing incident
- There isn't a clear start hour for when revocation procedures occurred
- Counting it generously from 2024-03-21 (revocation start) to 2024-03-26 23:59 gives us 1243, when CA reports 1241?
6: certSIGN
Issue: certSIGN: Certificates with incorrect Subject attribute order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886624
Impacted: 625
Missed at 5d: 306 (48.96%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- Casualty of the fuzziness of the precise revocation start time, the 306 certs are revoked within 68 minutes after the deadline
7: CFCA
Issue: CFCA: certificate basicConstraints extension not marked as critical
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886135
Impacted: 2095*
Missed at 5d: 2089 (99.71%)
Missed at 15d: 1268 (60.53%)
Missed at 30d: 526 (25.11%)
- Note: This is an ongoing incident
- *Incident provides 2098 SHA256 sums, however this is 2095 de-duped
8: Chunghwa Telecom
Issue: Chunghwa Telecom: Wrong Extended Key Usage setting by GTLSCA
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1887096
Impacted: 6453
Missed at 5d: 6450 (99.95%)
Missed at 15d: 6450 (99.95%)
Missed at 30d: 6448 (99.92%)
- Note: This is an ongoing incident
- The CRL is 6315 entries long, and this incident is 6298 of them currently
9: D-Trust
Issue: D-Trust: LDAP-URL in Subscriber Certificate Authority Information Access field
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1884714
Impacted: 2601*
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- *2 certificates are not CT-logged and not included in this figure
10: Entrust (1)
Issue: Entrust: EV TLS Certificate cPSuri missing
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883843
Impacted: 26668
Missed at 5d: 22838 (85.64%)
Missed at 15d: 20554 (77.07%)
Missed at 30d: 16655 (62.45%)
- Note: This is an ongoing incident
- This is included based off of comments from the CA
- 15d figure is taken from a comment at 14 days
- 30d figure is taken from a comment at 31 days
- No detailed analysis is provided, the scale of this is out of scope
11: Entrust (2)
Issue: Entrust: clientAuth TLS Certificates without serverAuth EKU
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886467
Impacted: 1176
Missed at 5d: 1076 (91.50%)
Missed at 15d: 1062 (90.31%)
Missed at 30d: 998 (84.87%)
- No caveats
12: Entrust (3)
Issue: Entrust: CPS typographical (text placement) error
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1890896
Impacted: 6008
Missed at 5d: 6008* (100%)
Missed at 15d: 6008* (100%)
Missed at 30d: 6008* (100%)
- CA has refused to revoke as per the Baseline Requirements and their own CPS
- No detailed analysis is provided due to this
13: Firmaprofesional
Issue: Firmaprofesional: Policy Qualifiers other than id-qt-cps present for certificate
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1889420
Impacted: 499
Missed at 5d: 1 (0.20%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- No caveats
14: FNMT
Issue: FNMT: Certificates issued included Policy qualifiers other than id-qt-cps
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1875942
Impacted: 712
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- CA provided serial numbers speeding up this report
15: Hongkong Post
Issue: Hongkong Post: TLS certificates with Certificate Policies extension that does not assert http scheme
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1886406
Impacted: 1176
Missed at 5d: 1160 (98.64%)
Missed at 15d: 1160 (98.64%)
Missed at 30d: 1160 (98.64%)
- Note: This is an ongoing incident
16: NETLOCK
Issue: NETLOCK: Policy Qualifiers other than id-qt-cps is included in TLS certificates
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1889570
Impacted: 522
Missed at 5d: 521 (99.81%)
Missed at 15d: 255 (48.85%)
Missed at 30d: N/A
- Note: This is an ongoing incident
- No 30d figure as incident ongoing, at time of writing (22d)
17: Sectigo
Issue: Sectigo: EV Certificate issuance with incorrect subject:serialNumber attribute value
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1891245
Impacted: 166
Missed at 5d: 0 (0.00%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- CA provided serial numbers speeding up this report
18: Telekom Security
Issue: Telekom Security: TLS certificates with basicConstraints not marked as critical
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1875820
Impacted: 816
Missed at 5d: 336 (41.18%)
Missed at 15d: 0 (0.00%)
Missed at 30d: 0 (0.00%)
- No caveats
19: TWCA
Issue: TWCA: TLS certificates with non-critical basicConstraints
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1885132
Impacted: 16481
Missed at 5d: 4330* (22.23%)
Missed at 15d: 1933* (11.73%)
Missed at 30d: 1846* (11.20%)
- CA provided serial numbers speeding up this report
- However incident data included expired certificates inflating the missed certificate figures
20: VikingCloud
Issue: VikingCloud: OV Precertificates with incorrect Subject RDN encoding order
Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1883779
Impacted: 3208
Missed at 5d: 3123 (97.35%)
Missed at 15d: 2433 (75.84%)
Missed at 30d: 1596 (49.75%)
- Strictly speaking this is two incidents in one: additional small batch of EV certs were discovered 21 days into start of revocation
I hope this information is useful and gives an insight into how CAs are performing their incident response currently.
As mentioned any questions or comments feel free to comment publicly or privately.
- Wayne
Mike Shaver
Apr 27, 2024, 5:12:19 PMApr 27
to Wayne, dev-secur...@mozilla.org
Thanks, Wayne. I think this sort of analysis is quite valuable for constructing a reliable history of behaviour when evaluating CA operational effectiveness.
Where should it be kept longer-term? I wonder if there should be a per-root journal generated/maintained, to better help identify patterns where it may be appropriate to apply additional scrutiny to CAs, or where additional clarity around expected practices would be valuable. (Not to necessarily support CAs offering "lack of BR clarity" as an excuse for poor behaviour.)
Mike
--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/da25a846-1bb5-4abf-84fe-38e9086a6ac1n%40mozilla.org.
Reply all
Reply to author
Forward
0 new messages