Un-incorporated SCTs from CNNIC

456 views
Skip to first unread message

Brendan McMillion

unread,
Jul 28, 2018, 1:57:41 PM7/28/18
to Certificate Transparency Policy, crypto
Hello ct-policy@

Attached is what I believe to be an un-incorporated SCT from CNNIC for this certificate. crt.sh doesn't show that the log contains the certificate, and my get-proof-by-hash requests fail with status "400 Bad Request" and error message "Couldn't find hash."
310505000.cnnic.json
310505000.pem

Brendan McMillion

unread,
Aug 4, 2018, 8:38:53 PM8/4/18
to Certificate Transparency Policy, crypto
Here's another, for this certificate.
265186742.cnnic.json
265186742.pem

Brendan McMillion

unread,
Aug 7, 2018, 2:19:46 PM8/7/18
to Certificate Transparency Policy, crypto
Here is a third SCT, for this certificate. Please let me know if I should continue posting these as I find them, or if I should discard them.
628021802.cnnic.json
628021802.pem

Alex Gaynor

unread,
Aug 7, 2018, 2:22:30 PM8/7/18
to bre...@cloudflare.com, Certificate Transparency Policy, cry...@cloudflare.com
Since no one else is saying anything, I think continuing to post these is valuable, in that it highlights an ongoing issue with wide applicability.

It's alarming to me that neither CNNIC, nor Google, have said anything since this thread opened 10 days ago.

Alex

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CABP-pSQB%3DNye7BXA0BhKk%3DToALLDWBKhF78woWXGE8Pfauc-WA%40mail.gmail.com.

Kat Joyce

unread,
Aug 7, 2018, 2:24:12 PM8/7/18
to Brendan McMillion, Certificate Transparency Policy, crypto
Hi Brendan,

Thank you for posting these. We have been double checking your findings as they've come in, and I believe the Chrome guys plan to post a response soon :)

Just wanted to write a quick acknowledgement message to let you know this is not being ignored, and is definitely valuable!

Thanks again,
Kat

Ryan Sleevi

unread,
Aug 7, 2018, 2:39:57 PM8/7/18
to Alex Gaynor, Brendan McMillion, Certificate Transparency Policy, cry...@cloudflare.com
On Tue, Aug 7, 2018 at 2:22 PM Alex Gaynor <aga...@mozilla.com> wrote:
Since no one else is saying anything, I think continuing to post these is valuable, in that it highlights an ongoing issue with wide applicability.

It's alarming to me that neither CNNIC, nor Google, have said anything since this thread opened 10 days ago.

Was this been reported to the CNNIC log operator e-mail? We do not presently require that Log Operators monitor all activity on ct-policy@, nor have we ever.
 

Alex

On Tue, Aug 7, 2018 at 2:19 PM 'Brendan McMillion' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
Here is a third SCT, for this certificate. Please let me know if I should continue posting these as I find them, or if I should discard them.

On Sat, Aug 4, 2018 at 5:38 PM, Brendan McMillion <bre...@cloudflare.com> wrote:
Here's another, for this certificate.

On Sat, Jul 28, 2018 at 10:57 AM, Brendan McMillion <bre...@cloudflare.com> wrote:
Hello ct-policy@

Attached is what I believe to be an un-incorporated SCT from CNNIC for this certificate. crt.sh doesn't show that the log contains the certificate, and my get-proof-by-hash requests fail with status "400 Bad Request" and error message "Couldn't find hash."


--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CABP-pSQB%3DNye7BXA0BhKk%3DToALLDWBKhF78woWXGE8Pfauc-WA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

Brendan McMillion

unread,
Aug 7, 2018, 3:23:40 PM8/7/18
to rsl...@chromium.org, Alex Gaynor, Certificate Transparency Policy, crypto
> Was this been reported to the CNNIC log operator e-mail?

No, I hadn't reported it to them. I sent an email to their contact address just now.


On Tue, Aug 7, 2018 at 11:39 AM, Ryan Sleevi <rsl...@chromium.org> wrote:
On Tue, Aug 7, 2018 at 2:22 PM Alex Gaynor <aga...@mozilla.com> wrote:
Since no one else is saying anything, I think continuing to post these is valuable, in that it highlights an ongoing issue with wide applicability.

It's alarming to me that neither CNNIC, nor Google, have said anything since this thread opened 10 days ago.

Was this been reported to the CNNIC log operator e-mail? We do not presently require that Log Operators monitor all activity on ct-policy@, nor have we ever.
 

Alex

On Tue, Aug 7, 2018 at 2:19 PM 'Brendan McMillion' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
Here is a third SCT, for this certificate. Please let me know if I should continue posting these as I find them, or if I should discard them.

On Sat, Aug 4, 2018 at 5:38 PM, Brendan McMillion <bre...@cloudflare.com> wrote:
Here's another, for this certificate.

On Sat, Jul 28, 2018 at 10:57 AM, Brendan McMillion <bre...@cloudflare.com> wrote:
Hello ct-policy@

Attached is what I believe to be an un-incorporated SCT from CNNIC for this certificate. crt.sh doesn't show that the log contains the certificate, and my get-proof-by-hash requests fail with status "400 Bad Request" and error message "Couldn't find hash."


--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

Devon O'Brien

unread,
Aug 8, 2018, 3:25:08 PM8/8/18
to Certificate Transparency Policy, rsl...@chromium.org, aga...@mozilla.com, cry...@cloudflare.com

Hi Brendan,


Thanks for these reports; we absolutely do value your bringing this to our attention. We wanted to investigate this matter and confirm before responding.


We have confirmed that these certificates are not incorporated in the CNNIC CT Log, which means that the Log is in violation of its MMD on multiple counts.


We’ve reached out to CNNIC, but while we wait to hear from them, we had a couple of questions for you regarding this find:


  1. How did you encounter these unincorporated entries? These certificates, while from the Google Compliance Monitor, were initially issued to Nimbus2020 and Argon2020 logs. Since you have the SCTs, this suggests you were the first to submit these entries to CNNIC’s Log.

  2. Is this effort a part of your CT Log monitoring scheme, and if so, are you performing this same checks on all pending/qualified Logs?


On Tuesday, August 7, 2018 at 12:23:40 PM UTC-7, Brendan McMillion wrote:
> Was this been reported to the CNNIC log operator e-mail?

No, I hadn't reported it to them. I sent an email to their contact address just now.

On Tue, Aug 7, 2018 at 11:39 AM, Ryan Sleevi <rsl...@chromium.org> wrote:
On Tue, Aug 7, 2018 at 2:22 PM Alex Gaynor <aga...@mozilla.com> wrote:
Since no one else is saying anything, I think continuing to post these is valuable, in that it highlights an ongoing issue with wide applicability.

It's alarming to me that neither CNNIC, nor Google, have said anything since this thread opened 10 days ago.

Was this been reported to the CNNIC log operator e-mail? We do not presently require that Log Operators monitor all activity on ct-policy@, nor have we ever.
 

Alex

On Tue, Aug 7, 2018 at 2:19 PM 'Brendan McMillion' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
Here is a third SCT, for this certificate. Please let me know if I should continue posting these as I find them, or if I should discard them.

On Sat, Aug 4, 2018 at 5:38 PM, Brendan McMillion <bre...@cloudflare.com> wrote:
Here's another, for this certificate.

On Sat, Jul 28, 2018 at 10:57 AM, Brendan McMillion <bre...@cloudflare.com> wrote:
Hello ct-policy@

Attached is what I believe to be an un-incorporated SCT from CNNIC for this certificate. crt.sh doesn't show that the log contains the certificate, and my get-proof-by-hash requests fail with status "400 Bad Request" and error message "Couldn't find hash."


--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

Brendan McMillion

unread,
Aug 8, 2018, 4:19:26 PM8/8/18
to Devon O'Brien, Certificate Transparency Policy, rsl...@chromium.org, Alex Gaynor, crypto
> How did you encounter these unincorporated entries?

Merkle Town estimates the uptime and response time of add-(pre-)chain endpoints by occasionally choosing a certificate we expect the log would accept, and submitting it. Typically no more than 4 in 10 minutes, but CNNIC definitely sees much less than that because they trust so few roots that it's hard to find certificates.

> Is this effort a part of your CT Log monitoring scheme, and if so, are you performing this same checks on all pending/qualified Logs?

All listed logs are monitored in this way.


To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

Kat Joyce

unread,
Aug 9, 2018, 6:54:58 AM8/9/18
to Brendan McMillion, Devon O'Brien, Certificate Transparency Policy, rsl...@chromium.org, Alex Gaynor, crypto
On Wed, Aug 8, 2018 at 9:19 PM 'Brendan McMillion' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
> How did you encounter these unincorporated entries?

Merkle Town estimates the uptime and response time of add-(pre-)chain endpoints by occasionally choosing a certificate we expect the log would accept, and submitting it. Typically no more than 4 in 10 minutes, but CNNIC definitely sees much less than that because they trust so few roots that it's hard to find certificates.

Just chiming in as I had a thought about this when I read it - if the goal of this is only to measure uptime and response time for those end points, you could submit a certificate that the Log already contains.
 

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

Ben Laurie

unread,
Aug 9, 2018, 8:45:34 AM8/9/18
to Kat Joyce, bre...@cloudflare.com, Devon O'Brien, Certificate Transparency Policy, Ryan Sleevi, aga...@mozilla.com, cry...@cloudflare.com
On Thu, 9 Aug 2018 at 11:54, 'Kat Joyce' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
On Wed, Aug 8, 2018 at 9:19 PM 'Brendan McMillion' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
> How did you encounter these unincorporated entries?

Merkle Town estimates the uptime and response time of add-(pre-)chain endpoints by occasionally choosing a certificate we expect the log would accept, and submitting it. Typically no more than 4 in 10 minutes, but CNNIC definitely sees much less than that because they trust so few roots that it's hard to find certificates.

Just chiming in as I had a thought about this when I read it - if the goal of this is only to measure uptime and response time for those end points, you could submit a certificate that the Log already contains.

"Response time" presumably includes time taken to merge?
 

Kat Joyce

unread,
Aug 9, 2018, 9:50:00 AM8/9/18
to be...@google.com, bre...@cloudflare.com, asymm...@google.com, ct-p...@chromium.org, rsl...@chromium.org, aga...@mozilla.com, cry...@cloudflare.com
On Thu, Aug 9, 2018 at 1:45 PM Ben Laurie <be...@google.com> wrote:
On Thu, 9 Aug 2018 at 11:54, 'Kat Joyce' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
On Wed, Aug 8, 2018 at 9:19 PM 'Brendan McMillion' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:
> How did you encounter these unincorporated entries?

Merkle Town estimates the uptime and response time of add-(pre-)chain endpoints by occasionally choosing a certificate we expect the log would accept, and submitting it. Typically no more than 4 in 10 minutes, but CNNIC definitely sees much less than that because they trust so few roots that it's hard to find certificates.

Just chiming in as I had a thought about this when I read it - if the goal of this is only to measure uptime and response time for those end points, you could submit a certificate that the Log already contains.

"Response time" presumably includes time taken to merge?

As Logs don't merge right away (that's the whole point of SCTs and the MMD right?) I wouldn't expect response time to include the time to merge.   However, Pierre did point out that the response time will vary depending on the code path being hit, which will be different if fetching a pre-existing SCT vs creating a new one and adding the cert to the queue to be merged, so yup, there may be variance in response time between adding a new cert and adding a pre-existing one.

Brendan McMillion

unread,
Aug 9, 2018, 3:15:42 PM8/9/18
to Kat Joyce, Ben Laurie, Devon O'Brien, Certificate Transparency Policy, rsl...@chromium.org, Alex Gaynor, crypto
> Just chiming in as I had a thought about this when I read it - if the goal of this is only to measure uptime and response time for those end points, you could submit a certificate that the Log already contains.

As you say in your later response, submitting new and old certificates trigger different code paths, which is why we submit both old and new certificates. It exposes interesting phenomena, like how the new Argon/Xenon logs are 4-5x slower to respond for new certificates than old, while most other logs have comparable response times in both cases. Or, on the other hand, how the Nimbus logs are actually slightly faster to respond to new certificates than old.

> "Response time" presumably includes time taken to merge?

No, response time is the how long from when the certificate is sent until an SCT comes back. An RP has to wait for the log to respond with the SCT, but isn't particularly affected by time-to-merge.


To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

Brendan McMillion

unread,
Aug 12, 2018, 4:01:06 PM8/12/18
to Kat Joyce, Ben Laurie, Devon O'Brien, Certificate Transparency Policy, rsl...@chromium.org, Alex Gaynor, crypto
349437580.cnnic.json
349437580.pem
635640010.cnnic.json
635640010.pem

Brendan McMillion

unread,
Aug 18, 2018, 6:20:46 PM8/18/18
to Kat Joyce, Ben Laurie, Devon O'Brien, Certificate Transparency Policy, rsl...@chromium.org, Alex Gaynor, crypto
628352817.cnnic.json
628352817.pem

Devon O'Brien

unread,
Aug 21, 2018, 1:12:09 PM8/21/18
to Certificate Transparency Policy, katj...@google.com, be...@google.com, asymm...@google.com, rsl...@chromium.org, aga...@mozilla.com, cry...@cloudflare.com

From July 28, 2018 to August 18, 2018, it was reported on several instances to ct-p...@chromium.org that the CNNIC CT Log had failed to incorporate multiple certificates into the Log’s merkle tree within 24 hours of issuing a corresponding SCT. In each case, the certificate entries remain unincorporated at time of reporting.


In each of these circumstances, the certificate in question was issued from the Google Merge Delay Intermediate 1 CA for the purpose of performing compliance monitoring of CT Logs other than CNNIC’s. These certificates were not directly logged by Google, but rather independently submitted by a third party for the purpose of CT Log monitoring. Brendan, the original reporter has indicated that these certificates were used for monitoring due to the difficulty of finding unlogged certificates chaining to a root accepted by the CNNIC CT Log.


Timeline:

July 28 - Initial report of SCT corresponding to an unincorporated certificate entry


August 4 - Second report of SCT corresponding to an unincorporated certificate entry, distinct from previous


August 7 - Third report of SCT corresponding to an unincorporated certificate entry, distinct from all previous


August 8 - Chrome reached out to CNNIC at their specified Log Operator email address and have received no response from CNNIC on either the public discussion list or via email.


August 12 - Fourth and fifth reports of SCTs corresponding to unincorporated certificate entries, distinct from all previous


August 18 - Sixth report of SCT corresponding to an unincorporated certificate entry, distinct from all previous


August 20 - Chrome reached out to CNNIC again at their specified Log Operator email address and have not yet received a response


Due to this pattern of events, coupled with the CNNIC CT Log Operator’s lack of response to or acknowledgement of this issue, the Chrome Security team is proposing to disqualify the CNNIC CT Log. This is not a decision we make lightly and we welcome input from the CT community as well as any relevant facts that may not yet have been presented pertaining to these incidents. We will wrap up discussion 1 week from this post (August 28) and shortly thereafter announce a decision on this matter.

Devon O'Brien

unread,
Sep 4, 2018, 4:17:32 PM9/4/18
to Certificate Transparency Policy, katj...@google.com, be...@google.com, asymm...@google.com, rsl...@chromium.org, aga...@mozilla.com, cry...@cloudflare.com

This matter has gone unacknowledged by CNNIC for over a month and there has been no new information provided since our August 21 post announcing our plan to disqualify the CNNIC CT Log. The demonstrable lack of inclusion of certificate entries for which CNNIC CT Log has issued SCTs is troubling enough, but the subsequent complete lack of acknowledgement or response to this issue indicates that the CNNIC CT Log Operators are not able or willing to uphold the requirements specified in Chromium’s CT Log Policy.


Because of this, the CNNIC CT Log (https://ctserver.cnnic.cn) will no longer be a qualified Log, effective September 18, 2018. No SCT from the CNNIC CT Log - past, present or future - will count towards the requirement that at least one SCT is qualified at time of check and, as such, will not be appropriate for inclusion within the TLS extension or embedded within a stapled OCSP response. SCTs that are dated prior to September 18, 2018 and embedded within existing will continue to count as once or currently qualified, but certificates issued on-or-after September 18, 2018 will need to ensure that there are a sufficient number of SCTs from logs which are qualified or pending qualification at time of issuance.


What does this mean for site operators


If you are delivering SCTs embedded in the certificate, this should require no action on your part. All certificates previously issued, and which contain SCTs issued by the CNNIC CT Log, that complied with the Certificate Transparency policy will continue to do so. While SCTs from the CNNIC CT Log will not be considered as being from a log qualified at the time of check, the presence of qualified Google logs in the certificate will ensure there is no disruption. If you refresh or renew your certificate, your CA should be including sufficient and diverse SCTs from other logs that it should require no action on your part.


If you are delivering SCTs embedded in the OCSP response, via OCSP stapling, then your CA must take appropriate action to update their OCSP pipeline to include at least one SCT from a non-Google operated log that is qualified at time of check. Once done, you must refresh the OCSP response stapled to the connection. Alternatively, you may also enable the TLS extension mechanism to deliver additional SCTs from non-Google operated logs, in order to ensure that at least one SCT from a non-Google log, qualified at time of check, is presented.


If you are delivering SCTs via a TLS extension, SCTs issued by the CNNIC CT Log will not be considered qualified at the time of check. As such, you must begin to serve SCTs from a different, non-Google log by September 18, 2018, in order to satisfy the CT requirements.


What does this mean for CAs


If you are embedding SCTs in your OCSP response, you must issue new OCSP responses before September 18, 2018, which replace the CNNIC SCTs with SCTs issued from another non-Google log. Your customers must then begin to serve this new response. Alternatively, your customers must include SCTs delivered via the TLS extension until you are able to update the SCTs included in your responses.


If you are embedding SCTs in your certificates, SCTs from the CNNIC CT Log for newly issued certificates will not count towards the minimum requirements after September 18, 2018, either in the number of SCTs from logs once or currently qualified or as a non-Google log. You must provide SCTs from a diverse set of logs which are qualified at the time of issuance. SCTs from the CNNIC CT Log will be ignored.


Reply all
Reply to author
Forward
0 new messages