StartCom Log misbehaving: Failure to incorporate SCTs

[Ryan Sleevi]

As a continuing part of improving compliance monitoring of the CT Ecosystem, Google has been investigating the presentation of SCTs by servers and certificates and whether or not they are incorporated within the Log, as required.

During this investigation, a pattern of incidents has been discovered with the StartCom Log ( https://bugs.chromium.org/p/chromium/issues/detail?id=611672 ) that show a sustained pattern of non-compliant behaviour. We understand this pattern has also recently been observed by other participants in the CT ecosystem.

Some certificates have never been included in the log, while others have appeared within a tree head substantially later than permitted. Both are violations of the MMD property, but are both concerning. 

These findings have been collected at https://docs.google.com/document/d/1VUS84NjsAjPcz8rutWfK1jX36cc5QN7QDXl1pSymA4Q/edit?usp=sharing

We'd greatly appreciate the thoughtful feedback from the community regarding this incident.

Our proposed steps are as follows:

- StartCom to provide an Incident Report regarding this matter, including both the underlying cause, why StartCom fails to proactively detect it from their own systems monitoring, and any facts surrounding the SCT for https://crt.sh/?id=84153049 and its substantial delay in inclusion

- Freeze (Disqualify) the log on [a date to be determined].

  - Any SCTs issued on or after that date will not count towards "qualified at time of certificate issuance" or "qualified at time of check"

  - Any SCTs issued prior to that will count as "once qualified" (disqualified prior to check)

As noted, we welcome feedback on this proposal, and look to draw discussion on the proposed next steps to a conclusion within the next few days, to provide greater clarity to the ecosystem as to what actions are required. We hope others will take the opportunity to verify our data, as well as share any additional information they may find relevant to such discussions.

From an impact to end users:

- The requirement of multiple SCTs ensures that such certificates have been logged in one or more other (non-StartCom) logs. This example underscores the importance of CT redundancy, by ensuring that a single failure does not undermine the transparency requirements or goals. As such, the security of end users has not been undermined by this incident alone.

From an impact to site operators:

- If adopted as proposed, site operators employing the TLS extension will need to obtain SCTs from a different non-Google log to serve on or after [a date to be determined]. SCTs from StartCom's log after that date will no longer be accepted.

From an impact to CAs:

- If embedding SCTs in OCSP responses, CAs will need to obtain SCTs from a different non-Google log to serve on or after [a date to be determined]. SCTs from StartCom's log after that date will no longer be accepted.

- If embedding SCTs in Certificates:

  - If the only SCTs included were from Logs that were once qualified, but no SCTS are qualified at time of check, the certificate will no longer be CT Qualified. As a practical matter, this means any certificates which only embedded SCTs from the Google Aviator Log and StartCom log would be affected.

  - If at least one SCT is from a log still qualified in Chrome, and the certificate otherwise met the CT Qualified requirement, there will be no change to the CT Qualified status. However, CAs should be aware that if the remaining CT Logs included are all disqualified, that certificate will no longer be CT Qualified.

  - CAs should update their systems no later than [a date to be determined] to obtain SCTs from one or more non-Google logs other than the StartCom log, to ensure newly issued certificates meet the requirement that the certificates are qualified at time of issuance.

What are the factors we considered in this proposal:

- As captured during the previous discussion of Aviator, in https://groups.google.com/a/chromium.org/d/msg/ct-policy/lsZqdM_u7DI/qAsiRHkgCAAJ , an MMD violation may or may not be an appropriate reason to freeze or disqualify a log, depending on the surrounding factors.

- In this particular case, what we observe appears to be a sustained practice of 'losing' certs and failing to incorporate them, along with a troubling 'retroactive' logging, without any corresponding incident report or discussion.

- Based on external data provided, which we believe will be shared soon, we believe this incident is still ongoing and unresolved. As such, a corrective step to 'freeze' the log provides mitigation against future risk of undisclosed certificates.

- Unlike some of the other discussions regarding uptime, the MMD failure is accompanied by cryptographic evidence of failure, so there is no ambiguity as to the expectations.

That said, we look forward to StartCom's Incident Report, so that as a collective ecosystem, the situation and facts surrounding this Incident can be used to improve and enhance Logs' implementations, including that of any future log that StartCom may run, which they are welcome to do so.

[Kurt Roeckx]

On Tue, Dec 12, 2017 at 04:37:39PM -0500, Ryan Sleevi wrote:
>
> Our proposed steps are as follows:
> - StartCom to provide an Incident Report regarding this matter, including
> both the underlying cause, why StartCom fails to proactively detect it from
> their own systems monitoring, and any facts surrounding the SCT for
> https://crt.sh/?id=84153049 and its substantial delay in inclusion
> - Freeze (Disqualify) the log on [a date to be determined].
> - Any SCTs issued on or after that date will not count towards "qualified
> at time of certificate issuance" or "qualified at time of check"
> - Any SCTs issued prior to that will count as "once qualified"
> (disqualified prior to check)

I'm not sure what you mean with this last line. SCTs issued before
that day will still be used to get to the total required after
that day? Other text in your mail seem to indicate that it won't.

Not actually incorparting SCTs means I don't actually trust that
log. But as long there is an SCT from a log I still trust, I guess
that's fine.


Kurt

[Ryan Sleevi]

On Tue, Dec 12, 2017 at 5:32 PM, Kurt Roeckx <ku...@roeckx.be> wrote:

On Tue, Dec 12, 2017 at 04:37:39PM -0500, Ryan Sleevi wrote:
>
> Our proposed steps are as follows:
> - StartCom to provide an Incident Report regarding this matter, including
> both the underlying cause, why StartCom fails to proactively detect it from
> their own systems monitoring, and any facts surrounding the SCT for
> https://crt.sh/?id=84153049 and its substantial delay in inclusion
> - Freeze (Disqualify) the log on [a date to be determined].
>   - Any SCTs issued on or after that date will not count towards "qualified
> at time of certificate issuance" or "qualified at time of check"
>   - Any SCTs issued prior to that will count as "once qualified"
> (disqualified prior to check)

I'm not sure what you mean with this last line. SCTs issued before
that day will still be used to get to the total required after
that day? Other text in your mail seem to indicate that it won't.

The definition of Chrome's current CT policy requires at least one SCT be qualified at time of check, while permitting, if embedded, at least one SCT from a Google log once or currently qualified, one SCT from a non-Google log once or currently qualified, and a total number of SCTs related to the lifetime. The logs must be qualified at time of issuance of the certificate.

Thus, disqualifying on a particular date means that existing SCTs retain their "once or currently qualified" status (as they were "once qualified"), while future SCTs would not meet the "once or currently qualified" definition (because the log would not be qualified at time of issuance, as evidenced by the SCT being produced after that timestamp)

I'm not sure if that makes it clearer - put differently, existing SCTs work (even though they're met with skepticism), new SCTs don't work, and the system is fine because the other (still qualified) logs provide redundancy guarantees as to the level of transparency.

 

Not actually incorparting SCTs means I don't actually trust that
log. But as long there is an SCT from a log I still trust, I guess
that's fine.

Exactly. 

[startcom ct operator]

Because of what had happened, we decide to freeze Startcom 's log after 24 hours.
This freeze include follow interface:

• add-chain
• add-pre-chain

other interface is still provide services.

And we will provide incident report after we find out  the reason.

[Brendan McMillion]

Thank you for starting this thread. Cloudflare started monitoring this
log slightly less than three weeks ago. Since then, we've submitted
2170 chains to it (0 precerts) and of those, 2 of the returned SCTs
have not been incorporated into an STH. Both of these are attached.

> --
> You received this message because you are subscribed to the Google Groups
> "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ct-policy+...@chromium.org.
> To post to this group, send email to ct-p...@chromium.org.
> To view this discussion on the web visit
> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/272cfd5a-b280-4e2f-b6d9-d6f94c9d675c%40chromium.org.

[Andrew Ayer]

I have audited every single embedded SCT found in public Certificate
Transparency logs. I found 85 pre-certificate SCTs that StartCom (log
ID NLtq1sPfnAPuqKSZ/3iRSGydXlysktAfe/0bzhnbSO8=) failed to incorporate.

I've attached evidence of these SCTs. It's a JSON array of objects
containing:

* sct: The SCT (in JSON, per RFC6962 Section 4.1)
* precert:
* issuer_key_hash: the issuer key hash, in base64
* tbs_certificate: the pre-certificate TBSCertificate, in base64

Regards,
Andrew

[Iñigo Barreira]

Hi all,

we´re still investigating the issue and will try tro provide an incident report soon. 

Regards

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20171218083653.f833215b28457fb2ada0bfaa%40andrewayer.name.