Re: Possible SCT inclusion violation for DigiCert Nessie2023 log

[Chris Thompson]

Re-sending to ct-p...@chromium.org as my original message failed. See our message about a potential MMD inclusion violation below.

On Mon, Aug 7, 2023 at 12:15 PM Chris Thompson <cth...@google.com> wrote:

Chrome's SCT auditing system detected a possible MMD inclusion violation for DigiCert Nessie2023. We are reporting this to ct...@digicert.com and including ct-p...@chromium.org for public discussion.

When trying to validate the inclusion proof, we are seeing Nessie2023 returning "400 Bad Request" errors, which is preventing us from verifying inclusion. We have ingested all of Nessie2023's entries through the STH of 2023-08-07 05:57:12.198+00:00. We have no record of this SCT in our ingestion logs currently.

Chrome’s CT Log Policy (https://goo.gl/chrome/ct-log-policy) requires certificate entries to be incorporated within 24 hours of issuance. Serving HTTP status codes other than 200 for prolonged periods of time is also considered to be an outage since it interferes with confirming the previous requirement. We’d appreciate it if DigiCert could investigate this issue and report back on this list as soon as possible.

- Chris (on behalf of the Chrome Certificate Transparency team)

Alert details

Log name: DigiCert Nessie2023 Log

Inclusion deadline: Wed, 02 Aug 2023 20:14:16 +0000

Leaf hash: +3TL6p2CdpUXLwWIiw2xtrI08680tBoljwgcuxhr3cM=

-----BEGIN CERTIFICATE-----

MIIFejCCBGKgAwIBAgIRAMY7rUKRrOl6CT0mR1rLWfcwDQYJKoZIhvcNAQELBQAw

RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM

TEMxEzARBgNVBAMTCkdUUyBDQSAxRDQwHhcNMjMwODAxMTkxNDEwWhcNMjMxMDMw

MjAwMDU5WjAmMSQwIgYDVQQDExttaWRhdGxhbnRpY3Bvb2xidWlsZGVycy5jb20w

ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQClTN9YkETICVzdBPt+c5zp

+2PBzWSMVZ2Fv6YI+DBA5Sv4YTKCRcphyDmX3wbzEyVOp27YHV9sVphmE6+3jc/g

Yx7wtp73/cxF/3PrrnJn7AvUPovmjNkdOGz0jXWqP8b/qx0M3v97s82BmljxrYgM

H0kAVRIbMuz2+skzOnd2rYb6RGqVE8LgWokZAp+0q31J+b6HhNsDAZg0s7ZbN+Js

rWaD62+RC3bvvSE+HSc3uJzSEYmS5UlQuYJn4Z0TJ2bsjieAI/My1MNr+zYKJxIh

xP26K8lZ8fFfzbrOeZviucQP+Z3w2caARa0LWOwkmemPGrdsgc7AgzyawnNPfRed

AgMBAAGjggKBMIICfTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH

AwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUmqmiGkR1qy3z5CjJF6OwIkOTZFMw

HwYDVR0jBBgwFoAUJeIYDrJXkZQq5dRdhpCD3lOzuJIweAYIKwYBBQUHAQEEbDBq

MDUGCCsGAQUFBzABhilodHRwOi8vb2NzcC5wa2kuZ29vZy9zL2d0czFkNC9jYXlT

WnNYY1BGYzAxBggrBgEFBQcwAoYlaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMv

Z3RzMWQ0LmRlcjAmBgNVHREEHzAdghttaWRhdGxhbnRpY3Bvb2xidWlsZGVycy5j

b20wIQYDVR0gBBowGDAIBgZngQwBAgEwDAYKKwYBBAHWeQIFAzA8BgNVHR8ENTAz

MDGgL6AthitodHRwOi8vY3Jscy5wa2kuZ29vZy9ndHMxZDQvMzNiNGp3MXFvZGMu

Y3JsMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHYA6D7Q2j71BjUy51covIlryQPT

y9ERa+zraeF3fW0GvW4AAAGJsryU7wAABAMARzBFAiEA1uF9E6IuiM6qkSYiDNuw

xVWDq6rc5XWKb5g2bvvcI+sCIFbdLZ6P+XZ0nmXac3/VNuw7+kCtlz1SSiOL/Ex0

mQ4tAHUAs3N3B+GEUPhjhtYFqdwRCUp5LbFnDAuH3PADDnk2pZoAAAGJsryqUgAA

BAMARjBEAiBCMxy7YwazOG0jWgVQJcara5LzxuxJf5Pl7ZQfaWZX7wIgTroQlQMB

Pd/a4vL/gVQp2Mlm5hTLVYkizwwTvffDWSAwDQYJKoZIhvcNAQELBQADggEBABgg

diAWXJVG4Xu0aHwTj0pbXT+tRjrspnVHy1FKFTYmmTL1lSMidLsbyAZeJUgkvzWn

XDgd30ZHRd2a0iqBlDZFfq2CVCRpC3wEartetG+E+LN5w6B51obaTw8k12PFzhIE

5GvLETLM7cV3nyWkTCNoJU9ui4PunypwlnpT92i/Hq9qmJsHRc7SE5/GiTfRGqpl

o6S/OuqCaZWN+lmS5mAOGnJEuM8++8XHsdiRq+0jhYbU3QgIwHIawhjSdpU7P9UV

tqGLdoIP++QzotUtAmv0c4+iDfcurG9jfys/e4AlHMluGb0OuvuKsOehPbbPHv6k

az0jUWWiQOFkhGxGcrY=

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIFjDCCA3SgAwIBAgINAgCOsgIzNmWLZM3bmzANBgkqhkiG9w0BAQsFADBHMQsw

CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU

MBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAw

MDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp

Y2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFENDCCASIwDQYJKoZIhvcNAQEBBQAD

ggEPADCCAQoCggEBAKvAqqPCE27l0w9zC8dTPIE89bA+xTmDaG7y7VfQ4c+mOWhl

UebUQpK0yv2r678RJExK0HWDjeq+nLIHN1Em5j6rARZixmyRSjhIR0KOQPGBMUld

saztIIJ7O0g/82qj/vGDl//3t4tTqxiRhLQnTLXJdeB+2DhkdU6IIgx6wN7E5NcU

H3Rcsejcqj8p5Sj19vBm6i1FhqLGymhMFroWVUGO3xtIH91dsgy4eFKcfKVLWK3o

2190Q0Lm/SiKmLbRJ5Au4y1euFJm2JM9eB84Fkqa3ivrXWUeVtye0CQdKvsY2Fka

zvxtxvusLJzLWYHk55zcRAacDA2SeEtBbQfD1qsCAwEAAaOCAXYwggFyMA4GA1Ud

DwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0T

AQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUJeIYDrJXkZQq5dRdhpCD3lOzuJIwHwYD

VR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYG

CCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcw

AoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQt

MCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsME0G

A1UdIARGMEQwCAYGZ4EMAQIBMDgGCisGAQQB1nkCBQMwKjAoBggrBgEFBQcCARYc

aHR0cHM6Ly9wa2kuZ29vZy9yZXBvc2l0b3J5LzANBgkqhkiG9w0BAQsFAAOCAgEA

IVToy24jwXUr0rAPc924vuSVbKQuYw3nLflLfLh5AYWEeVl/Du18QAWUMdcJ6o/q

FZbhXkBH0PNcw97thaf2BeoDYY9Ck/b+UGluhx06zd4EBf7H9P84nnrwpR+4GBDZ

K+Xh3I0tqJy2rgOqNDflr5IMQ8ZTWA3yltakzSBKZ6XpF0PpqyCRvp/NCGv2KX2T

uPCJvscp1/m2pVTtyBjYPRQ+QuCQGAJKjtN7R5DFrfTqMWvYgVlpCJBkwlu7+7KY

3cTIfzE7cmALskMKNLuDz+RzCcsYTsVaU7Vp3xL60OYhqFkuAOOxDZ6pHOj9+OJm

YgPmOT4X3+7L51fXJyRH9KfLRP6nT31D5nmsGAOgZ26/8T9hsBW1uo9ju5fZLZXV

VS5H0HyIBMEKyGMIPhFWrlt/hFS28N1zaKI0ZBGD3gYgDLbiDT9fGXstpk+Fmc4o

lVlWPzXe81vdoEnFbr5M272HdgJWo+WhT9BYM0Ji+wdVmnRffXgloEoluTNcWzc4

1dFpgJu8fF3LG0gl2ibSYiCi9a6hvU0TppjJyIWXhkJTcMJlPrWx1VytEUGrX2l0

JDwRjW/656r0KVB02xHRKvm2ZKI03TglLIpmVCK3kBKkKNpBNkFt8rhafcCKOb9J

x/9tpNFlQTl7B39rJlJWkR17QnZqVptFePFORoZmFzM=

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIFVzCCAz+gAwIBAgINAgPlk28xsBNJiGuiFzANBgkqhkiG9w0BAQwFADBHMQsw

CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU

MBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAw

MDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp

Y2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjEwggIiMA0GCSqGSIb3DQEBAQUA

A4ICDwAwggIKAoICAQC2EQKLHuOhd5s73L+UPreVp0A8of2C+X0yBoJx9vaMf/vo

27xqLpeXo4xL+Sv2sfnOhB2x+cWX3u+58qPpvBKJXqeqUqv4IyfLpLGcY9vXmX7w

Cl7raKb0xlpHDU0QM+NOsROjyBhsS+z8CZDfnWQpJSMHobTSPS5g4M/SCYe7zUjw

TcLCeoiKu7rPWRnWr4+wB7CeMfGCwcDfLqZtbBkOtdh+JhpFAz2weaSUKK0Pfybl

qAj+lug8aJRT7oM6iCsVlgmy4HqMLnXWnOunVmSPlk9orj2XwoSPwLxAwAtcvfaH

szVsrBhQf4TgTM2S0yDpM7xSma8ytSmzJSq0SPly4cpk9+aCEI3oncKKiPo4Zor8

Y/kB+Xj9e1x3+naH+uzfsQ55lVe0vSbv1gHR6xYKu44LtcXFilWr06zqkUspzBmk

MiVOKvFlRNACzqrOSbTqn3yDsEB750Orp2yjj32JgfpMpf/VjsPOS+C12LOORc92

wO1AK/1TD7Cn1TsNsYqiA94xrcx36m97PtbfkSIS5r762DL8EGMUUXLeXdYWk70p

aDPvOmbsB4om3xPXV2V4J95eSRQAogB/mqghtqmxlbCluQ0WEdrHbEg8QOB+DVrN

VjzRlwW5y0vtOUucxD/SVRNuJLDWcfr0wbrM7Rv1/oFB2ACYPTrIrnqYNxgFlQID

AQABo0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4E

FgQU5K8rJnEaK0gnhS9SZizv8IkTcT4wDQYJKoZIhvcNAQEMBQADggIBAJ+qQibb

C5u+/x6Wki4+omVKapi6Ist9wTrYggoGxval3sBOh2Z5ofmmWJyq+bXmYOfg6LEe

QkEzCzc9zolwFcq1JKjPa7XSQCGYzyI0zzvFIoTgxQ6KfF2I5DUkzps+GlQebtuy

h6f88/qBVRRiClmpIgUxPoLW7ttXNLwzldMXG+gnoot7TiYaelpkttGsN/H9oPM4

7HLwEXWdyzRSjeZ2axfG34arJ45JK3VmgRAhpuo+9K4l/3wV3s6MJT/KYnAK9y8J

ZgfIPxz88NtFMN9iiMG1D53Dn0reWVlHxYciNuaCp+0KueIHoI17eko8cdLiA6Ef

MgfdG+RCzgwARWGAtQsgWSl4vflVy2PFPEz0tv/bal8xa5meLMFrUKTX5hgUvYU/

Z6tGn6D/Qqc6f1zLXbBwHSs09dR2CQzreExZBfMzQsNhFRAbd03OIozUhfJFfbdT

6u9AWpQKXCBfTkBdYiJ23//OYb2MI3jSNwLgjt7RETeJ9r/tSQdirpLsQBqvFAnZ

0E6yove+7u7Y/9waLd64NnHi/Hm3lCXRSHNboTXns5lndcEZOitHTtNCjv0xyBZm

2tIMPNuzjsmhDYAPexZ3FL//2wmUspO8IFgV6dtxQ/PeEMMA3KgqlbbC1j+Qa3bb

bP6MvPJwNQzcmRk13NfIRmPVNnGuV/u3gm3c

-----END CERTIFICATE-----

[Andrew Ayer]

I am also seeing numerous unincorporated SCTs from Nessie 2023, for
example the SCTs embedded in these certificates:

https://crt.sh/?sha256=2219d2fc3b455c114c29fd919345f0e7d98e40088fa14be10f03c43f884347f6 (https://nessie2023.ct.digicert.com/log/ct/v1/get-proof-by-hash?hash=s9a%2B6o8IsgQtfctggiP%2FBl%2FLo%2BFf0A7HbjWIMNRhYb0%3D&tree_size=284218335)

https://crt.sh/?sha256=b5a1e2c87a1114364813efefbf24d50f2a8aa07c04e869b442686e1fc1fa0d19 (https://nessie2023.ct.digicert.com/log/ct/v1/get-proof-by-hash?hash=XuEGJxbmc2e1SqU7Foj6Mu%2BxB0Rm5QbbtVH%2F2UgTaSI%3D&tree_size=284237629)

https://crt.sh/?sha256=fa1370bb05a3fc5ddd03d4d642fb3e711f66e9395ec9ff3739e45849dbbcc35a (https://nessie2023.ct.digicert.com/log/ct/v1/get-proof-by-hash?hash=heSQAv8zhHiTPPQIKBhEWnIcPVq%2BAFPOEVRSsY1ZAEc%3D&tree_size=284237629)

The SCTs were issued at around the same time as the unincorporated SCTs
reported by Google and Cloudflare. It's not just the proof endpoints
which are failing; my monitor has not observed these Merkle Tree leafs
being returned by get-entries either.

Regards,
Andrew

On Mon, 7 Aug 2023 12:19:32 -0700
"'Chris Thompson' via Certificate Transparency Policy"

<ct-p...@chromium.org> wrote:

> Re-sending to ct-p...@chromium.org as my original message failed.
> See our message about a potential MMD inclusion violation below.
>

> On Mon, Aug 7, 2023 at 12:15___PM Chris Thompson <cth...@google.com>

> wrote:
>
> > Chrome's SCT auditing system detected a possible MMD inclusion
> > violation for DigiCert Nessie2023. We are reporting this to
> > ct...@digicert.com and including ct-p...@chromium.org for public
> > discussion.
> >
> > When trying to validate the inclusion proof, we are seeing
> > Nessie2023 returning "400 Bad Request" errors, which is preventing
> > us from verifying inclusion. We have ingested all of Nessie2023's
> > entries through the STH of 2023-08-07 05:57:12.198+00:00. We have
> > no record of this SCT in our ingestion logs currently.
> >

> > Chrome___s CT Log Policy (https://goo.gl/chrome/ct-log-policy)

> > requires certificate entries to be incorporated within 24 hours of
> > issuance. Serving HTTP status codes other than 200 for prolonged
> > periods of time is also considered to be an outage since it

> > interferes with confirming the previous requirement. We___d

> --
> You received this message because you are subscribed to the Google
> Groups "Certificate Transparency Policy" group. To unsubscribe from
> this group and stop receiving emails from it, send an email to
> ct-policy+...@chromium.org. To view this discussion on the
> web visit
> https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CANLpndFAcnpqbt9qLk-m-Ze%2BkaGBdbwwd8BkRCJz1Xp5tJS3qg%40mail.gmail.com.

[Jeremy Rowley]

Posting here as well as replying to the email. 

Acknowledged and investigating. So far, it looks like the log is getting hit with too many requests a second to respond to them. We think its still merging SCTs correctly though. Wlil post additional details as I know them.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20230807140147.60b68bbe6c04b4eff27dbd52%40andrewayer.name.

[Jeremy Rowley]

We confirmed that there are SCTs missing from Nessies database. We are unsure why. This does mean that Nessie missed its MMD for these certs. We are still investigating and will report back when we figure out the root cause.

[Chris Thompson]

Thanks for the updates Jeremy.

We're still seeing "400 Bad Request" responses when trying to check inclusion proofs, so do also let us know when the load issues have abated.

- Chris

[Chris Thompson]

Yes, we think the best next step would be to pause accepting new certificates into the log to reduce the potential impact on newly issued certificates, as we will very likely need to distrust the log (the path to recovery and being convinced that we can fully enumerate the affected certificates seems challenging at best).

Best of luck with the continued investigation.

- Chris

On Tue, Aug 8, 2023 at 2:10 PM Jeremy Rowley <jeremy...@digicert.com> wrote:

Given that the MMD was missed, this log should be distrusted per the CT policy. Would you like us to stop accepting new submissions?