Venafi has produced inconsistent STHs

[Andrew Ayer]

Venafi (log ID rDua7X+pZ0dXFZ5tfVdWcvnZgQCUHpve/+yhMTt1eC0=) has signed
the following STHs (also attached):

{
"tree_size": 90161,
"timestamp": 1488320541206,
"sha256_root_hash": "I7PzqT8cEPjlfgOx3/2VxnuO5BTIJ6Zf8OM8DAwGhuM=",
"tree_head_signature": "BAEBAAgJjfMWJ0agR0VCpB1v8UJs3X8H3dPU8C0NmUMtMAdAm27tgiDcpDECHCMrJbIIy7v+y1p4zVGC6eWf1Tew+W+O32WinshHb9th7lLNlQ5yJUKW5UtMTJxW/BiFpHO75+WXsajb4EyTPkoJ1M2qxDX/cK/hSb2ar0W7G9weRaw7WetwEAG7pv2j/tnUUGXHWfnNkg6f40GkwSaGfY9Xw9gaZDeUawvS8T61qODsaZprWDsBAWaClaaelrmlx0kkZBnMP/LKKSlywrTNEt3Ow1oKBqT3nL5A2fTPoRfsF+1OGyy2iokX2pI1ZLBRS4v4rqMce/dgzXxrS6OUUINHfLA="
}

{
"tree_size": 90167,
"timestamp": 1488321064440,
"sha256_root_hash": "sMnqsYxLYyKdF7QNqplWgnIJk6WBsuqlEGo0aNHHhRg=",
"tree_head_signature": "BAEBAIXJw+INMcxEzrsW3+tzgxiE2jP8sJZN5HYwO3MmuFHy5t57lJqPADYEFBIfx7cEEGyp0RQLb6Wonvkq4xhsANt0xXCQQw1STxRKixnaYtSnEkoVJfWXPAVyjUqFPx+ryOHtdocMGlf60rt4UmGK0rhy9XTCrktYFkzG9ZPrGujGDvXIeBnIaarbU0L5g224rsdLvDXwpRopHpHinomsc5W+ReA+eQ38WGBHkz33e/+YPJQGd/L3EaDN3da0VIEYNHY5AGW/Yq7rU6cD+dHH9mL1OK971/HZ362aEnBBZ47zi/1OzANeKdT1tyB+db7VpXhScJxw1o0G3T0V0P+lM4o="
}

However, it is unable to provide a valid consistency proof between
these STHs. For reference, here's the consistency proof provided by
https://ctlog.api.venafi.com/ct/v1/get-sth-consistency?first=90161&second=90167:

{
"consistency" : [
"r+r7qrYh4i9lWa1nzVO86nj+9c6BGZq5i36IRzgPGl4=",
"NPLfppxk5a1xQZb/tmCK3oJ/WFflOaxx4pNE49Tf+8Y=",
"U0IGCc5N616P/zKPQVkFZkYdRoSBmLipCgspE0vNLeE=",
"kZiqCHdVXWXLAmFtPPm6tNUBm3ciQY7KTkBIPdPyP50=",
"im9E++S7hURbeazY1bHGqxx6/5zYmDmpLhCU3lAvGOM=",
"VH8nj3DKzwv9v3bcQeDKjw4a0H+GbIm1SIxU0z3J/1E=",
"2eQpspAlOaHniWW6exNICgPfge2u0BHXAU0bJoCxy+c=",
"jsEyi+238H+/SNiDNmoNeGTPeMkBEeM55WNKuuPrfeY=",
"rcxt0aKSfv5MGCDJSwt6WDZffq9Nmp4SpQm5Xr376Lo="
]
}

It may also be of interest that the certificate Venafi is currently
serving to me at log index 90160 is not the same certificate that I
previously recorded Venafi as having at this index.

Regards,
Andrew

[Ryan Sleevi]

Thanks for this heads up, Andrew.

I'm hoping the Venafi team can provide feedback, while as a community, we determine appropriate next steps.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20170228160048.16407cf750429c906b86d319%40andrewayer.name.

[Steve Topletz]

Hello,

Yes, Venafi is aware of this issue and has been tracking it since the AWS S3 outage earlier today. While the log server is functioning as expected now , we ran into a database inconsistency issue as part of the effort to get back online as quickly as possible. To be clear, we have confirmed that NO certificate data has been lost. We are in the process of identifying/confirming root cause, and will post a full incident response, as well as what we are doing to negate the possibility of this impacting us again.

Regards,

Steve Topletz

[Steve Topletz]

SUMMARY:

On February 28th of 2017 at 22:23:32 GMT the AWS S3 outage caused Venafi ‘ctlog.api.venafi.com’ log to generate and publish inconsistent STHs. This represented a violation of section 3 of RFC 6962 which states:

A log is a single, ever-growing, append-only Merkle Tree of such certificates.

This was triggered by automatic data restoration from S3 due to the recovery process of a terminated instance on an Amazon EC2 auto-scaling group. While this did not result in the loss of any logged certificates, as confirmed by load balancer logs, it did result in an inconsistent tree head being served from the log for approximately two minutes.

IMPACT:

The inconsistent tree head was served to nine monitors.

ROOT CAUSE:

The following is a chronological account (times in GMT) of the cause for the inconsistent STH generation.

17:36 The last successful backup of the CT logs were picked up and published. 

17:38 The Amazon S3 API failed when performing backups of the log, and continued to fail with the database continuing to update.

18:42 The correct signed tree head was made available. (timestamp 1488307345708)

18:55 The instance running CT log was automatically terminated by the AWS auto-scaling group.

22:22 Venafi CT log went out of synchronization and signed the incorrect tree head because it pulled the last successful backup which did not reflect the current state of the database.

22:23 Venafi CT log server made the incorrect STH available. (timestamp 1488320541206)

22:25 Venafi became aware of the incorrect STH and stopped the instance.

22:31 Venafi restored the correct backup and the server generated a new correct STH.

22:32 Venafi CT log server made the correct STH available. (timestamp 1488321064440)

REMEDIATION AND PREVENTION:

The current architecture of the Venafi CT server assumes periodic back-ups of the certificate log data in AWS S3, which leaves us susceptible to the possibility of incomplete archives in the event of outages like what we saw with Amazon this week. While no certificate data was lost, in our urgency to restore services to a functional state, we missed checking on the consistency of the backed-up data prior to restore operations. In retrospect, we should have performed a manual merge from our last back-up to ensure that we didn’t miss any data, which, while potentially slower, would negate the possibility of inconsistent tree heads.

 

In addition to implementing this procedure in the unlikely event of another S3 outage, we are evaluating architectural changes, and mitigating the dependency on a single source of failure like S3. Our new log server, submitted to Google earlier this year, is built on a distributed CT log infrastructure that is not predisposed to these types of outages and their resulting inconsistencies. We will also be reaching out to Root CAs impacted by this incident, including the remediation steps we have taken, once we have determined a course of action.

[Ryan Sleevi]

On Wed, Mar 1, 2017 at 10:13 PM, Steve Topletz <steve....@venafi.com> wrote:

SUMMARY:

On February 28th of 2017 at 22:23:32 GMT the AWS S3 outage caused Venafi ‘ctlog.api.venafi.com’ log to generate and publish inconsistent STHs. This represented a violation of section 3 of RFC 6962 which states:

A log is a single, ever-growing, append-only Merkle Tree of such certificates.

This was triggered by automatic data restoration from S3 due to the recovery process of a terminated instance on an Amazon EC2 auto-scaling group. While this did not result in the loss of any logged certificates, as confirmed by load balancer logs, it did result in an inconsistent tree head being served from the log for approximately two minutes.

IMPACT:

The inconsistent tree head was served to nine monitors.

ROOT CAUSE:

The following is a chronological account (times in GMT) of the cause for the inconsistent STH generation.

17:36 The last successful backup of the CT logs were picked up and published. 

17:38 The Amazon S3 API failed when performing backups of the log, and continued to fail with the database continuing to update.

18:42 The correct signed tree head was made available. (timestamp 1488307345708)

18:55 The instance running CT log was automatically terminated by the AWS auto-scaling group.

22:22 Venafi CT log went out of synchronization and signed the incorrect tree head because it pulled the last successful backup which did not reflect the current state of the database.

Hi Steve,

Thanks for sharing this level of detail, as it's useful for both log operators and the ecosystem to understand the risks and challenges.

I'm having a little trouble understanding the timeline here with your remark that this did not result in a loss of any logged certificates, so I'm just trying to work to understand.

Are you saying you have all certificates (and corresponding SCTs) issued from the time period 17:38 to 22:25 still available? That is, that you can demonstrate an inclusion proof (to the incorrect STH) for all of those certificates (with the 'bad' SCTs), with the fullness of the 'bad' tree, and similarly, demonstrate a subsequent inclusion proof that all of those certificates are incorporated into the STH generated 22:32 (or later), with the fullness of the 'good' tree?

Can you state how many certificates this affected?

[Ryan Sleevi]

On Thu, Mar 2, 2017 at 1:33 AM, Ryan Sleevi <rsl...@chromium.org> wrote:

Hi Steve,

Thanks for sharing this level of detail, as it's useful for both log operators and the ecosystem to understand the risks and challenges.

I'm having a little trouble understanding the timeline here with your remark that this did not result in a loss of any logged certificates, so I'm just trying to work to understand.

Are you saying you have all certificates (and corresponding SCTs) issued from the time period 17:38 to 22:25 still available? That is, that you can demonstrate an inclusion proof (to the incorrect STH) for all of those certificates (with the 'bad' SCTs), with the fullness of the 'bad' tree, and similarly, demonstrate a subsequent inclusion proof that all of those certificates are incorporated into the STH generated 22:32 (or later), with the fullness of the 'good' tree?

Can you state how many certificates this affected?

Just wanting to touch base on this point, as we determine the next step.

It would seem that, on the basis of the evidence presented, it's appropriate to disqualify Venafi. What's unclear, and where part of the question came from, is whether consistency to the STH produced at 1488307345708 can be produced, or whether it's necessary to disqualify at an earlier STH.

I'm curious how others feel about this proposed course of action.

[Andrew Ayer]

Hi Ryan,

I agree that disqualification is appropriate. I would ask that Venafi
provide the STH at timestamp 1488307345708 so that I and other monitors
can verify its consistency. Assuming no inconsistency can be proven, I
think it's appropriate to disqualify at that STH.

Regards,
Andrew

[Kurt Roeckx]

The STH before that would be at 1488300146134 (2017-02-28
16:42:26.134+00), with a tree size of 90155.


Kurt

[Ben Laurie]

I think this is a super interesting case.

If I am understanding correctly, Venafi's log now contains all certificates they issued an SCT for.

The incorrect STH was issued for a log that consisted of a subset of certificates accepted.

Later STHs include all those certificates and more, but in a different order, and hence have an STH that cannot be linked to the incorrect STH. (Note: this is deduced from the nature of the failure)

However, if it can be shown that the incorrect STH corresponds to a subset of certs now logged, then it seems to me that this entirely meets the intent of the system, if not the letter.

And hence I would argue against disqualification, pending the above demonstration.

 

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACvaWvYgVWPuRvSdMtXh2127HQ8frOkcLOjBgGmc-5Tn6u9J1A%40mail.gmail.com.

[Ben Laurie]

On 6 March 2017 at 21:06, Ben Laurie <be...@google.com> wrote:

On 6 March 2017 at 19:01, Ryan Sleevi <rsl...@chromium.org> wrote:

On Thu, Mar 2, 2017 at 1:33 AM, Ryan Sleevi <rsl...@chromium.org> wrote:

Hi Steve,

Thanks for sharing this level of detail, as it's useful for both log operators and the ecosystem to understand the risks and challenges.

I'm having a little trouble understanding the timeline here with your remark that this did not result in a loss of any logged certificates, so I'm just trying to work to understand.

Are you saying you have all certificates (and corresponding SCTs) issued from the time period 17:38 to 22:25 still available? That is, that you can demonstrate an inclusion proof (to the incorrect STH) for all of those certificates (with the 'bad' SCTs), with the fullness of the 'bad' tree, and similarly, demonstrate a subsequent inclusion proof that all of those certificates are incorporated into the STH generated 22:32 (or later), with the fullness of the 'good' tree?

Can you state how many certificates this affected?

Just wanting to touch base on this point, as we determine the next step.

It would seem that, on the basis of the evidence presented, it's appropriate to disqualify Venafi. What's unclear, and where part of the question came from, is whether consistency to the STH produced at 1488307345708 can be produced, or whether it's necessary to disqualify at an earlier STH.

I'm curious how others feel about this proposed course of action.

I think this is a super interesting case.

If I am understanding correctly, Venafi's log now contains all certificates they issued an SCT for.

The incorrect STH was issued for a log that consisted of a subset of certificates accepted.

Later STHs include all those certificates and more, but in a different order, and hence have an STH that cannot be linked to the incorrect STH. (Note: this is deduced from the nature of the failure)

However, if it can be shown that the incorrect STH corresponds to a subset of certs now logged, then it seems to me that this entirely meets the intent of the system, if not the letter.

And hence I would argue against disqualification, pending the above demonstration.

BTW, obviously the certificates in question and the times at which they were logged should be otherwise unremarkable. :-)

[Ryan Sleevi]

On Mon, Mar 6, 2017 at 4:06 PM, 'Ben Laurie' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

I think this is a super interesting case.

If I am understanding correctly, Venafi's log now contains all certificates they issued an SCT for.

I don't believe that's a correct understanding at this time.

 

The incorrect STH was issued for a log that consisted of a subset of certificates accepted.

Later STHs include all those certificates and more, but in a different order, and hence have an STH that cannot be linked to the incorrect STH. (Note: this is deduced from the nature of the failure)

However, if it can be shown that the incorrect STH corresponds to a subset of certs now logged, then it seems to me that this entirely meets the intent of the system, if not the letter.

And hence I would argue against disqualification, pending the above demonstration.

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

[Ben Laurie]

I'm not sure what you mean by a bad SCT?

 

[Ryan Sleevi]

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

While I appreciate the perspective of a 'recoverable' error, I think this unduly places burden on implementers - whether clients or monitors - to recognize and appropriately handle this scenario, and so I don't think it's necessarily reasonable or desirable to explore that line of reasoning. While there may be elements of less critical impact - as discussed in the past, a blown MMD _may_ be such an incident - producing a split view of the log is arguably one for which a log has difficulty recovering from. This is also consistent with past action and past discussion, so I'm not sure what new details or policy changes would accommodate the path you proposed. 

[Ben Laurie]

On 6 March 2017 at 21:30, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

I'm asking what a bad SCT is.

An SCT is a promise to include into a log. It is not related to any particular STH. Hence my confusion...

[Ryan Sleevi]

On Mon, Mar 6, 2017 at 4:36 PM, 'Ben Laurie' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

On 6 March 2017 at 21:30, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

I'm asking what a bad SCT is.

An SCT is a promise to include into a log. It is not related to any particular STH. Hence my confusion...

A 'bad' SCT in this case would be a promise issued after the production of the 'bad' STH. We can know - from the existence of the STH - the number of SCTs produced up to that point. However, any SCTs issued between the production of two STHs cannot be publicly known - ergo, they could be omitted from the log.

Imagine an 'evil' log (and I'm not accusing Venafi of this, I'm highlighting the challenge)

SCT A

SCT B

SCT C

STH (covering A-B-C)

SCT D

SCT E

SCT F

"Bad" STH (covering D-E-F)

SCT Z

SCT E

SCT D

SCT F

"Good" STH ( covering E-D-F)

We can show that the 'bad' and 'good' STHs fully incorporate the SCTs issued for certificates D/E/F, that's true - but the community cannot know about SCT Z unless/until it's produced, and no consistency proof can be provided for it.

I can understand and appreciate that you may argue this is true even under 'good' conditions, but I think the timeline of events provided makes it possible for this in a way that cannot be detected by comparing 'bad' STH to 'good' STH, and may represent additional operational failure.

[Andrew Ayer]

On Mon, 6 Mar 2017 21:06:22 +0000

"'Ben Laurie' via Certificate Transparency Policy"
<ct-p...@chromium.org> wrote:

Hi Ben,

I agree that the intent of logging all certificates has not
been violated. However, one of the most appealing aspects of CT
is that it allows programmatic verification that a log is meeting
its security properties. If Venafi is not disqualified, it will no
longer be possible to programmatically verify its append-only property.
Any monitor/auditor that sees the inconsistent STH will raise the alarm
that Venafi is misbehaving. (Thanks to gossip, this STH is never going
away.) In theory, we could develop an STH revocation system so that
monitors/auditors know to ignore this STH. (The system would also need
to convey the previous good STH so monitors can rewind.) Such a
system would take time to develop and add complexity to CT. Is it
worth it? Why can't logs be expected to be robust enough not to
accidentally sign STHs?

For recoverable policy violations, such as downtime, I'm in favor
of considering disqualification on a case-by-case basis. However, once
a log signs an STH it can't take it back. Therefore, signing
inconsistent STHs is not a recoverable error and should always lead to
disqualification.

Regards,
Andrew

[Steve Topletz]

Ryan, et al

This is to provide additional information about Venafi’s outage.  The sequence of events is described in Venafi’s post from March, 1^st.

 

Please note, between 1488320541206 and 1488321064440 no SCTs were issued. The tree signed at 1488321064440 is identical to 1488307345708. Please find below the STH blobs for 1488300146134 and 1488307345708 and their consistency proof.

[image001.png]

 

Incident Timeline: 

[image002.png] 

Specifically:

 

16:42 – 1488300146134 (tree size: 90,155):

080012220A20AC3B9AED7FA9674757159E6D7D575672F9D98100941E9BDEFFECA1313B75782D18D6FBE7ACA82B20ABC0052A205B645ACD435D07270BC2E5765D23D3759408E342C2CDE02D76716EC75FFAB512328702080410011A800245D60157117BA5F0B8BC30560B407F327BF92C6940C06E320598623E28A04504479B67CBD89B128F5C8D1AA0B22A4210A8BEBD8D197BF9F02AFE69F95BCC1B86D82228FF7B37467A02B521EA8F5CA100F653B03E788B599810CFCCB3F8118DDFF386583002D5B6F48EC7EE1B49DBCB59710BD3D91481D6FB8D2CD1D356259742045677055AA50638D74DBD30366E6DC8DA0BDABE2F47B31E3EA5FC64574A026A6712732891E2D9BAA3D293CECADC8CAC69124E4EB21EB6EE41F0F6C80A1D998B7A4197BF724047AD99D8C93FFCB4799DEF424083A8DDB26C2456293907B833D3EBA38DBD2B6F1B6F3EAB2009B876AB5D22FF44A9F9BDA3E0B6FF76749A944930

 

18:42 – 1488307345708 (tree size: 90,167): 080012220A20AC3B9AED7FA9674757159E6D7D575672F9D98100941E9BDEFFECA1313B75782D18ACB29FB0A82B20B7C0052A20B0C9EAB18C4B63229D17B40DAA995682720993A581B2EAA5106A3468D1C78518328702080410011A80024881C1C4D26EB6F1F23CE8629D11F97B0AEAB7A2CC273F9564E1A1D00693156A20B329B0EB16572C6F4D6D517FADDAC43943C06DCA34B0F1C5E0708B5D7B6D341244521B49CF026E575E240C76D739FF312EBB3D4524217899FDF3594521EA7C381322781053A8585ABED4A02FFF173171C9D7B1D562C71FAC2C273090D4EF21E8F87EC13D96AFECDFF3C5C2082261582FE360CD73B6F4C042CDCD17CEEEFD4A83FA5855BC945935A47FE0E23028037F43BF06987E6EC194F2129772A218E5BCDF8E94C9635817D76839D83CADDCF82DF4EFB8AD5C37F3AC9FE85CF8D36C4A4084A12C5B6085B7F25AD8D7C264390D7A94CBE46612AFDDC3AE888723065B8B25

 

22:32 – 1488321064440 (tree size: 90,167):

080012220A20AC3B9AED7FA9674757159E6D7D575672F9D98100941E9BDEFFECA1313B75782D18F8DBE4B6A82B20B7C0052A20B0C9EAB18C4B63229D17B40DAA995682720993A581B2EAA5106A3468D1C78518328702080410011A800285C9C3E20D31CC44CEBB16DFEB73831884DA33FCB0964DE476303B7326B851F2E6DE7B949A8F00360414121FC7B704106CA9D1140B6FA5A89EF92AE3186C00DB74C57090430D524F144A8B19DA62D4A7124A1525F5973C05728D4A853F1FABC8E1ED76870C1A57FAD2BB7852618AD2B872F574C2AE4B58164CC6F593EB1AE8C60EF5C87819C869AADB5342F9836DB8AEC74BBC35F0A51A291E91E29E89AC7395BE45E03E790DFC586047933DF77BFF983C940677F2F711A0CDDDD6B45481183476390065BF62AEEB53A703F9D1C7F662F538AF7BD7F1D9DFAD9A127041678EF38BFD4ECC035E29D4F5B7207E75BED5A57852709C70D68D06DD3D15D0FFA5338A

 

And here is the consistency proof (sizes 90155 and 90167):

 

{ "consistency": [ "tEgqDD0zLL4KViqAMmnzvh\/7TjlIYJk\/fhSe6cS8Jo4=", "p4eTTY4JYQxhXzcG3iFH1imhqglNbbpCzZmp6Ozw9wM=", "fb63f4aoOOFIArLA0JfdJwfB78tecSJwnVYLb5RhbN4=", "Lz2zMdy+Yti8LjZ9O0m7cKZTwP4bbNZ7glCRF7UGBHE=", "7LXxCtSf42LbuFjKdYX1vdOnNgS3y\/5c+09mDQkC6DU=", "jl3sEafdkLxR0QYG5\/+34P7+QFSoIehfuYZujm+E6GA=", "VH8nj3DKzwv9v3bcQeDKjw4a0H+GbIm1SIxU0z3J\/1E=", "2eQpspAlOaHniWW6exNICgPfge2u0BHXAU0bJoCxy+c=", "jsEyi+238H+\/SNiDNmoNeGTPeMkBEeM55WNKuuPrfeY=", "rcxt0aKSfv5MGCDJSwt6WDZffq9Nmp4SpQm5Xr376Lo=" ] }

 

 Also, attached are the certificate entries missing from the bad STH:

 

Certificate Entry 1:

AAAAAAFahec6SQABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA9IwggPOoAMCAQICEFHdYzrohqzQXz92eOeK4MMwDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzE1MTRaFw0yMDAyMjkxNzE1MTRaMCYxCzAJBgNVBAYTAkVTMRcwFQYDVQQDDA53d3cuYXhhc3ViLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALq5Ldz8hYgxxy\/QobBuB0tjXXjEFEI4Bq\/bB2jqx+0ut2F\/natsF3U7WSxFkHQxyhQ3AwOxsEGb5K0f3BlSbBpY0XFDZIKrUT+dsPRifiSBjNL1qXo690r2+NMs6WoqmpE0EB3UMfB1WXSgJAx+qeNiCCpgB5B8LZbYWUDf912S\/iLZQz9poGBHwQMO9CvVn5eYrmtSZMhFBLGVbfKTbD2jqXbjp5ioSMMLjvYgEFfAIka+RKw\/bOEl5jbi2WPvg+Mq+Qe8UuM8pmADXL3+\/qFDe0bn6u5aLkg3ge42M\/3\/HxYjuf\/tiLlKo0lUQdNt5UdldT6jgo11seCbepPWPHcCAwEAAaOCAbwwggG4MA4GA1UdDwEB\/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwCQYDVR0TBAIwADAdBgNVHQ4EFgQUzvFSUz7nUX4EzdlBvx6W5CDp2sMwHwYDVR0jBBgwFoAU15FOAcSwv\/jIZ5NEnOcz+q2TDK8wbwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5jb20wOQYIKwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5zZXJ2ZXIxLmNydDA4BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2Etc2VydmVyMS5jcmwwGQYDVR0RBBIwEIIOd3d3LmF4YXN1Yi5jb20wIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMFEGA1UdIARKMEgwCAYGZ4EMAQIBMDwGCysGAQQBgbU3AQIFMC0wKwYIKwYBBQUHAgEWH2h0dHBzOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xpY3kAAA==

 

Certificate Entry 2:

AAAAAAFahgdibgABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA\/0wggP5oAMCAQICEEhDIMaMJhjh2m1\/AlJEbKgwDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzUxMjlaFw0yMDAyMjkxNzUxMjlaMCUxCzAJBgNVBAYTAlNFMRYwFAYDVQQDDA1iYW5hbmZpc2suY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzCn01bi0msFPsdsDMJIP38ulDqUEJyP\/f8ibST2qSPKRG+WRTsCzREFH+QkLK+OgYF3lq7lSKLetOvk+KwP\/4\/aq56x+dAZ4shFnrTQEBvji\/+fUa7kyY4z6XoRnSmAv3feAOxHVDf9EcnW\/5uiyBWQofw7QkVT4Vb0TXkNMTIZAumXrAptjtd61meZAQraCO7AQqkrf38qRJzflmLDXdcI3yW\/2iXKnqzUmL9VVF0yC73rSAEOz3oi8gcARmDFqOfuXn7+37raLGSgdllrphnXJ\/ReV0+I2of8wgzIZ\/XLduqrantNlMGyq98tJGC\/V22YE1Ag3dO4IRxv96perVwIDAQABo4IB6DCCAeQwDgYDVR0PAQH\/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAJBgNVHRMEAjAAMB0GA1UdDgQWBBRzVmmCuPdBBj7Xy3Eu8JSuCEMygjAfBgNVHSMEGDAWgBTXkU4BxLC\/+Mhnk0Sc5zP6rZMMrzBvBggrBgEFBQcBAQRjMGEwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbTA5BggrBgEFBQcwAoYtaHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc2NhLnNlcnZlcjEuY3J0MDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3NjYS1zZXJ2ZXIxLmNybDBFBgNVHREEPjA8gg1iYW5hbmZpc2suY29tghF3d3cuYmFuYW5maXNrLmNvbYIYYXV0b21hdGlvbi5iYW5hbmZpc2suY29tMCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzBRBgNVHSAESjBIMAgGBmeBDAECATA8BgsrBgEEAYG1NwECBTAtMCsGCCsGAQUFBwIBFh9odHRwczovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5AAA=

 

Certificate Entry 3:

AAAAAAFahdOFfwABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMABPEwggTtoAMCAQICED7CHba2CHrr94tmyrCLJUQwDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzAyMzRaFw0yMDAyMjkxNzAyMzRaMCsxCzAJBgNVBAYTAlJVMRwwGgYDVQQDDBN3d3cudGVsdGFtYi5wZXJtLnJ1MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuuEYyZuCPM5oQcf\/m6GKGuufjJ33doHaX8EPU0G\/6Fuzu8TvfjXjK3s5pOfDNuCq0C9dq2v2XssJQa1uo937orJ6BScd+oNJ9TPY8Ps0QcAf99Z3bvZscOUauRCL4TV9hF53kDr\/8VLHdCMU55HlCbkW0rGmvypT0MhTwhkJ61xt6GNsCvxm4amL5faeHWc+HtWAUWmAxVWZakyQQbCrBBNot4XEx6Wyl2ethKjBDMVm3sqMczC7IrEiG7SQuB1uC+tQtNpgVUlTx8xMMgRUpb6uWMnmWguomnTL5Syr8QsMBnIQnIvehrOBlxDU8pV95wEvAvjbP3Dq7FNOeGK0qjh4TAAENbbZMOhEcKIGZKl7ESyLP+qMfSG2aC8Iik7y6yevEy8O713nnT7ynm8dWHDdgjQY20I+e\/Bg0ooC32jnfpse\/wPEvkeiQqn0YM6YkjoztOcjJdCpXBBFpkEta0HaEf0QzQm+ixYfXINnPFTTUb0EJE8sosLg1B4CspUvXlC+b0P0aT+V\/i9bnZQOmGY7JCVMFmBDcYA6qA9bKeqcAqoH417+y176uxPLOcLIDZ8Ffo51Tsp3KE7X+qZOgGbENyOjszR5wRJzHCYzKwMVMyiUn2d6EMQ419vB9\/gbzIXs\/Tg4DmLhUo25jEL1DORmmkIkAaegq\/1Aaf93g0cCAwEAAaOCAdYwggHSMA4GA1UdDwEB\/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwCQYDVR0TBAIwADAdBgNVHQ4EFgQUy7gyH9tHJJrlknwJI3baC7wid00wHwYDVR0jBBgwFoAU15FOAcSwv\/jIZ5NEnOcz+q2TDK8wbwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5jb20wOQYIKwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5zZXJ2ZXIxLmNydDA4BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2Etc2VydmVyMS5jcmwwMwYDVR0RBCwwKoITd3d3LnRlbHRhbWIucGVybS5ydYITdnBuLnRlbHRhbWIucGVybS5ydTAjBgNVHRIEHDAahhhodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS8wUQYDVR0gBEowSDAIBgZngQwBAgEwPAYLKwYBBAGBtTcBAgUwLTArBggrBgEFBQcCARYfaHR0cHM6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeQAA

 

Certificate Entry 4:

AAAAAAFahcwfewABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA+4wggPqoAMCAQICEEBTOqyyEFzaBu6HHaQDVfswDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNjU1MDRaFw0yMDAyMjkxNjU1MDRaMDQxCzAJBgNVBAYTAlVTMSUwIwYDVQQDDBxnaXQtaW50cm8uZGVycmVraGFycmlzb24uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAomL05zT2sGFpwQ7AdIpDEFDwXnKgwFYteGqFFNpGmLQ30U3NWXG2Wp4Ima7QvqL7aLhHvsZVu7DCcN\/VaxfjWBREDoxEiEcAcrDYS+YFNlFsHxZ+2mmNhRJ\/mBSOCN2Kd7HgdE97Bd1L4h6iR1j5Nxjw8eitzPD5Oh6D5Xnj3t8wLQqTBkcGqa6cIf+L2uFm5DJeRafd4pMbEpzLsiH+hzwIzdkG82HS+qApYsF5+BWCO1KHyiXmSeOR0nO8kLd\/Gk\/7bNVhApWzR5mI99dEcFn0TWylkW\/RISPVp9drfJRg6gHT5k2RLuO32XP58r6IopLOSJdjKJyx12Vc+E1xtQIDAQABo4IByjCCAcYwDgYDVR0PAQH\/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAJBgNVHRMEAjAAMB0GA1UdDgQWBBQw0c1jfd25r879khVSJV60GDoT7zAfBgNVHSMEGDAWgBTXkU4BxLC\/+Mhnk0Sc5zP6rZMMrzBvBggrBgEFBQcBAQRjMGEwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbTA5BggrBgEFBQcwAoYtaHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc2NhLnNlcnZlcjEuY3J0MDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3NjYS1zZXJ2ZXIxLmNybDAnBgNVHREEIDAeghxnaXQtaW50cm8uZGVycmVraGFycmlzb24uY29tMCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzBRBgNVHSAESjBIMAgGBmeBDAECATA8BgsrBgEEAYG1NwECBTAtMCsGCCsGAQUFBwIBFh9odHRwczovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5AAA=

 

Certificate Entry 5:

AAAAAAFahfmaewABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA+QwggPgoAMCAQICEBvpdto94iTZYqsJ6soyKMswDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxODAzNDdaFw0yMDAyMjkxODAzNDdaMCQxCzAJBgNVBAYTAkNaMRUwEwYDVQQDDAxtYWlsLm11em8uY3owggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCmmkov+HgPvDRylo70YtEEQbTXOZ\/SAJwZQTVxdv3LijpXHJNPJx9a8mInL8bXlY48mRf1NWKx869tK70WGG5tTDEAaBOVlw57ewAkJ33Byx4lfRMDKsj+gkgdn+bvVyv9Sgv6pAz2q1bOiAqHXE1C++NGNgNeuCZ6T013nvRoTtTE+NnFAB2Pwd3dO7wdwU+R3EbjUAW9ePdbKD3ucsnG33rYm33HByOM80\/6THmC5a\/+UFNqAfTIto0VZo\/zTM02ldj\/Cab8ocORRixBatmiOVQgcrNV3DMiKiPT9xTkEzp3Ld0iR06zYBj4oYP4VG+8DqfFZ0WJygGagqh867OFAgMBAAGjggHQMIIBzDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMAkGA1UdEwQCMAAwHQYDVR0OBBYEFD8hNDiwPuVDXKOIuLUElebEnCC9MB8GA1UdIwQYMBaAFNeRTgHEsL\/4yGeTRJznM\/qtkwyvMG8GCCsGAQUFBwEBBGMwYTAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Auc3RhcnRzc2wuY29tMDkGCCsGAQUFBzAChi1odHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zY2Euc2VydmVyMS5jcnQwOAYDVR0fBDEwLzAtoCugKYYnaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2NhLXNlcnZlcjEuY3JsMC0GA1UdEQQmMCSCDG1haWwubXV6by5jeoIUYXV0b2Rpc2NvdmVyLm11em8uY3owIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMFEGA1UdIARKMEgwCAYGZ4EMAQIBMDwGCysGAQQBgbU3AQIFMC0wKwYIKwYBBQUHAgEWH2h0dHBzOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xpY3kAAA==

 

Certificate Entry 6:

AAAAAAFahemOVAABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA9YwggPSoAMCAQICEDqrJVoAnRG6cTJqzmH32lswDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzEzMzBaFw0yMDAyMjkxNzEzMzBaMCgxCzAJBgNVBAYTAkZJMRkwFwYDVQQDDBBmdy52aXJyYW50YWxvLmZpMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsVZIM6CJ07hAQQxIkgHdloTVX4jLblhP2rCruV9Q5aBbrz7HlBQscPpO\/Btu2NY5qVexeMoR9KFGbEystSQ2GBNhy4+1Cvbqpgh8Dco1gdWMgpIjLLyDnvU3s6T4tr5\/A3ApiJg2cqRqDf9jDM47wBVig\/BGAn83Wd4\/VyuMoajEzoz\/mK5+KdHL0AFZdngUrQ8ZT2zrMBNjpCnnktgPP8\/PV4zelWy+F4\/dfXP1nZii0AR0mVu1dW8c0weCA9yGsoA0LzoGq2x3ZiSF8jsucaezlT6iAMOPOwDCO6IH+59\/OiVSh9xidGz+VdDkcjt5TnBrfIsbXYas80poT8PyZwIDAQABo4IBvjCCAbowDgYDVR0PAQH\/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAJBgNVHRMEAjAAMB0GA1UdDgQWBBSVOeKX8yQHgvtctjcY16lHrJ1nbTAfBgNVHSMEGDAWgBTXkU4BxLC\/+Mhnk0Sc5zP6rZMMrzBvBggrBgEFBQcBAQRjMGEwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbTA5BggrBgEFBQcwAoYtaHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc2NhLnNlcnZlcjEuY3J0MDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3NjYS1zZXJ2ZXIxLmNybDAbBgNVHREEFDASghBmdy52aXJyYW50YWxvLmZpMCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzBRBgNVHSAESjBIMAgGBmeBDAECATA8BgsrBgEEAYG1NwECBTAtMCsGCCsGAQUFBwIBFh9odHRwczovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5AAA=

 

Summary:

 

In summary, please note that:

1         This issue was triggered by virtue of the AWS S3 outage, not an operator error.

2         No certificate data was lost – ALL of the certificate data is in the good/latest STH(s).

3         We have issued no SCTs during the outage or during the time that the instance with the “bad” head was active. We are happy to share any data we have to support this claim.

4         In the event of a similar issue in the future, we have identified the steps that we will take to prevent inconsistent STHs from ever being published.

[Tom Ritter]

On 6 March 2017 at 13:30, Andrew Ayer <ag...@andrewayer.name> wrote:
> On Mon, 6 Mar 2017 14:01:55 -0500
> Ryan Sleevi <rsl...@chromium.org> wrote:
>> It would seem that, on the basis of the evidence presented, it's
>> appropriate to disqualify Venafi. What's unclear, and where part of
>> the question came from, is whether consistency to the STH produced
>> at 1488307345708 can be produced, or whether it's necessary to
>> disqualify at an earlier STH.
>>
>> I'm curious how others feel about this proposed course of action.
>
> Hi Ryan,
>
> I agree that disqualification is appropriate.

Me too.

I'm excited to see Gen2 get qualified though, and I hope Venafi
considers re-upping with a second replacement log.

-tom

[Andrew Ayer]

Based on the latest information provided by Venafi, the tree at
1488307345708 is identical to the tree at 1488321064440, which is one
of the inconsistent STHs I produced in my original report. Therefore,
the last good STH is actually the one at 1488300146134 (tree size
90155) - this is last point at which all monitors had the same view of
the log.

However, it's unclear why the last good STH matters for a disqualified
log. Will Chrome downright ignore any SCT with a later timestamp, even
for determining the diversity requirement for embedded SCTs?

Regards,
Andrew

[Ryan Sleevi]

On Tue, Mar 7, 2017 at 12:26 AM, Andrew Ayer <ag...@andrewayer.name> wrote:

Based on the latest information provided by Venafi, the tree at
1488307345708 is identical to the tree at 1488321064440, which is one
of the inconsistent STHs I produced in my original report.  Therefore,
the last good STH is actually the one at 1488300146134 (tree size
90155) - this is last point at which all monitors had the same view of
the log.

However, it's unclear why the last good STH matters for a disqualified
log.  Will Chrome downright ignore any SCT with a later timestamp, even
for determining the diversity requirement for embedded SCTs?

Regards,
Andrew

Andrew,

Does the "Certificate Transparency in Chrome" policy, linked to from https://www.chromium.org/Home/chromium-security/certificate-transparency , not sufficiently address this?

"“Once or currently qualified” means that the log was qualified or pending qualification at the time of

certificate issuance, that the log was accepted prior to the time of check, but that the log may have

been disqualified following acceptance, prior to the time of check."

The 'last good' STH is used (in conjunction with other, still qualified, SCTs) to determine whether or not the log was qualified at time of certificate issuance. Disqualification can be retroactive to the point of the last consistent view - meaning the potential for some subset of certificates that may need to be reissued - or it can be forward looking - meaning CAs need to migrate to new logs to satisfy the SCT requirements.

[Ben Laurie]

On 6 March 2017 at 21:42, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:36 PM, 'Ben Laurie' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

On 6 March 2017 at 21:30, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

I'm asking what a bad SCT is.

An SCT is a promise to include into a log. It is not related to any particular STH. Hence my confusion...

A 'bad' SCT in this case would be a promise issued after the production of the 'bad' STH. We can know - from the existence of the STH - the number of SCTs produced up to that point. However, any SCTs issued between the production of two STHs cannot be publicly known - ergo, they could be omitted from the log.

But that's always true - i.e. a log can always issue an SCT and not log the corresponding cert. The only way you ever find out is to see the SCT, which is also the case here, isn't it?

 

Imagine an 'evil' log (and I'm not accusing Venafi of this, I'm highlighting the challenge)

SCT A

SCT B

SCT C

STH (covering A-B-C)

SCT D

SCT E

SCT F

"Bad" STH (covering D-E-F)

SCT Z

SCT E

SCT D

SCT F

"Good" STH ( covering E-D-F)

We can show that the 'bad' and 'good' STHs fully incorporate the SCTs issued for certificates D/E/F, that's true - but the community cannot know about SCT Z unless/until it's produced, and no consistency proof can be provided for it.

I can understand and appreciate that you may argue this is true even under 'good' conditions,

I do.

 

but I think the timeline of events provided makes it possible for this in a way that cannot be detected by comparing 'bad' STH to 'good' STH, and may represent additional operational failure. 

In terms of knowing about Z, I don't see how this error changes anyone's position at all.

 

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACvaWvaxyxidTDCoQ5XLmWHimqhif9%3DcuA9kKyQ3TTj84Y2z%2Bg%40mail.gmail.com.

[Ryan Sleevi]

On Tue, Mar 7, 2017 at 8:23 AM, Ben Laurie <be...@google.com> wrote:

I can understand and appreciate that you may argue this is true even under 'good' conditions,

I do.

 

but I think the timeline of events provided makes it possible for this in a way that cannot be detected by comparing 'bad' STH to 'good' STH, and may represent additional operational failure. 

In terms of knowing about Z, I don't see how this error changes anyone's position at all.

Before we continue this discussion much further, it may be helpful to take a step back and re-evaluate:

Can you concretely argue/advocate why you believe disqualification for violating the consistency of the Merkle Tree is inappropriate? Can you indicate what costs you believe will be born by the various stakeholders - monitor/auditors, clients, CAs, the ecosystem - both with and without your proposed course of action?

I'm extremely uncomfortable with the notion of allowing "known bad" SCTs into the ecosystem, and the trust that undermines. Further, I'm extremely opposed to introducing the idea of "SCT revocation", a cost that is unreasonable to request of Chrome, let alone for Chrome to request of the broader CT ecosystem.

[Andrew Ayer]

On Tue, 7 Mar 2017 01:52:58 -0500
Ryan Sleevi <rsl...@chromium.org> wrote:

> On Tue, Mar 7, 2017 at 12:26 AM, Andrew Ayer <ag...@andrewayer.name>
> wrote:
> >
> > Based on the latest information provided by Venafi, the tree at
> > 1488307345708 is identical to the tree at 1488321064440, which is
> > one of the inconsistent STHs I produced in my original report.
> > Therefore, the last good STH is actually the one at 1488300146134
> > (tree size
> > 90155) - this is last point at which all monitors had the same view
> > of the log.
> >
> > However, it's unclear why the last good STH matters for a
> > disqualified log. Will Chrome downright ignore any SCT with a
> > later timestamp, even for determining the diversity requirement for
> > embedded SCTs?
> >
> > Regards,
> > Andrew
> >
>
> Andrew,
>
> Does the "Certificate Transparency in Chrome" policy, linked to from
> https://www.chromium.org/Home/chromium-security/certificate-transparency ,
> not sufficiently address this?

It wasn't clear how the STH played a part, hence my question. The bit
I was missing was that disqualification could be retroactive. Thanks
for clearing that up with your explanation.

Regards,
Andrew

[Ryan Sleevi]

On Tue, Mar 7, 2017 at 12:00 PM, Andrew Ayer <ag...@andrewayer.name> wrote:

It wasn't clear how the STH played a part, hence my question.  The bit
I was missing was that disqualification could be retroactive. Thanks
for clearing that up with your explanation.

Regards,
Andrew

Right, I totally see where that came from, and I think it'll be necessary to take a stab to fix (feel free to propose yourself, if it'd help)

The overall goal is this - for any log recognized as 'qualified' or 'disqualified' in Chrome, our goal is that you should be able to assure the inclusion of all SCTs by obtaining inclusion proofs to the provided STH (delivered by Google). Any failure to provide for this is reasonable grounds for disqualification, modulo the discussions of the timeliness of this proof. That is, the MMD is effectively the window for how long to wait before you can obtain this proof.

The Venafi case is interesting, and I'm hoping someone can formulate the right questions (and I believe some of the folks at Google who work on CT are drafting them), since, as Ben Laurie is highlighting, it's arguably possible to show this goal is met; that is, every SCT Venafi issued during the 'bad' STH is possible to obtain an inclusion proof to the 'good' STH (they just have differing entry IDs from the bad and good trees)

This is a credit to Venafi for maintaining sufficient logging to be able to demonstrate this, modulo the 'unknown' element Ben and I were discussing, because it potentially creates the opportunity for trust, but that opportunity must be weighed against both the immediate and ecosystem risk - hence, why I posed the question to this list as to whether disqualification is appropriate in this specific case.

If you can imagine another scenario, where a log is unable to provide inclusion proofs for all issued SCTs (whether they realize it or not), then I think it'd be necessary and advisable to 'retroactively' disqualify, so that we maintain the property of all SCTs (up to time X) have inclusion proofs.

I'm open to suggestions as to how to clarify that with respect to 'qualified at time of issuance' being a state that can retroactively change in the event of violations of the policy, as appropriate to achieve the end state of always-verifiable consistency.

[Paul Hadfield]

Hi Andrew,

have you/CertSpotter retained a copy of the Venafi get-entries response for the range of entries from 90155 to 90161, captured at the time you spotted the unverifiable STH@90161 ? Or records from which that response could be reconstructed?  If so I'd like to see them.

I'm interested to see if the entries returned match those Google's monitor retrieved.

The two monitors that I help operate at Google didn't retrieve Venafi's unverifiable 90161 STH, since we were unlucky enough for both systems to sample on either side of the 2 minute period when it was live.

thanks,

Paul

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20170307090021.2f7ba950457c201e2436ef77%40andrewayer.name.

[Kurt Roeckx]

On Tue, Mar 07, 2017 at 05:19:36PM +0000, 'Paul Hadfield' via Certificate Transparency Policy wrote:
> Hi Andrew,
>
> have you/CertSpotter retained a copy of the Venafi get-entries response for
> the range of entries from 90155 to 90161, captured at the time you spotted
> the unverifiable STH@90161 ? Or records from which that response could be
> reconstructed? If so I'd like to see them.
>
> I'm interested to see if the entries returned match those Google's monitor
> retrieved.
>
> The two monitors that I help operate at Google didn't retrieve Venafi's
> unverifiable 90161 STH, since we were unlucky enough for both systems to
> sample on either side of the 2 minute period when it was live.

I've attached what I've downloaded, which I think are the bad
ones:
-rw-r--r-- 1 kurt kurt 43606 2017-02-28 22:23:56 +0000 entries_90155_90160


Kurt