Venafi has produced inconsistent STHs

[Andrew Ayer]

Venafi (log ID rDua7X+pZ0dXFZ5tfVdWcvnZgQCUHpve/+yhMTt1eC0=) has signed
the following STHs (also attached):

{
"tree_size": 90161,
"timestamp": 1488320541206,
"sha256_root_hash": "I7PzqT8cEPjlfgOx3/2VxnuO5BTIJ6Zf8OM8DAwGhuM=",
"tree_head_signature": "BAEBAAgJjfMWJ0agR0VCpB1v8UJs3X8H3dPU8C0NmUMtMAdAm27tgiDcpDECHCMrJbIIy7v+y1p4zVGC6eWf1Tew+W+O32WinshHb9th7lLNlQ5yJUKW5UtMTJxW/BiFpHO75+WXsajb4EyTPkoJ1M2qxDX/cK/hSb2ar0W7G9weRaw7WetwEAG7pv2j/tnUUGXHWfnNkg6f40GkwSaGfY9Xw9gaZDeUawvS8T61qODsaZprWDsBAWaClaaelrmlx0kkZBnMP/LKKSlywrTNEt3Ow1oKBqT3nL5A2fTPoRfsF+1OGyy2iokX2pI1ZLBRS4v4rqMce/dgzXxrS6OUUINHfLA="
}

{
"tree_size": 90167,
"timestamp": 1488321064440,
"sha256_root_hash": "sMnqsYxLYyKdF7QNqplWgnIJk6WBsuqlEGo0aNHHhRg=",
"tree_head_signature": "BAEBAIXJw+INMcxEzrsW3+tzgxiE2jP8sJZN5HYwO3MmuFHy5t57lJqPADYEFBIfx7cEEGyp0RQLb6Wonvkq4xhsANt0xXCQQw1STxRKixnaYtSnEkoVJfWXPAVyjUqFPx+ryOHtdocMGlf60rt4UmGK0rhy9XTCrktYFkzG9ZPrGujGDvXIeBnIaarbU0L5g224rsdLvDXwpRopHpHinomsc5W+ReA+eQ38WGBHkz33e/+YPJQGd/L3EaDN3da0VIEYNHY5AGW/Yq7rU6cD+dHH9mL1OK971/HZ362aEnBBZ47zi/1OzANeKdT1tyB+db7VpXhScJxw1o0G3T0V0P+lM4o="
}

However, it is unable to provide a valid consistency proof between
these STHs. For reference, here's the consistency proof provided by
https://ctlog.api.venafi.com/ct/v1/get-sth-consistency?first=90161&second=90167:

{
"consistency" : [
"r+r7qrYh4i9lWa1nzVO86nj+9c6BGZq5i36IRzgPGl4=",
"NPLfppxk5a1xQZb/tmCK3oJ/WFflOaxx4pNE49Tf+8Y=",
"U0IGCc5N616P/zKPQVkFZkYdRoSBmLipCgspE0vNLeE=",
"kZiqCHdVXWXLAmFtPPm6tNUBm3ciQY7KTkBIPdPyP50=",
"im9E++S7hURbeazY1bHGqxx6/5zYmDmpLhCU3lAvGOM=",
"VH8nj3DKzwv9v3bcQeDKjw4a0H+GbIm1SIxU0z3J/1E=",
"2eQpspAlOaHniWW6exNICgPfge2u0BHXAU0bJoCxy+c=",
"jsEyi+238H+/SNiDNmoNeGTPeMkBEeM55WNKuuPrfeY=",
"rcxt0aKSfv5MGCDJSwt6WDZffq9Nmp4SpQm5Xr376Lo="
]
}

It may also be of interest that the certificate Venafi is currently
serving to me at log index 90160 is not the same certificate that I
previously recorded Venafi as having at this index.

Regards,
Andrew

[Ryan Sleevi]

Thanks for this heads up, Andrew.

I'm hoping the Venafi team can provide feedback, while as a community, we determine appropriate next steps.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20170228160048.16407cf750429c906b86d319%40andrewayer.name.

[Steve Topletz]

Hello,

Yes, Venafi is aware of this issue and has been tracking it since the AWS S3 outage earlier today. While the log server is functioning as expected now , we ran into a database inconsistency issue as part of the effort to get back online as quickly as possible. To be clear, we have confirmed that NO certificate data has been lost. We are in the process of identifying/confirming root cause, and will post a full incident response, as well as what we are doing to negate the possibility of this impacting us again.

Regards,

Steve Topletz

[Steve Topletz]

SUMMARY:

On February 28th of 2017 at 22:23:32 GMT the AWS S3 outage caused Venafi ‘ctlog.api.venafi.com’ log to generate and publish inconsistent STHs. This represented a violation of section 3 of RFC 6962 which states:

A log is a single, ever-growing, append-only Merkle Tree of such certificates.

This was triggered by automatic data restoration from S3 due to the recovery process of a terminated instance on an Amazon EC2 auto-scaling group. While this did not result in the loss of any logged certificates, as confirmed by load balancer logs, it did result in an inconsistent tree head being served from the log for approximately two minutes.

IMPACT:

The inconsistent tree head was served to nine monitors.

ROOT CAUSE:

The following is a chronological account (times in GMT) of the cause for the inconsistent STH generation.

17:36 The last successful backup of the CT logs were picked up and published. 

17:38 The Amazon S3 API failed when performing backups of the log, and continued to fail with the database continuing to update.

18:42 The correct signed tree head was made available. (timestamp 1488307345708)

18:55 The instance running CT log was automatically terminated by the AWS auto-scaling group.

22:22 Venafi CT log went out of synchronization and signed the incorrect tree head because it pulled the last successful backup which did not reflect the current state of the database.

22:23 Venafi CT log server made the incorrect STH available. (timestamp 1488320541206)

22:25 Venafi became aware of the incorrect STH and stopped the instance.

22:31 Venafi restored the correct backup and the server generated a new correct STH.

22:32 Venafi CT log server made the correct STH available. (timestamp 1488321064440)

REMEDIATION AND PREVENTION:

The current architecture of the Venafi CT server assumes periodic back-ups of the certificate log data in AWS S3, which leaves us susceptible to the possibility of incomplete archives in the event of outages like what we saw with Amazon this week. While no certificate data was lost, in our urgency to restore services to a functional state, we missed checking on the consistency of the backed-up data prior to restore operations. In retrospect, we should have performed a manual merge from our last back-up to ensure that we didn’t miss any data, which, while potentially slower, would negate the possibility of inconsistent tree heads.

 

In addition to implementing this procedure in the unlikely event of another S3 outage, we are evaluating architectural changes, and mitigating the dependency on a single source of failure like S3. Our new log server, submitted to Google earlier this year, is built on a distributed CT log infrastructure that is not predisposed to these types of outages and their resulting inconsistencies. We will also be reaching out to Root CAs impacted by this incident, including the remediation steps we have taken, once we have determined a course of action.

[Ryan Sleevi]

On Wed, Mar 1, 2017 at 10:13 PM, Steve Topletz <steve....@venafi.com> wrote:

SUMMARY:

On February 28th of 2017 at 22:23:32 GMT the AWS S3 outage caused Venafi ‘ctlog.api.venafi.com’ log to generate and publish inconsistent STHs. This represented a violation of section 3 of RFC 6962 which states:

A log is a single, ever-growing, append-only Merkle Tree of such certificates.

This was triggered by automatic data restoration from S3 due to the recovery process of a terminated instance on an Amazon EC2 auto-scaling group. While this did not result in the loss of any logged certificates, as confirmed by load balancer logs, it did result in an inconsistent tree head being served from the log for approximately two minutes.

IMPACT:

The inconsistent tree head was served to nine monitors.

ROOT CAUSE:

The following is a chronological account (times in GMT) of the cause for the inconsistent STH generation.

17:36 The last successful backup of the CT logs were picked up and published. 

17:38 The Amazon S3 API failed when performing backups of the log, and continued to fail with the database continuing to update.

18:42 The correct signed tree head was made available. (timestamp 1488307345708)

18:55 The instance running CT log was automatically terminated by the AWS auto-scaling group.

22:22 Venafi CT log went out of synchronization and signed the incorrect tree head because it pulled the last successful backup which did not reflect the current state of the database.

Hi Steve,

Thanks for sharing this level of detail, as it's useful for both log operators and the ecosystem to understand the risks and challenges.

I'm having a little trouble understanding the timeline here with your remark that this did not result in a loss of any logged certificates, so I'm just trying to work to understand.

Are you saying you have all certificates (and corresponding SCTs) issued from the time period 17:38 to 22:25 still available? That is, that you can demonstrate an inclusion proof (to the incorrect STH) for all of those certificates (with the 'bad' SCTs), with the fullness of the 'bad' tree, and similarly, demonstrate a subsequent inclusion proof that all of those certificates are incorporated into the STH generated 22:32 (or later), with the fullness of the 'good' tree?

Can you state how many certificates this affected?

[Ryan Sleevi]

On Thu, Mar 2, 2017 at 1:33 AM, Ryan Sleevi <rsl...@chromium.org> wrote:

Hi Steve,

Thanks for sharing this level of detail, as it's useful for both log operators and the ecosystem to understand the risks and challenges.

I'm having a little trouble understanding the timeline here with your remark that this did not result in a loss of any logged certificates, so I'm just trying to work to understand.

Are you saying you have all certificates (and corresponding SCTs) issued from the time period 17:38 to 22:25 still available? That is, that you can demonstrate an inclusion proof (to the incorrect STH) for all of those certificates (with the 'bad' SCTs), with the fullness of the 'bad' tree, and similarly, demonstrate a subsequent inclusion proof that all of those certificates are incorporated into the STH generated 22:32 (or later), with the fullness of the 'good' tree?

Can you state how many certificates this affected?

Just wanting to touch base on this point, as we determine the next step.

It would seem that, on the basis of the evidence presented, it's appropriate to disqualify Venafi. What's unclear, and where part of the question came from, is whether consistency to the STH produced at 1488307345708 can be produced, or whether it's necessary to disqualify at an earlier STH.

I'm curious how others feel about this proposed course of action.

[Andrew Ayer]

Hi Ryan,

I agree that disqualification is appropriate. I would ask that Venafi
provide the STH at timestamp 1488307345708 so that I and other monitors
can verify its consistency. Assuming no inconsistency can be proven, I
think it's appropriate to disqualify at that STH.

Regards,
Andrew

[Kurt Roeckx]

The STH before that would be at 1488300146134 (2017-02-28
16:42:26.134+00), with a tree size of 90155.


Kurt

[Ben Laurie]

I think this is a super interesting case.

If I am understanding correctly, Venafi's log now contains all certificates they issued an SCT for.

The incorrect STH was issued for a log that consisted of a subset of certificates accepted.

Later STHs include all those certificates and more, but in a different order, and hence have an STH that cannot be linked to the incorrect STH. (Note: this is deduced from the nature of the failure)

However, if it can be shown that the incorrect STH corresponds to a subset of certs now logged, then it seems to me that this entirely meets the intent of the system, if not the letter.

And hence I would argue against disqualification, pending the above demonstration.

 

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACvaWvYgVWPuRvSdMtXh2127HQ8frOkcLOjBgGmc-5Tn6u9J1A%40mail.gmail.com.

[Ben Laurie]

On 6 March 2017 at 21:06, Ben Laurie <be...@google.com> wrote:

On 6 March 2017 at 19:01, Ryan Sleevi <rsl...@chromium.org> wrote:

On Thu, Mar 2, 2017 at 1:33 AM, Ryan Sleevi <rsl...@chromium.org> wrote:

Hi Steve,

Thanks for sharing this level of detail, as it's useful for both log operators and the ecosystem to understand the risks and challenges.

I'm having a little trouble understanding the timeline here with your remark that this did not result in a loss of any logged certificates, so I'm just trying to work to understand.

Are you saying you have all certificates (and corresponding SCTs) issued from the time period 17:38 to 22:25 still available? That is, that you can demonstrate an inclusion proof (to the incorrect STH) for all of those certificates (with the 'bad' SCTs), with the fullness of the 'bad' tree, and similarly, demonstrate a subsequent inclusion proof that all of those certificates are incorporated into the STH generated 22:32 (or later), with the fullness of the 'good' tree?

Can you state how many certificates this affected?

Just wanting to touch base on this point, as we determine the next step.

It would seem that, on the basis of the evidence presented, it's appropriate to disqualify Venafi. What's unclear, and where part of the question came from, is whether consistency to the STH produced at 1488307345708 can be produced, or whether it's necessary to disqualify at an earlier STH.

I'm curious how others feel about this proposed course of action.

I think this is a super interesting case.

If I am understanding correctly, Venafi's log now contains all certificates they issued an SCT for.

The incorrect STH was issued for a log that consisted of a subset of certificates accepted.

Later STHs include all those certificates and more, but in a different order, and hence have an STH that cannot be linked to the incorrect STH. (Note: this is deduced from the nature of the failure)

However, if it can be shown that the incorrect STH corresponds to a subset of certs now logged, then it seems to me that this entirely meets the intent of the system, if not the letter.

And hence I would argue against disqualification, pending the above demonstration.

BTW, obviously the certificates in question and the times at which they were logged should be otherwise unremarkable. :-)

[Ryan Sleevi]

On Mon, Mar 6, 2017 at 4:06 PM, 'Ben Laurie' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

I think this is a super interesting case.

If I am understanding correctly, Venafi's log now contains all certificates they issued an SCT for.

I don't believe that's a correct understanding at this time.

 

The incorrect STH was issued for a log that consisted of a subset of certificates accepted.

Later STHs include all those certificates and more, but in a different order, and hence have an STH that cannot be linked to the incorrect STH. (Note: this is deduced from the nature of the failure)

However, if it can be shown that the incorrect STH corresponds to a subset of certs now logged, then it seems to me that this entirely meets the intent of the system, if not the letter.

And hence I would argue against disqualification, pending the above demonstration.

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

[Ben Laurie]

I'm not sure what you mean by a bad SCT?

 

[Ryan Sleevi]

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

While I appreciate the perspective of a 'recoverable' error, I think this unduly places burden on implementers - whether clients or monitors - to recognize and appropriately handle this scenario, and so I don't think it's necessarily reasonable or desirable to explore that line of reasoning. While there may be elements of less critical impact - as discussed in the past, a blown MMD _may_ be such an incident - producing a split view of the log is arguably one for which a log has difficulty recovering from. This is also consistent with past action and past discussion, so I'm not sure what new details or policy changes would accommodate the path you proposed. 

[Ben Laurie]

On 6 March 2017 at 21:30, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

I'm asking what a bad SCT is.

An SCT is a promise to include into a log. It is not related to any particular STH. Hence my confusion...

[Ryan Sleevi]

On Mon, Mar 6, 2017 at 4:36 PM, 'Ben Laurie' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

On 6 March 2017 at 21:30, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

I'm asking what a bad SCT is.

An SCT is a promise to include into a log. It is not related to any particular STH. Hence my confusion...

A 'bad' SCT in this case would be a promise issued after the production of the 'bad' STH. We can know - from the existence of the STH - the number of SCTs produced up to that point. However, any SCTs issued between the production of two STHs cannot be publicly known - ergo, they could be omitted from the log.

Imagine an 'evil' log (and I'm not accusing Venafi of this, I'm highlighting the challenge)

SCT A

SCT B

SCT C

STH (covering A-B-C)

SCT D

SCT E

SCT F

"Bad" STH (covering D-E-F)

SCT Z

SCT E

SCT D

SCT F

"Good" STH ( covering E-D-F)

We can show that the 'bad' and 'good' STHs fully incorporate the SCTs issued for certificates D/E/F, that's true - but the community cannot know about SCT Z unless/until it's produced, and no consistency proof can be provided for it.

I can understand and appreciate that you may argue this is true even under 'good' conditions, but I think the timeline of events provided makes it possible for this in a way that cannot be detected by comparing 'bad' STH to 'good' STH, and may represent additional operational failure.

[Andrew Ayer]

On Mon, 6 Mar 2017 21:06:22 +0000

"'Ben Laurie' via Certificate Transparency Policy"
<ct-p...@chromium.org> wrote:

Hi Ben,

I agree that the intent of logging all certificates has not
been violated. However, one of the most appealing aspects of CT
is that it allows programmatic verification that a log is meeting
its security properties. If Venafi is not disqualified, it will no
longer be possible to programmatically verify its append-only property.
Any monitor/auditor that sees the inconsistent STH will raise the alarm
that Venafi is misbehaving. (Thanks to gossip, this STH is never going
away.) In theory, we could develop an STH revocation system so that
monitors/auditors know to ignore this STH. (The system would also need
to convey the previous good STH so monitors can rewind.) Such a
system would take time to develop and add complexity to CT. Is it
worth it? Why can't logs be expected to be robust enough not to
accidentally sign STHs?

For recoverable policy violations, such as downtime, I'm in favor
of considering disqualification on a case-by-case basis. However, once
a log signs an STH it can't take it back. Therefore, signing
inconsistent STHs is not a recoverable error and should always lead to
disqualification.

Regards,
Andrew

[Steve Topletz]

Ryan, et al

This is to provide additional information about Venafi’s outage.  The sequence of events is described in Venafi’s post from March, 1^st.

 

Please note, between 1488320541206 and 1488321064440 no SCTs were issued. The tree signed at 1488321064440 is identical to 1488307345708. Please find below the STH blobs for 1488300146134 and 1488307345708 and their consistency proof.

[image001.png]

 

Incident Timeline: 

[image002.png] 

Specifically:

 

16:42 – 1488300146134 (tree size: 90,155):

080012220A20AC3B9AED7FA9674757159E6D7D575672F9D98100941E9BDEFFECA1313B75782D18D6FBE7ACA82B20ABC0052A205B645ACD435D07270BC2E5765D23D3759408E342C2CDE02D76716EC75FFAB512328702080410011A800245D60157117BA5F0B8BC30560B407F327BF92C6940C06E320598623E28A04504479B67CBD89B128F5C8D1AA0B22A4210A8BEBD8D197BF9F02AFE69F95BCC1B86D82228FF7B37467A02B521EA8F5CA100F653B03E788B599810CFCCB3F8118DDFF386583002D5B6F48EC7EE1B49DBCB59710BD3D91481D6FB8D2CD1D356259742045677055AA50638D74DBD30366E6DC8DA0BDABE2F47B31E3EA5FC64574A026A6712732891E2D9BAA3D293CECADC8CAC69124E4EB21EB6EE41F0F6C80A1D998B7A4197BF724047AD99D8C93FFCB4799DEF424083A8DDB26C2456293907B833D3EBA38DBD2B6F1B6F3EAB2009B876AB5D22FF44A9F9BDA3E0B6FF76749A944930

 

18:42 – 1488307345708 (tree size: 90,167): 080012220A20AC3B9AED7FA9674757159E6D7D575672F9D98100941E9BDEFFECA1313B75782D18ACB29FB0A82B20B7C0052A20B0C9EAB18C4B63229D17B40DAA995682720993A581B2EAA5106A3468D1C78518328702080410011A80024881C1C4D26EB6F1F23CE8629D11F97B0AEAB7A2CC273F9564E1A1D00693156A20B329B0EB16572C6F4D6D517FADDAC43943C06DCA34B0F1C5E0708B5D7B6D341244521B49CF026E575E240C76D739FF312EBB3D4524217899FDF3594521EA7C381322781053A8585ABED4A02FFF173171C9D7B1D562C71FAC2C273090D4EF21E8F87EC13D96AFECDFF3C5C2082261582FE360CD73B6F4C042CDCD17CEEEFD4A83FA5855BC945935A47FE0E23028037F43BF06987E6EC194F2129772A218E5BCDF8E94C9635817D76839D83CADDCF82DF4EFB8AD5C37F3AC9FE85CF8D36C4A4084A12C5B6085B7F25AD8D7C264390D7A94CBE46612AFDDC3AE888723065B8B25

 

22:32 – 1488321064440 (tree size: 90,167):

080012220A20AC3B9AED7FA9674757159E6D7D575672F9D98100941E9BDEFFECA1313B75782D18F8DBE4B6A82B20B7C0052A20B0C9EAB18C4B63229D17B40DAA995682720993A581B2EAA5106A3468D1C78518328702080410011A800285C9C3E20D31CC44CEBB16DFEB73831884DA33FCB0964DE476303B7326B851F2E6DE7B949A8F00360414121FC7B704106CA9D1140B6FA5A89EF92AE3186C00DB74C57090430D524F144A8B19DA62D4A7124A1525F5973C05728D4A853F1FABC8E1ED76870C1A57FAD2BB7852618AD2B872F574C2AE4B58164CC6F593EB1AE8C60EF5C87819C869AADB5342F9836DB8AEC74BBC35F0A51A291E91E29E89AC7395BE45E03E790DFC586047933DF77BFF983C940677F2F711A0CDDDD6B45481183476390065BF62AEEB53A703F9D1C7F662F538AF7BD7F1D9DFAD9A127041678EF38BFD4ECC035E29D4F5B7207E75BED5A57852709C70D68D06DD3D15D0FFA5338A

 

And here is the consistency proof (sizes 90155 and 90167):

 

{ "consistency": [ "tEgqDD0zLL4KViqAMmnzvh\/7TjlIYJk\/fhSe6cS8Jo4=", "p4eTTY4JYQxhXzcG3iFH1imhqglNbbpCzZmp6Ozw9wM=", "fb63f4aoOOFIArLA0JfdJwfB78tecSJwnVYLb5RhbN4=", "Lz2zMdy+Yti8LjZ9O0m7cKZTwP4bbNZ7glCRF7UGBHE=", "7LXxCtSf42LbuFjKdYX1vdOnNgS3y\/5c+09mDQkC6DU=", "jl3sEafdkLxR0QYG5\/+34P7+QFSoIehfuYZujm+E6GA=", "VH8nj3DKzwv9v3bcQeDKjw4a0H+GbIm1SIxU0z3J\/1E=", "2eQpspAlOaHniWW6exNICgPfge2u0BHXAU0bJoCxy+c=", "jsEyi+238H+\/SNiDNmoNeGTPeMkBEeM55WNKuuPrfeY=", "rcxt0aKSfv5MGCDJSwt6WDZffq9Nmp4SpQm5Xr376Lo=" ] }

 

 Also, attached are the certificate entries missing from the bad STH:

 

Certificate Entry 1:

AAAAAAFahec6SQABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA9IwggPOoAMCAQICEFHdYzrohqzQXz92eOeK4MMwDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzE1MTRaFw0yMDAyMjkxNzE1MTRaMCYxCzAJBgNVBAYTAkVTMRcwFQYDVQQDDA53d3cuYXhhc3ViLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALq5Ldz8hYgxxy\/QobBuB0tjXXjEFEI4Bq\/bB2jqx+0ut2F\/natsF3U7WSxFkHQxyhQ3AwOxsEGb5K0f3BlSbBpY0XFDZIKrUT+dsPRifiSBjNL1qXo690r2+NMs6WoqmpE0EB3UMfB1WXSgJAx+qeNiCCpgB5B8LZbYWUDf912S\/iLZQz9poGBHwQMO9CvVn5eYrmtSZMhFBLGVbfKTbD2jqXbjp5ioSMMLjvYgEFfAIka+RKw\/bOEl5jbi2WPvg+Mq+Qe8UuM8pmADXL3+\/qFDe0bn6u5aLkg3ge42M\/3\/HxYjuf\/tiLlKo0lUQdNt5UdldT6jgo11seCbepPWPHcCAwEAAaOCAbwwggG4MA4GA1UdDwEB\/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwCQYDVR0TBAIwADAdBgNVHQ4EFgQUzvFSUz7nUX4EzdlBvx6W5CDp2sMwHwYDVR0jBBgwFoAU15FOAcSwv\/jIZ5NEnOcz+q2TDK8wbwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5jb20wOQYIKwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5zZXJ2ZXIxLmNydDA4BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2Etc2VydmVyMS5jcmwwGQYDVR0RBBIwEIIOd3d3LmF4YXN1Yi5jb20wIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMFEGA1UdIARKMEgwCAYGZ4EMAQIBMDwGCysGAQQBgbU3AQIFMC0wKwYIKwYBBQUHAgEWH2h0dHBzOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xpY3kAAA==

 

Certificate Entry 2:

AAAAAAFahgdibgABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA\/0wggP5oAMCAQICEEhDIMaMJhjh2m1\/AlJEbKgwDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzUxMjlaFw0yMDAyMjkxNzUxMjlaMCUxCzAJBgNVBAYTAlNFMRYwFAYDVQQDDA1iYW5hbmZpc2suY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzCn01bi0msFPsdsDMJIP38ulDqUEJyP\/f8ibST2qSPKRG+WRTsCzREFH+QkLK+OgYF3lq7lSKLetOvk+KwP\/4\/aq56x+dAZ4shFnrTQEBvji\/+fUa7kyY4z6XoRnSmAv3feAOxHVDf9EcnW\/5uiyBWQofw7QkVT4Vb0TXkNMTIZAumXrAptjtd61meZAQraCO7AQqkrf38qRJzflmLDXdcI3yW\/2iXKnqzUmL9VVF0yC73rSAEOz3oi8gcARmDFqOfuXn7+37raLGSgdllrphnXJ\/ReV0+I2of8wgzIZ\/XLduqrantNlMGyq98tJGC\/V22YE1Ag3dO4IRxv96perVwIDAQABo4IB6DCCAeQwDgYDVR0PAQH\/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAJBgNVHRMEAjAAMB0GA1UdDgQWBBRzVmmCuPdBBj7Xy3Eu8JSuCEMygjAfBgNVHSMEGDAWgBTXkU4BxLC\/+Mhnk0Sc5zP6rZMMrzBvBggrBgEFBQcBAQRjMGEwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbTA5BggrBgEFBQcwAoYtaHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc2NhLnNlcnZlcjEuY3J0MDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3NjYS1zZXJ2ZXIxLmNybDBFBgNVHREEPjA8gg1iYW5hbmZpc2suY29tghF3d3cuYmFuYW5maXNrLmNvbYIYYXV0b21hdGlvbi5iYW5hbmZpc2suY29tMCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzBRBgNVHSAESjBIMAgGBmeBDAECATA8BgsrBgEEAYG1NwECBTAtMCsGCCsGAQUFBwIBFh9odHRwczovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5AAA=

 

Certificate Entry 3:

AAAAAAFahdOFfwABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMABPEwggTtoAMCAQICED7CHba2CHrr94tmyrCLJUQwDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzAyMzRaFw0yMDAyMjkxNzAyMzRaMCsxCzAJBgNVBAYTAlJVMRwwGgYDVQQDDBN3d3cudGVsdGFtYi5wZXJtLnJ1MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuuEYyZuCPM5oQcf\/m6GKGuufjJ33doHaX8EPU0G\/6Fuzu8TvfjXjK3s5pOfDNuCq0C9dq2v2XssJQa1uo937orJ6BScd+oNJ9TPY8Ps0QcAf99Z3bvZscOUauRCL4TV9hF53kDr\/8VLHdCMU55HlCbkW0rGmvypT0MhTwhkJ61xt6GNsCvxm4amL5faeHWc+HtWAUWmAxVWZakyQQbCrBBNot4XEx6Wyl2ethKjBDMVm3sqMczC7IrEiG7SQuB1uC+tQtNpgVUlTx8xMMgRUpb6uWMnmWguomnTL5Syr8QsMBnIQnIvehrOBlxDU8pV95wEvAvjbP3Dq7FNOeGK0qjh4TAAENbbZMOhEcKIGZKl7ESyLP+qMfSG2aC8Iik7y6yevEy8O713nnT7ynm8dWHDdgjQY20I+e\/Bg0ooC32jnfpse\/wPEvkeiQqn0YM6YkjoztOcjJdCpXBBFpkEta0HaEf0QzQm+ixYfXINnPFTTUb0EJE8sosLg1B4CspUvXlC+b0P0aT+V\/i9bnZQOmGY7JCVMFmBDcYA6qA9bKeqcAqoH417+y176uxPLOcLIDZ8Ffo51Tsp3KE7X+qZOgGbENyOjszR5wRJzHCYzKwMVMyiUn2d6EMQ419vB9\/gbzIXs\/Tg4DmLhUo25jEL1DORmmkIkAaegq\/1Aaf93g0cCAwEAAaOCAdYwggHSMA4GA1UdDwEB\/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwCQYDVR0TBAIwADAdBgNVHQ4EFgQUy7gyH9tHJJrlknwJI3baC7wid00wHwYDVR0jBBgwFoAU15FOAcSwv\/jIZ5NEnOcz+q2TDK8wbwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5jb20wOQYIKwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5zZXJ2ZXIxLmNydDA4BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2Etc2VydmVyMS5jcmwwMwYDVR0RBCwwKoITd3d3LnRlbHRhbWIucGVybS5ydYITdnBuLnRlbHRhbWIucGVybS5ydTAjBgNVHRIEHDAahhhodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS8wUQYDVR0gBEowSDAIBgZngQwBAgEwPAYLKwYBBAGBtTcBAgUwLTArBggrBgEFBQcCARYfaHR0cHM6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeQAA

 

Certificate Entry 4:

AAAAAAFahcwfewABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA+4wggPqoAMCAQICEEBTOqyyEFzaBu6HHaQDVfswDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNjU1MDRaFw0yMDAyMjkxNjU1MDRaMDQxCzAJBgNVBAYTAlVTMSUwIwYDVQQDDBxnaXQtaW50cm8uZGVycmVraGFycmlzb24uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAomL05zT2sGFpwQ7AdIpDEFDwXnKgwFYteGqFFNpGmLQ30U3NWXG2Wp4Ima7QvqL7aLhHvsZVu7DCcN\/VaxfjWBREDoxEiEcAcrDYS+YFNlFsHxZ+2mmNhRJ\/mBSOCN2Kd7HgdE97Bd1L4h6iR1j5Nxjw8eitzPD5Oh6D5Xnj3t8wLQqTBkcGqa6cIf+L2uFm5DJeRafd4pMbEpzLsiH+hzwIzdkG82HS+qApYsF5+BWCO1KHyiXmSeOR0nO8kLd\/Gk\/7bNVhApWzR5mI99dEcFn0TWylkW\/RISPVp9drfJRg6gHT5k2RLuO32XP58r6IopLOSJdjKJyx12Vc+E1xtQIDAQABo4IByjCCAcYwDgYDVR0PAQH\/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAJBgNVHRMEAjAAMB0GA1UdDgQWBBQw0c1jfd25r879khVSJV60GDoT7zAfBgNVHSMEGDAWgBTXkU4BxLC\/+Mhnk0Sc5zP6rZMMrzBvBggrBgEFBQcBAQRjMGEwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbTA5BggrBgEFBQcwAoYtaHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc2NhLnNlcnZlcjEuY3J0MDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3NjYS1zZXJ2ZXIxLmNybDAnBgNVHREEIDAeghxnaXQtaW50cm8uZGVycmVraGFycmlzb24uY29tMCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzBRBgNVHSAESjBIMAgGBmeBDAECATA8BgsrBgEEAYG1NwECBTAtMCsGCCsGAQUFBwIBFh9odHRwczovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5AAA=

 

Certificate Entry 5:

AAAAAAFahfmaewABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA+QwggPgoAMCAQICEBvpdto94iTZYqsJ6soyKMswDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxODAzNDdaFw0yMDAyMjkxODAzNDdaMCQxCzAJBgNVBAYTAkNaMRUwEwYDVQQDDAxtYWlsLm11em8uY3owggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCmmkov+HgPvDRylo70YtEEQbTXOZ\/SAJwZQTVxdv3LijpXHJNPJx9a8mInL8bXlY48mRf1NWKx869tK70WGG5tTDEAaBOVlw57ewAkJ33Byx4lfRMDKsj+gkgdn+bvVyv9Sgv6pAz2q1bOiAqHXE1C++NGNgNeuCZ6T013nvRoTtTE+NnFAB2Pwd3dO7wdwU+R3EbjUAW9ePdbKD3ucsnG33rYm33HByOM80\/6THmC5a\/+UFNqAfTIto0VZo\/zTM02ldj\/Cab8ocORRixBatmiOVQgcrNV3DMiKiPT9xTkEzp3Ld0iR06zYBj4oYP4VG+8DqfFZ0WJygGagqh867OFAgMBAAGjggHQMIIBzDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMAkGA1UdEwQCMAAwHQYDVR0OBBYEFD8hNDiwPuVDXKOIuLUElebEnCC9MB8GA1UdIwQYMBaAFNeRTgHEsL\/4yGeTRJznM\/qtkwyvMG8GCCsGAQUFBwEBBGMwYTAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Auc3RhcnRzc2wuY29tMDkGCCsGAQUFBzAChi1odHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zY2Euc2VydmVyMS5jcnQwOAYDVR0fBDEwLzAtoCugKYYnaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2NhLXNlcnZlcjEuY3JsMC0GA1UdEQQmMCSCDG1haWwubXV6by5jeoIUYXV0b2Rpc2NvdmVyLm11em8uY3owIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMFEGA1UdIARKMEgwCAYGZ4EMAQIBMDwGCysGAQQBgbU3AQIFMC0wKwYIKwYBBQUHAgEWH2h0dHBzOi8vd3d3LnN0YXJ0c3NsLmNvbS9wb2xpY3kAAA==

 

Certificate Entry 6:

AAAAAAFahemOVAABFbs+o+IxVNTHBpjNQYfX\/TBnxPC+OWLYxQLEtqkrAfMAA9YwggPSoAMCAQICEDqrJVoAnRG6cTJqzmH32lswDQYJKoZIhvcNAQELBQAweDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MSYwJAYDVQQDEx1TdGFydENvbSBDbGFzcyAxIERWIFNlcnZlciBDQTAeFw0xNzAyMjgxNzEzMzBaFw0yMDAyMjkxNzEzMzBaMCgxCzAJBgNVBAYTAkZJMRkwFwYDVQQDDBBmdy52aXJyYW50YWxvLmZpMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsVZIM6CJ07hAQQxIkgHdloTVX4jLblhP2rCruV9Q5aBbrz7HlBQscPpO\/Btu2NY5qVexeMoR9KFGbEystSQ2GBNhy4+1Cvbqpgh8Dco1gdWMgpIjLLyDnvU3s6T4tr5\/A3ApiJg2cqRqDf9jDM47wBVig\/BGAn83Wd4\/VyuMoajEzoz\/mK5+KdHL0AFZdngUrQ8ZT2zrMBNjpCnnktgPP8\/PV4zelWy+F4\/dfXP1nZii0AR0mVu1dW8c0weCA9yGsoA0LzoGq2x3ZiSF8jsucaezlT6iAMOPOwDCO6IH+59\/OiVSh9xidGz+VdDkcjt5TnBrfIsbXYas80poT8PyZwIDAQABo4IBvjCCAbowDgYDVR0PAQH\/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAJBgNVHRMEAjAAMB0GA1UdDgQWBBSVOeKX8yQHgvtctjcY16lHrJ1nbTAfBgNVHSMEGDAWgBTXkU4BxLC\/+Mhnk0Sc5zP6rZMMrzBvBggrBgEFBQcBAQRjMGEwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbTA5BggrBgEFBQcwAoYtaHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc2NhLnNlcnZlcjEuY3J0MDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3NjYS1zZXJ2ZXIxLmNybDAbBgNVHREEFDASghBmdy52aXJyYW50YWxvLmZpMCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzBRBgNVHSAESjBIMAgGBmeBDAECATA8BgsrBgEEAYG1NwECBTAtMCsGCCsGAQUFBwIBFh9odHRwczovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5AAA=

 

Summary:

 

In summary, please note that:

1         This issue was triggered by virtue of the AWS S3 outage, not an operator error.

2         No certificate data was lost – ALL of the certificate data is in the good/latest STH(s).

3         We have issued no SCTs during the outage or during the time that the instance with the “bad” head was active. We are happy to share any data we have to support this claim.

4         In the event of a similar issue in the future, we have identified the steps that we will take to prevent inconsistent STHs from ever being published.

[Tom Ritter]

On 6 March 2017 at 13:30, Andrew Ayer <ag...@andrewayer.name> wrote:
> On Mon, 6 Mar 2017 14:01:55 -0500
> Ryan Sleevi <rsl...@chromium.org> wrote:
>> It would seem that, on the basis of the evidence presented, it's
>> appropriate to disqualify Venafi. What's unclear, and where part of
>> the question came from, is whether consistency to the STH produced
>> at 1488307345708 can be produced, or whether it's necessary to
>> disqualify at an earlier STH.
>>
>> I'm curious how others feel about this proposed course of action.
>
> Hi Ryan,
>
> I agree that disqualification is appropriate.

Me too.

I'm excited to see Gen2 get qualified though, and I hope Venafi
considers re-upping with a second replacement log.

-tom

[Andrew Ayer]

Based on the latest information provided by Venafi, the tree at
1488307345708 is identical to the tree at 1488321064440, which is one
of the inconsistent STHs I produced in my original report. Therefore,
the last good STH is actually the one at 1488300146134 (tree size
90155) - this is last point at which all monitors had the same view of
the log.

However, it's unclear why the last good STH matters for a disqualified
log. Will Chrome downright ignore any SCT with a later timestamp, even
for determining the diversity requirement for embedded SCTs?

Regards,
Andrew

[Ryan Sleevi]

On Tue, Mar 7, 2017 at 12:26 AM, Andrew Ayer <ag...@andrewayer.name> wrote:

Based on the latest information provided by Venafi, the tree at
1488307345708 is identical to the tree at 1488321064440, which is one
of the inconsistent STHs I produced in my original report.  Therefore,
the last good STH is actually the one at 1488300146134 (tree size
90155) - this is last point at which all monitors had the same view of
the log.

However, it's unclear why the last good STH matters for a disqualified
log.  Will Chrome downright ignore any SCT with a later timestamp, even
for determining the diversity requirement for embedded SCTs?

Regards,
Andrew

Andrew,

Does the "Certificate Transparency in Chrome" policy, linked to from https://www.chromium.org/Home/chromium-security/certificate-transparency , not sufficiently address this?

"“Once or currently qualified” means that the log was qualified or pending qualification at the time of

certificate issuance, that the log was accepted prior to the time of check, but that the log may have

been disqualified following acceptance, prior to the time of check."

The 'last good' STH is used (in conjunction with other, still qualified, SCTs) to determine whether or not the log was qualified at time of certificate issuance. Disqualification can be retroactive to the point of the last consistent view - meaning the potential for some subset of certificates that may need to be reissued - or it can be forward looking - meaning CAs need to migrate to new logs to satisfy the SCT requirements.

[Ben Laurie]

On 6 March 2017 at 21:42, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:36 PM, 'Ben Laurie' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

On 6 March 2017 at 21:30, Ryan Sleevi <rsl...@chromium.org> wrote:

On Mon, Mar 6, 2017 at 4:25 PM, Ben Laurie <be...@google.com> wrote:

However, as described, there's still the possibility of one or more bad SCTs introduced after the production of the 'bad' STH, so I'm not sure the argument fully applies.

I'm not sure what you mean by a bad SCT?

Bad STH - an STH inconsistent with the state of the log, for which one or more SCTs may have been issued after that, and for which no consistency proof can be demonstrated towards, since no STH following those SCTs was issued.

I'm asking what a bad SCT is.

An SCT is a promise to include into a log. It is not related to any particular STH. Hence my confusion...

A 'bad' SCT in this case would be a promise issued after the production of the 'bad' STH. We can know - from the existence of the STH - the number of SCTs produced up to that point. However, any SCTs issued between the production of two STHs cannot be publicly known - ergo, they could be omitted from the log.

But that's always true - i.e. a log can always issue an SCT and not log the corresponding cert. The only way you ever find out is to see the SCT, which is also the case here, isn't it?

 

Imagine an 'evil' log (and I'm not accusing Venafi of this, I'm highlighting the challenge)

SCT A

SCT B

SCT C

STH (covering A-B-C)

SCT D

SCT E

SCT F

"Bad" STH (covering D-E-F)

SCT Z

SCT E

SCT D

SCT F

"Good" STH ( covering E-D-F)

We can show that the 'bad' and 'good' STHs fully incorporate the SCTs issued for certificates D/E/F, that's true - but the community cannot know about SCT Z unless/until it's produced, and no consistency proof can be provided for it.

I can understand and appreciate that you may argue this is true even under 'good' conditions,

I do.

 

but I think the timeline of events provided makes it possible for this in a way that cannot be detected by comparing 'bad' STH to 'good' STH, and may represent additional operational failure. 

In terms of knowing about Z, I don't see how this error changes anyone's position at all.

 

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACvaWvaxyxidTDCoQ5XLmWHimqhif9%3DcuA9kKyQ3TTj84Y2z%2Bg%40mail.gmail.com.

[Ryan Sleevi]

On Tue, Mar 7, 2017 at 8:23 AM, Ben Laurie <be...@google.com> wrote:

I can understand and appreciate that you may argue this is true even under 'good' conditions,

I do.

 

but I think the timeline of events provided makes it possible for this in a way that cannot be detected by comparing 'bad' STH to 'good' STH, and may represent additional operational failure. 

In terms of knowing about Z, I don't see how this error changes anyone's position at all.

Before we continue this discussion much further, it may be helpful to take a step back and re-evaluate:

Can you concretely argue/advocate why you believe disqualification for violating the consistency of the Merkle Tree is inappropriate? Can you indicate what costs you believe will be born by the various stakeholders - monitor/auditors, clients, CAs, the ecosystem - both with and without your proposed course of action?

I'm extremely uncomfortable with the notion of allowing "known bad" SCTs into the ecosystem, and the trust that undermines. Further, I'm extremely opposed to introducing the idea of "SCT revocation", a cost that is unreasonable to request of Chrome, let alone for Chrome to request of the broader CT ecosystem.

[Andrew Ayer]

On Tue, 7 Mar 2017 01:52:58 -0500
Ryan Sleevi <rsl...@chromium.org> wrote:

> On Tue, Mar 7, 2017 at 12:26 AM, Andrew Ayer <ag...@andrewayer.name>
> wrote:
> >
> > Based on the latest information provided by Venafi, the tree at
> > 1488307345708 is identical to the tree at 1488321064440, which is
> > one of the inconsistent STHs I produced in my original report.
> > Therefore, the last good STH is actually the one at 1488300146134
> > (tree size
> > 90155) - this is last point at which all monitors had the same view
> > of the log.
> >
> > However, it's unclear why the last good STH matters for a
> > disqualified log. Will Chrome downright ignore any SCT with a
> > later timestamp, even for determining the diversity requirement for
> > embedded SCTs?
> >
> > Regards,
> > Andrew
> >
>
> Andrew,
>
> Does the "Certificate Transparency in Chrome" policy, linked to from
> https://www.chromium.org/Home/chromium-security/certificate-transparency ,
> not sufficiently address this?

It wasn't clear how the STH played a part, hence my question. The bit
I was missing was that disqualification could be retroactive. Thanks
for clearing that up with your explanation.

Regards,
Andrew

[Ryan Sleevi]

On Tue, Mar 7, 2017 at 12:00 PM, Andrew Ayer <ag...@andrewayer.name> wrote:

It wasn't clear how the STH played a part, hence my question.  The bit
I was missing was that disqualification could be retroactive. Thanks
for clearing that up with your explanation.

Regards,
Andrew

Right, I totally see where that came from, and I think it'll be necessary to take a stab to fix (feel free to propose yourself, if it'd help)

The overall goal is this - for any log recognized as 'qualified' or 'disqualified' in Chrome, our goal is that you should be able to assure the inclusion of all SCTs by obtaining inclusion proofs to the provided STH (delivered by Google). Any failure to provide for this is reasonable grounds for disqualification, modulo the discussions of the timeliness of this proof. That is, the MMD is effectively the window for how long to wait before you can obtain this proof.

The Venafi case is interesting, and I'm hoping someone can formulate the right questions (and I believe some of the folks at Google who work on CT are drafting them), since, as Ben Laurie is highlighting, it's arguably possible to show this goal is met; that is, every SCT Venafi issued during the 'bad' STH is possible to obtain an inclusion proof to the 'good' STH (they just have differing entry IDs from the bad and good trees)

This is a credit to Venafi for maintaining sufficient logging to be able to demonstrate this, modulo the 'unknown' element Ben and I were discussing, because it potentially creates the opportunity for trust, but that opportunity must be weighed against both the immediate and ecosystem risk - hence, why I posed the question to this list as to whether disqualification is appropriate in this specific case.

If you can imagine another scenario, where a log is unable to provide inclusion proofs for all issued SCTs (whether they realize it or not), then I think it'd be necessary and advisable to 'retroactively' disqualify, so that we maintain the property of all SCTs (up to time X) have inclusion proofs.

I'm open to suggestions as to how to clarify that with respect to 'qualified at time of issuance' being a state that can retroactively change in the event of violations of the policy, as appropriate to achieve the end state of always-verifiable consistency.

[Paul Hadfield]

Hi Andrew,

have you/CertSpotter retained a copy of the Venafi get-entries response for the range of entries from 90155 to 90161, captured at the time you spotted the unverifiable STH@90161 ? Or records from which that response could be reconstructed?  If so I'd like to see them.

I'm interested to see if the entries returned match those Google's monitor retrieved.

The two monitors that I help operate at Google didn't retrieve Venafi's unverifiable 90161 STH, since we were unlucky enough for both systems to sample on either side of the 2 minute period when it was live.

thanks,

Paul

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20170307090021.2f7ba950457c201e2436ef77%40andrewayer.name.

[Kurt Roeckx]

On Tue, Mar 07, 2017 at 05:19:36PM +0000, 'Paul Hadfield' via Certificate Transparency Policy wrote:
> Hi Andrew,
>
> have you/CertSpotter retained a copy of the Venafi get-entries response for
> the range of entries from 90155 to 90161, captured at the time you spotted
> the unverifiable STH@90161 ? Or records from which that response could be
> reconstructed? If so I'd like to see them.
>
> I'm interested to see if the entries returned match those Google's monitor
> retrieved.
>
> The two monitors that I help operate at Google didn't retrieve Venafi's
> unverifiable 90161 STH, since we were unlucky enough for both systems to
> sample on either side of the 2 minute period when it was live.

I've attached what I've downloaded, which I think are the bad
ones:
-rw-r--r-- 1 kurt kurt 43606 2017-02-28 22:23:56 +0000 entries_90155_90160


Kurt

[Andrew Ayer]

On Tue, 7 Mar 2017 17:19:36 +0000
Paul Hadfield <hadf...@google.com> wrote:

> have you/CertSpotter retained a copy of the Venafi get-entries
> response for the range of entries from 90155 to 90161, captured at
> the time you spotted the unverifiable STH@90161 ? Or records from
> which that response could be reconstructed? If so I'd like to see
> them.

Hi Paul,

Unfortunately, I am not currently storing the necessary information.
However, I do know the leaf hashes:

90155 p4eTTY4JYQxhXzcG3iFH1imhqglNbbpCzZmp6Ozw9wM=
90156 UMJNeHdASgUToTiMFjmqcKvhx275E+6s8uCJBtMuqHc=
90157 r+r7qrYh4i9lWa1nzVO86nj+9c6BGZq5i36IRzgPGl4=
90158 NPLfppxk5a1xQZb/tmCK3oJ/WFflOaxx4pNE49Tf+8Y=
90159 /QLcMe6AYC4MlZ4mVDHMMPV/m/hRdiYlAqED8Vg4roI=
90160 7lbvrvKd/3nP3UM18vnHAYxwmriy3r8QQ3kF1K6KK8s=

Since Venafi claims these entries are all in the other view of their log,
it should be possible to find the preimages of these hashes there.

Regards,
Andrew

[Steve Topletz]

Paul,

  Here is the entry mapping between the two datasets:

New	Temp
90156	90156
90160	90157
90161	90158
90163	90159
90165	90160

Steve

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

[Ben Laurie]

I'm not suggesting that "known bad" SCTs are permitted, but I am still confused by what you mean by a "bad SCT" (or now, a "known bad SCT"). AFAICS, you mean an SCT that has not been included in the log, but if that's what you mean, I don't see what this situation does to make it any easier to create such an SCT or any harder to detect it.

My point is that the consistency guarantee we are looking for, in a somewhat abstract sense, is that future logs contain everything that past logs did. The easiest way for this to happen is for logs to be strictly append-only, but it is also possible to achieve the property less strictly, and it seems in this case a less-strict consistency is demonstrable. Hence my argument against disquailification.

Running logs that are 100% reliable is _provably_ impossible, so we must expect mis-steps. My view is that where those mis-steps are explained, fixed and demonstrably do no harm, they should not lead to disqualification, which is enormously costly to the log operator (for example, ask the CT team what it cost to deal with Aviator, it was not pretty), and, I suspect, the ecosystem as a whole.

 

Now, I agree that permitting this less-strict consistency introduces cost for all parties. In this case, I suspect the cost can be boiled down to "there's one STH you should just pretend doesn't exist, but don't worry, we have manually verified that its OK", which doesn't seem like a hard thing to add to most s/w.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACvaWvZ8F%2BY9OuUxi5TLVynniWCLnHXucEOGMKWkYKOXN9EHjQ%40mail.gmail.com.

[Andrew Ayer]

On Tue, 7 Mar 2017 12:14:14 -0500
Ryan Sleevi <rsl...@chromium.org> wrote:

> The overall goal is this - for any log recognized as 'qualified' or
> 'disqualified' in Chrome, our goal is that you should be able to
> assure the inclusion of all SCTs by obtaining inclusion proofs to the
> provided STH (delivered by Google). Any failure to provide for this
> is reasonable grounds for disqualification, modulo the discussions of
> the timeliness of this proof. That is, the MMD is effectively the
> window for how long to wait before you can obtain this proof.
>
> The Venafi case is interesting, and I'm hoping someone can formulate
> the right questions (and I believe some of the folks at Google who
> work on CT are drafting them), since, as Ben Laurie is highlighting,
> it's arguably possible to show this goal is met; that is, every SCT
> Venafi issued during the 'bad' STH is possible to obtain an inclusion
> proof to the 'good' STH (they just have differing entry IDs from the
> bad and good trees)

This goal is flawed. Consider a malicious log that presents two views:
one to Google, and another to monitors. The view presented to Google
incorporates all submitted certificates within the MMD. The view
presented to monitors omits misissued certificates. This would render
CT useless, as monitors would fail to detect the misissued certificates.
Yet it would meet the goal stated above since it would be possible to
obtain inclusion proofs from every SCT to an STH delivered by Google.
If caught, the malicious log could even claim the alternative view
was a mistake caused by an operational problem.

Venafi may not be malicious, but their actions are indistinguishable
from those of a malicious log. They've presented a different view of
the log to some monitors, and from the perspective of those monitors,
Venafi has now issued over 1,000 SCTs and counting that are absent from
the log. Even though Venafi is once again presenting the same view to
everyone, it doesn't help the monitors which saw the bad view.
Auditing monitors like Cert Spotter are rejecting the new entries
because they do not produce the expected root hash and are therefore
not part of the Merkle Tree as the monitor sees it. Non-auditing
monitors are now missing certificates between indexes 90157 and 90160
inclusive and don't even realize it.

I don't think this case is interesting. It is causing the problems
that one would expect to be caused by presenting a split view of the
log, either maliciously or non-maliciously. Venafi should be
disqualified, and the CT Log Policy should be updated to prescribe
automatic disqualification for any log that produces inconsistent STHs,
as I can't imagine any scenario under which a log could recover from its
production of inconsistent STHs.

Regards,
Andrew

[Ryan Sleevi]

On Wed, Mar 8, 2017 at 11:20 AM, Andrew Ayer <ag...@andrewayer.name> wrote:

This goal is flawed.  Consider a malicious log that presents two views:
one to Google, and another to monitors.  The view presented to Google
incorporates all submitted certificates within the MMD.  The view
presented to monitors omits misissued certificates.  This would render
CT useless, as monitors would fail to detect the misissued certificates.
Yet it would meet the goal stated above since it would be possible to
obtain inclusion proofs from every SCT to an STH delivered by Google.
If caught, the malicious log could even claim the alternative view
was a mistake caused by an operational problem.

Apologies for phrasing it poorly, as I do agree with what you've stated here.

 

Venafi may not be malicious, but their actions are indistinguishable
from those of a malicious log. They've presented a different view of
the log to some monitors, and from the perspective of those monitors,
Venafi has now issued over 1,000 SCTs and counting that are absent from
the log. Even though Venafi is once again presenting the same view to
everyone, it doesn't help the monitors which saw the bad view.
Auditing monitors like Cert Spotter are rejecting the new entries
because they do not produce the expected root hash and are therefore
not part of the Merkle Tree as the monitor sees it.  Non-auditing
monitors are now missing certificates between indexes 90157 and 90160
inclusive and don't even realize it.

Right, and I think that's something that Ben is not fully appreciating. I think Ben is operating on the basis that the system can 'fix' itself by obtaining inclusion proofs to the latest STH, and may have designed a monitor that does that - I don't know. You've taken an alternative approach - to reject such certificates the moment an issue is detected, and to continue rejecting it (e.g. no "alert the badness but also accept the badness" mode)

I don't think this case is interesting.  It is causing the problems
that one would expect to be caused by presenting a split view of the
log, either maliciously or non-maliciously.  Venafi should be
disqualified, and the CT Log Policy should be updated to prescribe
automatic disqualification for any log that produces inconsistent STHs,
as I can't imagine any scenario under which a log could recover from its
production of inconsistent STHs.

Here's where I disagree. I think it is interesting, because the conversation here with Ben advocating against disqualification, and myself (and, seemingly, a number of others) advocating for disqualification, is that either approach presupposes certain behaviours of monitors. My understanding (perhaps incorrectly) of Ben's position is that monitors should accept the 'new' STH - even if it has now split from a consistent state - and merely alert when such splits are detected. It's interesting because I'm not sure that he's wrong here in how monitors 'could' work, but I'm not sure he's right here in how monitors 'should' work. You've obviously taken a different approach, as a monitor, and as such, there's a tension in the ecosystem with how monitors should handle such inconsistent views.

To the second half of your message - automatic disqualification - I think that largely hinges on the above discussion about how monitors should behave / be expected to behave and whether changes are warranted or desired. 

[Ryan Sleevi]

On Wed, Mar 8, 2017 at 10:56 AM, 'Ben Laurie' via Certificate Transparency Policy <ct-p...@chromium.org> wrote:

Running logs that are 100% reliable is _provably_ impossible, so we must expect mis-steps. My view is that where those mis-steps are explained, fixed and demonstrably do no harm, they should not lead to disqualification, which is enormously costly to the log operator (for example, ask the CT team what it cost to deal with Aviator, it was not pretty), and, I suspect, the ecosystem as a whole.

I think this is the crux of our continued disagreement for the past several years.

I agree we should learn from these missteps, and adjust policies as appropriate when we can be confident that these missteps did no harm. However, I also think that we should err on the side of caution, and attempt to operate beyond approach, weighing the ecosystem impact.

You're positing that the impact of disqualification - 6 (for the time in question) to 1,000 certificates (following qualification) - is greater than the impact to monitors and relying parties, both in terms of software and trust in the ecosystem. I think I'd have to disagree in this case.

 

Now, I agree that permitting this less-strict consistency introduces cost for all parties. In this case, I suspect the cost can be boiled down to "there's one STH you should just pretend doesn't exist, but don't worry, we have manually verified that its OK", which doesn't seem like a hard thing to add to most s/w.

I'm very uncomfortable with this statement, and think this approach to problem solving is unlikely to work out for the benefit of the ecosystem. This is yet another argument in favor of disqualification, because if "we" are going to require changes to software, it should be ideally with adequate time and discussion. Alternatively, we are effectively communicating that CT was not as well thought out as it hoped to be, and as a consequence, everyone should be prepared to update their code at a moments' notice. I don't think that's practical or realistic for code - although it may be and arguably is for the set of qualified logs - and so the consistent approach is to disqualify the log, at the last known good point, which will cause these 1,006 certificates to potentially fail policy compliance, and then collaboratively investigate what the norms and expectations are and whether "pretend this STH doesn't exist" is consistent with the description and goals of CT. 

[Ryan Sleevi]

In the effort of reducing any further ecosystem disruption in the event of disqualification, future or retroactive, I think we should try to reach a conclusion within the next day.

If anyone has any perspectives to share - whether agreeing or disagreeing with the various positions advocated on this thread - I think it would be great to hear 

[Andrew Ayer]

It is not so straightforward for a monitor to simply accept a 'new'
view. A monitor can use an inclusion proof to reconstruct the state of
the tree, but if it wants to avoid missing certificates (a hard
requirement for a monitor in my opinion), it needs to rewind its
position in the log to some point before the fork.

If a monitor retains old STHs, it can try each previous STH in turn
until it finds one with a valid consistency proof to the log's latest
STH, and restart monitoring from that position.

However, there is otherwise no need for monitors to retain old STHs, and
the self-hosted version of Cert Spotter does not, since it is intended
to be lightweight and not require lots of storage. If a monitor is not
retaining old STHs, it would need to obtain rewind instructions from
a trusted source or rewind to the beginning of the log and rescan the
entire thing.

For these reasons, implementing rewinding is rather unappealing to me.
Also, a monitor implementer reading RFC6962 or RFC6962bis would
not get the impression that rewinding is necessary, since violating the
append-only property is something that logs are just not supposed to do.

On top of this, there are the challenges I mentioned previously with
revoking STHs so they don't cause false alarms when gossiped around.

Before we go down this route, I think we need to consider:

1. Can we provide better guidance to log operators to help them avoid
signing inconsistent STHs?

2. If signing inconsistent STHs is unavoidable, is disqualifying a log
too disruptive to the ecosystem?

3. If so, can we reduce the disruption to an acceptable level (e.g. by
making it easier to spin up a replacement quickly)?

Regards,
Andrew

[Steve Topletz]

SUMMARY:

On March 13th of 2017 at 16:05:48 GMT a load balancer error caused Venafi ‘ctlog.api.venafi.com’ log to generate and publish inconsistent STHs. This represented a violation of section 3 of RFC 6962 which states:

A log is a single, ever-growing, append-only Merkle Tree of such certificates.

The external manifestation was similar to the previous availability event on 02/28 however in this case the database file could not be found in the backup bucket so the server started with an empty database and the head was signed with a tree size of 0.

 

The cause was a concurrency in the back-up procedure during a termination of the instance which caused the pointer to the datafile to be updated before the database file itself was uploaded to S3. The backup process is executed in a single thread on the server and each backup should have waited for the previous one to finish before starting but there must be a codepath that does not do this.

IMPACT:

The inconsistent tree head was served via approximately 100 GET requests.

ROOT CAUSE:

The following is a chronological account (times in GMT) of the cause for the inconsistent STH generation.

15:59:56 – successful backup (updated last backup pointer to 16:03:03)

16:03:03 – unsuccessful backup.

16:05:15 – new instance is unable to find a backup at 16:03:03 and instantiates an empty database.

16:05:34 – STH with size of the tree 0 is published.

16:05:48 – STH visible externally.

16:07:38 – instance removed from the load balancer.

16:40:01 – correct STH visible

22:45:xx – the missing entry from 16:03:03 was added to the log and published

REMEDIATION AND PREVENTION:

We are currently in the process of discussing a mitigation strategy, but wanted to alert the community as soon as possible.

[Andrew Ayer]

Hi Steve,

On Mon, 13 Mar 2017 17:38:56 -0700 (PDT)
Steve Topletz <steve....@venafi.com> wrote:

> On March 13th of 2017 at 16:05:48 GMT a load balancer error caused
> Venafi ‘ ctlog.api.venafi.com’ log to generate and publish
> inconsistent STHs.

Is "STHs" meant to be plural here? Elsewhere in your report, only a
single erroneous STH, with a tree size of 0, is mentioned.

Regards,
Andrew

[Steve Topletz]

No, a single STH tree size 0.

ST

[Al Cutter]

Hi Steve

On Tue, Mar 14, 2017 at 5:40 PM, Steve Topletz <steve....@venafi.com> wrote:

No, a single STH tree size 0.

I don't suppose it had a zero timestamp too, did it? ...

 

ST

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/F0D7E127-B543-41B0-89F3-DE5D1212F178%40venafi.com.

[Martijn Croonen]

I think this is the STH in question. My monitor saw it for the first time at 2017-03-13 16:06:29:

{

  "tree_size": 0,

  "timestamp": 1489421121050,

  "sha256_root_hash": "47DEQpj8HBSa+\/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",

  "tree_head_signature": "BAEBAJSY7ONUrcYLhgv3iZEbJssXngLNk4xhTO9AlrC6JK+cATbtkTHDhSiVS9JP1wNSPn9kg4rNkkauYKngNxhTVYBhoq5mwYicuokOeX20bRo36NmgKBkO+QcF4LfJlEe68JN0v9uYxmUdng0GFJCfWjP8lJ+BkMkFoVmCn18RPlIPPFHGSIbN0QBnU\/HX9Z2t7JsydER9fx6yjpz7DZBmkHYe6hda+z5sRqakHN1x3yp3qEYTb\/YAN9gI565lYwJ5uPfcZ4sH+6Bg+pSDOrt2IBnSR01vrlbBSiH8PjKVM2kdQ1KFwUDsniexP+dysmeJnbyeMi8xX2iGvGpik2JKWZY="

}

This was followed by a new STH at 2017-03-13 16:40:28 with tree_size 91950:

{

  "tree_size": 91950,

  "timestamp": 1489422043339,

  "sha256_root_hash": "ODMU0N0p6swDLznJ1xrBBindXAALz3d8lWrVRwLxYXs=",

  "tree_head_signature": "BAEBADYsu2429on5VPRgaJO\/XpuicKBCGIPCJoQLgpBKyg2ky7yBX0+xdj6m8f7nmlLBjZxrSrO9KtxCIwLmHp\/7G3bWwwDqtvL6alFs6kAwOerzoNcn7rkorrcV6pOXDGbSyiC4jllAO3erDLmjqpoTRWRC\/DcNJHNs5grBq+eGSBfhsmSn1ZS8\/vf16bGxW2yhVJpYp90RCXAU5TAp4v0k9kqGWMSSLTe\/CHY+VAPapJRB5ukK1E04GVhy+OnRN8cdfxwZ8tLBpPzbIBeMkkX+s2dakBoZnoB5hGPHH2awpff\/OHOITdjb3NF5NsiklYjzb6a8YRzXyceHzUI96XTrvi8="

}

The last STH my monitor saw before the invalid STH was with tree_size 91945:

{

  "tree_size": 91945,

  "timestamp": 1489413595694,

  "sha256_root_hash": "aA1xRITy5jpCH+hPnBI236+z4wSjBgIQCDaOXfOu9\/g=",

  "tree_head_signature": "BAEBAJQmdlQ91QsJS4+9q3IHdtyxXI\/0zeGd0PVNUq2JZe628S8QnGJWxp73bkqo+astAYMcI7F6aT5C7sBgEtDniMeySPT1W6ZQEoMgkwiI92z2j5tUnZgr8KDi7IV4085rVB1fbAyU7CpKZuzJ0zpJ1X57oSGQdEN7CsALRlclw74VyEx+qeBpZBvaE86kQRYs0E4K4MEsOHGa6xw8+\/KB3iR0Pg6gAs3QhyPCzMnUqIOXWY7MuhI4tJ\/FrsnDocFBr6r62t175FB81iSn91Qq8Vu1d3rHQryAhHL\/bPfrkUIkoORUJC6XMhw1Qy7SW+MF+5eAgGRu10JykSitbGPhffk="

}

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACM%3D_OcHcoRd82XCnpHeorLzGCiU6zFm46LaxRLDoZzfaEqD%2Bg%40mail.gmail.com.

[Steve Topletz]

Sadly no. J

 

From: 'Al Cutter' via Certificate Transparency Policy <ct-p...@chromium.org>
Reply-To: Al Cutter <a...@google.com>
Date: Tuesday, March 14, 2017 at 10:50 AM
To: Steve Topletz <steve....@venafi.com>
Cc: Certificate Transparency Policy <ct-p...@chromium.org>
Subject: Re: [ct-policy] Re: Venafi has produced inconsistent STHs

 

Hi Steve

 

On Tue, Mar 14, 2017 at 5:40 PM, Steve Topletz <steve....@venafi.com> wrote:

No, a single STH tree size 0.

 

I don't suppose it had a zero timestamp too, did it? ...

 

ST

On 3/13/17, 5:54 PM, "Andrew Ayer" <ag...@andrewayer.name> wrote:

    Hi Steve,

    On Mon, 13 Mar 2017 17:38:56 -0700 (PDT)
    Steve Topletz <steve....@venafi.com> wrote:

    > On March 13th of 2017 at 16:05:48 GMT a load balancer error caused
    > Venafi ‘ ctlog.api.venafi.com’ log to generate and publish
    > inconsistent STHs.

    Is "STHs" meant to be plural here?  Elsewhere in your report, only a
    single erroneous STH, with a tree size of 0, is mentioned.

    Regards,
    Andrew


--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/F0D7E127-B543-41B0-89F3-DE5D1212F178%40venafi.com.

 

--

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To post to this group, send email to ct-p...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACM%3D_OcHcoRd82XCnpHeorLzGCiU6zFm46LaxRLDoZzfaEqD%2Bg%40mail.gmail.com.

[Ryan Sleevi]

To wrap this thread up, I've posted https://groups.google.com/a/chromium.org/d/msg/ct-policy/KMAcNT3asTQ/UgJ70hvBBQAJ

As I mentioned in that mail, but want to stress here, Venafi's really set a new bar for disclosure and details, and the ability to reconstruct the certificates from the logs shows an attention to detail and robustness that I think all log operators should strive for.

This is very much an unfortunate situation. While it's great that Venafi has another log in the compliance processing phase, and I hope they'll continue to see it through to completion, it seemed appropriate to err on the side of caution here, especially given the implementation state of monitors.

So where do we go from here?

I would love to gather more feedback - from log operators, from monitors, from other CT clients - about what changes should happen in the ecosystem to mitigate such issues, if at all possible. Running a CT Log Server should not an impossible goal - and we should definitely take advantage of this situation to figure out if and how these sorts of things can be recovered from and/or mitigated.

I'll also be trying to follow-up with some of the action items related from our CT Policy Days, as I think this again highlights the opportunities and benefits of a more agile response in the ecosystem - which hopefully will make such incidents 'recoverable' through the establishment of additional logs (for example).