Old STH from Venafi

Kurt Roeckx

unread,

Apr 24, 2017, 6:24:33 PM4/24/17

to ct-p...@chromium.org

Today I received an STH from Venafi that was over 1 day old,
which is kind of unexpected, but I don't think there is anything
wrong with it. What I saw was:
tree_size | timestamp | first_seen | diff
-----------+----------------------------+-------------------------------+-----------------------
9758801 | 2017-04-23 00:11:24.765+00 | 2017-04-23 00:11:26.500233+00 | 00:00:01.735233
9746580 | 2017-04-22 23:41:33.207+00 | 2017-04-22 23:41:47.073396+00 | 00:00:13.866396
9746580 | 2017-04-22 23:41:29.999+00 | 2017-04-22 23:41:31.62064+00 | 00:00:01.62164
9727131 | 2017-04-22 23:26:16.22+00 | 2017-04-22 23:26:20.46088+00 | 00:00:04.24088
9727131 | 2017-04-22 23:11:32.215+00 | 2017-04-22 23:11:41.594376+00 | 00:00:09.379376
9727131 | 2017-04-22 23:11:24.611+00 | 2017-04-22 23:11:28.768941+00 | 00:00:04.157941
9727131 | 2017-04-22 23:11:24.587+00 | 2017-04-24 06:08:25.839592+00 | 1 day 06:57:01.252592
9727131 | 2017-04-22 22:51:41.737+00 | 2017-04-22 22:52:08.324793+00 | 00:00:26.587793
9727131 | 2017-04-22 22:50:17.287+00 | 2017-04-22 22:50:28.349308+00 | 00:00:11.062308
9727131 | 2017-04-22 22:47:03.646+00 | 2017-04-22 22:49:52.559005+00 | 00:02:48.913005
9718760 | 2017-04-22 22:22:59.255+00 | 2017-04-22 22:23:19.48791+00 | 00:00:20.23291
9717745 | 2017-04-22 22:01:58.432+00 | 2017-04-22 22:20:16.213508+00 | 00:18:17.781508

What is also strange is that there are 2 STHs within 0.024 second.

Or ordered by first_seen:
10507800 | 2017-04-24 06:34:20.515+00 | 2017-04-24 06:34:51.893492+00 | 00:00:31.378492
10483512 | 2017-04-24 06:25:44.773+00 | 2017-04-24 06:25:48.496335+00 | 00:00:03.723335
10483512 | 2017-04-24 06:18:37.701+00 | 2017-04-24 06:18:50.572158+00 | 00:00:12.871158
9727131 | 2017-04-22 23:11:24.587+00 | 2017-04-24 06:08:25.839592+00 | 1 day 06:57:01.252592
10483512 | 2017-04-24 06:04:20.205+00 | 2017-04-24 06:04:56.371312+00 | 00:00:36.166312
10483512 | 2017-04-24 06:02:43.426+00 | 2017-04-24 06:03:00.233033+00 | 00:00:16.807033
10472472 | 2017-04-24 05:31:17.851+00 | 2017-04-24 05:31:37.50333+00 | 00:00:19.65233

Earlier this month I also saw an STH with timestamp of 1970-01-01 00:00:00+00, a tree size
of 0, the sha256 root hash was empty and the tree head signature was 0x00000000. I didn't
store the STH itself as I received it, so I'm not sure what they really send, but it passed
the json parsing.

This is all from ctlog-gen2.api.venafi.com.

Kurt

Rob Stradling

unread,

Aug 23, 2017, 6:07:08 AM8/23/17

to Kurt Roeckx, ct-p...@chromium.org

https://ct.grahamedgecombe.com/logs/30 reports some further instances
over the past few months of ctlog-gen2.api.venafi.com returning STHs
that are older than the MMD.

--
Rob Stradling
Senior Research & Development Scientist
COMODO - Creating Trust Online

Kat Joyce

unread,

Aug 23, 2017, 7:34:59 AM8/23/17

to Rob Stradling, ctlog...@venafi.com, Kurt Roeckx, Certificate Transparency Policy, Ryan Sleevi

Thank you to you two for raising this. Having dug into our data, we have also seen this behaviour from the Log at https://ctlog-gen2.api.venafi.com. Specifically:

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-07-22 01:55:19 | 1970-01-01 00:00:00 | 416857.922045 | tree_size: 0 |

| .365+00:00 | +00:00 | 83331 | timestamp: 0 |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-07-14 07:40:59 | 1970-01-01 00:00:00 | 416671.683226 | tree_size: 0 |

| .615+00:00 | +00:00 | 38888 | timestamp: 0 |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-07-14 03:38:42 | 2017-07-11 11:39:20 | 63.9894802777 | tree_size: 42353410 |

| .89+00:00 | .761+00:00 | 7778 | timestamp: 1499773160761 |

| | | | sha256_root_hash: "xVUBIdVkKtNSDKzl/l/SUH2VwAJ48l/5K6iMYchJ1HQ=" |

| | | | tree_head_signature: "BAMARjBEAiBaoxwu04smLW6JxBduXA+PV+xnRZm8wLJG16Id0hH |

| | | | uCAIgJMz0UTkWIFsBFZlhMl5EDdFgyQsmmlazVVfpx7/ab7g=" |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-07-14 03:36:30 | 2017-07-11 23:10:55 | 52.4266447222 | tree_size: 42589676 |

| .968+00:00 | .047+00:00 | 22221 | timestamp: 1499814655047 |

| | | | sha256_root_hash: "V7W4KW81WbNBTFqC7FzLWdOLGxUSrZIzACjJ0aigYi4=" |

| | | | tree_head_signature: "BAMARzBFAiBc6ClNRIS1GlvWpiY3X392C21H0ZzrcLwDwf0exOX |

| | | | 2TgIhAMov6gZIAChHXjYkFkW4f+mgRohQhogy8HtjZgWO3R1a" |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-06-24 05:17:43 | 1970-01-01 00:00:00 | 416189.295369 | tree_size: 0 |

| .329+00:00 | +00:00 | 16665 | timestamp: 0 |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-06-24 05:09:42 | 1970-01-01 00:00:00 | 416189.161763 | tree_size: 0 |

| .347+00:00 | +00:00 | 05554 | timestamp: 0 |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-06-22 20:02:17 | 1970-01-01 00:00:00 | 416156.03807 | tree_size: 0 |

| .052+00:00 | +00:00 | | timestamp: 0 |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-06-18 23:25:15 | 2017-06-03 17:37:11 | 365.801033888 | tree_size: 25620693 |

| .493+00:00 | .771+00:00 | 88887 | timestamp: 1496511431771 |

| | | | sha256_root_hash: "CriCvS4OnFPt4hHtDvJXgRYcJItZWM7kX7s8Vb6+a/0=" |

| | | | tree_head_signature: "BAMARzBFAiBsHYi2s5+U8o8yYE/ChzV0IrVvN0p1YWAgSE/TizC |

| | | | yzwIhALgqNrvHSlmO+HASeTR26Jn+YJRUB1B9aPPdULsogqeH" |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-06-18 23:23:15 | 2017-06-03 17:37:11 | 365.767606111 | tree_size: 25620693 |

| .153+00:00 | .771+00:00 | 11109 | timestamp: 1496511431771 |

| | | | sha256_root_hash: "CriCvS4OnFPt4hHtDvJXgRYcJItZWM7kX7s8Vb6+a/0=" |

| | | | tree_head_signature: "BAMARzBFAiBsHYi2s5+U8o8yYE/ChzV0IrVvN0p1YWAgSE/TizC |

| | | | yzwIhALgqNrvHSlmO+HASeTR26Jn+YJRUB1B9aPPdULsogqeH" |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-05-30 19:40:51 | 2017-05-24 22:06:03 | 141.580037777 | tree_size: 22245541                                                       |

| .236+00:00          | .1+00:00            | 77779         | timestamp: 1495663563100                                                  |

|                     |                     |               | sha256_root_hash: "b8GjZGYHgKZoC//QcQCwP328paqTDXrDo0SlF9MKP5k="          |

|                     |                     |               | tree_head_signature: "BAMASDBGAiEAzY6adfUQvdi5exLuhcvl4Gpz5YcDDvxONGOYidD |

|                     |                     |               | rEWYCIQCLyOpAdNDQbt90w7Na6CI9u2JfJlI8lKY0R4kIZFbJmA=="                    |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-04-22 23:00:20 | 2017-04-18 19:20:47 | 99.6592086111 | tree_size: 7577126                                                        |

| .907+00:00          | .756+00:00          | 11112         | timestamp: 1492543247756                                                  |

|                     |                     |               | sha256_root_hash: "glagN9qrhZI2hmrsBfEyxJ5Vq8bcfsF0B/hf3Sfv3AY="          |

|                     |                     |               | tree_head_signature: "BAMARzBFAiBzNGiRW+6R7WVpBvRKnykhxGU2iEs0tNKEL5JP/ii |

|                     |                     |               | jmAIhAPE9cbUZsnLp5Y4oDFd6k82iUpBTDWvb+boq+lXKn4Cu"                        |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-04-17 08:15:49 | 2017-04-09 03:08:43 | 197.1183125   | tree_size: 1898045                                                        |

| .354+00:00          | .429+00:00          |               | timestamp: 1491707323429                                                  |

|                     |                     |               | sha256_root_hash: "q8qOcgnWGBovMuaglLII9x91IUhOAbclreXAUswZhOA="          |

|                     |                     |               | tree_head_signature: "BAMASDBGAiEA8uSk59Dbumott4uSGgEmGMOGEzUgqT1ndBJYIM8 |

|                     |                     |               | aBUsCIQD6Ucl8/ir79AbT4I6BRwAEwnutbhvcciKJrphVW4uJWg=="                    |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-04-17 08:11:49 | 2017-04-09 03:08:43 | 197.051560277 | tree_size: 1898045                                                        |

| .046+00:00          | .429+00:00          | 77778         | timestamp: 1491707323429                                                  |

|                     |                     |               | sha256_root_hash: "q8qOcgnWGBovMuaglLII9x91IUhOAbclreXAUswZhOA="          |

|                     |                     |               | tree_head_signature: "BAMASDBGAiEA8uSk59Dbumott4uSGgEmGMOGEzUgqT1ndBJYIM8 |

|                     |                     |               | aBUsCIQD6Ucl8/ir79AbT4I6BRwAEwnutbhvcciKJrphVW4uJWg=="                    |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-04-17 08:07:48 | 2017-04-09 03:08:43 | 196.984800833 | tree_size: 1898045                                                        |

| .712+00:00          | .429+00:00          | 33334         | timestamp: 1491707323429                                                  |

|                     |                     |               | sha256_root_hash: "q8qOcgnWGBovMuaglLII9x91IUhOAbclreXAUswZhOA="          |

|                     |                     |               | tree_head_signature: "BAMASDBGAiEA8uSk59Dbumott4uSGgEmGMOGEzUgqT1ndBJYIM8 |

|                     |                     |               | aBUsCIQD6Ucl8/ir79AbT4I6BRwAEwnutbhvcciKJrphVW4uJWg=="                    |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-04-17 08:01:48 | 2017-04-09 03:08:43 | 196.884657222 | tree_size: 1898045                                                        |

| .195+00:00          | .429+00:00          | 22222         | timestamp: 1491707323429                                                  |

|                     |                     |               | sha256_root_hash: "q8qOcgnWGBovMuaglLII9x91IUhOAbclreXAUswZhOA="          |

|                     |                     |               | tree_head_signature: "BAMASDBGAiEA8uSk59Dbumott4uSGgEmGMOGEzUgqT1ndBJYIM8 |

|                     |                     |               | aBUsCIQD6Ucl8/ir79AbT4I6BRwAEwnutbhvcciKJrphVW4uJWg=="                    |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

| 2017-04-11 05:42:30 | 2017-04-09 09:32:07 | 44.1731763888 | tree_size: 2004442                                                        |

| .457+00:00          | .022+00:00          | 88891         | timestamp: 1491730327022                                                  |

|                     |                     |               | sha256_root_hash: "VJ0gbXAPV/pF4reB9ySb9+PRXTT+aTLiksok4UZdAdU="          |

|                     |                     |               | tree_head_signature: "BAMASDBGAiEA8Ko/qNBI/0pmK/asH0AHD0dqmEmkZc6uVHDM08c |

|                     |                     |               | N0qECIQDWBDBeDBLcQQyLqD5Ywgh1x8958lyAbc4+yUPDT0DjKA=="                    |

+---------------------+---------------------+---------------+---------------------------------------------------------------------------+

(TimestampDiff is in hours)

RFC 6962 states "Each log MUST produce on demand a Signed Tree Head that is no older than the Maximum Merge Delay", and the Chrome Log Policy states that "Log Operators must ... Conform to RFC 6962". So, I suppose we better begin a discussion about what the appropriate action (if any) should be taken as a result of this behaviour, and invite the operators of the Venafi Logs to comment on the behaviour that multiple parties have now seen from their Log.

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+unsubscribe@chromium.org.
To post to this group, send email to ct-p...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/cabb68c7-923d-d876-afcf-10f4b31626d9%40comodo.com.

Andrew Ayer

unread,

Aug 23, 2017, 2:19:31 PM8/23/17

to Kat Joyce, 'Kat Joyce' via Certificate Transparency Policy, Rob Stradling, ctlog...@venafi.com, Kurt Roeckx, Ryan Sleevi

On Wed, 23 Aug 2017 12:34:16 +0100
"'Kat Joyce' via Certificate Transparency Policy"

<ct-p...@chromium.org> wrote:

> RFC 6962 states "Each log MUST produce on demand a Signed Tree Head
> that is no older than the Maximum Merge Delay", and the Chrome Log
> Policy states that "Log Operators must ... Conform to RFC 6962". So,
> I suppose we better begin a discussion about what the appropriate
> action (if any) should be taken as a result of this behaviour, and
> invite the operators of the Venafi Logs to comment on the behaviour
> that multiple parties have now seen from their Log.

Although this is a violation of RFC 6962, the STHs are nevertheless
consistent, so this is a recoverable violation. Venafi can fix the
problem, and monitors can continue using the log as if nothing bad ever
happened.

As a point of comparison, Pilot and Aviator both logged certificates
with invalid signatures, in violation of RFC 6962. Even though this was
not a recoverable violation, Pilot and Aviator were not disqualified.

For these reasons, Venafi shouldn't be disqualified, although they
should fix the problem and provide an incident report.

Regards,
Andrew

Hari Nair

unread,

Aug 25, 2017, 12:46:37 AM8/25/17

to Certificate Transparency Policy, katj...@google.com, rob.st...@comodo.com, ctlog...@venafi.com, ku...@roeckx.be, sle...@google.com, ag...@andrewayer.name

Hello,

Here's a Venafi incident report as requested (also attached as PDF).

--------------

SUMMARY:

On August 23rd Venafi was notified that its CT log server https://ctlog-gen2.api.venafi.com had intermittently published several Signed Tree Heads (STHs) that exceeded the stated Maximum Merge Delay (MMD) of 24 hours. As a result of logging a higher than expected volume of certificates, Venafi's CT log cluster was impacted by individual server restarts, with increasing re-synchronization duration, over an extended period of time that in turn led intermittently to a distortion of the cluster's quorum resulting in the STH publication.

IMPACT:

Venafi violated the MMD requirement as specified in RFC 6962 section 3.5: "Each log MUST produce on demand a Signed Tree Head that is no older than the Maximum Merge Delay".

At no time did Venafi's log server publish STHs inconsistent with the underlying Merkle tree.

ROOT CAUSE:

With only minor differences, Venafi operates its CT log infrastructure based on Google's open source reference implementation (https://github.com/google/certificate-transparency). Therefore, an STH is only issued after a majority of instances in the cluster agrees on an STH's validity. As a result of the Venafi's CT log operating above its workload capacity because of a significantly higher than expected volume of certificates sporadic server outages and restarts with increasing database re-synchronization times led, in some cases, to a distorted cluster majority as witnessed by the publication of STHs with timestamp of 1970-01-01 00:00:00 and empty tree size. Also, intermittently, a majority of servers agreed on an outdated version of the log database leading to the witnessed publication of outdated STHs. See for example https://ct.grahamedgecombe.com/logs/30 for samples of published STHs violating the stated MMD.

REMEDIATION AND PREVENTION:

Venafi has changed its CT cluster auto-scaling management and is further investigating optimization of the server join time into the cluster. Also, Venafi is in the process of testing, with the intention to publish a pull request for the reference implementation, an improvement which prevents the publication of STHs outside the MMD.

-------------

We will post an update once testing has been done and we have confirmed the improvement to the reference implementation.

Thanks,

Hari Nair

Aug23_Venafi_CT_Incident_Report_Summary.pdf

Ryan Sleevi

unread,

Aug 28, 2017, 1:43:05 PM8/28/17

to Hari Nair, Certificate Transparency Policy, Kat Joyce, Rob Stradling, ctlog...@venafi.com, Kurt Roeckx, Andrew Ayer

Thanks for the update, Hari.

I believe it's in line with both the text and the spirit of the policy to consider this a policy violation, but one that does not result in Log distrust. As the CT ecosystem grows, we want to learn from the challenges Log Operators face, and use that as an opportunity to both improve Logs and improve the policies around Logs to account for evolving Best Practices.

On a technical level, I think that, should the testing and implementation work out, further technical details about the nature of the issue (and how the remediation addresses it) would be both useful and valuable to the community.

On the monitoring side, we'll be working to incorporate further improvements to our monitoring and alerting. Further, this provides an excellent opportunity for those that do monitor logs to explore collaboration on a set of technical criteria that both can and should be monitored, both with respect to RFC6962 and with respect to the stated policies.

On a policy level, I think that should there be ongoing/persistent issues, or multiple issues, it may be necessary to consider distrusting the Log, but as long as Log Operators are not fatally violating the contracts of a well-behaving log (such as inconsistent or split views), there opportunity to learn from these issues and improve the ecosystem outweighs the direct and indirect risks to users.

Does this seem like an appropriate conclusion for the community? What other expectations should we set or questions should we seek to have answered, as we look to apply this more generally. Further, should the policy be updated to clarify this, beyond its existing acknowledgement that policy violations do not necessarily result in distrust, but may?

Richard Salz

unread,

Aug 28, 2017, 1:48:06 PM8/28/17

to Ryan Sleevi, Hari Nair, Certificate Transparency Policy, Kat Joyce, Rob Stradling, ctlog...@venafi.com, Kurt Roeckx, Andrew Ayer

> Does this seem like an appropriate conclusion for the community?

Yes. As long as no inconsistent STH was published it should be okay. What you do about post-facto STH violations might be hard, I think.

Andrew Ayer

unread,

Aug 28, 2017, 2:08:49 PM8/28/17

to Certificate Transparency Policy

Yes, this conclusion seems appropriate to me. I agree that
ongoing/persistent/multiple issues should warrant distrust even if
the issues are recoverable. Also, as the ecosystem matures the bar
should be raised and fewer violations should be tolerated before a log
is distrusted, as log operators should have more experience by then,
and the need for log stability will begin to outweigh the value of yet
another incident report.

> What other expectations should we set or questions should we seek to have
> answered, as we look to apply this more generally. Further, should
> the policy be updated to clarify this, beyond its existing
> acknowledgement that policy violations do not necessarily result in
> distrust, but may?

I think it would be very useful for the policy to draw a distinction
between recoverable and irrecoverable errors, the latter of which
will almost certainly lead to distrust. A distinction would help log
operators prioritize. For example, if a log suffers a database meltdown,
the operator should prefer a careful restoration over a rushed restoration
that risks signing an inconsistent STH, even if it means the log is down
for longer.

Regards,
Andrew

Reply all

Reply to author

Forward